Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

code coverage suite #848

Open
wants to merge 47 commits into
base: devnet-ready
Choose a base branch
from
Open

code coverage suite #848

wants to merge 47 commits into from

Conversation

sam0x17
Copy link
Contributor

@sam0x17 sam0x17 commented Oct 9, 2024

adds code coverage suite to the repo

  • find and parse all pallets dynamically and in parallel
  • code to import missing sections of pallets defined with #[import_section(..)] / #[pallet_section]
  • collect metadata on all extrinsics/calls in all pallets, in parallel
  • collect metadata on free functions defined within pallets (non-calls)
  • find all tests in the repo in parallel
  • find all benchmark tests (v2 syntax) in the repo in parallel
  • upgrade existing benchmarking v1 tests to v2 (only one file, quick)
  • add a quick little lint disallowing benchmarking v1 syntax and enable this lint in build.rs
  • collect metadata and coverage info for hooks, might be impractical, happy path would be if people commonly enter hooks manually within tests, then you can detect tests that do this and expect one covering each hook state, possibly for each pallet
  • analyze tests and generate coverage statistics based on what functions are/are not mentioned across all tests
  • display code coverage results in the terminal from the build script (using sam's build-print crate)
  • by default don't run code coverage unless RUN_COVERAGE env var is present (from within build.rs)

future work:

  • take function complexity into account (number of distinct terminal points = needs more coverage)
  • use conventional code coverage tools for everything that isn't a pallet and then roll this information into the final report
  • once 100% coverage is achieved, have CI fail if coverage drops below 100% and flag the specific functions/extrinsics/whatever that are untested
  • take function call graph into account -- if function or extrinsic A calls function or extrinsic B, then coverage on A at least partially covers B

@sam0x17 sam0x17 added the blue team defensive programming, CI, etc label Oct 9, 2024
@sam0x17 sam0x17 self-assigned this Oct 9, 2024
@sam0x17 sam0x17 requested a review from unconst as a code owner October 9, 2024 16:42
@sam0x17 sam0x17 changed the title test coverage suite code coverage suite Oct 9, 2024
@sam0x17 sam0x17 assigned sam0x17 and keithtensor and unassigned sam0x17 Oct 9, 2024
@sam0x17 sam0x17 added the no-spec-version-bump PR does not contain changes that requires bumping the spec version label Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blue team defensive programming, CI, etc no-spec-version-bump PR does not contain changes that requires bumping the spec version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants