Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tooling for generating coverage #293

Open
maxammann opened this issue Oct 18, 2023 · 6 comments · May be fixed by #296
Open

Add tooling for generating coverage #293

maxammann opened this issue Oct 18, 2023 · 6 comments · May be fixed by #296

Comments

@maxammann
Copy link
Contributor

It would be great to have tooling like "cargo fuzz coverage ", that creates a binary with coverage instrumentation, then runs it on the corpus and generates an LLVM .profdata file.

This can then be used to generate an HTML report using:

cargo cov -- show fuzz/target/<target triple>/release/my_compiler \
    --format=html \
    -instr-profile=fuzz/coverage/my_compiler/coverage.profdata \
    > index.html

From my perspective this is maybe one of the last key things that are missing from test-fuzz that are available in cargo-fuzz.

@smoelius
Copy link
Collaborator

I think it should be not-to-hard to implement something like this.

cargo-test-fuzz already has --replay corpus, replay-queue, etc. Moreover, the way those options are implemented is by running a test binary. I think we could repurpose that code to run the test binary under cargo-llvm-cov. So we could then have --coverage corpus, --coverage queue, etc.

We would have to insist that the user has installed cargo-llvm-cov, but we already do that for cargo-afl, so it's probably okay.

One thing I am uncertain about is: what is the best interface for specifying the toolchain? My understanding is that cargo-llvm-cov requires a nightly compiler.

One approach could be to check whether the current rustc is nightly (using a trick like this). If it's not, then print a warning like "the currently installed nightly toolchain will be used," and go with that. But I am open to other ideas.

Please let me know if what I've written is significantly different from what you were imagining.

@maxammann
Copy link
Contributor Author

I think we could repurpose that code to run the test binary under cargo-llvm-cov. So we could then have --coverage corpus, --coverage queue, etc.

That sounds good!

We would have to insist that the user has installed cargo-llvm-cov, but we already do that for cargo-afl, so it's probably okay.

I also think that is fine. We just have to pass the essential options along. cargo-fuzz currently just generates a profdata file and then you are supposed to work this that. I think this may be preferable over actually generating HTML reports, so users still have control.
Supporting every feature of llvm-cov is probably hard to maintain in the long run.

One thing I am uncertain about is: what is the best interface for specifying the toolchain? My understanding is that cargo-llvm-cov requires a nightly compiler.

I will test that. Something was stabilized around ~1.60 with regards to coverage.

If it's not, then print a warning like "the currently installed nightly toolchain will be used," and go with that.

cargo-fuzz also requires nightly. So maybe we can check how they do it.


I belive a major blocker with the approach you mentinoed is that LLVM will overwrite the profile data. So if you rerun a binary, all the previous data is lost. So either you collect many files and then combine them or run all tests in one single binary. I feel like the first approach works better and only relies on some temporary storage.

@smoelius
Copy link
Collaborator

So either you collect many files and then combine them...

I don't know how cargo-llvm-cov does it, but I think it has something like this "built in": https://github.com/taiki-e/cargo-llvm-cov#merge-coverages-generated-under-different-test-conditions

But your overall point that this idea requires further investigation is certainly on the mark.

@maxammann
Copy link
Contributor Author

maxammann commented Oct 19, 2023

I just experimented a little bit with cargo-llvm-cov and it seems to require a quite default layout of the target/ directory. At least that is the case for binary target like they are used by cargo-fuzz.

With cargo fuzz the target directory is in fuzz/target. Binaries in there are named after the target, e.g. target_1.

I got it working using the following environment:

export RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target'
export CARGO_LLVM_COV="1"
export CARGO_LLVM_COV_SHOW_ENV="1"
export CARGO_LLVM_COV_TARGET_DIR="/project/fuzz/target/"
export LLVM_PROFILE_FILE="$CARGO_LLVM_COV_TARGET_DIR/fuzz-%p-%8m.profraw"

Then I did from the fuzz/ directory:

cargo +nightly fuzz run fuzz_target_1 -- -runs=0 corpus/fuzz_target_1
cargo llvm-cov report --profile release --target x86_64-unknown-linux-gnu

I'm continuing now to see how to integrate it into test-fuzz.

@smoelius
Copy link
Collaborator

🙌

@maxammann
Copy link
Contributor Author

maxammann commented Oct 19, 2023

Also got it working 🎉

  1. cargo test-fuzz
  2. Export:
export CARGO_LLVM_COV="1"
export CARGO_LLVM_COV_SHOW_ENV="1"
export CARGO_LLVM_COV_TARGET_DIR="/project/target"
export LLVM_PROFILE_FILE="/project/target/$project_dir-%p-%8m.profraw"
export RUSTFLAGS="-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target"
  1. cargo test-fuzz --no-instrumentation --replay corpus
  2. cargo llvm-cov report --profile debug --html --hide-instantiations
  3. rm target/*.profraw

The report looks slightly weird though:

image

@maxammann maxammann linked a pull request Oct 20, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants