Coverage

[oli-obk]

Figure out a way to run gcov on miri, so we know if we aren't testing something or if some hack becomes obsolete

[cgettys-microsoft]

I think this is relatively doable nowadays, as a far bit of infrastructure in this area has stabilized:

Instrumentation-based Code Coverage

The LLVM coverage data in particular is pretty nice.

Might be a good first issue now?

[RalfJung]

Maybe not a good "first" issue, since I assume it will involve some fighting with the build system. But yeah seems worth promoting this a bit so maybe someone can look into it.

What is not clear at all to me is how to best get this into CI. Ideally we'd somehow know when a line that used to be covered is no longer covered, but that requires keeping state across CI runs... not sure how that would work / what people usually do here.

[Mandragorian]

I would be interested in trying this. I would need to do some research first though. I haven't done anything similar before.

[RalfJung]

Yeah, definitely -- we'd want to know what is even possible in this space and what the final result would look like before investing into any kind of polished solution.

[Mandragorian]

I finally found some time to work on this. Just by adding RUSTFLAGS="-C instrument-coverage" in the ./miri test command:

[mandragore@archibald miri]$ git diff
diff --git a/miri-script/src/util.rs b/miri-script/src/util.rs
index f5a6a8188..45f587dfa 100644
--- a/miri-script/src/util.rs
+++ b/miri-script/src/util.rs
@@ -188,7 +188,7 @@ impl MiriEnv {
     }
 
     pub fn test(&self, crate_dir: impl AsRef<OsStr>, args: &[String]) -> Result<()> {
-        self.cargo_cmd(crate_dir, "test").args(args).run()?;
+        self.cargo_cmd(crate_dir, "test").args(args).env("RUSTFLAGS", "-C instrument-coverage").run()?;
         Ok(())
     }

I was able to generate the profraw files. Then after install cargo-binutils and running:

[root@e4aaf2fc8c9d miri]# cargo-profdata -- merge -sparse default_*.profraw  -o cooked.profdata
[root@e4aaf2fc8c9d miri]# cargo-cov -- report --instr-profile=cooked.profdata --object target/debug/miri  -ignore-filename-regex="root/.cargo/registry/*|root/.rustup/*|/rust/deps/*|target/*"

(the root in the regex above is because I use a container to run tests)

I got this report:

Filename                                          Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
alloc_addresses/mod.rs                                147                16    89.12%          24                 4    83.33%         303                 7    97.69%           0                 0         -
alloc_addresses/reuse_pool.rs                          44                 0   100.00%           6                 0   100.00%          93                 0   100.00%           0                 0         -
alloc_bytes.rs                                         43                 8    81.40%          15                 4    73.33%          70                 6    91.43%           0                 0         -
bin/miri.rs                                           331               115    65.26%          37                19    48.65%         523               164    68.64%           0                 0         -
borrow_tracker/mod.rs                                 123                10    91.87%          37                 1    97.30%         288                15    94.79%           0                 0         -
borrow_tracker/stacked_borrows/diagnostics.rs         176                18    89.77%          42                 1    97.62%         342                27    92.11%           0                 0         -
borrow_tracker/stacked_borrows/item.rs                 26                 1    96.15%           8                 0   100.00%          46                 1    97.83%           0                 0         -
borrow_tracker/stacked_borrows/mod.rs                 406                37    90.89%          44                 1    97.73%         666                41    93.84%           0                 0         -
borrow_tracker/stacked_borrows/stack.rs               157                 2    98.73%          18                 0   100.00%         270                 1    99.63%           0                 0         -
borrow_tracker/tree_borrows/diagnostics.rs            193                12    93.78%          33                 1    96.97%         382                18    95.29%           0                 0         -
borrow_tracker/tree_borrows/mod.rs                    221                37    83.26%          28                 1    96.43%         410                29    92.93%           0                 0         -
borrow_tracker/tree_borrows/perms.rs                  148                31    79.05%          34                 5    85.29%         227                40    82.38%           0                 0         -
borrow_tracker/tree_borrows/tree.rs                   224                30    86.61%          48                 5    89.58%         521                30    94.24%           0                 0         -
borrow_tracker/tree_borrows/unimap.rs                  60                23    61.67%          28                11    60.71%         129                51    60.47%           0                 0         -
clock.rs                                               37                 1    97.30%          10                 0   100.00%          72                 1    98.61%           0                 0         -
concurrency/cpu_affinity.rs                            23                 6    73.91%           7                 0   100.00%          54                 9    83.33%           0                 0         -
concurrency/data_race.rs                              622                69    88.91%         102                 1    99.02%        1106                24    97.83%           0                 0         -
concurrency/init_once.rs                               47                47     0.00%          10                10     0.00%          77                77     0.00%           0                 0         -
concurrency/range_object_map.rs                        38                 1    97.37%          11                 0   100.00%          61                 1    98.36%           0                 0         -
concurrency/sync.rs                                   258                46    82.17%          56                 5    91.07%         559                20    96.42%           0                 0         -
concurrency/thread.rs                                 377                59    84.35%          97                14    85.57%         695                78    88.78%           0                 0         -
concurrency/vector_clock.rs                           119                57    52.10%          33                 6    81.82%         234               101    56.84%           0                 0         -
concurrency/weak_memory.rs                            184                20    89.13%          23                 0   100.00%         336                 8    97.62%           0                 0         -
diagnostics.rs                                        264                27    89.77%          18                 2    88.89%         487                42    91.38%           0                 0         -
eval.rs                                               159                47    70.44%          10                 2    80.00%         315                47    85.08%           0                 0         -
helpers.rs                                            547               145    73.49%         104                24    76.92%         849               155    81.74%           0                 0         -
intrinsics/atomic.rs                                  219                53    75.80%          14                 0   100.00%         215                10    95.35%           0                 0         -
intrinsics/mod.rs                                     499               131    73.75%           4                 0   100.00%         315                14    95.56%           0                 0         -
intrinsics/simd.rs                                    910               242    73.41%           6                 0   100.00%         590                56    90.51%           0                 0         -
machine.rs                                            503                98    80.52%          89                 7    92.13%        1005               103    89.75%           0                 0         -
mono_hash_map.rs                                       40                 4    90.00%          14                 2    85.71%          60                 8    86.67%           0                 0         -
operator.rs                                            74                15    79.73%           8                 0   100.00%          85                 4    95.29%           0                 0         -
provenance_gc.rs                                       89                14    84.27%          27                 6    77.78%         141                22    84.40%           0                 0         -
range_map.rs                                           80                14    82.50%          14                 3    78.57%         147                21    85.71%           0                 0         -
shims/alloc.rs                                        102                20    80.39%           8                 0   100.00%         110                 1    99.09%           0                 0         -
shims/backtrace.rs                                    156                48    69.23%           6                 0   100.00%         167                 9    94.61%           0                 0         -
shims/env.rs                                           61                21    65.57%          10                 2    80.00%          72                21    70.83%           0                 0         -
shims/extern_static.rs                                 54                26    51.85%           4                 0   100.00%          63                11    82.54%           0                 0         -
shims/foreign_items.rs                                911               300    67.07%          18                 3    83.33%         688               114    83.43%           0                 0         -
shims/io_error.rs                                      96                43    55.21%          15                 3    80.00%         110                32    70.91%           0                 0         -
shims/native_lib.rs                                    99                43    56.57%           6                 0   100.00%         117                34    70.94%           0                 0         -
shims/os_str.rs                                       115                90    21.74%          24                16    33.33%         234               160    31.62%           0                 0         -
shims/panic.rs                                        102                38    62.75%           7                 0   100.00%         139                16    88.49%           0                 0         -
shims/time.rs                                         176               106    39.77%          16                11    31.25%         289               136    52.94%           0                 0         -
shims/tls.rs                                          145                77    46.90%          16                 7    56.25%         213                99    53.52%           0                 0         -
shims/unix/android/foreign_items.rs                    12                12     0.00%           2                 2     0.00%          20                20     0.00%           0                 0         -
shims/unix/env.rs                                     203                56    72.41%          17                 1    94.12%         207                15    92.75%           0                 0         -
shims/unix/fd.rs                                      264                93    64.77%          53                 9    83.02%         450               105    76.67%           0                 0         -
shims/unix/foreign_items.rs                          1342               593    55.81%           8                 3    62.50%         677               116    82.87%           0                 0         -
shims/unix/freebsd/foreign_items.rs                    81                81     0.00%           2                 2     0.00%          61                61     0.00%           0                 0         -
shims/unix/fs.rs                                     1017               562    44.74%          71                29    59.15%        1252               504    59.74%           0                 0         -
shims/unix/linux/epoll.rs                             275                73    73.45%          24                 4    83.33%         418                39    90.67%           0                 0         -
shims/unix/linux/eventfd.rs                            65                16    75.38%           6                 1    83.33%         135                 6    95.56%           0                 0         -
shims/unix/linux/foreign_items.rs                     192                69    64.06%           2                 0   100.00%         128                12    90.62%           0                 0         -
shims/unix/linux/mem.rs                                38                11    71.05%           1                 0   100.00%          49                 2    95.92%           0                 0         -
shims/unix/linux/sync.rs                              123                47    61.79%           1                 0   100.00%         126                30    76.19%           0                 0         -
shims/unix/macos/foreign_items.rs                     249               249     0.00%           2                 2     0.00%         149               149     0.00%           0                 0         -
shims/unix/macos/sync.rs                               58                58     0.00%           7                 7     0.00%          68                68     0.00%           0                 0         -
shims/unix/mem.rs                                      88                33    62.50%           2                 0   100.00%         119                20    83.19%           0                 0         -
shims/unix/solarish/foreign_items.rs                   85                85     0.00%           2                 2     0.00%          74                74     0.00%           0                 0         -
shims/unix/sync.rs                                    476               128    73.11%          44                 0   100.00%         576                28    95.14%           0                 0         -
shims/unix/thread.rs                                   74                19    74.32%           7                 0   100.00%          88                 1    98.86%           0                 0         -
shims/unix/unnamed_socket.rs                          144                42    70.83%           9                 1    88.89%         264                36    86.36%           0                 0         -
shims/wasi/foreign_items.rs                            22                22     0.00%           2                 2     0.00%          26                26     0.00%           0                 0         -
shims/windows/env.rs                                  147               147     0.00%          11                11     0.00%         172               172     0.00%           0                 0         -
shims/windows/foreign_items.rs                        773               773     0.00%           6                 6     0.00%         538               538     0.00%           0                 0         -
shims/windows/handle.rs                                54                54     0.00%          12                12     0.00%         100               100     0.00%           0                 0         -
shims/windows/sync.rs                                 139               139     0.00%           7                 7     0.00%         159               159     0.00%           0                 0         -
shims/windows/thread.rs                                56                56     0.00%           2                 2     0.00%          58                58     0.00%           0                 0         -
shims/x86/aesni.rs                                     99                28    71.72%           6                 0   100.00%         120                 1    99.17%           0                 0         -
shims/x86/avx.rs                                      330                86    73.94%           1                 0   100.00%         182                12    93.41%           0                 0         -
shims/x86/avx2.rs                                     481               144    70.06%           1                 0   100.00%         241                13    94.61%           0                 0         -
shims/x86/bmi.rs                                       64                11    82.81%           1                 0   100.00%          70                 2    97.14%           0                 0         -
shims/x86/gfni.rs                                     131                33    74.81%           3                 0   100.00%         105                 1    99.05%           0                 0         -
shims/x86/mod.rs                                     1072               309    71.18%          39                 1    97.44%         814                28    96.56%           0                 0         -
shims/x86/sha.rs                                      120                30    75.00%          13                 0   100.00%         135                 1    99.26%           0                 0         -
shims/x86/sse.rs                                      198                52    73.74%           2                 0   100.00%         114                 7    93.86%           0                 0         -
shims/x86/sse2.rs                                     334                91    72.75%           2                 0   100.00%         193                 8    95.85%           0                 0         -
shims/x86/sse3.rs                                      34                 9    73.53%           1                 0   100.00%          31                 2    93.55%           0                 0         -
shims/x86/sse41.rs                                    142                44    69.01%           1                 0   100.00%          97                 3    96.91%           0                 0         -
shims/x86/sse42.rs                                    401               115    71.32%           7                 0   100.00%         249                14    94.38%           0                 0         -
shims/x86/ssse3.rs                                    132                43    67.42%           1                 0   100.00%          76                 4    94.74%           0                 0         -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                               18920              6661    64.79%        1567               281    82.07%       22216              4299    80.65%           0                 0         -

I guess we can create a coverage command for the miri script to automate the above? Or a flag in the test command?

[RalfJung]

That's cool!

Yeah ./miri test --coverage sounds good to me.

[Mandragorian]

I created this draft PR #3954

It requires that cargo-binutils is installed to generated the reports. It's so that we use the correct llvm version, compatible with what rustc is using. There is also llvm_profparser that tarpaulin is using. Maybe we can use that if we want to generate more complicated reports. But as a first step maybe the PR is good enough as it is.