Can't get code coverage working with VS2019

[bgrainger]

I'm interested in contributing tests to increase code coverage (for one or more of these issues: https://github.com/dotnet/corefx/labels/increase-code-coverage).

I've set up a Windows 10 development machine with VS2019 Preview, as per https://github.com/dotnet/corefx/wiki/Development-environment-setup-for-Windows.

In src\System.Data.Common\tests I can execute dotnet msbuild /t:BuildAndTest /p:ForceRunTests=true /p:ConfigurationGroup=Release and run all the tests. But if I add /p:Coverage=true (following the instructions at https://github.com/dotnet/corefx/blob/master/Documentation/building/code-coverage.md), it fails with the following message:

  ----- start 18:00:22.66 ===============  To repro directly: =====================================================
  pushd C:\Code\dotnet\corefx\artifacts\bin\tests\System.Data.Common.Tests\netcoreapp-Windows_NT-Release-x64\
  "C:\Code\dotnet\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\dotnet.exe" tool run coverlet "System.Data.Common.Tests.dll" --target ""C:\Code\dotnet\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\dotnet.exe"" --targetargs "xunit.console.dll System.Data.Common.Tests.dll -xml testResults.xml -nologo -notrait category=nonnetcoreapptests -notrait category=nonwindowstests -notrait category=OuterLoop -notrait category=failing" --format "opencover" --output "coverage.xml" --threshold "0" --threshold-type "line,branch,method" --use-source-link --exclude-by-file "C:\Code\dotnet\corefx\src\Common\src\System\SR.*" --exclude-by-file "C:\Code\dotnet\corefx\src\Common\src\System\NotImplemented.cs" --include-directory "C:\Code\dotnet\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9" --include "[System.Data.Common]*"
  "C:\Code\dotnet\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\dotnet.exe" tool run reportgenerator "-reports:coverage.xml" "-targetdir:C:\Code\dotnet\corefx\artifacts\bin\tests\System.Data.Common.Tests\netcoreapp-Windows_NT-Release-x64\report" "-reporttypes:Html" "-verbosity:Info"
  popd
  ===========================================================================================================
    It was not possible to find any installed dotnet SDKs
    Did you mean to run dotnet SDK commands? Please install dotnet SDK from:
        https://aka.ms/dotnet-download
    It was not possible to find any installed dotnet SDKs
    Did you mean to run dotnet SDK commands? Please install dotnet SDK from:
        https://aka.ms/dotnet-download
  ----- end 18:00:22.69 ----- exit code -2147450735 ----------------------------------------------------------

I note that the report URL in the coverage page is not currently working: https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/Code_Coverage_Report/

Also, there are some issues that suggest that the coverage infrastructure may be getting overhauled: #960, #29411. This makes me unsure if I should expect coverage to be working or not.

Is it possible to run coverage locally? If so, is there something I need to change in my setup to get it working?

[bgrainger]

The problem appears to be related to set DOTNET_MULTILEVEL_LOOKUP=0 in RunTests.cmd.

With this present, the "It was not possible to find any installed dotnet SDKs" error message is displayed.

With that environment variable unset, dotnet tool instead prints an error message that dotnet tool run is not a known command. This appears to be because my dotnet program, "C:\Code\dotnet\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\dotnet.exe" has version 2.2.300.

[bgrainger]

If install .NET Core SDK 3.0.100-preview5-011568, then run the individual commands from RunTests.cmd, without the environment variables, I can get a coverage report.

This unblocks me for testing coverage locally, but the original issue of the command in the documentation (dotnet msbuild /p:Coverage=true) not working is still a problem.

[ahsonkhan]

@ViktorHofer has been working on getting this issue resolved. Thanks for reporting it.

cc @ericstj

[ViktorHofer]

The fix for this is in arcade now and will be consumable in corefx with the next auto update.

[ericstj]

@ViktorHofer was this fixed by dotnet/corefx#38200?

[ViktorHofer]

This should already be fixed now.

[ericstj]

/cc @buyaa-n @ahsonkhan

[ViktorHofer]

correct.

[bgrainger]

Confirmed this is fixed; thanks!