Request Delegate Generator code counts as user code for code coverage

[martincostello]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

If <EnableRequestDelegateGenerator>true</EnableRequestDelegateGenerator> is enabled for a project using a minimal code coverage metric, such as with Coverlet's Threshold setting, the generated code counts towards the user's code.

This can then cause branch coverage metrics to fail due to the additional error handling etc. that is generated for the user that should be considered to be effectively "framework code".

Expected Behavior

The source-generated code should be marked with [CompilerGenerated]/[ExcludeFromCodeCoverage] or similar so that existing code coverage tools' heuristics to ignore non-user code does not "penalise" the user for uncovered branches of the emitted code around their request delegate(s) that they did not themselves write.

Steps To Reproduce

1. Clone martincostello/website@94c0f14
2. Run build.ps1

Observe that the build fails with the following error:

Passed!  - Failed:     0, Passed:    92, Skipped:     0, Total:    92, Duration: 1 m 23 s - Website.Tests.dll (net8.0)

Calculating coverage result...
  Generating report 'C:\Coding\martincostello\website\artifacts\coverage\coverage.cobertura.xml'
  Generating report 'C:\Coding\martincostello\website\artifacts\coverage\coverage.json'

+---------+--------+--------+--------+
| Module  | Line   | Branch | Method |
+---------+--------+--------+--------+
| Website | 78.14% | 56.33% | 89.24% |
+---------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 78.14% | 56.33% | 89.24% |
+---------+--------+--------+--------+
| Average | 78.14% | 56.33% | 89.24% |
+---------+--------+--------+--------+

C:\Users\Martin.Costello\.nuget\packages\coverlet.msbuild\3.2.0\build\coverlet.msbuild.targets(71,5): error : The minimum branch coverage is below the specified 62 [C:\Coding\martincostello\website\tests\Website.Tests\Website.Tests.csproj]
C:\Users\Martin.Costello\.nuget\packages\coverlet.msbuild\3.2.0\build\coverlet.msbuild.targets(71,5): error :  [C:\Coding\martincostello\website\tests\Website.Tests\Website.Tests.csproj]
C:\Users\Martin.Costello\.nuget\packages\coverlet.msbuild\3.2.0\build\coverlet.msbuild.targets(71,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in /_/src/coverlet.msbuild.tasks/CoverageResultTask.cs:line 250 [C:\Coding\martincostello\website\tests\Website.Tests\Website.Tests.csproj]
Exception: C:\Coding\martincostello\website\build.ps1:82

With the request delegate generator disabled, the coverage is as below:

Passed!  - Failed:     0, Passed:    92, Skipped:     0, Total:    92, Duration: 30 s - Website.Tests.dll (net8.0)

Calculating coverage result...
  Generating report '/home/runner/work/website/website/artifacts/coverage/coverage.cobertura.xml'
  Generating report '/home/runner/work/website/website/artifacts/coverage/coverage.json'

+---------+--------+--------+--------+
| Module  | Line   | Branch | Method |
+---------+--------+--------+--------+
| Website | 93.02% | 69.91% | 96.18% |
+---------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 93.02% | 69.91% | 96.18% |
+---------+--------+--------+--------+
| Average | 93.02% | 69.91% | 96.18% |
+---------+--------+--------+--------+

Exceptions (if any)

No response

.NET Version

8.0.100-preview.4.23260.5

Anything else?

No response

[martincostello]

I'm going to have a quick look at pushing up a PR to fix this.

[captainsafia]

Thanks for filing this issue! I noticed that in the PR you linked the suspect code blogs are classes that are marked as file private (see here). It doesn't seem right to me that code coverage tools would detect types with this scope, although I'm not an expert on how these particular implementations work.

If the type is marked file private and the file is marked generated then it feels sufficient to omit them from code coverage. My motivation here is to reduce the amount of generated code attributes that we have to emit in the output source.

Any thoughts on why code coverage behaves this way with file-scoped types?

[martincostello]

I think that tools like coverlet just "absorb" all of the compiled code in the user's assembly and then use attributes and/or name Regexes to decide whether they should ignore stuff, rather than inspecting the source code itself for things like the file private modifier.

I'll tweak my setup locally and see what gets emitted into my HTML coverage reports that's "extra" that shouldn't be included.

If there's something in the compiled code to detect this sort of thing, then maybe this is actually an issue with coverlet and it should be looking for and self-excluding anything that heuristic would detect.

[mitchdenny]

The fact that it is file private doesn't matter. Code coverage tools will just report on whether the code is visited by the test and if is marks it as covered. We will need to add the generated code attributes.

[martincostello]

For completeness, this is what I get in my HTML coverage reports: