Add option to set threshold only for total coverage

[Jeky-v]

Add option to set threshold only for total coverage.
So run will be succeeded when total coverage more then specified threshold, even if some of projects have coverage 0, or below specified threshold.
Something like
dotnet test /p:CollectCoverage=true /p:TotalThreshold=80

[ovargas]

This could be pretty cool, but what about specifying the threshold you expect by type, something like:

dotnet test /p:CollectCoverage=true /p:Threshold=80,90 /p:ThresholdType=line,branch

From what I saw in the usage, to get the those warnings you have to execute the command twice.

[vantheshark]

I'm running 2 tests assemblies and merge the result for the 2nd. It'd be nice to have the threshold on a particular module or summary threshold

[maikschoeneich]

I would also love to see a threshold type total-line or something like this. So I can fail on an overall coverage.

[maikschoeneich]

With my change we have the first step into the right direction I think:

@vanthoainguyen: You can either fail on a specific module as well as on the total coverage.
@ovargas: you can specify the total type which should fail

[tonerdo]

@maikschoeneich @Jeky-v @ovargas @vanthoainguyen what do you think about adding a ThresholdStat option? It's values could be Average|Minimum|Maximum|Total

Minimum => The module with the lowest coverage must be greater than or equal to the specified threshold
Maximum => The module with the highest coverage must be greater than or equal to the specified threshold
Average => The average coverage of all the modules must be greater than or equal to the specified threshold
Total => The total coverage of all the modules must be greater than or equal to the specified threshold

This will as usual take the specified ThresholdType into consideration and we can easily add more use cases people might need in the future without having to add a new option each time

What do you guys think?

[vantheshark]

Hi @tonerdo ,
I believe most people will run coverage for test assemblies which somehow are related to each other.
For example the solution with below projects:
Project.Core
Project.Service
Project.Core.Tests
Project.Service.Tests

Assume the Project.Service.dll reference the Project.Core.dll, the coverage for the service test assembly will likely cover some of the part of the Project.Core.dll. Therefore, I believe the threshold on total coverage will be really handy as It will fail the build once the coverage drop no matter what changes you make to the any dlls.

Apparently threshold on each assemblies would be awesome but I don't think it's as important as threshold on the total coverage which should be easier to implement.

Regarding your question, I personally don't like implicit/assumption, I'd prefer to specify the module name, but it's just me :) so I would not do the Maximum and Minimum. However, the Average and Total could be helpful. Again, I don't know how you will do it but the threshold on the summary (total or average) is clearly a demand feature :P as I simply want to fail the build when some dudes don't write unit tests :)

[tonerdo]

Thanks for the feedback @vanthoainguyen, the first implementation of this feature can start with just Average and Total and leave room for adding new ones in the future

[tonerdo]

You should be able to use the newly added --threshold-stat op /p:ThresholdStat to specify how you'd like the threshold comparison to happen. Valid values are minimum, average, total. Available in the latest releases:

• https://www.nuget.org/packages/coverlet.msbuild/2.5.0
• https://www.nuget.org/packages/coverlet.console/1.4.0