/p:MergeWith does not merge but adds

[pape77]

I'm testing using coverlet over a solution that has 3 test projects which i want to combine into a single .opencover.xml file

Since there's currently no direct solution for this. What I tried to do is generate the coverage for one of the test projects, output its result into an "converage.json" file and then apply a MergeWith param in the following dotnet test commands to merge into that same json file and set the output format to opencover

Problem is, that since the first project(ProjectName.Api) includes other two as references (ProjectName.Application, ProjectName.Data), the report is generated as following:

Test run for /build/test/ProjectName.Api.Tests/bin/Debug/netcoreapp2.0/ProjectName.Api.Tests.dll(.NETCoreApp,Version=v2.0)

Results File: /results/_8e5df6e97674_2018-09-28_11_10_36.trx

Total tests: 15. Passed: 15. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 2.4645 Seconds

Calculating coverage result...
  Generating report '/results/coverage.json'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
|ProjectName.Api         | 24.4%  | 28.6%  | 33.3%  |
+----------------------+--------+--------+--------+
| ProjectName.Data        | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+
|ProjectName.Application | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+

So it includes two other coverages for the referenced projects with 0% coverage.
This wouldn't be a problem if /p:MergeWith param actually did a proper merge.

But when I run the next dotnet test (over the ProjectName.Application project) with the MergeWith param pointing to this recently generated coverage.json file, I get this:

Test run for /build/test/ProjectName.Application.Tests/bin/Debug/netcoreapp2.0/ProjectName.Application.Tests.dll(.NETCoreApp,Version=v2.0)

Starting test execution, please wait...
Results File: /results/_5d6c3e7d1c85_2018-09-28_11_10_47.trx

Total tests: 4. Passed: 4. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.7143 Seconds

Calculating coverage result...
  Generating report '/results/coverage.json'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
| ProjectName.Data        | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+
| ProjectName.Application | 100%   | 60%    | 100%   |
+----------------------+--------+--------+--------+
| ProjectName.Api         | 24.4%  | 28.6%  | 33.3%  |
+----------------------+--------+--------+--------+
| ProjectName.Data        | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+
| ProjectName.Application | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+

So it just adds the recent coverage to the existing ones in coverage.json. Causing a duplicated coverage in ProjectName.Application and ProjectName.Data (because again, ProjectName.Application references ProjectName.Data, so it adds a coverage 0% report)

Any solutions to this? Or am I missusing something?

Or even a way for coverlet for avoid including referenced project coverage?

[pape77]

So my final expectation is to run this:

dotnet test ./tests/ProjectName.Api.Tests/ProjectName.Api.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage

 dotnet test ./${SolutionName} -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

 dotnet test ./${SolutionName} -l trx -r /results /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

Which is equivalent to run dotnet test on each proj:

dotnet test ./tests/ProjectName.Api.Tests/ProjectName.Api.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage

 dotnet test ./tests/ProjectName.Application.Tests/ProjectName.Application.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

 dotnet test ./tests/ProjectName.Data.Tests/ProjectName.Application.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

 dotnet test ./${SolutionName} -l trx -r /results /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

[pape77]

Update: @tonerdo Looking into the Merging code 45bc5bc

I see you merge by Module key, then by Document key, etc.
In this case the Module key will be different since the dll is referenced from different paths

So the coverage.json looks like:

"/build/test/ProjectName.Api.Tests/bin/Debug/netcoreapp2.0/ProjectName.Application.dll": {
    "/build/src/ProjectName.Application/Services/Provider.cs": {
      "ProjectName.Application.Services.Provider": {
                 "System.Void ProjectName.Application.Services.KeyProvider::.ctor(ProjectName.Data.Repositories.Contracts.IProjectRepository,Microsoft.Extensions.Caching.Memory.IMemoryCache)": {
          "Lines": {
            "18": 0,
            "19": 0,
            "20": 0,
            "21": 0,
            "22": 0
          },
          "Branches": []
        }
      }
  ...
 }
},

...

"/build/test/ProjectName.Application.Tests/bin/Debug/netcoreapp2.0/ProjectName.Application.dll": {
    "/build/src/ProjectName.Application/Services/Provider.cs": {
      "ProjectName.Application.Services.Provider": {
        "System.Void ProjectName.Application.Services.KeyProvider::.ctor(ProjectName.Data.Repositories.Contracts.IProjectRepository,Microsoft.Extensions.Caching.Memory.IMemoryCache)": {
          "Lines": {
            "18": 4,
            "19": 4,
            "20": 4,
            "21": 4,
            "22": 4
          },
          "Branches": []
        }
      }
   ...
  }
}

So module keys are different but they reference the same dll file. This is why they are just added and not merged.

Any chance it can look into same dll names for merging instead of the whole key? (via Path.GetFileName perhaps?)

[pape77]

Pr created for this #204

[tonven]

When this will be pushed to coverlet.msbuild nuget package?

[pape77]

@Tonvengo Hopefully after #211 is merged! haha (otherwise the mergeWith will break)

Hopefully @tonerdo sees this first :)

[pape77]

I will reopen until it's there on the pkg

[tonven]

:) nice

Great, thanks for the feature, hopefully it gets a release soon.

However, I'm wondering, wouldn't it be an even better solution to extend this feature into some kind of /p:AggregateResultsAt that would create the first file if it doesn't exist, append all other results to it, and then finally convert it to other formats if required.

In my use case (VSTS build with multiple test projects), I could have a single step that would look like this:

dotnet test **/*[Tt]ests/*.csproj
/p:CollectCoverage=true
/p:AggregateResultsAt=/somepath/
/p:CoverletOutputFormat=opencover,cobertura

Thank you guys for your work!

[pape77]

@vlef without doubts. That was my next step, because if the file doesn't exist it fails. And it's annoying because when running a dotnet test on a solution you want it to also generate the file for the first test run.

I haven't got the time to look into that TBH. But maybe @tonerdo can take a look when he is free. Or if you find it you can also make a pr for it :)

Cheers

[tonerdo]

Definitely sounds like a good idea @vlef @pape77. For starters I could just make the MergeWith parameter ignore when the specified file doesn't exist and just assume it's on a first run