Code Coverage Result Calculation takes longer time

[praneeth820903]

Hi,
For our project using Coverlet 2.0 to get the code coverage results, thank you for such a nice Library.
Between calculating coverage results and generating the report it takes approximately 6 minutes on average. Our average build time has gone up from 3 minutes to 11+ minutes. Is there any way in reducing the Calculating Code Coverage and Report Generation time. With 1.0 we did not have this problem, the overall time for unit tests and code coverage was not more than 3 minutes.

Recent Build Log output for ref:
2018-06-13T13:36:41.7491920Z Step 4/15 : RUN /bin/bash -c "dotnet --version dotnet sln ./MyProjectName.sln remove docker-compose.dcproj dotnet restore ./MyProjectName.sln dotnet publish ./MyProjectName.sln --configuration Release -o ./obj/Docker/publish dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --configuration Release --no-build ./tests/Services/UnitTests/UnitTests.csproj if [[ -f "/build/tests/Services/UnitTests/coverage.xml" ]]; then cp -f -T /build/tests/Services/UnitTests/coverage.xml /build/src/Services/MyProject/MyProject.API/obj/Docker/publish/API_UnitTests_coverage.xml; fi"
...
2018-06-13T13:38:18.2159990Z Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
2018-06-13T13:38:18.2174940Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-06-13T13:38:18.2181180Z
2018-06-13T13:38:18.3384270Z Starting test execution, please wait...
2018-06-13T13:38:21.3001400Z [39;49m[xUnit.net 00:00:02.1583989] Discovering: UnitTests
2018-06-13T13:38:21.4789480Z [39;49m[39;49m[xUnit.net 00:00:02.3576858] Discovered: UnitTests
2018-06-13T13:38:21.4856690Z [39;49m[39;49m[xUnit.net 00:00:02.3651608] Starting: UnitTests
2018-06-13T13:40:56.2712730Z [39;49m[39;49m[xUnit.net 00:02:37.1488980] Finished: UnitTests
2018-06-13T13:40:56.4501810Z [39;49m
2018-06-13T13:40:56.4518150Z [39;49mTotal tests: 610. Passed: 610. Failed: 0. Skipped: 0.
2018-06-13T13:40:56.4534620Z [39;49m[39;49m[32mTest Run Successful.
2018-06-13T13:40:56.4549900Z [39;49m[39;49mTest execution time: 2.6347 Minutes
2018-06-13T13:40:56.5515730Z [39;49m[?1h=
2018-06-13T13:40:56.5528900Z Calculating coverage result...
2018-06-13T13:46:45.9403570Z Generating report '/build/tests/Services/UnitTests/coverage.xml'
...
2018-06-13T13:46:46.5420020Z -- Report generation completed

[latop2604]

Not related to coverlet, but you should use --no-restore switch in addition to --no-build

[praneeth820903]

With --no-restore there is no change in the time still it's around 6 minutes, I feel it's about how the coverage result is calculated.
2018-06-14T15:10:46.4316560Z Calculating coverage result...
2018-06-13T15:16:35.8000070Z Generating report '/build/tests/Services/UnitTests/coverage.xml'

[tonerdo]

@praneeth820903 my next priority on Coverlet is improving its performance

[praneeth820903]

@tonerdo Thank you

[petli]

I've looked at this now (since coverlet 2.0.1 was unfortunately unusable on our codebase) and have managed to cut both runtimes and analysis time dramatically. (Unfortunately the firewall at the office blocks github SSH access, so I can't send a pull request until later tonight when I'm home on my own network. Sorry for the tease.)

I've added a performance test that instruments a reasonably large source file (5000 lines, 1000 methods). In 2.0.1 analysis was O(n^2) or so, with this fix it is negligable (run times in seconds):

Uninstrumented		Original code		Changed code
Iterations	Test run time	Test run time	Analysis time	Test run time	Analysis time
50	1.9	2.5	4
100	2	3.5	28	2.5	0
150	2	4.5	68
200	2.1	5.5	104	3	0
10000	2.3	-	-	45	0

On the code in our work project I see a halfing of test runtime compared to 2.0.1, and analysis is immediate instead of 30 minutes going on infinity.

[tonerdo]

@petli quick question, what units are the times in? Seconds? Minutes? Hours? Also, what do "iterations" represent?

[petli]

@tonerdo All numbers are in seconds. Iterations is the number i put into the InlineData parameter to PerformanceTest.RunTest and thus controlled how many times the code was run through: https://github.com/tonerdo/coverlet/pull/134/files#diff-ce9987212a5f0c4a99ee1547a0731eaf

[roytoo]

@tonerdo Any estimate on when this fix will be released as a new version?
Not rushing you, just asking for timing feedback to evaluate if it would be worthwhile to build and deploy this fix locally until it is released in a new version as a NuGet package. :-)

[tonerdo]

@roytoo, just doing some last minute checks. Will make a new release in a couple of hours

[tonerdo]

@roytoo find the new release here: https://www.nuget.org/packages/coverlet.msbuild/2.1.0

[roytoo]

Thanks @tonerdo , I've already updated to use 2.1.0 and it is significantly faster.

Thanks to @petli as well for diagnosing and submitting the PR with the performance improvements. I had taken a quick look last month but wasn't able to spot the O(n^2) performance issue he found.

[petli]

Thanks @tonerdo for pushing out the new version!

[praneeth820903]

Thanks @tonerdo for pushing out the new version!