Skip to content

Performance regressions due to GC Regions #63930

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline fe8422cdff450941bc3ba4b120576e8822459df4
Compare ef7ff071099d185a5f66b55966866fcc964e287c
Diff Diff

Regressions in BenchmarksGame.KNucleotide_9

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 56.91 ms 62.13 ms 1.09 0.08 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'BenchmarksGame.KNucleotide_9*'

Payloads

Baseline
Compare

Histogram

BenchmarksGame.KNucleotide_9.RunBench


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 62.132452575 > 59.921577299062506.
IsChangePoint: Marked as a change because one of 12/7/2021 9:49:54 AM, 1/11/2022 2:18:34 PM, 1/18/2022 5:19:20 AM falls between 1/8/2022 8:22:56 PM and 1/18/2022 5:19:20 AM.
IsRegressionStdDev: Marked as regression because -22.718367898215927 (T) = (0 -62105451.3935966) / Math.Sqrt((395282169985.69183 / (32)) + (919908986491.3501 / (33))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (33) - 2, .025) and -0.07917867059962681 = (57548812.89405839 - 62105451.3935966) / 57548812.89405839 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline fe8422cdff450941bc3ba4b120576e8822459df4
Compare ef7ff071099d185a5f66b55966866fcc964e287c
Diff Diff

Regressions in System.Collections.CtorFromCollectionNonGeneric<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 23.05 μs 24.77 μs 1.07 0.00 False
Stack - Duration of single invocation 21.92 μs 23.80 μs 1.09 0.03 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<Int32>.Queue(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.767194148235916 > 24.231584690585265.
IsChangePoint: Marked as a change because one of 1/11/2022 2:18:34 PM, 1/18/2022 5:19:20 AM falls between 1/8/2022 8:22:56 PM and 1/18/2022 5:19:20 AM.
IsRegressionStdDev: Marked as regression because -32.38804485194529 (T) = (0 -24850.402096972426) / Math.Sqrt((15720.062709109392 / (32)) + (83195.71292542249 / (33))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (33) - 2, .025) and -0.07704352403223026 = (23072.792828220732 - 24850.402096972426) / 23072.792828220732 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.796464449949188 > 22.789753051162776.
IsChangePoint: Marked as a change because one of 1/11/2022 2:18:34 PM, 1/18/2022 5:19:20 AM falls between 1/8/2022 8:22:56 PM and 1/18/2022 5:19:20 AM.
IsRegressionStdDev: Marked as regression because -19.54017533902972 (T) = (0 -23993.40668573774) / Math.Sqrt((29843.46554165946 / (32)) + (399530.12372097117 / (33))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (33) - 2, .025) and -0.10253203806089282 = (21762.094757751234 - 23993.40668573774) / 21762.094757751234 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline fe8422cdff450941bc3ba4b120576e8822459df4
Compare ef7ff071099d185a5f66b55966866fcc964e287c
Diff Diff

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NewOperator_Array - Duration of single invocation 33.53 ns 35.75 ns 1.07 0.06 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Byte>.NewOperator_Array(length: 1000)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.75326767840259 > 35.15473559389209.
IsChangePoint: Marked as a change because one of 1/11/2022 2:18:34 PM, 1/18/2022 5:19:20 AM falls between 1/8/2022 8:22:56 PM and 1/18/2022 5:19:20 AM.
IsRegressionStdDev: Marked as regression because -17.408231406640766 (T) = (0 -35.88087983769851) / Math.Sqrt((0.4308079323728019 / (32)) + (0.2601888585838306 / (33))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (33) - 2, .025) and -0.07629459721512617 = (33.3374151747477 - 35.88087983769851) / 33.3374151747477 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline fe8422cdff450941bc3ba4b120576e8822459df4
Compare ef7ff071099d185a5f66b55966866fcc964e287c
Diff Diff

Regressions in System.Diagnostics.Perf_Activity

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ActivityAllocations - Duration of single invocation 157.30 ns 175.74 ns 1.12 0.03 True
EnumerateActivityLinkTagsLarge - Duration of single invocation 9.47 μs 10.39 μs 1.10 0.04 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Diagnostics.Perf_Activity*'

Payloads

Baseline
Compare

Histogram

System.Diagnostics.Perf_Activity.ActivityAllocations(idFormat: Hierarchical)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 175.74116709608057 > 164.34910749867822.
IsChangePoint: Marked as a change because one of 1/11/2022 2:18:34 PM, 1/18/2022 5:19:20 AM falls between 1/8/2022 8:22:56 PM and 1/18/2022 5:19:20 AM.
IsRegressionStdDev: Marked as regression because -31.019760794326274 (T) = (0 -179.71304496587595) / Math.Sqrt((7.147932236036577 / (32)) + (9.979676620655603 / (33))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (33) - 2, .025) and -0.14306553699152283 = (157.22024604019597 - 179.71304496587595) / 157.22024604019597 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Diagnostics.Perf_Activity.EnumerateActivityLinkTagsLarge

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.392829875135723 > 9.942452779283839.
IsChangePoint: Marked as a change because one of 12/16/2021 8:45:41 PM, 1/12/2022 5:25:46 AM, 1/18/2022 5:19:20 AM falls between 1/8/2022 8:22:56 PM and 1/18/2022 5:19:20 AM.
IsRegressionStdDev: Marked as regression because -9.311624945530044 (T) = (0 -9863.282544360974) / Math.Sqrt((17788.074286222753 / (34)) + (92154.89737234468 / (31))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (31) - 2, .025) and -0.059119325068669744 = (9312.720777445424 - 9863.282544360974) / 9312.720777445424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions