Skip to content

[Perf] Linux/x64: 10 Regressions on 4/17/2023 6:40:18 PM #16690

@performanceautofiler

Description

@performanceautofiler

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Compare 9aefa9daa141bb7d9ba3f2b373d4b050c9b243fe
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in Benchstone.BenchI.BubbleSort

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 13.98 μs 16.66 μs 1.19 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.BubbleSort*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.BubbleSort.Test


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.658781892561645 > 14.701866402841699.
IsChangePoint: Marked as a change because one of 4/17/2023 1:54:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -153.16241279580066 (T) = (0 -16689.517611894484) / Math.Sqrt((618.071970332376 / (34)) + (863.4151245252449 / (3))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (3) - 2, .025) and -0.19122848291582448 = (14010.341299968573 - 16689.517611894484) / 14010.341299968573 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Compare 9aefa9daa141bb7d9ba3f2b373d4b050c9b243fe
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 32.74 ns 41.61 ns 1.27 0.63 False
Parse - Duration of single invocation 34.85 ns 55.31 ns 1.59 0.64 False
Parse - Duration of single invocation 29.62 ns 45.14 ns 1.52 0.64 False
Parse - Duration of single invocation 32.51 ns 62.35 ns 1.92 0.60 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.TryParse(value: "32767")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.60801255486559 > 35.383080307333366.
IsChangePoint: Marked as a change because one of 4/17/2023 1:54:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -4.867418475011111 (T) = (0 -42.003359650420606) / Math.Sqrt((4.703727716160876 / (33)) + (6.219593841269267 / (3))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (3) - 2, .025) and -0.20845021318753362 = (34.75803900901154 - 42.003359650420606) / 34.75803900901154 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Int16.Parse(value: "-32768")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.310154545609116 > 36.84920275796996.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -4.94434288066354 (T) = (0 -60.040635983106384) / Math.Sqrt((9.224378537163398 / (35)) + (44.7549092610125 / (2))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (2) - 2, .025) and -0.6443100118549693 = (36.514182575202895 - 60.040635983106384) / 36.514182575202895 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Int16.Parse(value: "0")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.137692037892045 > 31.175075851605392.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -16.136560146275635 (T) = (0 -45.962324640048685) / Math.Sqrt((5.087574564505079 / (34)) + (1.3600378570792637 / (2))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (2) - 2, .025) and -0.47012223074977505 = (31.26428787938776 - 45.962324640048685) / 31.26428787938776 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Int16.Parse(value: "32767")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 62.349159704857215 > 35.05085057111022.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -14.244584051658638 (T) = (0 -60.5671271903529) / Math.Sqrt((3.874091858800674 / (34)) + (6.351279765501148 / (2))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (2) - 2, .025) and -0.7438709986317713 = (34.73142637148816 - 60.5671271903529) / 34.73142637148816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Compare 9aefa9daa141bb7d9ba3f2b373d4b050c9b243fe
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.MathBenchmarks.Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Max - Duration of single invocation 4.37 μs 4.72 μs 1.08 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Double*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Double.Max


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.7246419905144075 > 4.590642361590055.
IsChangePoint: Marked as a change because one of 4/17/2023 1:54:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -60.99946862483759 (T) = (0 -4727.158473363669) / Math.Sqrt((843.8994465338607 / (34)) + (18.170442292055185 / (3))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (3) - 2, .025) and -0.07724313445729859 = (4388.200139929554 - 4727.158473363669) / 4388.200139929554 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Compare 9aefa9daa141bb7d9ba3f2b373d4b050c9b243fe
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Collections.IndexerSetReverse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IList - Duration of single invocation 5.81 μs 6.58 μs 1.13 0.38 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSetReverse<Int32>.IList(Size: 512)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.575333374423472 > 6.14265784724513.
IsChangePoint: Marked as a change because one of 4/17/2023 1:54:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -6.16439574384889 (T) = (0 -6792.449780814076) / Math.Sqrt((99031.42270704478 / (33)) + (36915.49467640371 / (3))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (3) - 2, .025) and -0.12647941604711804 = (6029.803726595536 - 6792.449780814076) / 6029.803726595536 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Compare 9aefa9daa141bb7d9ba3f2b373d4b050c9b243fe
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse3 - Duration of single invocation 161.51 ns 176.86 ns 1.10 0.27 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Parse3


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 176.86332079447192 > 171.45890788116245.
IsChangePoint: Marked as a change because one of 4/17/2023 1:54:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -2.0718539292452953 (T) = (0 -187.8179065847211) / Math.Sqrt((46.00785749593048 / (34)) + (273.5937390357462 / (3))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (3) - 2, .025) and -0.11872318359622708 = (167.88595189469936 - 187.8179065847211) / 167.88595189469936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Compare 9aefa9daa141bb7d9ba3f2b373d4b050c9b243fe
Diff Diff
Configs AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 31.81 ns 48.44 ns 1.52 0.69 False
Parse - Duration of single invocation 38.14 ns 49.27 ns 1.29 0.67 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Byte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Byte.Parse(value: "0")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.442604495652844 > 32.13360356781855.
IsChangePoint: Marked as a change because one of 2/16/2023 3:28:45 AM, 4/17/2023 1:54:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -19.97950445458046 (T) = (0 -49.11717427390863) / Math.Sqrt((9.057586314823975 / (34)) + (1.5246280098980298 / (3))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (3) - 2, .025) and -0.5576515247569722 = (31.532838695466232 - 49.11717427390863) / 31.532838695466232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Byte.Parse(value: "255")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.27370689275745 > 34.63949147109729.
IsChangePoint: Marked as a change because one of 2/14/2023 9:29:25 PM, 4/17/2023 6:40:18 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -21.644043043609884 (T) = (0 -48.72163261037674) / Math.Sqrt((8.11892158472017 / (34)) + (0.6095720265323639 / (2))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (2) - 2, .025) and -0.48704907541745585 = (32.763970884215254 - 48.72163261037674) / 32.763970884215254 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions