Skip to content

[Perf] Linux/x64: 27 Regressions on 8/17/2022 6:09:21 PM #74395

@performanceautofiler

Description

@performanceautofiler

There are some large wasm regressions here; as much as a 10 x slow down. Unfortunately, there seems to be some kind of glitch in the reporting, and the narrowest I can get the change range is this: 543bcc5...a250bfe.

@lewing @radical @pavelsavara Any likely candidates in that date range?

Automated issue reporting below.

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Uri_IsNotMatch - Duration of single invocation 444.95 ns 592.31 ns 1.33 0.06 False
Uri_IsMatch - Duration of single invocation 455.72 ns 505.99 ns 1.11 0.05 False
Backtracking - Duration of single invocation 2.45 μs 2.93 μs 1.19 0.10 False
Backtracking - Duration of single invocation 2.45 μs 2.91 μs 1.19 0.10 False
Uri_IsNotMatch - Duration of single invocation 446.25 ns 581.71 ns 1.30 0.05 False
Uri_IsNotMatch - Duration of single invocation 447.94 ns 593.46 ns 1.32 0.04 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.3093690327008 > 465.23264235782176.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -39.47621223602034 (T) = (0 -594.5364016454982) / Math.Sqrt((78.92038015570309 / (14)) + (73.27831635677934 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.32709221691484786 = (447.99931313563445 - 594.5364016454982) / 447.99931313563445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 505.9861520887883 > 482.7014638466213.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/25/2022 6:06:34 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.329767862712716 (T) = (0 -514.4415933301957) / Math.Sqrt((16.440001607480827 / (14)) + (123.32550738351799 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.12037177742305741 = (459.17043225905917 - 514.4415933301957) / 459.17043225905917 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9266354356553093 > 2.591927353654952.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/29/2022 4:06:39 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -50.14154390540031 (T) = (0 -2934.2504087143566) / Math.Sqrt((556.432351718883 / (14)) + (417.26555557386655 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18845364681210353 = (2468.964958444245 - 2934.2504087143566) / 2468.964958444245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9101414399726724 > 2.5780127447204717.
IsChangePoint: Marked as a change because one of 6/24/2022 5:01:29 PM, 6/25/2022 2:40:01 AM, 6/29/2022 4:06:39 AM, 7/12/2022 12:16:14 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -61.70631510275718 (T) = (0 -2923.812224534253) / Math.Sqrt((407.1945204507192 / (14)) + (236.0815117136678 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18619294374645873 = (2464.870694053968 - 2923.812224534253) / 2464.870694053968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 581.7136123036038 > 471.5091607783785.
IsChangePoint: Marked as a change because one of 6/25/2022 5:15:47 AM, 6/26/2022 6:11:06 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -65.94725454140149 (T) = (0 -592.9345145576187) / Math.Sqrt((24.613985181656737 / (14)) + (26.562821064295214 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3181628970020087 = (449.81884705310074 - 592.9345145576187) / 449.81884705310074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 593.4594763018782 > 471.43001759852376.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/24/2022 6:48:53 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -48.543170908408 (T) = (0 -596.0567739968475) / Math.Sqrt((34.7262092388468 / (14)) + (56.62787404726851 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3178926565327463 = (452.2802149645615 - 596.0567739968475) / 452.2802149645615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadSpanEmptyLoop - Duration of single invocation 92.65 μs 99.48 μs 1.07 0.09 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 1.70 μs 1.91 μs 1.13 0.07 False
ReadSpanEmptyLoop - Duration of single invocation 4.25 μs 4.95 μs 1.16 0.09 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 11.52 μs 12.59 μs 1.09 0.12 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 112.87 μs 118.86 μs 1.05 0.07 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Reader*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.47909983507735 > 97.49529907633277.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.875672080579108 (T) = (0 -101229.28427961336) / Math.Sqrt((265137.5448871564 / (14)) + (5976092.35420695 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.08769210576340399 = (93067.95897775218 - 101229.28427961336) / 93067.95897775218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.90914647431983 > 1.777974693072054.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -24.12292592652499 (T) = (0 -1927.1512790997767) / Math.Sqrt((611.4742780301284 / (14)) + (471.64347753333664 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.1398571366423715 = (1690.6954539728583 - 1927.1512790997767) / 1690.6954539728583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.946905717995979 > 4.460535892543025.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/25/2022 5:15:47 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.376124247128779 (T) = (0 -4757.429574447925) / Math.Sqrt((4758.814727171611 / (14)) + (8231.09975483114 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.11985725605871154 = (4248.246415968665 - 4757.429574447925) / 4248.246415968665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.58755040658602 > 12.38535184349092.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -6.466117999484596 (T) = (0 -12591.240550799725) / Math.Sqrt((69596.82198162597 / (14)) + (79394.98687585884 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06418279234267017 = (11831.840019778581 - 12591.240550799725) / 11831.840019778581 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 118.8626892992424 > 118.55053428507193.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -7.506023118864004 (T) = (0 -121572.12886191734) / Math.Sqrt((4192742.5510894423 / (14)) + (6482122.752315378 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06649265512776761 = (113992.4670623754 - 121572.12886191734) / 113992.4670623754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Format_MultipleArgs - Duration of single invocation 529.02 ns 850.06 ns 1.61 0.14 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Format_MultipleArgs


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 850.0557408302284 > 547.6817332098481.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/27/2022 8:54:23 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -86.40554379355675 (T) = (0 -848.1242802325862) / Math.Sqrt((88.87602574889881 / (14)) + (69.21223243326581 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.6173858106947179 = (524.3796963127123 - 848.1242802325862) / 524.3796963127123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_Environment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExpandEnvironmentVariables - Duration of single invocation 382.06 ns 1.06 μs 2.76 0.11 False
SetEnvironmentVariable - Duration of single invocation 194.79 ns 661.73 ns 3.40 0.16 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Environment.ExpandEnvironmentVariables


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.05539799053797 > 404.12774324999464.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -100.66329094480135 (T) = (0 -1058.1419394441955) / Math.Sqrt((68.03089949492843 / (14)) + (358.51775965172686 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -1.7472293516620847 = (385.16694603747425 - 1058.1419394441955) / 385.16694603747425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Environment.SetEnvironmentVariable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 661.725814258165 > 213.09369271099766.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -164.42300266168527 (T) = (0 -647.4959303229203) / Math.Sqrt((49.2398659816711 / (14)) + (35.11775717258779 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -2.2433183359130027 = (199.6399561379005 - 647.4959303229203) / 199.6399561379005 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 24.67 ns 36.76 ns 1.49 0.21 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", comparisonType: OrdinalIgnoreCase)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.76482912607041 > 26.19735640294146.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -67.5375793028699 (T) = (0 -36.38052307472043) / Math.Sqrt((0.037995025208353364 / (14)) + (0.23599627187678154 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.4615325383200185 = (24.892037721266604 - 36.38052307472043) / 24.892037721266604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ICollection - Duration of single invocation 186.65 μs 2.42 ms 12.95 0.01 False
Stack - Duration of single invocation 148.17 μs 2.98 ms 20.11 0.01 False
Queue - Duration of single invocation 186.46 μs 2.42 ms 12.98 0.02 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'
### Histogram

System.Collections.ContainsFalse<Int32>.ICollection(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4180025503826528 > 196.43006513020836.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -373.14299335258266 (T) = (0 -2427206.700892857) / Math.Sqrt((1083311.6941798208 / (14)) + (323590191.29539 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -11.95486324505923 = (187358.7281454749 - 2427206.700892857) / 187358.7281454749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9792487988095235 > 155.34581797022406.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -42.64814865820476 (T) = (0 -2532821.408927845) / Math.Sqrt((874624.8610895702 / (14)) + (28136948852.7768 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.092038104366907 = (148187.20818792962 - 2532821.408927845) / 148187.20818792962 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4195034313186814 > 195.68580234160373.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -91.92109792877557 (T) = (0 -2470159.070670352) / Math.Sqrt((714329.7028732462 / (14)) + (5552447103.319803 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -12.216076602459486 = (186905.62600179392 - 2470159.070670352) / 186905.62600179392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Load - Duration of single invocation 277.65 μs 294.38 μs 1.06 0.09 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 294.3815035812672 > 286.28722408634746.
IsChangePoint: Marked as a change because one of 7/1/2022 12:02:05 PM, 7/10/2022 5:59:30 PM, 7/18/2022 3:18:54 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.519909182959154 (T) = (0 -300647.38905054185) / Math.Sqrt((42979330.934151694 / (14)) + (35851042.04585498 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.09181732773423353 = (275364.18539395486 - 300647.38905054185) / 275364.18539395486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions