Skip to content

[Perf] Changes at 5/30/2022 5:18:40 PM #70019

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 18.02 ns 20.91 ns 1.16 0.04 False
ParseSpan - Duration of single invocation 12.60 ns 13.90 ns 1.10 0.04 False
TryParse - Duration of single invocation 18.60 ns 20.90 ns 1.12 0.02 False
TryFormat - Duration of single invocation 10.07 ns 14.28 ns 1.42 0.02 False
Parse - Duration of single invocation 12.48 ns 14.67 ns 1.18 0.05 False
TryFormat - Duration of single invocation 6.22 ns 8.87 ns 1.43 0.08 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.Parse(value: "4294967295")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.90963580322242 > 18.784444736706895.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -12.874258442904152 (T) = (0 -20.685657621666795) / Math.Sqrt((0.3930459224307238 / (63)) + (0.12439460999108248 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.15731933657549754 = (17.873768257320886 - 20.685657621666795) / 17.873768257320886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.ParseSpan(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.904170501323676 > 13.446475611467925.
IsChangePoint: Marked as a change because one of 4/8/2022 10:19:59 AM, 4/16/2022 5:36:25 PM, 5/30/2022 9:21:47 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -23.580213507568608 (T) = (0 -13.949503114739542) / Math.Sqrt((0.0714804080330334 / (61)) + (0.011735460891892903 / (5))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (5) - 2, .025) and -0.11145069804060415 = (12.550716949776868 - 13.949503114739542) / 12.550716949776868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryParse(value: "4294967295")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.90375599341185 > 19.736056437729772.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -12.10778235242679 (T) = (0 -20.64415202822326) / Math.Sqrt((0.23215992784562625 / (63)) + (0.09534218330846458 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.12416607372545044 = (18.36396997803821 - 20.64415202822326) / 18.36396997803821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryFormat(value: 4294967295)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.282920456766071 > 10.581393941692605.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -160.85537211932368 (T) = (0 -14.246516574728334) / Math.Sqrt((0.0259529440371242 / (63)) + (0.0009960487206765827 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.44501742446885745 = (9.859062135506706 - 14.246516574728334) / 9.859062135506706 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.Parse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.667848528809003 > 13.221865998475938.
IsChangePoint: Marked as a change because one of 3/26/2022 4:52:22 AM, 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -13.61347973216338 (T) = (0 -14.532805035709623) / Math.Sqrt((0.0620460584752799 / (63)) + (0.05094260911968003 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.14358556218507731 = (12.70810468080887 - 14.532805035709623) / 12.70810468080887 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryFormat(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.874208546011742 > 6.533805810690146.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -37.744626161663 (T) = (0 -9.017894593080138) / Math.Sqrt((0.061713030120306656 / (63)) + (0.01775431853011536 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.5328427078851965 = (5.883118043808798 - 9.017894593080138) / 5.883118043808798 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseSpan - Duration of single invocation 29.28 ns 35.90 ns 1.23 0.02 False
TryParse - Duration of single invocation 28.96 ns 36.10 ns 1.25 0.02 False
TryParse - Duration of single invocation 28.67 ns 35.58 ns 1.24 0.00 False
TryParse - Duration of single invocation 13.85 ns 15.26 ns 1.10 0.02 False
ParseSpan - Duration of single invocation 29.42 ns 36.85 ns 1.25 0.01 False
ParseSpan - Duration of single invocation 29.17 ns 36.29 ns 1.24 0.01 False
Parse - Duration of single invocation 29.15 ns 35.93 ns 1.23 0.01 False
Parse - Duration of single invocation 28.73 ns 35.85 ns 1.25 0.00 False
Parse - Duration of single invocation 13.71 ns 14.99 ns 1.09 0.03 False
ParseSpan - Duration of single invocation 14.11 ns 15.13 ns 1.07 0.01 False
TryParseSpan - Duration of single invocation 29.44 ns 36.22 ns 1.23 0.00 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.TryParseSpan(value: "9223372036854775807")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.900716326473166 > 30.429464294572007.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -73.05297825945446 (T) = (0 -35.907973090939) / Math.Sqrt((0.5706683460185307 / (62)) + (4.6626617063356646E-05 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.24277360223567693 = (28.893414718773123 - 35.907973090939) / 28.893414718773123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryParse(value: "-9223372036854775808")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.09783981842236 > 30.554607480463783.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -66.8531781225283 (T) = (0 -35.9529600503081) / Math.Sqrt((0.14025398858402163 / (63)) + (0.023082790265517565 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.22730376522629492 = (29.29426362810755 - 35.9529600503081) / 29.29426362810755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryParse(value: "9223372036854775807")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.58142235169034 > 30.234881416460922.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -83.2199470412605 (T) = (0 -35.6899605880321) / Math.Sqrt((0.07790683962682238 / (63)) + (0.015909159782171292 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.23238340922205664 = (28.960111212923117 - 35.6899605880321) / 28.960111212923117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryParse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.261934127988493 > 14.483981674952444.
IsChangePoint: Marked as a change because one of 3/26/2022 4:52:22 AM, 3/30/2022 6:37:35 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -29.580994960848166 (T) = (0 -15.289749918526136) / Math.Sqrt((0.05031466535805899 / (63)) + (0.00537326502029719 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.1092075814195988 = (13.784390022792516 - 15.289749918526136) / 13.784390022792516 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.ParseSpan(value: "-9223372036854775808")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.84727049929392 > 30.9086275873624.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -61.244756338756474 (T) = (0 -36.696407848342055) / Math.Sqrt((0.05749269576901382 / (62)) + (0.040243882518615326 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.24979800756393505 = (29.36187097935088 - 36.696407848342055) / 29.36187097935088 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.ParseSpan(value: "9223372036854775807")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.29325311558419 > 30.5373217957792.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -190.03316591378754 (T) = (0 -36.28182081910127) / Math.Sqrt((0.05515825758478877 / (62)) + (0.0017737955038416974 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.2524429686785497 = (28.96884067893499 - 36.28182081910127) / 28.96884067893499 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.Parse(value: "-9223372036854775808")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.92793820701734 > 30.631258337420388.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -22.17634450076978 (T) = (0 -36.13059356432015) / Math.Sqrt((0.10494217907618374 / (63)) + (0.2656123200248028 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.22600500179268837 = (29.470184470282987 - 36.13059356432015) / 29.470184470282987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.Parse(value: "9223372036854775807")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.854973534418235 > 30.160064026406275.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -48.9795648018665 (T) = (0 -35.832072835745805) / Math.Sqrt((0.1487416779463161 / (63)) + (0.04823235737302959 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.22791523998032076 = (29.181226577430596 - 35.832072835745805) / 29.181226577430596 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.Parse(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.988327610326593 > 14.418160884654744.
IsChangePoint: Marked as a change because one of 3/26/2022 4:52:22 AM, 3/30/2022 6:37:35 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -25.158217370356574 (T) = (0 -15.078662593838393) / Math.Sqrt((0.03530098336008355 / (63)) + (0.0065332247519817 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.09565640768781127 = (13.762218235604754 - 15.078662593838393) / 13.762218235604754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.ParseSpan(value: "12345")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.133668691470513 > 14.799796602882415.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -7.896044870754496 (T) = (0 -15.421821462857048) / Math.Sqrt((0.04045834132847233 / (62)) + (0.12558167550094462 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11802891738947521 = (13.793759019101222 - 15.421821462857048) / 13.793759019101222 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryParseSpan(value: "-9223372036854775808")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.21639956941242 > 30.92121567458197.
IsChangePoint: Marked as a change because one of 4/21/2022 3:06:05 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -51.92687470020792 (T) = (0 -36.35896690761533) / Math.Sqrt((0.09227762044301657 / (62)) + (0.053350921379235965 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.24729341424862317 = (29.150291737504432 - 36.35896690761533) / 29.150291737504432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 14.38 μs 17.12 μs 1.19 0.01 False
Span - Duration of single invocation 1.09 μs 1.27 μs 1.17 0.41 False
IDictionary - Duration of single invocation 15.69 μs 18.64 μs 1.19 0.01 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<String>.Dictionary(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.117625870364577 > 14.93440068308716.
IsChangePoint: Marked as a change because one of 3/20/2022 12:44:50 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -20.604820487853146 (T) = (0 -16981.715544424256) / Math.Sqrt((21146.982497953864 / (63)) + (51471.054983084076 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.19115401617742828 = (14256.52376921067 - 16981.715544424256) / 14256.52376921067 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.Span(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2705983781760568 > 1.171387688564955.
IsChangePoint: Marked as a change because one of 5/1/2022 10:22:11 AM, 5/7/2022 5:14:33 AM, 5/27/2022 6:33:44 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -9.40347035776082 (T) = (0 -1348.54392768287) / Math.Sqrt((5185.886876618465 / (48)) + (3293.5958691364126 / (18))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (18) - 2, .025) and -0.1350156056116734 = (1188.1280935834566 - 1348.54392768287) / 1188.1280935834566 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.640563153698977 > 16.3060764320747.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -148.06350576110805 (T) = (0 -18617.460411816468) / Math.Sqrt((18390.8119899223 / (63)) + (422.0202055953375 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.19819563663150466 = (15537.913711784044 - 18617.460411816468) / 15537.913711784044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Booleans

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBooleans - Duration of single invocation 1.28 ms 1.41 ms 1.10 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Booleans*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Booleans.WriteBooleans(Formatted: False, SkipValidation: False)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4083138362847225 > 1.3371767802884615.
IsChangePoint: Marked as a change because one of 4/20/2022 2:49:46 PM, 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -19.000157394402965 (T) = (0 -1409275.8037444036) / Math.Sqrt((2465533646.103263 / (62)) + (701569.5314954243 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.09321824855532522 = (1289107.4637719821 - 1409275.8037444036) / 1289107.4637719821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 9380113a892ec86d147e6e7506a10eda407a82be
Compare 842b6a87aaa6b6f25555541f541d5ee202d3972a
Diff Diff

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 11.43 μs 13.80 μs 1.21 0.01 False
Dictionary - Duration of single invocation 10.31 μs 12.54 μs 1.22 0.01 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.IDictionary(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.798332819383258 > 12.003789028223215.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -73.01182758091328 (T) = (0 -13750.654563387116) / Math.Sqrt((19799.46109350553 / (63)) + (1767.874673691752 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.18991870092908086 = (11555.961388497124 - 13750.654563387116) / 11555.961388497124 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.AddGivenSize&lt;String&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.54316092908695 > 10.837595829547027.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -47.4391453766967 (T) = (0 -12511.993589076936) / Math.Sqrt((13933.601442032314 / (63)) + (5380.782194242741 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.20508843688281608 = (10382.635171109532 - 12511.993589076936) / 10382.635171109532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Labels

arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtime

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions