-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Closed
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)Linux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime
Milestone
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Tests.Perf_UInt32
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Parse - Duration of single invocation | 18.02 ns | 20.91 ns | 1.16 | 0.04 | False | |||||
| ParseSpan - Duration of single invocation | 12.60 ns | 13.90 ns | 1.10 | 0.04 | False | |||||
| TryParse - Duration of single invocation | 18.60 ns | 20.90 ns | 1.12 | 0.02 | False | |||||
| TryFormat - Duration of single invocation | 10.07 ns | 14.28 ns | 1.42 | 0.02 | False | |||||
| Parse - Duration of single invocation | 12.48 ns | 14.67 ns | 1.18 | 0.05 | False | |||||
| TryFormat - Duration of single invocation | 6.22 ns | 8.87 ns | 1.43 | 0.08 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt32*'Payloads
Histogram
System.Tests.Perf_UInt32.Parse(value: "4294967295")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.90963580322242 > 18.784444736706895.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -12.874258442904152 (T) = (0 -20.685657621666795) / Math.Sqrt((0.3930459224307238 / (63)) + (0.12439460999108248 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.15731933657549754 = (17.873768257320886 - 20.685657621666795) / 17.873768257320886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt32.ParseSpan(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.904170501323676 > 13.446475611467925.
IsChangePoint: Marked as a change because one of 4/8/2022 10:19:59 AM, 4/16/2022 5:36:25 PM, 5/30/2022 9:21:47 AM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -23.580213507568608 (T) = (0 -13.949503114739542) / Math.Sqrt((0.0714804080330334 / (61)) + (0.011735460891892903 / (5))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (61) + (5) - 2, .025) and -0.11145069804060415 = (12.550716949776868 - 13.949503114739542) / 12.550716949776868 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt32.TryParse(value: "4294967295")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.90375599341185 > 19.736056437729772.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -12.10778235242679 (T) = (0 -20.64415202822326) / Math.Sqrt((0.23215992784562625 / (63)) + (0.09534218330846458 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.12416607372545044 = (18.36396997803821 - 20.64415202822326) / 18.36396997803821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt32.TryFormat(value: 4294967295)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.282920456766071 > 10.581393941692605.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -160.85537211932368 (T) = (0 -14.246516574728334) / Math.Sqrt((0.0259529440371242 / (63)) + (0.0009960487206765827 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.44501742446885745 = (9.859062135506706 - 14.246516574728334) / 9.859062135506706 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt32.Parse(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.667848528809003 > 13.221865998475938.
IsChangePoint: Marked as a change because one of 3/26/2022 4:52:22 AM, 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -13.61347973216338 (T) = (0 -14.532805035709623) / Math.Sqrt((0.0620460584752799 / (63)) + (0.05094260911968003 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.14358556218507731 = (12.70810468080887 - 14.532805035709623) / 12.70810468080887 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt32.TryFormat(value: 12345)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.874208546011742 > 6.533805810690146.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -37.744626161663 (T) = (0 -9.017894593080138) / Math.Sqrt((0.061713030120306656 / (63)) + (0.01775431853011536 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.5328427078851965 = (5.883118043808798 - 9.017894593080138) / 5.883118043808798 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Tests.Perf_Int64
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryParseSpan - Duration of single invocation | 29.28 ns | 35.90 ns | 1.23 | 0.02 | False | |||||
| TryParse - Duration of single invocation | 28.96 ns | 36.10 ns | 1.25 | 0.02 | False | |||||
| TryParse - Duration of single invocation | 28.67 ns | 35.58 ns | 1.24 | 0.00 | False | |||||
| TryParse - Duration of single invocation | 13.85 ns | 15.26 ns | 1.10 | 0.02 | False | |||||
| ParseSpan - Duration of single invocation | 29.42 ns | 36.85 ns | 1.25 | 0.01 | False | |||||
| ParseSpan - Duration of single invocation | 29.17 ns | 36.29 ns | 1.24 | 0.01 | False | |||||
| Parse - Duration of single invocation | 29.15 ns | 35.93 ns | 1.23 | 0.01 | False | |||||
| Parse - Duration of single invocation | 28.73 ns | 35.85 ns | 1.25 | 0.00 | False | |||||
| Parse - Duration of single invocation | 13.71 ns | 14.99 ns | 1.09 | 0.03 | False | |||||
| ParseSpan - Duration of single invocation | 14.11 ns | 15.13 ns | 1.07 | 0.01 | False | |||||
| TryParseSpan - Duration of single invocation | 29.44 ns | 36.22 ns | 1.23 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int64*'Payloads
Histogram
System.Tests.Perf_Int64.TryParseSpan(value: "9223372036854775807")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.900716326473166 > 30.429464294572007.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -73.05297825945446 (T) = (0 -35.907973090939) / Math.Sqrt((0.5706683460185307 / (62)) + (4.6626617063356646E-05 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.24277360223567693 = (28.893414718773123 - 35.907973090939) / 28.893414718773123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryParse(value: "-9223372036854775808")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.09783981842236 > 30.554607480463783.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -66.8531781225283 (T) = (0 -35.9529600503081) / Math.Sqrt((0.14025398858402163 / (63)) + (0.023082790265517565 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.22730376522629492 = (29.29426362810755 - 35.9529600503081) / 29.29426362810755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryParse(value: "9223372036854775807")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.58142235169034 > 30.234881416460922.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -83.2199470412605 (T) = (0 -35.6899605880321) / Math.Sqrt((0.07790683962682238 / (63)) + (0.015909159782171292 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.23238340922205664 = (28.960111212923117 - 35.6899605880321) / 28.960111212923117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryParse(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.261934127988493 > 14.483981674952444.
IsChangePoint: Marked as a change because one of 3/26/2022 4:52:22 AM, 3/30/2022 6:37:35 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -29.580994960848166 (T) = (0 -15.289749918526136) / Math.Sqrt((0.05031466535805899 / (63)) + (0.00537326502029719 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.1092075814195988 = (13.784390022792516 - 15.289749918526136) / 13.784390022792516 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.ParseSpan(value: "-9223372036854775808")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.84727049929392 > 30.9086275873624.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -61.244756338756474 (T) = (0 -36.696407848342055) / Math.Sqrt((0.05749269576901382 / (62)) + (0.040243882518615326 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.24979800756393505 = (29.36187097935088 - 36.696407848342055) / 29.36187097935088 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.ParseSpan(value: "9223372036854775807")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.29325311558419 > 30.5373217957792.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -190.03316591378754 (T) = (0 -36.28182081910127) / Math.Sqrt((0.05515825758478877 / (62)) + (0.0017737955038416974 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.2524429686785497 = (28.96884067893499 - 36.28182081910127) / 28.96884067893499 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.Parse(value: "-9223372036854775808")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.92793820701734 > 30.631258337420388.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -22.17634450076978 (T) = (0 -36.13059356432015) / Math.Sqrt((0.10494217907618374 / (63)) + (0.2656123200248028 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.22600500179268837 = (29.470184470282987 - 36.13059356432015) / 29.470184470282987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.Parse(value: "9223372036854775807")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.854973534418235 > 30.160064026406275.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -48.9795648018665 (T) = (0 -35.832072835745805) / Math.Sqrt((0.1487416779463161 / (63)) + (0.04823235737302959 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.22791523998032076 = (29.181226577430596 - 35.832072835745805) / 29.181226577430596 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.Parse(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.988327610326593 > 14.418160884654744.
IsChangePoint: Marked as a change because one of 3/26/2022 4:52:22 AM, 3/30/2022 6:37:35 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -25.158217370356574 (T) = (0 -15.078662593838393) / Math.Sqrt((0.03530098336008355 / (63)) + (0.0065332247519817 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.09565640768781127 = (13.762218235604754 - 15.078662593838393) / 13.762218235604754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.ParseSpan(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.133668691470513 > 14.799796602882415.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -7.896044870754496 (T) = (0 -15.421821462857048) / Math.Sqrt((0.04045834132847233 / (62)) + (0.12558167550094462 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.11802891738947521 = (13.793759019101222 - 15.421821462857048) / 13.793759019101222 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryParseSpan(value: "-9223372036854775808")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.21639956941242 > 30.92121567458197.
IsChangePoint: Marked as a change because one of 4/21/2022 3:06:05 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -51.92687470020792 (T) = (0 -36.35896690761533) / Math.Sqrt((0.09227762044301657 / (62)) + (0.053350921379235965 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.24729341424862317 = (29.150291737504432 - 36.35896690761533) / 29.150291737504432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Collections.CreateAddAndClear<String>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Dictionary - Duration of single invocation | 14.38 μs | 17.12 μs | 1.19 | 0.01 | False | |||||
| Span - Duration of single invocation | 1.09 μs | 1.27 μs | 1.17 | 0.41 | False | |||||
| IDictionary - Duration of single invocation | 15.69 μs | 18.64 μs | 1.19 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.CreateAddAndClear<String>*'Payloads
Histogram
System.Collections.CreateAddAndClear<String>.Dictionary(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.117625870364577 > 14.93440068308716.
IsChangePoint: Marked as a change because one of 3/20/2022 12:44:50 AM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -20.604820487853146 (T) = (0 -16981.715544424256) / Math.Sqrt((21146.982497953864 / (63)) + (51471.054983084076 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.19115401617742828 = (14256.52376921067 - 16981.715544424256) / 14256.52376921067 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<String>.Span(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2705983781760568 > 1.171387688564955.
IsChangePoint: Marked as a change because one of 5/1/2022 10:22:11 AM, 5/7/2022 5:14:33 AM, 5/27/2022 6:33:44 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -9.40347035776082 (T) = (0 -1348.54392768287) / Math.Sqrt((5185.886876618465 / (48)) + (3293.5958691364126 / (18))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (18) - 2, .025) and -0.1350156056116734 = (1188.1280935834566 - 1348.54392768287) / 1188.1280935834566 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<String>.IDictionary(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.640563153698977 > 16.3060764320747.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -148.06350576110805 (T) = (0 -18617.460411816468) / Math.Sqrt((18390.8119899223 / (63)) + (422.0202055953375 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.19819563663150466 = (15537.913711784044 - 18617.460411816468) / 15537.913711784044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Booleans
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| WriteBooleans - Duration of single invocation | 1.28 ms | 1.41 ms | 1.10 | 0.03 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Booleans*'Payloads
Histogram
System.Text.Json.Tests.Perf_Booleans.WriteBooleans(Formatted: False, SkipValidation: False)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4083138362847225 > 1.3371767802884615.
IsChangePoint: Marked as a change because one of 4/20/2022 2:49:46 PM, 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -19.000157394402965 (T) = (0 -1409275.8037444036) / Math.Sqrt((2465533646.103263 / (62)) + (701569.5314954243 / (3))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (62) + (3) - 2, .025) and -0.09321824855532522 = (1289107.4637719821 - 1409275.8037444036) / 1289107.4637719821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Collections.AddGivenSize<String>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IDictionary - Duration of single invocation | 11.43 μs | 13.80 μs | 1.21 | 0.01 | False | |||||
| Dictionary - Duration of single invocation | 10.31 μs | 12.54 μs | 1.22 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.AddGivenSize<String>*'Payloads
Histogram
System.Collections.AddGivenSize<String>.IDictionary(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.798332819383258 > 12.003789028223215.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -73.01182758091328 (T) = (0 -13750.654563387116) / Math.Sqrt((19799.46109350553 / (63)) + (1767.874673691752 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.18991870092908086 = (11555.961388497124 - 13750.654563387116) / 11555.961388497124 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.AddGivenSize<String>.Dictionary(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.54316092908695 > 10.837595829547027.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -47.4391453766967 (T) = (0 -12511.993589076936) / Math.Sqrt((13933.601442032314 / (63)) + (5380.782194242741 / (3))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (63) + (3) - 2, .025) and -0.20508843688281608 = (10382.635171109532 - 12511.993589076936) / 10382.635171109532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Metadata
Metadata
Assignees
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-linuxLinux OS (any supported distro)Linux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtime

















_1.png)
_2.png)
_3.png)

_1.png)
_2.png)