-
Notifications
You must be signed in to change notification settings - Fork 4
Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Collections.TryAddGiventSize<String>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Dictionary - Duration of single invocation | 9.87 μs | 11.00 μs | 1.11 | 0.02 | True |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryAddGiventSize<String>*'Payloads
Histogram
System.Collections.TryAddGiventSize<String>.Dictionary(Count: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.99983260233918 > 10.379600335817404.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -27.985454026071398 (T) = (0 -11161.806655564626) / Math.Sqrt((18171.264563411456 / (21)) + (62396.81664550035 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.12941891050354545 = (9882.787114471275 - 11161.806655564626) / 9882.787114471275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsMatch - Duration of single invocation | 108.82 ns | 124.07 ns | 1.14 | 0.06 | False | |||||
| IsMatch - Duration of single invocation | 69.41 ns | 82.95 ns | 1.20 | 0.11 | False | |||||
| IsMatch - Duration of single invocation | 118.61 ns | 139.03 ns | 1.17 | 0.05 | False | |||||
| IsMatch - Duration of single invocation | 114.24 ns | 129.23 ns | 1.13 | 0.06 | False | |||||
| IsMatch - Duration of single invocation | 124.22 ns | 137.47 ns | 1.11 | 0.07 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 10, Options: None)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 124.07029333694665 > 112.18285357223827.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/15/2022 3:27:46 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -17.659561207622726 (T) = (0 -125.78850456044614) / Math.Sqrt((11.069921722261116 / (21)) + (26.65810365859347 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.16801605427731614 = (107.69415719912769 - 125.78850456044614) / 107.69415719912769 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 1, Options: Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 82.94520972092253 > 73.08364853189485.
IsChangePoint: Marked as a change because one of 4/22/2022 7:11:21 PM, 5/9/2022 11:56:12 AM, 5/11/2022 9:09:01 PM, 5/18/2022 4:10:31 AM, 5/27/2022 6:33:44 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -5.1530953800046 (T) = (0 -85.95155262244992) / Math.Sqrt((81.60278929021568 / (21)) + (4.037012106153998 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13556524854349644 = (75.69054506792408 - 85.95155262244992) / 75.69054506792408 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 139.02711273942995 > 124.77740986557704.
IsChangePoint: Marked as a change because one of 5/11/2022 9:09:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -14.528949200403131 (T) = (0 -139.5461346234724) / Math.Sqrt((26.038615258660815 / (21)) + (13.25445868812534 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.146142049397908 = (121.75291421929668 - 139.5461346234724) / 121.75291421929668 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.2276694595754 > 119.90458456311633.
IsChangePoint: Marked as a change because one of 5/11/2022 9:09:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -19.881590939517157 (T) = (0 -129.93844950740427) / Math.Sqrt((0.602214722474547 / (21)) + (29.928751821323413 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13641773496881968 = (114.34039218947022 - 129.93844950740427) / 114.34039218947022 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.46837103762078 > 130.7152144980496.
IsChangePoint: Marked as a change because one of 5/10/2022 1:42:57 AM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -24.0720493565256 (T) = (0 -140.60649842535577) / Math.Sqrt((4.633622733854685 / (21)) + (7.967504829284167 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.11744425971045498 = (125.82864621971285 - 140.60649842535577) / 125.82864621971285 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Tests.Perf_Double
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Parse - Duration of single invocation | 115.29 ns | 137.06 ns | 1.19 | 0.07 | False | |||||
| Parse - Duration of single invocation | 117.15 ns | 142.74 ns | 1.22 | 0.07 | False | |||||
| TryParse - Duration of single invocation | 111.68 ns | 143.84 ns | 1.29 | 0.12 | False | |||||
| TryParse - Duration of single invocation | 52.00 ns | 66.79 ns | 1.28 | 0.07 | False | |||||
| TryParse - Duration of single invocation | 105.78 ns | 139.11 ns | 1.32 | 0.12 | False | |||||
| Parse - Duration of single invocation | 52.43 ns | 59.04 ns | 1.13 | 0.03 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Double*'Payloads
Histogram
System.Tests.Perf_Double.Parse(value: "1.7976931348623157e+308")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.0568498266847 > 118.42904804825866.
IsChangePoint: Marked as a change because one of 4/11/2022 1:47:52 PM, 4/14/2022 8:50:18 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -10.334340350362416 (T) = (0 -131.80169251558272) / Math.Sqrt((40.321951925641216 / (21)) + (25.28302716541836 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.1387831577717734 = (115.73906025575191 - 131.80169251558272) / 115.73906025575191 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.Parse(value: "-1.7976931348623157e+308")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 142.74121511431122 > 120.99850435845686.
IsChangePoint: Marked as a change because one of 3/30/2022 6:37:35 PM, 4/11/2022 1:47:52 PM, 4/14/2022 8:50:18 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -14.275495274950462 (T) = (0 -139.68790711519569) / Math.Sqrt((40.29464348616143 / (21)) + (16.917087763661282 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.18107050910384775 = (118.27228437122324 - 139.68790711519569) / 118.27228437122324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 143.83730024294363 > 116.80401521030954.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -13.652462853232594 (T) = (0 -140.33222749282402) / Math.Sqrt((61.162080317859434 / (21)) + (26.22889934670685 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.2196501308672507 = (115.05941248334771 - 140.33222749282402) / 115.05941248334771 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.TryParse(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 66.79212207249705 > 54.53303994018309.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -19.83714901356965 (T) = (0 -60.52895437722216) / Math.Sqrt((0.16938005838816028 / (21)) + (9.845877897051679 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.1722997763865099 = (51.63265881001553 - 60.52895437722216) / 51.63265881001553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.TryParse(value: "1.7976931348623157e+308")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 139.10875162319203 > 111.68270757256214.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/16/2022 5:36:25 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -22.877109968049588 (T) = (0 -134.3378070688334) / Math.Sqrt((17.450951657834228 / (21)) + (25.889819900769293 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.24537840120063906 = (107.86906769807601 - 134.3378070688334) / 107.86906769807601 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.Parse(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.04179914094483 > 55.24612904387697.
IsChangePoint: Marked as a change because one of 5/11/2022 9:09:01 PM, 5/20/2022 10:18:01 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -24.738934628181553 (T) = (0 -58.10612080608749) / Math.Sqrt((0.4033465286847956 / (21)) + (1.1662041490321697 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.09568655713110737 = (53.031700013030864 - 58.10612080608749) / 53.031700013030864 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Tests.Perf_Int32
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryFormat - Duration of single invocation | 8.07 ns | 9.38 ns | 1.16 | 0.03 | False | |||||
| TryParse - Duration of single invocation | 20.36 ns | 26.02 ns | 1.28 | 0.04 | False | |||||
| TryParse - Duration of single invocation | 15.75 ns | 21.29 ns | 1.35 | 0.03 | False | |||||
| ParseHex - Duration of single invocation | 17.38 ns | 18.38 ns | 1.06 | 0.05 | False | |||||
| TryParse - Duration of single invocation | 19.17 ns | 28.61 ns | 1.49 | 0.04 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'Payloads
Histogram
System.Tests.Perf_Int32.TryFormat(value: 12345)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.37799856842049 > 8.498350861161882.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/5/2022 12:13:02 AM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -17.724318634580737 (T) = (0 -8.919489304094956) / Math.Sqrt((0.0004444891519090147 / (21)) + (0.1115883659989248 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.10302164759379544 = (8.086413647050827 - 8.919489304094956) / 8.086413647050827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int32.TryParse(value: "2147483647")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.017896092048296 > 21.365472259432316.
IsChangePoint: Marked as a change because one of 4/8/2022 8:59:46 PM, 4/9/2022 9:54:15 AM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -40.85155008877778 (T) = (0 -25.79118472708775) / Math.Sqrt((0.13984427597728366 / (67)) + (0.08814828878702677 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.28602654588064363 = (20.05493961979338 - 25.79118472708775) / 20.05493961979338 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int32.TryParse(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.290688502785713 > 16.553673520973263.
IsChangePoint: Marked as a change because one of 3/30/2022 6:37:35 PM, 4/28/2022 4:25:37 PM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -29.32840935389612 (T) = (0 -20.88541396048852) / Math.Sqrt((0.24087354605105246 / (67)) + (0.17797818274137664 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.38503233096067585 = (15.079369263533462 - 20.88541396048852) / 15.079369263533462 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int32.ParseHex(value: "7FFFFFFF")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.38267020153511 > 18.260850227222154.
IsChangePoint: Marked as a change because one of 4/25/2022 5:27:54 PM, 4/28/2022 4:25:37 PM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -10.065469884070335 (T) = (0 -18.748892800050733) / Math.Sqrt((0.13664069342588522 / (67)) + (0.08526251015378862 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.08012283860872224 = (17.35811162386001 - 18.748892800050733) / 17.35811162386001 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int32.TryParse(value: "-2147483648")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.606607283299482 > 20.818654764652777.
IsChangePoint: Marked as a change because one of 3/30/2022 12:01:24 PM, 5/3/2022 11:42:39 AM, 6/6/2022 10:51:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -17.791275883165387 (T) = (0 -28.143571248773707) / Math.Sqrt((0.35942667907018855 / (67)) + (0.9290557314094756 / (5))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (67) + (5) - 2, .025) and -0.3819865555001683 = (20.364576729610796 - 28.143571248773707) / 20.364576729610796 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Tests.Perf_Decimal
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryParse - Duration of single invocation | 75.22 ns | 86.88 ns | 1.16 | 0.06 | False | |||||
| Parse - Duration of single invocation | 69.16 ns | 84.46 ns | 1.22 | 0.07 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Decimal*'Payloads
Histogram
System.Tests.Perf_Decimal.TryParse(value: "123456.789")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.88126959512908 > 77.02922650063351.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -13.264538653424081 (T) = (0 -86.42769899558687) / Math.Sqrt((13.735816049828893 / (21)) + (16.80500262717185 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.17953956201292576 = (73.27240372344524 - 86.42769899558687) / 73.27240372344524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Decimal.Parse(value: "123456.789")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 84.45614574277016 > 71.40289225034365.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -25.353179878127523 (T) = (0 -81.27025313299572) / Math.Sqrt((1.8328652754728065 / (21)) + (7.075182700492638 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.17413267063858984 = (69.21726578717401 - 81.27025313299572) / 69.21726578717401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | 9380113a892ec86d147e6e7506a10eda407a82be |
| Compare | 842b6a87aaa6b6f25555541f541d5ee202d3972a |
| Diff | Diff |
Regressions in System.Linq.Tests.Perf_Enumerable
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Distinct - Duration of single invocation | 2.12 μs | 2.39 μs | 1.12 | 0.03 | True | |||||
| Count - Duration of single invocation | 256.41 ns | 285.33 ns | 1.11 | 0.19 | False | |||||
| ToDictionary - Duration of single invocation | 1.88 μs | 2.28 μs | 1.21 | 0.18 | False | |||||
| Except - Duration of single invocation | 2.73 μs | 3.05 μs | 1.12 | 0.13 | False | |||||
| Intersect - Duration of single invocation | 3.31 μs | 3.62 μs | 1.09 | 0.09 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'Payloads
Histogram
System.Linq.Tests.Perf_Enumerable.Distinct(input: IEnumerable)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3851745810055864 > 2.240086085214206.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -34.51877392427077 (T) = (0 -2426.1893046715304) / Math.Sqrt((268.2467991658867 / (21)) + (3033.478339031773 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.13757536552800953 = (2132.7723667305386 - 2426.1893046715304) / 2132.7723667305386 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Linq.Tests.Perf_Enumerable.Count(input: IEnumerable)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 285.3330527840488 > 269.24427716325647.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -16.601026435680915 (T) = (0 -296.96140209897493) / Math.Sqrt((0.10455203912235275 / (21)) + (301.69007842752325 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.1574385101341319 = (256.5677567308185 - 296.96140209897493) / 256.5677567308185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Linq.Tests.Perf_Enumerable.ToDictionary(input: IEnumerable)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2768355278276857 > 1.9744065370017274.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -68.68267413778204 (T) = (0 -2252.493686408211) / Math.Sqrt((217.01272602105442 / (21)) + (1001.2959460703461 / (51))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (51) - 2, .025) and -0.20036395357270662 = (1876.5089368970096 - 2252.493686408211) / 1876.5089368970096 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Linq.Tests.Perf_Enumerable.Except(input: IEnumerable)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0540807682563815 > 2.868999124943706.
IsChangePoint: Marked as a change because one of 5/31/2022 6:06:22 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -15.6746404826657 (T) = (0 -3044.42224242225) / Math.Sqrt((5971.528708813805 / (29)) + (3857.198357257714 / (43))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (43) - 2, .025) and -0.0971207152152939 = (2774.920024935293 - 3044.42224242225) / 2774.920024935293 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Linq.Tests.Perf_Enumerable.Intersect(input: IEnumerable)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.6226016893554216 > 3.477522068783281.
IsChangePoint: Marked as a change because one of 6/1/2022 7:21:14 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -11.158577653665644 (T) = (0 -3585.9630530058885) / Math.Sqrt((9699.321246184028 / (33)) + (3477.4937798221026 / (39))) is less than -1.9944371117706434 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (39) - 2, .025) and -0.06485465055320434 = (3367.5610574109237 - 3585.9630530058885) / 3367.5610574109237 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
_1.png)






















