Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig #92942
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 22.04 |
| Queue | TigerUbuntu |
| Baseline | f4fa6554ace1ea713d1106dc29f9de3b99c2c92b |
| Compare | 876f763a8ff8345b61897ff6297876445b2b484f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Collections.IterateForEach<String>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 36.09 μs | 40.61 μs | 1.13 | 0.19 | False | ||||
| 2.20 μs | 2.39 μs | 1.08 | 0.09 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach<String>*'Payloads
System.Collections.IterateForEach<String>.ImmutableDictionary(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.607407156808044 > 37.089893448601295.
IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/15/2023 9:01:53 PM, 9/28/2023 10:19:19 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -11.236408533541496 (T) = (0 -39493.35415456238) / Math.Sqrt((943458.4853290913 / (34)) + (1529798.422734647 / (19))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (19) - 2, .025) and -0.10328420471377898 = (35796.174717110174 - 39493.35415456238) / 35796.174717110174 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.IterateForEach<String>.ConcurrentStack(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.387808932717352 > 2.3085860846963.
IsChangePoint: Marked as a change because one of 8/28/2023 1:23:07 PM, 9/15/2023 8:45:09 AM, 9/19/2023 8:08:41 AM, 9/28/2023 2:34:07 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -16.551421431608766 (T) = (0 -2387.7381153497513) / Math.Sqrt((4070.8571388419746 / (38)) + (13.31218331867806 / (18))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (18) - 2, .025) and -0.0775787348001714 = (2215.836335887362 - 2387.7381153497513) / 2215.836335887362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 22.04 |
| Queue | TigerUbuntu |
| Baseline | f4fa6554ace1ea713d1106dc29f9de3b99c2c92b |
| Compare | 876f763a8ff8345b61897ff6297876445b2b484f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 1.85 ms | 2.24 ms | 1.21 | 0.33 | False | ||||
| 1.61 ms | 1.94 ms | 1.21 | 0.41 | False | ||||
| 1.86 ms | 2.07 ms | 1.11 | 0.45 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Payloads
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2449590656370657 > 1.9542073608851673.
IsChangePoint: Marked as a change because one of 9/28/2023 10:19:19 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -11.648669240398894 (T) = (0 -2166888.4753188854) / Math.Sqrt((4262293862.298161 / (35)) + (8191364725.816323 / (19))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (19) - 2, .025) and -0.1446953787735456 = (1892982.635817524 - 2166888.4753188854) / 1892982.635817524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: None)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9430794095052082 > 1.7182877414062498.
IsChangePoint: Marked as a change because one of 9/28/2023 10:19:19 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -10.057369289460228 (T) = (0 -1936425.0974881868) / Math.Sqrt((7492289159.796342 / (34)) + (11152469588.025946 / (19))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (19) - 2, .025) and -0.17312251799620365 = (1650658.876445208 - 1936425.0974881868) / 1650658.876445208 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: None)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0714250809151786 > 1.9340903322916665.
IsChangePoint: Marked as a change because one of 9/28/2023 10:19:19 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -10.211522379110676 (T) = (0 -2103754.6285187663) / Math.Sqrt((10609781681.739275 / (35)) + (7167586246.510971 / (19))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (19) - 2, .025) and -0.1449648439200695 = (1837396.7023441903 - 2103754.6285187663) / 1837396.7023441903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 22.04 |
| Queue | TigerUbuntu |
| Baseline | f4fa6554ace1ea713d1106dc29f9de3b99c2c92b |
| Compare | 876f763a8ff8345b61897ff6297876445b2b484f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 16.24 ms | 18.43 ms | 1.13 | 0.12 | False | ||||
| 22.95 ms | 27.87 ms | 1.21 | 0.21 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'Payloads
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: NonBacktracking)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.429721516666664 > 16.69790454625.
IsChangePoint: Marked as a change because one of 9/24/2023 8:25:02 PM, 9/28/2023 10:19:19 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -18.877123352706093 (T) = (0 -18327106.116156835) / Math.Sqrt((299020111257.0226 / (36)) + (50275826454.47573 / (19))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (19) - 2, .025) and -0.12081683441778807 = (16351562.140549852 - 18327106.116156835) / 16351562.140549852 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: None)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.87251505 > 23.039335678125.
IsChangePoint: Marked as a change because one of 9/26/2023 11:02:05 PM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -8.740315008046933 (T) = (0 -25513424.58827134) / Math.Sqrt((810639697982.4038 / (28)) + (2154563205186.1409 / (26))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (26) - 2, .025) and -0.12937647731383523 = (22590717.179583665 - 25513424.58827134) / 22590717.179583665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 22.04 |
| Queue | TigerUbuntu |
| Baseline | f4fa6554ace1ea713d1106dc29f9de3b99c2c92b |
| Compare | 876f763a8ff8345b61897ff6297876445b2b484f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 58.11 ms | 65.30 ms | 1.12 | 0.02 | True | ||||
| 1.96 ms | 2.16 ms | 1.10 | 0.19 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'Payloads
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.29767571153846 > 61.09831479375.
IsChangePoint: Marked as a change because one of 9/28/2023 10:19:19 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -127.3965043846384 (T) = (0 -65769797.70447755) / Math.Sqrt((23320381574.356968 / (36)) + (54862236968.17504 / (19))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (19) - 2, .025) and -0.1301616834793032 = (58195034.09635989 - 65769797.70447755) / 58195034.09635989 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Twain", Options: None)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1603478074776787 > 2.021658105924479.
IsChangePoint: Marked as a change because one of 9/28/2023 10:19:19 AM, 10/2/2023 10:24:51 PM falls between 9/24/2023 12:29:05 AM and 10/2/2023 10:24:51 PM.
IsRegressionStdDev: Marked as regression because -11.575749488278916 (T) = (0 -2172740.487405355) / Math.Sqrt((3960745995.094912 / (35)) + (6213756494.9701805 / (19))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (19) - 2, .025) and -0.12584855161166092 = (1929869.2388910216 - 2172740.487405355) / 1929869.2388910216 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository