-
Notifications
You must be signed in to change notification settings - Fork 4
Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 96b849ecfc8c535eafd01e0ae82c8a54488aacfb |
| Compare | c71ad8cfdd0065395b01b3af6c06a0214f04c249 |
| Diff | Diff |
Regressions in System.Memory.ReadOnlySpan
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IndexOfString - Duration of single invocation | 437.59 ns | 514.54 ns | 1.18 | 0.12 | False | |||||
| IndexOfString - Duration of single invocation | 289.04 ns | 348.50 ns | 1.21 | 0.11 | False | |||||
| IndexOfString - Duration of single invocation | 320.20 ns | 445.41 ns | 1.39 | 0.13 | False | |||||
| IndexOfString - Duration of single invocation | 231.69 ns | 279.19 ns | 1.21 | 0.17 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'Payloads
Histogram
System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 514.5388159226967 > 452.18107559005864.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/22/2022 11:54:51 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -20.355749884721657 (T) = (0 -514.7970980871519) / Math.Sqrt((60.70143357557084 / (33)) + (132.0208005668129 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.1811320467917317 = (435.8505888359202 - 514.7970980871519) / 435.8505888359202 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 348.5012465477906 > 304.0939800708053.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/18/2022 12:43:31 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -17.23444724225909 (T) = (0 -334.9141477296196) / Math.Sqrt((29.76659841787175 / (33)) + (50.46295192405744 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.14351702320542387 = (292.880771281229 - 334.9141477296196) / 292.880771281229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 445.4095278610983 > 338.8064296772343.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -27.353568489859335 (T) = (0 -432.9844174022045) / Math.Sqrt((39.320636739028686 / (33)) + (140.72479757709223 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.327699618565273 = (326.1162474913507 - 432.9844174022045) / 326.1162474913507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonType: InvariantCultureIgnoreCase)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 279.1868331193049 > 246.48819031441124.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -16.802923377085133 (T) = (0 -282.12044769546003) / Math.Sqrt((41.222897278141836 / (33)) + (51.495536715894346 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.17738445278311885 = (239.61625026437162 - 282.12044769546003) / 239.61625026437162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository