Skip to content

[Perf] Linux/x64: 4 Regressions on 12/30/2022 6:15:13 AM #11293

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 96b849ecfc8c535eafd01e0ae82c8a54488aacfb
Compare c71ad8cfdd0065395b01b3af6c06a0214f04c249
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 437.59 ns 514.54 ns 1.18 0.12 False
IndexOfString - Duration of single invocation 289.04 ns 348.50 ns 1.21 0.11 False
IndexOfString - Duration of single invocation 320.20 ns 445.41 ns 1.39 0.13 False
IndexOfString - Duration of single invocation 231.69 ns 279.19 ns 1.21 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 514.5388159226967 > 452.18107559005864.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/22/2022 11:54:51 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -20.355749884721657 (T) = (0 -514.7970980871519) / Math.Sqrt((60.70143357557084 / (33)) + (132.0208005668129 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.1811320467917317 = (435.8505888359202 - 514.7970980871519) / 435.8505888359202 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 348.5012465477906 > 304.0939800708053.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/18/2022 12:43:31 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -17.23444724225909 (T) = (0 -334.9141477296196) / Math.Sqrt((29.76659841787175 / (33)) + (50.46295192405744 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.14351702320542387 = (292.880771281229 - 334.9141477296196) / 292.880771281229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 445.4095278610983 > 338.8064296772343.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -27.353568489859335 (T) = (0 -432.9844174022045) / Math.Sqrt((39.320636739028686 / (33)) + (140.72479757709223 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.327699618565273 = (326.1162474913507 - 432.9844174022045) / 326.1162474913507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonType: InvariantCultureIgnoreCase)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 279.1868331193049 > 246.48819031441124.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/30/2022 6:06:02 AM, 1/3/2023 6:29:42 AM falls between 12/23/2022 9:40:21 PM and 1/3/2023 6:29:42 AM.
IsRegressionStdDev: Marked as regression because -16.802923377085133 (T) = (0 -282.12044769546003) / Math.Sqrt((41.222897278141836 / (33)) + (51.495536715894346 / (10))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (10) - 2, .025) and -0.17738445278311885 = (239.61625026437162 - 282.12044769546003) / 239.61625026437162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions