-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Closed
Labels
area-VM-coreclrruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
Milestone
Description
`### Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | ac84ea6e241ad2d2cde346144b2c4b3a5d64fa1d |
| Compare | 0b12d37843e7165fb4c8b794186f19ef43af6c73 |
| Diff | Diff |
| tighter diff | a1bc0f3...5371203 |
Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| EnumerateUsingIndexer - Duration of single invocation | 1.74 μs | 1.85 μs | 1.06 | 0.01 | False | |||||
| 2.29 μs | 2.96 μs | 1.29 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*'Payloads
Histogram
System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfStrings)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.847930218805441 > 1.826793768692464.
IsChangePoint: Marked as a change because one of 12/2/2021 4:22:41 PM, 1/28/2022 4:34:00 PM, 3/17/2022 6:54:53 PM, 3/22/2022 12:26:23 PM falls between 3/13/2022 11:50:20 AM and 3/22/2022 12:26:23 PM.
IsRegressionStdDev: Marked as regression because -37.567265158486116 (T) = (0 -1844.6392731408614) / Math.Sqrt((179.10677767316412 / (27)) + (4.721594038302771 / (5))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (5) - 2, .025) and -0.05939220196144539 = (1741.2241375059637 - 1844.6392731408614) / 1741.2241375059637 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: Json400KB)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9598147370059142 > 2.4018696254624405.
IsChangePoint: Marked as a change because one of 1/28/2022 4:34:00 PM, 3/15/2022 12:53:39 AM, 3/22/2022 12:26:23 PM falls between 3/13/2022 11:50:20 AM and 3/22/2022 12:26:23 PM.
IsRegressionStdDev: Marked as regression because -120.6428218804647 (T) = (0 -2935.2304895714733) / Math.Sqrt((19.852319641720772 / (16)) + (437.8173014793286 / (16))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (16) - 2, .025) and -0.2817622489671951 = (2289.996051870456 - 2935.2304895714733) / 2289.996051870456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | ac84ea6e241ad2d2cde346144b2c4b3a5d64fa1d |
| Compare | 0b12d37843e7165fb4c8b794186f19ef43af6c73 |
| Diff | Diff |
Regressions in System.Collections.ContainsTrue<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| HashSet - Duration of single invocation | 4.99 μs | 6.01 μs | 1.20 | 0.05 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsTrue<Int32>*'Payloads
Histogram
System.Collections.ContainsTrue<Int32>.HashSet(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.013932003309152 > 5.256438464261174.
IsChangePoint: Marked as a change because one of 12/7/2021 3:27:58 PM, 12/8/2021 11:18:41 PM, 12/27/2021 6:47:12 AM, 3/17/2022 6:54:53 PM, 3/22/2022 12:26:23 PM falls between 3/13/2022 11:50:20 AM and 3/22/2022 12:26:23 PM.
IsRegressionStdDev: Marked as regression because -19.80322266903542 (T) = (0 -6053.083872496654) / Math.Sqrt((1148.5014152204074 / (27)) + (13486.517429206551 / (5))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (5) - 2, .025) and -0.2066314747169855 = (5016.51415475997 - 6053.083872496654) / 5016.51415475997 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Metadata
Metadata
Assignees
Labels
area-VM-coreclrruntime-coreclrspecific to the CoreCLR runtimespecific to the CoreCLR runtimetenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark

_1.png)