Regressions in Devirtualization.EqualityComparer #88667
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | Windows 10.0.19042 |
| Queue | OwlWindows |
| Baseline | e3ae9042413a0a43e13e4272f7d8fb5d977edceb |
| Compare | 9ea317f17d1ef5988921f611f80834d15a3e8070 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in Devirtualization.EqualityComparer
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| ValueTupleCompare - Duration of single invocation | 1.84 ns | 3.84 ns | 2.08 | 0.01 | True | |||
| ValueTupleCompareNoOpt - Duration of single invocation | 4.47 ns | 6.09 ns | 1.36 | 0.04 | True | |||
| ValueTupleCompareCached - Duration of single invocation | 3.12 ns | 4.34 ns | 1.39 | 0.20 | False | |||
| ValueTupleCompareWrapped - Duration of single invocation | 1.86 ns | 3.81 ns | 2.05 | 0.16 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Devirtualization.EqualityComparer*'Payloads
Devirtualization.EqualityComparer.ValueTupleCompare
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.843112840283866 > 2.2906849067852604.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/5/2023 3:16:33 PM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -57.42751590463879 (T) = (0 -3.8374712675164275) / Math.Sqrt((0.026791380736852954 / (26)) + (1.1981683562958218E-05 / (15))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (15) - 2, .025) and -0.92517646236273 = (1.9933088433913102 - 3.8374712675164275) / 1.9933088433913102 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
Devirtualization.EqualityComparer.ValueTupleCompareNoOpt
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.092543341559851 > 4.704558109630318.
IsChangePoint: Marked as a change because one of 7/5/2023 3:16:33 PM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -848.1624026883707 (T) = (0 -6.094328851821734) / Math.Sqrt((5.678715287955347E-05 / (26)) + (2.235689394598058E-05 / (15))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (15) - 2, .025) and -0.36385144554699234 = (4.4684696942030335 - 6.094328851821734) / 4.4684696942030335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
Devirtualization.EqualityComparer.ValueTupleCompareCached
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.340094943627221 > 2.9171170060329317.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 7/5/2023 3:16:33 PM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -33.57170136745841 (T) = (0 -4.339886342672489) / Math.Sqrt((0.04238543685962824 / (26)) + (6.660613160837092E-06 / (15))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (15) - 2, .025) and -0.4542803144749973 = (2.984215834785064 - 4.339886342672489) / 2.984215834785064 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Devirtualization.EqualityComparer.ValueTupleCompareWrapped
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.8107148712347003 > 2.5720030080921883.
IsChangePoint: Marked as a change because one of 5/19/2023 10:55:27 PM, 7/1/2023 6:36:20 PM, 7/5/2023 3:16:33 PM, 7/10/2023 4:47:56 PM falls between 7/1/2023 10:14:52 PM and 7/10/2023 4:47:56 PM.
IsRegressionStdDev: Marked as regression because -26.814634495440476 (T) = (0 -3.81425130162482) / Math.Sqrt((0.10037157665979843 / (26)) + (1.4754199600727505E-05 / (15))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (15) - 2, .025) and -0.7757393334723831 = (2.1479792837421767 - 3.81425130162482) / 2.1479792837421767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository