-
Notifications
You must be signed in to change notification settings - Fork 4
Closed
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 486b4d1a36aef5bbe6a77bb2c3412772e712961e |
| Compare | afacf6a99e1b98331af958caec638bd93dc46ffe |
| Diff | Diff |
Improvements in System.Buffers.Text.Tests.Utf8ParserTests
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryParseInt16 - Duration of single invocation | 10.78 ns | 5.39 ns | 0.50 | 0.76 | False | |||||
| TryParseBool - Duration of single invocation | 16.56 ns | 1.15 ns | 0.07 | 0.75 | False | |||||
| TryParseUInt16 - Duration of single invocation | 16.56 ns | 6.55 ns | 0.40 | 0.77 | False | |||||
| TryParseByte - Duration of single invocation | 11.17 ns | 9.86 ns | 0.88 | 0.57 | False | |||||
| TryParseUInt16 - Duration of single invocation | 16.95 ns | 7.87 ns | 0.46 | 0.70 | False | |||||
| TryParseUInt16 - Duration of single invocation | 17.33 ns | 6.93 ns | 0.40 | 0.76 | False | |||||
| TryParseUInt32Hex - Duration of single invocation | 16.25 ns | 8.09 ns | 0.50 | 0.68 | False | |||||
| TryParseInt16 - Duration of single invocation | 16.27 ns | 8.09 ns | 0.50 | 0.74 | False | |||||
| TryParseSByte - Duration of single invocation | 12.32 ns | 11.02 ns | 0.89 | 0.74 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'Payloads
Histogram
System.Buffers.Text.Tests.Utf8ParserTests.TryParseInt16(value: 0)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 5.390942466723847 < 10.242046945361183.
IsChangePoint: Marked as a change because one of 5/12/2022 12:13:09 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 4.442486490391021 (T) = (0 -7.451073949771431) / Math.Sqrt((8.506872078816755 / (299)) + (3.8911188824618006 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.28811798871794575 = (10.466725990663145 - 7.451073949771431) / 10.466725990663145 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseBool(value: true)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.15472238026061 < 8.778549514179586.
IsChangePoint: Marked as a change because one of 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 8.709935879029002 (T) = (0 -2.7170759584087643) / Math.Sqrt((7.55233344367303 / (299)) + (3.582687052990574 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.6759261477863492 = (8.384125839987512 - 2.7170759584087643) / 8.384125839987512 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt16(value: 0)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.545813817855214 < 10.975991555464086.
IsChangePoint: Marked as a change because one of 5/13/2022 2:27:51 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 10.693979116918369 (T) = (0 -6.484345111133795) / Math.Sqrt((9.626767771749465 / (299)) + (0.9332209018812935 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.3780876871677709 = (10.426462022601974 - 6.484345111133795) / 10.426462022601974 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseByte(value: 255)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.85968658514099 < 10.24274620254418.
IsChangePoint: Marked as a change because one of 5/12/2022 12:48:56 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 4.000064021725237 (T) = (0 -7.214503313253841) / Math.Sqrt((8.086167217735246 / (299)) + (4.301945741143833 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.2826521709461865 = (10.057189861116354 - 7.214503313253841) / 10.057189861116354 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt16(value: 65535)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.868914993411778 < 16.098240301599045.
IsChangePoint: Marked as a change because one of 5/12/2022 12:48:56 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 6.414829884304966 (T) = (0 -8.20950616200433) / Math.Sqrt((9.271827487061138 / (299)) + (2.1824311607323366 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.2900988896597187 = (11.564295424286934 - 8.20950616200433) / 11.564295424286934 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt16(value: 12345)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.932224071872212 < 14.267176068885467.
IsChangePoint: Marked as a change because one of 5/12/2022 12:13:09 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 2.5663263342096037 (T) = (0 -9.52534278896083) / Math.Sqrt((8.161972047470423 / (299)) + (5.429227090415783 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.17623516370914508 = (11.56318207493579 - 9.52534278896083) / 11.56318207493579 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt32Hex(value: 0)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.086948872743806 < 10.060131040623139.
IsChangePoint: Marked as a change because one of 5/16/2022 8:28:13 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 8.328526895412972 (T) = (0 -6.931582492959426) / Math.Sqrt((8.472663533235636 / (299)) + (1.669125850069101 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.357146647716058 = (10.782525234305403 - 6.931582492959426) / 10.782525234305403 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseInt16(value: 32767)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.087271461313206 < 13.53438811212587.
IsChangePoint: Marked as a change because one of 5/13/2022 2:27:51 PM, 5/30/2022 8:02:30 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 2.9731752458608876 (T) = (0 -10.243630063622911) / Math.Sqrt((8.81893008128026 / (299)) + (5.7010868090261955 / (8))) is greater than 1.967772354556772 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (8) - 2, .975) and 0.20002460499501046 = (12.804931411120489 - 10.243630063622911) / 12.804931411120489 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseSByte(value: 127)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.015243588700532 < 11.705952479697734.
IsChangePoint: Marked as a change because one of 5/14/2022 8:55:12 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsImprovementStdDev: Marked as improvement because 3.7420754850836926 (T) = (0 -7.813110333745113) / Math.Sqrt((9.160951031220463 / (299)) + (6.141073240386201 / (9))) is greater than 1.9677467375895035 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (9) - 2, .975) and 0.28795977262784134 = (10.972849613539422 - 7.813110333745113) / 10.972849613539422 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository








