.NET 7.0 RC1 Microbenchmarks Performance Study Report #76320
Description
Data
This time we have covered following configurations comparing .NET 7.0 Preview RC1 vs. .NET 7.0 Preview 7:
- architectures: arm64, x64, x86
- Unix: ubuntu 18.04, macOS Monterey 12.3, macOS Monterey 12.3.1, macOS Monterey 12.4
- Windows: 10 & 11
| Operating System | Bit | Processor Name |
|---|---|---|
| Windows 11 | Arm64 | Unknown processor |
| Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz |
| macOS Monterey 12.3 | Arm64 | Apple M1 Max |
| Windows 10 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz |
| Windows 10 | X64 | Intel Core i7-6700 CPU 3.40GHz (Skylake) |
| Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores |
| Windows 11 | X64 | Intel Core i7-9700 CPU 3.00GHz |
| ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz |
| ubuntu 18.04 | X64 | Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge) |
| macOS Monterey 12.3.1 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) |
| macOS Monterey 12.4 | X64 | Intel Core i5-4278U CPU 2.60GHz (Haswell) |
Most of the benchmarks were run on bare-metal machines, some were executed via WSL.
This would not be possible without the help from: @adamsitnik, @AndyAyersMS, @brianrob, @dakersnar, @jkotas, @kunalspathak who contributed their results and time. An addtional thank you to @dakersnar @mrsharm and @jozkee for shadowing the creation of this report and helping with the vetting of the results!
The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!
Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview 7 report can be found here.
Notable Improvements
- A lot of improvements such as
System.Memory.Span<Char>.LastIndexOfAnyValues(Size: 512),System.Collections.ContainsFalse<Int32>.*,System.Tests.Perf_Array.IndexOfShort,System.Tests.Perf_String.Replace_Charand other benchmarks -
System.Perf_Convert.ToBase64CharArray(binaryDataSize: 1024, formattingOptions: None)- Detected and reported here: [Perf] Improvement on 8/5/2022 7:33:33 PM perf-autofiling-issues#7250
- Improved by: Vectorize Convert.ToBase64CharArray and TryToBase64Chars #73320
-
System.Tests.Perf_Environment.GetLogicalDrives- Detected and reported here: [Perf] Improvement on 8/4/2022 12:05:09 AM perf-autofiling-issues#7262
- Improved by: Convert GetAllMountPoints to UnmanagedCallersOnly #73278
-
System.Text.Json.Serialization.Tests.ReadJson<ArrayList>.DeserializeFromUtf8Bytes -
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "/w+/s+Holmes/s+/w+", Options: NonBacktracking)- Improved by: Fixes for NonBacktracking NFA mode #72199
-
System.Reflection.Attributes.GetCustomAttributes*- Detected by: [Perf] Changes at 7/14/2022 8:11:01 PM perf-autofiling-issues#6964
- Improved by: Refactor segment size computation #71178
Regressions
-
A lot of regressions such as
System.Tests.Perf_Enum.GetNamefrom "More precise writebarriers for Regions"- Introduced by More precise writebarrier for regions #67389
- Investigation: [Perf] Windows/x64: 188 Regressions from GC changes #74014
-
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(),System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, Ordinal, False)) -
System.Text.Json.Tests.Utf8JsonReaderCommentsTestson arm64 -
*.CastingPerf.*benchmarks -
System.IO.Compression.Gzip- Introduced by Update zlib, zlib-intel dependencies #73082
- Investigation: Regressions in System.IO.Compression.Gzip #73389
By Design
-
Various improvements and regressions for
System.IO.Compression.Brotli.*(level:Optimal, *)CompressionLevel.Optimalwas changed in Fix CompressionLevel.Optimal for Brotli #72266
-
Various regressions in XmlSerializer, e.g.
MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>.XmlSerializer_ -
System.Collections.Concurrent.IsEmpty<String>.Dictionary(Size: 512)- Introduced by Vectorize IndexOfAnyExcept<T>(T value) #73488 - by design according to Vectorize IndexOfAnyExcept<T>(T value) #73488 (comment)
-
System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsyncand other GC-intensive benchmarks- Introduced by make SIP not on by default #73434
- Explained in Regressions from "make SIP not on by default" #73784
Already Solved
- Regressions in Mono because of Vectorize {Last}IndexOf{Any} and {Last}IndexOfAnyExcept without code duplication #73768
- Regressions in
String.Replace- Seem to be initially caused by GC Regions but got covered via Optimized string.Replace(char, char) #67049, see String.Replace(char, char) now slower in some cases #74771 (comment)
- Regressions in
System.Numerics.Tests.Perf_BigInteger.Parse
Noise, Flaky or Multimodal
A lot of benchmarks showed up in the report generated by the tool, but were not actual regressions, they mostly are the same as in the previous report
Some regressions looked real but turned out to be stale PGO related and fixed with PGO updates, e.g. #72866
Statistics
Total: 52498
Same: 72.28 %
Slower: 10.49 %
Faster: 8.19 %
Noise: 8.97 %
Unknown: 0.07 %
Statistics per Architecture
| Architecture | Same | Slower | Faster | Noise | Unknown |
|---|---|---|---|---|---|
| Arm64 | 60.01 % | 10.68 % | 21.05 % | 8.19 % | 0.06 % |
| X64 | 75.87 % | 10.92 % | 3.51 % | 9.62 % | 0.08 % |
| X86 | 88.90 % | 2.09 % | 5.91 % | 3.09 % | 0.00 % |
Statistics per Operating System
| Operating System | Same | Slower | Faster | Noise | Unknown |
|---|---|---|---|---|---|
| macOS Monterey 12.3 | 49.64 % | 26.33 % | 15.48 % | 8.51 % | 0.04 % |
| Windows 11 | 74.27 % | 3.55 % | 13.74 % | 8.37 % | 0.07 % |
| macOS Monterey 12.3.1 | 83.02 % | 2.81 % | 4.02 % | 10.05 % | 0.11 % |
| macOS Monterey 12.4 | 89.84 % | 2.33 % | 3.49 % | 4.28 % | 0.07 % |
| ubuntu 18.04 | 52.44 % | 32.90 % | 2.29 % | 12.26 % | 0.11 % |
| Windows 10 | 83.60 % | 3.74 % | 3.65 % | 9.00 % | 0.02 % |
Statistics per Namespace
| Namespace | Same | Slower | Faster | Noise | Unknown |
|---|---|---|---|---|---|
| BenchmarksGame | 84.62 % | 11.36 % | 4.03 % | 0.00 % | 0.00 % |
| Benchstone.BenchF | 87.69 % | 10.77 % | 1.54 % | 0.00 % | 0.00 % |
| Benchstone.BenchI | 86.22 % | 9.29 % | 4.49 % | 0.00 % | 0.00 % |
| Benchstone.MDBenchF | 87.69 % | 7.69 % | 4.62 % | 0.00 % | 0.00 % |
| Benchstone.MDBenchI | 85.38 % | 9.23 % | 5.38 % | 0.00 % | 0.00 % |
| Devirtualization | 84.62 % | 3.08 % | 1.54 % | 10.77 % | 0.00 % |
| Exceptions | 79.06 % | 14.74 % | 6.20 % | 0.00 % | 0.00 % |
| FractalPerf | 92.31 % | 7.69 % | 0.00 % | 0.00 % | 0.00 % |
| GuardedDevirtualization | 84.43 % | 0.00 % | 0.18 % | 15.38 % | 0.00 % |
| Inlining | 69.23 % | 0.00 % | 0.00 % | 30.77 % | 0.00 % |
| Interop | 74.36 % | 15.38 % | 10.26 % | 0.00 % | 0.00 % |
| JetStream | 92.31 % | 7.69 % | 0.00 % | 0.00 % | 0.00 % |
| Layout | 92.31 % | 7.69 % | 0.00 % | 0.00 % | 0.00 % |
| Lowering | 92.31 % | 0.00 % | 0.00 % | 7.69 % | 0.00 % |
| MicroBenchmarks.Serializers | 87.25 % | 11.12 % | 1.64 % | 0.00 % | 0.00 % |
| Microsoft.AspNetCore.Server.Kestrel.Performance | 71.59 % | 17.05 % | 11.36 % | 0.00 % | 0.00 % |
| Microsoft.Extensions.Caching.Memory.Tests | 79.34 % | 14.05 % | 6.61 % | 0.00 % | 0.00 % |
| Microsoft.Extensions.Configuration.Xml | 90.91 % | 6.82 % | 2.27 % | 0.00 % | 0.00 % |
| Microsoft.Extensions.DependencyInjection | 76.25 % | 12.32 % | 10.26 % | 1.17 % | 0.00 % |
| Microsoft.Extensions.Logging | 77.40 % | 13.51 % | 5.97 % | 3.12 % | 0.00 % |
| Microsoft.Extensions.Primitives.Performance | 59.09 % | 2.27 % | 5.68 % | 32.95 % | 0.00 % |
| Microsoft.Extensions.Primitives | 68.18 % | 4.55 % | 7.07 % | 20.20 % | 0.00 % |
| PerfLabTests | 70.77 % | 19.42 % | 5.96 % | 3.85 % | 0.00 % |
| PerfLabTests.CastingPerf2 | 60.00 % | 36.92 % | 3.08 % | 0.00 % | 0.00 % |
| SciMark2 | 84.62 % | 7.69 % | 7.69 % | 0.00 % | 0.00 % |
| SIMD | 88.89 % | 7.69 % | 3.42 % | 0.00 % | 0.00 % |
| Span | 90.15 % | 7.69 % | 2.15 % | 0.00 % | 0.00 % |
| StoreBlock | 77.18 % | 0.26 % | 3.33 % | 19.23 % | 0.00 % |
| System.Buffers.Binary.Tests | 79.81 % | 6.73 % | 4.81 % | 8.65 % | 0.00 % |
| System.Buffers.Tests | 83.15 % | 10.11 % | 4.47 % | 2.27 % | 0.00 % |
| System.Buffers.Text.Tests | 79.55 % | 5.35 % | 4.81 % | 10.29 % | 0.00 % |
| System.Collections | 76.38 % | 11.13 % | 11.47 % | 1.02 % | 0.00 % |
| System.Collections.Concurrent | 73.54 % | 10.68 % | 9.71 % | 6.07 % | 0.00 % |
| System.Collections.Tests | 71.96 % | 12.83 % | 9.01 % | 6.20 % | 0.00 % |
| System.ComponentModel.Tests | 56.20 % | 4.96 % | 38.84 % | 0.00 % | 0.00 % |
| System.ConsoleTests | 81.82 % | 9.09 % | 9.09 % | 0.00 % | 0.00 % |
| System.Diagnostics | 70.00 % | 10.00 % | 20.00 % | 0.00 % | 0.00 % |
| System.Diagnostics.Tracing | 86.36 % | 2.27 % | 11.36 % | 0.00 % | 0.00 % |
| System.Formats.Cbor.Tests | 73.43 % | 9.09 % | 17.48 % | 0.00 % | 0.00 % |
| System.Formats.Tar.Tests | 77.27 % | 13.64 % | 9.09 % | 0.00 % | 0.00 % |
| System.Globalization.Tests | 79.88 % | 10.33 % | 8.84 % | 0.96 % | 0.00 % |
| System | 79.13 % | 9.22 % | 9.22 % | 2.43 % | 0.00 % |
| System.IO.Compression | 66.67 % | 13.76 % | 19.57 % | 0.00 % | 0.00 % |
| System.IO.MemoryMappedFiles.Tests | 79.55 % | 15.91 % | 4.55 % | 0.00 % | 0.00 % |
| System.IO.Pipelines.Tests | 79.55 % | 9.09 % | 11.36 % | 0.00 % | 0.00 % |
| System.IO.Pipes.Tests | 84.85 % | 6.06 % | 9.09 % | 0.00 % | 0.00 % |
| System.IO.Tests | 77.48 % | 15.07 % | 5.54 % | 1.91 % | 0.00 % |
| System.Linq.Tests | 76.24 % | 18.81 % | 4.41 % | 0.54 % | 0.00 % |
| System.MathBenchmarks | 82.26 % | 15.88 % | 0.87 % | 0.99 % | 0.00 % |
| System.Memory | 65.03 % | 10.58 % | 9.38 % | 15.01 % | 0.00 % |
| System.Net.Http.Tests | 72.52 % | 27.10 % | 0.38 % | 0.00 % | 0.00 % |
| System.Net.NetworkInformation.Tests | 88.64 % | 2.27 % | 9.09 % | 0.00 % | 0.00 % |
| System.Net.Primitives.Tests | 72.31 % | 9.50 % | 5.37 % | 12.81 % | 0.00 % |
| System.Net.Security.Tests | 88.27 % | 8.64 % | 3.09 % | 0.00 % | 0.00 % |
| System.Net.Sockets.Tests | 77.36 % | 22.64 % | 0.00 % | 0.00 % | 0.00 % |
| System.Net.Tests | 84.85 % | 9.09 % | 6.06 % | 0.00 % | 0.00 % |
| System.Net.WebSockets.Tests | 72.73 % | 27.27 % | 0.00 % | 0.00 % | 0.00 % |
| System.Numerics.Tests | 54.70 % | 2.04 % | 9.08 % | 34.18 % | 0.00 % |
| System.Reflection | 40.00 % | 11.79 % | 45.90 % | 2.31 % | 0.00 % |
| System.Runtime.InteropServices.Tests | 77.27 % | 13.64 % | 4.55 % | 4.55 % | 0.00 % |
| System.Runtime.Serialization.Formatters.Tests | 81.82 % | 18.18 % | 0.00 % | 0.00 % | 0.00 % |
| System.Security.Cryptography.Primitives.Tests.Performance | 77.27 % | 19.32 % | 3.41 % | 0.00 % | 0.00 % |
| System.Security.Cryptography.Tests | 85.86 % | 12.12 % | 2.02 % | 0.00 % | 0.00 % |
| System.Security.Cryptography.X509Certificates.Tests | 81.82 % | 18.18 % | 0.00 % | 0.00 % | 0.00 % |
| System.Tests | 71.59 % | 11.64 % | 7.17 % | 9.60 % | 0.00 % |
| System.Text.Encodings.Web.Tests | 65.91 % | 24.24 % | 7.95 % | 1.89 % | 0.00 % |
| System.Text.Json.Document.Tests | 79.64 % | 15.82 % | 4.55 % | 0.00 % | 0.00 % |
| System.Text.Json.Reader.Tests | 79.55 % | 13.64 % | 6.82 % | 0.00 % | 0.00 % |
| System.Text.Json.Serialization.Tests | 68.90 % | 16.27 % | 14.83 % | 0.00 % | 0.00 % |
| System.Text.Json.Tests | 87.80 % | 8.81 % | 3.39 % | 0.00 % | 0.00 % |
| System.Text | 76.41 % | 13.33 % | 10.26 % | 0.00 % | 0.00 % |
| System.Text.RegularExpressions.Tests | 72.04 % | 14.01 % | 12.77 % | 0.00 % | 1.17 % |
| System.Text.Tests | 76.88 % | 15.42 % | 5.93 % | 1.78 % | 0.00 % |
| System.Threading.Channels.Tests | 76.52 % | 14.39 % | 9.09 % | 0.00 % | 0.00 % |
| System.Threading.Tasks.Tests | 75.76 % | 18.18 % | 3.03 % | 3.03 % | 0.00 % |
| System.Threading.Tasks | 78.57 % | 11.04 % | 6.49 % | 3.90 % | 0.00 % |
| System.Threading.Tests | 73.94 % | 11.28 % | 5.73 % | 9.06 % | 0.00 % |
| System.Xml.Linq | 75.60 % | 13.88 % | 3.35 % | 7.18 % | 0.00 % |
| System.Xml.Tests | 87.27 % | 9.09 % | 3.64 % | 0.00 % | 0.00 % |
| V8.Crypto | 84.62 % | 7.69 % | 7.69 % | 0.00 % | 0.00 % |
| V8.Richards | 84.62 % | 7.69 % | 7.69 % | 0.00 % | 0.00 % |
| XmlDocumentTests.XmlDocumentTests | 75.76 % | 9.09 % | 9.09 % | 6.06 % | 0.00 % |
| XmlDocumentTests.XmlNodeListTests | 54.55 % | 22.73 % | 9.09 % | 13.64 % | 0.00 % |
| XmlDocumentTests.XmlNodeTests | 36.36 % | 4.55 % | 0.00 % | 59.09 % | 0.00 % |
| System.Drawing.Tests | 92.50 % | 1.67 % | 3.33 % | 2.50 % | 0.00 % |
| HardwareIntrinsics.RayTracer | 90.00 % | 0.00 % | 0.00 % | 10.00 % | 0.00 % |
Big thanks to everyone involved!