Massive performance regression .NET 9 vs .NET 10

[tomrus88]

Description

I've seeing massive performance regression when targeting .NET 10 instead of .NET 9 with below repro code

using System.Diagnostics;

namespace ConsoleApp17
{
    internal class Program
    {
        struct Foo
        {
            public int Value;
        }


        static long counter;

        static void Main(string[] args)
        {
            var startTime = Stopwatch.StartNew();

            var fakeData = Enumerable.Range(0, 4200000).Select(i => new Foo { Value = i });

            int outputEntriesPerBlock = 157;

            for (var i = 0; i < 26000; i++)
            {
                var blockEntries = fakeData.Skip(i * outputEntriesPerBlock).Take(outputEntriesPerBlock).ToArray();

                for (var j = 0; j < blockEntries.Length; j++)
                {
                    DoSomething(blockEntries[j]);
                }
            }

            Console.WriteLine($"Work took {startTime.Elapsed}, result {counter}");
        }

        static void DoSomething(Foo foo)
        {
            counter += foo.Value;
        }
    }
}

https://github.com/wowdev/TACTSharp/blob/ed6fb1b7bd3220afea1c3845c2dbe797f553ac1c/TACTSharp/GroupIndex.cs#L93-L111 code where that problem was found initially

Project file

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net9.0;net10.0</TargetFrameworks>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <PublishAot>true</PublishAot>
  </PropertyGroup>

</Project>

Please note that PublishAot bit in a project is important because it apparently turns on UseSizeOptimizedLinq option.

Configuration

Win11 x64, .NET 10 Preview 3

Regression?

Yes.

Data

On my system i'm getting following results:

// .NET 10 
Work took 00:03:45.9408366, result 8331359959000
// .NET 9 or on .NET 10 with <UseSizeOptimizedLinq>false</UseSizeOptimizedLinq> 
Work took 00:00:00.0924906, result 8331359959000

So it's around 2443 times slower on .NET 10.

Analysis

Enabled UseSizeOptimizedLinq option (triggered by enabling PublishAot) seems to be what contributes to the problem.

[Clockwork-Muse]

// .NET 9 or on .NET 10 with <UseSizeOptimizedLinq>false</UseSizeOptimizedLinq>

... because you're trading one optimization for another (size of binary vs speed it runs at). There isn't necessarily a "fix" here for us, other than recommending you add that flag.

That said...

var blockEntries = fakeData.Skip(i * outputEntriesPerBlock).Take(outputEntriesPerBlock).ToArray();

... save yourself some trouble and just use Chunk():

foreach (var block in fakeData.Chunk(outputEntriesPerBlock))
{
    foreach (var entry in block) 
    {
        DoSomething(entry);
    }
}

This is likely to be performant on both versions, because you're not constantly re-reading the source data. It's also going to be safer, if your source ever ends up being some sort of read-once stream.

---

From a quick look at your code, I would recommend you look at Pipelines and Channels.

Also, BinaryPrimitives.ReverseEndianness() should almost never be called directly - instead you should be using methods that explicitly specify the target endianness. Failure to do so results in files that only work on a single endianness.

[tomrus88]

because you're trading one optimization for another (size of binary vs speed it runs at). There isn't necessarily a "fix" here for us, other than recommending you add that flag

I'm not trading anything, I haven't even touched any of those settings in a first place, it's all basically defaults. All I did was changing target framework from .NET 9 to .NET 10 to see this performance regression... And I bet I won't be alone when .NET 10 releases.

I know that code is not perfect (it's not even my code, I'm just a potential user). What I do know is that code should not suddenly become 2000x slower when upgrading to new version.

[Symbai]

I'm not trading anything, I haven't even touched any of those settings in a first place, it's all basically defaults. All I did was changing target framework from .NET 9 to .NET 10

But you said it yourself in your first post. In .NET 10 using AOT UseSizeOptimizedLinq is probably enabled. In which case you get size optimzed linq instead. So yes, there is a performance regression from .NET 9 to .NET 10 if you built with AOT and you upgrade without disabling UseSizeOptimizedLinq. But this seems to be intended, otherwise UseSizeOptimizedLinq wouldn't be enabled by default. So Clockwork-Muse is right when saying "thre isn't necessarily a fix here for us".

[bencyoung-Fignum]

These kind of controls are always painful, especially as most people don't know they exist and especially when the default changes "silently". E.g. choosing between a 10x bigger app (or whatever) and a 2000x slower one isn't really a tradeoff. It seems more to be exposing issues in AOT compilation and trying to work around them in a very blunt way

[tomrus88]

But this seems to be intended, otherwise UseSizeOptimizedLinq wouldn't be enabled by default. So Clockwork-Muse is right when saying "thre isn't necessarily a fix here for us".

So you are saying it's intended for user to have 2500x slower code just to save a few bytes in generated binary?

Published repro nativeaot app sizes:
.NET 9: 1656832 bytes
.NET 10: 1540608 bytes (slow, UseSizeOptimizedLinq=true)
.NET 10: 1776128 bytes (fast, UseSizeOptimizedLinq=false)

So I win whopping 235520 bytes (~15% of app size on .NET 10), but app now runs almost 2500 times slower and this is OK? Really?

These kind of controls are always painful, especially as most people don't know they exist and especially when the default changes "silently". E.g. choosing between a 10x bigger app (or whatever) and a 2000x slower one isn't really a tradeoff. It seems more to be exposing issues in AOT compilation and trying to work around them in a very blunt way

Exactly. It is probably not even documented anywhere. The only reason I know it exist is because when I mentioned this issue in C# discord, someone told me to try it out.

[HighPerfDotNet]

So yes, there is a performance regression from .NET 9 to .NET 10

And that's the problem which should not be happening, especially THAT much slower - even -10% is unacceptable.

[Fabi]

I'm not trading anything, I haven't even touched any of those settings in a first place, it's all basically defaults. All I did was changing target framework from .NET 9 to .NET 10

But you said it yourself in your first post. In .NET 10 using AOT UseSizeOptimizedLinq is probably enabled. In which case you get size optimzed linq instead. So yes, there is a performance regression from .NET 9 to .NET 10 if you built with AOT and you upgrade without disabling UseSizeOptimizedLinq. But this seems to be intended, otherwise UseSizeOptimizedLinq wouldn't be enabled by default. So Clockwork-Muse is right when saying "thre isn't necessarily a fix here for us".

Going from under a second to up to lets say 5 minutes is not a regression. That is a big, unacceptable bug in the size optimized linq. Especially when it does not even give you a huge size optimization.
That is a high priority bug that needs to be fixed.

[GerardSmit]

The default value of UseSizeOptimizedLinq is always true for NativeAOT:

runtime/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets

Line 41 in 957aaac

<UseSizeOptimizedLinq Condition="'$(UseSizeOptimizedLinq)' == ''">true</UseSizeOptimizedLinq>

Wouldn't it be better to set UseSizeOptimizedLinq based on what's configured in OptimizationPreference?

OptimizationPreference	UseSizeOptimizedLinq
Size	true
Speed	false
Not provided	false

The OptimizationPreference-property is documented: https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/optimizing (maybe add UseSizeOptimizedLinq to this page as well).

When you upgrade the project to .NET 10 and have OptimizationPreference set to Size, it makes sense that .NET made improvements to improve the size-reduction. The developer chose for size.
When you upgrade the project and don't have anything set, you won't get a regression.

[elgonzo]

@Symbai, @Clockwork-Muse

that's such a silly argument to make and is entirely based on the preposterous assumption that a majority of .NET developers prefer paying a huge performance penalty for some savings in executable size -- because what else could explain the claim of this performance regression being a part of an intentional change of default build settings? Who in the world would believe that...?

Performance regressions of this order of magnitude, especially when submarined into the runtime or build tool-chain as the new default, are not trade-offs -- they are regression bugs. Great that you think "there isn't necessarily a fix here for us" -- what a glorious future awaits .NET (Looking forward to the day when i can run a .NET app on a Commodore 64 -- at the same slow speed as on a modern 5...6 GHz CPU... 😂)

[HighPerfDotNet]

it makes sense that .NET made improvements to improve the size-reduction. The developer chose for size.

No, it does not make sense that your application speed gets murdered performance wise even if size went down 10 times, which it did not. It's a time bomb, real killer that can easily get missed - because it's just a safe upgrade to next .Net.

Choosing size historically (in most languages) did very little and very rarely had significant performance reductions, nobody expects getting totally killed, which is what is happening here.