Major compiler slowdown due to changes in .NET GC between .NET 7 preview2 and .NET 7 preview3

[vzarytovskii]

Build.cmd -c Debug is getting stuck for some time (5+ minutes on my local machine) on generating tables:

          327 shift/reduce conflicts
          393 reduce/reduce conflicts
          consider setting precedences explicitly using %left %right and %nonassoc on terminals and/or setting explicit precedence on rules using %prec
  writing tables to log
          1881 states
          326 nonterminals
          205 terminals
          1087 productions
          #rows in action table: 1881

I've seen reports of this from other people, we should investigate. Can reproduce it roughly every other clean build.

Update 1:
Also, after terminating the build, and doing it again (without cleaning), it is stuck here for about 10 minutes already:

  FSharp.DependencyManager.Nuget -> C:\Users\vlza\code\fsharp\artifacts\bin\FSharp.DependencyManager.Nuget\Proto\netstandard2.0\FSharp.DependencyManager.Nuget.dll

Update 2: From the dump, it appears it spends a lot of time here:

C:\Users\vlza\code\fsharp\.dotnet\dotnet.exe  "C:\Users\vlza\code\fsharp\.dotnet\sdk\7.0.100-preview.7.22377.5\FSharp\fsc.dll" @"C:\Users\vlza\AppData\Local\Temp\tmp1a940496fb5f480798d2a8cccb94abe4.rsp"

[vzarytovskii]

@auduchinok You've mentioned that you have seen some slowness too, right?

[nojaf]

We are seeing this one right now.

[vzarytovskii]

So, judging by the tracing, it is doing something in background (i.e. not completely stuck), but is very slow.

[auduchinok]

@auduchinok You've mentioned that you have seen some slowness too, right?

Yes, it looked like the same place in the tables generation. I've used build.cmd -noVisualStudio on a clean local copy.

[vzarytovskii]

So, when building with .NET7-p7:

And when with .NET7-p2 (dev17.3 branch for example):

Spends almost 3 times less building compiler.

Will try and collect snapshot diffs.

Scale is different, but it's clear that it goes from the same stages in beginning (parsing probably?) and the end (emitting assembly?).

[vzarytovskii]

Some experimenting:
Same fsharp compiler repo commit (HEAD).
Build with preview2 of SDK - proto build is ok ~90s
Build with preview3 of SDK - proto build is very slow (8m and counting)

[dsyme]

From discussion with @Maoni0 and @mangod9

• The problem is related to .NET 7 enabling GC regions by default, particularly for "workstation GC". For example the initial compilation of the F# compiler:

  With .NET 6.0:

  > dotnet.exe C:\Program Files\dotnet\sdk\6.0.400\FSharp\fsc.dll"  @..\..\..\log.txt
  TIME:  0.4 Delta:  0.3 Mem:  98 G0:   6 G1:  3 G2:  2 [Import mscorlib and FSharp.Core.dll]
  TIME:  9.5 Delta:  9.1 Mem: 339 G0:  65 G1: 24 G2:  4 [Parse inputs]
  TIME:  9.5 Delta:  0.0 Mem: 339 G0:   0 G1:  0 G2:  0 [Import non-system references]
  TIME: 32.2 Delta: 22.7 Mem: 746 G0: 1063 G1: 163 G2:  1 [Typecheck]
   ...
  

  With .NET 7.0 Preview 7 on exactly same inputs

  > dotnet.exe "C:\Program Files\dotnet\sdk\7.0.100-preview.7.22377.5\FSharp\fsc.dll"  @..\..\..\log.txt
  TIME:  0.3 Delta:  0.2 Mem: 104 G0:   6 G1:  4 G2:  2 [Import mscorlib and FSharp.Core.dll]
  TIME:  8.3 Delta:  8.0 Mem: 373 G0:  65 G1: 31 G2: 16 [Parse inputs]
  TIME:  8.3 Delta:  0.0 Mem: 373 G0:   0 G1:  0 G2:  0 [Import non-system references]
  TIME: 223.4 Delta: 215.0 Mem: 832 G0: 1063 G1: 651 G2: 616 [Typecheck]
  

  Note the number of Gen2 collections in this phase of the compilation has gone up from 1 to 616.

• The very large slowdown disappears if we turn on ServerGC for fsc - this can be done globally via an environment variable or we can add it as a runtimeconfig.json for the F# compiler in the .NET SDK. However this is problematic as ServerGC is about 30% slower for the F# compiler.

[dsyme]

Repro.

1. Get this repo and checkout the current main:

   git clone https://github.com/dotnet/fsharp
   git checkout 75e1f5f1900a351ca036081cf2d1ed453ddf5ef1
   

2. .\build -noVisualStudio

   NOTE: this unfortunately takes about 15 minutes because the slowdown in the SDK compiler, but it gets all the right bits in place for the next step.

3. Then

    cd src\Compiler
   

4. Copy https://gist.github.com/dsyme/07499d160e3d16e60aa2af03742ba63e to args.txt and adjust paths to nugets (sorry)

5. Do the compilation with whichever F# compiler and .NET runtime you like, e.g.

    dotnet "C:\Program Files\dotnet\sdk\7.0.100-preview.7.22377.5\FSharp\fsc.dll"  @args.txt
   

   Or

    dotnet "C:\Program Files\dotnet\sdk\6.0.400\FSharp\fsc.dll"  @args.txt
   

   Note these run on .NET 7 and .NET 6 respectively. You can Ctrl-C after the "Typecheck" phase is reported, no need to wait

You can modify the settings these compilers are using:

To turn off GC regions for .NET 7

set COMPLUS_GCName=clrgc.dll

To turn on Server GC

Edit the appropriate fsc.runtimeconfig.json e.g.

 {
   "runtimeOptions": {
     ...
     "configProperties": {
       ...
       "System.GC.Server": true
     }
   }
 }

[vzarytovskii]

I guess as a temporary resolution, we should switch to old GC (via the runtimeconfig, if possible).

[TIHan]

Fascinating, I saw this happen yesterday too when building the compiler with my own built runtime to get SuperPMI collections.