ARM64 CG2 compilation of System.Text.Json crashes on JIT assert 'block->bbWeight > BB_ZERO_WEIGHT'

[trylek]

OS: Windows
Architecture: arm64
Example run: https://dev.azure.com/dnceng/public/_build/results?buildId=1136542&view=logs&j=438f2a33-0bac-577f-c1e5-b7956f9ac284&t=437a67a3-60d6-56bd-4640-6cece310571b

Diagnostic info:

16 / 257 (6%, 1 failed): failed in 9686 msecs, exit code -2147483645 = 0x80000003, expected 0: dotnet.exe D:\workspace\_work\1\s\artifacts\bin\coreclr\windows.arm64.Checked\x64\crossgen2\crossgen2.dll @D:\workspace\_work\1\s\artifacts\tests\coreclr\obj\windows.arm64.Checked\crossgen.out\System.Text.Json.dll.rsp
20 / 257 (6%, 1 failed): launching: D:\workspace\_work\1\s\.dotnet\dotnet.exe D:\workspace\_work\1\s\artifacts\bin\coreclr\windows.arm64.Checked\x64\crossgen2\crossgen2.dll @D:\workspace\_work\1\s\artifacts\tests\coreclr\obj\windows.arm64.Checked\crossgen.out\System.Data.OleDb.dll.rsp
  D:\workspace\_work\1\s\src\coreclr\jit\fgdiagnostic.cpp:2700
  Assertion failed 'block->bbWeight > BB_ZERO_WEIGHT' in 'System.Text.Json.JsonReaderHelper:IndexOfOrLessThan(byref,ubyte,ubyte,ubyte,int):int' during 'Optimize layout' (IL size 1021)

/cc @dotnet/crossgen-contrib @dotnet/jit-contrib

[mangod9]

assume this is a recent regression?

[trylek]

FWIW, on OSX ARM64 there's another non-deterministic issue, the CG2 compiler sometimes crashes on a nullref when internally sorting dependency nodes; that is demonstrated in the OSX arm64 leg of the same run as quoted in the title:

      Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at ILCompiler.Sorting.Implementation.MergeSortCore`5.ParallelSort(TDataStructure arrayToSort, Int32 index, Int32 length, TComparer comparer)
         --- End of inner exception stack trace ---
         at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
         at ILCompiler.Sorting.Implementation.MergeSortCore`5.ParallelSortApi(TDataStructure arrayToSort, TComparer comparer)
         at ILCompiler.MetadataManager.GetCompiledMethods(EcmaModule moduleToEnumerate, CompiledMethodCategory methodCategory)
         at ILCompiler.DependencyAnalysis.NodeFactory.EnumerateCompiledMethods(EcmaModule moduleToEnumerate, CompiledMethodCategory methodCategory)+MoveNext()
         at ILCompiler.DependencyAnalysis.ReadyToRun.ExceptionInfoLookupTableNode.LayoutMethodsWithEHInfo()
         at ILCompiler.DependencyAnalysis.ReadyToRun.ExceptionInfoLookupTableNode.ShouldSkipEmittingObjectNode(NodeFactory factory)
         at ILCompiler.DependencyAnalysis.ReadyToRun.HeaderNode.GetData(NodeFactory factory, Boolean relocsOnly)
         at ILCompiler.DependencyAnalysis.ReadyToRunObjectWriter.EmitPortableExecutable()
         at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile)
         at ILCompiler.Program.RunSingleCompilation(Dictionary`2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary`2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, CompilerTypeSystemContext typeSystemContext)
         at ILCompiler.Program.Run(String[] args)
         at ILCompiler.Program.Main(String[] args)

[mangod9]

I think the osx arm64 NRE failure is a known issue with the runtime, not specifically related to cg2 itself.

[trylek]

I see the first repro of the OSX arm64 bug in Steve MacLean's run from May 5 (last Wednesday):

https://dev.azure.com/dnceng/public/_build/results?buildId=1124050&view=ms.vss-test-web.build-test-results-tab&runId=34195084&resultId=100691&paneView=debug

I'm looking for the first occurrence of the Windows bug.

[trylek]

OK, so I see the first occurrence of the Windows bug in @BruceForstall's run from May 6:

https://dev.azure.com/dnceng/public/_build/results?buildId=1127214&view=results

[davidwrighton]

@AndyAyersMS is the expert for issues with basic block weight.

[AndyAyersMS]

I'll take a look at the windows arm64 failure.

[AndyAyersMS]

We end up with a negative block count and that leads to the assert. Suspect the issue is in fgConnectFallThrough.

BB56 [0106]  2       BB54,BB57             0.82  5123    [000..000)-> BB59 ( cond )                     i internal bwd bwd-target IBC 
BB57 [0107]  1       BB56                  0.82  5123    [000..000)-> BB56 ( cond )                     i internal bwd IBC 
BB59 [0109]  3       BB54,BB56,BB57        0.47  2915    [000..000)-> BB74 (always)                     i internal IBC 

Decided to reverse conditional branch at block BB56 branch to BB59 because of IBC profile data
fgFindInsertPoint(regionIndex=0, putInTryRegion=true, startBlk=BB01, endBlk=BB00, nearBlk=BB59, jumpBlk=BB00, runRarely=false)
Relocated uncommon block BB57 by reversing conditional jump at BB56
Relocated block [BB57..BB57] inserted after BB59
New Basic Block BB79 [0137] created.
Setting edge weights for BB57 -> BB79 to [-0.0002441406 .. 0]
Added an unconditional jump to BB59 after block BB57

BB56 [0106]  2       BB54,BB57             0.82  5123    [000..000)-> BB57 ( cond )                     i internal bwd bwd-target IBC 
BB59 [0109]  3       BB54,BB56,BB79        0.47  2915    [000..000)-> BB74 (always)                     i internal IBC 
BB57 [0107]  1       BB56                  0.82  5123    [000..000)-> BB56 ( cond )                     i internal bwd IBC 
BB79 [0137]  1       BB57                -2e-08     0    [???..???)-> BB59 (always)                     internal IBC

[AndyAyersMS]

Root cause seems to be upstream, when we compute edge weights we're willing to set a min edge weight to something below zero (and root cause of that is inconsistent profile data, which we have to tolerate).