Test failure JIT/Methodical/xxobj/sizeof/sizeof64_Target_64Bit_and_arm_il_r/sizeof64_Target_64Bit_and_arm.cmd

[JulieLeeMSFT]

Failed in run: runtime-coreclr outerloop 20230123.1

Failed tests:

R2R-CG2 windows arm Checked @ Windows.11.Arm64.Open
  - JIT/Methodical/xxobj/sizeof/sizeof64_Target_64Bit_and_arm_il_r/sizeof64_Target_64Bit_and_arm.cmd

Error message:

Fatal error. Internal CLR error. (0x80131506)
at System.GC.AllocateNewArray(IntPtr, Int32, GC_ALLOC_FLAGS)
at System.Text.StringBuilder.ExpandByABlock(Int32)
at System.Text.StringBuilder.Append(Char, Int32)
at System.Text.StringBuilder.Append(Char)
at System.Diagnostics.StackTrace.ToString(TraceFormat, System.Text.StringBuilder)
at System.Diagnostics.StackTrace.ToString(TraceFormat)
at System.Exception.get_StackTrace()
at System.Exception.ToString()
at Internal.JitInterface.CorInfoImpl.AllocException(System.Exception)
at Internal.JitInterface.CorInfoImpl._resolveToken(IntPtr, IntPtr*, Internal.JitInterface.CORINFO_RESOLVED_TOKEN*)
at Internal.JitInterface.CorInfoImpl.JitCompileMethod(IntPtr ByRef, IntPtr, IntPtr, IntPtr, Internal.JitInterface.CORINFO_METHOD_INFO ByRef, UInt32, IntPtr ByRef, UInt32 ByRef)
at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(ILCompiler.DependencyAnalysis.IMethodNode, Internal.IL.MethodIL)
at Internal.JitInterface.CorInfoImpl.CompileMethod(ILCompiler.DependencyAnalysis.ReadyToRun.MethodWithGCInfo, ILCompiler.Logger)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOneMethod|5(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>, Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOnThread|4(Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompilationThread|3(System.Object)
at System.Threading.Thread.StartCallback()

Return code:      1
Raw output file:      C:hwA74E094BwB27F0950uploads\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_routput.txt
Raw output:
BEGIN EXECUTION
sizeof64_Target_64Bit_and_arm.dll
TestLibrary.dll
2 file(s) copied.
11:39:15.16
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2sizeof64_Target_64Bit_and_arm.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp"   -r:C:hwA74E094BwB27F0950eJITMethodicalMethodical_r2IL-CG2*.dll  -r:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2*.dll"
Emitting R2R PE file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
" "dotnet" "C:hwA74E094Bp
2rdump
2rdump.dll" --header --sc --in C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll --out C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.r2rdump --val"
11:39:16.88
11:39:16.90
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2TestLibrary.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp"   -r:C:hw

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<<Main>$>g__TestExecutor7|0_6(<>c__DisplayClass0_0&)

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Failed in run: runtime-coreclr outerloop 20230123.1

Failed tests:

R2R-CG2 windows arm Checked @ Windows.11.Arm64.Open
  - JIT/Methodical/xxobj/sizeof/sizeof64_Target_64Bit_and_arm_il_r/sizeof64_Target_64Bit_and_arm.cmd

Error message:

Fatal error. Internal CLR error. (0x80131506)
at System.GC.AllocateNewArray(IntPtr, Int32, GC_ALLOC_FLAGS)
at System.Text.StringBuilder.ExpandByABlock(Int32)
at System.Text.StringBuilder.Append(Char, Int32)
at System.Text.StringBuilder.Append(Char)
at System.Diagnostics.StackTrace.ToString(TraceFormat, System.Text.StringBuilder)
at System.Diagnostics.StackTrace.ToString(TraceFormat)
at System.Exception.get_StackTrace()
at System.Exception.ToString()
at Internal.JitInterface.CorInfoImpl.AllocException(System.Exception)
at Internal.JitInterface.CorInfoImpl._resolveToken(IntPtr, IntPtr*, Internal.JitInterface.CORINFO_RESOLVED_TOKEN*)
at Internal.JitInterface.CorInfoImpl.JitCompileMethod(IntPtr ByRef, IntPtr, IntPtr, IntPtr, Internal.JitInterface.CORINFO_METHOD_INFO ByRef, UInt32, IntPtr ByRef, UInt32 ByRef)
at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(ILCompiler.DependencyAnalysis.IMethodNode, Internal.IL.MethodIL)
at Internal.JitInterface.CorInfoImpl.CompileMethod(ILCompiler.DependencyAnalysis.ReadyToRun.MethodWithGCInfo, ILCompiler.Logger)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOneMethod|5(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>, Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOnThread|4(Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompilationThread|3(System.Object)
at System.Threading.Thread.StartCallback()

Return code:      1
Raw output file:      C:hwA74E094BwB27F0950uploads\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_routput.txt
Raw output:
BEGIN EXECUTION
sizeof64_Target_64Bit_and_arm.dll
TestLibrary.dll
2 file(s) copied.
11:39:15.16
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2sizeof64_Target_64Bit_and_arm.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp"   -r:C:hwA74E094BwB27F0950eJITMethodicalMethodical_r2IL-CG2*.dll  -r:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2*.dll"
Emitting R2R PE file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
" "dotnet" "C:hwA74E094Bp
2rdump
2rdump.dll" --header --sc --in C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll --out C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.r2rdump --val"
11:39:16.88
11:39:16.90
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2TestLibrary.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp"   -r:C:hw

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<<Main>$>g__TestExecutor7|0_6(<>c__DisplayClass0_0&)

Author:	JulieLeeMSFT
Assignees:	markples
Labels:	area-CodeGen-coreclr, blocking-outerloop
Milestone:	-

[JulieLeeMSFT]

@markples, it is blocking outerloop. PTAL with a high priority.

[JulieLeeMSFT]

@trylek, PTAL. We are seeing the same error message from different test cases. #11360 and #81118.
Edit: corrected the issue number to 11360z

Fatal error. Internal CLR error. (0x80131506)
at System.GC.AllocateNewArray(IntPtr, Int32, GC_ALLOC_FLAGS)
at System.Text.StringBuilder.ExpandByABlock(Int32)
at System.Text.StringBuilder.Append(Char, Int32)
at System.Text.StringBuilder.Append(Char)
at System.Diagnostics.StackTrace.ToString(TraceFormat, System.Text.StringBuilder)
at System.Diagnostics.StackTrace.ToString(TraceFormat)
at System.Exception.get_StackTrace()
at System.Exception.ToString()
at Internal.JitInterface.CorInfoImpl.AllocException(System.Exception)
at Internal.JitInterface.CorInfoImpl._resolveToken(IntPtr, IntPtr*, Internal.JitInterface.CORINFO_RESOLVED_TOKEN*)
at Internal.JitInterface.CorInfoImpl.JitCompileMethod(IntPtr ByRef, IntPtr, IntPtr, IntPtr, Internal.JitInterface.CORINFO_METHOD_INFO ByRef, UInt32, IntPtr ByRef, UInt32 ByRef)
at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(ILCompiler.DependencyAnalysis.IMethodNode, Internal.IL.MethodIL)
at Internal.JitInterface.CorInfoImpl.CompileMethod(ILCompiler.DependencyAnalysis.ReadyToRun.MethodWithGCInfo, ILCompiler.Logger)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOneMethod|5(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>, Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOnThread|4(Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompilationThread|3(System.Object)
at System.Threading.Thread.StartCallback()

[markples]

A meta-question about the testing here - when I look at the devops job - https://dev.azure.com/dnceng-public/public/_build/results?buildId=144935&view=logs&j=454a72be-7065-5174-fb75-00cb418aebaf&t=46d7521e-696b-5674-5766-9cffa0247852 - the log file for this test isn't listed. The test is there in the devops test results integration, and if I go to https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata and manually find the logs I can see the failure there. Is there a problem with how test merging reports the overall job status back to devops?

Separately, and more directly relevant: do we have definitive instructions on how to repro a crossgen failure like this, especially one with "R2R-CG2 windows arm Checked @ Windows.11.Arm64.Open" that mentions both arm and arm64? I have a bunch of things to try, but I don't know how long it will take until one works and I have a repro to work with.

[JulieLeeMSFT]

Moving @AaronRobinsonMSFT ‘s comment to the right issue: #81120 (comment)