C000007B in CoreCLR Pri0 Test Run Windows_NT arm checked

[trylek]

First known occurrence:

https://dev.azure.com/dnceng/public/_build/results?buildId=462402&view=results

Commit info:

HEAD is now at 7dc30fc1 Merge 03693d335194dd76177cec10ac159f32de973082 into 0a29c61468f60f815119e52e477d0154351a4abf

Proximate failure:

C:\dotnetbuild\work\100ccf8f-adbf-49ff-87b2-37c7a2ef03ce\Work\c74808db-757a-43c3-a1f2-ec1c841e07ff\Exec>C:\dotnetbuild\work\100ccf8f-adbf-49ff-87b2-37c7a2ef03ce\Payload\CoreRun.exe C:\dotnetbuild\work\100ccf8f-adbf-49ff-87b2-37c7a2ef03ce\Payload\xunit.console.dll JIT\Stress\JIT.Stress.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -trait TestGroup=JIT.Stress 

C:\dotnetbuild\work\100ccf8f-adbf-49ff-87b2-37c7a2ef03ce\Work\c74808db-757a-43c3-a1f2-ec1c841e07ff\Exec>set _commandExitCode=-1073741701 

C000007B = STATUS_INVALID_IMAGE_FORMAT

As this happens on ARM, a usual cause is erroneously mixing up ARM and Intel binaries.

[trylek]

/cc @dagood

[danmoseley]

@jkotas can this be closed now you reverted the commit?

[jkotas]

We have been dealing with several different CI breaks. The revert fixed the failures in runtime-live-build (Runtime Build Windows_NT x86 release).

This issue tracks failures in CoreCLR Pri0 Test Run Windows_NT arm. It is not fixed yet. I was not able to pin point it to any single commit. I believe that it was likely introduced by some kind of ambient infrastructure change.

[trylek]

I experimentally tried to roll back two commits merged in between the last passing and the first failing run that seemed potentially related (Leandro's change to assembly resolution logic and JanV's merge of Viktor's build cleanup) but, as JanK rightly pointed out, sadly neither fixed the regression.

Today I tried a different approach: I checked out a private branch off the "old coreclr" repo master, I backported Jarret's change re-enabling ARM runs and I ran an "old coreclr PR pipeline" on the change:

https://dev.azure.com/dnceng/public/_build/results?buildId=466805&view=logs&j=41021207-15b4-5953-02cc-987654ff0f7b&t=786f87fb-d4be-5ad9-2d0b-e89af519eba6

Apparently it failed with exactly the same failure as the new runs. I believe this confirms that the failure is more likely related to the ambient state of the ARM machines than to "real" pipeline changes, just as JanK speculated in his previous response. It's probably time to start talking to infra guys, initially perhaps @ilyas1974, @MattGal or @Chrisboh, but I doubt anyone will be able to shed more light on this before the end of the holidays.

[ilyas1974]

Where there specific questions you had about the ARM hardware being used? The current hardware being used for Windows ARM64 runs is running a newer build of Windows that we were previously using as well as Python 3.7.0 - the x86 version.