FreeBSD tests crash with HRESULT: 0x8007FF02 from #106724

[Thefrank]

Description

From #106724 forward tests on FreeBSD now all creash with HRESULT: 0x8007FF02

These do not show up in /var/log/messages on either the jail or the host. dmesg also free of errors from this. Nothing SIGXXXX.
Builds otherwise complete without issue.

Reproduction Steps

As initially discovered:
./build.sh -ci -c Release -subset Clr+Mono+Host.Native+Host.Tools+Host.Pkg+Libs+Libs.Tests+Packs --test
Double checked with (requires #106302):
src/tests/build.sh Release /p:LibrariesConfiguration=Release -rebuild -runtests

Expected behavior

Tests to run either pass/fail

Actual behavior

"exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill."

or

"Failed to create CoreCLR, HRESULT: 0x8007FF02"

Regression?

Yes

Known Workarounds

None?

Configuration

FreeBSD-x64 both 13.3 and 14.1
System has about ~200G of free RAM during this. Jails run without resource quotas.

Other information

Public log for both FreeBSD 13.3 and 14.1: https://dev.azure.com/IFailAt/freebsd-dotnet-runtime-nightly/_build/results?buildId=1625&view=results BinLog are published to artifacts

commit listed is 6df7807 (after PR listed)

git bisect log

git bisect start
# status: waiting for both good and bad commits
# good: [21fe57174af159f7bcb39f591cd9fbb96a36bfcb] COMObject invoke through reflection (#106677)
git bisect good 21fe57174af159f7bcb39f591cd9fbb96a36bfcb
# status: waiting for bad commit, 1 good commit known
# bad: [6df7807617cff3e4b4eba7bb33c8ad00c2b1cf1c] Add missing .alt_entry to CoreCLR *_FakeProlog methods (#106744)
git bisect bad 6df7807617cff3e4b4eba7bb33c8ad00c2b1cf1c
# good: [a3cbf8c2534040faa4664e45c19a0ba877bb5c5c] Grab back some performance in `RuntimeTypeHandle::GetElementType` (#106730)
git bisect good a3cbf8c2534040faa4664e45c19a0ba877bb5c5c
# good: [4e8892ea65fe85593b1853907d965fb49d324ce9] JIT: Add Non-Faulting behaviour for Sve.LoadVector*NonFaulting*() (#106648)
git bisect good 4e8892ea65fe85593b1853907d965fb49d324ce9
# good: [cfa0cc5b0282c29597f10b6c2fdceab159e0be81] Inject IJW Copy Constructor calls in the JIT instead of in the IL stubs (#106424)
git bisect good cfa0cc5b0282c29597f10b6c2fdceab159e0be81
# good: [cfa8da560f674d035a26e9b3b07067933d46e44c] Avoid signed overflow in DBG_FlushInstructionCache (#105918)
git bisect good cfa8da560f674d035a26e9b3b07067933d46e44c
# bad: [92e41b6b2cf7aea845fcfc9a439dc4e191c07428] Add callback-based Encode to AsnWriter
git bisect bad 92e41b6b2cf7aea845fcfc9a439dc4e191c07428
# bad: [27ee590d1f96dc590f8ad85e76b7b8a6ad000b2d] pal init: InitializeFlushProcessWriteBuffers() before first thread to improve start time (#106724)
git bisect bad 27ee590d1f96dc590f8ad85e76b7b8a6ad000b2d
# first bad commit: [27ee590d1f96dc590f8ad85e76b7b8a6ad000b2d] pal init: InitializeFlushProcessWriteBuffers() before first thread to improve start time (#106724)

[harisokanovic]

@Thefrank , can you recommend a guide for building the v9 SDK on FreeBSD? https://github.com/dotnet/dotnet/ fails with ./prep-source-build.sh: line 199: /dotnet-dotnet/.dotnet/dotnet: No such file or directory.

[Thefrank]

Building the SDK requires bootstraping from Linux or using an old SDK to build the new one. If you just want a prebuilt we have those too.

Building:
Bootstraping will require: #105004 #105587 and optionally #105587. Native builds only need #105587. If you want a collection of scripts for native building, there is this

I have not tried to build from VMR for net9 as the VMR tends to require more work than "bootstrap from tags" or "use old SDK to build new SDK" :(

Downloading:
Both I and @sec have net9p7 SDKs prebuilt: Here or Here

The former repo is what I use for my builds; the later by @sec also has ARM64 versions and will work just as well for either AMD64 or ARM64. I cherry-pick missing/needed PRs into my SDKs (e.g., the ones listed above). If you want an SDK with 0 changes @sec's are better.

Please let me know how else I can help with this!

[JulieLeeMSFT]

Not a codegen issue. Removing codegen label.

[cperciva]

FreeBSD dev here. Just to confirm, this is running in native FreeBSD, not via the Linux emulation layer?

[cperciva]

Does this process ever call execve? The "process has called MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED" flag gets cleared when we execve and that would result in MEMBARRIER_CMD_PRIVATE_EXPEDITED returning EPERM later.

[Thefrank]

It is running on native FreeBSD (no linux kernel modules, linux service not running) on both host and jail. allow.mlock=1 is set for the jail where the build system is located.

[cperciva]

@Thefrank And does is call execve at any point?

[Thefrank]

@cperciva Yes. I have attached a truss.txt (truss -fadeH -o truss.txt /root/runtime/artifacts/log/repro/ilasm/PortablePdb/IlasmPortablePdbTests/repro_IlasmPortablePdbTests.sh). Test selected at random.
truss.txt