EventListenerThreadPool test failing in nativeaot outerloop runs

[MichalStrehovsky]

BEGIN EXECUTION
/datadisks/disk1/work/A6FD09A7/p/nativeaottest.sh /datadisks/disk1/work/A6FD09A7/w/A12B08E5/e/tracing/eventlistener/EventListenerThreadPool/ EventListenerThreadPool.dll ''
Test Failed: Did not see all of the expected events.
ThreadPoolIOPack: 0
ThreadPoolIOEnqueue: 0
ThreadPoolIODequeue: 0
Xunit.Sdk.EqualException: Assert.Equal() Failure: Values differ
Expected: 100
Actual:   -1
   at Xunit.Assert.Equal[T](T, T, IEqualityComparer`1) + 0x1a1
   at __GeneratedMainWrapper.Main() + 0x38
Expected: 100
Actual: 101
END EXECUTION - FAILED

This is consistently failing in the Pri0 outerloop runs on all architectures.

Last good run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=745467&view=results
First bad run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=746036&view=results

From this I suspected #103675 and a revert PR at #105543 confirmed the suspicion (notice Pri0 legs are green in the revert PR). Note that PR reverts two commits because there was a merge conflict I didn't bother resolving, but the other commit is from yesterday and this is failing for a week.

Cc @eduardo-vp @kouvel

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[MichalStrehovsky]

Disabling the test in #105557 to get clean(er) outerloops again.

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[davmason]

I helped @eduardo-vp looked in to this failure, it doesn't seem like it is specific to the event he added.

@LakshanF is this scenario (using an EventListener to listen to native runtime events) expected to work under NativeAOT?

When I debugged the failing test under NativeAOT I saw it fail in this callstack:

	EventListenerThreadPool.exe!S_P_CoreLib_System_Diagnostics_Tracing_EventPipePayloadDecoder__DecodePayload() Line 18	Unknown
 	EventListenerThreadPool.exe!S_P_CoreLib_System_Diagnostics_Tracing_NativeRuntimeEventSource__ProcessEvent() Line 60	Unknown
 	EventListenerThreadPool.exe!S_P_CoreLib_System_Diagnostics_Tracing_EventPipeEventDispatcher__DispatchEventsToEventListeners() Line 169	Unknown
 	EventListenerThreadPool.exe!S_P_CoreLib_System_Diagnostics_Tracing_EventPipeEventDispatcher___c__DisplayClass12_0___StartDispatchTask_b__0() Line 139	Unknown
 	EventListenerThreadPool.exe!S_P_CoreLib_System_Threading_ExecutionContext__RunInternal() Line 179	Unknown
 	EventListenerThreadPool.exe!S_P_CoreLib_System_Threading_Tasks_Task__ExecuteWithThreadLocal() Line 2342	Unknown
 	EventListenerThreadPool.exe!S_P_CoreLib_System_Threading_Thread__StartThread() Line 446	Unknown
 	EventListenerThreadPool.exe!S_P_CoreLib_System_Threading_Thread__ThreadEntryPoint() Line 226	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!00007ffc2e58af28()	Unknown

The failure is because m_eventData is not populated for NativeRuntimeEventSource and we use m_eventData as a template for how to parse the appropriate events coming from the runtime.

I was able to debug through and see the code for creating the metadata seems to be running as expected, except we don't actually have the metadata in memory after. The code in question is EventSource.CreateManifestAndDescriptors. We call AddEventDescriptor for the events here, but the change doesn't seem to actually happen to the original array after returning, even though it is passed by ref.

@MichalStrehovsky are you able to help determine if this is a bad codegen issue or some other? Maybe the debugger is not showing the right memory?

[MichalStrehovsky]

Since this reproes on all architectures and all OSes, I'd consider a codegen issue less likely.

We run many of the tracing tests in the src/tests/tracing suite so my assumption is that this works in general but then again I would not be able to tell a difference between an event source and a metric, so take it with a grain of salt. @LakshanF knows most of native AOT specific aspects of this and any known issues.