[EventPipe] Add EventPipeBuffer corruption diagnostics #118874

mdh1418 · 2025-08-19T01:51:16Z

On rare occasions, EventPipeBuffers may be corrupted during their lifetimes (the reason is currently unknown, but the main suspects are corruption by an external process or some uncaught use-after-free scenario). Once an EventPipeEventInstance is written into an EventPipeBuffer, there are no validation mechanisms guaranteeing that the data is uncorrupted by the time the event instance is read from the buffer. As such, there has been instances where the in-process event listener has been observed to hit an Access Violation due to EventPipeInternal_GetNextEvent returning a non-NULL pointer to corrupted bytes.

In order to better diagnose whether EventPipeBuffers in these scenarios are being corrupted internally or externally, this PR aims two opt-in EventPipeBuffer guarding mechanisms:

Header and Footer Guard signatures - To detect corruption and overrun
Memory Virtual Protection - To limit instances where some internals corrupt the EventPipeBuffer, helping distinguish between an internal and external corruption.

Behind an EventPipeBufferGuardLevel config switch (DOTNET_EventPipeBufferGuardLevel envvar), this PR adds two levels of increasing EventPipeBuffer protection.

DOTNET_EventPipeBufferGuardLevel	Behavior
0	No Protection, default
1	EventPipeBuffer Header and Footer guard signatures are active, and RaiseFailFastException triggers upon signature corruption detection. Memory Protection set to ReadOnly once the `EventPipeBuffer` is converted to read-only
2	In addition to level 1 behavior, EventPipeBuffer memory is set to ReadWrite during event writes, and ReadOnly at all other times after allocation

Header and Footer Guard Details

In the buffer header, we inject a magic + data relevant to the EventPipeBuffer's creation (timestamp, writing thread, and event sequence number), so in the event that the header/footer is partially corrupted, the remaining bytes can provide context for that buffer.

In the buffer footer, we inject a magic, it's inverse for a quick integrity check, a checksum computed from the header's identifiable bytes with a salt, and finally padding bytes to help detect buffer overrun + quick visual marker.

Note: To maintain EventPipeEventInstance 8-byte alignments, 32-bytes for each of the header and footer was determined to be a small enough overhead that provided a good starting point to diagnose buffer corruption.

Testing

Performed manual testing with a debugger, corrupting an EventPipeBuffer's guards before buffer_manager_move_next_event_any_thread during a call to ep_buffer_manager_get_next_event for an in-proc EventListener.

Previously, EventPipeBuffer's were not truly read-only when converted, as metadata IDs would be generated on the fly and written into the EventPipeEventInstance during event block writing. Instead, pass in the computed metadata ID separately, allowing for buffers to be protected with ClrVirtualProtect.

Copilot

Pull Request Overview

This PR adds diagnostic mechanisms to EventPipeBuffer to detect and diagnose memory corruption issues. The implementation introduces configurable buffer guard levels with header/footer signatures and memory protection to help distinguish between internal and external corruption sources.

Key changes include:

Addition of configurable EventPipeBufferGuardLevel with three protection levels (0-2)
Implementation of header and footer guard structures with magic values and checksums
Integration of memory protection using virtual memory APIs when guards are enabled

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
ep-types-forward.h	Adds forward declarations for guard structures and enums for protection levels
ep-rt.h	Adds runtime interface declarations for memory protection and fatal error handling
ep-file.c	Refactors to pass metadata_id parameter directly instead of storing in event instance
ep-event-instance.h/.c	Removes metadata_id field from EventPipeEventInstance structure
ep-buffer.h/.c	Core implementation of buffer guards with header/footer structures and validation
ep-buffer-manager.h/.c	Integrates guard level configuration and passes it to buffer allocation
ep-block.h/.c	Updates to accept metadata_id as parameter instead of extracting from event instance
ep-rt-*.h	Platform-specific implementations of memory protection and fatal error functions
clrconfigvalues.h	Adds configuration value for EventPipeBufferGuardLevel

Copilot · 2025-08-19T01:52:17Z

src/native/eventpipe/ep-buffer.c

+		// [ uint64 magic2 ][ uint64 magic2_inv ][ uint64 checksum ] and remaining bytes (if any) left as 0xEB.
+		uint8_t *footer_base_bytes = instance->limit - EP_BUFFER_FOOTER_GUARD_SIZE;
+		EventPipeBufferFooterGuard *footer = (EventPipeBufferFooterGuard *)footer_base_bytes;
+		memset (footer, 0xEB, EP_BUFFER_FOOTER_GUARD_SIZE);


The magic number 0xEB should be defined as a named constant (e.g., EP_BUFFER_FOOTER_PADDING_BYTE) to improve code maintainability and make its purpose clear.

Suggested change

memset (footer, 0xEB, EP_BUFFER_FOOTER_GUARD_SIZE);

memset (footer, EP_BUFFER_FOOTER_PADDING_BYTE, EP_BUFFER_FOOTER_GUARD_SIZE);

Copilot · 2025-08-19T01:52:17Z

src/native/eventpipe/ep-buffer.c

+
+	// Verify the remaining bytes in the footer guard are the fill value 0xEB.
+	for (size_t i = 0; i < sizeof(footer->padding); i++) {
+		if (footer->padding[i] != 0xEB)


The hardcoded 0xEB value should match the named constant suggested for line 56 to ensure consistency and maintainability.

Suggested change

if (footer->padding[i] != 0xEB)

if (footer->padding[i] != EP_BUFFER_FOOTER_PADDING_FILL)

Copilot · 2025-08-19T01:52:18Z

src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.h

+void
+ep_rt_fatal_error_with_message (const ep_char8_t *message)
+{
+	if (message != nullptr) {


Should use 'is not null' instead of '!= nullptr' according to the coding guidelines for null checks.

Copilot · 2025-08-19T01:52:18Z

src/native/eventpipe/ep-buffer.c

+
+	// Header trailing padding bytes should still be 0x00
+	for (size_t i = 0; i < sizeof(header->padding); i++) {
+		if (header->padding[i] != 0x00)


The hardcoded 0x00 value should be defined as a named constant (e.g., EP_BUFFER_HEADER_PADDING_BYTE) for consistency with the footer padding approach.

Suggested change

if (header->padding[i] != 0x00)

// Header trailing padding bytes should still be EP_BUFFER_HEADER_PADDING_BYTE

for (size_t i = 0; i < sizeof(header->padding); i++) {

if (header->padding[i] != EP_BUFFER_HEADER_PADDING_BYTE)

noahfalk

Looks good to me. A few nits inline and then then main item is we should make sure this works for NativeAOT too. I'm hoping we do that by shifting some of the runtime specific callouts into new minipal APIs.

noahfalk · 2025-10-14T22:28:51Z

src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.h

+    size_t length,
+    EventPipePageProtection protection)
+{
+    return true;


@jkotas - is there some guiding principle on what we add to minipal and what should stay out?

Today EventPipe code relies on runtime-specific callouts for various OS functionality but I'm hoping we can shift that trajectory to depend more on minipal directly. This spot seems like an appealing example where we'd like to invoke an OS API but the existing pattern takes us through runtime-specific wrappers first. Adding VirtualProtect to minipal and using it feels like a good approach to me but I want to make sure I'm not abusing the intent of minipal.

Good candidates for minipal are unambiguous trivial methods used from number of different places. "get current time stamp" is a perfect example.

minipal is not meant to wrap everything with platform specific implementation.

I am not sure whether mprotect is a good candidate for minimap. Most places that call mprotect tend to come with their own unique requirements.

This spot seems like an appealing example where we'd like to invoke an OS API but the existing pattern takes us through runtime-specific wrappers first.

This was setup years ago to work around build system limitations. It does not make sense. It would make a lot more sense for eventpipe to have OS-specific PALs and avoid trying to fit into runtime-specific wrappers.

Thanks Jan! We'll create our own EventPipe PAL as it sounds like these aren't appropriate for minipal.

The EventPipe runtime shim was originally put into place to make majority of EventPipe code runtime agnostic, but still reuse runtime unique implementation of low-level artifacts like IO, lock, threading, atomics, mainly keeping the EventPipe<->runtime interaction stable for CoreCLR while porting code from C++ to C and integrated with Mono, originally it even reused each runtime container implementation, but since then, I broken out that part into native/containers and if we don't think there is much value continue using runtime specific implementations of these low level artifacts shimmed by EventPipe runtime layer, then we should probably move towards one EventPipe OS PAL source file shared by Mono/CoreCLR/NAOT.

There will still be some runtime specific things that will stay in the runtime shim layer, but it will be smaller, and it will be simpler to run EventPipe standalone, making it simpler to get our low-level native runtime tests running outside of runtime. I believe some of these things would potentially end up in minipal, we already have some artifacts that could be used by EventPipe in minipal like, mutex, hig-res timers, utf8-ucs2 conversions and I had some volatile/atomics functions in another PR that would suite minipal and EventPipe as well.

I had an ambition for a long time to get our low-level native EventPipe and container tests currently under Mono, https://github.com/dotnet/runtime/tree/main/src/mono/mono/eventpipe/test, running as a runtime test (as a shared native library or separate binary executed from test) running on all runtimes, part of that was to make EventPipe less dependent on runtime specific artifacts, moving towards one EventPipe OS PAL shared by all runtimes. If we believe that is a good direction, then doing that work could start moving us towards an EventPipe OS PAL.

noahfalk · 2025-10-14T22:52:13Z

src/coreclr/nativeaot/Runtime/eventpipe/ep-rt-aot.h

+void
+ep_rt_fatal_error_with_message (const ep_char8_t *message)
+{
+    /* Not implemented, no-op */


We should make sure NativeAOT has an implementation of this too. Ideally similar to above we could have a shared minipal_raise_fatal_error() function that no longer needs runtime-specific callouts. Doing that depends on the fatal error handling staying simple, runtime agnostic, and directly aligning with underlying OS APIs.

fatal error handling staying simple, runtime agnostic, and directly aligning with underlying OS APIs.

Fatal error handling includes runtime-specific crash dump and watson logic currently...

noahfalk · 2025-10-14T23:15:09Z

src/native/eventpipe/ep-buffer-manager.c

 		instance->remaining_sequence_point_alloc_budget = sequence_point_allocation_budget;
 	}

+	buffer_guard_level = EP_MIN (ep_rt_config_value_get_buffer_guard_level (), EP_BUFFER_GUARD_LEVEL_PROTECT_OUTSIDE_WRITES);


It looks like this is trying to clamp the range into valid enum values. I'd suggest define EP_BUFFER_GUARD_LEVEL_MAX = EP_BUFFER_GUARD_LEVEL_PROTECT_OUTSIDE_WRITES in the enum definition, then use that MAX value here. It better conveys the intent and its more likely to get updated correctly if we ever change the levels in the future.

noahfalk · 2025-10-14T23:23:16Z

src/native/eventpipe/ep-buffer.c

 		buffer->current_read_event = NULL;
+
+	if (buffer->buffer_guard_level != EP_BUFFER_GUARD_LEVEL_NONE)
+		ep_buffer_ensure_guard_consistency (buffer);


I'd move the consistency check after the vprotect call to guarantee there is no time window where the memory can get corrupted after we check it and before it is protected.

noahfalk · 2025-10-14T23:26:35Z

src/native/eventpipe/ep-buffer.c

+	if (buffer->buffer_guard_level != EP_BUFFER_GUARD_LEVEL_NONE)
+		ep_buffer_ensure_guard_consistency (buffer);
+
+	if ((buffer->buffer_guard_level >= EP_BUFFER_GUARD_LEVEL_PROTECT_OUTSIDE_WRITES) &&


This if can be nested under the one at line 128 to make it clear that this code isn't on the hot path. The compiler may not be allowed to optimize that on its own if it has to conservatively assume that buffer->buffer_guard_level could change asynchronously.

if(level != NONE) { ... if(level >= PROTECT_OUTSIDE_WRITES) ... }

noahfalk · 2025-10-14T23:29:36Z

src/native/eventpipe/ep-buffer.c

+	if (buffer->buffer_guard_level != EP_BUFFER_GUARD_LEVEL_NONE)
+		ep_buffer_ensure_guard_consistency (buffer);
+
+	if ((buffer->buffer_guard_level >= EP_BUFFER_GUARD_LEVEL_PROTECT_ON_READONLY) &&


same thing, this if can be nested under the one at line 259 to ensure the compiler can keep it off the hot path.

jkotas · 2025-10-15T04:09:27Z

src/mono/mono/eventpipe/ep-rt-mono.h

+	uint32_t buffer_guard_level = 0;
+	gchar *value = g_getenv ("DOTNET_EventPipeBufferGuardLevel");
+	if (!value)
+		value = g_getenv ("COMPlus_EventPipeBufferGuardLevel");


We should not need to support for COMPlus_ for any new variables.

(We are keeping COMPlus_ path for now for existing variables. Ideally, we will delete it at some point.)

jkotas · 2025-10-15T04:56:41Z

On rare occasions, EventPipeBuffers may be corrupted during their lifetimes (the reason is currently unknown, but the main suspects are corruption by an external process or some uncaught use-after-free scenario).

I would bet on a race condition that leads to use-after-free.

We should have an issue that describes the observed problem and what we have found about the nature of the corruption so far. Once we trace it down, we may consider deleting this instrumentation or at least simplifying it - depending on what we find.

mdh1418 added 6 commits August 18, 2025 20:43

[ClrConfig] Add EventPipeBufferGuardLevel Config Switch

fb9b360

[EventPipeBuffer] Add Buffer Guard macros

6df06bc

[EventPipeBuffer] Inject buffer guards

a16ece5

[EventPipeBuffer] Ensure buffer guard consistent or FailFast

6a74500

[EventPipeBuffer] Add buffer ReadOnly protection

af7729e

mdh1418 added this to the 11.0.0 milestone Aug 19, 2025

mdh1418 requested review from Copilot and noahfalk August 19, 2025 01:51

mdh1418 requested review from MichalStrehovsky, lateralusX, steveisok and vitek-karas as code owners August 19, 2025 01:51

github-actions bot added the area-Tracing-coreclr label Aug 19, 2025

dotnet-policy-service bot assigned mdh1418 Aug 19, 2025

Copilot AI reviewed Aug 19, 2025

View reviewed changes

mdh1418 added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Aug 19, 2025

build-analysis bot mentioned this pull request Aug 19, 2025

System.Data.OleDb.Tests timeout in net48 x86 Release leg #87783

Closed

mdh1418 removed the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Oct 7, 2025

noahfalk reviewed Oct 14, 2025

View reviewed changes

jkotas reviewed Oct 15, 2025

View reviewed changes

mdh1418 mentioned this pull request Nov 3, 2025

[UserEvents] Add end-to-end runtime test #121316

Open

	memset (footer, 0xEB, EP_BUFFER_FOOTER_GUARD_SIZE);
	memset (footer, EP_BUFFER_FOOTER_PADDING_BYTE, EP_BUFFER_FOOTER_GUARD_SIZE);

	if (footer->padding[i] != 0xEB)
	if (footer->padding[i] != EP_BUFFER_FOOTER_PADDING_FILL)

[EventPipe] Add EventPipeBuffer corruption diagnostics #118874

Are you sure you want to change the base?

[EventPipe] Add EventPipeBuffer corruption diagnostics #118874

Uh oh!

Conversation

mdh1418 commented Aug 19, 2025

Header and Footer Guard Details

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

noahfalk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkotas Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lateralusX Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkotas commented Oct 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jkotas Oct 15, 2025 •

edited

Loading

lateralusX Oct 29, 2025 •

edited

Loading