Implement process and instance isolation #4498

jamescrosswell · 2025-09-05T03:15:08Z

Resolves #2033, Resolves #1067

Note

I'm not 100% happy with the structure of this code - it impacted quite a few mostly unrelated tests so it's kind of "leaky" at the moment. @Flash0ver if you've got ideas on alternative ways this could be done, I'm all ears.

Interprocess Locking

This turns out to be surprisingly difficult.

Something like a named Mutex could be used for this. However Mutexes are thread affine so they need to be released on the same thread that acquired them. Other than setting aside a dedicated thread for the lifetime of the Hub, there's no easy way to make that work.

A named Semaphore would be another option, but these aren't supported on macOS.

In the end I went with the "lock file" solution that @lucas-zimerman suggested. Keeping a lock file specific to each cache directory and opening this with FileShare.None gives us a kind of file based named mutex that we can use to avoid having multiple instances of the Hub trying to process files from the same cache directory.

…dotnet into isolated-cache

codecov · 2025-09-11T22:46:54Z

Codecov Report

❌ Patch coverage is 74.62687% with 34 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (version6@130650d). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/Sentry/Internal/Http/CachingTransport.cs	61.11%	17 Missing and 4 partials ⚠️
src/Sentry/Internal/CacheDirectoryCoordinator.cs	81.96%	9 Missing and 2 partials ⚠️
src/Sentry/Internal/ReadOnlyFilesystem.cs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             version6    #4498   +/-   ##
===========================================
  Coverage            ?   76.36%           
===========================================
  Files               ?      392           
  Lines               ?    14741           
  Branches            ?     2969           
===========================================
  Hits                ?    11257           
  Misses              ?     2772           
  Partials            ?      712

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bruno-garcia · 2025-10-05T23:20:37Z

@sentry review

src/Sentry/Internal/CacheDirectoryCoordinator.cs

seer-by-sentry · 2025-10-05T23:25:28Z

src/Sentry/Internal/Http/CachingTransport.cs

+        {
+            throw new InvalidOperationException("Cache directory already locked.");
+        }


The cache coordinator acquisition failure throws an InvalidOperationException, but this could make it impossible to create multiple SDK instances in some scenarios. Consider making this non-fatal and falling back to a non-cached transport or using a different isolation strategy.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

question: Could this be a severe issue?

But was should happen if this fails?
I believe the code path comes from creating a SentryClient via the Hub.

But we don't have too much control in this CachingTransport-.ctor here.

Hmmm ...
I wonder if we should avoid trying to Acquire the Lock-File in the .ctor,
but instead in the Initialize ... or the Worker-Task ... but then the error may happen quite late.

🤔 something is troubling me but I can't quite put my finger on it just yet 🤔

This is how the SDK currently works... so I don't believe it's an issue. If it is, I think we could address in a separate PR.

sentry-dotnet/src/Sentry/Internal/Http/CachingTransport.cs

Lines 81 to 83 in ee30f23

_isolatedCacheDirectoryPath =

options.TryGetIsolatedCacheDirectoryPath() ??

throw new InvalidOperationException("Cache directory or DSN is not set.");

seer-by-sentry · 2025-10-05T23:25:29Z

src/Sentry/Internal/ReadWriteFileSystem.cs

+    /// <remarks>
+    /// This method can throw all of the same exceptions that the <see cref="FileStream"/> constructor can throw. The
+    /// caller is responsible for handling those exceptions.
+    /// </remarks>
+    public override bool TryCreateLockFile(string path, out Stream fileStream)
+    {
+        // Note that FileShare.None is implemented via advisory locks only on macOS/Linux... so it will stop
+        // other .NET processes from accessing the file but not other non-.NET processes. This should be fine
+        // in our case - we just want to avoid multiple instances of the SDK concurrently accessing the cache
+        fileStream = new FileStream(
+            path,
+            FileMode.OpenOrCreate,
+            FileAccess.ReadWrite,
+            FileShare.None);
+        return true;
+    }


The TryCreateLockFile method can throw all FileStream constructor exceptions but always returns true. This is inconsistent with the interface design pattern where 'Try' methods typically return false on failure rather than throwing. Consider catching exceptions and returning false, or rename the method to indicate it can throw.

Suggested change

/// <remarks>

/// This method can throw all of the same exceptions that the <see cref="FileStream"/> constructor can throw. The

/// caller is responsible for handling those exceptions.

/// </remarks>

public override bool TryCreateLockFile(string path, out Stream fileStream)

{

// Note that FileShare.None is implemented via advisory locks only on macOS/Linux... so it will stop

// other .NET processes from accessing the file but not other non-.NET processes. This should be fine

// in our case - we just want to avoid multiple instances of the SDK concurrently accessing the cache

fileStream = new FileStream(

path,

FileMode.OpenOrCreate,

FileAccess.ReadWrite,

FileShare.None);

return true;

}

public override bool TryCreateLockFile(string path, out Stream fileStream)

{

try

{

fileStream = new FileStream(

path,

FileMode.OpenOrCreate,

FileAccess.ReadWrite,

FileShare.None);

return true;

}

catch

{

fileStream = Stream.Null;

return false;

}

}

_{Did we get this right? 👍 / 👎 to inform future reviews.}

This is consistent with the behaviour for CreateFileStream and makes it easier to surface errors to callers... so I'd rather stick with the API we have.

src/Sentry/SentrySdk.cs

src/Sentry/Internal/CacheDirectoryCoordinator.cs

Flash0ver · 2025-11-03T20:26:04Z

test/Sentry.Tests/HubTests.cs

+        if (options.Transport is CachingTransport cachingTransport)
+        {
+            cachingTransport.Dispose(); // Release cache lock so that the cacheDirectory can be removed


question: Don't we need to do this for every test?

The method CapturesEventWithContextKey_Implementation does it too?

I think we only need to do it when options.CacheDirectoryPath is set... it's just to ensure the cache dir gets cleaned up. Most tests aren't using a cache dir.

Flash0ver · 2025-11-03T20:27:15Z

test/Sentry.Tests/SentryClientTests.cs

        var cachingTransport = Assert.IsType<CachingTransport>(_fixture.SentryOptions.Transport);
        _ = Assert.IsType<FakeTransport>(cachingTransport.InnerTransport);
+
+        cachingTransport.Dispose(); // Release cache lock so that the cacheDirectory can be removed


question: don't we need to do this on a Test-Class level instead, so that it applies to every Test-Method?

It's only necessary because we set CacheDirectoryPath on line 1628.

It's a bit messy but only really a problem in tests, I think... SDK users won't need to worry about this.

Flash0ver · 2025-11-03T20:37:58Z

src/Sentry/Internal/Http/CachingTransport.cs

+        {
+            throw new InvalidOperationException("Cache directory already locked.");
+        }


question: Could this be a severe issue?

But was should happen if this fails?
I believe the code path comes from creating a SentryClient via the Hub.

But we don't have too much control in this CachingTransport-.ctor here.

Hmmm ...
I wonder if we should avoid trying to Acquire the Lock-File in the .ctor,
but instead in the Initialize ... or the Worker-Task ... but then the error may happen quite late.

🤔 something is troubling me but I can't quite put my finger on it just yet 🤔

src/Sentry/Internal/Http/CachingTransport.cs

Flash0ver · 2025-11-03T20:46:59Z

src/Sentry/Internal/Http/CachingTransport.cs

+
+            _options.LogDebug("found abandoned cache: {0}", dir);
+            using var coordinator = new CacheDirectoryCoordinator(dir, _options.DiagnosticLogger, _options.FileSystem);
+            if (!coordinator.TryAcquire())


question: Could we run into a concurrency issue?

E.g.

this recycler has started and tries to acquire a lock-file

in the .ctor we have also acquired a lock-file already, in a directory below

now we try to move files that contain a lock-file that we still have a reference via Stream on

Essentially now retrying 3 times without any effect?
But I may misunderstand something there 🤔

The lock files sit in the parent/root cache directory. The isolated cache folders are all children of that root folder, so none of the operations on the isolated cache folders should affect the locks. I've added a code comment to make this clearer:

sentry-dotnet/src/Sentry/Internal/CacheDirectoryCoordinator.cs

Lines 22 to 23 in ada094b

// Note this creates a lock file in the cache directory's parent directory... not in the cache directory itself

_lockFilePath = $"{cacheDir}.lock";

github-actions · 2025-11-05T01:04:49Z

	Messages
📖	Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against efaa83e

jamescrosswell · 2025-11-05T03:30:06Z

When I changed the merge target, it didn't automatically trigger a build, so I triggered this manually:
https://github.com/getsentry/sentry-dotnet/actions/runs/19088039748

All green 🟢

jamescrosswell · 2025-11-05T03:38:33Z

@Flash0ver I think the only remaining piece of feedback to address (not sure what happened to it) is how we handle upgrading from previous SDK versions, if people have files in their cache when they do so (since we're changing the cache directory).

Options:

Deal with it via upgrade docs, in the Breaking Changes section of the changelog
Modify the code to automatically import envelopes from the cache root directory (we could maybe remove that code in a future v7 release).

I think since this could potentially result in lost client data, it's worth opting for door number 2 here... but only if we're broadly happy with the PR aside from that.

I addressed all the review feedback and changed the merge target to version 6.

jamescrosswell and others added 21 commits September 5, 2025 15:12

Implement process and instance isolation

83a3c75

Experimenting with named mutexes and semaphores

5e2c12e

Format code

7382cac

Update CachingTransport.cs

ec60ca1

Update CacheDirectoryCoordinator.cs

dcec5e0

Switched to a file lock

220884b

Update Sentry.csproj

180ff53

Format code

03c998c

Fixed tests

71ed31b

Update CachingTransportTests.cs

d5dc55a

Merge branch 'isolated-cache' of https://github.com/getsentry/sentry-…

3ae4d96

…dotnet into isolated-cache

Update CHANGELOG.md

0046933

.

d6dd817

Fixed mobile tests

90ede6b

Format code

c26c5bd

Windows tests

39be526

Create CacheDirectoryCoordinatorTests.cs

c85161f

Update SentryOptionsTests.cs

536d9e2

Merge branch 'isolated-cache' of https://github.com/getsentry/sentry-…

477361b

…dotnet into isolated-cache

Added tests for SalvageAbandonedCacheSessions

ee30f23

Update CacheDirectoryCoordinatorTests.cs

cd0848d

jamescrosswell marked this pull request as ready for review September 11, 2025 22:25

jamescrosswell requested a review from Flash0ver as a code owner September 11, 2025 22:25

This comment was marked as outdated.

Sign in to view

jamescrosswell added 2 commits September 12, 2025 10:30

Merge branch 'main' into isolated-cache

b763698

Update CHANGELOG.md

a7b0656

jamescrosswell added 2 commits September 19, 2025 22:26

Merge branch 'main' into isolated-cache

78f1a3a

Update CHANGELOG.md

29fa252

seer-by-sentry bot reviewed Oct 5, 2025

View reviewed changes

src/Sentry/Internal/CacheDirectoryCoordinator.cs Outdated Show resolved Hide resolved

seer-by-sentry bot reviewed Oct 5, 2025

View reviewed changes

src/Sentry/Internal/CacheDirectoryCoordinator.cs Outdated Show resolved Hide resolved

seer-by-sentry bot reviewed Oct 5, 2025

View reviewed changes

src/Sentry/SentrySdk.cs Outdated Show resolved Hide resolved

Flash0ver reviewed Nov 3, 2025

View reviewed changes

jamescrosswell added 2 commits November 5, 2025 13:51

Review feedback

ada094b

Merge remote-tracking branch 'origin/version6' into isolated-cache

efaa83e

jamescrosswell changed the base branch from main to version6 November 5, 2025 01:04

jamescrosswell requested a review from Flash0ver November 5, 2025 03:30

	_isolatedCacheDirectoryPath =
	options.TryGetIsolatedCacheDirectoryPath() ??
	throw new InvalidOperationException("Cache directory or DSN is not set.");

	// Note this creates a lock file in the cache directory's parent directory... not in the cache directory itself
	_lockFilePath = $"{cacheDir}.lock";

Uh oh!

Implement process and instance isolation #4498

Are you sure you want to change the base?

Implement process and instance isolation #4498

Conversation

jamescrosswell commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Interprocess Locking

Uh oh!

This comment was marked as outdated.

Uh oh!

codecov bot commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bruno-garcia commented Oct 5, 2025

Uh oh!

Uh oh!

Uh oh!

seer-by-sentry bot Oct 5, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seer-by-sentry bot Oct 5, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jamescrosswell Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 5, 2025

Uh oh!

jamescrosswell commented Nov 5, 2025

Uh oh!

jamescrosswell commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jamescrosswell commented Sep 5, 2025 •

edited

Loading

codecov bot commented Sep 11, 2025 •

edited

Loading

jamescrosswell Nov 4, 2025 •

edited

Loading