SqlSnapshotStore with autoinitialization stops if DB is temporrarily inaccessible

[balcko]

Hi,
we are using Akka.Net version 1.3.12 on production with clustering and SqlServer persistence.
Application is hosted as a service on Windows server with target framework net461.

Even after the fix of akkadotnet/Akka.Persistence.SqlServer#104 we still encountered an issue, that persistent actors could not start after the planned DB maintenance end. The only solution for us was to restart actor systems in whole cluster via pbm.
In logs we have found following errors related to SnapshotStore:

• Circuit Breaker is open; calls are failing fast
• Error during snapshot store initialization

I have visually debugged the SqlServerSnapshotStore code and found two issues there, which probably caused the actor to stop:

1. akka.net/src/core/Akka.Persistence/Snapshot/SnapshotStore.cs

   Line 105 in c8fa67c

   _breaker.WithCircuitBreaker(() => DeleteAsync(saveSnapshotFailure.Metadata));

If the DB becomes unavailable and saving snapshot fails so many times that circuit breaker opens, this line of code will immediately throw even without awaiting the task. This causes the actor to restart. Issue 2 happens afterwards

2. If autoinitiliaze setting is on (our case), after the actor restart, parent SqlSnapshotStore starts initialization. Since DB is still unavailable, it fails and as a result actor is permanently stopped on the line

   akka.net/src/contrib/persistence/Akka.Persistence.Sql.Common/Snapshot/SqlSnapshotStore.cs

   Line 139 in 8f9d19f

   Context.Stop(Self);

Now the whole actor system is screwed, since all new persistent actors can not start.
Fix could be following:

1. exception should definitely not restart the actor (maybe just log exception and swallow it). Another possible approach would be to skip the whole "rollback" delete after the failed save, since it will anyway probably fail and missed snapshot save should not matter, since snapshotting should be used only for optimizing replays and in this case if event has been successfully persisted before, then missed snapshot save won't affect event sourcing.

2. Actor should not stop if autoinitialize fails, the error should be logged and autoinitialization should be retried after some delay.

[ondrejpialek]

This is a duplicate of #3665, but you may close the original ticket as there is more info here (or I can copy over the details from the other).

[Aaronontheweb]

Thanks! I'll be looking into this.

[Aaronontheweb]

@ondrejpialek @balcko so the issue with the CircuitBreaker described in solution 1 was already addressed and fixed in Akka.NET v1.3.13: #3754 - I'd strongly recommend upgrading to 1.3.14 to get the latest fixes there.

I can work on the back-off issue with auto-initialize = on as that fix has merit on its own, but an immediate fix is available.

[ondrejpialek]

We turned off auto-initialize on staging and production so no longer affected by that problem (which made the issue more prevalent I guess?).

We did not encounter these issues since turning that off, but seeing the CircuitBreaker is still susceptible we will update soon :) Thanks!

[balcko]

Great, thanks, we will update to the latest version.
Regarding the auto-initialization, it is not that critical, but would be nice to have it for the future,thanks.