Some persistent actors are stuck with RecoveryTimedOutException after circuit breaker opens

[object]

This issue looks similar to #3870, however it happens when using latest version of Akka.NET.

OS: Windows Server 2016
Platform: .NET Core 3.1
Akka.NET packages: 1.4.0-beta14 (used in a cluster)

Scenario:

1. Akka.Persistence.SqlServer.Journal.BatchingSqlServerJournal raises exception with message "Circuit Breaker is open; calls are failing fast", most likely due to a temporary db outage

2. Attempt to recover state of some persistent actors fail with RecoveryTimedOutException. Here's a typical sequence of events, taken from our log:

Started (Akka.Pattern.BackoffOnRestartSupervisor)
now supervising akka://Oddjob/system/sharding/upload/0/ps~msui30002111/msc:ps~msui30002111
now watched by [akka://Oddjob/system/sharding/upload/0/ps~msui30002111#1585193596]
now watched by [akka://Oddjob/system/recoveryPermitter#1099929798
Spawned MediaSetController actor
now watched by [akka://Oddjob/system/sharding/upload/0#1224240942]
Started (Akkling.Persistence.FunPersistentActor`1[System.Object])
Restoring state from snapshot
(after 1 minute)
["", null, "Akka.Persistence.RecoveryTimedOutException: Recovery timed out, didn't get event within 60s, highest sequence number seen 312."] {AckInfo} {Exception}
Passivating started on entity "ps~msui30002111"
received AutoReceiveMessage <Terminated>: [akka://Oddjob/system/sharding/upload/0/ps~msui30002111#1585193596] - ExistenceConfirmed=True
Entity stopped after passivation ["ps~msui30002111"]

3. Once a persistent actor fails with such exception, it is stuck until the system is restarted. Other actors may be successfully recovered.

[object]

And here's an extract from our HOCON:

akka {
  persistence{
    journal {
      plugin = "akka.persistence.journal.sql-server"
      sql-server {
        class = "Akka.Persistence.SqlServer.Journal.BatchingSqlServerJournal, Akka.Persistence.SqlServer"
        recovery-event-timeout = 60s
        schema-name = dbo
        table-name = EventJournal
        auto-initialize = off
      }
    }
    snapshot-store {
      plugin = "akka.persistence.snapshot-store.sql-server"
      sql-server {
        class = "Akka.Persistence.SqlServer.Snapshot.SqlServerSnapshotStore, Akka.Persistence.SqlServer"
        serializer = protobuf
        schema-name = dbo
        table-name = SnapshotStore
        auto-initialize = off
      }
    }
  }
}

[Aaronontheweb]

Think I have an idea on how to reproduce this with SQL Server - it's tough to reproduce in this repository with Sqlite because I can't easily simulate a database outage there. Should be doable with SQL Server though.

[object]

I once managed to trigger this scenario by simply executing a long running query that locked the whole EventJournal table. After the query was complete, persistent actors that failed during its execution were still stuck and I had to restart the service to be able to recover them.

[Aaronontheweb]

@object https://github.com/akkadotnet/Akka.Persistence.SqlServer/blob/dev/src/Akka.Persistence.SqlServer.Tests/SqlServerFixture.cs - we can add a method to stop the Docker container we use for integration testing in here and a second method to start it again. That would be a pretty robust way of doing it.

[ismaelhamed]

This still stands akkadotnet/Akka.Persistence.SqlServer#114 (comment)

[Aaronontheweb]

@ismaelhamed that would do it

[object]

@Aaronontheweb we are not using Docker for integration tests yet but stopping a container looks like a way to reproduce this scenario.

[Aaronontheweb]

I'm going to leave this issue open (edit: meaning myself and my team can't get on it right away) for the time being as we're pretty tied up with the 1.4.0 release (trying to get a release candidate with a stable API shipped today) - but I think we can get this reproduced and patched in short order.