Epic: Add Connectivity Health Checks for Akka.Persistence Plugins

[Aaronontheweb]

Background

Akka.Persistence health checks currently rely solely on CircuitBreaker-based readiness checks. These only detect failures when live operations actually fail, missing backend connectivity issues during idle periods.

Problem

• Silent failures during idle periods: Database outages overnight go undetected until first user interaction at 8am
• No proactive monitoring: Operations teams cannot respond to infrastructure issues before they impact users
• Gap between readiness and liveness: Current implementation serves as readiness check only - relies on the CircuitBreaker which only trips when actual persistence operations fail

Solution

Add connectivity checks as a new capability for persistence plugins - active probes that verify backend connectivity regardless of recent operation activity.

Inspired by community feedback around business hour bound applications that need proactive database monitoring.

API Design

Minimal - use all defaults

builder.WithJournal(
    new SqlServerJournalOptions { ... },
    journal => journal
        .WithHealthCheck()              // Existing readiness check (CircuitBreaker)
        .WithConnectivityCheck());      // NEW connectivity check (active probe)

With customization

builder.WithJournal(
    new SqlServerJournalOptions { ... },
    journal => journal
        .WithHealthCheck()
        .WithConnectivityCheck(
            unHealthyStatus: HealthStatus.Unhealthy,
            name: "custom-journal-connectivity"));

Same pattern for snapshot stores

builder.WithSnapshot(
    new SqlServerSnapshotOptions { ... },
    snapshot => snapshot
        .WithHealthCheck()
        .WithConnectivityCheck());

Configuration Parameters

Parameter	Type	Default	Description
unHealthyStatus	HealthStatus	Unhealthy	Status when connectivity check fails
name	string?	Akka.Persistence.{Journal|SnapshotStore}.{id}.Connectivity	Health check name for ASP.NET Core
tags	(hardcoded)	[akka, persistence, journal/snapshot-store, connectivity]	Will be customizable via #680

Target Plugins

Epic should include implementation in:

• Akka.Persistence.Sql.Hosting (SQL Server, PostgreSQL, SQLite)
• Akka.Persistence.Azure.Hosting (separate sub-issue)
• Akka.Persistence.MongoDb.Hosting (separate sub-issue)
• Akka.Persistence.Redis.Hosting (separate sub-issue)

Key Design Decisions

1. Separate from base classes: No changes to AsyncWriteJournal / SnapshotStore in core Akka.Persistence
2. Opt-in: Users explicitly call .WithConnectivityCheck() to enable
3. Independent: Can use readiness check, connectivity check, or both
4. Plugin-specific: Each plugin implements own lightweight probe
5. Follows existing patterns: Uses IAkkaHealthCheck, AkkaHealthCheckRegistration, and builder chaining
6. Consistent parameters: Matches WithHealthCheck() signature structure

Related Enhancements

• Enhancement: Add tags parameter to AkkaPersistenceJournalBuilder.WithHealthCheck() and AkkaPersistenceSnapshotBuilder.WithHealthCheck() #680: Add customizable tags parameter to existing WithHealthCheck() methods

Success Criteria

• Connectivity checks successfully detect database/backend unavailability
• Checks can be added without changing existing code patterns
• Minimal performance overhead from probes
• Consistent implementation across all persistence plugins
• Unit tests covering normal and error paths
• Documentation updated with examples
• Backward compatible - no breaking changes
• Ready for v1.5.54+ release

Notes

• Connectivity checks should run independently of persistence operation activity
• This addresses use cases where database failures occur during idle periods and must be detected proactively
• Probe timeout and tags customization planned as follow-up enhancements