feat(remote): add custom certificate validation callbacks and fix asymmetric hostname validation

[Aaronontheweb]

Problem Statement

Recent mTLS security improvements (PRs #7851, #7897, #7891) provide strong foundational security, but expose two issues:

Issue 1: Asymmetric Hostname Validation

• Client-side: Validates server certificate hostname when validate-certificate-hostname = true ✅
• Server-side: IGNORES the setting when validating client certificates ❌

Impact: In a cluster with per-node certificates, servers accept any client cert with valid chain, regardless of hostname. Compromised nodes could connect with stolen certs.

Example: Cluster (node1.example.com, node2.example.com) with validate-certificate-hostname = true

• Client → Server: Validates hostname correctly ✅
• Server → Client: Accepts ANY valid cert chain ❌

Issue 2: Limited Validation Scenarios

Current config-based validation only supports:

• Chain validation (CA trust)
• Hostname matching
• Certificate presence

Users cannot easily implement custom validation for:

• Certificate pinning (whitelist known cert thumbprints)
• Subject/Issuer matching (organizational CA validation)
• Peer identity verification
• Custom business rules

Proposed Solution

1. Custom Validation Callbacks (Primary)

Add high-level CertificateValidationCallback delegate that users can implement or compose from helpers:

// User-facing delegate (more useful than RemoteCertificateValidationCallback)
public delegate bool CertificateValidationCallback(
    X509Certificate2 certificate,
    X509Chain chain,
    string remotePeer,        // Actual peer identifier
    SslPolicyErrors errors,
    ILoggingAdapter log);     // For diagnostics

// Usage: provide custom validator
var setup = new DotNettySslSetup(cert, suppressValidation: false,
    requireMutualAuthentication: true,
    customValidator: MyValidationStrategy);

2. Pre-Built Helper Functions

Make common scenarios easy without writing custom logic:

// All available via CertificateValidation factory
CertificateValidation.ValidateChain(log)                 // CA chain validation
CertificateValidation.ValidateHostname(expectedHostname, log)  // CN/SAN matching
CertificateValidation.PinnedCertificate(thumbprints)    // Certificate pinning
CertificateValidation.ValidateSubject(pattern, log)     // Subject DN matching (with wildcards)
CertificateValidation.ValidateIssuer(pattern, log)      // Issuer DN matching
CertificateValidation.Combine(validators...)            // Compose multiple validators
CertificateValidation.ChainPlusThen(customCheck, log)   // Chain validation + custom logic

3. Real-World Examples

Production TLS with pinning:

var validator = CertificateValidation.Combine(
    CertificateValidation.ValidateChain(Log),
    CertificateValidation.ValidateHostname(log: Log),
    CertificateValidation.PinnedCertificate(knownThumbprints)
);

Internal cluster with per-node certs:

var validator = CertificateValidation.Combine(
    CertificateValidation.ValidateChain(Log),
    CertificateValidation.ValidateHostname(log: Log),
    CertificateValidation.ValidateIssuer("CN=Our-Internal-CA", Log)
);

Custom peer validation:

var validator = CertificateValidation.ChainPlusThen(
    customCheck: (cert, chain, peer) =>
    {
        // Only accept certs for authorized nodes
        var cn = cert.GetNameInfo(X509NameType.DnsName, false);
        return authorizedNodeNames.Contains(cn);
    },
    log: Log
);

Implementation Details

Changes Required

1. DotNettySslSetup.cs: Add CustomValidator property (new optional constructor parameter)
2. DotNettyTransportSettings.cs:
   • Define CertificateValidationCallback delegate
   • Implement CertificateValidation helper factory class
3. DotNettyTransport.cs:
   • Update SetClientPipeline() to use custom validator when provided
   • Update SetServerPipeline() to use custom validator when provided
   • Fix asymmetry bug: Apply hostname validation bidirectionally
4. Tests: Full unit test suite for each helper (CertificateValidationHelpersSpec)

Backward Compatibility

• ✅ Existing config-based validation still works (via TlsValidationCallbacks)
• ✅ Falls back to TlsValidationCallbacks if no custom validator provided
• ✅ New parameter is optional

Benefits

• ✅ Maximum flexibility - Users can implement any validation logic
• ✅ Great ergonomics - Common cases have easy pre-built helpers
• ✅ Composable - Helpers combine with Combine() and ChainPlusThen()
• ✅ Discoverable - IDE shows all available helpers
• ✅ Fixes asymmetry - Custom validators apply bidirectionally (both client and server)
• ✅ Idiomatic .NET - Similar to Polly policies, authorization attributes
• ✅ Testable - Each helper independently testable

Related PRs

• feat(remote): implement mutual TLS authentication support #7851 (mutual TLS enforcement)
• Fix TLS hostname validation bug and add configurable validation #7897 (hostname validation)
• Improve TLS/SSL certificate error messages during handshake failures #7891 (improved error messages)

Acceptance Criteria

• CertificateValidationCallback delegate defined
• CertificateValidation factory class with all 7 helpers implemented
• DotNettySslSetup updated with CustomValidator parameter
• DotNettyTransport adapted to use custom validators bidirectionally
• Asymmetry bug fixed: server-side hostname validation respects setting
• Comprehensive test suite: CertificateValidationHelpersSpec
• Backward compatibility verified
• Documentation updated with examples

[Aaronontheweb]

Resolved via #7915