Fix TLS hostname validation bug and add configurable validation (#7897)

* Fix TLS hostname validation bug and add configurable validation (#7893)

## Summary

This commit addresses GitHub issue #7893 by fixing a critical TLS hostname validation bug and introducing a configurable, type-safe validation system.

## Changes

### Bug Fix
- **Fixed DotNettyTransport.cs:355** - TLS client was incorrectly validating against the client's own certificate DNS name instead of the remote server address
  - Changed from `certificate.GetNameInfo(X509NameType.DnsName, false)` to `remoteAddress.Host`

### New Configuration
- Added `validate-certificate-hostname` config option to Remote.conf
  - Default: `false` (disabled for backward compatibility and mutual TLS flexibility)
  - When enabled: Traditional TLS hostname validation (CN/SAN must match target hostname)
  - When disabled: Only validates certificate chain, ignores hostname mismatches
  - Useful for: Mutual TLS with per-node certificates, IP-based connections, dynamic service discovery

### Type-Safe Validation System
- Introduced enums to prevent primitive confusion in security-critical code:
  - `ChainValidationMode` enum (ValidateChain, IgnoreChainErrors)
  - `HostnameValidationMode` enum (ValidateHostname, IgnoreHostnameMismatch)
- Created `TlsValidationCallbacks` static factory class with:
  - Main `Create()` method accepting enum parameters and logging adapter
  - Convenience methods: `ValidateFull()`, `ValidateChainOnly()`, `ValidateHostnameOnly()`, `AcceptAll()`
  - Detailed error logging with filtered SslPolicyErrors
- Makes validation flags independent and composable
- Replaces ~35 lines of inline callback code with 3 lines of self-documenting factory calls

### Updated SslSettings
- Added `ValidateCertificateHostname` property
- Updated all constructors to accept the new property
- Updated `Create()` method to read from HOCON config

### Test Coverage
- Extended `DotNettyMutualTlsSpec` with 4 new test cases:
  - `Hostname_validation_disabled_should_allow_different_certificates()` - Different certs work with validation disabled
  - `Hostname_validation_enabled_should_reject_different_certificates()` - Different certs fail with validation enabled
  - `Same_certificate_should_connect_in_mutual_tls()` - Typical mutual TLS scenario
  - `Hostname_validation_default_should_be_disabled()` - Verifies backward compatibility

## Technical Details

Chain validation and hostname validation are now fully independent:
- `suppressValidation=true` disables chain/CA validation (for self-signed certs)
- `validateCertificateHostname=true/false` controls hostname matching (for per-node certs, IPs)

This allows testing hostname validation with self-signed certificates by using `suppressValidation=true, validateCertificateHostname=true`.

Fixes #7893

* Extend DotNettySslSetup to expose all SSL/TLS configuration options

## Summary

Extended `DotNettySslSetup` programmatic API to expose the full SSL/TLS configuration, including the newly added hostname validation setting and the existing RequireMutualAuthentication setting that was previously only available via HOCON.

## Changes

### DotNettySslSetup API
- Added `RequireMutualAuthentication` property (was missing from programmatic API)
- Added `ValidateCertificateHostname` property (new setting from #7893)
- Added comprehensive XML documentation for all properties and constructors
- Added backward-compatible constructors:
  - 2-parameter: Defaults to RequireMutualAuthentication=true, ValidateCertificateHostname=false
  - 3-parameter: Defaults to ValidateCertificateHostname=false
  - 4-parameter: Full control over all settings
- Updated `Settings` property to pass all 4 parameters to `SslSettings` constructor

### Integration Tests
- Added 3 integration tests in `DotNettySslSetupSpec`:
  - Verify 2-parameter setup configures effective DotNettyTransportSettings with expected defaults
  - Verify 3-parameter setup configures effective settings with specified RequireMutualAuth
  - Verify 4-parameter setup configures effective settings with all specified values
- Tests verify the actual consumption path: ActorSystem → DotNettyTransportSettings.Create() → setup.Value.Settings
- Tests validate that setup values correctly override HOCON defaults

## Backward Compatibility

All existing code using the 2-parameter constructor continues to work with the same defaults:
- RequireMutualAuthentication: true (matches previous HOCON-only behavior)
- ValidateCertificateHostname: false (matches new HOCON default)

The setup is properly consumed in `DotNettyTransportSettings.Create(ActorSystem)` which retrieves the setup via `system.Settings.Setup.Get<DotNettySslSetup>()` and calls `setup.Value.Settings` to get the fully configured `SslSettings` object.

* Update security documentation for hostname validation feature

## Changes

- Explained the new independent validation system (chain vs hostname)
- Added details about default certificate stores used (Windows, Linux, macOS)
- Documented the `validate-certificate-hostname` setting with use cases
- Added validation mode combination table
- Included configuration examples for both P2P and client-server scenarios
- Added comprehensive troubleshooting for hostname validation errors
- Documented enhanced TLS error messages from v1.5.52
- Reduced emoji usage for more professional tone
- Added links to Microsoft documentation and RFC specifications

## Key Documentation Updates

### New Sections
- Certificate Validation: Independent Control
- Hostname Validation setting explanation
- Validation Mode Combinations table
- Configuration with Hostname Validation Enabled
- Enhanced error message examples

### Troubleshooting Additions
- RemoteCertificateNameMismatch errors (hostname validation failures)
- UntrustedRoot errors (chain validation failures)
- Understanding TLS Error Messages section with real examples
- Multiple fix options for each error scenario

### Technical Details
- Explained which OS certificate stores are used by default
- Referenced RFC 5280 and RFC 6125 for validation standards
- Clarified that suppress-validation only controls chain validation
- Clarified that hostname validation is separate and optional

* Fix markdown linting and spellcheck issues in security docs

- Add blank lines around all fenced code blocks
- Add language specifiers to code blocks (text, hocon, bash, powershell)
- Change dash lists to asterisk lists for consistency
- Add 'hostnames' to spellcheck dictionary
- Emphasize that hostname validation defaults to false (disabled)

* Improve documentation heading structure and title case

- Remove technical setting names from headings
- Use descriptive section titles instead
- Change subheadings to 'Enabled/Disabled' pattern
- Move technical details into content body
- Fix title case linting issues

This makes the documentation more scannable and separates conceptual
sections from implementation details.

* added api approvals

Author: Aaronontheweb

docs/articles/remoting/security.md

@@ -36,6 +36,7 @@
         "Hasher",
         "Hipsterize",
         "HOCON",
+        "hostnames",
         "journaled",
         "Kubernetes",
         "lifecycles",

docs/cSpell.json

@@ -863,7 +863,11 @@ namespace Akka.Remote.Transport.DotNetty
     public sealed class DotNettySslSetup : Akka.Actor.Setup.Setup
     {
         public DotNettySslSetup(System.Security.Cryptography.X509Certificates.X509Certificate2 certificate, bool suppressValidation) { }
+        public DotNettySslSetup(System.Security.Cryptography.X509Certificates.X509Certificate2 certificate, bool suppressValidation, bool requireMutualAuthentication) { }
+        public DotNettySslSetup(System.Security.Cryptography.X509Certificates.X509Certificate2 certificate, bool suppressValidation, bool requireMutualAuthentication, bool validateCertificateHostname) { }
         public System.Security.Cryptography.X509Certificates.X509Certificate2 Certificate { get; }
+        public bool RequireMutualAuthentication { get; }
         public bool SuppressValidation { get; }
+        public bool ValidateCertificateHostname { get; }
     }
 }

src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveRemote.DotNet.verified.txt

@@ -863,7 +863,11 @@ namespace Akka.Remote.Transport.DotNetty
     public sealed class DotNettySslSetup : Akka.Actor.Setup.Setup
     {
         public DotNettySslSetup(System.Security.Cryptography.X509Certificates.X509Certificate2 certificate, bool suppressValidation) { }
+        public DotNettySslSetup(System.Security.Cryptography.X509Certificates.X509Certificate2 certificate, bool suppressValidation, bool requireMutualAuthentication) { }
+        public DotNettySslSetup(System.Security.Cryptography.X509Certificates.X509Certificate2 certificate, bool suppressValidation, bool requireMutualAuthentication, bool validateCertificateHostname) { }
         public System.Security.Cryptography.X509Certificates.X509Certificate2 Certificate { get; }
+        public bool RequireMutualAuthentication { get; }
         public bool SuppressValidation { get; }
+        public bool ValidateCertificateHostname { get; }
     }
 }

src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveRemote.Net.verified.txt

@@ -30,7 +30,7 @@ public DotNettyMutualTlsSpec(ITestOutputHelper output) : base(ConfigurationFacto
         {
         }
 
-        private static Config CreateConfig(bool enableSsl, bool requireMutualAuth, bool suppressValidation = false, string certPath = null)
+        private static Config CreateConfig(bool enableSsl, bool requireMutualAuth, bool suppressValidation = false, string certPath = null, bool? validateCertificateHostname = null)
         {
             var config = ConfigurationFactory.ParseString($@"
                 akka {{
@@ -49,10 +49,15 @@ private static Config CreateConfig(bool enableSsl, bool requireMutualAuth, bool
                 return config;
 
             var escapedPath = (certPath ?? ValidCertPath).Replace("\\", "\\\\");
+            var hostnameValidationConfig = validateCertificateHostname.HasValue
+                ? $"validate-certificate-hostname = {(validateCertificateHostname.Value ? "on" : "off")}"
+                : "";
+
             var ssl = $@"
                 akka.remote.dot-netty.tcp.ssl {{
                     suppress-validation = {(suppressValidation ? "on" : "off")}
                     require-mutual-authentication = {(requireMutualAuth ? "on" : "off")}
+                    {hostnameValidationConfig}
                     certificate {{
                         path = ""{escapedPath}""
                         password = ""{Password}""
@@ -275,6 +280,165 @@ await Assert.ThrowsAsync<AskTimeoutException>(async () =>
             }
         }
 
+        [Fact(DisplayName = "Different certificates with hostname validation disabled should connect successfully")]
+        public async Task Hostname_validation_disabled_should_allow_different_certificates()
+        {
+            // Per-node certificates should work when hostname validation is disabled
+            // Note: Using suppressValidation=true to bypass chain validation since test certs are self-signed
+            // This isolates the hostname validation logic we're testing
+            ActorSystem server = null;
+            ActorSystem client = null;
+
+            try
+            {
+                // Server with one certificate, hostname validation disabled
+                var serverConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ValidCertPath, validateCertificateHostname: false);
+                server = ActorSystem.Create("ServerSystem", serverConfig);
+                InitializeLogger(server, "[SERVER] ");
+
+                var serverEcho = server.ActorOf(Props.Create(() => new EchoActor()), "echo");
+                var serverAddr = RARP.For(server).Provider.DefaultAddress;
+                var serverEchoPath = new RootActorPath(serverAddr) / "user" / "echo";
+
+                // Client with different certificate, hostname validation disabled
+                var clientConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ClientCertPath, validateCertificateHostname: false);
+                client = ActorSystem.Create("ClientSystem", clientConfig);
+                InitializeLogger(client, "[CLIENT] ");
+
+                // Should successfully connect because hostname validation is disabled
+                var response = await client.ActorSelection(serverEchoPath).Ask<string>("hello", TimeSpan.FromSeconds(5));
+                Assert.Equal("hello", response);
+            }
+            finally
+            {
+                if (client != null)
+                    Shutdown(client, TimeSpan.FromSeconds(10));
+                if (server != null)
+                    Shutdown(server, TimeSpan.FromSeconds(10));
+            }
+        }
+
+        [Fact(DisplayName = "Different certificates with hostname validation enabled should fail with name mismatch")]
+        public async Task Hostname_validation_enabled_should_reject_different_certificates()
+        {
+            // When hostname validation is enabled, different certificates should fail with RemoteCertificateNameMismatch
+            // Note: Using suppressValidation=true to bypass chain validation and test hostname validation specifically
+            ActorSystem server = null;
+            ActorSystem client = null;
+
+            try
+            {
+                // Server with one certificate, hostname validation enabled
+                var serverConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ValidCertPath, validateCertificateHostname: true);
+                server = ActorSystem.Create("ServerSystem", serverConfig);
+                InitializeLogger(server, "[SERVER] ");
+
+                var serverEcho = server.ActorOf(Props.Create(() => new EchoActor()), "echo");
+                var serverAddr = RARP.For(server).Provider.DefaultAddress;
+                var serverEchoPath = new RootActorPath(serverAddr) / "user" / "echo";
+
+                // Client with different certificate, hostname validation enabled
+                var clientConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ClientCertPath, validateCertificateHostname: true);
+                client = ActorSystem.Create("ClientSystem", clientConfig);
+                InitializeLogger(client, "[CLIENT] ");
+
+                // Should fail because hostname in certificate doesn't match connection target (127.0.0.1)
+                await Assert.ThrowsAsync<AskTimeoutException>(async () =>
+                {
+                    await client.ActorSelection(serverEchoPath).Ask<string>("hello", TimeSpan.FromSeconds(3));
+                });
+            }
+            finally
+            {
+                if (client != null)
+                    Shutdown(client, TimeSpan.FromSeconds(10));
+                if (server != null)
+                    Shutdown(server, TimeSpan.FromSeconds(10));
+            }
+        }
+
+        [Fact(DisplayName = "Same certificate should connect successfully (typical mutual TLS scenario)")]
+        public async Task Same_certificate_should_connect_in_mutual_tls()
+        {
+            // Typical mutual TLS: Both nodes use the same shared certificate
+            // Hostname validation disabled because we're using IPs/per-node certs
+            ActorSystem server = null;
+            ActorSystem client = null;
+
+            try
+            {
+                // Server with same certificate, hostname validation disabled (typical for mutual TLS)
+                var serverConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ValidCertPath, validateCertificateHostname: false);
+                server = ActorSystem.Create("ServerSystem", serverConfig);
+                InitializeLogger(server, "[SERVER] ");
+
+                var serverEcho = server.ActorOf(Props.Create(() => new EchoActor()), "echo");
+                var serverAddr = RARP.For(server).Provider.DefaultAddress;
+                var serverEchoPath = new RootActorPath(serverAddr) / "user" / "echo";
+
+                // Client with same certificate, hostname validation disabled
+                var clientConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ValidCertPath, validateCertificateHostname: false);
+                client = ActorSystem.Create("ClientSystem", clientConfig);
+                InitializeLogger(client, "[CLIENT] ");
+
+                // Should successfully connect - typical mutual TLS scenario
+                var response = await client.ActorSelection(serverEchoPath).Ask<string>("hello", TimeSpan.FromSeconds(5));
+                Assert.Equal("hello", response);
+            }
+            finally
+            {
+                if (client != null)
+                    Shutdown(client, TimeSpan.FromSeconds(10));
+                if (server != null)
+                    Shutdown(server, TimeSpan.FromSeconds(10));
+            }
+        }
+
+        [Fact(DisplayName = "Hostname validation unspecified should default to disabled (backward compatibility)")]
+        public async Task Hostname_validation_default_should_be_disabled()
+        {
+            // When validate-certificate-hostname is not specified, it should default to false
+            // Note: Using suppressValidation=true to bypass chain validation and test hostname default behavior
+            ActorSystem server = null;
+            ActorSystem client = null;
+
+            try
+            {
+                // Server without specifying hostname validation (should default to false)
+                var serverConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ValidCertPath, validateCertificateHostname: null);
+                server = ActorSystem.Create("ServerSystem", serverConfig);
+                InitializeLogger(server, "[SERVER] ");
+
+                var serverEcho = server.ActorOf(Props.Create(() => new EchoActor()), "echo");
+                var serverAddr = RARP.For(server).Provider.DefaultAddress;
+                var serverEchoPath = new RootActorPath(serverAddr) / "user" / "echo";
+
+                // Client with different certificate, hostname validation unspecified (should default to false)
+                var clientConfig = CreateConfig(enableSsl: true, requireMutualAuth: true, suppressValidation: true,
+                    certPath: ClientCertPath, validateCertificateHostname: null);
+                client = ActorSystem.Create("ClientSystem", clientConfig);
+                InitializeLogger(client, "[CLIENT] ");
+
+                // Should successfully connect because hostname validation defaults to disabled
+                var response = await client.ActorSelection(serverEchoPath).Ask<string>("hello", TimeSpan.FromSeconds(5));
+                Assert.Equal("hello", response);
+            }
+            finally
+            {
+                if (client != null)
+                    Shutdown(client, TimeSpan.FromSeconds(10));
+                if (server != null)
+                    Shutdown(server, TimeSpan.FromSeconds(10));
+            }
+        }
+
         private sealed class EchoActor : ReceiveActor
         {
             public EchoActor()

src/core/Akka.Remote.Tests/Transport/DotNettyMutualTlsSpec.cs

@@ -105,6 +105,96 @@ await Assert.ThrowsAsync<RemoteTransportException>(async () =>
             });
         }
 
+        [Fact(DisplayName = "DotNettySslSetup with 2 parameters should configure effective DotNettyTransportSettings with defaults (RequireMutualAuth=true, ValidateHostname=false)")]
+        public void Two_parameter_setup_should_configure_transport_settings_with_defaults()
+        {
+            var certificate = new X509Certificate2(ValidCertPath, Password, X509KeyStorageFlags.DefaultKeySet);
+            var sslSetup = new DotNettySslSetup(certificate, suppressValidation: true);
+
+            var actorSystemSetup = ActorSystemSetup.Empty
+                .And(BootstrapSetup.Create().WithConfig(ConfigurationFactory.ParseString(@"
+akka {
+  actor.provider = ""Akka.Remote.RemoteActorRefProvider,Akka.Remote""
+  remote.dot-netty.tcp {
+    port = 0
+    hostname = ""127.0.0.1""
+    enable-ssl = true
+  }
+}")))
+                .And(sslSetup);
+
+            using var sys = ActorSystem.Create("test", actorSystemSetup);
+
+            // Verify that DotNettyTransportSettings.Create uses the setup correctly
+            var settings = DotNettyTransportSettings.Create(sys);
+
+            Assert.True(settings.EnableSsl);
+            Assert.Equal(certificate, settings.Ssl.Certificate);
+            Assert.True(settings.Ssl.SuppressValidation);
+            Assert.True(settings.Ssl.RequireMutualAuthentication); // default from 2-param constructor
+            Assert.False(settings.Ssl.ValidateCertificateHostname); // default from 2-param constructor
+        }
+
+        [Fact(DisplayName = "DotNettySslSetup with 3 parameters should configure effective DotNettyTransportSettings with specified RequireMutualAuth and default ValidateHostname=false")]
+        public void Three_parameter_setup_should_configure_transport_settings()
+        {
+            var certificate = new X509Certificate2(ValidCertPath, Password, X509KeyStorageFlags.DefaultKeySet);
+            var sslSetup = new DotNettySslSetup(certificate, suppressValidation: false, requireMutualAuthentication: false);
+
+            var actorSystemSetup = ActorSystemSetup.Empty
+                .And(BootstrapSetup.Create().WithConfig(ConfigurationFactory.ParseString(@"
+akka {
+  actor.provider = ""Akka.Remote.RemoteActorRefProvider,Akka.Remote""
+  remote.dot-netty.tcp {
+    port = 0
+    hostname = ""127.0.0.1""
+    enable-ssl = true
+  }
+}")))
+                .And(sslSetup);
+
+            using var sys = ActorSystem.Create("test", actorSystemSetup);
+
+            // Verify that DotNettyTransportSettings.Create uses the setup correctly
+            var settings = DotNettyTransportSettings.Create(sys);
+
+            Assert.True(settings.EnableSsl);
+            Assert.Equal(certificate, settings.Ssl.Certificate);
+            Assert.False(settings.Ssl.SuppressValidation);
+            Assert.False(settings.Ssl.RequireMutualAuthentication); // explicitly set to false
+            Assert.False(settings.Ssl.ValidateCertificateHostname); // default from 3-param constructor
+        }
+
+        [Fact(DisplayName = "DotNettySslSetup with 4 parameters should configure effective DotNettyTransportSettings with all specified values")]
+        public void Four_parameter_setup_should_configure_transport_settings_with_all_values()
+        {
+            var certificate = new X509Certificate2(ValidCertPath, Password, X509KeyStorageFlags.DefaultKeySet);
+            var sslSetup = new DotNettySslSetup(certificate, suppressValidation: true, requireMutualAuthentication: false, validateCertificateHostname: true);
+
+            var actorSystemSetup = ActorSystemSetup.Empty
+                .And(BootstrapSetup.Create().WithConfig(ConfigurationFactory.ParseString(@"
+akka {
+  actor.provider = ""Akka.Remote.RemoteActorRefProvider,Akka.Remote""
+  remote.dot-netty.tcp {
+    port = 0
+    hostname = ""127.0.0.1""
+    enable-ssl = true
+  }
+}")))
+                .And(sslSetup);
+
+            using var sys = ActorSystem.Create("test", actorSystemSetup);
+
+            // Verify that DotNettyTransportSettings.Create uses the setup correctly
+            var settings = DotNettyTransportSettings.Create(sys);
+
+            Assert.True(settings.EnableSsl);
+            Assert.Equal(certificate, settings.Ssl.Certificate);
+            Assert.True(settings.Ssl.SuppressValidation);
+            Assert.False(settings.Ssl.RequireMutualAuthentication); // explicitly set to false
+            Assert.True(settings.Ssl.ValidateCertificateHostname); // explicitly set to true
+        }
+
         #region helper classes / methods
 
         protected override void AfterAll()

src/core/Akka.Remote.Tests/Transport/DotNettySslSetupSpec.cs

@@ -565,6 +565,18 @@ akka {
         # Set to false only if your environment cannot support client certificate authentication.
         # Default: true (secure by default)
         require-mutual-authentication = true
+
+        # Enable or disable certificate hostname validation during TLS handshake.
+        # When true: Traditional TLS hostname validation is performed (certificate CN/SAN must match target hostname)
+        # When false: Only validates certificate chain against CA, ignores hostname mismatches
+        #
+        # Set to false for scenarios such as:
+        #   - Mutual TLS with per-node certificates in P2P clusters
+        #   - IP-based connections where certificates use DNS names
+        #   - Service discovery with dynamic addresses
+        #
+        # Default: false (disabled for backward compatibility and mutual TLS flexibility)
+        validate-certificate-hostname = false
       }
     }