SslStream API improvements for enhanced use cases

[wfurt]

Background and Motivation

This covers several use cases discussed in various runtime, aspnetcore and YARP threads.
This allow to implement new functionality as well as it allows callers to control some internal aspects of SslStream and X509Chain used internally.

Proposed API

namespace System.Security
{
+    public sealed class SslStreamCertificateContext
+    {
+          public static SslStreamCertificateContext CreateForServer(
+            X509Certificate2 target,
+            X509Certificate2Collection? additionalCertificates); 
+    }

     public delegate bool RemoteCertificateValidationCallback(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors);
     public delegate X509Certificate ServerCertificateSelectionCallback(object sender, string hostName);
+    public delegate ValueTask<SslServerAuthenticationOptions> ServerOptionsSelectionCallback(SslStream sender, string hostName, CancellationToken cancellationToken);

     public class SslServerAuthenticationOptions
     {
         RemoteCertificateValidationCallback RemoteCertificateValidationCallback { get; set; }
         X509Certificate ServerCertificate { get; set; }
         ServerCertificateSelectionCallback ServerCertificateSelectionCallback { get; set; };
+        SslStreamCertificateContext ServerCertificateContext  { get; set; };
     }

      public class SslStream 
      {
          public Task AuthenticateAsServerAsync(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken = default);
          public void AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions);

          // Add overloaded with delegate returning SslServerAuthenticationOptions instead of passing SslServerAuthenticationOptions in.
+         public Task AuthenticateAsServerAsync(ServerOptionsSelectionCallback optionCallback, CancellationToken cancellationToken = default);
          // property to show name requested by client
+         public virtual string HostName { get; }
    }
}

SslStreamCertificateContext
There are cases when caller passes server certificate to AuthenticateAsServer(). That is single certificate and SslStream is trying to build complete chain, possibly doing synchronous HTTP calls behind the scenes. There is no coordination or visibility and that can create issues especially when the sites are not reachable or flaky. The intermediates are caches on disk, but that may not work as expected with docker and Kuberneties.

We talk about alternatives (like using just X509CertificateCollection or X509Chain) and decide to go with proposal of opaque object. That gives use option to extend it later and use is as state holder later when we need to add OCSP support, session cache and possibly other features.

This adds third (and hopefully last) way how to specify server certificate. e.g. new full context, existing X509Certificate and existing ServerCertificateSelectionCallback().

ServerOptionsSelectionCallback
There is desire to configure more properties (like SslProtocols) per specific server-name.
Great examples may be Kestrel and YARP when all HTTP servers may be on the same port yet expecting different policies. Perhaps legacy server allowing Tls1.x and more secured one requiring Tls12 or above. Since the server name client is trying to connect to is not known until first message from client, this is currently very difficult to implement. There are cases when users would sniff and parse incoming TLS frames to get the information soon enough.

The other use case new delegate is trying to solve is better ability to perform IO in selection callback. Current delegate is synchronous and that can be a problem if there is need to fetch certificate from database of key vault. There were also cases, when somebody want to inject audit logs to database and that leads to Async -> Sync -> Async path.

With proposed ValueTask, the callback can finish synchronously or synchronously, return one certificate or whole context and it can also set crypto parameters one would normally set on SslServerAuthenticationOptions.
#31097
#35844
dotnet/aspnetcore#21300

HostName
When AuthenticateAsClient() is called, it is mandatory that TargetHost is set. (either via SslClientAuthenticationOptions or via some convenience overloads). However, when RemoteCertificateValidationCallback() runs, there is no easy way to know what target SslStream was trying to connect to. We already store that information internally in SslStream and this would expose it as public property.

On server, this property holds name requested by client or it is empty string if SNI extension was not used.

The name can be DNS name in dot notation or it can be IP address as string.
Alternatively we can name this TargetHost to be consistent with terminology in SslClientAuthenticationOptions.
#27619

Alternative Designs

Since we have ServerCertificateSelectionCallback we could add ServerOptionsSelectionCallback there as well. It feels strange to have callback to select options part of the option.

There was some discussion around name HostName property.

Right now, the asks are to make properties configurable based on SNI (server name). If we want to make it more extensible for future we can consider one of the following:

1. pass ClientHello in as raw data
   ReadOnlySpan<byte> clientHello That would allows callback to process any other extension from the request.

2. use some container like:

public delegate ValueTask<SslServerAuthenticationOptions> ServerOptionsSelectionCallback(SslStream sender, SslClientHelloInfo clientHello, CancellationToken cancellationToken);

readonly struct SslClientHelloInfo
{
    string ServerName;
    SslProtocols SslProtocols;
    //maybe
    List<SslApplicationProtocol> ApplicationProtocols;
    // maybe
    ReadOnlySpan<byte> RawClientHello;
}

HTTP/2 specification requires certain ciphers to be blocked and this would be path to fulfill that expectation. e.g. If client asks for h2 and we are willing to do that, cipher policy could be used on platforms with support. (so as we can disable renegotiation)

The caveat is that ALPN is list so we would need to allocate. That may be negligible comparing to cost of crypto in SSL/TLS handshake.

Usage Examples

The expectation is that this can be used with PFX files or to-be-approved #31944

Caller can get and verify full chain once (when possible) and then use across multiple SslStream. If this needs to be done per server name, once would use AuthenticateAsServerAsync(optionCallback, cancellationToken);

    SslStreamCertificateContext CretateCertificateContext(ReadOnlySpan<char> certificate, ReadOnlySpan<char> certPem key, ReadOnlySpan<char> chainPem)
    {
            X509Certificate2 serverCertificate =  X509Certificate2.CreateFromPem(certificate, key);
            X509Certificate2Collection extra = new X509Certificate2Collection(serverCertificate);
            extra.ImportFromPem(chainPem)
 
            X509Chain chain = new X509Chain()
            chain.ChainPolicy.ApplicationPolicy.Add(serverAuthOid);
            chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
            chain.ChainPolicy.DisableCertificateFetch = true;
            foreach (X509Certificate intermediate in extra)
            {
                chain.ChainPolicy.ExtraStore.Add(intermediate);
            }
             
            if (chain.Build())
            {
                // We have full chain in PEM data
                return SslStreamCertificateContext.CreateForServer(serverCertificate, chain.ChainElements);
            }
 
            // Log warning ???
            // Try to build chain with network access.
            chain.ChainPolicy.DisableCertificateFetch = false;
            chain.ChainPolicy.UrlRetrievalTimeout = TimeSpan.FromSeconds(_maxFetchTime);
            if (chain.Build())
            {
                // Build was successful. Intermediate certificates will be cached now.
                return SslStreamCertificateContext.CreateForServer(serverCertificate, chain.ChainElements);
            }
 
            // Failed to fetch chain parts.
            // do you favorite error mitigation here
            return null;
    }
 
    void PrepareServer(String name, ReadOnlySpan<char> certificate, ReadOnlySpan<char> certPem key, ReadOnlySpan<char> chain)
    {
            var options = new SslServerAuthenticationOptions()
            // customize options per site
            options.ServerCertificateContext = CretateCertificateContext(certificate, key, chain);
            _configurations.Add(name, options);
    }
 
    ValueTask<SslServerAuthenticationOptions> ServerOptionsSelectionCallback(SslStream sender, string hostname, CancellationToken cancellationToken)
    {
            if (!_configurations.TryGet(name, out SslServerAuthenticationOptions options))
            {
                // unknown site?
                return ValueTask.FromException(new Exception());
            }
 
            return new ValueTask<SslServerAuthenticationOptions>(options);
    }

Risks

This is primarily meant for advanced use cases. It feels like it would be easy to mess up in async callback.

This should be pretty much what we agreed on @bartonjs @davidfowl @halter73
Let me know if you think it is ready for API review.

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[scalablecory]

Your proposed API is messed up, looks like some methods are outside of SslStream.

[scalablecory]

Are there any other interesting parameters in the client hello that we might want to pass (maybe not now, but in future) beyond the hostName that is in the new delegate? Would it make more sense to pass a new class instead that we could add additional properties to later?

[scalablecory]

Having the sync API use an async callback isn't great. Can a second delegate be added for sync?

[halter73]

Considering this a new server-only API, would we be OK with not adding a sync version of AuthenticateAsServerAsync(ServerOptionsSelectionCallback, CancellationToken)?

[davidfowl]

I think @scalablecory means and overload that takes a synchronous callback as well

[halter73]

That's how I understood @scalablecory's comment as well, but no need to have a overload with a sync callback if there's no sync overload.

[vcsjones]

Just so I am clear, there is no expectation that X509Chain has to be used? It seems that the API allows for this, so I'm just making sure I understand correctly.

If I am saying "here's my leaf, here's an ordered set of additional intermediates" I would not expect X509Chain to be used at all - I would expect the server to just accept it as-is and send it down in the Certificate certificate_list part of the handshake (after ServerHello). Path-building, revocation, trust, policy, etc all the responsibilities of the client / user agent.

[bartonjs]

Just so I am clear, there is no expectation that X509Chain has to be used

My thoughts, at least, are that it's used once, when this object is created, so that it can normalize the inputs (e.g. remove duplicates, make sure the order is correct, etc).

[wfurt]

I added AuthenticateAsServer(ServerOptionsSelectionCallback optionCallback) as we just added client sync complements. We can certainly do that later. As far as the delegate, I was expecting that on synchronous path, that would finish synchronously returning the "value" part of ValueTask. That is still possible even with the AuthenticateAsServerAsync is everything is ready in memory.