[API Proposal]: Add Wrap and Unwrap methods to NegotiateAuthentication API

[filipnavara]

Background and motivation

In #69920 the initial NegotiateAuthentication API surface was reviewed and approved. This was intentionally scaled down to a minimum viable proposal to unblock client-side and server-side HTTP authentication scenarios without resorting to reflection on internal API surface.

This follows with addition to the API surface to support additional scenarios:

1. Implementation of NTLM and Negotiate SASL authentication mechanisms as used by SMTP, IMAP, LDAP and other protocols. This is currently done internally in the SmtpClient class. Additionally it would be useful for external libraries like MailKit for identical scenarios.
2. Bi-directional authenticated communication between two parties after the initial authentication was negotiated. This is what NegotiateStream does today but also what projects like .NET/C# client for Apache Kudu need.
3. Specifying the allowed impersonation (eg. delegation) for the credentials used by the client. Thus is necessary prerequisite to enable scenarios for SQL Server client library.
4. Handling server-side scenarios with KDC proxy (Kerberos Domain Controller exposed on HTTPS endpoint for external authentication).

API Proposal

Wrap/Unwrap (signing and encryption)

Additional methods are added that closely reflect the GSS_Wrap and GSS_Unwrap methods in the GSSAPI specification. They map most of the functionality save for the qop input parameter which was never exposed by the internal APIs in .NET and which is commonly set to 0 (default QOP protection) in the native APIs.

The main problematic point is how to express the buffer allocation semantics in a concise way. For Wrap the size of the encrypted content is not known before hand and the operation modifies an internal context state so it needs to succeed in one go, ie. it cannot return "buffer too small" error and let the caller retry. We want to allow the caller to reuse the buffer as long as it is big enough. The IBufferWriter<byte> interface seems to convey these semantics quite cleanly. For Unwrap we know the unwrapped data are at most as big as the input wrapped data. On Windows we can efficiently do the unwrapping inline within the same buffer and it would be nice to expose it to the caller.

namespace System.Net.Security;

/// <summary>
/// Represents a stateful authentication exchange that uses the Negotiate, NTLM or Kerberos security protocols
/// to authenticate the client or server, in client-server communication.
/// </summary>
public sealed class NegotiateAuthentication : IDisposable
{
    /// <summary>
    /// Wrap an input message with signature and optionally with an encryption.
    /// </summary>
    /// <param name="input">Input message to be wrapped.</param>
    /// <param name="outputWriter">Buffer writter where the wrapped message is written.</param>
    /// <param name="isEncrypted">
    /// On input specifies whether encryption is requested.
    /// On output specifies whether encryption was applied in the wrapping.
    /// </param>
    /// <returns>
    /// <see cref="NegotiateAuthenticationStatusCode.Completed" /> on success, other
    /// <see cref="NegotiateAuthenticationStatusCode" /> values on failure.
    /// </returns>
    /// <remarks>
    /// Like the <see href="https://datatracker.ietf.org/doc/html/rfc2743#page-65">GSS_Wrap</see> API
    /// the authentication protocol implementation may choose to override the requested value in the
    /// isEncrypted parameter. This may result in either downgrade or upgrade of the protection level.
    /// </remarks>
    /// <exception cref="InvalidOperationException">Authentication failed or has not occurred.</exception>
    public NegotiateAuthenticationStatusCode Wrap(ReadOnlySpan<byte> input, IBufferWriter<byte> outputWriter, ref bool isEncrypted);
    
    /// <summary>
    /// Unwrap an input message with signature or encryption applied by the other party.
    /// </summary>
    /// <param name="input">Input message to be unwrapped.</param>
    /// <param name="outputWriter">Buffer writter where the unwrapped message is written.</param>
    /// <param name="isEncrypted">
    /// On output specifies whether the wrapped message had encryption applied.
    /// </param>
    /// <returns>
    /// <see cref="NegotiateAuthenticationStatusCode.Completed" /> on success.
    /// <see cref="NegotiateAuthenticationStatusCode.MessageAltered" /> if the message signature was
    /// invalid.
    /// <see cref="NegotiateAuthenticationStatusCode.InvalidToken" /> if the wrapped message was
    /// in invalid format.
    /// Other <see cref="NegotiateAuthenticationStatusCode" /> values on failure.
    /// </returns>
    /// <exception cref="InvalidOperationException">Authentication failed or has not occurred.</exception>
    public NegotiateAuthenticationStatusCode Unwrap(ReadOnlySpan<byte> input, IBufferWriter<byte> outputWriter, out bool isEncrypted);

    /// <summary>
    /// Unwrap an input message with signature or encryption applied by the other party.
    /// </summary>
    /// <param name="input">Input message to be unwrapped. On output contains the decoded data.</param>
    /// <param name="unwrappedOffset">Offset in the input buffer where the unwrapped message was written.</param>
    /// <param name="unwrappedLength">Length of the unwrapped message.</param>
    /// <param name="isEncrypted">
    /// On output specifies whether the wrapped message had encryption applied.
    /// </param>
    /// <returns>
    /// <see cref="NegotiateAuthenticationStatusCode.Completed" /> on success.
    /// <see cref="NegotiateAuthenticationStatusCode.MessageAltered" /> if the message signature was
    /// invalid.
    /// <see cref="NegotiateAuthenticationStatusCode.InvalidToken" /> if the wrapped message was
    /// in invalid format.
    /// Other <see cref="NegotiateAuthenticationStatusCode" /> values on failure.
    /// </returns>
    /// <exception cref="InvalidOperationException">Authentication failed or has not occurred.</exception>
    public NegotiateAuthenticationStatusCode UnwrapInplace(Span<byte> input, out int unwrappedOffset, out int unwrappedLength, out bool isEncrypted);
}

Impersonation level / mutual authentication

The internal NTAuthentication API used ContextFlagsPal enum to specify additional requests such as signing, encryption, or mutual authentication to the authentication provider. Exposing the enum itself on public API was deemed inappropriate because it was mixing flags for client-side and server-side authentication. Many of the flags were mutually exclusive or specific to Schannel (TLS) authentication not related to the new API. The minimum viable prototype was to expose the required protection level (none, signing, encryption and signing). This API suggestion extends the NegotiateAuthenticationClientOptions and NegotiateAuthentication with additional properties that facilitate scenarios related to impersonation and mutual authentication.

namespace System.Net.Security;

public class NegotiateAuthenticationClientOptions
{
    /// <summary>
    /// Indicates that mutual authentication is required between the client and server.
    /// </summary>
    public bool RequireMutualAuthentication { get; set; }
    
    /// <summary>
    /// One of the <see cref="TokenImpersonationLevel" /> values, indicating how the server
    /// can use the client's credentials to access resources.
    /// </summary>
    public System.Security.Principal.TokenImpersonationLevel AllowedImpersonationLevel { get; set; }
}

public class NegotiateAuthentication
{
    /// <summary>
    /// One of the <see cref="TokenImpersonationLevel" /> values, indicating the negotiated
    /// level of impresonation.
    /// </summary>
    public System.Security.Principal.TokenImpersonationLevel ImpersonationLevel { get; }
}

KDC proxy support and server-side validation

In the KDC proxy scenario the problem is to specifying how channel binding validation is performed. The client connects to a HTTPS endpoint of the KDC proxy and thus the channel binding used in the authentication may not match what the server expects. In the native SSPI methods this is represented by the ASC_REQ_ALLOW_MISSING_BINDINGS and ASC_REQ_PROXY_BINDINGS flags used for different scenarios. On managed side these are exposed by the ExtendedProtectionPolicy class along with other policy options such as list of service principal names.

There are two different ways to expose the underlying functionality. The minimum viable way is just directly exposing these two flags, it is described below in the Alternative Designs section. The proposal here adds ExtendedProtectionPolicy and RequiredImpersonationLevel to the server-side options and moves the validation responsibility into the NegotiateAuthentication class.

The validation of channel bindings and target name (SPN), or collectively the extended security policy, would be moved into the last step of GetOutgoingBlob in the authentication flow. It would report the status back as either TargetUnknown or BadBinding. Similarly, the ImpersonationLevel would be validated against RequiredImpresonationLevel and return the ImpersonationValidationFailed error if the check fails.

In Kerberos scenarios the SPN is already validated by the system libraries. This would simply align NTLM to expose the same level of validation but implemented in managed code. This is currently done in NegotiateStream and HttpListener with almost identical code that could be shared.

public class NegotiateAuthenticationServerOptions
{
    /// <summary>
    /// Indicates extended security and validation policies.
    /// </summary>
    public System.Security.Authentication.ExtendedProtectionPolicy? Policy { get; set; }

    /// <summary>
    /// One of the <see cref="TokenImpersonationLevel" /> values, indicating how the server
    /// can use the client's credentials to access resources.
    /// </summary>
    public System.Security.Principal.TokenImpersonationLevel RequiredImpersonationLevel { get; set; }
}

public enum NegotiateAuthenticationStatusCode
{
    /// <status>Validation of RequiredProtectionLevel against negotiated protection level failed.</status>
    /// <remarks>Part of original API proposal, just not enforced in the managed code yet</remarks>
    SecurityQosFailed,
    /// <status>Validation of the target name failed</status>
    TargetUnknown,
    /// <status>Validation of the impersonation level failed</status>
    ImpersonationValidationFailed,
}

API Usage

On client-side the API usage would be identical to #69920 with the addition of the new options used for the authentication specified in NegotiateAuthenticationClientOptions.

Server-side validation

On server-side the API usage would also be conceptually identical but the validation code would be changed slightly.

For example, within NegotiateStream.AuthenticateAsServer[Async] the extended protection policy would be passed into NegotiateAuthenticationServerOptions. Instead of performing manual validation after the handshake an error code would be directly returned from the GetOutgoingBlob API and transformed into appropriate error response for the caller:

message = negotiateAuthentication.GetOutgoingBlob(message, out var statusCode);

// Simplified validation:
if (statusCode == NegotiateAuthenticationStatusCode.BadBinding ||
    statusCode == NegotiateAuthenticationStatusCode.TargetUnknown ||
    statusCode == NegotiateAuthenticationStatusCode.ImpersonationValidationFailed ||
    statusCode == NegotiateAuthenticationStatusCode.SecurityQosFailed)
{
    exception = statusCode switch
    {
        NegotiateAuthenticationStatusCode.BadBinding =>
            new AuthenticationException(SR.net_auth_bad_client_creds_or_target_mismatch),
        NegotiateAuthenticationStatusCode.TargetUnknown =>
            new AuthenticationException(SR.net_auth_bad_client_creds_or_target_mismatch),
        NegotiateAuthenticationStatusCode.ImpersonationValidationFailed =>
            new AuthenticationException(SR.Format(SR.net_auth_context_expectation, _expectedImpersonationLevel.ToString(), PrivateImpersonationLevel.ToString())),
        _ => // NegotiateAuthenticationStatusCode.SecurityQosFailed
            exception = new AuthenticationException(SR.Format(SR.net_auth_context_expectation, result.ToString(), _expectedProtectionLevel.ToString())),
    };

    int statusCode = ERROR_TRUST_FAILURE;
    message = new byte[sizeof(long)];
    BinaryPrimitives.WriteInt64LittleEndian(message, statusCode);
    await SendAuthResetSignalAndThrowAsync<TIOAdapter>(message, exception, cancellationToken).ConfigureAwait(false);
    Debug.Fail("Unreachable");
}
else if (statusCode == NegotiateAuthenticationStatusCode.Completed)
{
    // Signal remote party that we are done
    ...
}
else if (statusCode != NegotiateAuthenticationStatusCode.ContinueNeeded)
{
    // Signal remote side on a failed attempt.
    ...
}

...

Alternative Designs

KDC proxy support and server-side validation

Instead of exposing the full ExtendedProtectionPolicy as NegotiateAuthenticationServerOptions.Policy only the underlying system flags could be exposed. The full validation of SPNs or impersonation level would be left to the caller. No new error codes would be introduced.

namespace System.Net.Security;

public enum NegotiateAuthenticationBindingValidation
{
    /// <summary>
    /// Check the channel binding against the value in transport.
    /// </summary>
    Default,
    /// <summary>
    /// Allow missing channel binding on the transport.
    /// </summary>
    /// <remarks>Maps to the ASC_REQ_ALLOW_MISSING_BINDINGS SSPI flag.</remarks>
    AllowMissingBindings,
    /// <summary>
    /// Require channel binding but don't check its value.
    /// </summary>
    /// <remarks>Maps to the ASC_REQ_PROXY_BINDINGS SSPI flag.</remarks>
    ProxyBindings,
}

public class NegotiateAuthenticationServerOptions
{
    /// <summary>
    /// Indicates the required level of channel binding validation.
    /// </summary>
    public NegotiateAuthenticationBindingValidation BindingValidation { get; set; }
}

Risks

Currently NegotiateAuthentication is pretty light-weight wrapper over GSSAPI (non-Windows) and SSPI (Windows) native APIs. Moving part of the server-side validation into the wrapper risks that something is implemented incorrectly. On the other hand, the expectation is to reuse the existing code already present in NegotiateStream/HttpListener and thus make it easier for the caller to implement any of the advanced validation scenarios correctly.

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

In #69920 the initial NegotiateAuthentication API surface was reviewed and approved. This was intentionally scaled down to a minimum viable proposal to unblock client-side and server-side HTTP authentication scenarios without resorting to reflection on internal API surface.

This follows with addition to the API surface to support two additional scenarios:

• Implementation of NTLM and Negotiate SASL authentication mechanisms as used by SMTP, IMAP, LDAP and other protocols. This is currently done internally in the SmtpClient class. Additionally it would be useful for external libraries like MailKit for identical scenarios.
• Bi-directional authenticated communication between two parties after the initial authentication was negotiated. This is what NegotiateStream does today but also what projects like .NET/C# client for Apache Kudu need.

Two additional methods are added that closely reflect the GSS_Wrap and GSS_Unwrap methods in the GSSAPI specification. They map most of the functionality save for the qop input parameter which was never exposed by the internal APIs in .NET and which is commonly set to 0 (default QOP protection) in the native APIs.

API Proposal

namespace System.Net.Security;

/// <summary>
/// Represents a stateful authentication exchange that uses the Negotiate, NTLM or Kerberos security protocols
/// to authenticate the client or server, in client-server communication.
/// </summary>
public sealed class NegotiateAuthentication : IDisposable
{
    /// <summary>
    /// Wrap an input message with signature and optionally with an encryption.
    /// <summary>
    /// <param name="input">Input message to be wrapped.</param>
    /// <param name="outputWriter">Buffer writter where the wrapped message is written.</param>
    /// <param name="isConfidential">
    /// On input specifies whether confidentiality transformation (encryption) is requested.
    /// On output specifies whether the confidentiality transformation was applied in the wrapping.
    /// </param>
    /// <returns>
    /// <see cref="NegotiateAuthenticationStatusCode.Completed" /> on success, other
    /// <see cref="NegotiateAuthenticationStatusCode" /> values on failure.
    /// </returns>
    /// <remarks>
    /// Like the <see href="https://datatracker.ietf.org/doc/html/rfc2743#page-65">GSS_Wrap</see> API
    /// the authentication protocol implementation may choose to override the requested value in the
    /// <see cref="isConfidential" /> parameter. This may result in either downgrade or upgrade of the
    /// protection level.
    /// </remarks>
    public NegotiateAuthenticationStatusCode Wrap(ReadOnlySpan<byte> input, IBufferWriter<byte> outputWriter, ref bool isConfidential);
    
    /// <summary>
    /// Unwrap an input message with signature or encryption applied by the other party.
    /// <summary>
    /// <param name="input">Input message to be unwrapped.</param>
    /// <param name="outputWriter">Buffer writter where the unwrapped message is written.</param>
    /// <param name="isConfidential">
    /// On output specifies whether the wrapped message had confidentiality transformation applied.
    /// </param>
    /// <returns>
    /// <see cref="NegotiateAuthenticationStatusCode.Completed" /> on success.
    /// <see cref="NegotiateAuthenticationStatusCode.MessageAltered" /> if the message signature was
    /// invalid.
    /// <see cref="NegotiateAuthenticationStatusCode.InvalidToken" /> if the wrapped message was
    /// in invalid format.
    /// Other <see cref="NegotiateAuthenticationStatusCode" /> values on failure.
    /// </returns>
    public NegotiateAuthenticationStatusCode Unwrap(ReadOnlySpan<byte> input, IBufferWriter<byte> outputWriter, out bool isConfidential);
}

API Usage

Span<byte> messageLengthBuffer = new byte[4];
NegotiateAuthentication auth = new NegotiateAuthentication(
    new NegotiateAuthenticationClientOptions
    {
        Package = "NTLM",
        Credential = s_testCredentialRight,
        TargetName = "HTTP/foo",
        RequiredProtectionLevel = ProtectionLevel.EncryptAndSign
    });

// Do the initial authentication exchange through auth.GetOutgoingBlob
Authenticate(auth, stream);

// Exchange length-prefixed messages between client and server

ArrayBufferWriter<byte> output = new ArrayBufferWriter<byte>();
bool isConfidential = true;
NegotiateAuthenticationStatusCode statusCode;
// Encrypt and sign the word "hello"
statusCode = ntAuth.Wrap("hello"u8, output, ref isConfidential);
if (statusCode == NegotiateAuthenticationStatusCode.Completed)
{
    BinaryPrimitives.WriteInt32LittleEndian(messageLengthBuffer, output.WrittenCount);
    stream.Write(messageLengthBuffer);
    stream.Write(output.WrittenSpan);
    
    stream.ReadExactly(messageLengthBuffer);
    int messageLength = BinaryPrimitives.ReadInt32LittleEndian(messageLengthBuffer);

    // For simplicity I do allocations here but the APIs are designed to avoid them if desired
    byte[] messageBuffer = new byte[messageLength];
    stream.ReadExactly(messageBuffer);

    output.Clear(); // Reuse buffer
    statusCode = ntAuth.Unwrap(messageBuffer, output, out _);
    if (statusCode == NegotiateAuthenticationStatusCode.Completed)
    {
        Console.WriteLine("Got reply: " + Encoding.UTF8.GetString(output.WrittenSpan));
    }
}

Alternative Designs

No response

Risks

No response

Author:	filipnavara
Assignees:	-
Labels:	api-suggestion, area-System.Net.Security
Milestone:	-

[filipnavara]

Draft implementation available at https://github.com/filipnavara/runtime/tree/negotiate-wrap-api (based on PR #70720 which implements the initial shape of NegotiateAuthentication API). It adds the Wrap/Unwrap APIs and converts System.Net.Mail to use them instead of the internal NTAuthentication.

cc @wfurt
cc @xqrzd, would this work for your Kudu scenario?

[xqrzd]

It looks like it would work. I'll give it a try when it's ready.

[filipnavara]

I updated the proposal slightly to mention exceptions and rename the isConfidential parameter to isEncrypted to match the existing IsEncrypted property.

[karelz]

Triage: It would allow us to avoid Reflection in SqlClient.
If we make it before platform complete (7/12), we could take it into 7.0, otherwise Future.

[filipnavara]

Mainly pending an API review to finalize the public API shape. I have the implementation mostly ready and it should be really quick to tweak it to the final API shape. This would be useful for us (eM Client) to drop private interop bindings too.