Handshake failing when client specifies custom certificate chain in its TLSConfiguration

[keithcorbs]

Hi all,

I have built an HTTP client using the Swift NIO and Swift NIO SSL framework, and my client needs to call a server endpoint that issues certificates signed with a private internal certificate authority. Because of this, my HTTP client needs to provide the internal public certificates needed to complete the SSL handshake and complete the request.

Given the current documentation, I tried doing this with swift-nio-ssl in two different ways, and neither of them worked. I tried the following:

1. Providing each certificate individually in the certificate chain of the TLS configuration:

let cert1 = try OpenSSLCertificate(file: "/usr/local/share/ca-certificates/cert-1.pem", format: OpenSSLSerializationFormats.pem)
let cert2 = try OpenSSLCertificate(file: "/usr/local/share/ca-certificates/cert-2.pem", format: OpenSSLSerializationFormats.pem)
let cert3 = try OpenSSLCertificate(file: "/usr/local/share/ca-certificates/cert-3.pem", format: OpenSSLSerializationFormats.pem)
let certificateChain: [OpenSSLCertificateSource] = [.certificate(cert1), .certificate(cert2), .certificate(cert3)]
let tlsConfiguration = TLSConfiguration.forClient(certificateChain: certificateChain)

2. Providing a single certificate, which is just all of the above certificates concatenated together:

let allCerts = try OpenSSLCertificate(file: "/usr/local/share/ca-certificates/allCerts.pem", format: OpenSSLSerializationFormats.pem)
let certificateChain: [OpenSSLCertificateSource] = [.certificate(allCerts)]
let tlsConfiguration = TLSConfiguration.forClient(certificateChain: certificateChain)

And both options caused my request to fail with the following Error in our errorCaught function of our ChannelInboundHandler protocol implementation:

Error received from HTTP connection: handshakeFailed(NIOOpenSSL.OpenSSLError.sslError([Error: 335573126 error:14007086:SSL routines:CONNECT_CR_CERT:certificate verify failed]))

Debugging further into this, it seems to go into the low level details of SSL and it might be beyond my high level comprehension of what's going on. Both seem to be failing in similar fashion.

To provide context into a related working way to call this manually - I was able to verify that I can get a response back with the following curl command by specifying the concatenated certificate file, and I was expecting similar behavior in swift-nio-ssl by providing the concatenated certificate file to the TLSConfiguration:

curl 'https://private-service.com:443/privateServiceOperation' -H 'Host https://private-service.com:443' -H 'User-Agent: CurlRequest' -H 'Accept: */*' --cacert /usr/local/share/ca-certificates/allCerts.pem

I was wondering if this was a supported feature by swift-nio-ssl, and if not, could it be a supported feature? Let me know if you need more details.

Thank you,
Keith

[Lukasa]

You don't want to use certificateChain, you want to set a custom trustRoot instead. Try this:

let rootCertsFile = "/usr/local/share/ca-certificates/allCerts.pem"
let tlsConfiguration = TLSConfiguration.forClient(trustRoots: .file(rootCertsFile))

[keithcorbs]

I was able to verify this working with a call to our internal service by passing this into the trustRoots as a client! This came a bit from my naiveness of SSL, I believed what I had was classified as a certificate chain, but the chain itself is actually what should be used as a trust root from a client perspective.

Thanks for the response and recommendation!

[Lukasa]

No problem whatsoever, TLS is a complex protocol and the terminology can definitely get confusing.

[YourMJK]

If you are using TLSConfiguration.additionalTrustRoots and getting a similar handshake error, use TLSConfiguration.trustRoots instead, like described above.

Don't know why, but that fixed the issue in my case! If I understand the API correctly, it shouldn't make a difference – but it does.

It was using the BoringSSL implementation instead of OpenSSL, so the specific error was
handshakeFailed(NIOSSL.BoringSSLError.sslError([Error: 268435581 error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at swift-nio-ssl/Sources/CNIOBoringSSL/ssl/handshake.cc:288]))

[Lukasa]

Do you have any minimal reproducer you could share? This shouldn't be necessary.

[YourMJK]

@Lukasa
I set up a repository that demonstrates the issue here: https://github.com/YourMJK/niossl-trust-test

However, I'm unable to generate a certificate that works with AsyncHTTPClient.
The openssl command that I use is in the Makefile, not sure what the issue is.

But that's why my HTTP client example produces the error:

• certificate is not permitted for this usage when using TLSConfiguration.trustRoots and
• certificate is not trusted when using TLSConfiguration.additionalTrustRoots

I originally had the issue with MQTT NIO and a MQTT broker using TLS.
So my MQTT client example in the repo is fully working and demonstrates the issue (but needs a running MQTT broker).

[Lukasa]

Hmm, I'm weirdly struggling to get mosquitto to actually do TLS. Do you have an example config file that uses these certs that I can use?

[YourMJK]

Sure, the minimal mosquitto.conf is fairly straight-forward:

listener 8883

allow_anonymous true  # don't require username and password

certfile /path/to/cert.pem
keyfile /path/to/key.pem

Adjust the paths to point to the PEM files in my repo and then run it with with $ mosquitto -c mosquitto.conf

I set it up in my repository as well now, so you can just use the start script if you have mosquitto installed locally.

[Lukasa]

Great, thanks for filing this.

This boils down to a behavioural difference on Apple platforms between what happens if you set trustRoots and what happens if you set additionalTrustRoots. Today, we have the following matrix of possibilities:

	Apple OS	Other OS
trustRoots = .default, no additional trust roots	SecTrust for validation with default settings	BoringSSL for validation using PEM files found on disk
trustRoots = nil, no additional trust roots	SecTrust for validation with default settings	BoringSSL for validation using PEM files found on disk
trustRoots = .file, no additional trust roots	BoringSSL for validation using certs from passed file	BoringSSL for validation using certs from passed file
trustRoots = .certificates, no additional trust roots	BoringSSL for validation using passed certs	BoringSSL for validation using passed certs
trustRoots = .default, additional trust roots provided	SecTrust for validation with additional roots passed using SecTrustSetAnchorCertificates	BoringSSL for validation using PEM files on disk and additional roots
trustRoots = nil, additional trust roots provided	SecTrust for validation with additional roots passed using SecTrustSetAnchorCertificates	BoringSSL for validation using PEM files on disk and additional roots
trustRoots = .file, additional trust roots provided	BoringSSL for validation using certs from passed file and additional roots	BoringSSL for validation using certs from passed file and additional roots
trustRoots = .certificates, additional trust roots provided	BoringSSL for validation using passed certs and additional roots	BoringSSL for validation using passed certs and additional roots

Importantly, note that on Apple platforms if you only set additional trust roots, you will use those roots with the system validator, SecTrust. Alternatively, if you set trustRoots, you use those roots with the BoringSSL validator.

In cases where you see an issue, like this one, you can use the system log to find out what the problem is. Here, the system log in this case offers this log message:

default	14:03:04.231518+0100	mqtt-client	Trust evaluate failure: [leaf ServerAuthEKU]

The message being communicated here is that SecTrust doesn't trust the certificate chain because the leaf cert is missing the server authentication extended key usage bit. This can be changed by modifying the way you create your certificates. For example, if you're using swift-certificates then you can add the extended key usage directly:

let extensions = try Certificate.Extensions {
    Critical(
        BasicConstraints.isCertificateAuthority(maxPathLength: nil)
    )
    ExtendedKeyUsage([.serverAuth])
    SubjectAlternativeNames([.dnsName("niossl.test")])
}

In general, SecTrust is substantially stricter than BoringSSL's validator, so it will reject many chains that BoringSSL will accept.

In case it's helpful, a full certificate generator that works for your use-case is:

import ArgumentParser
import Crypto
import Foundation
import SwiftASN1
import X509

@main
struct CertGenerator: ParsableCommand {
    @Option
    var keyPath: String

    @Option
    var certPath: String

    func run() throws {
        let keyBytes = try String(contentsOf: URL(filePath: keyPath))
        let key = try Certificate.PrivateKey(pemEncoded: keyBytes)

        let subjectName = try DistinguishedName {
            CommonName("niossl.test")
        }
        let issuerName = subjectName

        let now = Date()
        let ski = SubjectKeyIdentifier(hash: key.publicKey)

        let extensions = try Certificate.Extensions {
            Critical(
                BasicConstraints.isCertificateAuthority(maxPathLength: nil)
            )
            try ExtendedKeyUsage([.serverAuth])
            SubjectAlternativeNames([.dnsName("niossl.test")])
            ski
            AuthorityKeyIdentifier(keyIdentifier: ski.keyIdentifier)
        }

        let certificate = try Certificate(
            version: .v3,
            serialNumber: Certificate.SerialNumber(),
            publicKey: key.publicKey,
            notValidBefore: now,
            notValidAfter: now.addingTimeInterval(60 * 60 * 24 * 365),
            issuer: issuerName,
            subject: subjectName,
            signatureAlgorithm: .sha256WithRSAEncryption,
            extensions: extensions,
            issuerPrivateKey: key
        )

        let pemEncodedCertificate = try certificate.serializeAsPEM()
        try Data(pemEncodedCertificate.pemString.utf8).write(to: URL(filePath: certPath))
    }
}

You aren't the first person to hit this in the last few weeks, so I think I'll add documentation for this behaviour. I'm somewhat reluctant to change it at this time, but if it's going to be weird it should at least be well-documented.

[Lukasa]

Documentation at #548

[YourMJK]

Thanks!

This makes it feel so obvious now.
Should have known that I can't get away with skipping the certificate extensions and that they would be checked during validation …

Sorry for causing so much effort.
But undoubtedly someone else that's equally as lazy as me will stumble over this in the future and will benefit from the documentation :)

[Lukasa]

There's absolutely no need for an apology: you weren't even the first person to report this problem to me this week. So it's clearly a problem that people encounter!