The remote certificate is invalid according to the validation procedure

[lukos]

Running a dotnet core 2.2 web api on Ubuntu 18.04 Docker image (mcr.microsoft.com/dotnet/core/aspnet:2.2-bionic). I am using MailKit (https://github.com/jstedfast/MailKit) to send an email and it works fine when I run locally on Windows 10 but fails on Linux:

System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure

Since it works on Windows OK, it seems that there is perhaps an OpenSSL issue, maybe due to 1.0 vs 1.1? MailKit calls into SslStream.AuthenticateAsClient, which is the start of the stack that fails. Since this is an email relay, I need some other way of debugging that rather than just that https works generally (which it does with curl).

I have tried the following to no avail:

1. Downloaded and updated the root CA stack using update-ca-certificates (no difference)
2. Used curl from the container itself to access the hello world lets encrypt web site to see if there is a problem with LE specifically (works fine)
3. Set environment variables for SSL_CERT_DIR (/usr/lib/ssl/certs) and SSL_CERT_FILE (/usr/lib/ssl/cert.pem) after symlinking the downloaded CA file into /usr/lib/ssl (no difference).

Can someone please give me some more debugging tips? Thanks.

[vcsjones]

I'm not familiar with MailKit, but after looking at their docs, it looks like they use the same ServerCertificateValidationCallback that's used elsewhere in .NET. You could then theoretically implement a validation callback that does additional logging, which should let you examine any validation errors. Something like:

static void Main(string[] args) {
    MailKit.Net.Smtp.SmtpClient client = default; //setup client
    client.ServerCertificateValidationCallback = (sender, cert, chain, errors) => {
        // Log errors
        // Log chain
        // Log cert
        return errors == SslPolicyErrors.None;
    };
    // Do thing with client.
}

You could also attempt to connect to the mail server using openssl directly and see what it produces, something like:

openssl s_client -connect smtp.example.com:587 -starttls smtp

[lukos]

So I tried with openssl and that looked to work OK. I didn't really know what I was doing afterwards but I could start the EHLO process with no issues.

I logged the results of the ServerCertificateValidationCallback, which shows RemoteCertificateChainErrors, the chain is shown below and it looks complete i.e. normal cert, intermediate and root but I don't know if there is an easy way to see this chain more like a browser shows is? I also returned "true" to ignore the errors and the message is sent without a problem.

[Subject]
  CN=ms9.ssmx.net

[Issuer]
  CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US

[Serial Number]
  031FC5DBAC17B9EB28B798EC5120EF57608E

[Not Before]
  11/15/2019 16:04:59

[Not After]
  02/13/2020 16:04:59

[Thumbprint]
  CE402F9055B9888E00F2525CC3F5DF0835D67BCA


[Subject]
  CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US

[Issuer]
  CN=DST Root CA X3, O=Digital Signature Trust Co.

[Serial Number]
  0A0141420000015385736A0B85ECA708

[Not Before]
  03/17/2016 16:40:46

[Not After]
  03/17/2021 16:40:46

[Thumbprint]
  E6A3B45B062D509B3382282D196EFE97D5956CCB


[Subject]
  CN=DST Root CA X3, O=Digital Signature Trust Co.

[Issuer]
  CN=DST Root CA X3, O=Digital Signature Trust Co.

[Serial Number]
  44AFB080D6A327BA893039862EF8406B

[Not Before]
  09/30/2000 21:12:19

[Not After]
  09/30/2021 14:01:15

[Thumbprint]
  DAC9024F54D8F6DF94935FB1732638CA6AD77C13

[lukos]

OK, so now I have some more information. I logged the chain element status and I saw this for my top level certificate:

unable to get certificate CRL

This should give me more to dig into.

[vcsjones]

If you are getting RemoteCertificateChainErrors, then take a look at chain.ChainStatus array to determine what parts of the chain is invalid.

[lukos]

Hi @vcsjones that's the error I posted above.

So apparently Lets Encrypt do not include a CRL for the server cert but use OCSP stapling instead so perhaps a better question is: Should the SSL connection be failing due to a CRL missing in the cert and not just fallback to OCSP?

Of course I can leave the server check bypassed or try and ignore that specific error but that sounds a bit smelly to me - I am hoping there is a proper way to get around this?

[davidsh]

Can you try using .NET Core 3.0? There were a lot of fixes to TLS/SSL handling and perhaps one of those fixes might solve your problem.

[wfurt]

Ping @lukos? 2.2 will be out of support this month.

[bartonjs]

In .NET Core 2.2 we didn't have support for OCSP on Linux. We added OCSP on Linux in .NET Core 3.0, so Let's Encrypt chains should cleanly build with revocation now.

If you're seeing otherwise, let us know.

[pawod]

I'm still gettting following errors targeting netcoreapp3.1 :

unable to get certificate CRL
unable to get local issuer certificate
Subject: CN=smtp.gmail.com, O=Google LLC, L=Mountain View, S=California, C=US
Issuer: CN=GTS CA 1O1, O=Google Trust Services, C=US

Minimal code example for repro with MailKit:

var client = new SmtpClient();
client.Connect("smtp.gmail.com", 587, SecureSocketOptions.StartTls);
client.Authenticate(_options.MailAccountName, _options.MailAccountPassword);
var message = new MimeMessage();
message.From.Add(author);
message.ReplyTo.Add(author);
message.To.Add(recipient);
message.Subject = subject;
message.Body = new TextPart(MimeKit.Text.TextFormat.Plain) { Text = text };
await client.SendAsync(message);

when checking against openssl everything is fine with the certs

openssl s_client -CApath /etc/ssl/certs/ -connect smtp.gmail.com:587 -starttls smtp
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com
   i:C = US, O = Google Trust Services, CN = GTS CA 1O1
 1 s:C = US, O = Google Trust Services, CN = GTS CA 1O1
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFizCCBHOgAwIBAgIRALAfsBFF7jp2CAAAAAAqsrIwDQYJKoZIhvcNAQELBQAw
QjELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczET
MBEGA1UEAxMKR1RTIENBIDFPMTAeFw0yMDAxMjEwODE0NTdaFw0yMDA0MTQwODE0
NTdaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
Ew1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgTExDMRcwFQYDVQQDEw5z
bXRwLmdtYWlsLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKg4
68gPXFngWV99XtDhpohHtfmkCFriYf6TJnc8kh0t8VnRjjwoDEt0uEkVyiTkAkjZ
cliqHcKf06h87fGmoA4FtnsKuBWhPeYPU76QR88tSmwDRI1TMcRJjRZmkRAypK+E
H3fHySbudMtdUnm5/G7WoWDtxKp8lAG8L2CHwgmDpgi2n/9TnQ2iZpu+L2SgDbbm
RXdSTo5RhkfDVZ8mVXqT61AFLX62x+0+GzoAh50Ye5145mC1ru7nH89Qno6xf6n+
XhTFmlJNbrORUCD5sHsehkEI4D/e+8O5ndIY7yz9G3uYwGhSm7kUBL5BLtYYH/1H
PyyN2WFiJ064xwcIKNECAwEAAaOCAlQwggJQMA4GA1UdDwEB/wQEAwIFoDATBgNV
HSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTbio0y2MGh
pZVCc2D6W2F1vaVdczAfBgNVHSMEGDAWgBSY0fhuEOvPm+xgnxiQG6DrfQn9KzBk
BggrBgEFBQcBAQRYMFYwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLnBraS5nb29n
L2d0czFvMTArBggrBgEFBQcwAoYfaHR0cDovL3BraS5nb29nL2dzcjIvR1RTMU8x
LmNydDAZBgNVHREEEjAQgg5zbXRwLmdtYWlsLmNvbTAhBgNVHSAEGjAYMAgGBmeB
DAECAjAMBgorBgEEAdZ5AgUDMC8GA1UdHwQoMCYwJKAioCCGHmh0dHA6Ly9jcmwu
cGtpLmdvb2cvR1RTMU8xLmNybDCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2ALIe
BcyLos2KIE6HZvkruYolIGdr2vpw57JJUy3vi5BeAAABb8diMFoAAAQDAEcwRQIg
Z/KVgKafdXDy+ApADniOxmKR/vwjwkGX3vFoSXQSKlkCIQCLumhFIcD0T1QA4auf
2pD2tVXvfQPRdoKihSfgVRCcgQB2AF6nc/nfVsDntTZIfdBJ4DJ6kZoMhKESEoQY
dZaBcUVYAAABb8diMH4AAAQDAEcwRQIhAJjOvO183KgUETiIpAt99NAvZPrY1KOp
xF49fkGWic/pAiBdLafQlkyz25SQvXu8rcz7pFH5x04L4cW4WtTdm6WpBDANBgkq
hkiG9w0BAQsFAAOCAQEAUVjgtKD1pxkisLV9VStkhyqpgd9JJ/l1W92AcF/RZ3Yf
/72ARCDXpfha3IMmVpqfmKUP03Yo7xM3GBHjgPxRtq1K/Hd8xk/ovVZ0ihugCwf2
8FXpEpcbuPm/QBdL9uZg8B+kT3UBsYggN9qrEvXLfODL6HwRHNMN1K5xaJyeQOf8
JZ8HX/JwTQS7iHGLajv9cT3ABUq4En8hy3KdGscV+fPs/qLdmYCDHpTmCYxW7G7q
Q0NKvXY0waiffFy0hxSnhMLoU4n/b4Kiv1AeWb1ZgtO4IY8O5f6wdQIXD/fcEjEr
UDMjAw4Ooy4Szy8gw9eXpot8yaFeGqwqyQML4MF28w==
-----END CERTIFICATE-----
subject=C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com

issuer=C = US, O = Google Trust Services, CN = GTS CA 1O1

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3271 bytes and written 429 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 SMTPUTF8

help(?)

[davidsh]

@bartonjs