HttpClientHandler.ServerCertificateCustomValidationCallback blows up on OSX

[niemyjski]

I just updated my Project to RTM and did testing on Windows and everything worked. I then went to test on OSX 10.11.5 (latest stable everything, dotnet, openssl) and ran into issues very quickly..

I'm getting The libcurl library in use (7.43.0) and its SSL backend (\"SecureTransport\") do not support custom handling of certificates. A libcurl built with OpenSSL is required. in stable latest everything on macOS...

"   at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)\n   at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()\n   at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\n   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\n   at Exceptionless.Submission.DefaultSubmissionClient.GetSettings(ExceptionlessConfiguration config, Int32 version, IJsonSerializer serializer) in /Users/blake/Code/Exceptionless.Net/src/Exceptionless/Submission/DefaultSubmissionClient.cs:line 89" [string]

5]  
blake@:~/Code/Exceptionless.Net$ brew info openssl
openssl: stable 1.0.2h (bottled) [keg-only]
SSL/TLS cryptography library

[2:16]  
lake@:~/Code/Exceptionless.Net$ brew install openssl
Warning: openssl-1.0.2h_1 already installed
blake@:~/Code/Exceptionless.Net$ brew link --force openssl
Warning: Already linked: /usr/local/Cellar/openssl/1.0.2h_1
To relink: brew unlink openssl && brew link openssl
blake@:~/Code/Exceptionless.Net$

I narrowed it down to defining a custom validation callback breaks the http client.. I can't even do a GET.

var handler = new HttpClientHandler { UseDefaultCredentials = true };
handler.ServerCertificateCustomValidationCallback = delegate { return true; };
var client = new HttpClient(handler, true);
client.GetAsync(url).GetAwaiter().GetResult();

All the code can be found here: http://github.com/exceptionless/Exceptionless.Net

[niemyjski]

Possibly references:

• https://github.com/dotnet/corefx/issues/7812
• A libcurl built with OpenSSL is required. wcf#1295

[davidsh]

cc: @stephentoub

[stephentoub]

Yes, to use several of the HttpClient features related to certificates, like ServerCertificateCustomValidationCallback, you need to be using a version of libcurl built against OpenSSL; the version you're using is the one that comes with macOS, built against the SecureTransport backend. We do not have the ability to plug in to that backend, hence we throw an exception to indicate that the callback won't be used.

[niemyjski]

@stephentoub I don't think this acceptable. I couldn't make any http request because of it and I'm using the install instructions on the main website which many many others are going to use (that are on osx). Could we get a check to see if it's supported like other handler features? Any request without validation callback is better than nothing.. I get why you are throwing it.. but most peoples http client code is going to be try catched and they are not going to see this...

[stephentoub]

I'm using the install instructions on the main website

We can look to augment the instructions / release notes with related details.
cc: @leecow

Any request without validation callback is better than nothing

I'm not understanding this part. If you want to avoid the error, you can avoid hooking up the callback. And if you've specified a callback, not invoking it could be a significant security hole, as we'd end up using the default validation of a security certificate rather than the validation the developer has explicitly requested via the callback.

but most peoples http client code is going to be try catched and they are not going to see this

I don't understand this part either. Can you elaborate?

Could we get a check to see if it's supported like other handler features?

You mean a capability property, like IsCustomServerCertificateValidationSupported? That's a reasonable request for the future. Thanks.

cc: @ericeil, @bartonjs

[niemyjski]

Yeah, I get all of your points and agree when it comes to security.

if (handler.SupportsServerCertificateValidationCallback) {}

[joshfree]

@stephentoub @bartonjs feel free to submit a PR to update the release notes / docs to make this more clear.

[atanasa]

How is this supposed to work on OSX? Install some package on OSX? Or there is no way to make it work on OSX?

[niemyjski]

I don't think there is any way for this to work on OSX currently (at least I haven't found a way)

[stephentoub]

I don't think there is any way for this to work on OSX currently (at least I haven't found a way)

I'm sure there are other ways, but I did:

brew install curl --with-openssl
brew link --force curl

and then with that library being used:

export DYLD_LIBRARY_PATH=/usr/local/lib/

I can successfully use these features...

stoubmac:~ stoub$ cd testapp
stoubmac:testapp stoub$ cat Program.cs 
using System;
using System.Net.Http;

namespace ConsoleApplication
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var handler = new HttpClientHandler { CheckCertificateRevocationList = true };
            var http = new HttpClient(handler);
            Console.WriteLine(http.GetAsync("https://www.bing.com").GetAwaiter().GetResult());
        }
    }
}
stoubmac:testapp stoub$ dotnet run
Project testapp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.

Unhandled Exception: System.PlatformNotSupportedException: The libcurl library in use (7.43.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.
   at System.Net.Http.CurlHandler.SslProvider.SetSslOptions(EasyRequest easy, ClientCertificateOption clientCertOption)
   at System.Net.Http.CurlHandler.EasyRequest.InitializeCurl()
   at System.Net.Http.CurlHandler.MultiAgent.ActivateNewRequest(EasyRequest easy)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ConsoleApplication.Program.Main(String[] args)
stoubmac:testapp stoub$ export DYLD_LIBRARY_PATH=/usr/local/lib/
stoubmac:testapp stoub$ dotnet run
Project testapp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Cache-Control: max-age=0, private
  Vary: Accept-Encoding
  Server: Microsoft-IIS/8.5
  P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
  Set-Cookie: SRCHD=AF=NOFORM; domain=.bing.com; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
  Set-Cookie: SRCHUID=V=2&GUID=05A7870E43E942F68BAC35BB8A4C52C6; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
  Set-Cookie: SRCHUSR=DOB=20160721; domain=.bing.com; expires=Sat, 21-Jul-2018 23:49:04 GMT; path=/
  Set-Cookie: _SS=SID=0A99E65098CC6D950303EF09991F6C8C; domain=.bing.com; path=/
  Set-Cookie: _EDGE_S=F=1&SID=0A99E65098CC6D950303EF09991F6C8C; path=/; httponly; domain=bing.com
  Set-Cookie: _EDGE_V=1; path=/; httponly; expires=Sat, 21-Jul-2018 23:49:04 GMT; domain=bing.com
  Set-Cookie: MUID=3441EE6881166AB927F3E73180C56B38; path=/; expires=Sat, 21-Jul-2018 23:49:04 GMT; domain=bing.com
  Set-Cookie: MUIDB=3441EE6881166AB927F3E73180C56B38; path=/; httponly; expires=Sat, 21-Jul-2018 23:49:04 GMT
  X-MSEdge-Ref: Ref A: 8C58C90CF2764DCC8D6E5F4EBDDA771C Ref B: 3F2B603999D10209F1CECA268112300C Ref C: Thu Jul 21 16:49:04 2016 PST
  Date: Thu, 21 Jul 2016 23:49:04 GMT
  Content-Type: text/html; charset=utf-8
}
stoubmac:testapp stoub$

[alanrenouf]

Fantastic, the export DYLD_LIBRARY_PATH=/usr/local/lib/ was what i was missing! Thanks

[jchannon]

What's the fix for CentOS?

[bartonjs]

@jchannon CentOS 6 had a curl-openssl package which would (have) work(ed). The problem is that the default libcurl package on CentOS uses Mozilla NSS as the TLS provider, and we don't have the ability to interpret their certificate pointers.

Normal CentOS curl:

$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3

Normal Ubuntu curl:

$ curl --version
curl 7.38.0 (x86_64-pc-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3

Note the Ubuntu one in this example is OpenSSL/1.0.1f, whereas the CentOS one uses NSS/3.19.1 Basic ECC.

Since curl doesn't have their own abstraction of certificates we would have to have support for each of the (currently 8) possible providers. Right now OpenSSL is the only one.

Since there's not a CentOS package for that anymore that I can find offhand; that might mean you have to build from source. Or maybe there's just something I'm missing in the CentOS package tree.