Cannot Establish Streaming Connection to FedRAMP Instance

[drewpc]

Describe the bug
Beginning around May 7, 2024, the Erlang Server SDK stopped connecting to the streaming FedRAMP server at https://stream.launchdarkly.us. This seems to be due to a cipher mismatch while negotiating the TLS connection. I believe what is happening is that the server now supports both TLS v1.2 and v1.3, but the Erlang Server SDK attempts to negotiate a TLS v1.3 connection with only ciphers for TLS v1.2. Wireshark packet captures show the client initiating a TLS connection and the server immediately closing with an Alert "close notify" packet.

The issue appears to be in the code that filters ciphers based on key exchange algorithms. This code filters out any ciphers for only TLS v1.2:

erlang-server-sdk/src/ldclient_config.erl

Lines 398 to 420 in 0b2efee

	-spec tls_base_options() -> [ssl:tls_client_option()].
	tls_base_options() ->
	DefaultCipherSuites = ssl:cipher_suites(default, 'tlsv1.2'),
	CipherSuites = ssl:filter_cipher_suites(DefaultCipherSuites, [
	{key_exchange, fun
	(ecdhe_ecdsa) -> true;
	(ecdhe_rsa) -> true;
	(_) -> false
	end
	},
	{mac, fun
	(sha) -> false;
	(_) -> true
	end
	}
	]),
	[{verify, verify_peer},
	{ciphers, CipherSuites},
	{depth, 3},
	{customize_hostname_check, [
	{match_fun, public_key:pkix_verify_hostname_match_fun(https)}
	]}].

If LaunchDarkly's intent is to only support TLS v1.2, then tls_base_options/0 must be updated to restrict the connection to TLS v1.2 only. If the intent is to support both TLS v1.2 and v1.3, then ciphers for both TLS v1.2 and v1.3 must be included in the list. Specific information can be found on the Erlang Hardening SSL page.

To reproduce
The following code samples are implemented in Elixir using iex -S mix.

This fails:

opts = %{
  :base_uri => 'https://sdk.launchdarkly.us',
  :events_uri => 'https://events.launchdarkly.us',
  :http_options => %{
    :tls_options => :ldclient_config.tls_basic_options()
  },
  :stream_uri => 'https://stream.launchdarkly.us'
}

:ldclient.start_instance(ld_sdk_key, :default, opts)

This succeeds by explicitly only allowing TLS v1.2 connections to the server:

opts = %{
  :base_uri => 'https://sdk.launchdarkly.us',
  :events_uri => 'https://events.launchdarkly.us',
  :http_options => %{
    :tls_options => :ldclient_config.tls_basic_options() ++ [
      versions: [:"tlsv1.2"]
    ]
  },
  :stream_uri => 'https://stream.launchdarkly.us'
}

:ldclient.start_instance(ld_sdk_key, :default, opts)

Expected behavior
Prior to May 7, 2024, these connections to the stream server succeeded and all flag data was retrieved.

Logs

18:49:12.887 [notice] Starting streaming update server for :default
18:49:12.891 [notice] Starting streaming connection to URL: ~c"https://stream.launchdarkly.us/all"
18:49:13.238 [warning] Error establishing streaming connection (:temporary): ~c"Could not open connection to host", will retry in 546 ms
18:49:13.786 [notice] Reconnecting streaming connection...

SDK version
version 3.2.0

Language version, developer tools
erlang 25.3.2
elixir 1.15.4-otp-25

OS/platform
macOS 14.4.1 and Ubuntu 20.04

[cwaldren-ld]

Hi @drewpc , appreciate you reporting this.

Is this currently impacting your operations, or are you in a testing/development phase with the Erlang SDK?

Filed internally as 244788.

[drewpc]

Thanks for the response! The workaround I suggested above is sufficient for us that it won't impact operations when we move into production in the next few weeks.

[kinyoklion]

@drewpc Version 3.3.0 of the SDK has been released and supports TLS 1.3 on OTP 23 and greater. When on an OTP version older than 23 it will limit the connection to TLS 1.2.

Thank you!
Ryan

[drewpc]

thank you so much for the quick response!