Regression in System.Data.SqlClient >= 4.7.0-preview7 on Docker

[rmja]

The newest versions of System.Data.SqlClient throws when connecting to SQL Azure when running inside a Docker linux container.

• 4.7.0-preview6.19303.8 does not throw
• 4.7.0-preview7.19362.9 and 4.7.0-preview8.19405.3 throws when run in Docker

I have tried both the debian and alpine images and both seems to produce the exception. The code works fine on Windows.
I have also not seen any issues when connecting to a "real" SQL Server, but I have not investigated this thoroughly.

The reproducing code is the following:

using System.Data.SqlClient;

namespace Repro145
{
    class Program
    {
        static void Main()
        {
            using (var connection = new SqlConnection(ConnectionString))
            {
                connection.Open();

                var command = new SqlCommand("SELECT * FROM Catalog.Device", connection);

                // System.Data.SqlClient 4.7.0-preview7.19362.9 and later throws when run in Docker
                // System.Data.SqlClient 4.7.0-preview6.19303.8 does not throw
                var reader = command.ExecuteReader();

                while (reader.Read()) ;

                connection.Close();
            }
        }

        private const string ConnectionString = "Server=tcp:xxx.database.windows.net,1433;Initial Catalog=xxx;Persist Security Info=False;User ID=xxx;Password=xxx;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;";
    }
}

.csproj file

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.9.2" />
    
    <!-- Does not throw -->
    <!--<PackageReference Include="System.Data.SqlClient" Version="4.7.0-preview6.19303.8" />-->
    
    <!-- Throws -->
    <PackageReference Include="System.Data.SqlClient" Version="4.7.0-preview7.19362.9" />
    
  </ItemGroup>

</Project>

Dockerfile

FROM mcr.microsoft.com/dotnet/core/runtime:3.0.0-preview8-buster-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:3.0.0-preview8-buster AS build
WORKDIR /src
COPY ["Repro145/Repro145.csproj", "Repro145/"]
RUN dotnet restore "Repro145/Repro145.csproj"
COPY . .
WORKDIR "/src/Repro145"
RUN dotnet build "Repro145.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "Repro145.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Repro145.dll"]

Full stacktrace

System.Data.SqlClient.SqlException
  HResult=0x80131904
  Message=A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught)
  Source=Core .Net SqlClient Data Provider
  StackTrace:
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Repro145.Program.Main() in C:\Source\repro145\Repro145\Program.cs:line 17

Inner Exception 1:
IOException: The decryption operation failed, see inner exception.

Inner Exception 2:
SslException: Decrypt failed with OpenSSL error - SSL_ERROR_SSL.

Inner Exception 3:
OpenSslCryptographicException: error:1408F10B:SSL routines:ssl3_get_record:wrong version number

This issue is related to dotnet/SqlClient#145 for the Microsoft.Data.SqlClient equivalent.

[roji]

/cc @David-Engel @cheenamalhotra @Wraith2

As there haven't been many actual changes to System.Data.SqlClient recently, are we thinking this is possibly a lower-level issue?

[Wraith2]

Looks like an ssl issue. All the changes I've made to the managed implementation are well above transport level and I don't think there was much else done in there. I know there have been changes to the managed SslStream for 3.0 for kestrel performance though so it might be related to those, or it might be environmental (ssl version differences?)

[davidsh]

@wfurt

@bartonjs Can you help here? This might be an OpenSsl related change/bug.

[bartonjs]

It means the server responded with something that wasn't TLS. Looks like the dockerfile used is from Debian, which tends to have some configuration to alter the default enabled ciphersuites; it's possible that this is a bad gateway reporting that there's no mutual ciphersuite.

Really someone needs to capture a network trace. It might even be enough to try from a similar environment to run openssl s_client -connect $HOST:$PORT -msg -servername $HOST with $HOST being the name of an AzureSql server and $PORT being whatever port SQL talks on. If that doesn't also fail with "ssl3_get_record:wrong version number" then the network capture is required. (If it does, then at least we can say what gibberish the server sent 😄)

[rmja]

@bartonjs Here is the output from the openssl client connect:

It all seems fine. Please note that 4.7.0-preview6.19303.8 (and its dependencies) does not produce this issue - even though the debian image is the same.

[bartonjs]

That command output doesn't show any response from the server, and the cipher is (NONE), meaning it isn't actually established. Weird. So looks like it'll require a tcpdump capture of failing to talk to SQL. (Ideally a success with the preview6 package and a failure with preview7 package)

[Wraith2]

Azure SQL requires ssl for all connections. Standard SQL requires ssl for login but not after that point unless you request it. The stack trace indicates that the error occurs during a read operation not a login so it looks as if the login (and thus ssl connection) was possible but that it is now failing while being used.

tcpdump seems like the best way to see what's different as suggested.

[rmja]

@Wraith2 year, I am not competent enough to provide such dumps, sorry.

[wfurt]

What is server you connecting to @rmja? I can check SSL handshake even without getting access to the database.
As far as the Debian and Alpine: what exact versions do you use? I can guide you how to get dumps but if it would be best if I can reproduce it locally.

[rmja]

@wfurt can you mail me at my-github-username@laesoe.org. Please include your IP/range so that I can open the firewall appropriately.

I use the versions mcr.microsoft.com/dotnet/core/runtime:3.0.0-preview8-buster-slimand mcr.microsoft.com/dotnet/core/runtime:3.0.0-preview8-alpine.

[wfurt]

I think there is regression between preview6 and preview7. I did not get to running in container yet but I can easily reproduce the linked issue.

I published example as self-contained app on Ubuntu 18.04 and than:

git checkout XXX
git clean -dfx
./build.sh
cp artifacts/bin/System.Data.SqlClient/netcoreapp-Unix-Debug/System.Data.SqlClient.dll XXXX

With that, tip of master fails, dotnet/corefx@88c5a3e works fine and dotnet/corefx@2469a91 fails.

That points to dotnet/corefx#35363 as culprit of this.

I did not change anything in self-contained app besides System.Data.SqlClient.dll and that points to the assembly it self.

I also have packet captures but since everything is wrapped in TDS, it is hard to decode the actual TLS messages.

cc: @saurabh500 @Gary-Zh @cheenamalhotra

[Wraith2]

That'll be me then. So the repro is unix client and azure server instance and then query any table and it'll blow up?

[wfurt]

yes, it seems like. However I did not try OSX or Windows. It would be easy to do if that helps.

[Wraith2]

Windows would use the native implementation and that wouldn't be affected by the changes in the PR you linked to. OSX would be affected. If you're bored and want to try it confirmation would be interesting but don't waste time you don't have on it.

I probably won't be able to look at this until the weekend but I'll see if I can repro and track it down.