Skip to content

OpenSSL stores refreshing blocks multiple threads causing lock contention and thread pool exhaustion. #113227

@baal2000

Description

@baal2000

Description

A .NET 8 application could experience lock contention and the thread pool exhaustion symptoms with the process dump showing threads congregating on a lock inside System.Security.Cryptography.X509Certificates.OpenSslCachedSystemStoreProvider.GetCollections().

From one of the collected dumps, there are 100s of threads waiting on

System.Security.Cryptography.X509Certificates.OpenSslCachedSystemStoreProvider.GetCollections()
System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.InitiateChain(Microsoft.Win32.SafeHandles.SafeX509Handle, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.X509Certificates.X509ChainTrustMode, System.DateTime, System.TimeSpan)
System.Security.Cryptography.X509Certificates.ChainPal.BuildChainCore(Boolean, System.Security.Cryptography.X509Certificates.ICertificatePal, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.X509Certificates.X509RevocationMode, System.Security.Cryptography.X509Certificates.X509RevocationFlag, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.X509Certificates.X509ChainTrustMode, System.DateTime, System.TimeSpan, Boolean)
System.Security.Cryptography.X509Certificates.ChainPal.BuildChain(Boolean, System.Security.Cryptography.X509Certificates.ICertificatePal, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.X509Certificates.X509RevocationMode, System.Security.Cryptography.X509Certificates.X509RevocationFlag, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.X509Certificates.X509ChainTrustMode, System.DateTime, System.TimeSpan, Boolean)
System.Security.Cryptography.X509Certificates.X509Chain.Build(System.Security.Cryptography.X509Certificates.X509Certificate2, Boolean)
System.Net.Security.SslStreamCertificateContext.Create(System.Security.Cryptography.X509Certificates.X509Certificate2, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, Boolean, System.Net.Security.SslCertificateTrust, Boolean)
System.Net.Security.SslStream.AcquireServerCredentials(Byte[] ByRef)
System.Net.Security.SslStream.GenerateToken(System.ReadOnlySpan`1<Byte>, Byte[] ByRef)
System.Net.Security.SslStream.ProcessTlsFrame(Int32, System.Net.Security.ProtocolToken ByRef)
System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]], System.Net.Security]](<ForceAuthenticationAsync>d__150`1<System.Net.Security.AsyncReadWriteAdapter> ByRef)
System.Net.Security.SslStream+<ProcessAuthenticationWithTelemetryAsync>d__147.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Security.SslStream+<ProcessAuthenticationWithTelemetryAsync>d__147, System.Net.Security]](<ProcessAuthenticationWithTelemetryAsync>d__147 ByRef)
Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.DoOptionsBasedHandshakeAsync(Microsoft.AspNetCore.Connections.ConnectionContext, System.Net.Security.SslStream, Microsoft.AspNetCore.Server.Kestrel.Core.Internal.TlsConnectionFeature, System.Threading.CancellationToken)
Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware+<OnConnectionAsync>d__18.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware+<OnConnectionAsync>d__18, Microsoft.AspNetCore.Server.Kestrel.Core]](<OnConnectionAsync>d__18 ByRef)
Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(Microsoft.AspNetCore.Connections.ConnectionContext)
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib]], Microsoft.AspNetCore.Server.Kestrel.Core]](<ExecuteAsync>d__8<System.__Canon> ByRef)
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1[[System.__Canon, System.Private.CoreLib]].ExecuteAsync()
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1[[System.__Canon, System.Private.CoreLib]].System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.Dispatch()
System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
DebuggerU2MCatchHandlerFrame

Configuration

  • .NET 8
  • Ubuntu 20.04
  • x64

Regression?

This is not a regression in .NET 8

Analysis

The issue is already addressed in .NET 9 through a fixed double-checking logic


Suggestion

Patch #102838 back to .NET 8.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions