-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Closed
Labels
Milestone
Description
Description
A .NET 8 application could experience lock contention and the thread pool exhaustion symptoms with the process dump showing threads congregating on a lock inside System.Security.Cryptography.X509Certificates.OpenSslCachedSystemStoreProvider.GetCollections().
From one of the collected dumps, there are 100s of threads waiting on
Line 94 in 0fdf85d
| lock (s_recheckStopwatch) |
System.Security.Cryptography.X509Certificates.OpenSslCachedSystemStoreProvider.GetCollections()
System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.InitiateChain(Microsoft.Win32.SafeHandles.SafeX509Handle, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.X509Certificates.X509ChainTrustMode, System.DateTime, System.TimeSpan)
System.Security.Cryptography.X509Certificates.ChainPal.BuildChainCore(Boolean, System.Security.Cryptography.X509Certificates.ICertificatePal, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.X509Certificates.X509RevocationMode, System.Security.Cryptography.X509Certificates.X509RevocationFlag, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.X509Certificates.X509ChainTrustMode, System.DateTime, System.TimeSpan, Boolean)
System.Security.Cryptography.X509Certificates.ChainPal.BuildChain(Boolean, System.Security.Cryptography.X509Certificates.ICertificatePal, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.OidCollection, System.Security.Cryptography.X509Certificates.X509RevocationMode, System.Security.Cryptography.X509Certificates.X509RevocationFlag, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, System.Security.Cryptography.X509Certificates.X509ChainTrustMode, System.DateTime, System.TimeSpan, Boolean)
System.Security.Cryptography.X509Certificates.X509Chain.Build(System.Security.Cryptography.X509Certificates.X509Certificate2, Boolean)
System.Net.Security.SslStreamCertificateContext.Create(System.Security.Cryptography.X509Certificates.X509Certificate2, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, Boolean, System.Net.Security.SslCertificateTrust, Boolean)
System.Net.Security.SslStream.AcquireServerCredentials(Byte[] ByRef)
System.Net.Security.SslStream.GenerateToken(System.ReadOnlySpan`1<Byte>, Byte[] ByRef)
System.Net.Security.SslStream.ProcessTlsFrame(Int32, System.Net.Security.ProtocolToken ByRef)
System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security]], System.Net.Security]](<ForceAuthenticationAsync>d__150`1<System.Net.Security.AsyncReadWriteAdapter> ByRef)
System.Net.Security.SslStream+<ProcessAuthenticationWithTelemetryAsync>d__147.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Security.SslStream+<ProcessAuthenticationWithTelemetryAsync>d__147, System.Net.Security]](<ProcessAuthenticationWithTelemetryAsync>d__147 ByRef)
Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.DoOptionsBasedHandshakeAsync(Microsoft.AspNetCore.Connections.ConnectionContext, System.Net.Security.SslStream, Microsoft.AspNetCore.Server.Kestrel.Core.Internal.TlsConnectionFeature, System.Threading.CancellationToken)
Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware+<OnConnectionAsync>d__18.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware+<OnConnectionAsync>d__18, Microsoft.AspNetCore.Server.Kestrel.Core]](<OnConnectionAsync>d__18 ByRef)
Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(Microsoft.AspNetCore.Connections.ConnectionContext)
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib]], Microsoft.AspNetCore.Server.Kestrel.Core]](<ExecuteAsync>d__8<System.__Canon> ByRef)
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1[[System.__Canon, System.Private.CoreLib]].ExecuteAsync()
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1[[System.__Canon, System.Private.CoreLib]].System.Threading.IThreadPoolWorkItem.Execute()
System.Threading.ThreadPoolWorkQueue.Dispatch()
System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
DebuggerU2MCatchHandlerFrame
Configuration
- .NET 8
- Ubuntu 20.04
- x64
Regression?
This is not a regression in .NET 8
Analysis
The issue is already addressed in .NET 9 through a fixed double-checking logic
Line 96 in d09a42b
| ret = s_nativeCollections; |
Line 97 in d09a42b
| elapsed = s_recheckStopwatch.Elapsed; |
Suggestion
Patch #102838 back to .NET 8.