[H/3] Stress run in Debug fails on assert

[ManickaP]

Easy to reproduce locally with: .\run-stress-debug-debug.ps1 -cancelRate 1 -http 3.0

We use duplex streaming with custom contents. So when the cancellation happens, response reading side gets cancelled and disposes Http3RequestStream which disposes the ArrayBuffer fields. But the writing side might not have reacted to the CancellationToken yet and the next thing it does is touching the ArrayBuffer:

runtime/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs

Line 465 in c5e7081

_sendBuffer.Discard(_sendBuffer.ActiveLength);

Nothing bad will happen as the next QuicStream.WriteAsync will kill this task.

Note that similar problem was encountered with these asserts:

runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs

Lines 716 to 718 in c5e7081

	// TODO: If there was and ongoing read/write when dispose was called, MsQuic events might be processed, but the continuations may not thus the state can still be in Ready.
	//Debug.Assert(_receiveTcs.IsCompleted);
	//Debug.Assert(_sendTcs.IsCompleted);

In general, we do not access Http3RequestStream fields in a thread-safe manner in case of duplex streaming which can cause problems when we Dispose the object from the reading side while still having writing side active.

Question is whether this is worth fixing or not. Either way, we should have a paper trail about the decision.

Process terminated. Assertion failed.
Expected 2 <= 0
   at System.Net.ArrayBuffer.Discard(Int32 byteCount) in C:\Users\mapichov\runtime\src\libraries\Common\src\System\Net\ArrayBuffer.cs:line 103
   at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs:line 484
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 179
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 3456
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncValueTaskMethodBuilder.cs:line 41
   at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 419
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 264
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ThreadPoolWorkQueue.cs:line 913
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\PortableThreadPool.WorkerThread.NonBrowser.cs:line 102
   at System.Threading.Thread.StartCallback() in C:\Users\mapichov\runtime\src\coreclr\System.Private.CoreLib\src\System\Threading\Thread.CoreCLR.cs:line 104

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Easy to reproduce locally with: .\run-stress-debug-debug.ps1 -cancelRate 1 -http 3.0

We use duplex streaming with custom contents. So when the cancellation happens, response reading side gets cancelled and disposes Http3RequestStream which disposes the ArrayBuffer fields. But the writing side might not have reacted to the CancellationToken yet and the next thing it does is touching the ArrayBuffer:

runtime/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs

Line 465 in c5e7081

_sendBuffer.Discard(_sendBuffer.ActiveLength);

Nothing bad will happen as the next QuicStream.WriteAsync will kill this task.

Note that similar problem was encountered with these asserts:

runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs

Lines 716 to 718 in c5e7081

	// TODO: If there was and ongoing read/write when dispose was called, MsQuic events might be processed, but the continuations may not thus the state can still be in Ready.
	//Debug.Assert(_receiveTcs.IsCompleted);
	//Debug.Assert(_sendTcs.IsCompleted);

In general, we do not access Http3RequestStream fields in a thread-safe manner in case of duplex streaming which can cause problems when we Dispose the object from the reading side while still having writing side active.

Question is whether this is worth fixing or not. Either way, we should have a paper trail about the decision.

Process terminated. Assertion failed.
Expected 2 <= 0
   at System.Net.ArrayBuffer.Discard(Int32 byteCount) in C:\Users\mapichov\runtime\src\libraries\Common\src\System\Net\ArrayBuffer.cs:line 103
   at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs:line 484
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 179
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 3456
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncValueTaskMethodBuilder.cs:line 41
   at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 419
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 264
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ThreadPoolWorkQueue.cs:line 913
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\PortableThreadPool.WorkerThread.NonBrowser.cs:line 102
   at System.Threading.Thread.StartCallback() in C:\Users\mapichov\runtime\src\coreclr\System.Private.CoreLib\src\System\Threading\Thread.CoreCLR.cs:line 104

Author:	ManickaP
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-