Akka.IO: redesign TcpConnection

[Aaronontheweb]

Changes

Addresses a significant number of reliability and performance issues with Akka.IO.Tcp

close #7634

Still need to add:

• Pull mode for TcpConnection - Akka.Streams.IO blows up without this
• Pull mode for TcpListener - same deal
• Better buffer management code for TcpConnection writing - if someone sends a message larger than the max frame size we'll hang currently. Better to paginate it.
• Need to restore the max-queue-size as byte-size semantics in our tests. I was against this initially but IMHO I think it's probably the more sensible "real measure" of resources

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

• This change follows the Akka.NET API Compatibility Guidelines.
• This change follows the Akka.NET Wire Compatibility Guidelines.
• I have reviewed my own pull request.
• Design discussion issue Akka.IO: TcpConnection does not propagate closure events reliably #7634
• Changes in public API reviewed, if any.

Latest dev Benchmarks

BenchmarkDotNet v0.13.12, Windows 11 (10.0.26100.3775)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.403
  [Host]  : .NET 8.0.14 (8.0.1425.11118), X64 RyuJIT AVX2
  LongRun : .NET 8.0.14 (8.0.1425.11118), X64 RyuJIT AVX2

Job=LongRun  Concurrent=True  Server=True  
InvocationCount=1  IterationCount=10  LaunchCount=3  
RunStrategy=Monitoring  UnrollFactor=1  WarmupCount=3  

Method	MessageLength	ClientsCount	Mean	Error	StdDev	Req/sec
ClientServerCommunication	10	1	53.038 μs	2.3749 μs	3.5546 μs	18,854.32
ClientServerCommunication	10	3	22.307 μs	0.9668 μs	1.4470 μs	44,829.77
ClientServerCommunication	10	5	15.343 μs	1.7903 μs	2.6797 μs	65,177.89
ClientServerCommunication	10	7	13.806 μs	1.5307 μs	2.2911 μs	72,433.14
ClientServerCommunication	10	10	11.569 μs	0.9201 μs	1.3771 μs	86,436.51
ClientServerCommunication	10	20	9.188 μs	0.8901 μs	1.3322 μs	108,841.90
ClientServerCommunication	10	30	9.048 μs	0.8720 μs	1.3051 μs	110,519.01
ClientServerCommunication	10	40	10.213 μs	0.4584 μs	0.6861 μs	97,915.46
ClientServerCommunication	100	1	57.817 μs	5.1364 μs	7.6880 μs	17,296.00
ClientServerCommunication	100	3	21.813 μs	0.9598 μs	1.4365 μs	45,843.55
ClientServerCommunication	100	5	16.136 μs	1.3610 μs	2.0370 μs	61,974.70
ClientServerCommunication	100	7	14.732 μs	2.3724 μs	3.5510 μs	67,878.50
ClientServerCommunication	100	10	11.896 μs	1.4770 μs	2.2107 μs	84,058.77
ClientServerCommunication	100	20	9.574 μs	0.8134 μs	1.2174 μs	104,445.29
ClientServerCommunication	100	30	9.879 μs	0.7771 μs	1.1631 μs	101,222.03
ClientServerCommunication	100	40	9.455 μs	0.9147 μs	1.3691 μs	105,759.12

This PR's Benchmarks

BenchmarkDotNet v0.13.12, Windows 11 (10.0.26100.3775)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.403
  [Host]  : .NET 8.0.14 (8.0.1425.11118), X64 RyuJIT AVX2
  LongRun : .NET 8.0.14 (8.0.1425.11118), X64 RyuJIT AVX2

Job=LongRun  Concurrent=True  Server=True
InvocationCount=1  IterationCount=10  LaunchCount=3
RunStrategy=Monitoring  UnrollFactor=1  WarmupCount=3

Method	MessageLength	ClientsCount	Mean	Error	StdDev	Req/sec
ClientServerCommunication	10	1	44.998 us	1.9392 us	2.9025 us	22,223.07
ClientServerCommunication	10	3	20.735 us	1.1877 us	1.7777 us	48,227.95
ClientServerCommunication	10	5	15.255 us	1.2650 us	1.8934 us	65,550.70
ClientServerCommunication	10	7	13.271 us	0.9893 us	1.4807 us	75,351.45
ClientServerCommunication	10	10	10.537 us	1.0431 us	1.5613 us	94,905.86
ClientServerCommunication	10	20	8.821 us	0.6686 us	1.0007 us	113,359.97
ClientServerCommunication	10	30	8.886 us	0.5094 us	0.7624 us	112,542.02
ClientServerCommunication	10	40	9.362 us	0.8531 us	1.2769 us	106,809.83
ClientServerCommunication	100	1	43.620 us	3.1453 us	4.7078 us	22,925.43
ClientServerCommunication	100	3	20.511 us	1.1551 us	1.7288 us	48,754.38
ClientServerCommunication	100	5	15.767 us	1.9737 us	2.9541 us	63,425.26
ClientServerCommunication	100	7	13.081 us	1.4413 us	2.1573 us	76,444.56
ClientServerCommunication	100	10	10.298 us	0.7608 us	1.1388 us	97,105.06
ClientServerCommunication	100	20	9.168 us	0.8832 us	1.3219 us	109,070.42
ClientServerCommunication	100	30	8.572 us	0.6353 us	0.9509 us	116,655.03
ClientServerCommunication	100	40	9.378 us	1.0401 us	1.5568 us	106,630.69

[Aaronontheweb]

The most significant difference in performance here is the single client test, which accurately measures single socket read / write performance - we went from 18k to 22k operations per second there, an improvement of ~18%

[Aaronontheweb]

Currently blocked on some original design issues - basically Akka.IO is missing some basic features you want for real TCP server applications, namely:

• Explicit system buffer sizes
• Explicit max frame sizes

The other issue I have with the settings code, in particular, is the question of how large the write buffer should be allowed to grow. In Akka.Remote we don't really constraint this at all outside of capping the number of ISystemMessages that can reside in the reliable delivery queue.

[Aaronontheweb]

So apparently SocketOptions exposes a way to configure send and receive buffer sizes already, but this is totally undocumented and not obvious. Not sure how we want to address that because nowhere are any system defaults of any kind documented - and these settings have to be squared against the maximum frame size too.

[Aaronontheweb]

Now that #7651 has been merged, we need to get some new numbers