Consider only reusing TCP port when hole punching

[mxinden]

Background

One can use the same port both listening for incoming connections and dialing outgoing connections. This is controlled via the SO_REUSEPORT socket option.

Port reuse is necessary for hole punching via TCP. It allows one to discover its outwards facing listening port via an outgoing TCP connection. The outgoing connection reuses the local listening port and is thus likely assigned the same outwards facing port as the outward facing listening port on the NAT.

When both endpoints of a connection send the TCP SYN simultaneously, it results not in two TCP connections but in one. On that TCP connections both endpoints assume to be the dialer (see sim-open spec for details.

Tie breaking on a simultaneous open TCP connection where both endpoints assume to be the dialer can today be done either via:

• DCUtR
• Multistream Select simultaneous open

On hole punched, i.e. coordinated, TCP simultaneous open connections tie breaking via DCUtR can be used. (Tie breaking could as well be done via Multistream Select simultaneous open here, though that is less efficient as it requires more roundtrips and the tie breaking via DCUtR is available for free.)

On non-coordinated, i.e. accidental, TCP simultaneous open connections only tie breaking via Multistream Select simultaneous open can be used. This is problematic as we would like to move away from Multistream Select, replacing it with Protocol Select (see #349). Protocol Select does not support tie breaking, see "TCP Simultaneous Open" section in Protocol Select specification.

Thus far we planned to simply not-support non-coordinated, i.e. accidental, TCP simultaneous open connections. In such cases, as both endpoint assume to be the dialer and as there is no tie breaking mechanism in place, the connection upgrade would simply fail. In #349 we argue that the low probability of such non-coordinated TCP simultaneous open connection is too low for it to be special cased.

Proposal

We could do port reuse on coordinated TCP hole-punched connections only and use a new port (default TCP behaviour) for all other outgoing TCP connections. This would prevent the case of uncoordinated TCP simultaneous open and thus dissolve the need for tie breaking on uncoordinated accidentally simultaneously dialed TCP connections.

There are two problems with this proposal:

1. How does one discover ones outwards-facing listening port? Previously done via non-holepunching outgoing TCP connections reusing the local listening port.
2. How does one keep the port-mapping of ones listening port on the NAT alive. Previously done via non-holepunching outgoing TCP connections reusing the local listening port.

Would it be worth special casing specific outgoing connections (e.g. used for AutoNAT and identify) and have them reuse the listening TCP port, despite not being a hole-punching connection?

Would it be safe to not do port-reuse when one is public, i.e. directly reachable?

---

Credit for the proposal goes to @hamamo.

[mxinden]

//CC @rkuhn as this is related to libp2p/rust-libp2p#2066.

[vyzo]

Why do we want this? It will break NAT type detection.

[rkuhn]

In ipfs-embed (on the unreleased sim-open branch) I worked around TCP simultaneous open in two ways:

• prefer to not bind for outgoing connections, i.e. using random source ports (as proposed above)
• if binding to the listen port is configured, recognise likely upgrade failures for non-coordinated simultaneous open by the nature of the raised error and redial with a randomised delay in that case

Discovery of an incoming peer’s likely listen address proceeds via the Identify protocol: when receiving information on the peer’s local listening ports, compute all combinations of those port numbers with the observed IP addresses, discard duplicates and known bad ones, and validate the remaining ones by attempting to dial them. This can of course not guess other forwarded ports on intermediate firewalls; my focus lay on non-NAT swarms within the same network or adequately configured networks (my software is made for the factory shop floor).

In general, I think it would be great if we could just use networks and foundational as they were designed — I’m still dreaming of IPv6 end-to-end connectivity (as I’m using from home). So as long as there’s a way to make libp2p not bind outgoing connections to a specific address or port, I’m happy.

---

For reference why I care about TCP simultaneous open: Actyx includes higher-level broadcast mechanisms used for peer discovery that rather frequently triggers connections in sufficient propinquity for this to be an issue in practice.

[mxinden]

Why do we want this?

To prevent accidental, i.e. non-coordinated, TCP simultaneous open. See:

We could do port reuse on coordinated TCP hole-punched connections only and use a new port (default TCP behaviour) for all other outgoing TCP connections. This would prevent the case of uncoordinated TCP simultaneous open and thus dissolve the need for tie breaking on uncoordinated accidentally simultaneously dialed TCP connections.

---

It will break NAT type detection.

Yes, this is as well described above:

There are two problems with this proposal:

1. How does one discover ones outwards-facing listening port? Previously done via non-holepunching outgoing TCP connections reusing the local listening port.
2. How does one keep the port-mapping of ones listening port on the NAT alive. Previously done via non-holepunching outgoing TCP connections reusing the local listening port.

There are ways around this, e.g. using SO_REUSEPORT on special connections doing the identify and AutoNAT protocol (see "Proposal" section). In addition, nodes that do know they are public, i.e. not behind a NAT, could disable port-reuse.

---

I don't think the consideration above is ideal, at least not quite yet. Also note that in case we can figure out the caveats and do decide it is worth doing, I will not have the capacity to push this effort anytime soon.

[vyzo]

there is nothing wrong with uncoordinated simopen, and it works with the simopen multistream extension; why do you want to prevent it?

…

On Mon, Jan 31, 2022, 16:11 Max Inden ***@***.***> wrote: Why do we want this? To prevent accidental, i.e. non-coordinated, TCP simultaneous open. See: We could do port reuse on coordinated TCP hole-punched connections only and use a new port (default TCP behaviour) for all other outgoing TCP connections. This would prevent the case of uncoordinated TCP simultaneous open and thus dissolve the need for tie breaking on uncoordinated accidentally simultaneously dialed TCP connections. ------------------------------ It will break NAT type detection. Yes, this is as well described above: There are two problems with this proposal: 1. How does one discover ones outwards-facing listening port? Previously done via non-holepunching outgoing TCP connections reusing the local listening port. 2. How does one keep the port-mapping of ones listening port on the NAT alive. Previously done via non-holepunching outgoing TCP connections reusing the local listening port. There are ways around this, e.g. using SO_REUSEPORT on special connections doing the identify and AutoNAT protocol (see "Proposal" section). In addition, nodes that do know they are public, i.e. not behind a NAT, could disable port-reuse. ------------------------------ I don't think the consideration above is ideal, at least not quite yet. Also note that in case we can figure out the caveats and do decide it is worth doing, I will not have the capacity to push this effort anytime soon. — Reply to this email directly, view it on GitHub <#389 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAI4STERZFZTL7KPANDU33UY2J7PANCNFSM5NBR75QQ> . You are receiving this because you commented.Message ID: ***@***.***>

[hamamo]

Hi, (note this is from memory of Max's talk, I heard about the plans to move to protocol select only then) The reason is to avoid round trips in the move from multistream select to protocol select. With multistream, the probem does not exist, that's true. For protocol select, the dialer role is always assumed to be taken by the peer initiating th connection, which means it both peers get this role in an uncoordinated simultaneous open. My suggestion was to reuse the incoming port for outgoing connections only when you know that you want to punch a hole, which requires previous coordination anyway, so there's no additional effort to find that out when you do a direct connect to some peer address. If you restrict outgoing port reuse to those cases where you know you need it, you don't need to prepare for it in the cases where you don't need it. But I need to make clear that I understand next to nothing about the specific requirements of p2p networking in the presence of NAT - my suggestion was based on experience with traditional TCP usage. Cheers, Hans-Martin 31. Januar 2022 16:08, "vyzo" ***@***.*** ***@***.******@***.***>)> schrieb: there is nothing wrong with uncoordinated simopen, and it works with the simopen multistream extension; why do you want to prevent it?

On Mon, Jan 31, 2022, 16:11 Max Inden ***@***.***> wrote: Why do we want this? To prevent accidental, i.e. non-coordinated, TCP simultaneous open. See: We could do port reuse on coordinated TCP hole-punched connections only and use a new port (default TCP behaviour) for all other outgoing TCP connections. This would prevent the case of uncoordinated TCP simultaneous open and thus dissolve the need for tie breaking on uncoordinated accidentally simultaneously dialed TCP connections. ------------------------------ It will break NAT type detection. Yes, this is as well described above: There are two problems with this proposal: 1. How does one discover ones outwards-facing listening port? Previously done via non-holepunching outgoing TCP connections reusing the local listening port. 2. How does one keep the port-mapping of ones listening port on the NAT alive. Previously done via non-holepunching outgoing TCP connections reusing the local listening port. There are ways around this, e.g. using SO_REUSEPORT on special connections doing the identify and AutoNAT protocol (see "Proposal" section). In addition, nodes that do know they are public, i.e. not behind a NAT, could disable port-reuse. ------------------------------ I don't think the consideration above is ideal, at least not quite yet. Also note that in case we can figure out the caveats and do decide it is worth doing, I will not have the capacity to push this effort anytime soon. — Reply to this email directly, view it on GitHub <#389 (comment) (#389 (comment))>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAI4STERZFZTL7KPANDU33UY2J7PANCNFSM5NBR75QQ (https://github.com/notifications/unsubscribe-auth/AAAI4STERZFZTL7KPANDU33UY2J7PANCNFSM5NBR75QQ)> . You are receiving this because you commented.Message ID: ***@***.***>

— Reply to this email directly, view it on GitHub (#389 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAWHUBIBN2JLCLQCBVIYOPTUY2QV7ANCNFSM5NBR75QQ). Triage notifications on the go with GitHub Mobile for iOS (https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or Android (https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you were mentioned.Message ID: ***@***.***>

[Winterhuman]

This issue is relevant to this discussion as well, reusing TCP ports was found to cause issues with some routers: ipfs/kubo#3320 (comment)

"...do port reuse on coordinated TCP hole-punched connections only" seems like the best solution to avoid the router issue.

[diegomrsantos]

How about something like that?

1. Start with port reuse disabled.
2. Start Autonat. If we are confident enough that we are behind a NAT, enable port reuse.
3. Learn about our public port after having enough outgoing connections with port reuse enabled.
4. Start Hole punching.

Seems we won't have non-coordinated TCP simultaneous open connections in this case as peers can't connect to us without hole punching.

[marten-seemann]

Not only does this break NAT type detection, but also normal address detection (via our STUN-like Identify mechanism). This is very important for AutoNAT v2.

We should not drop a crucial feature to get a slight performance advantage in a rare corner case.