Partial Messages Extension #631

MarcoPolo · 2025-07-15T01:31:43Z

Implements libp2p/specs#685

Commits are meaningful. Best to review commit by commit.

extensions.go

partialmessages/partialmsgs.go

MarcoPolo · 2025-09-24T17:53:49Z

thanks for the review @ppopth, friendly reminder you can see the latest changes with git range-diff master 7fc7eb495bd6 578e12f4afcd

sukunrt

LGTM 🎉

The bitmap is probably the common case, so add library support for it.

This allows a peer's local partsMetadata to merge with their remote partsMetadata. As long as the underlying PartsMetadata type supports CRDT-like addition, we can avoid sending the peer duplicates. Take the following example (credit Sukun): > Say: We have parts 1, 2, 5, 6 > > 1. Peer sends a message saying I have part 1, 2 > 2. We send parts 5, 6 and update the peerstate in PublishMessage to (1, 2, 5, 6) > 3. Concurrently with 2, we receive message from the peer saying I now have parts (1, 2, 3, 4) With this change we can now correctly store the fact that this peer should have parts 1,2,3,4,5,6. And in a way that does not assume a specific representation of the PartsMetadata.

and fix a bug where we would skip sending them full messages if they requested partial messages and we disabled partial messages.

if they requested partial messages on a given topic.

When a topic join happens concurrently with adding a new peer, we may send the topic join announcement before the hello packet which contains our extensions. This is a protocol violation. This change starts sending rpcs on new stream only after we've enqued a hello packet. We use UrgentPush to ensure that it's the first one sent.

instead of requiring a message object

We need some way to give peer feedback to gossipsub if the application is using partial messages. This adds a new PeerFeedback method that lets an application modify the peer score.

This lets us distinguish who can support a partial message request versus who requests partial message on a per topic basis.

Creates a mixed network where only one peer request partial, and the rest either support or don't

A peer initiated group state is a state that is initiatlized due to receiving a partial message RPC for which we have not (yet) published to. It's useful to have this state so we can remember what parts a peer has if we do publish to that group. Previously we limited the total number of peer initiated group states in a topic. This was agnostic to how many groups a single peer initiated, which meant that a single peer could exhaust the whole limit. This change tracks how many group states each peer has initialized and limits each peer to a configurable number of these states. The default total and per peer limits of 255 8 are likely good enough for most use cases, but the options exist for users to change this if they need to.

changes partialMessageStatePerTopicGroup to partialMessageStatePerGroupPerTopic to better reflect its usage.

MarcoPolo changed the base branch from master to marco/extensions July 17, 2025 21:01

MarcoPolo force-pushed the marco/extensions branch 2 times, most recently from 222428a to 5299023 Compare August 29, 2025 19:56

MarcoPolo force-pushed the marco/partial-messages-2 branch from 6a50510 to 8162df6 Compare August 29, 2025 20:44

MarcoPolo force-pushed the marco/partial-messages-2 branch from 8162df6 to 74b015e Compare September 10, 2025 21:21

MarcoPolo force-pushed the marco/extensions branch 3 times, most recently from 7402459 to 939defb Compare September 15, 2025 18:43

MarcoPolo force-pushed the marco/partial-messages-2 branch 2 times, most recently from a1d0543 to c5773aa Compare September 15, 2025 19:33

MarcoPolo force-pushed the marco/extensions branch 2 times, most recently from 4e4faae to 857c8cc Compare September 15, 2025 19:37

Base automatically changed from marco/extensions to master September 15, 2025 20:32

MarcoPolo force-pushed the marco/partial-messages-2 branch 2 times, most recently from a76890d to 96339bb Compare September 15, 2025 22:32

MarcoPolo marked this pull request as ready for review September 15, 2025 22:32

MarcoPolo requested a review from sukunrt September 15, 2025 22:32

jxs mentioned this pull request Sep 16, 2025

feat: implement gossipsub partial messages extension sigp/rust-libp2p#577

Open

ppopth reviewed Sep 23, 2025

View reviewed changes

extensions.go Outdated Show resolved Hide resolved

ppopth reviewed Sep 23, 2025

View reviewed changes

partialmessages/partialmsgs.go Show resolved Hide resolved

MarcoPolo force-pushed the marco/partial-messages-2 branch 3 times, most recently from 818f40d to 7fc7eb4 Compare September 23, 2025 21:05