Network shards (Attnet Revamp + DAS Distribution Columns)

configs/mainnet.yaml

@@ -135,9 +135,12 @@ MESSAGE_DOMAIN_VALID_SNAPPY: 0x01000000
 SUBNETS_PER_NODE: 2
 # 2**8 (= 64)
 ATTESTATION_SUBNET_COUNT: 64
-ATTESTATION_SUBNET_EXTRA_BITS: 0
-# ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS
-ATTESTATION_SUBNET_PREFIX_BITS: 6
+# The granularity of the network.
+NETWORK_TOPIC_COUNT: 64
+NETWORK_TOPIC_EXTRA_BITS: 0
+# ceillog2(NETWORK_TOPIC_COUNT) + NETWORK_TOPIC_EXTRA_BITS
+NETWORK_TOPIC_PREFIX_BITS: 6
+NETWORK_TOPIC_SHUFFLING_PREFIX_BITS: 3
 
 # Deneb
 # `2**7` (=128)

configs/minimal.yaml

@@ -136,9 +136,11 @@ MESSAGE_DOMAIN_VALID_SNAPPY: 0x01000000
 SUBNETS_PER_NODE: 2
 # 2**8 (= 64)
 ATTESTATION_SUBNET_COUNT: 64
-ATTESTATION_SUBNET_EXTRA_BITS: 0
-# ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS
-ATTESTATION_SUBNET_PREFIX_BITS: 6
+NETWORK_TOPIC_COUNT: 64
+NETWORK_TOPIC_EXTRA_BITS: 0
+# ceillog2(NETWORK_TOPIC_COUNT) + NETWORK_TOPIC_EXTRA_BITS
+NETWORK_TOPIC_PREFIX_BITS: 6
+NETWORK_TOPIC_SHUFFLING_PREFIX_BITS: 3
 
 # Deneb
 # `2**7` (=128)

specs/phase0/p2p-interface.md

@@ -191,7 +191,7 @@ This section outlines configurations that are used in this spec.
 |---|---|---|
 | `GOSSIP_MAX_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed gossip messages. |
 | `MAX_REQUEST_BLOCKS` | `2**10` (= 1024) | Maximum number of blocks in a single request |
-| `EPOCHS_PER_SUBNET_SUBSCRIPTION` | `2**8` (= 256) | Number of epochs on a subnet subscription (~27 hours) |
+| `EPOCHS_PER_SUBNET_SUBSCRIPTION` | `2**8` (= 256) | Number of epochs on an attestation subscription (~27 hours) |
 | `MIN_EPOCHS_FOR_BLOCK_REQUESTS` | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) | The minimum epoch range over which a node must serve blocks |
 | `MAX_CHUNK_SIZE` | `10 * 2**20` (=10485760, 10 MiB) | The maximum allowed size of uncompressed req/resp chunked responses. |
 | `TTFB_TIMEOUT` | `5` | The maximum duration in **seconds** to wait for first byte of request response (time-to-first-byte). |
@@ -200,10 +200,12 @@ This section outlines configurations that are used in this spec.
 | `MAXIMUM_GOSSIP_CLOCK_DISPARITY` | `500` | The maximum **milliseconds** of clock disparity assumed between honest nodes. |
 | `MESSAGE_DOMAIN_INVALID_SNAPPY` | `DomainType('0x00000000')` | 4-byte domain for gossip message-id isolation of *invalid* snappy messages |
 | `MESSAGE_DOMAIN_VALID_SNAPPY`  | `DomainType('0x01000000')` | 4-byte domain for gossip message-id isolation of *valid* snappy messages |
-| `SUBNETS_PER_NODE` | `2` | The number of long-lived subnets a beacon node should be subscribed to. |
 | `ATTESTATION_SUBNET_COUNT` | `2**6` (= 64) | The number of attestation subnets used in the gossipsub protocol. |
-| `ATTESTATION_SUBNET_EXTRA_BITS` | `0` | The number of extra bits of a NodeId to use when mapping to a subscribed subnet |
-| `ATTESTATION_SUBNET_PREFIX_BITS` | `int(ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS)` | |
+| `NETWORK_TOPIC_COUNT` | `ATTESTATION_SUBNET_COUNT` | The number of topics to generate for a node-id. Used in the gossipsub-protocol. |
+| `NETWORK_TOPIC_EXTRA_BITS` | `0` | The number of extra bits of a NodeId to use when mapping to a gossipsub topic |
+| `NETWORK_TOPIC_PREFIX_BITS` | `int(ceillog2(NETWORK_TOPIC_COUNT) + NETWORK_TOPIC_EXTRA_BITS)`. |
+| `NETWORK_TOPIC_SHUFFLING_PREFIX_BITS` | `3` | The number of bits used to shuffle nodes to a new topic within `EPOCHS_PER_SUBNET_SUBSCRIPTION`. |
+| `SUBNETS_PER_NODE` | `2` | The number of long-lived attestation subnets a beacon node should be subscribed to. |
 
 ### MetaData
 
@@ -1010,34 +1012,100 @@ Clients MAY connect to peers with the same `fork_digest` but a different `next_f
 Unless `ENRForkID` is manually updated to matching prior to the earlier `next_fork_epoch` of the two clients,
 these connecting clients will be unable to successfully interact starting at the earlier `next_fork_epoch`.
 
-### Attestation subnet subscription
+### Topic Backbone
 
-Because Phase 0 does not have shards and thus does not have Shard Committees, there is no stable backbone to the attestation subnets (`beacon_attestation_{subnet_id}`). To provide this stability, each beacon node should:
+In order for gossipsub to function, there must be a stable set of peers for
+each topic that remain subscribed for a long period of time (order 1
+day). This allows nodes to search for and maintain a selection of these
+long-lived peers in order to publish/receive messages on the topic. As some
+topics are transient for most nodes (i.e attestation subnets, DAS-related
+columns) it is necessary that we enforce each node on the network to facilitate the
+support of these topics by long-lived subscribing to them (and thereby validating and forwarding messages).
 
-* Remain subscribed to `SUBNETS_PER_NODE` for `EPOCHS_PER_SUBNET_SUBSCRIPTION` epochs.
-* Maintain advertisement of the selected subnets in their node's ENR `attnets` entry by setting the selected `subnet_id` bits to `True` (e.g. `ENR["attnets"][subnet_id] = True`) for all persistent attestation subnets.
-* Select these subnets based on their node-id as specified by the following `compute_subscribed_subnets(node_id, epoch)` function.
+To this end we define a mapping that relates a peer's node-id into one or many
+gossipsub topics for a given epoch. Every node MUST subscribe to their associated
+topic at any given epoch.
+
+The mapping is based on the first few
+`NETWORK_TOPIC_PREFIX_BITS` of a peer's node-id. Client implementors may wish
+to keep a uniform distribution of peers with these prefix-bits in order to
+maintain (on average) a uniform distribution of peers on the transient
+gossipsub topics (i.e attestation subnets).
+
+The mapping that links a node-id to a topic id is:
 
 ```python
-def compute_subscribed_subnet(node_id: NodeID, epoch: Epoch, index: int) -> SubnetID:
-    node_id_prefix = node_id >> (NODE_ID_BITS - ATTESTATION_SUBNET_PREFIX_BITS)
-    node_offset = node_id % EPOCHS_PER_SUBNET_SUBSCRIPTION
-    permutation_seed = hash(uint_to_bytes(uint64((epoch + node_offset) // EPOCHS_PER_SUBNET_SUBSCRIPTION)))
+def compute_network_topic(node_id: NodeID, epoch: Epoch, rotation_period: uint16) -> TopicID:
+    # NOTE: The rotation period is measured in epochs
+    # The main prefix bits to determine a network topic
+    topic_prefix = node_id >> (NODE_ID_BITS - NETWORK_TOPIC_PREFIX_BITS)
+    # Used to extract the total prefix bits (prefix + shuffling_bits)
+    shuffling_bit_size = (
+        NODE_ID_BITS
+        - NETWORK_TOPIC_PREFIX_BITS
+        - NETWORK_TOPIC_SHUFFLING_PREFIX_BITS
+    )
+    # The NETWORK_TOPIC_SHUFFLING_PREFIX_BITS that trail topic_prefix.
+    # These are used to stagger the rotation of network topics so that all
+    # nodes do not rotate from topics all at once.
+    # The larger the NETWORK_TOPIC_SHUFFLING_PREFIX_BITS the more granular the
+    # nodes will transition from one topic to another throughout a period.
+    shuffling_bits = (node_id >> shuffling_bit_size) % (1 << NETWORK_TOPIC_SHUFFLING_PREFIX_BITS)
+    # Calculates a multiplier that scales the shuffling prefix (assumed smaller
+    # than the rotation period) to be uniform throughout the entire rotation
+    # period. Can also be calculated as:
+    # rotation_period // 1 >> NETWORK_TOPIC_SHUFFLING_PREFIX_BITS
+    shuffling_multiplier = rotation_period >> NETWORK_TOPIC_SHUFFLING_PREFIX_BITS
+    # The epoch at which this node will rotate to a new topic
+    # This is distributed uniformly throughout the rotation period with a
+    # granularity based on the size of NETWORK_TOPIC_SHUFFLING_PREFIX_BITS
+    epoch_transition = (
+        (topic_prefix + (shuffling_bits * shuffling_multiplier)) % rotation_period
+    )
+    # A seed which changes every rotation period
+    # This enforces the rotation period and is staggered for each prefix so
+    # that nodes do not rotate from topics all at once
+    permutation_seed = hash(uint_to_bytes(uint64((epoch + epoch_transition) // rotation_period)))
+    # The resulting value that ultimately defines the network topic.
     permutated_prefix = compute_shuffled_index(
-        node_id_prefix,
-        1 << ATTESTATION_SUBNET_PREFIX_BITS,
+        topic_prefix,
+        1 << NETWORK_TOPIC_PREFIX_BITS,
         permutation_seed,
     )
-    return SubnetID((permutated_prefix + index) % ATTESTATION_SUBNET_COUNT)
+    return TopicID(permutated_prefix % NETWORK_TOPIC_COUNT)
 ```
 
+The `compute_network_topic` function is designed with the following
+desirable properties:
+* It uses only the first set of bits (defined by NETWORK_TOPIC_PREFIX_BITS and
+    NETWORK_TOPIC_SHUFFLING_PREFIX_BITS) of the
+  node-id. This allows for efficient discovery searches, by allowing clients to
+  search for specific nodes linked to a gossipsub topic based on the kademilia XOR
+  metric.
+* Nodes will maintain a topic for the `rotation_period` before
+  rotating to a new topic.
+* The rotation is staggered uniformly throughout the rotation period per topic.
+* No individual topic will suddenly rotate to another, rather a subset of nodes
+  per topic transition gradually throughout the rotation period.
+* The function is feasibly reversible. Prefixes can be calculated for desired
+  network topics for any given epoch, allowing nodes to search for these
+  prefixes.
+
+### Attestation Subnets
+
+The backbone structure for attestation subnets can be calculated from the
+network topic function above via the following:
+
 ```python
-def compute_subscribed_subnets(node_id: NodeID, epoch: Epoch) -> Sequence[SubnetID]:
-    return [compute_subscribed_subnet(node_id, epoch, index) for index in range(SUBNETS_PER_NODE)]
+def compute_subscribed_topic(node_id: NodeID, epoch: Epoch, rotation_period: EPOCHS_PER_SUBNET_SUBSCRIPTION) -> Sequence[SubnetID]:
+    network_topic_id = compute_network_topic(node_id, epoch, rotation_period)
+    return [SubnetId(network_topic_id + index) % ATTESTATION_SUBNET_COUNT for index in range(SUBNETS_PER_NODE)]
 ```
 
 *Note*: When preparing for a hard fork, a node must select and subscribe to subnets of the future fork versioning at least `EPOCHS_PER_SUBNET_SUBSCRIPTION` epochs in advance of the fork. These new subnets for the fork are maintained in addition to those for the current fork until the fork occurs. After the fork occurs, let the subnets from the previous fork reach the end of life with no replacements.
 
+*Note*: A node MUST subscribe to the subnets defined via `compute_subscribed_subnets` function, but MAY subscribe to more. If a node subscribes to extra subnets they SHOULD update their metadata and ENR bitfields to reflect the extra long-lived subscriptions.
+
 ## Design decision rationale
 
 ### Transport