Currently, half of our peers are nodes producing archived segments, with each segment containing 256 pieces. Nodes publish all new pieces to several other peers. A few observations can be made: a significant portion (perhaps over half) of our network is expected to be synced; new piece indexes are hashed, producing random keys for publication, which leads to multiple connections for each piece index. By publishing to multiple peers for each key, we generate substantial connection churn during regular operations.
I propose that we stop publishing each node's data to all farmers via Kademlia and, instead, leverage the fact that we have a large amount of similar data circulating within the system. We can divide our network into meshes, wherein each farmer connects to at least ?N=50? nodes, and nodes connect to ?F=50? farmers. These connections should be persistent. Nodes will publish their latest produced segment ID to known farmers (optionally, the publication could also include the available segment range on the node). Upon receiving this event, farmers should convert the segment ID into piece index keys and determine whether to populate their local L2 and Kademlia provider storage. The "get-piece" operation should remain the same, involving obtaining providers via Kademlia and then utilizing a request-response method. Gossipsub could be a potential candidate for the message propagation protocol. After implementing these modifications, we expect to significantly reduce connection congestion and increase the speed of new piece notifications. By publishing a single message instead of 256, we can avoid the need to establish connections that consume both pending and established connection pools. We don't need to disable the current publication for testing purposes. By increasing local connection limits, we can discreetly deploy gossipsub publications with smaller mesh sizes and test the new publication schema.
Open question:
Downsides:
Another idea to get rid of constant reconnections during pieces announcement would be introducing gossipsub subscriptions for piece provider announcements. If we set only one subscription topic the whole network would listen and propagate all messages. However, we could limit the propagation scale by introducing topics based on some ranges of the PeerId associated with the peer. For example, we could introduce ?N=256? different topics and send new piece_index_hash announcements only to the topic with a similar first byte and the receiver should subscribe to only a single topic respectively. This way we also limit necessary connections substantially and also it’s a general solution for both farmer and node.
Open question:
Downsides:
We could decrease get_piece_providers operations number by including hints in get_piece requests or responses for desired or available pieces, and possibly request and get multiple pieces without creating additional connections. A similar idea is to implement the peer_info protocol and exchange available piece indexes between peers and thus request multiple pieces from the known peer. A downside is that the solution will likely work only on small and medium-sized networks (the greater the network the less chance that peers contain more than one required piece), it will also require getting piece index for L2 cache entries (probably, announcing providers as put-record with artificial provider record with piece index along with the piece key).