What is the state of art on implementing a p2p message-broadcast network?

Question

I'm aware a quick Google wields tons of results and the literature on this topic is very rich, and that is exactly the problem. Among the universe of possible solutions I'm not sure which specifically is the best, up-to-date choice for my specific needs.

I'm trying to implement a p2p network over the internet with the only and one feature of broadcasting messages to online nodes. In order to connect to the network, you must be able to point to existing IPs. When you do, you discover some peers and keep an active connection with them. You then can send messages to every other node. There is no direct communication, every message sent is received by everyone else. I want this network to be as performant as possible, and work on top of UDP.

What are the names of some state of art algorithms to solve this problem on the shape here specified?

Eterne

If you broadcast, then you hit every host on a LAN without needing to know the individual IP addresses. – Phung

Oh, I mean for an app. Think of a mobile game. I want users to be able to broadcast their moves without involving a central server. – Eterne

Now, I am confused. Do you mean that you want to unicast packets to only the players? A broadcast is sent to every host on the LAN. A unicast is sent to a specific host, and a multicast is sent to a group of hosts that have subscribed to the multicast group. It sounds like you may want to use multicast, then each player host would listen for packets sent to the multicast group(s), and it will not bother the other hosts on the LAN. – Phung

Maybe we're talking about different things due to terminology. I don't know much about networking (thus this question), all I know is the problem I'm trying to solve. I want to create an app which, when opened, will connect to a network of computers running the same app. It will then be able to send messages that every other node will receive - that is what I'm calling "broadcast". I'm talking about a high-level reqs of my app (broadcasting messages, no central server I need to maintain/pay - think of a game), and asking for keywords/resources to start searching for solutions. There's no LAN. – Eterne

And I have given you some of that. If you want to do this as a group, the proper terminology is multicast. You application should join the multicast group, and it can send and receive messages within the multicast group without bothering other hosts that don't participate in the multicast group. Multicast is a selective form of broadcast. A broadcast will interrupt every host must process the messages. In multicast, you only interrupt the hosts subscribed to the multicast group. This avoids the need for a central server, but you could also use a central server if you want. – Phung

Now that you have added the requirement that this work over the Internet, you have removed the possibility of using broadcast or multicast since neither of those work on the Internet. You are going to require a central server for things like registration. – Phung

Answer 1

A broadcast of a single message in an overlay network over the internet is a fairly simple affair. You join the network, build a randomized or structured routing table of neighbors, flood neighbors with the message, they do the same with their neighbors, minus some pruning to avoid forwarding loops.

Complexity arises from additional constraints and requirements which vary from network to network. E.g. trust problems in anonymous settings, latency optimizations, optimizing high-bandwidth streams of many messages (p2p-assisted live video broadcasting), etc.

Urushiol

I'm glad to hear it is fairly simple but there are some complications that aren't obvious to me. You mean a random routing table; how exactly? How each node keeps a list of peers? For that, the initial "entry nodes" must have a list of all nodes, no? If you have any link/keyword that'd be helpful. In any case, thanks for the answer! – Eterne

useful keywords to punch into google scholar (can be combined to some extent): "overlay network", "structured overlay", "small world", "routing table", "neighbor list", "bootstrap"... in general bootstrapping and routing table maintenance are a separate problem from message propagation. you need to build your p2p algorithms piece by piece. – Urushiol

> how exactly? -- Here's a simple Node.js implementation of p2p broadcast network: github.com/will123195/p2p-broadcast – Changeable

Answer 2

If you want to learn about the concepts for setting up decentralized P2P architecture, you could take a look at the Dat Foundation, who provide an ecosystem of modules for P2P data sharing over the internet, such as Hypercore (raw P2P streams) and Hyperdrive (file transfer on top of hypercore). They have some good documentation on technical concepts, challenges and how they tackled them.

Some of the challenges in P2P designs:

• NAT routers and firewalls: overcome by implementing NAT Traversal and UDP hole punching (see: The State of NAT Traversal by ZeroTier)
• Discovering peers: Dat Project uses Gossiping as an efficient way for a swarm of peers to find each other on the network, and a simple (protobuf) wire protocol for communication (see: Hyperdiscovery and Hypercore Protocol).
• P2P data communication / synchronization: Hypercore implements append-only logs on each peer, and Merkle trees for aggregating and deduplicating data chunks from other peers in correct order

Both TCP, UDP, WebRTC and BittorrentDHT are supported by Dat Project.

Please read some of their specification docs for in-depth information on these concepts and the protocol design:

• Dat - Distributed Dataset Synchronization And Versioning (pdf)
• How Dat works
• Hypercore Protocol

Taveda

I just added more info on viable p2p technologies and using Dat Project in a decentralized video app on mobile: A: p2p video/audio chat on android using WebView – Taveda

Hi, interesting read, especially the article by ZeroTier was fun to read (though I guess almost anyone who ever had to design a P2P network doesn't really enjoy working with NATs). Could you update the link to the Dats Github? they seem broken @Arnold Schrijver – Dionne

Hi there @Dionne sorry for being late, but the answer is updated :) – Taveda