I want to build a decentralized, reddit-like system using P2P. What existing p2p library should I base it on?

Question

I want to build a decentralized, reddit-like system using P2P. Basically, I want to retain the basic capabilities of reddit, but make it decentralized, to make it more robust and immune to censorship. This will also allow people to develop different clients to match the way they want to browse it.

Could you recommend good p2p libraries to base my work on? They should be open-source, cross-platform, robust and easy to use. I don't care much about the language, I can adapt.

Vaivode

You should check out Distributed Hash Tables, there exist several implementations of production quality. Possibly you won't need much more...Also: I like your idea very much ;) – Giliane

Thanks! I like the idea too, but it seems like a very hard task... If you're interested in helping me out, you can subscribe to reddit/r/decentralized. I'll try to recruit more people when the idea gets clearer. – Vaivode

I warn you that historically distributed solutions have gained very little traction, and also the market is saturated with reddit clones. So, you will have to try damn hard to promote it and sign up enough initial users to maintain momentum. Good luck! – Cogency

I should add that your users will probably not be using any kind of roman script, so whatever you do, make sure unicode works properly throughout; maybe also provide a feature for people to filter what they see and search by the characterset in use? – Cogency

Why do you think decentralized solution have had no traction? Is there something intrisically wrong about them, or are they simply harder to build than centralized solutions? – Vaivode

I found this question researching the same idea. The biggest problem you'll face is people trying to game the system. Being centralized, Reddit can easily ensure one user gets one vote per comment and post, and that moderators have control over subreddits. A de-centralized version, by definition, would have no one with the authority. That's not saying it's impossible, but it's an extra problem you'll have to consider. – Bdellium

@Cerin: I agree! That's why I want to base my link selection system on personalized recommendations rather than democracy. The system will select links based on learning what the user likes and which users he trusts, not on vote counts. – Vaivode

I'm precisely working on this part of the system at the moment, developing it as a layer over reddit, with the idea of replacing reddit by a p2p system when the top layer is ready. – Vaivode

@static_rtti, Interesting idea. Focus on online learning algorithms. Vowpal Wabbit might be a good place to start. – Bdellium

One possible solution to limit vote spam is to make it computationally expensive to vote. If it takes 1 second to make a vote instead of 1 millisecond that's 1000x less spam votes to worry about ;) – Electronarcosis

@Martin: that's not a bad idea. How woud you implement it? – Vaivode

HashCash en.wikipedia.org/wiki/Hashcash is exactly such a system for email, I'd implement it exactly like that. – Electronarcosis

this is a good read on decentralized social networks 8bitmen.com/… – Publicness

Answer 1

Disclaimer: warning, self-promotion here !!!

Have you considered JXTA's latest release? It is probably sufficient for what you want to do. Else, we are working on a new P2P framework called Chaupal, but it is not operational yet.

EDIT

There is also what I call the quick-and-dirty UDP solution (which is not so dirty after all, I should call it minimal).

1. Just implement one server with a public address and start listening for UPD.
2. Peers located behind NATs contact the server which can read how their private IP address has been translated into a public IP address from the received datagrams.
3. You send that information back to the peer who can forward it to other peers. The server can also help exchanging this information between peers.
4. Then peers can communicate directly (one-to-one) by sending datagrams to these translated addresses.

Simple, easy to implement, but does not cover for lost datagrams, replays, out-of-order etc... (i.e., the typical stuff that TCP solves for you at the IP stack level).

Tilda

Thanks for your answer! I won't mark it as accepted immediately, because I want to offer a bounty for the question to get it as much attention as possible. On a side note, do you have any comments / suggestions as to how one might implement a decentralized, p2p-based, reddit-like system? – Vaivode

What you want to achieve is similar to FreeNet, so I would take a look at it for inspiration. – Outshine

No, what I want to do is very different from freenet. Freenet is basically an anonymous version of the web. I don't care much about anonymity. My goal is to allow people to exchange links and comments on these links, in a decentralized fashion. – Vaivode

What I meant is that when it comes to the underlying architecture (not what the user sees or the functionalities offered by your application), you should separate the communication level from the application level, which is what Freenet achieves. You can skip the anonymous part. – Outshine

Used JXTA myself and can recommend it :) – Megaton

Answer 2

I haven't had a chance to use it, but Telehash seems to have been made for this kind of application. Peer2Peer apps have a particular challenge dealing with the restrictions of firewalls... since Telehash is based on UDP, it's well suited for hole-punching through firewalls.

EDIT for static_rtti's comment:

If code velocity is a requirement libjingle has a lot of effort going into it, but is primarily geared towards XMPP. You can port off parts of the ICE code and at least get hole-punching. See the libjingle architecture overview for details about their implementation.

Polypary

Seems nice, thanks for answering! I'll investigate a bit more when I get the time. – Vaivode

@static_rtti, I responded above. – Polypary

Answer 3

Check out CouchDB. It's a decentralized web app platform that uses an HTTP API. People have used it to create "CouchApps" which are decentralized CouchDB-based applications that can spread in a viral nature to other CouchDB servers. All you need to know to write CouchApps is Javascript and learn the CouchDB API. You can read this free online book to learn more: http://guide.couchdb.org

The secret sauce to CouchDB is a Master-to-Master replication protocol that lets information spread like a virus. When I attended the first CouchConf, they demonstrated how efficient this is by throwing a "Couch Party" (which is where you have a room full of people replicating to the person next to them simulating an ad hoc network).

Also, all the code that makes a CouchApp work is public by default in special entities known as Design Documents.

P.S. I've been thinking of doing a similar project, but I don't have a lot of time to devote to it at the moment. GOD SPEED MY BOY!

Nealon

Thanks, I will definitely look into it. – Vaivode

No problem! If you have any questions, feel free to ask. I really think this is the ideal solution for a decentralized website. Also, there are mobile versions of CouchDB that can be embedded into Android & iOS apps so that you can actually take websites with you where ever you go. One last thing, I highly recommend this Google Tech Talk on "Ground Computing" with CouchDB: googlecode.blogspot.com/2009/09/… – Nealon