Distributing state across many machines

Question

I'm trying to write up a tool that requires knowledge of the state of other machines in a cluster (local LAN). This is for a network failover/high availability system similar to VRRP and corosync/openais, but I wish to contain more information (such as near real-time speed/performance characteristics) so devices can make more intelligent choices. This means using a protocol more complicated than a predetermine weight-based mechanism: by allowing all clustered machines to see the state of each other, they can communally agree on which is the most suitable to be the master device.

From my searches, I haven't found any (C, C++ or JavaME) libraries that offer a distributed state mechanism. Ideally, I'm looking for something that broadcasts/multicasts each individual machines state periodically so participating machines can build up a global state table and all can see who the master should be. State in this case is arbitrary key/value pairs.

I'd rather not re-invent any wheels so am curious to know if anyone here can point me in the right direction?

Sik

Have you checked Boost MPI. I think it is for distributed computing. But not sure if it will help you or not. – Saiz

Looks like a classic case of Brewer's CAP (Consistency, Availability and Partition) theorem. Reading up on that should provide you with a better understanding of the real problems. – Valeric

@Valeric - Not sure what you mean? I understand that it is impossible to guarantee each node has 100% knowledge of the state of all other participants, as outages will happen. Split horizons will occur etc. What I'm gunning for is a good, generic purpose framework, to accumulate and share state. If you look at the likes of OSPF and other various routing protocols you'll see that this technique is well used in specalised areas. – Sik

If you're talking about the state, you're already assuimg that the system is Consistent. You explicitly mention (high) Availability. Given that, I'd thought I'd point out the CAP theorem in a comment. It's not a framework of course, which is why I didn't put it in an answer. – Valeric

Thanks, makes sense. Consistency is the component of CAP that I definitely don't need: Eventual consistency and slightly stale state is acceptable in this case. – Sik

Answer 1

I'm not sure if there is any application for your purpose or not. But I know that you can write a simple program with MPI library and broadcast any information that you want.

all client's can send their state to root node, and the root node then broadcast the message.

functions that you need for this are:

MPI_Bcast
MPI_Send
MPI_Recv

there is lots of tutorial on C++/MPI on net, just google it!

Sousa

Answer 2

If I were you I'd investigate memcached (memcached.org) or one of the nosql variants.

Roanna

+1 It is a clean, simple solution, which I find appealing. On the other hand, all of the infrastructure gizmos from other systems have to be hand-built. – Noel

Answer 3

It sounds like Apache ZooKeeper might be a good match. It's distributed, hierarchical key-value store. To quote their Overview page:

ZooKeeper was designed to store coordination data: status information, configuration, location information, etc.

Here's an example of a simple Leader Election recipie, although it would require adaptation to determine a leader by some weighted criterion.

Accessory