Message Bus vs. Multicast

Question

I am working on an application which consists of several modules and requires them to share information with each other. Example: publish/subscribe scenario where a module publishes some information (e.g. a state variable) and modules interested in the particular information get it. Or a request/reply scenario where the interested module explicitly asks about the information and got answered.

I have been looking into different message bus implementations, namely D-bus, ØMQ, RabbitMQ and QPID (the latter two are based on AMQP). However then someone pointed out that instead of trying some complex and heavy message bus implementation, why don't I simply use multicast to solve the problem.

Lacking the experience to see if multicast can really solve my problem and to understand the pros and cons of the two solutions, I would kindly request experts to help me out. Would very much appreciate.

Mayberry

What message rate, size, persistence, and reliability requirements do you have? – Foulard

@ScottA. There are around 30 modules which need to share information. Message rate is not extremely high. It is mostly state information which is needed to be communicated to other modules e.g. when there is a change in state, or when a module requests value of a state variable. Reliability is not a big concern in the publish/subscribe scenario but in request/reply scenario. – Mayberry

How much information is being kept in the shared state? For Zookeeper, the use case is high reads, low writes, reasonably small amounts of data. It's not a messaging system but it can be abused as one. – Foulard

Answer 1

Having experience with both message buses and multicast in a large, high-volume production environment, and having talked with several experienced network engineers about the problems, I can say that you should avoid multicast like the plague unless you are broadcasting to very high numbers of nodes (hundreds).

If you're going to use multicast you have to understand that it is an unreliable protocol. Messages can go missing, they can be duplicated, and so on. You'll need to spend a lot of time getting the reliability protocol (retries, duplicate detection, resends) on top of multicast correct for it to be useful. There is a good anecdote about an army test of multicast commanded robotic tanks that I'm trying to find a reference for... basically when you are sending "turn right 90 degrees, turn right 90 degrees, fire" to a line of tanks and some of them receive only 1 turn right message and others get 3, it's a recipe for mayhem.

Depending on the sort of information you need to share there are several options.

If they are sharing configuration information, look at something like Zookeeper. It's reliable, lightweight, and simple to use. The most recent values of the shared state are always available and persisted. With a message bus you still need to have a resend protocol in case your module misses the last configuration message by being down.

With respect to a message bus, they can be complex. However, I would not put ZeroMQ in that category necessarily. It can emulate a message bus, but it's more of a point to point mechanism. I have not used it in production but the research and prototyping I've done with it was very favorable.

Another option might be a distributed data grid like Oracle Coherence, GridGain, GigaSpaces, and so on. Again, this is another application to install and maintain so your complexity goes up, but there are many uses for a data grid.

Another MQ option is HornetMQ. I haven't used it (we use two commercial MQs in-house, both Sonic and MQ Series), but I've seen some favorable comparisons.

D-Bus seems to be optimized for communication on a single machine, and the FAQ recommends looking elsewhere if you are doing peer-to-peer, clustering, or other similar things. Caveat: I have never used D-Bus so I'm basically regurgitating information I just read.

Foulard

Thanks a lot for your helpful reply. It is mostly state information the modules have to share. I will definitely have a look into Zookeeper. Unfortunately the other solutions you mentioned cannot be considered as software size is an issue, as it has to run on hardware with limited memory and CPU resources. – Mayberry

@JahanzebFarooq Zookeeper will probably have a similar footprint to something like HornetMQ. Again, it depends on what you're trading around. State information definitely sounds like a Zookeeper use case. ZeroMQ is also a good choice if you are looking for something lightweight, but it requires you adding a resend protocol on top. How many machines need to be communicating? – Foulard

In a typical scenario it is all on the same host. But in a more advanced scenario where a certain modules need to communicate between different hosts, number of hosts may range from 10 to 50. – Mayberry

Answer 2

Are you worried about packets/messages being dropped or lost? Message buses can handle or mitigate those concerns while multicast won't by default.

Chuipek

This seems like it should have been a comment. Gather any clarification you need via comments before you answer a question. – Sumner