DDS vs AMQP vs ZeroMQ [closed]

Question

I wanted a feedback on whether my evaluations and concerns are correct.

I have been reseaching the three, Data Distribution Service, AMQP and ZeroMQ for sometime now for building a data transport layer in a datacenter. All the three look promising, yet i encountered some blocking issues in few.

To give a context, my requirements are:

1. Scale upto 500+ physical nodes, 1000+ publishers and subscribers.
2. Support durable delivery of messages to take care of failing subscribers.
3. The aggregated throughput should be north of 1 Million messages / sec

Problems with AMQP:

1. The Broker architecture seems to be a bottleneck and central point of failure in the whole deployment setup. I can complicate my deployment by putting a federation and cluster to improve performance and availability of pending messages, but they still do not seem to be failproof.
2. The performance for durable queues seems to be very less. My sample application could only clock 6-7K messages / core / queue/ application.

Problems with ZeroMQ:

1. The Documentation seems to be a little wanting in depth.
2. The behavior of the system for pending messages seems to invite trouble in PUB/SUB model of communication. Please refer to : How zeromq handles slow consumers with PUB/SUB mode

OpenSplice DDS: I didnt find anything lacking in the DDS protocol except for adoption in the industry. Will like to know a first hand review on this product in terms of stability, performance or limitations.

Caballero

I will address your question later in depth but for now want to suggest that you add the tag "data-distribution-service" instead of "dds". The latter seems to have been hijacked by graphics programming. – Jolda

Thanks Holger, have updated the tags. – Caballero

Just to make the linked ZeroMQ discussion more subtle: For PUB/SUB, there is indeed a pre-settable per-subscriber limit on message queue size, over which messages are dropped. But there is also the ZMQ_SWAP option, with which you can enable disk swapping of messages. Quoting [1] "However, there is a way out. ØMQ provides something called a >>swap<<, which is a disk file that holds messages we can't store to the queue" [1] zguide.zeromq.org/chapter:all – Rubiaceous

@Caballero Regarding the slow consumer handling in ZeroMQ: Recent versions allow you to configure ZMQ_HWM: high water mark and ZMQ_SWAP: disk offload size. Check api.zeromq.org/master:zmq-getsockopt for details. – Escobar

Answer 1

I am surprised about your concerns on OpenSplice DDS adoption. OpenSplice DDS is today deployed on over several mission and business critical systems such as Naval Combat Management Systems, Military Vehicles, Air Traffic Control and Management, Metro, to High Frequency Auto-Trading. Just to give you another bit of info that should give you reassure you w.r.t. the adoption of the technology, the OMG DDS standard (the standard implemented by OpenSplice DDS) has been recommended by EUROCAE to exchange flight data plans across centers pan-european.

Let me know if you have any further questions either on the adoption or the technology.

-AC

Stint

Yea and i have been looking at the code of OpenSplice and it´s horrible. Also the API is a big mess and it is way to complicated to setup. OpenSplice was written by people that dont understand anything about software design principles. And it might be adopted by lots of companies because it works, but it is sure not the way to go if you want to evolve into the future. – Mouthpart

Answer 2

Take a look at this page. Lot of industries and companies are using DDS today.

Irvine