AMQP/RabbitMQ - Process messages sequentially

Question

I have one direct exchange. There is also one queue, bound to this exchange.

I have two consumers for that queue. The consumers are manually ack'ing the messages once they've done the corresponding processing.

The messages are logically ordered/sorted, and should be processed in that order. Is it possible to enforce that all messages are received and processed sequentially accross consumer A and consumer B? In other words, prevent A and B from processing messages at the same time.

Note: the consumers are not sharing the same connection and/or channel. This means I cannot use <channel>.basicQoS(1);.

Rationale of this question: both consumers are identicall. If one goes down, the other queue starts processing messages and everything keeps working without any required intervention.

Kerbstone

How about Single Active Consumer rabbitmq.com/consumers.html#single-active-consumer ? Multiple consumers bind but all messages go only to the very first one. If it dies, then messages are dispatched to the second one. You get your redundancy and processing order is preserved. – Walkover

Single Active Consumer does look cleaner than exclusive binds, but does require RabbitMQ 3.8 – Cressy

Answer 1

Usually the point of a MQ system is to distribute workload. Of course, there are some situations where processing of message N depends on result of processing the message N-1, or even the N-1 message itself.

If A and B can't process messages at the same time, then why not just have A or just B? As I see it, you are not saving anything with having 2 consumers in a way that one can work only when the other one is not...

In your case, it would be best to have one consumer but to actually do the parallelisation (not a word really) on the processing part.

Just to add that RMQ is distributing messages evenly to all consumers (in round-robin fashion) regardless on any criteria. Of course this is when prefetch is set to 1, which by default it is. More info on that here, look for "fair dispatch".

Stucco

Thank you for your insights. Coming back to your question then why not just have A or just B: You are correct in your understanding that A and B should not process messages at the same time. So it is indeed A or B. However, I thought it would be useful to have both A and B running: if A (or B) crashes, the system can continue without any (manual) intervention. I understand from you that my approach is not really possible. But the question then is: how can I facilitate a correct fail-over from A (or B) to B (or A)? – Kerbstone

You are welcome. Easier would be to have a monitoring agent (watchdog) for A which would restart it if it crashes, then to make a failover from A to B in whatever way. The messages don't get lost, they'll stay in the queue and will be delivered when the consumer is up again. – Stucco

OK, got it. Does RabbitMQ provide advanced fail-over functionalities? I know there is the management plugin and a REST API, but that's just monitoring (afaik). Are there any plugins/libraries/tools (RabbitMQ or 3rd party) for automatic fail-over which you would suggest to start with? – Kerbstone

I don't know for fail-over functionalities, this is the client side, so I don't think that the server should be taking care of this. Could be that there are some plugins, but I've never needed something like that so haven't ever looked it up. RabbitMQ does have a heartbeat (actually a AMQP feature) check - maybe this can be of some use to you. But a watchdog should be quite easy to write on any OS - one exaple: pull process list every n seconds see if consumer is there if not start it. – Stucco

OK, thank you. I will look into the heartbeat feature to start with. I'll leave this question open for a few days, in case somebody else has a more concrete answer. Thanks again. – Kerbstone

Answer 2

One approach to handling failover in a case where you want redundant consumers but need to process messages in a specific order is to use the exclusive consumer option when setting up the bind to the queue, and to have two consumers who keep trying to bind even when they can't get the exclusive lock.

The process is something like this:

1. Consumer A starts first and binds to the queue as an exclusive consumer. Consumer A begins processing messages from the queue.
2. Consumer B starts next and attempts to bind to the queue as an exclusive consumer, but is rejected because the queue already has an exclusive consumer.
3. On a recurring basis, consumer B attempts to get an exclusive bind on the queue but is rejected.
4. Process hosting consumer A crashes.
5. Consumer B attempts to bind to the queue as an exclusive consumer, and succeeds this time. Consumer B starts processing messages from the queue.
6. Consumer A is brought back online, and attempts an exclusive bind, but is rejected now.
7. Consumer B continues to process messages in FIFO order.

While this approach doesn't provide load sharing, it does provide redundancy.

Cressy

Answer 3

Even though this is already answered. May be this can help others. RabbitMQ has a feature known as Single Active Consumer, which matches your case.

We can have N consumers attached to a Queue but only 1 (one) of them will be actively consuming messages from the Queue. Fail-over happens only when active consumer fails.

Kindly take a look at the link https://www.rabbitmq.com/consumers.html#single-active-consumer

Thank you

Lordling