RabbitMQ consumer overload

Question

I`ve been reading about the principles of AMQP messaging confirms. (https://www.rabbitmq.com/confirms.html). Really helpful and wel written article but one particular thing about consumer aknowledgments is really confusing, here is the quote:

Another things that's important to consider when using automatic acknowledgement mode is that of consumer overload.

Consumer overload? Message queue is processed and kept in RAM by broker (if I understand it correctly). What overload is it about? Does consumer have some kind of second queue? Another part of that article is even more confusing:

Consumers therefore can be overwhelmed by the rate of deliveries, potentially accumulating a backlog in memory and running out of heap or getting their process terminated by the OS.

What backlog? How is this all works together? What part of job is done by consumer (besides consuming message and processing it of course)? I thought that broker is keeping queues alive and forwards the messages but now I am reading about some mysterious backlogs and consumer overloads. This is really confusing, can someone explain it a bit or at least point me to the good source?

Untried

Answer 1

I believe the documentation you're referring to deals with what, in my opinion, is sort of a design flaw in either AMQP 0-9-1 or RabbitMQ's implementation of it.

Consider the following scenario:

• A queue has thousands of messages sitting in it
• A single consumer subscribes to the queue with AutoAck=true and no pre-fetch count set

What is going to happen?

RabbitMQ's implementation is to deliver an arbitrary number of messages to a client who has not pre-fetch count. Further, with Auto-Ack, prefetch count is irrelevant, because messages are acknowledged upon delivery to the consumer.

In-memory buffers: The default client API implementations of the consumer have an in-memory buffer (in .NET it is some type of blocking collection (if I remember correctly). So, before the message is processed, but after the message is received from the broker, it goes into this in-memory holding area. Now, the design flaw is this holding area. A consumer has no choice but to accept the message coming from the broker, as it is published to the client asynchronously. This is a flaw with the AMQP protocol specification (see page 53).

Thus, every message in the queue at that point will be delivered to the consumer immediately and the consumer will be inundated with messages. Assuming each message is small, but takes 5 minutes to process, it is entirely possible that this one consumer will be able to drain the entire queue before any other consumers can attach to it. And since AutoAck is turned on, the broker will forget about these messages immediately after delivery.

Obviously this is not a good scenario if you'd like to get those messages processed, because they've left the relative safety of the broker and are now sitting in RAM at the consuming endpoint. Let's say an exception is encountered that crashes the consuming endpoint - poof, all the messages are gone.

How to work around this?

You must turn Auto-Ack off, and generally it is also a good idea to set reasonable pre-fetch count (usually 2-3 is sufficient).

Apologia

Thank you for such detailed answer! It makes much more sense now! What happens if I turn off Auto-Ack without specifying pre-fetch count? Wil consumer still be flooded with messages? Or I don`t quite understand the role of the pre-fetch count here? – Untried

I’d have to go look up to see, but in all reality, I think so. A prefetch count of more than 1 only makes sense in very limited cases, where the processing of the message takes about the same amount of time as it does to deliver (there aren’t too many use cases where that is true). I prefer no pre-fetch. – Apologia

Maybe this article is helpful: Work Queues (using the Java Client) – Darbydarce

Answer 2

Being able to signal back pressure is a basic problem in distributed systems. Without explicit acknowledgements, the consumer does not have any way to say "Slow down" to broker. With auto-ack on, as soon as the TCP acknowledgement is received by broker, it deletes the message from its memory/disk.

However, it does not mean that the consuming application has processed the message or have enough memory to store incoming messages. The backlog in the article is simply a data structure used to store unprocessed messages (in the consumer application)

Offertory