RabbitMQ Consume Messages in Batches and Ack them all at once

Question

I have a RabbitMQ queue in which I post thousands of messages. I need a background Service that will:

• Retrieve the messages in batches of 100 messages
• Store those 100 messages in a Database
• Ack all 100 messages at once
• Proceed with the next batch of 100 messages

I'm using the RabbitMQ Client to listen for messages, but I don't know how to "batch" them.

Does anyone has a working example on how to get my messagess 100-100 at a time and ACK them all at once after they have been saved in a DB?

Thanx a lot in advance

Crutch

Answer 1

You want to use the "prefetch" and "multi ack" features -

• Prefetch - https://www.rabbitmq.com/consumer-prefetch.html
• Multi-ack - https://www.rabbitmq.com/confirms.html#consumer-acks-multiple-parameter

This is what you'll do in your code:

• Open a connection and channel
• Set the channel prefetch to 100
• Start a consumer. The callback you register for your consumer will start being called with messages. You will have to save these messages in a list or other data structure
• Once you have received 100 messages, save the delivery tag of the 100th message and do your database work.
• Ack the 100 messages by setting "multi ack" to true and use the delivery tag of the 100th message.
• RabbitMQ will send the next 100 messages in the same manner

Combat

Is it possible that RabbitMQ stores the batches message and once hit the prefetch limit, send the batched messages to consumers? this way we don't need to store the received message in consumers one by one. – Eason

@ShahinGhasemi that is not how the AMQP protocol works. There shouldn't be an issue with storing messages one at a time in the appropriate data structure. You may also wish to investigate RabbitMQ streams - rabbitmq.com/streams.html – Combat

What would happen if there were less than 100 messages in queue? Would prefetch be able to notify the consumer of this or otherwise have the consumer know when there are no more messages? – Hutson

The consumer would have to determine when to stop waiting for the rest of the batch of messages. – Combat

is it possible to push to consumer first 100 messages bit make ack after each success message processing? – Roughhouse