How does RabbitMQ decide when it is time to delete a message?

Question

I am trying to understand the logic for message deletion in RabbitMQ.

My goal is to make messages persist even if there is not a client connected to read them, so that when clients reconnect the messages are waiting for them. I can use durable, lazy queues so that messages are persisted to disk, and I can use HA replication to ensure that multiple nodes get a copy of all queued messages.

I want to have messages go to two or more queues, using topic or header routing, and have one or more clients reading each queue.

I have two queues, A and B, fed by a header exchange. Queue A gets all messages. Queue B gets only messages with the "archive" header. Queue A has 3 consumers reading. Queue B has 1 consumer. If the consumer of B dies, but the consumers of A continue acknowledging messages, will RabbitMQ delete the messages or continue to store them? Queue B will not have anyone consuming it until B is restarted, and I want the messages to remain available for later consumption.

I have read a bunch of documentation so far, but still have not found a clear answer to this.

Sander

I hope I was able to answer your question! If not let me know what I can do to improve it! – Zygote

Answer 1

The simple answer is that messages consumed from one queue have no bearing on messages in another. Once you publish a message, the broker distributes copies to as many queues as appropriate - but they are true copies of the message and are absolutely unrelated from that point forward so far as the broker is concerned.

Messages enqueued into a durable queue remain until they are pulled by a consumer on the queue, and optionally acknowledged.

Note that there are specific queue-level and message-level TTL settings that could affect this. For example, if the queue has a TTL, and the consumer does not reconnect before it expires, the queue will evaporate along with all its messages. Similarly, if a message has been enqueued with a specific TTL (which can also be set as a default for all messages on a particular queue), then once that TTL passes, the message will not be delivered to the consumer.

Secondary Note In the case where a message expires on the queue due to TTL, it will actually remain on the queue until it is next up to be delivered.

Waldo

Answer 2

RabbitMQ will decide when to delete the messages upon acknowledgement.

Let's say you have a message sender:

var factory = new ConnectionFactory() { HostName = "localhost", Port = 5672, UserName = "guest", Password = "guest" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
    channel.QueueDeclare(queue: "hello",
                         durable: true,
                         exclusive: false,
                         autoDelete: false,
                         arguments: null);

    string message = "Hello World!";
    var body = Encoding.UTF8.GetBytes(message);

    channel.BasicPublish(exchange: "",
                         routingKey: "hello",
                         basicProperties: null,
                         body: body);
    Console.WriteLine(" [x] Sent {0}", message);
}

This will create a durable queue "hello" and send the message "Hello World!" to it. This is what the queue would look like after sending one message to it.

Now let's set up two consumers, one that acknowledges the message was received and one that doesn't.

channel.BasicConsume(queue: "hello",
                    autoAck: false,
                    consumer: consumer);

and

channel.BasicConsume(queue: "hello",
                    autoAck: true,
                    consumer: consumer);

If you only run the first consumer, the message will never be deleted from the queue, because the consumer states that the messages will only disappear from the queue if the client manually acknowledges them: https://www.rabbitmq.com/confirms.html

The second consumer however will tell the queue that it can safely delete all the messages it received, automatically/immediately.

If you don't want to automatically delete these messages, you must disable autoAck and do some manual acknowledgement using the documentation:

http://codingvision.net/tips-and-tricks/c-send-data-between-processes-w-memory-mapped-file (Scroll down to "Manual Acknowledgement").

channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);

Zygote

how can i make sure all subscribed services has processed the message before deleting a message. for example having acknowledgement for any subscriber and message only delets when all subscribers acknowledge. – Xenolith

Answer 3

There are different ways where RabbitMQ deletes the messages. Some of them are:

• After Ack from consumer
• Time-to-live(TTL) for that Queue reached.
• Time-to-live(TTL) for messages on that Queue reached.

The last two points state that RabbitMQ allows you to set TTL(Time-to-live) for both messages and queues. TTL can be set for a given queue by setting the x-message-ttl argument to queue.declare, or by setting the message-ttl policy. Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy.

A message that has been in the queue for longer than the configured TTL is said to be dead. Important point to note here is that a single message routed to different Queues can die at different times or sometimes never in each queue where it resides. The death of a message in one Queue has no impact on the life of same message in some other Queue

Slowpoke