Azure queue - can I verify a message will be read only once?

Question

I am using an Azure queue and have several different processes reading from the queue.
My system is built in a way that assumes each message is read only once.
This Microsoft article claims Azure queues have an at least once delivery guarantee which potentially means two processes can read the same message from the queue.
This StackOverflow thread claims that if I use GetMessage then the message becomes invisible to all other processes for the invisibility timeout.

Assuming I use GetMessage() and never exceed the message invisibility time before I DeleteMessage, can I assume I will get each message only once?

Downwind

I think you can assume that. – Pablopabon

I asked the following question to my Microsoft Azure contact: "I’m relying heavily on queue’s and want to make sure that if I GetMessage() and then DeleteMessage() , no other process will get the same message from the queue. According to this Microsoft article, queues have no At-Most-Once guarantee. Does this mean that two processes can read the same queue message and process it?" Response: Yes – this is exactly what it means. If you want that each message will be proceed exactly once you will have to use ServiceBus queue. – Downwind

Answer 1

I think there is a property in queue message named DequeueCount, which is the number of times this message has been dequeued. And it's maintained by queue service. I think you can use this property to identify whether your message had been read before.

https://learn.microsoft.com/en-us/dotnet/api/azure.storage.queues.models.queuemessage.dequeuecount?view=azure-dotnet

Hoffarth

What happens if processA checks DequeueCount and see it is 0. Then processA looses the CPU and processB checks DequeueCount and also finds it at 0. ProcessB then Dequeues the message and then processA wakes up again and also Dequeues the message. That way they both perform the business logic on the same queue message. Am I missing something here? – Downwind

@Gilad - the DequeueCount is set by the service when the item is dequeued, so threading issues don't apply. So you could achieve at-most-once semantics by dropping any messages with DequeueCount > 0. But I don't think there's any way to get exactly-once semantics with queues. – Anthroposophy

That's correct as 'breischl' said, the DequeueCount is maintained by queue service. It will be +1 once a message was dequeued. The queue service itself achieve at-least-once, and if you checked the DequeuCount > 0 then I think you achieve at-most-once. Only one thing, you'd better move the messages that dequeucount >0 to another 'poison message' instead of just delete them, as those messages probably means some background task was not finished yet. – Hoffarth

Is that DequeueCount consistent in concurrency scenarios though? Could two threads dequeue the same message simultaneously and both see it as DequeueCount: 0? – Elson

You cannot rely on DequeCount. @MikeAsdf scenario is valid. In concurrent reading scenarios Microsoft makes not guarantee. You need to use Service Bus – Fairweather

Answer 2

No. The following can happen:

• GetMessage()
• Add some records in a database...
• Generate some files...
• DeleteMessage() -> Unexpected failure (process that crashes, instance that reboots, network connectivity issues, ...)

In this case your logic was executed without calling DeleteMessage. This means, once the invisibility timeout expires, the message will appear in the queue and be processed once again. You will need to make sure that your process is idempotent:

Idempotence is the property of certain operations in mathematics and computer science, that they can be applied multiple times without changing the result beyond the initial application.

An alternative solution would be to use Service Bus Queues with the ReceiveAndDelete mode (see this page under How to Receive Messages from a Queue). If you receive the message it will be marked as consumed and never appear again. This way you can be sure it is delivered At-Most-Once (see the comparison with Storage Queues here). But then again, if something happens while your are processing the message (ie: server crashes, ...), you could loose valuable information.

Update:

This will simulate an At-Most-Once in storage queues. The message can arrive multiple times via GetMessage, but will only be processed once by your business logic (with the risk that some of your business logic will never execute).

• GetMessage()
• DeleteMessage()
• AddRecordsToDatabase()
• GenerateFiles()

Loseff

Assuming my process never crashes and DeleteMessage is always called within the invisibility period, can I assume no other process will get the same message? – Downwind

Yes (some content to satisfy the minimum character count) – Loseff

If this is the case I can do: queue.GetMessage(); immediately followed by queue.DeleteMessage(); in which case I have an At-Most-Once guarantee. If this is the case, why does Microsoft claim there is no At-Most-Once guarantee? – Downwind

Yes. Even if the DeleteMessage fails, you won't have executed any business logic so it would be safe to process the message again the next time. – Loseff

My point is as following: if what you're saying is true, why would Microsoft claim there is no At-Most-Once guarantee? – Downwind

The call to DeleteMessage() can still fail. And the Guest OS could crash before calling DeleteMessage(). And the host OS can crash. And the hardware can fail. There's NO guarantee DeleteMessage() will be called or will succeed. Ergo... there's no at-most-once delivery guarantee. – Tavey

True, but if he calls DeleteMessage immediately after the message arrives, you can assume the business logic executes at-most-once. (Note: the original comment did not use the term at-most-once) – Loseff

I asked my MS Azure contact and her response was: "if you want that each message will be proceed exactly once you will have to use ServiceBus queue." Unfortunate, but as it's from Microsoft I guess it's official... – Downwind

The answer from your Microsoft contact is incorrect. Sandrino is correct, and the logic described here gives "at most once" processing of messages. (This says nothing about delivery, just processing.) – Suppliant

To follow up on Sandrino's example: let's say my queue has itemA and I have two processes. It is possible that both processes will perform GetMessage() and get ItemA, but one of them (the slower one) will fail when calling DeleteMessage() and therefor will not execute the business logic. Is this correct? – Downwind