Getting duplicate messages from Azure Storage Queue

Question

I am getting duplicate messages from Azure Storage Queue. I am sure that my function is getting executed only once as I am logging the message in the database in the same function. After clicking on the button, I see that, message is queued in storage queue and in a second its dequeued(so again, it's not like that there are 2 same messages enqueued).

Any thoughts on this? Thanks in advance!

P.S - please note that it's not a Azure Service Bus where I can set the property RequiresDuplicateDetection to True to fix this issue.

Prosperus

Answer 1

For now, azure Storage queues do not support Duplicate detection.

There is no mechanism that can query a Storage queue and find out if a message with the same contents is already there or was there before. You can try to implement your own logic using some storage table, but that will not be reliable - as the entry into the table may succeed and then entry into the queue may fail - and now you would potentially have bad data in the table.

Your code should always assume that it can retrieve a message containing the same data that was already processed. This is because messages can come back to the queue when workers that are working on them crash or take too long.

As the storage queue does not support duplicate detection, you can give your voice to feedback.

For more details, you could refer to this article.

Forspent

Thanks Joey for the response. I know that there is no duplicate detection in Azure storage. What I wanted to know is that how people are preventing duplicate messages getting sent from Azure Storage (may be through code or something) or what is the possible way to identify why the workers are crashing and causing them to resend the message? – Prosperus

You can send the message to the list and eachtime before you send message, compare message with list. – Forspent

There are many possible reasons that causing crash and resend message. For example, when you send a large message and timeout. If you do not want to resend message, you can set maxDequeueCount as 0. – Forspent

Answer 2

You can send the message to Queue and when it is read from the Queue then you can delete it so no more duplication but this will be done on server-side from Azure Functions. From client-side you will face too many concurrency issues. If you do not want to delete then dequeue property usage would be the right way as well.

https://learn.microsoft.com/en-us/azure/storage/queues/storage-dotnet-how-to-use-queues?tabs=dotnet#de-queue-the-next-message

Automation

Answer 3

By looking at the official Microsoft Storage queues VS Service Bus queues Capabilities table here we see that Storage queues only guarantee At-Least-Once Delivery while the Service Bus queues guarantee also At-Most-Once using ReceiveAndDelete receive mode.

So if you are looking to guarantee At-Most-Once without the need for you to build the additional infrastructure components and checks you have to go for Service Bus queues.

Lovett