Hidden messages in Azure storage queue

Question

Sometimes there are some messages in Azure Queues that are not taken in charge by Azure Functions and also are not visible from StorageExplorer. These messages are created without any visibility delay.

Is there any way to know what do those messages contain, and why are they not processed by our Azure Functions?

In the image you can see that we have a message in queue but it is not visible in the list and it is there from hours.

Frock

did you check out poison messages. deadletter queues learn.microsoft.com/en-us/azure/app-service-web/… – Bannock

Can you see if your function has executed (and failed) recently? That number indicates either 'expired' or 'invisible' messages. 'Invisible' messages should re-appear in the queue when the InvisibilityTimeout expires. – Max

I don't have an answer for you but I can say that this happens because some process read the message and therefore it is currently invisible until the invisibility timeout expires or the process manually releases it. The purpose of this mechanism is to stop multiple processes from processing the same message. – Campaign

Answer 1

The Azure Queue API currently has no way to check invisible messages.

There are several situations in which a message will become invisible:

1. The message was added with an VisibilityTimeout in the Put Message request. The message will be invisible until this initial timeout expires.
2. The message has been retrieved (dequeued). Whenever a message is retrieved it will be invisible for the duration of the VisibilityTimeout specified by the Get Messages request, or 30 seconds by default.
3. The message has expired. Messages expire after 7 days by default, or after the MessageTTL specified in the Put Message request. Note: after a while these messages are automatically deleted, but until that point they are there as invisible message.

Use cases

Initial VisibilityTimeout

Messages are created with an initial VisibilityTimeout so that the message can be created now, but processed later (after the timeout expires), for whatever reason the creator has for wanting to delay this processing.

VisibilityTimeout on retrieving

The intended process for processing queue messages is:

1. The application dequeues one or more messages, optionally specifying the next VisibilityTimeout. This timeout should be bigger than the time it takes to process the message(s).
2. The application processes the message(s).
3. The application deletes the messages. When the processing fails the message(s) are not deleted.
4. Message(s) for which the process failed will become visible again as soon as their VisibilityTimeout expires, so that they can be re-tried. To prevent endless retries step 2. should start by checking the DequeueCount of the message: if it is bigger than the desired retry-count it should be deleted, instead of processed. It is good practice to copy such messages to a deadletter / poison queue (for example a queue with the original queue name plus a -poison suffix).

MessageTTL

By default messages have a time-to-live of 7 days. If the application processing cannot handle the amount of messages being added, a backlog could build up. Adjusting the TTL will determine what happens to such backlog. Alternatively the application could crash, so that the backlog would build up until the application would be started again.

Prose

Answer 2

It seems that the message is expired. The following steps could reproduce the issue, you could test it.

Add message with a short TTL

After the message has been expired

Parody

this should not be the case, because the messages have 7 days expiration – Frock

In above steps, specifying 7 seconds just for quickly reproduce the issue. – Parody

I tried your suggestion and it is correct. I'm wondering whether the expired messages will always remain there as invisible messages, or whether they are removed at some point. – Prose