Azure Storage Queue - processing messages on poison queue

Question

I've been using Azure Storage Queues to post messages too, then write the messages to a db table. However I've noticed that when an error occurs processing messages on the queue, the message is written to a poison queue.

Here is some background to the setup of my app:

Azure Web App -> Writes message to the queue

Azure function -> Queue trigger processes the message and writes the contents to a db

There was an issue with the db schema which caused the INSERTS to fail. Each message was retried 5 times, which I believe is the default for retrying queue messages, and after the 5th attempt the message was placed on the poison queue.

The db schema was subsequently fixed but now I've no way of processing the messages on the poison queue.

My question is can we recover messages written to the poison queue in order to process them and INSERT them into the db, and if so how?

Standfast

Does this answer your question? Azure: How to move messages from poison queue to back to main queue? – Tahiti

Answer 1

For your particular problem, I would recommend solution mentioned in question part of this post: Azure: How to move messages from poison queue to back to main queue?

Please note that name of poison queue == $"{queueName}-poison"

Cini

Not sure this is the best approach in case a 'corrupted' message gets recycled forever. – Subordinary

@Subordinary Agreed, that is why I said 'for this particular problem'. That code shall run only for a short time to pump poison messages caused by the recent glitch in and then be deleted. Code could also ignore very old (by min timestamp) messages which were not caused by the glitch. – Cini

I agree, for my particular scenario it would work to copy the message back to the original queue and process it again now that the db has been fixed. So thanks for this. However, how would this work in a production environment. There's a potential that another error could occur and as @Subordinary says, messages could get into an eternal loop of retries. I'm curious to find out what manual steps could be used if we notify someone when a message has been added to the queue. – Standfast

Answer 2

In my current project I've created something what is called: "Support functions" in the FunctionApp. It exposes a special HTTP endpoint with Admin authorization level that can be executed at any time.

Please See the code below, which solves the problem of reprocessing messages from the poison queue:

public static class QueueOperations
{
    [FunctionName("Support_ReprocessPoisonQueueMessages")]
    public static async Task<IActionResult> Support_ReprocessPoisonQueueMessages([HttpTrigger(AuthorizationLevel.Admin, "put", Route = "support/reprocessQueueMessages/{queueName}")]HttpRequest req, ILogger log,
        [Queue("{queueName}")] CloudQueue queue,
        [Queue("{queueName}-poison")] CloudQueue poisonQueue, string queueName)
    {
        log.LogInformation("Support_ReprocessPoisonQueueMessages function processed a request.");

        int.TryParse(req.Query["messageCount"], out var messageCountParameter);
        var messageCount = messageCountParameter == 0 ? 10 : messageCountParameter;

        var processedMessages = 0;
        while (processedMessages < messageCount)
        {
            var message = await poisonQueue.GetMessageAsync();
            if (message == null)
                break;

            var messageId = message.Id;
            var popReceipt = message.PopReceipt;

            await queue.AddMessageAsync(message); // a new Id and PopReceipt is assigned
            await poisonQueue.DeleteMessageAsync(messageId, popReceipt);
            processedMessages++;
        }

        return new OkObjectResult($"Reprocessed {processedMessages} messages from the {poisonQueue.Name} queue.");
    }
}

Alternatively it may be a good idea to create a new message with the additional metadata (as information that the message has already been processed in the past with no success - then it may be send to the dead letter queue).

Unaneled

Answer 3

Just point your Azure function to the poison queue and the items in that poison queue will be handled. More details here: https://briancaos.wordpress.com/2018/05/03/azure-functions-how-to-retry-messages-in-the-poison-queue/

Barkley

Answer 4

You have two options

1. Add another function that is triggered by messages added to the poison queue. You can try adding the contents to the db in this function. More details on this approach can be found here. Of course, if this function too fails to process the message you could check the dequeue count and post a notification that needs manual intervention.
2. Add an int 'dequeueCount' parameter to the function processing the queue and after say 5 retries log the failure instead of letting the message go the poison queue. For example you can send an email to notify that manual intervention is required.

Subordinary

Thanks, that's what I was looking for :) You mention manual intervention, and the link to the github docs also touches on this, but what type of manual steps could you take for a message whose dequeue count has been exceeded? For example, I'm using the Azure Storage Explorer and can see the message on the poison queue. However I don't seem to be able to do anything with the message. There is an option to dequeue the message, but I don't see where the message goes after this? I'm pretty new to azure functions (as you can probably guess) so trying to get a good understanding of it. – Standfast

I haven't tried it, but if you trigger another Azure function when a message is added to the poison queue and it process the msg successfully, I assume the msg will be deleted from the poison queue. In the end, isn't that what you want to achieve - process 'failed' msgs. – Subordinary

Ok so 2 functions, one for the main queue and another for the poison queue, doing the same processing. However if it is a problem other than the one I encountered and the message is corrupt, won't it always retry corrupt messages?? Or does the dequeue count kick in on a poison queue too? – Standfast

You can add a 'dequeueCount' int parameter to either of the functions to do some processing. You can add it to the original function and after 5 retries for example let it go the poison queue by default, or send an email for manual intervention. Check the 'Manual poison message handling' topic in the link provided in the answer. Use SendGrid output binding in the function to send an email programmatically. learn.microsoft.com/en-us/azure/azure-functions/… – Subordinary

Thanks, there's plenty there for me to go through and try out. I'll let you know how I get on :) – Standfast

Feel free to upvote and mark it as an answer if you think it helped you :) – Subordinary

I've tried to upvote, but it doesn't show up because I'm a newbie (although apparently it does register)! I'll mark it as an answer once I get playing about with it tomorrow when I'm in the office. – Standfast

Answer 5

You can use azure management studio(cerulean) and move the message from poison queue to actual queue. Highly recommended tool to access queues and blobs and do any production related activity also. https://www.cerebrata.com/products/cerulean

I am just user of the tool and no way affiliated, i recommended because it is very powerful, very useful and makes you very productive.

Click on move and message can be moved to the actual uploaded queue

Advisory

Answer 6

Azure Storage Explorer(version above 1.15.0) has now added support to move messages from one queue to another. This makes it possible to move all, or a selected set of messages, from the poison queue back to the original queue.

https://github.com/microsoft/AzureStorageExplorer/issues/1064

Entirely