Can I redrive messages from an SQS queue that is an onFailure target from a Lambda function back to that that function?

Question

I have an SQS queue (Q) that receives messages via the onFailure "destinations" setting from a Lambda function (F). The Lambda function is triggered by and EventBridge event bus "rule".

My question is: Can I configure the redrive policy of queue Q so that I can redrive messages directly to function F?

Currently, I have set the redrive allow policy to allowAll, but the "Start DLQ Redrive" button is disabled in the console. Looking at the configuration settings for the drive allow policy, I get the feeling that only other queues can be a target for a redrive.

What confuses me about this is that my goal here was to use the onFailure function of the "destinations" feature. Destinations can only be used when a function is called asynchronously and queues trigger lambdas synchronously. So if I were to put a queue in front of my lambda function F that could be a target for a redrive, then I would not be able to use the onFailure destination.

Monet

What's triggering the Lambda? – Imaret

An eventbridge event bus rule – Monet

Answer 1

It's not possible to send an event payload from Queue Q to Lambda F with redrive. Redrive works by sending messages from the DLQ back to the source queue, not directly to a Lambda target. Consider, too that the SQS message structure differs from that of EventBridge events, which would confuse your Lambda.

Check out Event replay as an alternative. Or add a Lambda to periodically read from the DLQ and resubmit the events.

Imaret

Interesting. That makes sense. There's an unfortunate tradeoff here. I wanted to use destinations to get the rich message, which includes the error data. That makes diagnosis of the problem so much easier. But now that I have that, there's now way to redrive the messages :(. – Monet

I wonder if another approach would be to add another queue that can trigger the lambda, which can serve as the redrive target for the DLQ. As you said, @Imaret that could confuse the lambda function since the event structure would be different, but maybe one could handle that argument polymorphism. Not ideal but would be so convenient to be able to redrive the rich messages. – Monet

@Monet Also possible. Be careful, though. A gotcha with argument polymorphism is that if Lambda F fails repeatedly on a single message, Lambda F will get passed a differently shaped, increasingly nested message each time the bum message travels through the redrive loop. If you're keen to keep the redrive, consider: Queue Q -> Redrive Q2 -> Resubmitter Lambda -> Bus, where the Resubmitter Lambda turns the redriven Queue message back into the proper event shape and puts it back into the Event Bus. – Imaret

that is a really good point about the almost recursive nesting. That'd be really ugly. I like your other suggestion as well. – Monet