SQS dead letter queue not triggered on AWS Lambda invocation errors

Question

I have an AWS Lambda function, which subscribes to a DynamoDB stream and is configured with an SQS dead letter queue (DLQ). I can see that the correct queue is configured in the Management Console. Also I took care to give my function permissions for sqs:SendMessage on my DLQ.

The subscription works, but still "hangs" on invocation errors as if no DLQ were configured. I.e., if there is a message, which leads to an unhandled exception, the function continues to retry this message until it's dropped from the stream. I can see that the number of invocation errors rises, and no DLQ errors are shown in the function's Cloudwatch dashboard. The SQS queue remains empty.

What I want is that failed messages get forwarded to my DLQ and the subscription continues to the next message. Any ideas?

Edit

As Jonathan Seed said below, DLQ's currently don't work with stream-based subscriptions. AWS Support confirmed that they're working on implementing this though.

Spent

"AWS Support confirmed that they're working on implementing this though." This has been implemented and released for public use as of November 2019: aws.amazon.com/about-aws/whats-new/2019/11/… Please see my answer for the correct way to handle failure on DynamoDB stream events. – Shaffer

Answer 1

I believe this is because DynamoDB streams are stream based event sources. The lambda documentation states that when dealing with stream based event sources "if a Lambda function fails, AWS Lambda attempts to process the erring batch of records until the time the data expires"

From my understanding, the lambda function will retry until the event is either processed successfully or expires and disappears from the stream, the event is never "discarded" by the lambda function, as they are in non-stream based event sources.

You may have to implement your own failure handling as a part of your main lambda function if you wish to discard certain events, posting the event manually to a queue/topic and returning succesfully.

Philtre

I hate to say it, but I think you are right. I wish the DLQ doc page would state explicitly, that they don't have any effect on stream subscriptions, but it makes sense. – Spent

Oh, btw, the trailing slash of the link in your answer breaks it. Couldn't correct it myself due to the 6 character minimum for edits. – Spent

I'm getting the same issue on Lambda as an SQS processor. – Bingo

Is this still true? Am facing same issue where lambda based on kinesis data stream is not putting records in dlq. – Cortez

This is no longer true, see answer by @Shaffer - in SLS you do it that way: serverless.com/framework/docs/providers/aws/events/… – Luther

Answer 2

• Using DynamoDB streams to trigger lambda means that you are using synchronous invocation. However, DLQ is only available for asynchronous invocations.
• The good news is that in November 2019, AWS published new error handling mechanisms for Kinesis and DynamoDB event source.

With this feature, you can configure a destination on failure. This destination can be an SNS topic, SQS queue, another lambda function, or an EventBridge event bus.

For adding this through the console UI,

1. Go to the lambda function
2. Click on the Add Destination button
3. Select Stream invocation
4. Select on failure condition
5. Select SQS queue as the destination and point it to the SQS that you want to use like a DLQ.

For adding it through cloudformation, follow this documentation.
I'll provide a basic example for the trigger that you need to attach to your lambda function:

LambdaTrigger:
  Type: AWS::Lambda::EventSourceMapping
  Properties:
    FunctionName: !GetAtt Lambda.Arn
    EventSourceArn: !GetAtt TableName.StreamArn
    DestinationConfig:
      OnFailure:
        Destination: !GetAtt DLQ.Arn

Shaffer

Answer 3

The destination on failure is good if you are interested only in Metadata which is ridiculous. I am not sure if there is a way to put the entire msg in the DLQ

Lallans

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Whit