Will Lambda receive SQS messages that have been consumed?

Question

I have a system design like this

SQS -> trigger -> Lambda -> if fails -> DLQ

pre condition

1. Lambda function using a try catch block , it won't throw any errors .
2. Lambda function never run out of memory , or timeout . (from Lambda monitoring)
3. Error count is 0 in Lambda monitoring
4. Never use SQS console to view messages
5. Lambda SQS batchSize set to 1
6. DLQ Maximum Receives set to 1
7. Lambda invocation about 60k

After running for a while

1. we found a few message in DLQ
2. message in DLQ has attributes ApproximateReceiveCount is 2 or bigger.

Is this as expected ？

In my opinion if no error throws in Lambda , DLQ message should always be zero .

Willena

Hello Yao, good question. Did you see any error on your lambda dashboard while your message is sent to DLQ ? – Thrice

Are you using the Amazon SQS console to view the messages at any time? If you view the Details of a message in the console, this counts as a Receive. – Aurthur

Never use Amazon SQS console to pull message . No Lambda errors found in Lambda monitoring, also can not found error logs in CloudWatch. – Willena

Are you saying that no CloudWatch Logs are being generated by the Lambda function, or you can see the logs being generated for each invocation but there is no indication of an error within the logs? – Aurthur

Answer 1

SQS is designed to provide at least once delivery. There is a chance that your message could be received by more than one lambda invocation.

If you need exactly once delivery, consider using a fifo queue.

Regarding the second part of your question - why are messages being written to the DLQ. The easiest way to troubleshoot that is to look for the lambda invocation logs which match some uniquely identifying aspect of the SQS message in the DLQ.

Crusty

Thank you . This may be the root cause . If a message delivery twice . ApproximateReceiveCount will increase to 2 ? – Willena

Potentially - remember that is still the ApproximateReceiveCount. It could actually have been delivered more or less than that. – Crusty

@YaoYang if this answered your question, would you please upvote it and mark it accepted so others can find it? Thanks! – Crusty

Answer 2

I have just been doing some preliminary research into a tangentially related question about Lambda and SQS. What I read from AWS documentation is that another reason for redelivery of a message is if the message has sat in its queue for long enough that the visibility timeout has passed.

This can happen if you have a higher level of concurrency than you expect.

See Asynchronous invocation - AWS Lambda about Lambda Asynchronous Invocation, which is--again--only tangentially related to your subject. But I believe the same idea applies and can be inferred from just the documentation I have already seen on SQS and redelivery:

Even if your function doesn't return an error, it's possible for it to receive the same event from Lambda multiple times because the queue itself is eventually consistent. If the function can't keep up with incoming events, events might also be deleted from the queue without being sent to the function. Ensure that your function code gracefully handles duplicate events, and that you have enough concurrency available to handle all invocations.

Sihon