AWS Lambda w/ SQS trigger, SQS Lambda Destinations, never adds to destination queue

Question

I have a simple lambda function that is triggered from a SQS queue and I'm using the new Lambda Destinations functionality.

It is set up to trigger from QUEUE_A, do some modification of the payload body, then send it to QUEUE_B on success, or QUEUE_ERRORS on failure.

QUEUE_B and QUEUE_ERRORS are set up as Destinations on the lambda function.

When I trigger the lambda from the CLI, I get a record on QUEUE_B with a good record, and on QUEUE_ERRORS on a bad record. So, it seems to be working.

But, when the lambda is triggered from SQS, I never get a record on QUEUE_B or QUEUE_ERRORS. A good record runs the lambda, and on a bad record it goes to QUEUE_A_DEADLETTER, which I do not want.

I have tried configuring QUEUE_A to have no retry/deadletter - if I do that, on a bad record it will just retry forever (no matter how low I set the visibility/retries).

What can I try next?

---

EDIT:
CloudWatch shows me exactly what I expect to see - I see good logs on a "good" record, and stack traces/exceptions on "bad" records, so it is not a problem within the function itself AFAIK.

---

EDIT: Replacing the SQS triggers and destinations with SNS triggers and destinations are working. So, I think this is related to SQS being sync and SNS being async? Does anybody know?

Vinegarette

Sounds like you are getting errors within the Lambda function. Have you looked at CloudWatch Logs to see the log output of the Lambda function? You could also add some additional debug to see whether the SendMessage() call succeeded. – Lavernlaverna

CloudWatch shows good logs on my "good" records, and exceptions on my "bad" records. Where is SendMessage() defined? I am not trying to push to SQS in the function code, just as a Destination. I also expect a "bad" record to go to QUEUE_ERRORS but it doesn't. – Vinegarette

Ah! It's using the new AWS Lambda Destinations functionality. Sorry, with that you won't need SendMessage(). – Lavernlaverna

Answer 1

Destinations will not be triggered if you invoke the Lambda function synchronously.

The primary use case of Destinations is to know about the async execution results of Lambda functions, primarily to get more visibility into the execution details like request and response contexts, payloads, exception stack traces etc. So if a Lambda was invoked synchronously (say, using cli or via an SQS trigger), no messages will be delivered to the Destinations endpoints.

When you used the CLI you would have used aws lambda invoke-async. Instead if you use aws lambda invoke (which executes Lambda synchronously), you will see the same problems, your destination endpoints will not receive the message.

You can keep your Destinations endpoints still as SQS (you will see a working example in the article above), but your Lambda trigger will have to change to an asynchronous one.

Flocky

Man, this is disappointing! Why can't we set failure destinations for SQS triggers? I've got a batch of lambda's that I'd like to all hit SNS if they encounter any error invocations. – Roentgenoscope

this help lot only thing we need to call as per latest cli command aws lambda invoke \ --function-name lambdaname \ --invocation-type Event \ --payload '{ "name": "Bob" }' \ response.json – Ngocnguyen

@CharlieSchliesser It looks like you can now, by setting the onFailure property of your EventSourceMapping to an SQS Queue. – Handful

Thank for this post. I spent to long on troubleshooting why my SQS queue did not trigger the destination SNS Topic through a lambda function. When looking again at the documentation (aws.amazon.com/blogs/compute/…) one can see that SQS is missing from the picture showing the possible inputs. – Harday

Answer 2

I got a similar issue with a function that is triggered from the API-Gateway: Lambda triggered from the API-Gateway with SQS as destanation and the SQS working only from invoke but not from the API.

to solve it, I GET Async from inside the function to the SQS URL

Mighty

What do you mean? Do you send a message to SQS explicitly from the code? – Drillstock