AWS - SQS Batch Size and Lambda approach

Question

If I understand correctly, batch size setting on Lambda decides how many messages to take in one sweep from the SQS. Therefore this JSON (taken from the test Lambda SQS);

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "FAIL",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {
      },
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:eu-west-1:123456789012:MyQueue",
      "awsRegion": "eu-west-1"
    }
  ]
}

There is Records array. And if I set batch size to 5, then if there are 5 messages in SQS, they will be included in array. If there are 10 messages, then Lambda will be invoked twice, with 5 messages in each Record.

I am now confused a bit, as to what approach to take. My Lambda is fairly simple. It will be Axios POST request to external service. If it errors out, I will throw an error. I could even use axios-retry, and make retries fairly easy.

Should I use batch in my case? Naively lookin, all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

Contrary, I would have to iterate via all messages and attempt an Axios request. What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

Maplemaples

Answer 1

all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

I don't think you need batch processing in above case.

What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

Message(s) won't be deleted from the queue if your Lambda errors, based on the SQS visibility timeout and redrive policy configuration. SQS will trigger Lambda again according to the configuration. If you have configured DLQ then after maxReceiveCount is reached, failed messages will be added to the DLQ and removed from the main queue.

Alwitt

In the lambda, I am doing a programmatic delete from the sqs queue after each one is processed. This should prevent it from resending the ones that succeeded. Then, if I got any error at all, I throw an exception to signal the ones that failed (that were not deleted) to be requeued – Treiber

What happens when you get one "bad" message in a batch of 10, and it's at position 5, and my Lambda throws an exception. If I have a DLQ configured, then after the maxReceiveCount is reached, is only message 5 put on the DLQ? Or would messages 6, 7, 8, 9, and 10 also be placed on the DLQ? This would assume messages 6-10 were involved with the batch of messages that included message 5 during each invocation. – Showbread

@Showbread SQS doesn't know which of the batch of 10 failed, so it has no option but to ultimately send all 10 to the DLQ. If you want more granularity, but still use batch processing, then your Lambda function can explicitly delete each successful message from the SQS queue, leaving only the unprocessed and failed messages in the SQS queue. For example, using npmjs.com/package/@middy/sqs-partial-batch-failure – Scriptural

SQS now has the provision to fail specific messages in the queue - #55498407 – Womanhood

Answer 2

What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

As per AWS documentation, SQS event source mapping now supports handling of partial failures out of the box. For this to work you have to include ReportBatchItemFailures in your EventSourceMapping configuration. So if third of five messages fail don't throw an exception, instead keep a track of the messageId of the failed message and continue processing 4th and 5th message. Return back

{
  "batchItemFailures": [
    { "itemIdentifier": "id3" }
  ]
}

where id3 is the messageId of the 3rd message as response to SQS. For further details refer the documentation and SO post

Womanhood