AWS SQS doesn't reliably trigger Lambda

Question

I've set up a small serverless app using Lambda and SQS. In my case i wanted to trigger a lambda every time a message is added to a SQS Queue.

functions in my serverless.yml

functions:
  collectGame:
    handler: js/collect.collectGame
    memorySize: 128
    timeout: 10
    events:
    - sqs:
        arn:
          Fn::GetAtt:
            - gameRequestQueue
            - Arn
    - http:
        method: post
        cors:
          origin: "https://my-api-url.com"
        path: get/game/{id}
        private: true
        request:
          parameters:
            paths:
              id:true

I tested the process by sending 31 Messages at once to the Queue but realized that only 9 Lambdas get executed (by looking into the cloudwatch logs). I looked into the Queue and can confirm that its being filled with all the messages and that its empty after the 9 Lambdas have been triggered.

I'd expect to have 31 Lambda executions but thats not the case. Anyone knows potential reasons why my Lambdas are not being triggered by the messages?

Vernalize

Does your lambda get invoked with multiple messages? – Deemphasize

Are you sure there were only 9 executions and not just 9 log groups in cloud watch? – Numbat

@ArtemArkhipov I tried it again and now i see 11 Invocations inside a total of 8 log groups – Vernalize

Answer 1

Your lambda function is probably being invoked with multiple messages. You should be able to set the BatchSize to 1 when you create the event source mapping, if you only want one message to be sent per lambda invocation

It looks like you are using the serverless framework. See their SQS event documentation for setting the batch size.

Deemphasize

Thanks a lot! That was the exact issue i was experiencing. – Vernalize

Thanks!! I have set BatchSize to 10 and lambda is not triggered, even when there are more than 10 messages in queue. After changing BatchSize to 1, it works again. – Traitor

Answer 2

For anyone using aws sam here is the link that mentions batch size: here, look for the subheading 'Configuring a Queue as an Event Source'. And here is the code that works for me to set this up in the yaml together with a DLQ:

  # add an event trigger in the properties section of your function
  Events:
    MySQSEvent:
      Type: SQS
      Properties:
        Queue: !GetAtt MySqsQueueName.Arn              
        BatchSize: 1

# then define the queue
  MySqsQueueName:
    Type: AWS::SQS::Queue  
    Properties: 
      VisibilityTimeout: 800
      ReceiveMessageWaitTimeSeconds: 10
      DelaySeconds: 10
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt MyDLQueue.Arn
        maxReceiveCount: 2

  # define a dead letter queue to handle bad messages
  MyDLQueue:
    Type: AWS::SQS::Queue  
    Properties: 
      VisibilityTimeout: 900

Hope this helps someone - this took me ages to work out for my app!

Caning

Answer 3

i was also facing the exact same issue. The problem was in my lambda function. If the batch size is more than 1, in that case in a single lambda invocation, multiple SQS messages will be passed to lambda (based on batch size), just handle all the messages in lambda (by iterating through all the messages).

check your event Records array for multiple messages.

{Records: [{..},{..},{..}]}

Sirrah