How to trigger lambda with some delay when message published to SQS?

Question

I have a lambda configured to be triggered when messages are published to SQS queue. Here is the SAM template for deployment.

  MyQueue:
    Type: AWS::SQS::Queue
    Properties:
        VisibilityTimeout: 180
        DelaySeconds: 90

  MyLambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../pathToCode
      Handler: index.handler
      Events:
        MySQSEvent:
          Type: SQS
          Properties:
              Queue: !GetAtt MyQueue.Arn

I am using DelaySeconds property of AWS::SQS::QUEUE which apparently doesn't work. My lambda get executed as soon as the message is published to queue. How can I put delay in it?

Hosmer

Well, that's working as expected. DelaySeconds will delay the delivery of the message to the queue in the specified amount of time. It doesn't mean it will be invisible for that amount of time, so as soon as the message hits the queue, your Lambda function tries to pick it up. I am afraid what you're trying to achieve is impossible. I really don't see a great benefit of doing it. Maybe if you refactor your architecture a little bit you won't need these types of workarounds – Crisis

@ThalesMinussi thanks for responding. My use-case is, I need to perform some action in another lambda after a minute of executing a specific task. So I push a message to queue(triggering another lambda) hoping it will be executed after a minute. Does it make any sense? – Hosmer

You can't achieve that with SQS I am afraid. One way I see to do it (without thinking too much) is to create an item in DynamoDB that contains the message. You could then have a cloud watch event hooked to a Lambda polling this DynamoDB table every 10 seconds for example, so everytime it sees a new item that has not yet run, you then invoke your Lambda. Now I return the question to you: does it make sense? – Crisis

I said 10 seconds but it was a random number. If you want to see it every minute, then run the cloud watch event every minute. However, it could fall into some grey area that your item would not be picked up for 1 minute and 59 seconds. If you can wait that long I think this should be OK. You could also add a column in your DynamoDB table that has the information around WHEN it should run. So if you run your CloudWatch Event and item.shouldRunAt is later than current time, don't do anything and wait for the next execution – Crisis

That could work, how about step function? What if I create a step function of which first step will just be a delay of one minute and after that my lambda can be executed. – Hosmer

I am not familiar with Step functions. @Ronyis has posted an answer around it. – Crisis

Answer 1

EDITED:

There are 3 ways you can consider for delaying message delivery from SQS to Lambda:

1. Configure DelaySeconds for the queue, so that new messages will be invisible to consumers (such as your Lambda) for a specific amount of time, up to 15 minutes
2. Configure MaximumBatchingWindowInSeconds in your Lambda event source (trigger). Your Lambda can aggregate messages for the specified duration (up to 5 minutes), or until reaching the BatchSize (up to 10,000 messages, 10 by default). This option configures a maximal waiting time for messages, while for some messages there can still be no waiting at all (when the max window time / size is reached).
3. Use a Step Functions state machine (with a Wait state), as proposed in the original answer. This gives the maximal flexibility, and the ability to wait up to 1 year.

Original answer:

The best solution is to use AWS Step Functions.

The lambda triggered by the SQS should execute a state machine, where the first step is the required amount of sleep, and the second one is the lambda invocation.

Geese

I have come to same conclusion. Thanks @Geese – Hosmer

@AffanShahab You may want to check out the batch window feature. See my answer. – Nomothetic

@ronyis: I'd love to understand why Step functions are a better solution than SQS delay with lambda? – Ebneter

@Ebneter updated the answer - depends on the use-case – Geese

Answer 2

In 2021, the best answer is to use the new "batch window" feature of SQS:

https://aws.amazon.com/about-aws/whats-new/2020/11/aws-lambda-now-supports-batch-windows-of-up-to-5-minutes-for-functions/

Using this, you can allow the SQS queue to gather messages for up to 5 minutes before the Lambda function will be invoked.

Nomothetic

Thanks for the updated answer, Its been long since I did this and not currently on that project. I'll check it if needed in future – Hosmer

MaximumBatchingWindowInSeconds says: "The maximum amount of time, in seconds, that Lambda spends gathering records before invoking the function.". The property is used to wait until the batchSize is reached. If you need to wait X minutes for the execution of each message in the queue, this solution will not work, for example, if the batchSize is 10 and there are 9 messages accumulated, when the next message arrives, the execution will be immediate without waiting for the necessary delay of x minutes. – Impeach

Answer 3

Delay Seconds : It delays the Receive Message call whenever you poll the Queue. But whenever you push it message into the queue, the message gets inserted immediately.

If you want your lambda to execute after a certain interval post the trigger, put a sleep(n) before you start your execution and increase the lambda timeout if required. As of now there's no direct way to delay the trigger, your lambda will be triggered as soon as the message is in the queue.

Candicandia

It is not an optimal solution, because lambda has costs binded to the execution time – Cameroncameroon

@Nikaidoh absolutely correct. It'll add an overhead cost based on the amount of memory being used for that particular amount of time. – Candicandia