SQS - Delivery Delay of 30 minutes

Question

From the documentation of SQS, Max time delay we can configure for a message to hide from its consumers is 15 minutes - http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html

Suppose if I need to hide the messages for a day, what is the pattern? For eg. I want to mimic a daily cron for doing some action.

Thanks

Dauphin

Have a look at alestic.com/2015/05/aws-lambda-recurring-schedule for daily cron. – Richardricharda

Using AWS:SQS to mimic a cron job is bad idea. AWS:SQS offers eventual delivery, it does not guarantee that message will be delivered exactly after the delay_seconds you specified. There could a second or more extra time taken(though extremely rare) and also duplicate deliveries. Same with using Visibility timeout field. Be careful. – Guardian

Answer 1

The simplest way to do this is as follows:

SQS.push_to_queue({perform_message_at : "Thursday November 2022"},delay: 15 mins)

Inside your worker

message = SQS.poll_messages
if message.perform_message_at > Time.now
   SQS.push_to_queue({perform_message_at : "Thursday November 
   2022"},delay:15 mins)
else
   process_message(message)
end

Basically push the message back to the queue with the maximum delay and only process it when its processing time is less than the current time.

HTH.

Dann

@flicflac, the total number of messages in the queue, will never exceed the number of jobs to be delayed, no question of anything getting clogged up. Are you one of the creators or promoters of the website you mentioned? Would be nice to clarify your position, as it consists of a paid service. Thanks! – Dann

Hey @user2076066, sorry I should have clarified. I ran into the same issue as you and the OP, and am building API's to solve this exact problem. By clogged up, the main issue with the proposed solution is that the consumer should only have 1 responsibility - processing the message. Here it has two, both re-queuing delayed messages, and processing messages. Over time, if you have long delays, it spends more time requeueing delayed messages than fulfilling its primary job: processing the message. Btw I'd love to chat some-time just to hear about your use-case more. – Grotesquery

My original comment was deleted but the point I made is still important. If you are delaying a message for a day, you are potentially queueing and re-queueing the same message ~100 times before it finally gets processed with this solution. This won't scale well and the AWS costs could be very expensive. – Grotesquery

Another option is that you can configure the delay on the SQS itself (either from the console or when creating a SQL). This might be useful if there are multiple producers/publishers for the same SQS. – Inhere

What's the argument against using a CRON job to only push the message when it should be performed instead? – Wholism

Answer 2

Visibility timeout can do up to 12 hours. I think you can hack something together where you process a message but don't delete it and next time it is processed its been 12 hours. So a queue with one message and visibility timeout of 12 hours. That gets you a 12 hour cron.

Dumortierite

This doesn't scale. There is a limit of 120,000 messages which can be invisible (aka in-flight) at any given time. – Myiasis

Another issue with this is as far as I know an invisible message is really invisible, even from the web console. That makes it harder to diagnose/monitor/alert your SQS queue. In that sense I believe using Cloudwatch or EventBridge is more proper a solution for this problem. – Darius

Answer 3

Cloudwatch is likely a better way to do it. You can use a createEvent API with the timer, and have it trigger either a lambda function or an API call to whatever comes next.

Another way to do is to use the "wait" utility in an AWS step function.

https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-wait-state.html

In any case, unless you are extremely sure you will never need anything more than 15 minutes, the SQS backdoor to add the delay seems hacky.

Stempien

How to use the wait utility with the sqs – Serviceberry

Answer 4

Re enqueuing messages into the same queue might not be the best route. It can be confusing in the event an issue arises, seeing a full queue (should it be full? or is the consumer not processing them?). The messages probably should only enter the queue when it should be processed within 15 min. Else you will be invoking thousands of invocations to simply re enqueue a message.

I would do the following:

Lambda worker 1: Receives new message, if item should be delayed more than 15 min, store in DynamoDB, else enqueue message.

Lambda worker 2: Every n(5?) minutes, poll DynamoDB for messages that should be processed within the next 15 minutes, enqueue the messages with the correct delay.

Simply add a index so querying for messages by execution date is fast, this should cover the ability to enqueue thousands of messages per second. If your throughput needs are higher than this, this may not be the best solution.

Lichee

Answer 5

If resource utilization efficiency is not a concern then a little tweak on using a visiblity-timout-queue can come in handy. This can let us have a delay upto 12 hours, but again this can be extended if we switch the message to another queue.

Amazon SQS visibility timeout feature in queue provides a interval based retry mechanism for the messages which are not removed from the queue (if the consumer is unable to process it) with default being set to 30 seconds. So, the same message will keep getting visible to a consumer again at 30 seconds intervals untill its removed from the queue.

SQS also provide a way to handle the messages which are not removed from queue using a dead letter queue which will queue the messages which are unable to process (or say not removed from the main queue). The number of retries after which the message goes into dead letter queue is configurable.

The catch here is in using the dead letter queue for processing instead of actual queue.

We can configure two queues Q1 and Q1-dead-queue. Q1 has Q1-dead-queue as it backup queue after 1 try with defaultVisibilityTimeout set as per requirement. Consumer for Q1 will just consume it (no processing) but won't remove it from Q1. This message will then wait for the configured visibility timeout (can be max 12 hours) and it will be put to Q1-dead-queue and consumer of dead queue can now process it in a timely manner.

Further more, the visbility timout is also configurable at message level.

Refs:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html

Atheism

Answer 6

Two thoughts.

1. Untested. Perhaps publish to and SNS topic that has no SQS queues. When delivery needs to happen, subscribe the queue to the topic. (I've not done this, I'm not sure if this would work as expected)
2. Push messages as files to a central store (like S3). Create a worker that looks at the time created timestamp and decides whether to publish them to a queue or not. If created >= 1d ago, publish.

Jacy

Any kind of worker. An ec2 vm is what I had in mind. – Jacy

Arun, AWS's elastic beanstalk supports directly linking worker instances to an SQS queue – Northeaster

Answer 7

You can do this by adding a DLQ with MaxReceives set to 1 on the first queue. Add a simple Lambda on the first queue and fail the message vi Lambda. So message will be moved to DLQ automatically and then you can consume from DLQ. Both primary queue and DLQ can have max 15 min delay, so finally you get 30 min delay.

So your consumer app receives the message after 30 minutes, without adding any custom logic on it.

Lumpkin

Answer 8

This was a challenge for us as well and I never found a perfect solution so I ended up building a service to address it. Obviously self promotion here but the system allows you to work around the DelaySeconds limitation and set arbitrary date/times at scale.

https://anticipated.io

Some of the challenges working with Step Functions are scale of registered machines (if your system had that requirement). If you use EventBridge to fire them you run out of allowable rulesets (limit is 200 as of this posting). Example: if you need to set 150,000 arbitrary events a month you run into limits quickly.

Directional