How to schedule on-demand tasks on aws serverless

Question

We are developing a serverless application on AWS. Currently, we have a use case in which we want a user to be able to create configurations for schedules to send them daily, weekly or monthly reports at a specific time on the day on which the report will be sent. We have multiple users, and each user can have multiple report schedule configurations.

I want to know, What is the best architecture for this feature on a serverless application on AWS?

My initial thoughts were to make a table in DB that holds the details of the schedules for each user and make an event rule which runs every minute to trigger a lambda function which checks this table and see if there are any jobs at this time to be triggered or not. and if there is any schedule at this time, it will send the report. But my leader had a concern for this solution as this solution is hitting the DB recurrently and this is a load on DB (we are using aurora MySQL).

Newmark

Answer 1

I'm not sure about what scale will you be operating. But I think you can use cloudwatch events which ultimately triggers the lambda function.

So your user's configuration can become a cloudwatch event with its specific rule and whenever the rule executes it triggers the lambda function.

I'd suggest you to go through this:

https://aws.amazon.com/blogs/mt/build-scheduler-as-a-service-amazon-cloudwatch-events-amazon-eventbridge-aws-lambda/

Looks like it's exactly what you are looking for.

Knot

Yes, I experienced this pattern before when building snoozing functionality. But this architecture is intending to create a specific rule for each configuration, but AWS Event Bridge has some limits per account, as an account can only have 100 Event buses, and each event bus can only have 300 rules, so one bus can only handle 300 schedule configuration. This architecture is not scalable as I also use these rules for other features. see the EventBridge quota per account here: link – Newmark

So that link has multiple approaches suggested. I believe the first one should work as it doesn't use Event Bridge; Here you create cloudwatch rules for every configuration and let them trigger the lambda. – Knot

@SalahHegazi the 300 rules per bus is the soft limit, I think you can increase it up to 2000. But 100 buses per account cannot be changed. I've added quota limits details in the following answer - https://mcmap.net/q/145764/-scalable-event-scheduler-for-possibly-hundreds-of-thousands-of-events – Brassard

Answer 2

Recently(10-Nov-2022) AWS launched a new service called EventBridge Scheduler and it's serverless. I think this is what you are looking for. Mainly in the schedule patterns, you can choose,

• One-time schedule
• Recurring schedule

One-time Schedule

---

Recurring Schedule

In the recurring schedule, you have two schedule types.

1. Cron-based schedule
2. Rate-based schedule

1. Cron-based schedule

2. Rate-based schedule

For both options in Recurring Schedule, you can set the Timeframe as well.

Brassard