My application is written in django f/w which uses celery and redis for asynchronous tasks. I would like to autoscale workers according load/no.of messages in queue. For this, I would like to make use of different options provided by AWS.
What is the replacement with AWS infrastructure for Celery and Redis?
ElastiCache is Amazon's Managed Service for an in-memory data store. Elasticache lets you provision a service that provides high-performance data-store functionality using either Redis or Memcached as a base.
I have not personally used Celery but I know if it as a message queue like RabbitMQ. In this case, the likely Managed Service alternative would be AmazonMQ. AmazonMQ uses Apache ActiveMQ under the hood but the API layer should abstract away most of the differences for you.
If you wanted to, you could probably get away with running Celery on AWS and not use their AmazonMQ Service. You could simply perform the install process on an EC2 instance or with ElasticBeanstalk. Or even run it in a Linux Container on something like ECS or Fargate.
If you were to use EC2, you could probably even get away with using an existing community Marketplace AMI with Celery already provisioned and presumably configurable with cloud-init data.
Here are blog posts and other questions from people setting up Celery-based queue up on AWS:
Hope this helps! If you need any additional information or support for this question feel free to reach out with any questions you may have!
There's an interesting article on aws around use of AWS batch along with celery workers for batch processing in this style.
https://aws.amazon.com/blogs/hpc/run-celery-workers-for-compute-intensive-tasks-with-aws-batch/
In summary
© 2022 - 2025 — McMap. All rights reserved.
