Celery: When should you choose Redis as a message broker over RabbitMQ?

Question

My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks?

Does that mean we should use Redis as a message broker IF we are already using it for something else?

Milliard

Celery clearly recommends using AMQP over Redis. I wouldn't use Redis. – Protomartyr

@Apero Though Rabbitmq has been supported longer than Redis (and is the default), both are listed as stable. I don't see a clear recommendation either way. I'd be curious to read about if you've seen otherwise, however. docs.celeryproject.org/en/master/getting-started/brokers/… – Semicolon

@DanilaGanchar the article mentioned: It is apparent that RabbitMQ takes 75% of Redis’ time to add a message and 86% of the time to process a message. why Redis is faster? – This

Updated link to the brokers in the docs: docs.celeryproject.org/en/stable/getting-started/… – Helladic

Answer 1

I've used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup.

• If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise...
• If you are using Redis for any other reason, go with Redis
• If you are hosting at AWS, go with Redis so that you can use a managed Redis as service
• If you hate complicated installs, go with Redis
• If you already have RabbitMQ installed, stay with RabbitMQ

In the past, I would have recommended RabbitMQ because it was more stable and easier to setup with Celery than Redis, but I don't believe that's true any more.

---

Update 2019

AWS now has a managed service that is equivalent to RabbitMQ called Amazon MQ, which could reduce the headache of running this as a service in production. Please comment below if you have any experience with this and celery.

Theosophy

One annoyance with using Redis / python 3.7 / Celery 4.2 is that the results backend doesn't work because async is now part of python ( see github.com/celery/celery/issues/4849 ) -- this should be fixed with celery 4.3 . I don't know if this also affects RabbitMQ. – Theosophy

However, Celery 4.3 is now released, so this is no longer an issue -- results backend works again with Redis. – Theosophy

You can also use Celery with Amazon SQS which is super simple to deploy and run – Puma

@Puma SQS support is experimental, though. I wonder what the unstable/missing bits are... – Requisite

As of July 2019, SQS is marked as stable. Yet, it is missing monitoring and control channels. Also, SQS behavior is a bit of counter-intuitive. We've got messages processed up to 14 times due to at-least-once delivery policy. – Madalene

According to this ticket github.com/celery/celery/issues/5001 you can not use Amazon MQ as replacement for RabbitMQ. They implement different verions of AMQP protocol (1.0 vs 0.9.1) – Dingus

Now Amazon MQ also offers rabbitmq brokers in addition to activemq, I've used it and it works fine. – Henrique

I've personally moved away from SQS so I don't know if this works/is appropriate for Celery, but for those reading along: SQS now supports FIFO queues that guarantee FIFO and exactly-once delivery. – Campball

Answer 2

The Redis broker gives tasks to workers in a fair round robin between different queues. Rabbit is FIFO always. For me, a fair round robin was preferable and I tried both. Rabbit seems a tad more stable though.

Shelve

"The Redis broker gives tasks to workers in a fair round robin between different queues." Can you provide a reference for this? – Meteor

docs.celeryq.dev/projects/kombu/en/v4.0.2/reference/… Believe me - I tried all the different transport options. I'm actually using SQS now and to me, it's the best one but it's also not round robin. – Shelve