Why use Celery instead of RabbitMQ?

Question

From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a worker could just listen to the MQ and execute the task when a message is received. This achieves exactly what Celery offers, so why need Celery at all?

Zerline

Answer 1

You are right, you don't need Celery at all. When you are designing a distributed system there are a lot of options and there is no right way to do things that fits all situations.

Many people find that it is more flexible to have pools of message consumers waiting for a message to appear on their queue, doing some work, and sending a message when the work is finished.

Celery is a framework that wraps up a whole lot of things in a package but if you don't really need the whole package, then it is better to set up RabbitMQ and implement just what you need without all the complexity. In addition, RabbitMQ can be used in many more scenarios besides the task queue scenario that Celery implements.

But if you do choose Celery, then think twice about RabbitMQ. Celery's message queueing model is simplistic and it is really a better fit for something like Redis than for RabbitMQ. Rabbit has a rich set of options that Celery basically ignores.

Linguiform

"Rabbit has a rich set of options that Celery basically ignores". That's true but a little misleading. You can for example set up routing rules in the Rabbit layer that are not controlled by Celery, but do affect the routing and consumption of Celery tasks. It's a design issue whether you want routing to be handled by the caller, a Celery custom router, or the exchange mechanism. It's true those Rabbit configs can be "invisible" to Celery but that doesn't mean they don't have a useful effect. – Wiebmer

Answer 2

Celery basically provides a nice interface to doing just what you said, and deals with all the configuration for you. Yes you could do it by hand, but you'd just be rewriting celery.

Oval

There's also the operations element. Huge parts of Celery is there for reliability (e.g. not crashing when a particular exception is serialized, etc), and managing workers, and clusters of workers. – Porush

Answer 3

In my opinion, it's easy to integrate celery with flower and other monitoring packages than RabbitMQ.

It all depends on the use case anyways...

If you don't need other functionalities celery provides, RabbitMQ would be an easy way out. Weighing out your options wouldn't be a bad idea...

Bamby