Equivalent of Celery in Node JS

Question

Please suggest an equivalent of Celery in Node JS to run asynchronous tasks. I have been able to search for the following:

1. (Later)
2. Kue (Kue),
3. coffee-resque (coffee-resque)
4. cron (cron)
5. node-celery(node celery)

I have run both manual and automated threads in background and interact with MongoDB.

node-celery is using redis DB and not Mongo DB. Is there any way I can change that?When I installed node-celery redis was installed as dependency.

I am new to celery, Please guide.Thanks.

Pandean

You want to use MongoDB as the broker or as the backend? Or, is it that the workers will interact with MongoDB? – Marcoux

I want to use MongoDB as backend, RabbitMQ/Redis as broker – Pandean

Answer 1

Celery is basically a RabbitMQ client. There are producers (tasks), consumers (workers) and AMQP message broker which delivers messages between tasks and workers.

Knowing that will enable you to write your own celery in node.js.

node-celery here is a library that enables your node process to work both as a celery client (Producer/Publisher) and a celery worker (Consumer).

See https://abhishek-tiwari.com/post/amqp-rabbitmq-and-celery-a-visual-guide-for-dummies

Rotifer

I think that you are not having into account all the things that celery solves for you. Is not just a tiny layer on top of AMQP. – Iceland

@Iceland +1, for 99% of users just using celery-node makes much more sense than trying to write your own from scratch as suggested in this answer. – Slumberland

@Slumberland celery-node didn't exist at the time of this answer, duh – Rotifer

@Rotifer Does it exist today? Allow me to rephrase: "for 99% of users just using celery-node makes much more sense than trying to write your own from scratch" – Slumberland

Answer 2

Edit-1/2018

My recommendation is not to use Kue now, as it seems to be a stalled project, use Celery instead. It is very well supported and maintained by the community and supports large number of use cases.

---

Old Answer

Go for Kue, it's a wholistic solution that resembles Celery in Python word; it has the concepts of: producers/consumers, delayed tasks, task retrial, task TTL, ability to round-robin tasks across multiple consumers listening to the same queue, etc.

Probably Celery is more advanced with more features with more brokers to support and you can use celery-node if you like, but, in my opinion, I think no need to go for a hybrid solution that requires installation of python and node when you can only use only language that's sufficient in 90% of the cases (unless necessary of course).

Singsong

Kue is old repo and not well maintained. – Meatman

That's an old post as well – Singsong

Kue is no longer maintained so this answer should probably be removed – Glede

Answer 3

It is also worth mentioning https://github.com/OptimalBits/bull. It is a fast, reliable, Redis-based queue written for stability and atomicity.

Bull 4 is currently in beta and has some nice features https://github.com/taskforcesh/bullmq

Fumigator

Warning: As of 2022, Bull does not support basic features like TTL – Slumberland

Answer 4

Go for Kue, it's a wholistic solution that resembles Celery in Python word; it has the concepts of: producers/consumers, delayed tasks, task retrial, task TTL, ability to round-robin tasks across multiple consumers listening to the same queue, etc.

Kue, after so much time have passed, still has the same old core issues unsolved:

• github.com/Automattic/kue/issues/514
• github.com/Automattic/kue/issues/130
• github.com/Automattic/kue/issues/53

If anyone reading this don't want to rewrite Kue, don't start with it. It's good for simple tasks. But if you want to deal with a lot of them, concurrent, or task chains (when one task creates another) - stop wasting your time.

I've wasted a month trying to debug Kue and still no success. The best choice was to change Kue for Pubs/sub Messaging queue on RabbitMQ and Rabbot (another RabbitMQ wrap up).

Personally, I haven't used Celery as much to put all in for it, but as I've been searching for Celery alternative and found how someone is advising for Kue just boiled my blood in veins.

If you want to send a delayed email (as in Kue example) you can go for whatever you'd like without worrying about errors. But if you want a reliable system task/message queue, don't even start with Kue. I'd personally go with 5. node-celery(node celery)

Gerrit

Thanks for caring enough to post your experience here. This made me laugh and at the same time sympathise -> "boiled my blood in veins". I have felt the exact same way when certain over-hyped web applications are mentioned. I have wasted so many months battling those applications people carelessly recommend just because they are the trend. – Farreaching

The answer that you are quoting is 4 years old, if you take an old post and base your solution on without validation first, then, that's your mistake ... Nowadays, I would recommend people to use Celery all the way, as it is a very well maintained library. – Singsong

We learn from our mistakes. At the end of 2016, when I was searching for a solution Kue seemed to be it. And only when you dig deep enough you see that it's not what you were looking for. Shared my hard feelings for others not to choose Kue (which is still placed as second on the question). And as I see, since mid 2017 Kue have never got back up. @Singsong thanks for updating you answer ;) – Outstare

Answer 5

In our experience, Kue was unreliable, losing jobs. Granted, we were using an older version, it's probably been fixed since. That was also during the period when TJ abandoned the project and the new maintainers hadn't been chosen. We switched to beanstalkd and have been very happy. We're using https://github.com/ceejbot/fivebeans as the node interface to beanstalkd.

Lenity

Answer 6

• Simple Solution

https://www.npmjs.com/package/node-cron

Code :

npm install --save node-cron

import cron from 'node-cron'

cron.schedule('* * * * *', () => {
  console.log('running a task every minute');
});

Butch