Why would running scheduled tasks with Celery be preferable over crontab?

Question

Considering Celery is already a part of the stack to run task queues (i.e. it is not being added just for running crons, that seems an overkill IMHO ).

How can its "periodic tasks" feature be beneficial as a replacement for crontab ? Specifically looking for following points.

• Major pros/cons over crontab
• Use cases where celery is better choice than crontab
• Django specific use case: Celery vs crontab to run django based periodic tasks, when celery has been included in the stack as django-celery for queing django tasks.

Aerobic

Biggest thing for me is that cron's env vars tend to diverge a lot from rest of app deployment - wsgi app server and celery worker. Having periodic tasks in celery avoids a lot of path & settings hell. But it's entirely a judgement call: my analysis is for a primarily django-oriented codebase & deployment setup, but a multi-language codebase with lots of other cronjobs might consider the cron environment "more standard" than the django one. – Sharice

This answer, though not exhaustive, provides some good points in this direction. And also its easier to manage/control tasks through django admin instead of logging into servers and manually editing the crontab. – Unloose

Answer 1

I've been using cron for a production website, and have switched to celery on a current project. I'm far more into celery than cron, here is why:

• Celery + Celerybeat has finer granularity than cron. Cron cannot run more than once a minute, while celery can (I have a task run every 90 seconds which checks an email queue to send messages, and another which cleans the online users list).
• A cron line has to call a script or a unique command, with absolute path and user info. Celery calls python functions, no need to write more than code.
• With celery, to deploy to another machine, you generally just have to pull/copy your code, which is generally in one place. Deploying with cron would need more work (you can automate it but...)
• I really find celery better suited than cron for routine cleaning (cache, database), and in general, for short tasks. Dumping a database is more a work for cron, however, because you don't want clutter the event queue with too long tasks.
• Not the least, Celery is easily distributed across machines.

Fifteen

+1. you can always add and manage periodic tasks from the admin site, no need to connect to the servers console. – Grandnephew

Some notes: Cron tasks that run frequently and can "overlap" should be run with flock when overlap is not desired. Celery tasks for such cases should have expiry time specified. Servers may have different default timezones and this should be considered when setting up cron. Celery can be configured to use specific timezone on app level with CELERY_TIMEZONE. Queue of celery tasks consumes memory or diskspace: you should be sure that you queue does not grow indefinetly. "Deploying with cron" can be automated to some degree with run-parts – Disillusionize

Answer 2

Celery is indicated any time you need to coordinate jobs across multiple machines, ensure jobs run even as machines are added or dropped from a workgroup, have the ability to set expiration times for jobs, define multi-step jobs with graph-style rather than linear dependency flow, or have a single repository of scheduling logic that operates the same across multiple operating systems and versions.

Lade