Airflow + celery or dask. For what, when?

Question

I read in the official Airflow documentation the following:

What does this mean exactly? What do the authors mean by scaling out? That is, when is it not enough to use Airflow or when would anyone use Airflow in combination with something like Celery? (same for dask)

Bigod

Answer 1

In Airflow terminology an "Executor" is the component responsible for running your task. The LocalExecutor does this by spawning threads on the computer Airflow runs on and lets the thread execute the task.

Naturally your capacity is then limited by the available resources on the local machine. The CeleryExecutor distributes the load to several machines. The executor itself publishes a request to execute a task to a queue, and one of several worker nodes picks up the request and executes it. You can now scale the cluster of worker nodes to increase overall capacity.

Finally, and not ready yet, there's a KubernetesExecutor in the works (link). This will run tasks on a Kubernetes cluster. This will not only give your tasks complete isolation since they're run in containers, you can also leverage the existing capabilities in Kubernetes to for instance auto scale your cluster so that you always have an optimal amount of resources available.

Taunton

For LocalExecutor, tasks are executed as subprocess: ...If it happens to be the LocalExecutor, tasks will be executed as subprocesses; in the case of CeleryExecutor and MesosExecutor, tasks are executed remotely... – Sayce

An undercommunicated feature of SO is that it's a wiki (at least in some sense). You're absolutely right, please feel free to edit the original answer. – Taunton

Answer 2

You may enjoy reading this comparison of dask to celery/airflow task managers http://matthewrocklin.com/blog/work/2016/09/13/dask-and-celery

Since you are not asking a specific question, general reading like that should be informative, and maybe you can clarify what you are after.

-EDIT-

Some people coming to this more recently may wish to look into prefect, which is a sort of rewritten airflow with dask in mind (comes in open-source core with paid enterprise features).

Argil

Answer 3

Following on @mdurant's answer, here is a tutorial I wrote with 3 Airflow DAG examples that use Dask. Hopefully this gives you some sense of when and how to use Dask together with Airflow.

https://coiled.io/blog/3-airflow-dag-examples-with-dask/

Vandalize