Deployment of Airflow Codebase

Question

We are in the process of streamlining our build/deployment pipelines for airflow codebase.

Does anyone have experience in build and deployment pipelines using CI/CD tools for apache airflow?

How, do you deploy your airflow codebase including DAGs/plugins/operators in different environments like test, staging, production etc.

How do you manage airflow.cfg config for each environments?

Where do you manage your configs for each environment.

Paranoid

Answer 1

We build all of our code into a Docker image (DAGs, plugins, different Python packages, different airflow.cfg files, etc.) that gets pushed up to our Kubernetes cluster. That same image runs everywhere, ensuring that dependencies stay locked down and each Airflow is configured best for it's use case (we run multiple Airflow instances on our Kubernetes cluster).

As far as CI/CD, since our deployment is pretty much just a docker push, we've used CircleCI without any issues.

For managing environments, we'll try to keep connections named the same across Airflows (e.g. redshift_conn) but with different credentials (dev Redshift vs prod Redshift). I think there should be more elegant solutions to that, but that's worked so far for us.

Kassia

Thanks, Viraj. Currently, we are running airflow without Kubernete. We are looking/searching for an elegant solution to manage same build artifacts for multiple environments and managing environment specific configurations. – Paranoid

Are you using any orchestration system? At my company, we used Mesos to deploy Airflow environments before migrating to Kubernetes - which has definitely made it easier to manage build artifacts and environments. – Kassia

In a few streams we are using Kubernetes, but not for Airflow. We are planning to use in Airflow as well. Not sure, how much time/effort it will take to move to Kybernetes. – Paranoid