Airflow inside docker running a docker container

Question

I have airflow running on an EC2 instance, and I am scheduling some tasks that spin up a docker container. How do I do that? Do I need to install docker on my airflow container? And what is the next step after. I have a yaml file that I am using to spin up the container, and it is derived from the puckel/airflow Docker image

Satan

I think installing docker inside a docker container is a little bit inception. I think it would be ideal to use ECS or Lambda or something serverless if possible. – Zildjian

Answer 1

I got a simpler solution working which just requires a short Dockerfile to build a derived image:

FROM puckel/docker-airflow

USER root
RUN groupadd --gid 999 docker \
    && usermod -aG docker airflow
USER airflow

and then

docker build -t airflow_image .
docker run -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /usr/bin/docker:/bin/docker:ro \
    -v /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7:ro \
    -d airflow_image

Jube

The main idea here is to match the docker container docker group id to the host's docker group id. So, for this to work you must have on your host a docker group with id 999, otherwise this will not work. – Legg

Answer 2

Finally resolved

My EC2 setup is running unbuntu Xenial 16.04 and using a modified the puckel/airflow docker image that is running airflow

Things you will need to change in the Dockerfile

Add USER root at the top of the Dockerfile

USER root

mounting docker bin was not working for me, so I had to install the

docker binary in my docker container

Install Docker from Docker Inc. repositories.

RUN curl -sSL https://get.docker.com/ | sh

search for wrapdocker file on the internet. Copy it into scripts directory in the folder where the Dockerfile is located. This starts the docker daemon inside airflow docker

Install the magic wrapper

ADD ./script/wrapdocker /usr/local/bin/wrapdocker RUN chmod +x /usr/local/bin/wrapdocker

add airflow as a user to the docker group so the airflow can run docker jobs

RUN usermod -aG docker airflow

switch to airflow user

USER airflow

Docker compose file or command line arguments to docker run

Mount docker socket from docker airflow to the docker image just installed

- /var/run/docker.sock:/var/run/docker.sock

You should be good to go !

Satan

Answer 3

You can spin up docker containers from your airflow docker container by attaching volumes to your container.

Example:

docker run -v /var/run/docker.sock:/var/run/docker.sock:ro -v /path/to/bin/docker:/bin/docker:ro your_airflow_image

You may also need to attach some libraries required by docker. This depends on the system you are running Docker on. Just read the error messages you get when running a docker command inside the container, it will indicate you what you need to attach.

Your airflow container will then have full access to Docker running on the host. So if you launch docker containers, they will run on the host running the airflow container.

Decasyllable

Thanks for your response. I had a to add a few more things - I will append the answer when I have a complete solution. I am running this on an EC2 instance running Ubuntu Xenial. My docker container is build on top of debian. Prob. for that reason, I had to add a docker binary in my Dockerfile and start the daemon. I am getting permission denied errors now, so next step would be to add airflow as a user to the nested docker instance so it can trigger docker run. – Satan

@AshishMarkanday Could you not attach the docker binary from the host to the container instead of adding it to the Dockerfile? – Mirilla

Definitely add it to the host or some other container orchestration like ECS or Kubernetes. Don't run docker within docker – Zildjian

This fixed my issue, I was wasting my time checking my DockerOperator, but actually we have to configure the Airflow worker. Thank you. – Mayfly