How to pull private docker image from GitLab container registry using DockerOperator in Airflow 2.0?

Question

I have installed Airflow 2.0 using Docker on AWS and trying to pull my own private docker image GitLab container registry using DockerOperator but getting permission denied.

How I can pull my docker image from GitLab and How I can create a connection for the Docker image GitLab registry?

Dag file link Docker yml file link

My dag Code:

java_test_scheduler = DockerOperator(
    task_id='Java-hello-world-test-scheduler',
    image='registry.gitlab.com/mobinalhassan/jamay_aeronova:latest',
    auto_remove=True,
    force_pull=True,
    dag=dag
)

I also tried by adding docker volums:

- /var/run/docker.sock:/var/run/docker.sock
- $HOME/.ssh:/root/.ssh:ro
- $HOME/.docker:/root/.docker

Error:

sock.connect(self.unix_socket) PermissionError: [Errno 13] Permission denied

Illnatured

Answer 1

Your dag is missing dag_conn_id definition when pulling from private docker registry:

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html

Woods

I have tried this but failed, There is no documentation on how to create a connection link... – Illnatured

This is the link I have created...docker://gitlab+68hzvkEtonsazVoxNHsy:[email protected] but not working, Can you explain that.. – Illnatured

Are you using airflow connections? airflow.apache.org/docs/apache-airflow/stable/howto/… then use your conn id (for example 'gitlab_conn_id') in dockeroperator initialization code as parameter docker_conn_id='gitlab_conn_id' – Woods

Yes! actually, I did the whole thing, setting up the connection and giving Id in DockerOperator as you have mentioned... – Illnatured

Answer 2

1. I have tried without docker you can check this answer for details on that
2. I wanted to keep least dependency on airflow and hence didn't create the connection from airflow whose details are mentioned in the above referenced answer link
3. As far as your dag link is concerned, I had tried similar approach but could not make it work with image='registry.gitlab.com/mobinalhassan/jamay_aeronova:latest', however it did work out when I removed the :latest and use it like image='registry.gitlab.com/mobinalhassan/jamay_aeronova',, try this out and let me know if it works
4. For full details on pulling private docker image, I have jotted down a step by step tutorial which you can refer at - this link

Hammond

Okay, let me try – Illnatured

@MobinAlHassan edited and added the link to the step by step tutorial – Hammond