Private images with Airflow KubernetesPodOperator

Question

I am getting started with Airflow and trying to use the KubernetesPodOperator, but I am having trouble with downloading images from private registries. I did some research but I couldn't find an answer to my problem.

Putting it simply: can I use private images from DockerHub with the KubernetesPodOperator?

Epigone

Have you managed to work this out? Can't make it work :-( – Garek

I haven't tried since. Airflow is quite buggy and not very good at heavy-duty jobs, so we kind of abandoned it... – Epigone

Answer 1

It looks like pod_generator.PodGenerator accepts some kind of object kube_config that knows about imagePullSecrets, but unfortunately KubernetesPodOperator doesn't provide any such kube_config to PodGenerator

As best I can tell, it's just an edge case that slipped through the cracks, although it looks like there is a Jira for that which matches up with a corresponding GitHub PR, but it isn't clear from looking at the changed files that it will 100% solve the problem you are describing. Perhaps weigh in on either the PR, or the Jira, or maybe even both, to ensure it is addressed.

Intersperse

Bummer! Oh well, at least Cloud Composer supports private repos from Cloud Container Registry. That will do it for now. – Epigone

Answer 2

Yes, you can. The KubernetesPodOperator accepts an imagePullSecret since version 1.10.1

:param image_pull_secrets: Any image pull secrets to be given to the pod.
                           If more than one secret is required, provide a
                           comma separated list: secret_a,secret_b
:type image_pull_secrets: str

References: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ https://github.com/apache/airflow/blob/master/airflow/contrib/operators/kubernetes_pod_operator.py

Vaclava

Answer 3

As of Airflow Version 5, to access a private image using KubernetesPodOperator you must create a Kubernetes Secret representing credentials to your image and provide this secret object as an argument to your KubernetesPodOperator.

More info here: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#how-to-use-private-images-container-registry

Cide