Airflow - ModuleNotFoundError: No module named 'kubernetes'

Question

I installed Python, Docker on my machine and am trying to import the from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator but when I connect the docker, I get the message that the module does not exist. I have already done the pip install apache-airflow[kubernetes] and I still have the same error. Is there a specific machine location that I should check if the library is actually installed? What can I do to solve this?

from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
import logging
import os
from airflow.utils.helpers import parse_template_string

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.utcnow(),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(
    'kubernetes_sample', default_args=default_args, schedule_interval=timedelta(minutes=10))


start = DummyOperator(task_id='run_this_first', dag=dag)

passing = KubernetesPodOperator(namespace='default',
                          image="Python:3.6",
                          cmds=["Python","-c"],
                          arguments=["print('hello world')"],
                          labels={"foo": "bar"},
                          name="passing-test",
                          task_id="passing-task",
                          get_logs=True,
                          dag=dag
                          )

failing = KubernetesPodOperator(namespace='default',
                          image="ubuntu:1604",
                          cmds=["Python","-c"],
                          arguments=["print('hello world')"],
                          labels={"foo": "bar"},
                          name="fail",
                          task_id="failing-task",
                          get_logs=True,
                          dag=dag
                          )

passing.set_upstream(start)
failing.set_upstream(start)

webserver_1 | Traceback (most recent call last): webserver_1 |
File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 377, in process_file webserver_1 | m = imp.load_source(mod_name, filepath) webserver_1 | File "/usr/local/lib/python3.6/imp.py", line 172, in load_source webserver_1 | module = _load(spec) webserver_1 | File "", line 684, in _load webserver_1 | File "", line 665, in _load_unlocked webserver_1 | File "", line 678, in exec_module webserver_1 | File "", line 219, in _call_with_frames_removed webserver_1 | File "/usr/local/airflow/dags/example_airflow.py", line 3, in webserver_1 | from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 21, in webserver_1 | from airflow.contrib.kubernetes import kube_client, pod_generator, pod_launcher webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 25, in webserver_1 | from kubernetes import watch, client webserver_1 | ModuleNotFoundError: No module named 'kubernetes'

Pontias

Try: pip install airflow['kubernetes'] – Shebat

@FelipeFB - did u resolve? – Ammadis

I just restart like @Kalil posted! – Pontias

i got same error upgrading from 1.10.3 to 1.10.4 but that pip suggestion does not fix it :( – Ammadis

Answer 1

Run the following

pip install apache-airflow[kubernetes]

Restart Airflow webserver and scheduler after that.

Billbillabong

i got same error upgrading from 1.10.3 to 1.10.4 but that pip suggestion does not fix it :( – Ammadis

Answer 2

It tries to import module named kubernetes. Try that:

pip install kubernetes

Gemeinschaft

I import this too... but nothing changed! – Pontias

Answer 3

try any of these imports. one of these got deprecated, so you might have a version that is no longer supported.

    #from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
    from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator

Menton

The second seems import one on my airflow version Broken DAG: [/airflow-dags/dags/dbt.py] Traceback (most recent call last): File "/airflow-dags/dags/dbt.py", line 3, in <module> from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator File "/home/airflow/.local/lib/python3.9/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 25, in <module> from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator # noqa ModuleNotFoundError: No module named 'airflow.providers.cncf.kubernetes' – Teddman

Answer 4

If you have both kubernetes and apache-airflow-providers-cncf-kubernetes installed, then there's a chance you are running into this error to another kubernetes.py local file that exists somewhere in your PYTHONPATH.

If this is the case, then simply renaming that file should do the trick.

Legionnaire