Airflow on Cloud Composer Cannot import module

Question

I'm running a DAG test_dag.py which is structured in the following way in my Google Cloud Storage Bucket.

gcs-bucket/
    dags/
        test_dag.py
        dependencies/
            __init__.py
            dependency_1.py
            module1/
                __init__.py
                dependency_2.py

Airflow detects the DAG, test_dag.py, which tries to import from depencies/dependency_1.py, (which imports successfully) and dependencies/module1/dependency_2.py which gives the error Broken DAG: [/home/airflow/gcs/dags/test_dag.py] module 'dependencies' has no attribute 'module1'.

The line causing this is from dependencies.module1 import dependency_2.

This seems to indicate to me that Cloud Composer is unable to import from a subdirectory within dependencies/, and when I look at their dependencies documentation here, the example they give is only one directory level down from /dags (and is only 1 file rather than being a full python package).

Here is the weird part though -- it runs successfully when I run this locally in Airflow (not on Cloud Composer). So I'm at a loss for why my imports would work locally but not on Cloud Composer.

I've also tried importing everything from within my __init__.py files, which gives me the same attribute error, and moving my dependencies a level up into gcs-bucket/ where they can't seems to be found at all.

When I print out __file__ from with my DAG I get /home/airflow/gcs/dags/test_dag.py and when I print sys.path I get:

['/usr/local/bin', '/opt/python3.6/lib/python36.zip', '/opt/python3.6/lib/python3.6', '/opt/python3.6/lib/python3.6/lib-dynload', '/opt/python3.6/lib/python3.6/site-packages', '/usr/local/lib/airflow', '/home/airflow/gcs/dags', '/etc/airflow/config', '/home/airflow/gcs/plugins']

I'm totally at a loss here, any help would be much appreciated. Thank you.

EDIT: It seems that Cloud Composer does not like when dependencies try to import other dependencies (see comments below). Wondering if there is some way around this?

Quinsy

Is it possible for you to make your dependencies code available? – Odin

It's a bit tough to do that to be honest, this is a minimum viable example but in reality the dependencies directory is a huge package of interconnected pieces. If there is a specific part you'd like to see I can copy it over. The reason I don't think it necessarily has to do with that code though, is that it runs fine when I run it locally, so I don't think the way the imports are structured is the issue here – Quinsy

I tried reproducing your problem but it worked fine for me. I could import both a module in a one level directory and in a 2 level directory – Odin

Can you check if all the dependencies and import names are right? – Odin

I’ll double check everything but can I ask two one follow ups to that — are you able to import dependency 1 from within dependency 2? And more importantly did you have to include a ‘setup.py’ file or anything else that I might have forgotten to make the imports work? – Quinsy

I just don’t understand how my imports can be wrong in cloud composer but it’s able to run locally on Airflow :/ – Quinsy

And to confirm, you put your dependency packages with /dags, correct? – Quinsy

The dependency directory is on my dags/ folder. Did you try to import dependency 1 inside dependency 2? I imported both on the dag. If you tried to import dependency 1 inside dependency 2 I think it changed the situation. If you confirm I can test it too – Odin

Yes I have done that I would be curious if it works on your end – Quinsy

I tested it now and had the same problem as you. Can this structure of dependencies be changed? – Odin

That's a relief. I'm glad to hear that I'm not totally crazy. The package is honestly so interwoven at this point that it would be a huge job to restructure it entirely. Do you happen to know of any alternative ways that this can work? Maybe through plugins or some other way of importing the dependency? – Quinsy

Can you confirm that you are trying to import dependence 1 within dependence 2? Because in your question you said "The line causing this is from dependencies.module1 import dependency_2." – Odin

@Quinsy - I just wondering if you resolve this issue? I'm facing the exact same problem :( – Stogy

@Stogy I did not. I was never able to get it to work – Quinsy

Anyone solved that? Seems impossible Cloud composer Is unable to del with custom modules – Zitazitah

Answer 1

Could you add __init__.py under dags/ folder and give a try?

Mayramays