how to import custom modules in Cloud Composer

Question

I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them.
Example : from src.kuzzle import KuzzleQuery

Structure:

• main.py
• src
  • kuzzle.py

I have imported my project folder in data storage and when i refreshed the UI of airflow composer i've got this error:

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/airflow/gcs/dags/quality-control/main.py", line 8, in <module>
    from src.kuzzle import KuzzleQuery, Tag
ModuleNotFoundError: No module named 'src' ```

Amado

did help you the answer? – Rudolfrudolfo

@RaulSaucedo I think this answer is correct for import errors in your local machine. This question is about import errors in Cloud Composer. I face the same issue... – Cool

Answer 1

GCP Composer is loading the dags folder as a python module, so you can just put your code there in a separate file or folder and it will work as usual referencing it from the dags.

dags/
  dag1.py
  util.py

Then you can reference util and import it from dag1 as:

from util import custom_function

If dags folder does not work for you composer also loads more directories like the ones for loading modules.

You can also check this question since it is also related.

Airflow dag dependencies not available to dags when running Google's Cloud Compose

Wirephoto

Answer 2

Try adding an __init__.py file inside src directory to make it a module.

Cool

Answer 3

You could check these options:

The path of the module is not correct

Probably, you would want to import a module file, but this module is not in the same directory.

For example:

Project Structure

core.py
folder_1
---module.py

We want to import the module.py

Core.py

import module.py #incorrect

Output:

ModuleNotFoundError: No module named 'module'

Core.py

import folder_1.module.py #correct

Output:

...Program finished with exit code 0

The library not Installed

If you want to import a module of a library which is not installed in your virtual environment before importing a library's module, you need to install it with the pip command. You could see this documentation:

For example, import Beautifulsoup4 library it is not installed

>>> from bs4 import BeautifulSoup
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'bs4'

Now, let's install the library and try to re-import it

Installing:

pip install beautifulsoup4
Collecting beautifulsoup4
  Using cached https://files.pythonhosted.org/packages/d1/41/e6495bd7d3781cee623ce23ea6ac73282a373088fcd0ddc809a047b18eae/beautifulsoup4-4.9.3-py3-none-any.whl
Requirement already satisfied: soupsieve>1.2; python_version >= "3.0" in /home/py/Desktop/seo_pro/seo_env/lib/python3.6/site-packages (from beautifulsoup4) (1.9.5)
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.9.3

Re-importing:

>>> from bs4 import BeautifulSoup
>>>

Setup your files as packages and modules

You need to properly set up your files as packages and modules.

Packages are a way of structuring Python’s module namespace by using “dotted module names”. For example, the module name A.B designates a submodule named B in a package named A. Just like the use of modules saves the authors of different modules from having to worry about each other’s global variable names, the use of dotted module names saves the authors of multi-module packages like NumPy or Pillow from having to worry about each other’s module names.

sound/                          Top-level package
      __init__.py               Initialize the sound package
      formats/                  Subpackage for file format conversions
              __init__.py
              wavread.py
              wavwrite.py
              aiffread.py
              aiffwrite.py
              auread.py
              auwrite.py
              ...
      effects/                  Subpackage for sound effects
              __init__.py
              echo.py
              surround.py
              reverse.py
              ...
      filters/                  Subpackage for filters
              __init__.py
              equalizer.py
              vocoder.py
              karaoke.py

You could see more information.

Rudolfrudolfo

This has nothing to do with the question. He wants to run his code in Cloud Composer. – Chambray