google.cloud namespace import error in __init__.py

Question

I have read through at least a dozen different stackoverflow questions that all present the same basic problem and have the same basic answer: either the module isn't installed correctly or the OP is doing the import wrong.

In this case, I am trying to do from google.cloud import secretmanager_v1beta1.

It works in my airflow container when I run airflow dags or if I run pytest tests/dags/test_my_dag.py. However, if I run cd dags; python -m my_dag or cd dags; python my_dag.py I get this error:

from google.cloud import secretmanager as secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)

I can add from google.cloud import bigquery in the line right above this line and that works OK. It appears to literally just be a problem with this particular package.

Why does it matter if pytest and airflow commands succeed? Because, I have another environment where I am trying to run dataflow jobs from the command-line and I get this same error. And unfortunately I don't think I can bypass this error in that environment for several reasons.

UPDATE 6

I have narrowed down the error to an issue with the google.cloud namespace and the secretmanager package within that namespace in the __init__.py file.

If I add from google.cloud import secretmanager to airflow/dags/__init__.py and then try to run python -m dags.my_dag.py, I receive this error but with a slightly different stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/workspace/airflow/dags/__init__.py", line 3, in <module>
    from google.cloud import secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)

OLD INFORMATION

I am 95% sure that it's still a path problem and that pytest and airflow are fixing something I'm not aware of that isn't handled when I try to manually run the python script.

Things I have tried:

cd /airflow; python setup.py develop --user
cd /airflow; pip install -e . --user
cd /airflow/dags; pip install -r requirements.txt --user

UPDATE

As per requests in the comments, here are the contents of requirements.txt:

boto3>=1.7.84
google-auth==1.11.2
google-cloud-bigtable==1.2.1
google-cloud-bigquery==1.24.0
google-cloud-spanner==1.14.0
google-cloud-storage==1.26.0
google-cloud-logging==1.14.0
google-cloud-secret-manager>=0.2.0
pycloudsqlproxy>=0.0.15
pyconfighelper>=0.0.7
pymysql==0.9.3
setuptools==45.2.0
six==1.14.0

And I accidentally omitted the --user flags from the pip and python installation command examples above. In my container environment everything is installed into the user's home directory using --user and NOT in the global site-packages directory.

UPDATE 2

I've added the following code to the file that is generating the error:

print('***********************************************************************************')
import sys
print(sys.path)
from google.cloud import secretmanager_v1beta1 as secretmanager
print('secretmanager.__file__: {}'.format(secretmanager.__file__))

From airflow list_dags:

['/home/app/.local/bin', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/workspace/airflow', '/usr/local/lib/python3.7/site-packages', '/workspace/airflow/dags', '/workspace/airflow/config', '/workspace/airflow/plugins']
secretmanager.__file__: /home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg/google/cloud/secretmanager_v1beta1/__init__.py

From python my_dag.py:

['/workspace/airflow/dags', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/icentris_ml_airflow-0.0.0-py3.7.egg', '/usr/local/lib/python3.7/site-packages']

UPDATE 3 tree airflow/dags

airflow/dags
├── __init__.py
├── __pycache__
│   ├── __init__.cpython-37.pyc
│   ├── bq_to_cs.cpython-37.pyc
│   ├── bq_to_wrench.cpython-37.pyc
│   ├── fetch_cloudsql_tables-bluesun.cpython-37.pyc
│   ├── fetch_cloudsql_tables.cpython-37.pyc
│   ├── fetch_app_tables-bluesun.cpython-37.pyc
│   ├── fetch_app_tables.cpython-37.pyc
│   ├── gcs_to_cloudsql.cpython-37.pyc
│   ├── gcs_to_s3.cpython-37.pyc
│   ├── lake_to_staging.cpython-37.pyc
│   ├── schedule_dfs_sql_to_bq-bluesun.cpython-37.pyc
│   ├── schedule_dfs_sql_to_bq.cpython-37.pyc
│   ├── app_to_bq_initial_load-bluesun.cpython-37.pyc
│   ├── app_to_lake-bluesun.cpython-37.pyc
│   └── app_to_lake.cpython-37.pyc
├── bq_to_wrench.py
├── composer_variables.json
├── my_ml_airflow.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── lake_to_staging.py
├── libs
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-37.pyc
│   │   ├── checkpoint.cpython-37.pyc
│   │   └── utils.cpython-37.pyc
│   ├── checkpoint.py
│   ├── io
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   └── __init__.cpython-37.pyc
│   │   └── gcp
│   │       ├── __init__.py
│   │       ├── __pycache__
│   │       │   ├── __init__.cpython-37.pyc
│   │       │   └── storage.cpython-37.pyc
│   │       └── storage.py
│   ├── shared -> /workspace/shared/
│   └── utils.py
├── requirements.txt
├── table_lists
│   └── table-list.json
└── templates
    └── sql
        ├── lake_to_staging.contacts.sql
        ├── lake_to_staging.orders.sql
        └── lake_to_staging.users.sql

11 directories, 41 files

UPDATE 4

I tried fixing it so that sys.path looked the same when running python dags/my_dag.py as it does when running airflow list_dags or pytest test_my_dag.py.

Still get the same error.

Looking at a more recent version of documentation, I noticed that you should be able to just do from google.cloud import secretmanager. Which gave me the same result (works with airflow and pytest, not when trying to run directly).

At this point, my best guess is that it has something to do with namespace magic, but I'm not sure?

Chagres

What's the content of your requirement.txt file? – Hogarth

Try pip install -U google-cloud-secret-manager. Add google-cloud-secret-manager==0.2.0 to requirements.txt. – Newsreel

Please leave a comment when you add additional information, we aren't notified of edits. – Hogarth

In the one that works, what is secretmanager.__file__? In the one that doesn't, what is sys.path? – Hogarth

@DustinIngram Update posted as per your comment. Thanks. – Chagres

OK, so the module is in '/home/app/.local/lib/python3.7/site-packages', which is on the path. This means that something of higher priority on the path is providing a google.cloud namespace. Of the higher priority paths, the only one that isn't included in the working path is '/workspace/airflow/dags'. What's in that directory? The output of tree /workspace/airflow/dags would be helpful. – Hogarth

@DustinIngram updated with output of tree command. – Chagres

Answer 1

After much trial and error, the issue is that currently one cannot import secretmanager from the google.cloud namespace if one has not previously imported another package from google.cloud.

Ex.

mod.py

from google.cloud import secretmanager # Fails with error

mod2.py

from google.cloud import bigquery
from google.cloud import secretmanager # Works because the first import initialized the namespace

Chagres

Answer 2

It have to be installed via terminal: pip install google-cloud-secret-manager Because package name is not secretmanager but google-cloud-secret-manager

Musetta

What worked for me in a Jupyter setting was pip3 install --user google-cloud-secret-manager – Amphicoelous

Answer 3

Similar to Noah's answer, this fixed the issue for me without having to import an unneeded module:

import google.cloud.secretmanager as secretmanager

Domett

Answer 4

I'm using Python 3.8.2 and google-cloud-secret-manager 2.7.1. The following line fixed the issue for me:

from google.cloud import secretmanager_v1beta1 as secretmanager

Martinson