Wrong dependency in Google Cloud SDK for google-auth?

Question

i am using google cloud storage and google cloud datastore api (locally, for now), in a Standard Environment app.

I am having a very weird behaviour trying to running my tests (with pytest): i have discovered that, during tests, the dev_appserver.fix_sys_path command run by pytest-beds that i am using (https://github.com/kaste/pytest-beds/blob/master/testbeds/plugin.py#L35) puts the bundled google-cloud libraries before my custom versions:

/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/antlr3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib
/Users/username/google-cloud-sdk/platform/google_appengine/lib/ipaddr
/Users/username/google-cloud-sdk/platform/google_appengine/lib/yaml-3.10
/Users/username/google-cloud-sdk/platform/google_appengine/lib/rsa
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules
/Users/username/google-cloud-sdk/platform/google_appengine/lib/httplib2
/Users/username/google-cloud-sdk/platform/google_appengine/lib/oauth2client_devserver
/Users/username/google-cloud-sdk/platform/google_appengine/lib/six-1.9.0
/Users/username/google-cloud-sdk/platform/google_appengine
/Users/username/google-cloud-sdk/platform/google_appengine/lib/simplejson
/Users/username/google-cloud-sdk/platform/google_appengine/lib/django-1.4
/Users/username/google-cloud-sdk/platform/google_appengine/lib/endpoints-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/jinja2-2.6
/Users/username/google-cloud-sdk/platform/google_appengine/lib/protorpc-1.0
/Users/username/google-cloud-sdk/platform/google_appengine/lib/PyAMF-0.6.1
/Users/username/google-cloud-sdk/platform/google_appengine/lib/markupsafe-0.15
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webob-1.2.3
/Users/username/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2
/Users/username/dev/project
/Users/username/.virtualenvs/project/bin
/Users/username/dev/project/app
/Users/username/.virtualenvs/project/lib/python27.zip
/Users/username/.virtualenvs/project/lib/python2.7
/Users/username/.virtualenvs/project/lib/python2.7/plat-darwin
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac
/Users/username/.virtualenvs/project/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/Extras/lib/python
/Users/username/.virtualenvs/project/lib/python2.7/lib-tk
/Users/username/.virtualenvs/project/lib/python2.7/lib-old
/Users/username/.virtualenvs/project/lib/python2.7/lib-dynload
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Users/username/.virtualenvs/project/lib/python2.7/site-packages

This causes the import of google.auth.crypt.base module import to fail, because it does not exist in the package version shipped with google-cloud-sdk, while i can see that version 1.3.0 on github, that is apparently the same version of the one shipped with the sdk, does.

Same for pyasn1_modules package: version 0.2.1 existing, shipped 0.11

The import errors I am having are:

from google.auth.crypt import base
ImportError: cannot import name base

and:

from pyasn1_modules import pem
ImportError: cannot import name pem

Last, but not least, the environment: MACOSX.

Google Cloud SDK 184.0.0
app-engine-python 1.9.65
app-engine-python-extras 1.9.63
beta 2017.09.15
bq 2.0.28
cloud-datastore-emulator 1.3.0
core 2018.01.05
gcloud
gsutil 4.28

UPDATE: apparently, i was using the google.cloud.storage while instead i should be using, according to the examples for standard environment (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/storage/appengine-client/main.py#L23), the package cloudstorage. I will change the code, try and update the post here.

UPDATE 2: i have created a specific github repo that should replicate the problem, just in case. Here it is:

https://github.com/brunoripa/gae_import_error

Mychael

I'm not a flex env user, so I'm not entirely sure about this, but the behaviour could be because the dev_appserver.py is designed for the standard env only, doesn't support flex env. Try to run your tests without using it. – Fallen

Well, since i am using appengine machinery, disabling the plugin (by running the tests with --no-gae) is not an option. Very weird behaviour :/ – Mychael

Hm, flex env is very different from the standard env in "using appengine machinery", I'd even dare say that it's not actually using it. A flex env app is a standalone app, a standard env app is just some python code executed by the GAE "machinery" (be it the real one or the dev_server.py one). – Fallen

Ok, sorry, my bad. The app i am talking about is a standard environment one (was confused with the flex one, which is working nicely). All i am describing is happening on standard environment (going to update the post). Sorry – Mychael

Some time passed, so i reply the comment here. The author of pytest-beds replied to the bug i filed, which was known to him. Find it here: github.com/kaste/pytest-beds/issues/4 – Mychael

Answer 1

If the problem is entirely on localhost during testing, you can run: pip install cryptography

so that the Google libraries won't even need pem to begin with.

Epithelium

Answer 2

We typically add libs to remote API with vendor system.

from google.appengine.ext import vendor
vendor.add('lib')

Module pyasn1_modules gets installed as a dependency when you install google-cloud and I am not sure of the reason behind it but, in remote API, I have observed it misses pem.py. The simple workaround would be:

# within remote API console

>>> import sys
>>> sys.modules.pop('pyasn1_modules')

<module 'pyasn1_modules' from '/Users/pjamkhande001/google-cloud-sdk/platform/google_appengine/lib/pyasn1_modules/pyasn1_modules/__init__.pyc'>

About your other issue of not being able to import from google.auth.crypt import base, you are required to gcloud auth login.

gcloud auth application-default login restores your default authentication credentials but if you want to login to the application as user, you have to gcloud auth login. Hope this helps.

Wilscam