pylint can't find google.cloud

Question

I've installed the Google Cloud SDK and want the code I'm writing to pass pylint. Unfortunately any time I import anything from google.* I get an error:

E: 10, 0: No name 'cloud' in module 'path/to/my/current/module.google' (no-name-in-module)
E: 10, 0: Unable to import 'google.cloud' (import-error)

Versions:

$: pylint --version
pylint 1.7.0, 
astroid 1.5.0
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4]

If I put a hook in pylint to print out the sys path I get nothing interesting. The google-cloud-sdk is in /usr/local/lib/python2.7/dist-packages so it should be able to find it.

['/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/pylint-1.7.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/backports.functools_lru_cache-1.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/configparser-3.5.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/singledispatch-3.4.0.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/editdistance-0.3.1-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/mccabe-0.6.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/astroid-1.5.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/wrapt-1.10.10-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/lazy_object_proxy-1.2.2-py2.7-linux-x86_64.egg', '/usr/lib/python2.7/dist-packages']

Does anyone know why it is looking in my local path for the "google" module and how I can fix it?

Updates with more detail about my environment:

The Google Cloud SDK modules in question are located at:

/usr/local/lib/python2.7/dist-packages/google

if I ls that directory it shows:

api auth cloud gapic gax iam ...

and if I follow down those paths all the modules are in the places I'd expect them to be given the import statements.

However, there are no __init__.py files, which makes me think that they're using implicit namespace packages. So it seems like the relevant question here is: how to make pylint recognize an implicit namespace package? The docs say it should "just work":

https://docs.pylint.org/en/latest/user_guide/run.html?highlight=re

For the record, the same problem shows up when using mypy.

Peng

Maybe Pylint issue. – Isodynamic

Answer 1

Update (2):

It turns out I may have a solution in a Google App Engine project I am working on:

def fixup_paths(path):
    """Adds GAE SDK path to system path and appends it to the google path
    if that already exists."""
    # Not all Google packages are inside namespace packages, which means
    # there might be another non-namespace package named `google` already on
    # the path and simply appending the App Engine SDK to the path will not
    # work since the other package will get discovered and used first.
    # This emulates namespace packages by first searching if a `google` package
    # exists by importing it, and if so appending to its module search path.
    try:
        import google
        google.__path__.append("{0}/google".format(path))
    except ImportError:
        pass

    sys.path.insert(0, path)

# and then call later in your code:
fixup_paths(path_to_google_sdk)

I really hope that helps!

Update (1):

Here is a possible solution I found: No module named cloud while using google.cloud import bigquery

In that post, the problem is a python 2.7 script is trying to import from google.cloud, and is failing due to the implicit namespace issue. The solution is:

Fortunately in Python 2.7 you can fix this easily by avoiding implicit imports. Just add this to the top of your file:

from __future__ import absolute_import

Original Answer:

Could you please post your import statement, the full path to the google/cloud.py or whatever you are trying to import, the contents of the google module from which you are trying to import, and the contents of your current module please?

Without that information it is hard to explore this issue. My initial hunch is that you are missing an __init__.py, either in your current module or in the google module. As shown in this post, your code can run fine without __init__.py in your module, but if pylint runs from outside your module then it may not see your module properly without that.

The reason I suspect a missing __init__.py in the google module is because in PEP 420 in python 3, they dropped the requirement for __init__.py, and I have seen questions resulting from people trying to import from google modules which are designed for python 3, and which omit the __init__.py that is required in python 2.

If none of those are the case, then maybe the module you are trying to import is located a few directories into the google-cloud-sdk directory that is located at /usr/local/lib/python2.7/dist-packages, as opposed to being an immediate child of the google-cloud-sdk directory, and the python path your code normally sees accounts for this, while the python path that pylint sees does not.

Penrod

Sure. Here is an example import statement: from google.cloud import storage – Peng

Sorry, I keep accidentally pressing enter :) I think you've hit the nail on the head with your comment about missing init.py, and importing libraries designed with Python3 in mind to Python2. I am working in 2.7 but yes, the Google Cloud SDK uses the PEP420 standard. Let me update the question to be more specific. (For what it's worth, I see the same problem when running mypy--can't find the module.) – Peng

Just to confirm, you do have an __init__.py or you don't in your project's directory? – Penrod

I do have init.py in my project's directory. If I add "from future import absolute_import" I get a different set of errors: E: 14, 0: No name 'cloud' in module 'google' (no-name-in-module) E: 14, 0: Unable to import 'google.cloud' (import-error) E: 15, 0: No name 'cloud' in module 'google' (no-name-in-module) E: 15, 0: Unable to import 'google.cloud' (import-error) which might be an improvement? – Peng

According to this link, you may find success if you delete all google-related packages from dist-packages and reinstall them. Or maybe you have another google package somewhere on your path that DOES have an __init.py__, which would throw off the implicit namespace. The advantage of implicit namespaces is you can load from many directories that have the same name as if they were one package, but if it finds that __init__.py, it stops looking for more packages. – Penrod

No luck :( Interestingly, flake8 works fine! So I might just switch to that and figure out mypy later. I'll keep chipping away at it. I'm going to award the bounty to you because you tried a lot of things but I won't mark as answered since no fix worked. I'll update if I find anything interesting. – Peng

Brendan, how would one use the code in your most recent update? I run pylint from the command line and you can't put Python code in pylintrc. – Longanimity

Answer 2

I ran into the same issue with Google's Cloud Shell Editor. To fix this, you just need to update .pylintrc per below.

1. Determine the location of the google package:

>>> from google.cloud import language_v1
>>> print(language_v1)
<module 'google.cloud.language_v1' from '/usr/local/lib/python3.7/dist-packages/google/cloud/language_v1/__init__.py'>

2. Specify the /usr/local/lib/python3.7/dist-packages/ path in your ~/.pylintrc.

[MASTER]
init-hook='import sys; sys.path.append("/usr/local/lib/python3.7/dist-packages/")'

Thanks to this tip!

Thrave