Google App Engine: ImportError: No module named appengine.ext

Question

I am trying to write a test for my GAE programme which uses the datastore. Following Google's Documentation, I see that I should be adding the path to my SDK into my PYTHONPATH. I did this using:

import sys
sys.path.remove('/usr/local/lib/python2.7/dist-packages')    # Has a 'google' module, which I want to be sure isn't interfering.
sys.path.insert(1,'/home/olly/google-cloud-sdk/platform/google_appengine')
sys.path.insert(1, '/home/olly/google-cloud-sdk/platform/google_appengine/lib/yaml/lib')

Then when the file is run:

Traceback (most recent call last):
 File "myapp_tests.py", line 20, in <module>
    from google.appengine.ext import ndb
ImportError: No module named appengine.ext

I have installed the SDK in the location above, and looking in /home/olly/google-cloud-sdk/platform/google_appengine/ I found the google folder, which has an __init__.py in it, along with appengine. Basically, the folder structure looks good to me, with them all being named correctly and having __init__.py files.

In an interactive console, after running the commands above, I found that I could run:

import google

no problem, but when I tried

import google.appengine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named appengine

It was my understanding that having the __init__.py() files in the directories meant that they could be imported as above. I also did a sudo find / --name "google", and the only thing that showed up that is also in my PYTHONPATH was the /usr/local/lib/python2.7/dist-packages, which I explicitly removed, and also inserted the rest of my paths in front of anyway.

I tried using GAE's own method of:

import dev_appserver
dev_appserver.fix_sys_path()

which added a whole lot of paths to sys.path, but still didn't help me make it work.

I also found that when I add '/home/olly/Servers/google_appengine/google' to my paths, I can run:

import appengine.ext

but running:

from appengine.ext import ndb

causes:

Traceback (most recent call last):
  File "booking_function_tests.py", line 16, in <module>
     from appengine.ext import ndb
  File "/home/olly/Servers/google_appengine/google/appengine/ext/ndb/__init__.py", line 7, in <module>
     from tasklets import *
  File "/home/olly/Servers/google_appengine/google/appengine/ext/ndb/tasklets.py", line 69, in <module>
     from .google_imports import apiproxy_stub_map
  File "/home/olly/Servers/google_appengine/google/appengine/ext/ndb/google_imports.py"    , line 11, in <module>
     from google3.storage.onestore.v3 import entity_pb
ImportError: No module named google3.storage.onestore.v3

Am I missing something really obvious? How should I go about importing ndb?

EDIT: I'm running the latest SDK (1.9.34), but I have the following code in my google_imports.py:

try:
  from google.appengine.datastore import entity_pb
  normal_environment = True
except ImportError:
  try:
    from google3.storage.onestore.v3 import entity_pb
    normal_environment = False
  except ImportError:
    # If we are running locally but outside the context of App Engine.
    try:
      set_appengine_imports()
      from google.appengine.datastore import entity_pb
      normal_environment = True
    except ImportError:
      raise ImportError('Unable to find the App Engine SDK. '
                        'Did you remember to set the "GAE" environment '
                        'variable to be the path to the App Engine SDK?')

Also, google.__path__ gives me the '/usr/local/lib/python2.7/dist-packages' path which I thought I removed earlier. Here is an excerpt of how I'm removing it:

import sys
sys.path.insert(1, '/home/olly/Servers/google_appengine')
sys.path.insert(1, '/home/olly/Servers/google_appengine/lib/yaml/lib')
sys.path.remove('/usr/local/lib/python2.7/dist-packages')

import google
print google.__path__
print sys.path


['/usr/local/lib/python2.7/dist-packages/google']
['/home/olly/Servers/google_appengine/myapp', '/home/olly/Servers/google_appengine/lib/yaml/lib', '/home/olly/Servers/google_appengine/google', '/home/olly/Servers/google_appengine', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode']

So my sys.path is updated, but import google seems to still be importing from the no-longer-there path, which would be the crux of my problem I guess. Do I need to reload the path or something?

Tremain

After the successful import google, what's the output of print google.__file__? – Payee

AttributeError: 'module' object has no attribute 'file'. path on the other hand is giving me the '/usr/local/lib/python2.7/dist-packages', which I though I removed. This is probably the source of my problems. Although when I print sys.path, it doesn't show up in there... – Tremain

For me, print google.__ file __ in the dev server's remote shell displays the path in my local filesystem. I'd strongly recommend following @billprin's advice and running your app in a virtualenv to avoid conflicts with anything on your machine. FWIW my test runner inserts the path to the sdk at the front of sys.path and then calls dev_appserver.fix_sys_path() and that works well. – Payee

In the end I just pip uninstall protobuf, which stopped the conflict for now. I don't particularly want to use a virtualenv at the moment, but will think about it. – Tremain

Answer 1

I run into these problems a lot less by always running inside a virtualenv.

I agree with snakecharmerrb you should get print google.__file__ or google.__path_ to figure out exactly what you're importing.

This snippet might also solve your problem:

import google

gae_dir = google.__path__.append('/path/to/appengine_sdk//google_appengine/google')
sys.path.insert(0, gae_dir) # might not be necessary

import google.appengine # now it's on your import path`

Convalescent

Answer 2

Which version of app engine SDK are you using? I am using the latest SDK (1.9.34). I find the below snippet in my ~/google_appengine/google/appengine/ext/ndb/google_imports.py file

try:
    from google3.storage.onestore.v3 import entity_pb
    normal_environment = False
except ImportError:
    # If we are running locally but outside the context of App Engine.
    try:
      set_appengine_imports()
      from google.appengine.datastore import entity_pb
      normal_environment = True
    except ImportError:
      raise ImportError('Unable to find the App Engine SDK. '
                    'Did you remember to set the "GAE" environment '
                    'variable to be the path to the App Engine SDK?')

But in your stack trace, after google3.storage import it doesn't seem to go inside the except clause.

So try the same code with latest SDK.

Splotch