No module named cloud while using google.cloud import bigquery

Question

i have built an app engine application to load data into bigquery table using google app engine launcher but when I run it on local host or on the cloud i get the No module named cloud while using google.cloud import bigquery error message in log file. I have installed the google cloud client library but it is still giving me the same error. please see below the code I am using

---main.py file contains

import argparse
import time
import uuid

from google.cloud import bigquery

def load_data_from_gcs(dataset_name, table_name, source):
    bigquery_client = bigquery.Client()
    dataset = bigquery_client.dataset(dataset_name)
    table = dataset.table(table_name)
    job_name = str(uuid.uuid4())

    job = bigquery_client.load_table_from_storage(
        job_name, table, source)

    job.begin()

    wait_for_job(job)

    print('Loaded {} rows into {}:{}.'.format(
        job.output_rows, dataset_name, table_name))


def wait_for_job(job):
    while True:
        job.reload()
        if job.state == 'DONE':
            if job.error_result:
                raise RuntimeError(job.error_result)
            return
        time.sleep(1)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('Test')
    parser.add_argument('mytable')
    parser.add_argument('gs://week/geninfo.csv')

    args = parser.parse_args()

    load_data_from_gcs(
        args.dataset_name,
        args.table_name,
        args.source)

--app.yaml file contains the following code

application: mycloudproject
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /favicon\.ico
  static_files: favicon.ico
  upload: favicon\.ico

- url: .*
  script: main.app

Please let me know what is missing or if I am doing something wrong here?

Shama

Possible duplicate of app engine project to load data into bigquery not working – Centralism

Can you share the output of python -v? This might help you see what modules you have loaded and from where. – Maeve

the output has too many characters on it and doesn't allow me to post it in here. is there any other way to share my file? – Shama

hi i have managed to save the output using the pastebin and is available through this link link – Shama

Answer 1

This can be a bit tricky. Google Cloud uses the new Python namespace format (if you look at the source you'll notice that there's no __init__.py in the directory structure).

This was changed in Python 3.3 with PEP-420

Fortunately in Python 2.7 you can fix this easily by avoiding implicit imports. Just add this to the top of your file:

from __future__ import absolute_import

Hope that helps.

Subcartilaginous

Answer 2

Find the directory containing google/cloud/..., and add that directory to the PYTHONPATH so that python can find it. See this post for details on how to add to PYTHONPATH. It outlines two common ways to do it:

Here's how to do it with a bash command:

export PYTHONPATH=$PYTHONPATH:/<path_to_modules>

Or you could append it to the path in your script:

# if the google/ directory is in the directory /path/to/directory/
path_to_look_for_module = '/path/to/directory/'
import sys
if not path_to_look_for_module in sys.path:
    sys.path.append(path_to_look_for_module)

If that doesn't work, here is some code I found in one of my projects for importing Google Appengine modules:

def fixup_paths(path):
    """Adds GAE SDK path to system path and appends it to the google path
    if that already exists."""
    # Not all Google packages are inside namespace packages, which means
    # there might be another non-namespace package named `google` already on
    # the path and simply appending the App Engine SDK to the path will not
    # work since the other package will get discovered and used first.
    # This emulates namespace packages by first searching if a `google` package
    # exists by importing it, and if so appending to its module search path.
    try:
        import google
        google.__path__.append("{0}/google".format(path))
    except ImportError:
        pass

    sys.path.insert(0, path)

# and then call later in your code:
fixup_paths(path_to_google_sdk)
from google.cloud import bigquery

Microsporophyll

Answer 3

It looks like you are trying to use the Cloud Datastore client library in a Google App Engine's standard environment. As documented in Google's documentation you should not be doing this. Instead, either use the NDB Client Library or do not use the standard environment.

Antalya

Answer 4

Are you sure you've updated to the latest version of the library? The version installed by pip may be out of date. Previously, the module was imported as:

from gcloud import bigquery

If that works, you're running an older version. To install the latest, I'd recommend pulling from the master in the github project.

Overijssel

Yes, I installed the library yesterday and this still doesn't work. I have also changed the import statement to the one you have suggested from gcloud import bigquery but the it still doesn't work and giving me an error message ImportError: No module named gcloud – Shama