gsutil not working if calling from pycharm with python and subprocess, "cannot import name _common"

Question

I have written a python script to automatize some gsutil operations. gsutil works fine if I run it by command line. But if I translate the same command with subprocess in python I get an error:

BUCKET_NAME = 'datastore-backup'
FOLDER_NAME = 'my_folder'

gcs_path = os.path.join('gs://', BUCKET_NAME, FOLDER_NAME)

files = subprocess.check_output(['gsutil', 'ls', gcs_path], stderr=sys.stdout)
print(files)

I get this error

Traceback (most recent call last):
  File "/Users/dario/Downloads/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 13, in <module>
    import bootstrapping
  File "/Users/dario/Downloads/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 32, in <module>
    import setup  # pylint:disable=g-import-not-at-top
  File "/Users/dario/Downloads/google-cloud-sdk/bin/bootstrapping/setup.py", line 55, in <module>
    from googlecloudsdk.core import properties
  File "/Users/dario/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 34, in <module>
    from googlecloudsdk.core.util import times
  File "/Users/dario/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/util/times.py", line 55, in <module>
    from dateutil.tz import _common as tz_common
ImportError: cannot import name _common

any help

Subtreasury

I tried to run your code in the Google Cloud Shell but couldn't replicate the error. It seems that it has something to do with your Python environment and not with Google Cloud Platform or gsutil command. I've seen this error appear when running the code in PyCharm. Could you confirm if you are indeed trying to run it in PyCharm? If so, check this, it might be helpful. – Tyranny

yes i am running this in pycharm..... boxed were already ticked as per solution proposed, no luck... – Subtreasury

I would recommend to change the tags of your question as this is not a Google Cloud Platform issue as long as the gsutil command runs without errors from your command line. This way, your question will reach people more suited to answer it. You could also search/post in the PyCharm community here – Tyranny

Possible duplicate of ImportError when running dev_appserver.py from pycharm – Dome

Answer 1

I have solved this problem for good. The source of the issue is when you call either subprocess.check or os.system(, the gsutil cli would use the wrong python.

In my case gsutil uses a system python binary that is 2.7. I uncovered this issue by inserting a print(tz) statement in /Users/dario/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/util/times.py. And it turns out that it is loading a Python2.7/site-packages version of dateutil.

So to fix this, you can specify the cloudsdk python binary via the following

export CLOUDSDK_PYTHON=<what-ever-python-distro-you-want>

pyCharm is a wonderful tool second to non. Hope this makes it easy for some to use the built-in runner/debugger of this powerful IDE.

Heideheidegger

Where did you specify that command? In the terminal? – Technocracy