Google cloud storage python client AttributeError: 'ClientOptions' object has no attribute 'scopes' occurs after deployment

Question

I am using cloud storage with App Engine Flex. Out of the blue i start getting this error message after deploy succeeds

The error is happening from these lines in my flask app.

from google.cloud import storage, datastore
client = storage.Client()

File "/home/vmagent/app/main.py", line 104, in _load_db
    client = storage.Client()
  File "/env/lib/python3.6/site-packages/google/cloud/storage/client.py", line 110, in __init__
    project=project, credentials=credentials, _http=_http
  File "/env/lib/python3.6/site-packages/google/cloud/client.py", line 250, in __init__
    Client.__init__(self, credentials=credentials, client_options=client_options, _http=_http)
  File "/env/lib/python3.6/site-packages/google/cloud/client.py", line 143, in __init__
    scopes = client_options.scopes or self.SCOPE
AttributeError: 'ClientOptions' object has no attribute 'scopes'

This is something to do with breaking upgrades made to grpcio and google-api-core and google-cloud-storage packages based on numerous SO threads. However, I cant figure out where this is happening.

My requirements.txt is as follows:

setuptools>=40.3
grpcio<=1.27.2
google-api-core<1.17.0
Flask

gevent>=0.13

gunicorn>=19.7.1
numpy>=1.18.0
numpy-financial
scipy>=1.4
pvlib>=0.7
google-cloud-storage==1.28.0
google-cloud-datastore==1.12.0
google-cloud-pubsub
pandas==1.0.5

my app.yaml is as follows:

service: app-preprod
runtime: custom
env: flex
entrypoint: gunicorn -t 600 -c gunicorn.conf.py -b :$PORT main:app


runtime_config:
   python_version: 3.6
manual_scaling:
    instances: 1
resources:
    cpu: 1
    memory_gb: 4
beta_settings:
    cloud_sql_instances: xxxx:europe-west6:component-cost
   
endpoints_api_service:
  name: apipreprod-dot-xxxx.appspot.com
  rollout_strategy: managed

Looking at the release histories, some new versions of google-cloud-storage etc were released a few days ago, but i have tried to maintain the same older version number.

The ridiculous thing is that with these exact same requirements.txt, i have an identical prod app engine that is working fine --- but that i had not redeployed for a week. Obviously, no problems at all with exactly the same versions of storage and datastore to run the client from my local machine.

--EDIT-- Apparently according to

https://github.com/googleapis/google-cloud-python/issues/10471

i should just add

google-cloud-core==1.3.0 to requirements.txt This seems a workaround --- any better permanent way of ensuring this break doesnt catch me unawares?

Tetrapody

Answer 1

This is due to https://github.com/googleapis/google-cloud-python/issues/10471.

I'd recommend upgrading google-cloud-core and google-api-core to the latest versions with the bugfix.

Mythopoeic

I am very fearful of upgrading to the latest, because in the past there are breaking changes especially, in grpcio and google-api-core which force us to rollback, thats why google-cloud-core is pinned to 1.3.0. For example, this one left me high and dry - github.com/googleapis/python-api-core/issues/47 – Tetrapody

Answer 2

This error message indicates that you would need to set an attribute “scopes” for your ClientOptions object.

I would like to suggest you take a look at this documentation for the ClientOptions. The “scopes” attribute should be “OAuth access token override scopes”. You may set it to None. Please review that part of your code.

On the other hand, make sure that your properly prepared your environment to use Datastore client libraries.

Academic