How can I resolve "HttpError: Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object., 401" error?

Question

The problem: I'm trying to read in a .gz JSON file that is stored in one of my project's cloud storage bucket using a google colab python notebook and I keep getting this error:

HttpError: Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object., 401

My code:

fs = gcsfs.GCSFileSystem(project='my-project')
with fs.open('bucket/path.json.gz') as f:
    gz = gzip.GzipFile(fileobj=f) 
    file_as_string = gz.read()
    json_a = json.loads(file_as_string)

I've tried all of these authentication methods and still get the same 401 error :

!gcloud auth login
!gcloud auth list
!gcloud projects list
!gcloud config set project 'myproject-id'

---

from google.colab import auth
auth.authenticate_user()

---

!gcloud config set account 'my GCP email'

---

!gcloud auth activate-service-account

---

!gcloud auth application-default login

---

!gsutil config

---

!gcloud config set pass_credentials_to_gsutil false

---

!gsutil config -a

I've also set my GCP IAM permissions to:

• Editor
• Owner
• Storage Admin
• Storage Object Admin
• Storage Object Creator
• Storage Object Viewer
• Storage Transfer Admin

Accalia

Start with gcloud init, then gcloud auth application-default login. If that does not work show the commands and the results in your question. – Incorporable

Answer 1

It's not entirely clear from your question but:

1. gcloud and Google SDKs both use Google's identity|auth platform but they don't share state. You usually (!) can't login using gcloud and expect code using an SDK to be authenticated too
2. @john-hanley correctly points out that one (often confusing) way to share state between gcloud and code using Google SDKs is to use gcloud auth application-default login. However, this only works because gcloud writes its state locally and code using Google SDKs when running as the same user on the same host, will be able to access this state. I think (!?) this won't work with browser-based collab
3. I'm unfamiliar with gcsfs.GCSFileSystem but, it is not a Google SDK. Unless its developers have been particularly thoughtful, it won't be able to leverage authentication done by the Google SDK using auth.authenticate_user().

So...

I think you should:

1. Ensure that your user account ([email protected] or whatever) has roles/storage.objectAdmin (or any predefined role that permits storage.objects.get).
2. Use google.collab.auth and auth.authenticate_user() to obtain credentials for the browser's logged-in user (i.e. [email protected]).
3. Use a Google Cloud Storage library, e.g. google-cloud-storage to access the GCS object. The Google library can leverage the credentials obtained in the previous step.

Update

Here's an example.

NOTE: it use the API Client Library rather than the Cloud Client Library but these are functionally equivalent.

Klehm