gsutil ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket even though I'm loggedin in gcloud

Question

I am trying to create an internal app to upload files to google cloud. I don't want each individual user or this app to log in so I'm using a service account. I login into the service account and everything is ok, but when I try to upload it gives me this error: ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket

As you can see I am logged in with a service account and my account and(neither service or personal) works

Rapp

I had to gcloud config set pass_credentials_to_gsutil true. – Mahican

Answer 1

I had similar problem, and as always, it took me 2 hours but the solution was trivial, if only it was written somewhere... I needed to login (or authorize, what suits you) to the gsutil in addition to being authorized to the gcloud. I thought they are linked or whatever, but nah. After I ran gsutil config and authorized via the provided link (and code that I pasted back to the console), it started working for me.

Note that I was also logged in to gcloud via a service account linked to my project and having the service account .json key saved locally (see gcloud auth activate-service-account --help.

Lannylanolin

YES! you are right, not documented anywhere, they just suggest you use gcloud to authenticate!?!? – Felicitation

If you run the gcloud wrapper script (.../bin/gsutil), it should use your gcloud credentials for gsutil. If you invoke the packaged gsutil directly (.../platform/gsutil/gsutil), it won't, and you'll need to configure your credentials via gsutil config. See my answer below. – Hooded

Now This can be also done through google cloud auth gcloud auth login – Chivalric

This was the key for me gcloud auth activate-service-account pointed to my service account .json file – Spurling

I don't understand what the solution is. I have authenticated via gcloud auth login and still gcsfs does not use my account. – Magen

@Magen It is possible Google has changed the authentication system for gcloud since this answer was posted. I recently installed gcloud on a new computer, I (for a change) had now problems with not being able to get Windows to find gsutil. I think I messed around with the PATH, but I honestly can't remember all too well. – Dugger

It is wrong to say that this is not documented. If you follow Install the gcloud CLI of GCP you are asked to do this as step 7: To initialize the gcloud CLI, run gcloud init: which will open the browser for a login. I only ignored that opened browser window, thinking that it would already run by the command, but that is far from not being documented. – Simons

@questionto42standswithUkraine It has been three years since my reply, so maybe they added the info, but I can't vouch that it was there back then. :) – Dugger

Answer 2

gcloud auth login solved my issue. You need both steps:

gcloud auth login
gcloud auth application-default login

Appendage

I did a pip install gcloud. After gcloud is installed, I run gcloud auth login and I get bash: gcloud: command not found... – Gluttonous

This worked perfectly, so much simpler than the accepted answer. (and also I didn't seem to need the second line, running this in the online command prompt.) – Starks

This solution worked for me! Thanks. I tried only use "gcloud auth login", which doesn't work, after also run "gcloud auth application-default login" it finally worked – Tower

solved my issue this one here 👍 – Bunco

this solved my issue as well ! – Bemoan

this method worked for me, in my case I was authenticate in CLI but couldn't read/write on gap bucket from pandas, using this method it worked! thanks – Pippin

didn't work at all – Petterson

This is great, simple but effective. solved my 2 days firefighting issue – Lushy

If you follow Install the gcloud CLI of GCP you are asked to do this as step 7: To initialize the gcloud CLI, run gcloud init: which will open the browser for a login. If you login there, you do not need the commands of this answer anymore. Therefore, it is all about just following the GCP SDK guide step by step. – Simons

working with these tools is a nightmare – Josh

Answer 3

It happened to me because I had an incomplete initialisation while running gcloud init. I reinitialised the configuration using gcloud init command and it worked fine.

Ligialignaloes

Similar: I had to run gcloud auth list to determine I had no user logged in (a surprise to me), then gcloud auth login '[email protected]' – Amperage

If you follow Install the gcloud CLI of GCP you are asked to do this as step 7: To initialize the gcloud CLI, run gcloud init: which will open the browser for a login. I only ignored that opened browser window, thinking that it would already run by the command, not sure whether you mean that with "incomplete initialisation". – Simons

Answer 4

I can only think of a few things that might cause you to see this error:

1. Maybe you have an alias set up to a standalone installation of gsutil (which doesn't share credentials with gcloud)?
   Edit: it's also possible you're invoking the wrong gsutil entry point - make sure you're using <path-to-cloud-sdk>/google-cloud-sdk/bin/gsutil, and not <path-to-cloud-sdk>/google-cloud-sdk/platform/gsutil/gsutil. The platform path will not automatically know about your configured gcloud auth options.

2. Maybe your service account credentials have moved/are invalid now? If your boto file is referring to a keyfile path and the keyfile was moved, this might happen.

3. Maybe the gcloud boto file (that gcloud created to use with gsutil when you ran gcloud auth login) is gone. You can run gsutil version -l to see if it's shown in your config path. If gcloud's boto file is present, you should see a line similar to this:

   config path(s): /Users/Daniel/.config/gcloud/legacy_credentials/[email protected]/.boto

You can run gsutil version -l to get a bit more info and look into the possibilities above. In particular, these attributes from the output will probably be the most helpful: using cloud sdk, pass cloud sdk credentials to gsutil, config path(s), and gsutil path.

Hooded

Answer 5

Use this command to resolve some issues

gsutil config

Follow the browser to get a code, then set it in your terminal.

Cronyism

Answer 6

I had the same issue, tried to do gsutil config then it recommended me gcloud auth login which opened google in the browser. After i logged in, i could download with gsutil cp -r gs://my_bucket/Directory local_save_path the entire bucket and save it locally.

Pfennig

Answer 7

I faced the same problem. It took me two days to get this thing working. I am writing about the whole setup. please refer to step 2 for the answer to the question. FYI my OS is windows 10

Step 1: Firstly, I faced problems installing gcloud and this is what i did. The script(.\google-cloud-sdk\install.bat) which is supposed to add gcloud to the path was not working due to permission issues. I had to add the path manually in two places 1) In the system variables, to the "PATH" variable i added the path to the gcloud bin which should look like - C:\Users\774610\google-cloud-sdk\bin - in my case 2) Additionally gcloud needs python so to the "PATHEXT" variable i appended ".PY" at the end. After Performing these tasks gcloud started working.

Step 2: Even though gcloud is working, maven is not able to connect to cloud storage and the error was "401 Anonymous caller does not have storage.objects.list access to bucket" I was pretty sure i did login to my account and selected the correct project. I also tried adding environment variable as shown in this documentation "https://cloud.google.com/docs/authentication/getting-started" Nothing seemed to be working even though all the credentials were perfectly setup.

while going through the gcloud documentation I came across this command - "gcloud auth application-default login" which was exactly what i needed.

Refer here for difference between gcloud auth login and gcloud auth application default login In short what this command does is it obtains your credentials via a web flow and stores them in 'the well-known location for Application Default Credentials' and any code/SDK you run will be able to find the credentials automatically

After this, maven was successfully able to connect to google storage and do its stuff.

Hope this helps, thanks

Andonis

Answer 8

Does your service account actually have the required permission? The role(s) that will give you this permission are roles/storage.objectViewer / roles/storage.objectAdmin / roles/storage.admin.

Please ensure the service account actually have the permissions in your Cloud Console and then it should work.

--- UPDATE ---

Since you have the correct permission in the account, there it's likely the correct account wasn't used in the gsutil command. This can happen if you have multiple installations of your gsutil tool, please ensure your gsutil has the correct path point to a .BOTO file. There's a similar issue reported on the github repo. You can see if the solution there works.

Ultimately, you can use a new machine / vm with a fresh install to test it out to see if it works. You can this easily by going to the Cloud Console and using the Cloud Shell. No real installation needed, should be very simple to test.

This should work and it will basically isolate your issue (to that of multiple installation) on your original machine. After that, you basically just have to do a clean install to fix it.

Mezereum

It wouldn't say 'Anonymous caller' if the user was authenticated. It would say 'Caller' (or perhaps have the email address of the caller). – Negrete

Yes, it does. It has the same permissions as my personal account which I can use from the web but not from console – Rapp

I see what you mean. In that case, it's likely an installation issue (multiple gsutil version). You can test that with a clean install or use a new machine. Detailed instructions in my edited answer. – Mezereum

Answer 9

If you installed gsutil using python (without gcloud SDK), it may help to run gsutil config and complete steps of initialisation.

Ghislainegholston

Despite gsutil was coming from the standard installation, I still had to perform gsutil config and it authenticated the tool. Although my problem was described by mhouglum's answer – Execute

Answer 10

Thank you for all the replies.
I would like to share my own experience.
I had to login under the user which is defined when installing Gitlab Runner.
By default, the user indicated in the installation doc is : "gitlab-runner".
So, first, I added a password on this user:

passwd gitlab-runner

then :

su - gitlab-runner
gcloud auth login
gcloud auth application-default login

The issue is solved.
Maybe there is a better way, by directly putting the Google auth files under /home/gitlab-runner

Lumenhour

Answer 11

I faced same issue. I used

gcloud auth login

and follow the link

Foist

This is a duplicate of existing answers. – Byers

Answer 12

If you are using a service account you need first to authorize it, otherwise gsutil won't have the permission to read/write

 gcloud auth activate-service-account --key-file=service_account_file.json

Teniers

Answer 13

Personally, I had an account with proper permissions registered but I got that error as well despite verifying that my account was running using "sudo gcloud init"

What solved it for me was navigating to the ~/.gutil directory and writing the following

sudo chown jovyan:jovyan * 

which let my JupyterLab terminal run, not from root, but from default jovyan. After that it used my account, not Anonymous caller

Endospore

Answer 14

Here is another way to edit roles:

gsutil iam ch allUsers:objectViewer gs://tf-learn-objectdetection

Fore more documentation:

gsutil iam help

Congius

Answer 15

Use gcloud auth login

1. Goto mention link
2. Copy Verification code
3. Paste Verification code

Diagonal

This is a duplicate of existing answers. – Byers

Answer 16

In my case, even after using gsutils solutions discussed in other answers, I got the error. After checking other google search results, I found out that the reason was that I was authenticating with "my user" while running the gsutils as the root.

Thanks to the answer in the gsutils page in github: https://github.com/GoogleCloudPlatform/gsutil/issues/457

Tejada

Answer 17

Let me expain what helped me step by step:

First my requirement is to enable CORS, but faced the asked issue, So I followed the below steps:

On Browser side:

1. Open google cloud console on your browser.
2. Open Cloud shell editor.
3. Type gcloud auth login.
4. Now it will show an command with an url.
5. Copy that command Don't close browser.

On PC GCloud software side:

1. Download GCloud Sdk Installer.exe
2. Open GCLoud in your pc It will ask you to sign In via browser
3. Signin with correct email id
4. Select your project from the shown list
5. Paste the previously copied command
6. Again it will ask you to signIn
7. Select the proper account to sign in
8. Now the GCloud cmd will show you another command with url as output
9. Copy the output Open your browser, then paste it.

Done! It will show like You are now logged in as [email protected]

Now I'm able to set CORS without any exception. Hope these steps will be helpfull for someone who is new to the issue.

Steffaniesteffen

Answer 18

Looks like account information is not stored with gsutil

Step 1: gsutil config

Step 2: copy url in browser

Step3: select account and grant permission

Step 4: Copy key and share it in gsutil promt "step1 will be asking for this key to proceed"

Step 5: Run command whose access was denied

Thank you Petr Krýže!!! you saved my day...

Holliholliday

Answer 19

For me the problem was: I've installed gsutil following the instructions at https://cloud.google.com/storage/docs/gsutil_install#deb and afterwards I've used pip to install dependencies from a project and one of them was gsutil. Both have concurred and the solution was to uninstall it from python modules:

pip uninstall gsutil

Ginsberg

Answer 20

In my case I got this error message as I was trying to read with Pandas a csv file in GCS, outside from a GCP environment:

import pandas as pd

path = "gs://createbucket123/my.csv"
pd.read_csv(path)

as per Read csv from Google Cloud storage to pandas dataframe

I'll try to look into how to authenticate my Pandas DF with GCP.

Prizefight