Cloud ML Service account cannot access Cloud Storage and is not listed in IAM & admin panel

Question

When creating a new version of an ML Engine Model with the command

gcloud ml-engine versions create 'v1' --model=model_name --origin=gs://path_to_model/1/ --runtime-version=1.4

I recieve the following error:

ERROR: (gcloud.ml-engine.versions.create) FAILED_PRECONDITION: Field: version.deployment_uri Error: Read permissions are required for Cloud ML service account cloud-ml-service@**********.iam.gserviceaccount.com to the model file gs://path_to_model/1/saved_model.pb.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: Read permissions are required for Cloud ML service account cloud-ml-service@**********.iam.gserviceaccount.com to the model file gs://path_to_model/1/saved_model.pb.
  field: version.deployment_uri

This service account is not listed in the IAM & admin panel and does not belong to my project, so I don't want to grant permissions for this account manually.

Has anyone else also experienced this? Any suggestions on what I should do?

Additional information:

• The google storage bucket has storage class regional and location europe-west1.

• I already tried to disable (and re-enable) the ML Engine service with the command

  gcloud services disable ml.googleapis.com
  

  but this resulted in the following error:

  ERROR: (gcloud.services.disable) The operation with ID tmo-acf.********-****-****-****-************ resulted in a failure.
  

Updated information:

• The storage bucket does not belong to a different project.

• The command

  gcloud iam service-accounts get-iam-policy cloud-ml-service@**********.iam.gserviceaccount.com
  

  gives the error:

  ERROR: (gcloud.iam.service-accounts.get-iam-policy) PERMISSION_DENIED: Permission iam.serviceAccounts.getIamPolicy is required to perform this operation on service account projects/-/serviceAccounts/cloud-ml-service@**********.iam.gserviceaccount.com.
  

  The dash in the path projects/-/serviceAccounts/... in this error message seems very wrong to me.

PROBLEM HAS BEEN SOLVED

I was finally able to disable the ML Engine service after removing all my models. After re-enabling the service I got a new service account which shows up in my IAM & admin panel and is able to access my cloud storage.

Adaptive

Answer 1

If someone finds this issue, @freeCris wrote the solution in the question. I decided to write this down as I read all the documentation in the answers to find nothing useful and then realized he wrote how to solve it in the question itself.

For those wanting to fix this, just run (make sure you don't have resources in ML Engine such as models and versions):

gcloud services disable ml.googleapis.com

And then run:

gcloud services enable ml.googleapis.com

You'll get a new service account that this time is listed in your IAM console. Just add it to your GCS bucket and it'll work now.

Gingrich

Answer 2

I think the problem was, that you tried to create the model under a different project, which was not associated with that bucket you tried to reach. So you used the service account of that different project to access the bucket, that's why it did not have any permissions and did not appear in you AMI.

If that happens again or if anybody else has that problem, you can check your projects with gcloud projects list and change it with gcloud config set project <project name>.

Cerise

The original question clarifies and checks that it's the same project. My company only has a single project and I'm running into this issue right now. (the solution presented doesn't work -- I think it's a bug on Google's end) – Marchioness

+1 because it was the only efficient solution to work export PROJECT_ID=YOUR_PROJECT_ID gcloud config set project $PROJECT_ID – Rokach

Answer 3

Yes, that service account doesn't belong to your project. You can know the service account for the Cloud ML Engine. For deploying on ML Engine, you will need to grant read access to your model files on gcs to that service account. Here is the documentation on how you can do that: https://cloud.google.com/ml-engine/docs/access-control#permissions_required_for_storage

This might also be useful: https://cloud.google.com/ml-engine/docs/working-with-data#using_a_cloud_storage_bucket_from_a_different_project

Anastos

To clarify: I do not know which project this service account belongs to or where it comes from. The bucket does not belong to a different project. If i try to get the iam-policy for this account it results in an error (see updated question). I do not want to give read access to an account of unknown source. – Adaptive

Cloud ML Engine doesn't run in the user project. Hence the service account is not in your project. It is unique to your project though, but managed by the cloud ML Engine service. – Luteolin