gcloud beta run deploy fails after sucessfully uploading image, fails to enable API

Question

gcloud beta run deploy used to work but now I'm getting an error:

$ gcloud beta run deploy $PROJECT --image $IMAGE_NAME --platform=managed --region us-central1 --project $PROJECT --add-cloudsql-instances $PROJECT-db
...
DONE
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ID                                    CREATE_TIME                DURATION  SOURCE                                                                                 IMAGES                                    STATUS
abcdefj-higj-lmnopquer-uvw-xyz  2019-06-29T13:59:07+00:00  1M4S      gs://$PROJECT_cloudbuild/source/XYZ123.96-aae829d50a2e43a29dce44d1f93bafbc.tgz  gcr.io/$PROJECT/$PROJECT (+1 more)  SUCCESS
API [sql-component.googleapis.com] not enabled on project 
[$PROJECT]. Would you like to enable and retry (this will take a 
few minutes)? (y/N)?  y

Enabling service [sql-component.googleapis.com] on project [$PROJECT]...
ERROR: (gcloud.beta.run.deploy) INVALID_ARGUMENT: Invalid operation name operations/noop.DONE_OPERATION, refers to an already DONE operation

I've checked the APIs from the console, both Cloud SQL Admin and Cloud SQL APIs are enabled. I've also tried disabling them and run the deploy command again, but to no avail.

More info:

1. The SQL server instance is part of the same project. Changing the --add-cloudsql-instances parameter to the connection name ($PROJECT:$REGION:$SQLNAME) has no effect

2. Manually enabling the server has no effect: gcloud services enable sql-component.googleapis.com --project XXX

3. Removing the --add-cloudsql-instances parameter and the server deploys successfully.

4. This works: gcloud sql connect $PROJECTDB --user=root --quiet # NOTE: ($PROJECTDB) is the same parameter as --add-cloudsql-instances above

Kirkman

1) Are you trying to add a SQL Server instance from another project? If yes, does the service account that you are using for Cloud Run have permissions to that SQL instance in the other project? By default Cloud Run uses the Compute Engine Default Service Account. 2) Try manually enabling the service as it may be taking too long for Cloud Run gcloud services enable sql-component.googleapis.com --project XXX 4) What is the value of $PROJECT-db. It should look like $PROJECT:$REGION:$GCP_SQL_NAME. Example: development-123456:us-central1:mysqldb – Heaviness

1) nope, it's the same project 2) oh right! that command returned no error. But gcloud beta run deploy still fails with the same error as before. 4) Ok I just had the name of the db, changing that to the instance name didn't work. – Kirkman

Update your question with the new details. You either have a service account issue or an SQL instance name issue. Try removing the --add-cloudsql-instances command line option to make sure you can deploy and that your service responds on $PORT. Once you know that works, debug the Cloud SQL issue. – Heaviness

Thanks, updated. So the deploy step works if I omit --add-cloudsql-instances. Do you think then it's a service account issue? The docs say that cloud run will use the default service account if not specified, and that account already has Editor permissions... – Kirkman

Is Cloud SQL in the same Region as Cloud Run? – Heaviness

Make sure that this command is returning the same "connection name" as you are using in the deploy command gcloud sql instances describe <instance_name> – Heaviness

As a test, create a new service account. Attach the Cloud SQL Admin role. Use the --service-account command line option to specify the service account. cloud.google.com/sdk/gcloud/reference/beta/run/… – Heaviness

If the service account works, then download the role to Cloud SQL Client – Heaviness

Is there anything logged in Stackdriver logs for Cloud Run that indicates the failure reason? – Heaviness

Same region, same connection string. Something interesting I just noticed is that when I deployed WITHOUT the --add-cloudsql-instances, the daemon deployed correctly and somehow has connectivity to the SQL instance. Not sure how that works! I'm guessing this is a bug in the cloud run interface... (accidentally hit enter, let me try deploying a few code changes to confirm) – Kirkman

OK just confirmed it's working... it seems as though gcloud beta run deploy with --add-cloudsql-instance internally setup the db configuration, and fail. Then when we redeploy, omitting the cloudsql configuration parameter will NOT clear the previous configuration and allow the deployment to complete. – Kirkman

@JohnHanley Thanks for your help :-) If you add the resolution as an answer I'll give you credit for it... Cheers – Kirkman

@JohnHanley Could you post your solution as an Answer? – Electrolytic

Answer 1

For this problem there were two issues:

1. Enabling API services. I recommend enabling services prior to running Cloud Run deploy as this can take longer than Cloud Run might allow. Run this command first: gcloud services enable sql-component.googleapis.com
2. The Cloud SQL connection name was incorrect. Specifying the correct name helps.

The format of the Cloud SQL connection name is: $PROJECT:$REGION:$GCP_SQL_NAME.

Example: development-123456:us-central1:mysqldb

This command will return information about the Cloud SQL instance including the connection name:

gcloud sql instances describe <instance_name>

Note. Cloud Run has several commands for specifying the Cloud SQL instance to attach.

--add-cloudsql-instances - This option appends the specified connection name.

--set-cloudsql-instances - This option replaces the current Cloud SQL connection name.

If you are not deploying a new version to Cloud Run, it is not necessary to use the --add-cloudsql-instances option as the value persists. I prefer to use the --set-cloudsql-instances option to clearly specify the Cloud SQL instances.

Cloud Run supports multiple Cloud SQL instances. You can have add more than one connection name.

Heaviness

Thanks! Changing --add to --set solved my problem! – Zachariah

Answer 2

There seems to be a bug in gcloud v253.0.0 when deploying deploy Cloud Run services with Cloud SQL instances (requires Gmail log-in).

Once I downgraded to gcloud v251.0.0, I got rid of the "API [sql-component.googleapis.com] not enabled" error message and was able to deploy Cloud Run services with Cloud SQL instances again.

$ gcloud components update --version 251.0.0

UPDATE, July 17, 2019: The issue is fixed in Cloud SDK 254.0.0. If you upgrade to the latest version now, deploying Cloud Run services with Cloud SQL instances should work:

$ gcloud components update

Theda

Can confirm: I found this answer by Googling for "sql-component.googleapis.com noop.DONE_OPERATION" after getting a similar error when following the instructions here: cloud.google.com/run/docs/configuring/connect-cloudsql By downgrading from 253 to 251 things worked. – Sputnik

Confirmed as well: I was having the same issue and solved it by downgrading to 251.0.0 thanks @Martin – Remuneration