Serverless VPC access connector is in a bad shape

Question

Our project is using a Serverless VPC access connector to allow access to DB over private IP from cloud functions and cloud runs. It was working flawlessly for a few months, but today I tried to deploy one of the functions that use such a connector and I got the message:

VPC connector projects/xxxx/locations/us-central1/connectors/vpc-connector is not ready yet or does not exist. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.

I went to the Serverless VPC access view and found out that indeed the connector has a red marking on it. When I hover on it it says

Connector is in a bad state, manual deletion recommended

but I don't know for what reason, Link to logs doesn't show anything for the past 3 months.

I tried to google about the such error but without success. I also tried to search through logs but also didn't find anything relevant.

I'm looking for any hints:

• Why it happened?
• How to fix it? I don't want to recreate the connector, it is related to many functions, and cloud runs

Malediction

You have tips in the doc, especially if you have IP range override, if you have private service connect, or other stuff like that. – Gustav

Answer 1

As the issue was blocking us from the deployment of cloud functions I was forced to recreate the connector.

But this time API returned an error:

Error: Error waiting to create Connector: Error waiting for Creating Connector: Error code 7, message: Operation failed: Google APIs Service Agent (<PROJECT_NUMBER>@cloudservices.gserviceaccount.com) needs editor role in the project.

After adding such permission old connector started to work again...

Before there was no such requirement, but it changed in meantime.

Spooky, one time something works other not.

Malediction

I am having the same problem. I don't trust myself to recreate the connector since if I do it wrong, I'll be bringing down our live site. I tried adding editor permissions to any related service account and was unsuccessful. Will ask my GCP consultant. I agree GCP has stuff working one day and completely broke the next, it's happened so many times to our prod site it's kinda a scary reality. – Crib

Turns out it got solved on its own. Google had some resource issues and I think that caused it. After waiting 24 hours it fixed itself. – Crib

I've encountered the same problem while tearing down my infra using pulumi (similar to terraform). I also saw the "VPC access connector is in a bad shape" and the API also returns for me the same error about adding the editor role. When I go the Google Cloud Console UI to delete the connector is also fails to delete with the same error as the API ... which was really unexpected as I'm logged in with my personal account that has the Owner role.... – Plashy

@KevinDanikowski thanks for the info, really valuable feedback. It's good to know that I'm not doing anything wrong on my side. Did the GCP consultant recommend any next steps from Google or provide any advice or workarounds? If more people encounter this on StackOverflow maybe it's worth raising a bug with GCP (if one hasn't been raised already?). – Plashy

@JamesLawson Honestly i've found that lots bugs like this with GCP. I don't trust them for reliability. I've had load balancers fail for no reason a few times, had to recreate it to fix it. This one wasn't easy to recreate so I'm glad it fixed itself. i think the only solution is to rebuild it tho. Otherwise, just wait. in my case, it didn't stop working, we just couldn't connect to launch new cloudrun containers. so we were able to wait it out – Crib

Answer 2

This week, I also got the Error code 7 when I attempted to destroy my VPC connector implementation. Initially, I tried to delete it with my Terraform configuration, and then via the GCP Console UI. Both didn't work. The full error was as follows:

Error waiting for Deleting Connector: Error code 7, message: Operation failed: Forbidden

What probably caused my problem was the IAM policy implementation. During the development, I first created the VPC connector, and then the IAM policy. When I implemented the policy it deleted the full IAM policy for the full GCP project.

This way, I deleted the GCP service account attached to the vpcaccess.googleapis.com, which is service-<PROJECT_NUMBER>@gcp-sa-vpcaccess.iam.gserviceaccount.com. By re-adding the service account and give it the roles/vpcaccess.serviceAgent role, I could delete my VPC access connector without the above error.

Hope this helps.

Homosexual

Answer 3

The only resolution I can find is to add the Editor role before adding the connector, with its 8963 excess permissions.

The error message Error code 7, message: Operation failed: Google APIs Service Agent (<PROJECT_NUMBER>@cloudservices.gserviceaccount.com) needs editor role in the project. is surprising as basic roles are not even allowed in some organization policies. You would think a more specific role should be sufficient to create a VPC serverless connector.

You can't add conditions like Expiry to a basic role so don't forget to close this security hole before the SecOps team comes after you.

If your VPC is in the service project of a Shared VPC then you will need to add this role to the host project before creating the connector. That's not clear in the message. Giving a service account from a service project Editor role on the network host project might raise a few eyebrows in SecOps if they haven't already blocked such a path. That would force you to put the connector in the host project instead.

You will also need to apply the "roles/compute.networkUser" to a bunch of service accounts on the host project's subnet that this connector sits on. Don't apply these bindings to the project but directly to the subnet (which may be on the host project),

This is not a definitive list of service accounts as it depends on what's using the connector. App engine default service account for the first, cloud functions for the second) :

• "serviceAccount:${var.project_id}@appspot.gserviceaccount.com"
• "serviceAccount:service-${var.project_number}@gcf-admin-robot.iam.gserviceaccount.com"
• "serviceAccount:${var.project_number}@cloudservices.gserviceaccount.com"
• "serviceAccount:service-${var.project_number}@gcp-sa-- vpcaccess.iam.gserviceaccount.com"

You can't see subnet IAM bindings on the console, but gcloud can show them

 gcloud compute networks  subnets get-iam-policy projects/VPC_HOST_PROJECT/regions/REGION/subnetworks/my-subnet 

If any of the permissions are not set, the VPC will be created but left in a bad state. You need Serverless VPC Access Admin role to delete it (or Editor role), but even that won't help if one of your services is actively using the connector and you get a resource in use error.

Good luck [re]creating one.

Nonfulfillment