Terraform claims backend GCS bucket doesn't exist

Question

Terraform 1.4.5 on Windows, authenticating as individual user via gcloud auth application-default login. I suddenly started getting this error a few hours whenever I do any operating involving the state file stored in a Google Cloud Storage bucket:

Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn't exist

My backend.tf file has not changed in several months. It simply lists the GCS bucket name and prefix:

terraform {
  backend "gcs" {
    bucket = "my-bucket-name"
    prefix = "my-module-name"
  }
}

I've tried the usual fixes, such as running terraform init -reconfigure and deleting the .terraform hidden directory, but always get the same error message.

This did happen around the time there were some permission changes and cleanup tasks done by the group that manages our GCP projects. But, the bucket still exists and I have read/write access to it.

Anisometric

Answer 1

Found the root cause of this error is gcloud's default project had been deleted. This doesn't affect Terraform directly, but was influencing the quota project, which can be viewed in file $HOME/.config/gcloud/application_default_credentials.json (Linux/Mac) or $env:APPDATA\gcloud\application_default_credentials.json (Windows)

To fix, I did this:

gcloud auth login
gcloud config set project my-main-project-id
gcloud auth application-default login

This should also do the trick:

gcloud auth application-default set-quota-project my-main-project-id

Anisometric

set-quota-project is the solution. The other commands might solve the problem due to side effects (setting the project id) but might fail later. The technical reason is that the CLI uses a Google OAuth Client ID that is not part of your ORG/Project which also means it is not part of your billing account. So you have to tell the CLI which project is to be billed. – Gifted

Ahhh thank you. I never quite understood what the quota project was, or if I should care about it. That makes sense. – Anisometric