Can I automatically enable APIs when using GCP cloud with terraform?

Question

I am very new to GCP with terraform and I want to deploy all my modules using centralized tools.

Is there any way to remove the step of enabling google API's every time so that deployment is not interrupted?

Karilla

Can you add more details? Why do you need to enable APIs everytime? you should need to do once for a project – Cope

I think I see the purpose of the question. Imagine I create a brand new project and wish to populate that project with assets. I might want to use Terraform. However before my application can actually run, I may need to enable some APIs and it would be great if the Terraform script could do that too as I would consider that "infrastructure setup". – Adolfo

hello @pradeep My question is that I am using Jenkins as a centralized tool to deploy my modules in the GCP so during my deployment I have to enable the required APIs due to this it interrupts my process of deployment so there is any bypass to this. – Karilla

You can enable all the required API before starting application deployment using Terraform or gcloud commands. API can be enabled at project creation time or anytime after that. – Cope

Answer 1

There is a Terraform resource definition called "google_project_service" that allows one to enable a service (API). This is documented at google_project_service.

An example of usage appears to be:

resource "google_project_service" "project" {
  project = "your-project-id"
  service = "iam.googleapis.com"
}

Adolfo

not enabling serviceusage.googleapis.com, what could be the issue? – Hymnal

Answer 2

instead of using count as suggested by @pradeep you may also loop over the services in question:

variable "gcp_service_list" {
  description ="The list of apis necessary for the project"
  type = list(string)
  default = [
    "cloudresourcemanager.googleapis.com",
    "serviceusage.googleapis.com"
  ]
}

resource "google_project_service" "gcp_services" {
  for_each = toset(var.gcp_service_list)
  project = "your-project-id"
  service = each.key
}

Gifferd

Cleanest solution on this page. Should probably be the accepted answer. – Velasquez

Error when reading or editing Project Service nvoi-playground/cloudresourcemanager.googleapis.com: Request List Project Services xxx` returned error: Batch request and retried single request ...` is this expected ? – Bigname

Answer 3

Yes , you can use google_project_service resource to enable one API at a time. You can use count or other loop methods to enable multiple APIs. You would need project editor/owner role to do this.

# Enable services in newly created GCP Project.
resource "google_project_service" "gcp_services" {
  count   = length(var.gcp_service_list)
  project = google_project.demo_project.project_id
  service = var.gcp_service_list[count.index]

  disable_dependent_services = true
}

You can find the complete example here.

Cope

Answer 4

For those reading this in 2022, enabling serviceusage and cloudresourcesmanager automatically from Terraform doesn't work as enabling those APIs through the API has a dependency on them being already enabled...

The solution is to do it through the gcloud command line:

# Use `gcloud` to enable:
# - serviceusage.googleapis.com
# - cloudresourcemanager.googleapis.com
resource "null_resource" "enable_service_usage_api" {
  provisioner "local-exec" {
    command = "gcloud services enable serviceusage.googleapis.com cloudresourcemanager.googleapis.com --project ${var.project_id}"
  }

  depends_on = [google_project.project]
}

# Wait for the new configuration to propagate
# (might be redundant)
resource "time_sleep" "wait_project_init" {
  create_duration = "60s"

  depends_on = [null_resource.enable_service_usage_api]
}

More details on https://medium.com/rockedscience/how-to-fully-automate-the-deployment-of-google-cloud-platform-projects-with-terraform-16c33f1fb31f

Postern

This seems like a huge oversight from the team at Terraform. Has no one raised this issue on Github yet? – Tarentarentum

This should be a bug on the TF github not a "work around" – Rockribbed

Answer 5

2022 - Sharing my personal experience of enabling services using code

Why you should not enable services using Terraform

1. Usually a cost is associated with enabling services and Billing account have to be linked to the services at time. Example, you want to create a static public ip or setup a Cloud CDN.

2. Enabling services using code is possible as suggested in https://cloud.google.com/service-usage/docs/enable-disable#gcloud and https://mcmap.net/q/395701/-can-i-automatically-enable-apis-when-using-gcp-cloud-with-terraform (Using Terraform null resource running gcloud command) but the additional challenge is, enabling a service does not happen in second or minute.

   Adding wait/sleep or dependency flow helped me for a while but, in the long run it complicated my code.

3. Cross dependency issues. Using Terraform code when I enabled services for Compute, then other developer were able to create resources. After sometime, when I no longer need and tried to terraform destroy I got several dependency issue.

TL;DR Enabling services is usually one-time task just like creating a billing account. As per my experience, I recommend not to automate such important things.

Below is added on April 2023

4. Distribute your TF Code. As suggested in by @FreshMike & @Juarez_Rudsatz, distribute your TF code into multiple folders.

   You can do this by product wise(Compute/Networking/IAM/ServiceAccounts/..) or module wise(Permissions/Admin/Module-A/Product-A/Product-B/..) or others depending on what suits your requirement.

   In my team project, we used a separate folder call test/setup and used this module https://registry.terraform.io/modules/terraform-google-modules/project-factory/google/latest/submodules/project_services to enable multiple APIs in one go.

Gelhar

That's why I would suggest to split into multiple TF runs. One for settings up APIs, Network, External IPs, and a second for users, compute resources, LBs, the whole shebang. Now if you need to destroy you can destroy without affecting the one time stuff. – Hammerhead

Thanks for your comment. I'm just curious, did it work for you? Did mean having multiple independent terraform folders right? – Gelhar

It works well if used with multiple terraform states. As mentioned above, it's interesting to split into multiple folders/modules/states and segregate things ilike: - IAM roles/users/groups - IAM role/permission assignments - Networking - Storage - Database - Instance/Compute - App - Services – Tildie

How do you handle referencing resources which are in another folder? Do you just hardcode the values? Also, doesn't this introduce a required order of deployment - i.e. deploy this folder first, then that, etc.? – Conqueror

Answer 6

All the answers are excellent; however, I'd like to add a new scenario not yet covered.

I tried all the answers, e.g.,

resource "google_project_service" "x" {
  project = ...
  service = "compute.googleapis.com"
}

but the Compute Engine default service account ends up in the disabled state:

After a lot of investigation, I found the culprit to be the way I created the project. I used the following open source module to create the project:

module "my_project" {
  source  = "terraform-google-modules/project-factory/google"
  ...
}

The open source module's underlying source code contains this snippet:

resource "google_project_default_service_accounts" "default_service_accounts" {
  count          = upper(var.default_service_account) == "KEEP" ? 0 : 1
  action         = upper(var.default_service_account)
  ...

where the var's default value is this:

variable "default_service_account" {
  description = "Project default service account setting: can be one of `delete`, `deprivilege`, `disable`, or `keep`."
  default     = "disable"

The combined result is equivalent to the sample code at google_project_default_service_accounts:

resource "google_project_default_service_accounts" "my_project" {
  project = "my-project-id"
  action = "DISABLE"
  ...
}

Therefore, the solution is:

module "my_project" {
  source  = "terraform-google-modules/project-factory/google"
  default_service_account = "KEEP"  # <<<<====== solution
  ...
}

Caveat: The security best practice is to disable the default SAs and create your own custom SAs, I suppose. So, my solution above is to solve a mystery, not to be considered the security best practice.

Selective

Hi, this is an amazing post. I very much like detailed work. Kudos! – Gelhar