Terraform check if resource exists before creating it

Question

Is there a way in Terraform to check if a resource in Google Cloud exists prior to trying to create it?

I want to check if the following resources below exist in my CircleCI CI/CD pipeline during a job. I have access to terminal commands, bash, and gcloud commands. If the resources do exist, I want to use them. If they do not exist, I want to create them. I am doing this logic in CircleCI's config.yml as steps where I have access to terminal commands and bash. My goal is to create my necessary infrastructure (resources) in GCP when they are needed, otherwise use them if they are created, without getting Terraform errors in my CI/CD builds.

If I try to create a resource that already exists, Terraform apply will result in an error saying something like, "you already own this resource," and now my CI/CD job fails.

Below is pseudo code describing the resources I am trying to get.

resource "google_artifact_registry_repository" "main" {
  # this is the repo for hosting my Docker images
  # it does not have a data source afaik because it is beta
}

For my google_artifact_registry_repository resource. One approach I have is to do a Terraform apply using a data source block and see if a value is returned. The problem with this is that the google_artifact_registry_repository does not have a data source block. Therefore, I must create this resource once using a resource block and every CI/CD build thereafter can rely on it being there. Is there a work-around to read that it exists?

resource "google_storage_bucket" "bucket" {
  # bucket containing the folder below
}

resource "google_storage_bucket_object" "content_folder" {
  # folder containing Terraform default.tfstate for my Cloud Run Service
}

For my google_storage_bucket and google_storage_bucket_object resources. If I do a Terraform apply using a data source block to see if these exist, one issue I run into is when the resources are not found, Terraform takes forever to return that status. It would be great if I could determine if a resource exists within like 10-15 seconds or something, and if not assume these resources do not exist.

data "google_storage_bucket" "bucket" {
  # bucket containing the folder below
}

output bucket {
  value = data.google_storage_bucket.bucket
}

When the resource exists, I can use Terraform output bucket to get that value. If it does not exist, Terraform takes too long to return a response. Any ideas on this?

Gael

The ideal path forward here is to import these resources into the state and manage them with TF. – Evanevander

I tried that. The problem I ran into was the case where I was importing a resource (cloud storage bucket) that did not exist. Terraform import would hang, or it took so long I didn't wait around for it to complete. It was more minutes that I wanted this script to wait. If the resource doesn't exist and I know that, then I can create the resource. If the resource exists, then I can import it. I need to know if the resource exists first to determine if the import is necessary. – Gael

Answer 1

TF does not have any build in tools for checking if there are pre-existing resources, as this is not what TF is meant to do. However, you can create your own custom data source.

Using the custom data source you can program any logic you want, including checking for pre-existing resources and return that information to TF for future use.

Endospore

Thank you for sharing this. I was not aware of external data sources in Terraform. I can see examples using bash scripts, python, and JavaScript. I wrote a quick bash script to call gsutil to get my GCP Cloud Storage bucket. I think this will work. Thanks! – Gael

@Endospore : Seems there's link rot on the doc link: "This documentation page doesn't exist for version 2.3.2 of the external provider." – Cabotage

Probably: registry.terraform.io/providers/hashicorp/external/latest/docs/… – Lipski

Answer 2

Thanks to the advice of Marcin, I have a working example of how to solve my problem of checking if a resource exists in GCP using Terraform's external data sources. This is one way that works. I am sure there are other approaches.

I have a CircleCI config.yml where I have a job that uses run commands and bash. From bash, I will init/apply a Terraform script that checks if my resource exists, like so below.

data "external" "get_bucket" {
  program = ["bash","gcp.sh"]
  query = {
    bucket_name = var.bucket_name
  }
}

output "bucket" {
  value = data.external.get_bucket.result.name
}

Then in my gcp.sh, I use gsutil to get my bucket if it exists.

#!/bin/bash

eval "$(jq -r '@sh "BUCKET_NAME=\(.bucket_name)"')"
bucket=$(gsutil ls gs://$BUCKET_NAME)

if [[ ${#bucket} -gt 0 ]]; then
  jq -n --arg name "" '{name:"'$BUCKET_NAME'"}'
else
  jq -n --arg name "" '{name:""}'
fi

Then in my CircleCI config.yml, I put it all together.

terraform init
terraform apply -auto-approve -var bucket_name=my-bucket
bucket=$(terraform output bucket)

At this point I check if the bucket name is returned and determine how to proceed based on that.

Gael

So in fact you can do the same without terraform? You can call you bash script at the yaml file level instead calling terraform for that. As I understand that you call terraform 2 times in your case? 1st to know if exist then 2nd time with your previous output as input of the 2nd call – Gallbladder

Answer 3

There is a way to check if a resource already exists before creating the resource. But you should be aware of whether it exists. Using this approach, you need to know if the resource exists. If the resource does not exist, it'll give you an error.

I will demonstrate it by create/reading data from an Azure Resource Group. First, create a boolean variable azurerm_create_resource_group. You can set the value to true if you need to create the resource; otherwise, if you just want to read data from an existing resource, you can set it to false.

variable "azurerm_create_resource_group" {
    type = bool
}

Next up, get data about the resource using the ternary operator supplying it to count, next do the same for creating the resource:

data "azurerm_resource_group" "rg" {
    count = var.azurerm_create_resource_group == false ? 1 : 0
    name = var.azurerm_resource_group
}

resource "azurerm_resource_group" "rg" {
    count         = var.azurerm_create_resource_group ? 1 : 0
    name          = var.azurerm_resource_group
    location      = var.azurerm_location
}

The code will create or read data from the resource group based on the value of the var.azurerm_resource_group. Next, combine the data from both the data and resource sections into a locals.

locals {
        resource_group_name = element(coalescelist(data.azurerm_resource_group.rg.*.name, azurerm_resource_group.rg.*.name, [""]), 0) 
        location            = element(coalescelist(data.azurerm_resource_group.rg.*.location, azurerm_resource_group.rg.*.location, [""]), 0)
    }

Another way of doing it might be using terraformer to import the infra code.

I hope this helps.

Fourier

That doesn't check if the resource exists; it asserts that it does or doesn't exist, and then adjusts behaviour accordingly. To check if a resource exists, terraform would need to support non-failing/conditional data resources such that the calling module could check whether the data resource returned a valid result, but this functionality doesn't exist. – Aborn

And what if I'd need to delete the module instance that actually creates the resource group? It'd be deleted as well. I'm afraid there is no a good solution and approach should change – Rankins

Answer 4

This has been very useful.

1. Create a Powershell or shell script that does a lookup of the resource and returns true if it exists or false if it does not.

...
$schedules = (Get-AzAutomationSchedule -AutomationAccountName $automationAccountName -ResourceGroupName $resourceGroupName -Name $lookupName -ErrorAction SilentlyContinue)
if ($schedules.Count -gt 0) {
    Write-output '{ "exists": "1" }'
} else {
    Write-Output '{ "exists": "0" }'
}

2. Add a data element

data "external" "getSchedule" {
  program = ["pwsh", "${path.module}/getSchedule.ps1", "new-schedule"]
}

3. Use the Variable returned in count to add resrouce if true or not if false

resource "azurerm_automation_schedule" "new-schedule" {
  count = data.external.getSchedule.result.exists ? 0 : 1
...
}

Ljubljana

Answer 5

This is a particularly difficult problem.

I'm using Digital Ocean and they have a nice SDK that I can use to check whether a resource exists or not. So I thought let me first check if the resource exists, and if it does, let me create a data source against it so I can access its config. If it doesn't exist, I'll create it.

And bear in mind that this was happening on synth as that is when the terraform json is generated.

So that works great, the first time. The second time you run it however, when it does the check for the existence of the resource, it will find it present and thus it wont generate the iac config for this resource and thus when it does the diff to ascertain what it needs to deploy it will want to delete the resource.

So you have two options:

1. Either you simply accept that you're going to need to provid a parameter to link a data source with (if parameter is provided, data source is created, if not, resource is created) OR
2. You're going to need to add an extra flag when filtering for the existing resource which you can use to indicate whether it's a managed resource or not - so when doing the lookup you make sure to ignore any resources that don't have this flag. That way you get the best of both worlds but you do have to deal with a lot more complexity and could lead to issues if not everyone understands this mechanism. And furthermore, at least in Digital Ocean, not all resources support flags.

Translate

Answer 6

This work for me:

1. Create data

data "gitlab_user" "user" {
  for_each = local.users
  username = each.value.user_name 
}

2. Create resource

resource "gitlab_user" "user" {
  for_each       = local.users
  name           = each.key
  username       = data.gitlab_user.user[each.key].username != null ? data.gitlab_user.user[each.key].username : split("@", each.value.user_email)[0]
  email          = each.value.user_email
  reset_password = data.gitlab_user.user[each.key].username != null ? false : true
}

P.S.

Variable

variable "users_info" {
  type = list(
    object(
      {
        name         = string
        user_name    = string
        user_email   = string
        access_level = string
        expires_at   = string
        group_name   = string
      }
    )
  )
  description = "List of users and their access to team's groups for newcommers"
}

Locals

locals {
  users  = { for user in var.users_info : user.name => user }
}

Lichtenfeld

No, it doesn't work, I have tried with aws_iam_role – Presto