How to for_each through a list(objects) in Terraform 0.12
Asked Answered
W

9

163

I need to deploy a list of GCP compute instances. How do I loop for_each through the "vms" in a list of objects like this:

    "gcp_zone": "us-central1-a",
    "image_name": "centos-cloud/centos-7",
    "vms": [
      {
        "hostname": "test1-srfe",
        "cpu": 1,
        "ram": 4,
        "hdd": 15,
        "log_drive": 300,
        "template": "Template-New",
        "service_types": [
          "sql",
          "db01",
          "db02"
        ]
      },
      {
        "hostname": "test1-second",
        "cpu": 1,
        "ram": 4,
        "hdd": 15,
        "template": "APPs-Template",
        "service_types": [
          "configs"
        ]
      }
    ]    
}
Watercool answered 28/10, 2019 at 16:12 Comment(3)
Terraform uses a declarative language, consequently there are no if, for or while statements. You can simulate these however, to some extent, in some cases thanks to the countproperty. Read this excellent article about how to reproduce conditions or loops with Terraform: blog.gruntwork.io/…Weinreb
Beginning in Terraform 0.12.6 resources can use for_each blocks: 0.12.6 Changelog. For what it's worth, if/else conditionals have been in Terraform for a couple of versions now: Terraform docs: Conditional ExpressionsOrganize
Here a blog post containing all possible for loops explanation of for loops in terraform - #58595006Lucretialucretius
W
109

Seem's like I found what to do. If you pass not the maps of maps but the list of maps you can use such code

resource "google_compute_instance" "node" {
    for_each = {for vm in var.vms:  vm.hostname => vm}

    name         = "${each.value.hostname}"
    machine_type = "custom-${each.value.cpu}-${each.value.ram*1024}"
    zone         = "${var.gcp_zone}"

    boot_disk {
        initialize_params {
        image = "${var.image_name}"
        size = "${each.value.hdd}"
        }
    }

    network_interface {
        network = "${var.network}"
    }

    metadata = {
        env_id = "${var.env_id}"
        service_types = "${join(",",each.value.service_types)}"
  }
}

It will create actual number of instance and when you remove for example middle one of three(if you create three:)), terraform will remove what we asked.

Watercool answered 29/10, 2019 at 12:11 Comment(3)
This works well. The list(object)) is converted to a map that can be used as the for_each value. Just select a key (e.g hostname) that is uniqueSalep
Care to share what : vm.hostname => vm does in the for_each line: for_each = {for vm in var.vms: vm.hostname => vm}?Disclaimer
@Disclaimer - This is a way of setting the "key" and "value" for the resource, where vm.hostname is the key and vm is the value. So in this example, we uniquely identify each resource in the loop by hostname. If you had multiple VMs in our list with the same hostname but different CPUs, then we might rewrite as "${vm.hostname}:${vm.cpu}" => vm. This allows terraform to track the resources by a key you define... IIRC Terraform tracks by index otherwise (which can be problematic if you reorder things)Pentheus
D
252

I work a lot with iterators in Terraform, they always gave me bad headaches. Therefore I identified five of the most common iterator patterns (code examples are given below), which helped me construct a lot of nice modules (source).

  1. Using for_each on a list of strings
  2. Using for_each on a list of objects
  3. Using for_each to combine two lists
  4. Using for_each in a nested block
  5. Using for_each as a conditional

Using for_each and a list of strings is the easiest to understand, you can always use the toset() function. When working with a list of objects you need to convert it to a map where the key is a unique value. The alternative is to put a map inside your Terraform configuration. Personally, I think it looks cleaner to have a list of objects instead of a map in your configuration. The key usually doesn't have a purpose other than to identify unique items in a map, which can thus be constructed dynamically. I also use iterators to conditionally deploy a resource or resource block, especially when constructing more complex modules.

1. Using for_each on a list of strings

locals {
  ip_addresses = ["10.0.0.1", "10.0.0.2"]
}

resource "example" "example" {
  for_each   = toset(local.ip_addresses)
  ip_address = each.key
}

2. Using for_each on a list of objects

locals {
  virtual_machines = [
    {
      ip_address = "10.0.0.1"
      name       = "vm-1"
    },
    {
      ip_address = "10.0.0.1"
      name       = "vm-2"
    }
  ]
}    

resource "example" "example" {
  for_each   = {
    for index, vm in local.virtual_machines:
    vm.name => vm # Perfect, since VM names also need to be unique
    # OR: index => vm (unique but not perfect, since index will change frequently)
    # OR: uuid() => vm (do NOT do this! gets recreated everytime)
  }
  name       = each.value.name
  ip_address = each.value.ip_address
}

3. Using for_each to make the Cartesian product of two lists

locals {
  domains = [
    "https://example.com",
    "https://stackoverflow.com"
  ]
  paths = [
    "/one",
    "/two",
    "/three"
  ]
}
    
resource "example" "example" {
  # Loop over both lists and flatten the result
  urls = flatten([
    for domain in local.domains : [
      for path in local.paths : {
        domain = domain
        path   = path
      }
    ]
  ]))
}

4. Using for_each on a nested block

# Using the optional() keyword makes fields null if not present
variable "routes" {
  type = list(
    name   = string
    path   = string
    config = optional(object({
      cache_enabled = bool
      https_only    = bool
    }))
  default = []
}

resource "example" "example" {
  name = ...
  
  dynamic "route" {
    for_each = {
      for route in var.routes :
      route.name => route
    }
    content {
      # Note: <top_level_block>.value.<object_key>
      name = route.value.name
    }
    
    dynamic "configuration" {
      # Note: <top_level_block>.value.<optional_object_key>
      for_each = route.value.config != null ? [1] : []
      content {
        cache_enabled = route.value.config.cache_enabled
        https_only    = route.value.config.https_only
      }
    }
  }

5. Using for_each as a conditional (particularly for dynamic blocks)

variable "deploy_example" {
  type        = bool
  description = "Indicates whether to deploy something."
  default     = true
}

# Using count and a conditional, for_each is also possible here.
# See the next solution using a for_each with a conditional.
resource "example" "example" {
  count      = var.deploy_example ? 0 : 1
  name       = ...
  ip_address = ...
}

variable "enable_logs" {
  type        = bool
  description = "Indicates whether to enable something."
  default     = false
}

resource "example" "example" {
  name       = ...
  ip_address = ...

  # Note: dynamic blocks cannot use count!
  # Using for_each with an empty list and list(1) as a readable alternative. 
  dynamic "logs" {
    for_each = var.enable_logs ? [] : [1]
    content {
      name     = "logging"
    }
  }
}
Dorr answered 7/6, 2021 at 12:2 Comment(6)
This is better than the official documentationCharacterize
In the conditional section, shouldn't the count line be var.deploy_something ? 0 : 1 or am I misunderstanding?Estabrook
@AndyMadge you are correct, I changed it accordingly!Dorr
One of the best stackoverflow answers for Terraform. I've bookmarked this and find myself coming back every other week. Thanks :)Humidity
@Humidity thanks, great to hear that :) This is why I am on stackoverflow in the first place.Dorr
This should be added to the official documentationLawman
W
109

Seem's like I found what to do. If you pass not the maps of maps but the list of maps you can use such code

resource "google_compute_instance" "node" {
    for_each = {for vm in var.vms:  vm.hostname => vm}

    name         = "${each.value.hostname}"
    machine_type = "custom-${each.value.cpu}-${each.value.ram*1024}"
    zone         = "${var.gcp_zone}"

    boot_disk {
        initialize_params {
        image = "${var.image_name}"
        size = "${each.value.hdd}"
        }
    }

    network_interface {
        network = "${var.network}"
    }

    metadata = {
        env_id = "${var.env_id}"
        service_types = "${join(",",each.value.service_types)}"
  }
}

It will create actual number of instance and when you remove for example middle one of three(if you create three:)), terraform will remove what we asked.

Watercool answered 29/10, 2019 at 12:11 Comment(3)
This works well. The list(object)) is converted to a map that can be used as the for_each value. Just select a key (e.g hostname) that is uniqueSalep
Care to share what : vm.hostname => vm does in the for_each line: for_each = {for vm in var.vms: vm.hostname => vm}?Disclaimer
@Disclaimer - This is a way of setting the "key" and "value" for the resource, where vm.hostname is the key and vm is the value. So in this example, we uniquely identify each resource in the loop by hostname. If you had multiple VMs in our list with the same hostname but different CPUs, then we might rewrite as "${vm.hostname}:${vm.cpu}" => vm. This allows terraform to track the resources by a key you define... IIRC Terraform tracks by index otherwise (which can be problematic if you reorder things)Pentheus
S
51

From Terraform 1.3, you can use the for_each and objects with modules like the following:

modules/google_compute_instance/variables.tf

variable "hosts" {
    type = map(object({
        cpu           = optional(number, 1)
        ram           = optional(number, 4)
        hdd           = optional(number, 15)
        log_drive     = optional(number, 300)
        template      = optional(string, "Template-New")
        service_types = list(string)
      }))
    }

modules/google_compute_instance/main.tf

resource "google_compute_instance" "gcp_instance" {
  for_each = {
    for key, value in var.hosts :
    key => value
  }

  hostname      = each.key
  cpu           = each.value.cpu
  ram           = each.value.ram
  hdd           = each.value.hdd
  log_drive     = each.value.log_drive
  template      = each.value.template
  service_types = each.value.service_types
}

servers.tf

module "gcp_instances" {
    source = "./modules/google_compute_instance"

    hosts = {
        "test1-srfe" = {
            hdd           = 20,
            log_drive     = 500,
            service_types = ["sql", "db01", "db02"]
        },
        "test1-second" = {
            cpu           = 2,
            ram           = 8,
            template      = "APPs-Template",
            service_types = ["configs"]
        },
    }
}

Of course, you can add as many variables as needed and use them in the module.

Scallop answered 30/10, 2019 at 11:16 Comment(5)
It's nice but variable is dynamically provided each time.Watercool
Common variables can be added separately as variables and reused as var.variable. The unique ones have to be declared separately anyway.Whencesoever
You're a hero. Finally I've found a proper example, from start to end.Bad
Code was updated to use optional variables with defaults from TF 1.3, and to use the host objects' keys as hostnames instead of duplicating the code.Whencesoever
Wonderful, this worked for me. for_each = { for key, value in local.instances : key => value } If your data type is tuple, this is the only way do it I think. Please correct me If I am wrong.Metropolitan
T
9

You can do the following:

for_each = toset(keys({for i, r in var.vms:  i => r}))
cpu = var.vms[each.value]["cpu"]

Assuming you had the following:

variable "vms" {
    type = list(object({
        hostname        = string
        cpu             = number
        ram             = number
        hdd             = number
        log_drive       = number
        template        = string 
        service_types   = list(string)
    }))
    default = [
        {
            cpu: 1
            ...
        }
    ]
}
Teter answered 8/7, 2020 at 13:46 Comment(0)
O
7

Using the for_each block is pretty new and there's not too much documentation. Some of the best info comes from their announcement blog post: https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/

Also make sure to check out the Dynamic Blocks section of their documentation: https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

From what your example looks like you need to have a set of values for each instance that is created so you'll have a map of maps:

Below is an example I created using Terraform 0.12.12:

variable "hostnames" {
    default = {
        "one" = {
            "name" = "one",
            "machine" = "n1-standard-1",
            "os" = "projects/coreos-cloud/global/images/coreos-stable-2247-5-0-v20191016",
            "zone" = "us-central1-a"
        },
        "two" = {
            "name" = "two",
            "machine" = "n1-standard-2",
            "os" = "projects/centos-cloud/global/images/centos-8-v20191018",
            "zone" = "us-central1-b"
        }
    }
}

resource "google_compute_instance" "default" {
    for_each = var.hostnames
    name         = each.value.name
    machine_type = each.value.machine
    zone         = each.value.zone

    boot_disk {
        initialize_params {
            image = each.value.os
        }
    }

    scratch_disk {
    }

    network_interface {
        network = "default"
    }
}

Terraform plan output:

Terraform will perform the following actions:

  # google_compute_instance.default["one"] will be created
  + resource "google_compute_instance" "default" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + deletion_protection  = false
      + guest_accelerator    = (known after apply)
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + machine_type         = "n1-standard-1"
      + metadata_fingerprint = (known after apply)
      + name                 = "one"
      + project              = (known after apply)
      + self_link            = (known after apply)
      + tags_fingerprint     = (known after apply)
      + zone                 = "us-central1-a"

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + mode                       = "READ_WRITE"
          + source                     = (known after apply)

          + initialize_params {
              + image  = "projects/coreos-cloud/global/images/coreos-stable-2247-5-0-v20191016"
              + labels = (known after apply)
              + size   = (known after apply)
              + type   = (known after apply)
            }
        }

      + network_interface {
          + address            = (known after apply)
          + name               = (known after apply)
          + network            = "default"
          + network_ip         = (known after apply)
          + subnetwork         = (known after apply)
          + subnetwork_project = (known after apply)
        }

      + scheduling {
          + automatic_restart   = (known after apply)
          + on_host_maintenance = (known after apply)
          + preemptible         = (known after apply)

          + node_affinities {
              + key      = (known after apply)
              + operator = (known after apply)
              + values   = (known after apply)
            }
        }

      + scratch_disk {
          + interface = "SCSI"
        }
    }

  # google_compute_instance.default["two"] will be created
  + resource "google_compute_instance" "default" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + deletion_protection  = false
      + guest_accelerator    = (known after apply)
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + machine_type         = "n1-standard-2"
      + metadata_fingerprint = (known after apply)
      + name                 = "two"
      + project              = (known after apply)
      + self_link            = (known after apply)
      + tags_fingerprint     = (known after apply)
      + zone                 = "us-central1-b"

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + mode                       = "READ_WRITE"
          + source                     = (known after apply)

          + initialize_params {
              + image  = "projects/centos-cloud/global/images/centos-8-v20191018"
              + labels = (known after apply)
              + size   = (known after apply)
              + type   = (known after apply)
            }
        }

      + network_interface {
          + address            = (known after apply)
          + name               = (known after apply)
          + network            = "default"
          + network_ip         = (known after apply)
          + subnetwork         = (known after apply)
          + subnetwork_project = (known after apply)
        }

      + scheduling {
          + automatic_restart   = (known after apply)
          + on_host_maintenance = (known after apply)
          + preemptible         = (known after apply)

          + node_affinities {
              + key      = (known after apply)
              + operator = (known after apply)
              + values   = (known after apply)
            }
        }

      + scratch_disk {
          + interface = "SCSI"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.
Organize answered 29/10, 2019 at 0:12 Comment(1)
So I need to convert list to map, as I undersand it change the index of list to key and it's gonna be map.Watercool
B
2

Yes this is possible, you need to use the for expression in Terraform to achieve this though, the for loop converts the list of objects into a value in which Terraform can loop over using for_each, without the for expression, Terraform cannot loop over the list of objects because there is no key value for Terraform to reference.

Below is a is a simple example:

# variables.tf
variable "nsg_rules" {
  description = "list of maps consisting of nsg rules"
  type = list(object({
    access                       = string
    destination_address_prefixes = list(string)
    destination_port_ranges      = list(string)
    direction                    = string
    name                         = string
    priority                     = number
    protocol                     = string
    source_address_prefixes      = list(string)
    source_port_range            = string
  }))
  default = [
    {
      access                       = "Deny"
      destination_address_prefixes = ["10.10.1.0/24", "10.10.2.0/24"]
      destination_port_ranges      = ["80"]
      direction                    = "Inbound"
      name                         = "DenyHTTPInbound"
      priority                     = 100
      protocol                     = "*"
      source_address_prefixes      = ["10.0.0.0/24"]
      source_port_range            = "*"
    },
    {
      access                       = "Deny"
      destination_address_prefixes = ["10.10.10.0/24", "10.10.11.0/24"]
      destination_port_ranges      = ["22"]
      direction                    = "Inbound"
      name                         = "DenySSHInbound"
      priority                     = 200
      protocol                     = "*"
      source_address_prefixes      = ["10.0.0.0/24"]
      source_port_range            = "*"
    }
  ]
}

Use the for expression wrapped in curl brackets to convert the variable value, each maps key will be given the value of each maps name input, for example the first map would be given a key of "DenyHTTPInbound"

resource "azurerm_network_security_rule" "nsg_rules" {
  for_each = { for rule in var.nsg_rules : rule.name => rule }
 
  access                       = each.value.access
  destination_address_prefixes = each.value.destination_address_prefixes
  destination_port_ranges      = each.value.destination_port_ranges
  direction                    = each.value.direction
  name                         = each.value.name
  network_security_group_name  = azurerm_network_security_group.nsg.name
  priority                     = each.value.priority
  protocol                     = each.value.protocol
  resource_group_name          = azurerm_resource_group.rg.name
  source_address_prefixes      = each.value.source_address_prefixes
  source_port_range            = each.value.source_port_range
}

ref: https://jimferrari.com/2023/02/13/loop-through-list-of-maps-objects-with-terraform/

Bedight answered 15/2, 2023 at 10:45 Comment(0)
Y
0

I took reference from the for_each example above and used below. This did not work for me, link below has details. Terraform for_each on custom registry

module "az"{
source="./modules/az"
vpc_id = module.vpc.vpc_id
for_each = toset(keys({for i,v in var.az_sub: i => v}))
availability_zone = var.az_sub[each.value]["az"]
public_cidr_block = var.az_sub[each.value]["public_cidr_block"]
private_cidr_block  =var.az_sub[each.value]["private_cidr_block"]
}

Error:module.az is object with 2 attributes If I replace for_each with actual values, the module is working perfectly.

Yonne answered 28/6, 2022 at 23:5 Comment(0)
P
0

This is a pretty confusing structure in terraform, but given:

variable services {
  type        = list(map(string))
  description = "services"
  default     = [
    {
      name          = "abc"
      target_port   = 9097
      health_port   = 3780
      health_code   = 200
      protocol      = "HTTP"
    },
    {
      name          = "def"
      target_port   = 8580
      health_port   = 3580
      health_code   = 200
      protocol      = "HTTP"
    },
    {
      name          = "ghi"
      target_port   = 80
      health_port   = 3680
      health_code   = 200
      protocol      = "HTTP"
    }
  ]
}

You iterate through resource as so:

resource "aws_lb_listener" "listeners" {
  for_each   = {
    for service in var.services: service.name => service
  }

  load_balancer_arn = aws_lb.internal.arn
  port              = each.value.target_port
  protocol          = each.value.protocol
  tags              = var.tags

You reference ANOTHER resource which uses a list of objects as so:

  default_action {
    type              = "forward"
    target_group_arn  = aws_lb_target_group.target_groups[each.value.name].id
  }

resource "aws_lb_target_group" "target_groups" {
  for_each   = {
    for service in var.services: service.name => service
  }

Notice since aws_lb_target_group is also using an array of maps, you must specify the map property when referencing from another resource, as shown above! That could trip people up.

And if you want to output the list of objects, you do as so:

output "alb_listener_arns" {
  value   = values(aws_lb_listener.listeners)[*].arn
}

output "target_group_ids" {
  value   = values(aws_lb_target_group.target_groups)[*].id
}
Pyromania answered 16/1, 2023 at 6:22 Comment(0)
K
0

I've kept together some snippets for remembering how to do do various list comprehension and looping in TF. Some examples, e.g. if given a users = { "foo": { a = true, b = []}} in a tfvar file.

Make from map into a list:

output "reminderHowToListComprehensionEach" {
   value = [for e in var.users : e]
}

Keep as a map:

output "reminderHowToListComprehensionKeyValue" {
    value = {for k,v in var.users : k => v}
}

Another way

output "reminderHowToListComprehensionKey" {
    value = [for k,v in var.users : k]
}

And more useful probably, filtering the map:

output "reminderHowToListComprehensionValueWithConditional" {
    value = [for k,v in var.users : v if !v["xyz_bool_attribute"]]
}

See also:

Kirwin answered 15/9, 2023 at 8:8 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.