Terraform stalls while trying to get IP addresses of multiple instances?

Question

So I am using terraform to provision ec2 instances and openstack instances. I am trying to reference the IP addresses of the instances I am creating because I need to run commands that use them (to set up consul). However after adding references to these variables terraform just stalls out and does absolutely nothing after I run a terraform apply or terraform plan:

Here is a sample of the resource block for what I am trying to run:

resource "aws_instance" "consul" {
  count = 3
  ami = "ami-ce5a9fa3"
  instance_type = "t2.micro"
  key_name = "ansible_aws"
  tags {
    Name = "consul"
  }

  connection {
    user = "ubuntu"
    private_key="${file("/home/ubuntu/.ssh/id_rsa")}"
    agent = true
    timeout = "3m"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y curl",
      "echo ${aws_instance.consul.0.private_ip} >> /home/ubuntu/test.txt",
      "echo ${aws_instance.consul.1.private_ip} >> /home/ubuntu/test.txt",
      "echo ${aws_instance.consul.2.private_ip} >> /home/ubuntu/test.txt"
    ]
  }
}

Update: so I tried running a similar command with my openstack cloud and got the same problem:

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y curl",
      "echo ${openstack_compute_instance_v2.consul.0.network.0.fixed_ip_v4}",
      "echo ${openstack_compute_instance_v2.consul.1.network.1.fixed_ip_v4}",
      "echo ${openstack_compute_instance_v2.consul.2.network.2.fixed_ip_v4}"
    ]
  }

So I have found if instead I use only one of the IP addresses then the other instances wont even be created until my first instance is created, like in the block below:

  provisioner "remote-exec" {
    inline = [
      "echo ${openstack_compute_instance_v2.consul.0.network.0.fixed_ip_v4}",
    ]
  }

I need all my instances to be created at the same time and have access to the IP addresses of all the other instances created as soon as they are created.

Fieldpiece

At a very quick glance that looks like that should do exactly what you want. What's wrong with it? – Toadflax

That shouldn't be a problem. The aws_instance.consul... part is saying to get the output from an aws_instance resource called consul. What happens if you remove the count and have a single "echo ${aws_instance.consul.private_ip} >> /home/ubuntu/test.txt"? – Toadflax

So I tried doing something very similar to this with my openstack cloud and got the same exact problem. after running terraform apply my terminal starts doing absolutely nothing. After waiting for it to do something for like 5 min I did a double control + c to exit and was then presented with a wall of errors. – Fieldpiece

So when I have a single instance and run "echo ${aws_instance.consul.private_ip}" it works fine, but with multiple instances terraform will just do absolutely nothing after I run a terraform apply or terraform plan – Fieldpiece

Answer 1

So taking a longer look at this it looks like it is actually an issue with Terraform where it gets locked into a dependency cycle.

That same issue also includes a workaround for this by using a null_resource to then connect to all of the provisioned instances and run a script against them.

For your use case you could use something like this:

resource "aws_instance" "consul" {
  count = 3
  ami = "ami-ce5a9fa3"
  instance_type = "t2.micro"
  key_name = "ansible_aws"
  tags {
    Name = "consul"
  }
}

resource "null_resource" "configure-consul-ips" {
  count = 3

  connection {
    user = "ubuntu"
    private_key="${file("/home/ubuntu/.ssh/id_rsa")}"
    agent = true
    timeout = "3m"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y curl",
      "sudo echo '${join("\n", aws_instance.consul.*.private_ip)}' > /home/ubuntu/test.txt"
    ]
  }
}

This should bring up 3 instances and then connect to them, install curl and then create a file with a new line separated list of the private IP addresses of the 3 instances.

Toadflax

I did notice that null resources cant run all the commands my other servers do, so essentially it is not reconnecting to my servers created in the previous resource block. Another solution was to create another server after the first 3 and run remote bash commands from that new server. It was pretty easy to work that into my terraform script, since I needed to create a load balancer instance anyway. So that 4th instance runs the commands I need and has access to the IP addresses of the first 3 servers. – Fieldpiece