terraform minor aws user_data change "forces replacement" - What Is The Best Resolution?

Question

I have made a minor change (comments only) to my AWS EC2 user_data bootstrap bash script. Terraform has detected this, and now wants to replace the EC2 with a new one. But this is a live service, and I'd rather not have to take it down, backup all the data, build a new server, and then restore the data - just to keep terraform happy!

Is there a simpler way of fixing this, by somehow bringing the code/statefile/aws in line so that I can use terraform to make other changes to the environment, but not have to worry about blowing away this server?

I've even tried editing the user data from the AWS console to try to bring it inline with what I think the changes should be, but this hasn't worked.

For now I'm using -target for specific changes, but this is probably not sustainable in the long run.

Mozzetta

Answer 1

If you want to prevent EC2 instance replacement when you changed your user_data you need to set user_data_replace_on_change = false:

user_data_replace_on_change - (Optional) When used in combination with user_data or user_data_base64 will trigger a destroy and recreate when set to true. Defaults to false if not set.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#user_data_replace_on_change

This option was initially implemented here

Slop

Answer 2

You can instruct the provider to ignore the changes to the user_data argument for that resource with the lifecycle meta-argument:

resource "aws_instance" "this" {
  # ...

  lifecycle {
    ignore_changes = [user_data]
  }
}

Diuretic

Thanks, I'd not thought of that one, and it is certainly better than -target, and I guess once the user_data does change for real I can just take the lifecycle statement out. My only problem is that I have a module for EC2's and if I changed the module that would affect all EC2s, not just the one I've currently got a problem with. – Mozzetta

An update, I tried to pass in user_data as a var to the module so I could ignore it for just the affected "aws_instance", but that failed spectacularly as lifecyle statements can't be dynamic (include vars). So that pretty much rules that out as an option. – Mozzetta

@PBurke Understood, and that is certainly a limitation, but this is currently your codified option here. – Diuretic

Answer 3

Per this issue https://github.com/hashicorp/terraform-provider-aws/issues/23315 the behavior now depends on https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#user_data_replace_on_change which defaults to false so that it will not force replacement.

I had the issue however that I wanted the user_data script to execute on a Terraform apply, so I created a script that would do this on update of that field.

Note that this expects to be run in an environment with 1) an AWS_PROFILE set with permissions to execute a Session Manager command, 2) Session Manager Plugin installed, and 3) the target instance must have SSM Agent installed (see this link)

In the Terraform:

resource "null_resource" "update-user-data" {
  depends_on = [aws_instance.this]

  triggers = {
    user_data_base64 = base64encode(data.template_file.user_data.rendered)
  }
  provisioner "local-exec" {
    command     = file("${path.module}/run-remote-shell-script.sh")
    interpreter = ["bash", "-c"]
    environment = {
      INSTANCE_NAME = var.name
    }
  }
}

run-remote-shell-script.sh:

#!/usr/bin/env bash

test -n "$INSTANCE_NAME" || (echo missing INSTANCE_NAME; exit 1)
test -n "$AWS_PROFILE" || (echo missing AWS_PROFILE; exit 1)

INSTANCE_ID=$(aws ec2 describe-instances \
 --filters Name=tag:Name,Values=$INSTANCE_NAME \
  --query 'Reservations[].Instances[].InstanceId' --output text)

echo $INSTANCE_ID

aws ec2 wait instance-running --instance-ids $INSTANCE_ID;
aws ssm send-command --instance-ids $INSTANCE_ID \
   --document-name AWS-RunShellScript --parameters '{"commands": ["sudo bash /var/lib/cloud/instance/user-data.txt"]}'
sleep 1

exit 0

Marilumarilyn