How to fire EC2 instances and upload/run a startup script on each of them?

Question

I want to automate the launch of a set of Linux EC2 instances.

Basically, I want to write a script/program that would :

• Instantiate N occurrences of a given AMI of mine.
• For each started instance, it would upload a customized script and let the script run into the instance.

Using VMWare, I would typically do that using vmrun or the Vix SDK.

What are the options in Amazon AWS/EC2?

Willaims

Answer 1

The answer depends a bit on what AMI you are running as the features provided are entirely AMI dependent.

The Amazon Linux AMIS and the official Ubuntu AMIs have the cloud-init package installed. This has a number of ways you can trigger startup actions, but the one that matches your request most closely (and my favorite because I invented it) is the concept of a user-data script.

You can simply pass any script (starting with the two characters #!) as the user-data when starting the EC2 instances. It will be run as root on the first boot of the instance.

For a specific example of how this works, I use this exact technique in my recent article: Uploading Known ssh Host Key in EC2 user-data Script

You also wanted to run more than one EC2 instance with the same script. The ec2-run-instances command and the related APIs and web console allow you to specify any number of instances to start with the same user-data. For example:

ec2-run-instances            \
  --instance-count 10        \
  --user-data-file $MYSCRIPT \
  --key $USER                \
  $SOMEAMI

If you are currently running an AMI that does not have cloud-init installed, you could do one of:

• Switch to an AMI that has cloud-init installed, or

• Build a custom version of your AMI that has cloud-init installed, or

• Write a more complicated wrapper script that makes a record of all of the instance ids after they are kicked off, waits for all of the instances to move to the running state, waits for the sshd to accept connections, uploads your startup script to each instance, and runs the startup script on each instance.

Phenobarbitone

One of the nice things with SO is that by the time you start finding some pointers, the guy whose name starts to appear in all your google queries answered your question personally. Awesome :-) – Willaims

I would avoid option 3 of the above answer. I have run into two different pitfalls with this approach. 1) An instance has an IP address you have previously seen, and the known_hosts entry doesn't match. 2) The machine uploading & controlling all the instances & startup scripts gets overloaded, and stuff breaks – Unjust

Best quote of SO: "and my favorite because I invented it" – Julianajuliane

Hrishikesh Kale: It depends which Debian AMIs you are running. The official Debian AMIs are being improved. They currently support user-data scripts and should eventually have full cloud-init software. – Phenobarbitone

Any way to do this with spot requests? – Triclinium

Sam: Yes, you can specify user-data to pass to spot instances. If it is an AMI that runs user-data scripts, then it will work as you desire. – Phenobarbitone

Note as of 6/14/2019, the CLI syntax for the user data flag is slightly different: --user-data file://my_script.sh instead of --user-data-file my_script.sh – Preamble

Answer 2

I have a tutorial to run the script in the "cloud-init" that runs each time AWS EC2 is startup.

• to set configuration file (AWS CentOS6) and

• run the scripts when you startup the EC2

To set configuration file on Linux you may refer configure cloud-init on AWS Linux.

I personally use AWS VPC/EBS that was setup based on Linux AMI, I didn't touch anything on the configuration file /etc/cloud/cloud.cfg but my boot script in the cloud-init runs well.

Mok