user-data (cloud-init) script not executing on EC2

Question

my user-data script

#!
set -e -x
echo `whoami`
su root
yum update -y
touch ~/PLEASE_WORK.txt

which is fed in from the command:

ec2-run-instances ami-05355a6c -n 1 -g mongo-group -k mykey -f myscript.sh -t t1.micro -z us-east-1a

but when I check the file /var/log/cloud-init.log, the tail -n 5 is:

[CLOUDINIT] 2013-07-22 16:02:29,566 - cloud-init-cfg[INFO]: cloud-init-cfg ['runcmd']
[CLOUDINIT] 2013-07-22 16:02:29,583 - __init__.py[DEBUG]: restored from cache type DataSourceEc2
[CLOUDINIT] 2013-07-22 16:02:29,686 - cloud-init-cfg[DEBUG]: handling runcmd with freq=None and args=[]
[CLOUDINIT] 2013-07-22 16:02:33,691 - cloud-init-run-module[INFO]: cloud-init-run-module ['once-per-instance', 'user-scripts', 'execute', 'run-parts', '/var/lib/cloud/data/scripts']
[CLOUDINIT] 2013-07-22 16:02:33,699 - __init__.py[DEBUG]: restored from cache type DataSourceEc2

I've also verified that curl http://169.254.169.254/latest/user-data returns my file as intended.

and no other errors or the output of my script happens. how do I get the user-data scrip to execute on boot up correctly?

Gravy

Answer 1

Cloud-init does not accept plain bash scripts, just like that. It's a beast that eats YAML file that defines your instance (packages, ssh keys and other stuff).

Using MIME you can also send arbitrary shell scripts, but you have to MIME-encode them.

$ cat my-boothook.txt
#!/bin/sh
echo "Hello World!"
echo "This will run as soon as possible in the boot sequence"

$ cat my-user-script.txt
#!/usr/bin/perl
print "This is a user script (rc.local)\n"

$ cat my-include.txt
# these urls will be read pulled in if they were part of user-data
# comments are allowed.  The format is one url per line
http://www.ubuntu.com/robots.txt
http://www.w3schools.com/html/lastpage.htm

$ cat my-upstart-job.txt
description "a test upstart job"
start on stopped rc RUNLEVEL=[2345]
console output
task
script
echo "====BEGIN======="
echo "HELLO From an Upstart Job"
echo "=====END========"
end script

$ cat my-cloudconfig.txt
#cloud-config
ssh_import_id: [smoser]
apt_sources:
 - source: "ppa:smoser/ppa"

$ ls
my-boothook.txt     my-include.txt      my-user-script.txt
my-cloudconfig.txt  my-upstart-job.txt

$ write-mime-multipart --output=combined-userdata.txt \
   my-boothook.txt:text/cloud-boothook \
   my-include.txt:text/x-include-url \
   my-upstart-job.txt:text/upstart-job \
   my-user-script.txt:text/x-shellscript \
   my-cloudconfig.txt

$ ls -l combined-userdata.txt 
-rw-r--r-- 1 smoser smoser 1782 2010-07-01 16:08 combined-userdata.txt

The combined-userdata.txt is the file you want to paste there.

More info here:

https://help.ubuntu.com/community/CloudInit

Also note, this highly depends on the image you are using. But you say it is really cloud-init based image, so this applies. There are other cloud initiators which are not named cloud-init - then it could be different.

Ison

ah I see - everything must be MIME encoded. I'm using the generic amazon linux AMI, but I see the cloud-init process in the startup logs, so I assumed it was the same as the ubuntu flavor. – Gravy

You don't have to MIME-encode scripts for ec2-run-instances. Cloud-init may require data in a particular format, but Amazon takes care of that for you - see docs.aws.amazon.com/AWSEC2/latest/UserGuide/… MIME/Base64/gzip encodings are optional - your script just needs to start with #! – Geniegenii

So the user data can start with #cloud-config or #!/bin/bash (or another shell script path). Both will work fine without the mime. If you want both the cloud-config and the script, you need the mime encoding as you described. – Christabella

How does include work? Can another cloud config file be referred? – Adrenal

Answer 2

Actually, cloud-init allows a single shell script as an input (though you may want to use a MIME archive for more complex setups).

The problem with the OP's script is that the first line is incorrect. You should use something like this:

#!/bin/sh

The reason for this is that, while cloud-init uses #! to recognize a user script, the operating system needs a complete shebang line in order to execute the script.

So what's happening in the OP's case is that cloud-init behaves correctly (i.e. it downloads and tries to run the script) but the operating system is unable to actually execute it.

---

See: Shebang (Unix) on Wikipedia

Reinareinald

The documentation on cloud-init is somewhat misleading here, where it states "Begins with: #! or Content-Type: text/x-shellscript when using a MIME archive." I can see how someone might not realize that #! needs to include the path after the shebang. – Ignominy

Not only misleading, but very poor to start off with. There are very few examples of how this has to be put together to work, and then it is broken for some images ? gcloud "just works". Why the lack of (or incorrect) documentation ? There should be an example for file use too, but even the syntax for this is not clear. – Powel

Answer 3

This is a couple years old now, but for others benefit I had the same issue, and it turned out that cloud-init was running twice, from inside /etc/rc3.d . Deleting these files inside the folder allowed the userdata to run correctly:

lrwxrwxrwx  1 root root   22 Jun  5 02:49 S-1cloud-config -> ../init.d/cloud-config
lrwxrwxrwx  1 root root   20 Jun  5 02:49 S-1cloud-init -> ../init.d/cloud-init
lrwxrwxrwx  1 root root   26 Jun  5 02:49 S-1cloud-init-local -> ../init.d/cloud-init-local

Merits

Answer 4

The problem is with cloud-init not allowing the user script to run on the next start-up.

First remove the cloud-init artifacts by executing: rm /var/lib/cloud/instances/*/sem/config_scripts_user

And then your userdata must look like this:

#!/bin/bash 
echo "hello!"

And then start you instance. It now works (tested).

Testa