Where to find logs for a cloud-init user-data script?

Question

I'm initializing spot instances running a derivative of the standard Ubuntu 13.04 AMI by pasting a shell script into the user-data field.

This works. The script runs. But it's difficult to debug because I can't figure out where the output of the script is being logged, if anywhere.

I've looked in /var/log/cloud-init.log, which seems to contain a bunch of stuff that would be relevant to debugging cloud-init, itself, but nothing about my script. I grepped in /var/log and found nothing.

Is there something special I have to do to turn logging on?

Division

did you check /var/log/boot.log ? CloudInit runs at boot time, so your 'prints' and errors should appear somewhere in that log – Montgolfier

All I get in boot.log is this: "cloud-init start running: Mon, 09 Sep 2013 06:39:55 +0000. up 44.94 seconds" – Division

what's is the script you're passing? If it doesn't have any outputs then it is normal for you not to see anything. – Montgolfier

@cjdcordeiro, the script does some apt-get commands, downloads and builds Redis and installs some R packages. All of that should produce some output. For good measure, I threw in some echo statements and tried with and without "set -x" with no discernible results. – Division

So, @cbare, this comment is going to be big, read it in the answers bellow. – Montgolfier

@Division want to update the answer? – Jackboot

Answer 1

so I tried to replicate your problem. Usually I work in Cloud Config and therefore I just created a simple test user-data script like this:

#!/bin/sh

echo "Hello World.  The time is now $(date -R)!" | tee /root/output.txt

echo "I am out of the output file...somewhere?"

yum search git    # just for fun

ls

exit 0

Notice that, with CloudInit shell scripts, the user-data "will be executed at rc.local-like level during first boot. rc.local-like means 'very late in the boot sequence'" After logging in into my instance (a Scientific Linux machine) I first went to /var/log/boot.log and there I found:

Hello World. The time is now Wed, 11 Sep 2013 10:21:37 +0200! I am

out of the file. Log file somewhere? Loaded plugins: changelog, kernel-module, priorities, protectbase, security, : tsflags, versionlock 126 packages excluded due to repository priority protections 9 packages excluded due to repository protections ^Mepel/pkgtags
| 581 kB 00:00

=============================== N/S Matched: git =============================== ^[[1mGit^[[0;10mPython.noarch : Python ^[[1mGit^[[0;10m Library c^[[1mgit^[[0;10m.x86_64 : A fast web interface for ^[[1mgit^[[0;10m

...

... (more yum search output)

...

bin etc lib lost+found mnt proc sbin srv tmp var

boot dev home lib64 media opt root selinux sys usr

(other unrelated stuff)

So, as you can see, my script ran and was rightly logged. Also, as expected, I had my forced log 'output.txt' in /root/output.txt with the content:

Hello World. The time is now Wed, 11 Sep 2013 10:21:37 +0200!

So...I am not really sure what is happening in you script. Make sure you're exiting the script with

exit 0   #or some other code

If it still doesn't work, you should provide more info, like your script, your boot.log, your /etc/rc.local, and your cloudinit.log. btw: what is your cloudinit version?

Montgolfier

I verified that this works on a CentOS AMI (ami-0027bf69) and does not work on the standard Ubuntu 13.04 AMI (ami-c30360aa). On Ubuntu, the output.txt gets written as expected, I changed the yum line to "apt-get install -y git". This AMI runs with BOOTLOGD_ENABLE=No, so no boot.log. Thanks, @cjdcordeiro! – Division

Hmm, I guess I'll call it verified that logging is either intentionally turned off or broken on the Ubuntu 13.04 AMI and/or in Cloud-Init 0.7.2. Thanks for the help! – Division

Yeah it is probably a Ubuntu AMI related question. – Montgolfier

Answer 2

The default location for cloud init user data is already /var/log/cloud-init-output.log, in AWS, DigitalOcean and most other cloud providers. You don't need to set up any additional logging to see the output.

Jackboot

Yes - see my comment on the my answer above, at the time this question was asked, that wasn't yet the default in all distributions. – Scarecrow

@michael Yep. I write new answers when things change and old answers don't apply anymore to keep Stack Overflow relevant. You're welcome to maintain your own answer to do the same. – Jackboot

Answer 3

You could create a cloud-config file (with "#cloud-config" at the top) for your userdata, use runcmd to call the script, and then enable output logging like this:

output: {all: '| tee -a /var/log/cloud-init-output.log'}

Scarecrow

Thanks Michael. I'll try that and report back when I start the next round. – Division

This file exists on my centos box. – Smith

@can - I recently noticed this myself when using an AmazonLinux AMI. I was using an Ubuntu AMI at the time I wrote this comment. I guess it got added in to the defaults in some other distributions. Very nice! – Scarecrow

Tested. This works and is extremely useful for debugging. – Chalmers