how to add crontab scheduler through user-data script in aws ec2?

Question

I am trying to add a crontab so that I can get the cloudwatch metrics of diskspace used and disk space utilization every 5 minutes through a user data script. Below is my user-data script:

   #!/bin/bash
sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA.x86_64
curl https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O
unzip CloudWatchMonitoringScripts-1.2.2.zip && rm CloudWatchMonitoringScripts-1.2.2.zip && cd aws-scripts-mon

crontab<<EOF
*/1 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl -mem-util --mem-used --mem-avail --swap-util --disk-space-util --disk-space-used --disk-space-avail --memory-units=megabytes --disk-path=/dev/xvda1 --from-cron
EOF

./mon-put-instance-data.pl -mem-util --mem-used --mem-avail --swap-util --disk-space-util --disk-space-used --disk-space-avail --memory-units=megabytes --disk-path=/dev/xvda1

All these steps are working properly when running from aws-terminal, also no steps are failing in cloud-init-logs. The first time I am able to get the cloud watch metrics but after that, they don't come through, so it means crontab is not working, how can this be fixed?

Leonorleonora

This actually worked for a user data script: https://mcmap.net/q/1170428/-crontab-in-aws-cloudformation-userdata – Remunerative

Answer 1

Don't use -e as that is edit mode, you want something like:

  (crontab -l 2>/dev/null || echo ""; echo "*/5 * * * * /path/to/job -with args") | crontab -

Update: Added empty echo pipe, to avoid user data stopping due to failure in missing crontab list for root (default running user for userdata), which would return non-zero and fail the script.

Note: Also be aware that Userdata by default runs only once, not every time the instance is rebooted. So changing userdata when the instance is stopped doesn't let you test different ways unless you modify as per this document. If set to run each time, the script above would also concatenate the same rule over and over to crontab!

Morphia

remove ) character or add ( before echo – Berberine

i didn't get from here what to change in my script – Leonorleonora

Answer 2

You can also use heredoc to achieve this in a cleaner way

crontab<<EOF
* * * * * script.sh
EOF

And if you want to append to the existing crontab, do the below

crontab<<EOF
$(crontab -l)
* * * * * script2.sh
EOF

Now list the crontab using

crontab -l

Also, the man page says that each user can have their own crontab, and though these are files in /var/spool/cron, they are not intended to be edited directly.

e.g. if you are creating cron as root user, the corresponding user's cron file would be

/var/spool/cron/root

Please see below in detail

[root@localhost ~]# crontab -l
no crontab for root

[root@localhost ~]# crontab<<EOF
*/5 * * * * script1.sh
EOF
[root@localhost ~]# crontab -l
*/5 * * * * script1.sh

[root@localhost ~]# crontab<<EOF
*/10 * * * * script2.sh
EOF
[root@localhost ~]# crontab -l
*/10 * * * * script2.sh

[root@localhost ~]# crontab<<EOF
$(crontab -l)
* * * * * script3.sh
EOF
[root@localhost ~]# crontab -l
*/10 * * * * script2.sh
* * * * * script3.sh

[root@localhost ~]# crontab<<EOF
$(crontab -l)
* * * * * script4.sh
EOF
[root@localhost ~]# crontab -l
*/10 * * * * script2.sh
* * * * * script3.sh
* * * * * script4.sh

[root@localhost ~]# cat /var/spool/cron/root
*/10 * * * * script2.sh
* * * * * script3.sh
* * * * * script4.sh
[root@localhost ~]#

In your case, it would look like

#!/bin/bash
sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA.x86_64
curl https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O
unzip CloudWatchMonitoringScripts-1.2.2.zip && rm CloudWatchMonitoringScripts-1.2.2.zip && cd aws-scripts-mon

crontab<<EOF
echo $'i*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl -mem-util --mem-used --mem-avail --swap-util --disk-space-util --disk-space-used --disk-space-avail --memory-units=megabytes --disk-path=/dev/xvda1 --from-cron\E:x\n'
EOF

./mon-put-instance-data.pl -mem-util --mem-used --mem-avail --swap-util --disk-space-util --disk-space-used --disk-space-avail --memory-units=megabytes --disk-path=/dev/xvda1

Berberine

I tried the approach from here but still it didn't work if i pass it through user-data script although it did work when i tried from terminal – Leonorleonora

maybe you are doing it wrong. Send the user data script in a neat way. use the above syntax as is. if it works on the terminal it must work in the script. see my updated answer and I have updated your question code block too. – Berberine

i have updated the change even if you also change this to:- crontab<<EOF */1 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl -mem-util --mem-used --mem-avail --swap-util --disk-space-util --disk-space-used --disk-space-avail --memory-units=megabytes --disk-path=/dev/xvda1 --from-cron EOF only the echo line is wrong in your case , rest line are same you have to remove and yes it is still not working – Leonorleonora

Answer 3

This post is old but i spent way too long finding this solution.

I was trying to automate adding a cronjob to an ec2 so that i didnt have to manually schedule the job or add the script. all of this was done through user data in the cloudformation. my solution required that i used Systems manager. U may need to change system manager to ec2-user.

To do this I used:

crontab -u ssm-user /path/to/script

I tried like 5 other solutions that worked in the console but didnt work in userdata. I really hope this helps others automate their deployments in the future

Caseose

how to use heredoc syntax with this, i want to add this programmatically from bash script – Classicist

Answer 4

for adding lines in the crontab i used these lines in my user-data script and it worked fine for me

sudo su

crontab -u ec2-user -l

echo -e "*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl -mem-util --mem-used --mem-avail --swap-util --disk-space-util --disk-space-used --disk-space-avail --memory-units=megabytes --disk-path=/dev/xvda1 --from-cron " | crontab -u ec2-user -

exit

the linked which help me to find the solution is https://serverfault.com/questions/296949/programmatically-add-entry-to-users-crontab

Leonorleonora

how to use heredoc syntax with this, i want to add this programmatically from bash script – Classicist

Answer 5

I had similar issue like others are saying here that these commands work fine from CLI but when added through userdata not working. After several hours of head scratching search finally realized I had to add line "#!/bin/bash" before crontab setup commands. This fixed my problem, some very well experienced people might know this already but if you are bit new to this realm may not know this. My script look like below.

# !/bin/bash
sudo su 
echo '*/1 * * * * <path-to-python> <path-to-python-script>' > /tmp/mycrontab.txt
sudo -u ubuntu bash -c 'crontab /tmp/mycrontab.txt'

Dichogamy

Answer 6

This is pretty old but nobody has posted this yet and it is perhaps the more correct solution.

echo '*/1 * * * * ec2-user ~/aws-scripts-mon/mon-put-instance-data.pl -mem-util --mem-used --mem-avail --swap-util --disk-space-util --disk-space-used --disk-space-avail --memory-units=megabytes --disk-path=/dev/xvda1 --from-cron' > /etc/cron.d/mon-put-instance-data

https://linuxize.com/post/scheduling-cron-jobs-with-crontab/#system-wide-crontab-files

Totality