How can I be alerted when an EBS instance is running out of space?

Question

I'm running a wordpress off of AWS, and I can't figure out how to monitor if the volume is running out of capacity. There are many options for monitoring other things, but I just want to know when we run low on space.

Swarthy

Amazon is just bad at doing basic things. – Commutable

Answer 1

There are no built in metrics for disk space - Because Amazon is looking at this 'from the outside', they don't know what you've done with an EBS volume: it could be part of a raid set, formatted in some exotic format etc.

Amazon has a system called CloudWatch that can be used to monitor AWS systems and resources. Luckily, CloudWatch supports custom metrics: you can use the CloudWatch APIs to add any data you want (and then create alerts off them).

Amazon provides some sample scripts that will publish (among other things) disk space utilisation to Cloudwatch. Do be aware that there is a small charge associated with using custom metrics - $0.50 per metric per month.

Chaste

But you get 10 free ones a month. – Garfish

Keep in mind that when you use these scripts you may want to provide --aws-credential-file=/home/ubuntu/monitoring-scripts/awscreds.template file filled with your AWS key and secret. That's something which could have been emphasized better in the documentation. – Parenthesis

@DimitryK, even better than providing a credentials file like you suggest, is to use an IAM Role. If you launch your instance associated with an Instance Profile which is associated with an IAM Role, those scripts (and the AWS CLI and the AWS SDKs and many other tools) are able to automatically obtain temporary credentials -- no need to deploy and manage (rotate, revoke, etc) credentials files manually anymore. – Eldridgeeldritch

The sample script link is now being redirected to 'What Are Amazon CloudWatch, Amazon CloudWatch Events, and Amazon CloudWatch Logs?' – Ilia

For details on how to setup metric on Windows Server 2016 see: #37441725 – Antiquity

Is it just me or is $0.50 for a single piece of text data absolutely and utterly absurd? This is basically the definition of rent seeking. I'd write a bash script to check df metrics and send an email or ping some endpoint when they get above a certain %, or use something like Prometheus. – Irrecusable

Answer 2

Selected answer will no longer work because:

The CloudWatch monitoring scripts are deprecated. We recommend that you use the CloudWatch agent to collect metrics and logs. - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html

I found this guide helpful at setting up custom cloudwatch metric such as disk monitoring: https://marbot.io/blog/monitoring-ec2-disk-usage.html

The only thing that the above is missing is setting up IAM role and attaching it to the EC2, guide for this is here (do this first, then follow the blog post): https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html

Per recommendation, summary steps below:

1. Set up IAM role per link here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html
2. Attach it to EC2 you want to monitor
3. ssh into your EC2
4. cd /tmp
5. wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm (to download the agent rpm)
6. sudo rpm -U amazon-cloudwatch-agent.rpm (to install)
7. sudo vi /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json (to edit config)
8. paste the config (in the link, no changes necessary)
9. sudo systemctl restart amazon-cloudwatch-agent (restart agent)
10. tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log (see if its working)
11. Navigate to cloudwatch console and set up alarm based on custom metric that will show up as CWAgent (thats the name specified in the config, more detail in the link on exact way to set up alarm)

Mccrary

Answer 3

I agree with Frederick Cheung's answer; however, I found this utility for Windows instances: http://www.eleven41.com/labs/cloudwatch-monitor-for-windows/. It runs as a service and will send the disk space and memory usage metrics to CloudWatch for you. From there, you'd just need to set up the alarm in CloudWatch.

Isaacs

Answer 4

I agree with Christopher Hinkle's agreement with Frederick Cheung's answer :)

Instead of using a utility where you still have to setup all the alerts yourself, try using Blue Matador. It will set up all the alerts for you, on all the mount points, on all the servers.

That said, you should also watch a lot more metrics than just the disk space. There's a guide called How to Monitor Amazon EBS with CloudWatch that goes over all the metrics and how you should approach monitoring them (if you don't use Blue Matador to automate them).

Longhorn