How to monitor EC2 instances by memory?

Question

Using Cloudwatch you can monitor your EC2 instances by several criteria, such as network usage, CPU usage, and so on…

Unfortunately, there is no metric for memory consumption. First of all, just out of curiosity, I would like to know, why? Can anybody explain why it is possible to, e.g., monitor CPU usage, but not memory usage? At least to me, that's not obvious.

And then, my actual question: Okay, given that Cloudwatch doesn't allow monitoring the EC2 instances' memory usage - what is the alternative? How should I setup an alarm if, e.g. > 80% of the memory of an instance is being used?

Ethnology

It's been 5 years since the question and AWS still doesn't have an easy and straight-forward way to monitor memory usage of EC2 instances :( What a shame. – Exarch

Answer 1

Memory and Disk specific statistics require AWS to monitor at the OS level rather than the host level, so that is why they leave it out by default. It will probably be added at some point but since it has been on the wish list for about 7 years, we can assume it's a very low priority item.

The recommended way to monitor memory usage is to create a custom Cloudwatch metric by using your own monitoring scripts on your instances. AWS have published documentation on how to achieve this on Linux instances with a set of (unsupported) scripts.

Once your instances are publishing the custom metrics, you will be able to attach alarms to them in CloudWatch.

Flyblown

CloudWatch recently released a collectd plugin that can publish collectd metrics to Cloudwatch and collectd can read memory usage (among many other things) aws.amazon.com/blogs/aws/new-cloudwatch-plugin-for-collectd – Hotze

Arguably, about every other third-party solution does this out of the box. Oh, and Azure and GCP respective monitoring agents also do this. AWS just doesn't give a single F* about this issue. – Merilynmeringue

Update for 2023: The technologies used by this answer are deprecated. See docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html (that article has a link to the replacement technology). – Jockey

Notably, even with CloudWatch agent collecting these metrics, they don't show up in the monitoring tab for the instance in the EC2 console. So be prepared for crappy UX... – Ghazi

Answer 2

Well, now the new CloudWatch agent can collect metrics like memory and disk usage, see the docs.

With this you can monitor this kind of metrics, but you will need to install and configure the agent in the instances.

Cockalorum

This is super awesome and helped me get my memory metrics that I wanted and AWS doesn't provide natively! Thanks! – Handbreadth

Answer 3

Well by default the AWS Ec2 does not provide memory metrics. So we have to install the cloud watch agent separately and then get the metrics

Steps:

1. Download the cloud watch agent from s3

   sudo wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

2. Install the cloud watch agent

   sudo rpm -U ./amazon-cloudwatch-agent.rpm

3. We need to write a config file in which we will specify the metrics we need to put.

   sudo vi /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

You need to insert the below snippet inside the amazon-cloudwatch-agent.json File

{
    "metrics": {
        "metrics_collected": {
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ],
                "metrics_collection_interval": 30
            }
        }
    }
}

4. Then after specifying the metric then we just need to start the cloud watch agent

   sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

5. You can go and test out if the metric is coming or not in the cloud watch page

Check:

Cloudwatch console --> All metrics --> under custom namespace --> CW Agent --> host --> ip of the instance server

Now you can view the memory usage and monitor the usage

AWS Documentation: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html

Rhinoceros

tested working, which should be the answer – Beeler

The custom namespace is CWAgent. If you don't see it, make sure you have added the Cloudwatch policy to the running instance (docs). You can also check the logs for any issue you may have: /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log or /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log – Fredella

make sure that following policy is also there CloudWatchAgentServerPolicy without it you will not be able to send data to CloudWatch – Uhf

Working in 2024 – Chamberlain

Answer 4

As of 2024, it has become relatively convenient to install and use the CloudWatch agent for EC2 instances. Make sure that the SSMAgent is installed on your instance. It's usually preinstalled: https://docs.aws.amazon.com/systems-manager/latest/userguide/ami-preinstalled-agent.html

Then, you can select your instance in the EC2 dashboard, switch to the "Monitoring" tab and select "Configure CloudWatch agent". The UI will guide you through the steps. You might need to adjust your instance's IAM role to include the CloudWatchAgentServerRole and AmazonSSMManagedInstanceCore IAM policy during the process. The additional graphs will also show in the monitoring tab.

Jocko

In 2024 this is the right answer, though the others are still helpful. Note that from "Instance > Monitoring" there's a toggle to turn on for "Include metrics in the CWAgent namespace" that should be turned on, and a button to the right of it for "Manage detailed monitoring". That's where you enable the CWAgent (today). Warning: turning this one incurs extra costs for CloudWatch – Harms