How to implement custom health checks for EC2 instances without using an ELB?

Question

Scenario:

• I'm running an EC2 instance behind an auto-scaling group, but I'm not using an ELB.
• Inside the EC2 instance, a docker container with a web server is running.

I would like to add a simple health check that the web server does still respond, so if the docker container goes down, the auto scaling group can replace the instance.

From what I see, only the ELB supports custom health checks. As I don't need an ELB, I wonder if it makes sense to run the health check inside the EC2 instance with a cron job. If the web server does not respond (locally), it can set the health status like this:

export INSTANCE=$(curl http://169.254.169.254/latest/meta-data/instance-id)
export AWS_DEFAULT_REGION=$(curl http://169.254.169.254/latest/dynamic/instance-identity/document|grep region|awk -F\" '{print $4}')
aws autoscaling set-instance-health --instance-id $INSTANCE --health-status Unhealthy

I think, it should work, but it looks a bit complicated, though. Is there a better way to implement custom health checks (without using an ELB)?

Pantheas

Looking at the AWS docs this seems to be the recommended way. -> docs.aws.amazon.com/autoscaling/latest/userguide/… – Mouser

The downside of running a cron on your EC2 instance is the case of your app not starting up correctly, crashing, etc... a case where the EC2 instance passes the instance health check, but your web server isn't responding correctly. For that reason, it would be better to run elsewhere, like perhaps a scheduled lambda function (docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html) Here's a simple example that could be adapted: marcelog.github.io/articles/… – Sibeal

Answer 1

In 2017, there is no direct support from AWS, only the API to set the health of an EC2 instance. Thus, the technique described in the question is the recommended way:

• Implement a custom health check (could be a shell script or whatever you choose) and run it periodically (via cron or whatever you choose)
• Use the autoscaling set-instance-health API to communicate the result to the auto-scaling group

AWS documentation on custom health checks:

If you have custom health checks, you can send the information from your health checks to Auto Scaling so that Auto Scaling can use this information. For example, if you determine that an instance is not functioning as expected, you can set the health status of the instance to Unhealthy. The next time that Auto Scaling performs a health check on the instance, it will determine that the instance is unhealthy and then launch a replacement instance.

Use the following set-instance-health command to set the health state of the specified instance to Unhealthy:

aws autoscaling set-instance-health --instance-id i-123abc45d –-health-status Unhealthy

Pantheas

The thing I don't quite understand is why bother setting the health status to unhealthy vs just terminating it directly. The latter would save a little time. Perhaps for tracking, so you can see the reason the instance was terminated was because it was unhealthy. – Sibeal

@MichaelRush As far as I know, there are 2 extra reasons: Either when using additional lifecycle hooks in the autoscaling group, or in situations when you still want a startup grace period. – Emilia