How to make an EC2 instance automatically reboot when ELB says it's unavailable?

Question

Every now and then Elastic Load Balancer (ELB) kicks out one of my servers for being unavailable. Which is good -- hardly any interruption.

Usually, just rebooting the instance through the AWS Console fixes the problem.

I would like to have my EC2 instance automatically reboot when it becomes unavailable.

What's the best way to do this? (it happens to be a Windows instance, but ideally that shouldn't matter)

Macdonell

Answer 1

I got a cheap EC2 micro instance and wrote a simple script that lists all my servers on EC2, and does a GET on each one to make sure that it's functioning properly. If a server isn't functioning properly, my script reboots it. I have the script running once every 10 minutes.

Woulda been slightly easier if there were a service out there to do this for me, but the system was easy to set up and the micro instance is cheap.

Macdonell

what if your micro instance crashes? – Lavernlaverna

that has happened. right now i get a Pingdom alert. however, ideally, i should have the production servers checking the micro, and the micro checking the production. sigh. – Macdonell

Care to share this script? – Jewbaiting

@Jewbaiting - it's a ColdFusion script, which I'm guessing wouldn't be much use to you. – Macdonell

Answer 2

Use autoscaling group and set it to keep 1 live instance. If your instance crashes it will spawn a new one from predefined instance image. It's easy and convenient.

Mathematical

While this is true, often rebooting the instance in question fixes the issue, and rebooting is much faster than launching a new instance… – Snazzy

Answer 3

Agree with @agmin that a lambda function is a great way to do this. You can configure a CloudWatch alarm to trigger the lambda function via an SNS topic.

I put this implementation together and it seems to be working well. It can be set up and configured entirely via the command line:

https://github.com/jcowley/elb-instance-reboot

Informed

Answer 4

Generally speaking you should have your load balancers set up with an auto scaling group. The auto scaling group will have information about how your server is configured. For example it will have an ami and possible user-data-file that will allow it to be configured without any user interaction. When you have these two services working together the load balancer will terminate your failed instance and start new instance immediately.

Seeing as your using a windows instance you could write a powershell script to configure your server and use CloudInit.NET to execute the script when the server starts. The last part of the script should enable the check on the load balancer.

Portly

This is good information but for various reasons I would still prefer that a simple reboot happen, vs a termination->new instance. – Macdonell

You can try to use the new ec2-describe-instance-status command and write a script that will reboot the instance when its down. This script would also need to add the instance back to the load balancer. Pretty easy script to write. Once the script is complete put it in a crontab and it will work as long as the machine where its running is working. aws.typepad.com/aws/2012/01/ec2-instance-status-checks.html – Portly

Answer 5

Write a lambda function!

It's cheaper than running a micro instance, and you can set it to run at whatever interval you'd like. You also get CloudWatch monitoring with it so you don't need to worry about checking the health of your micro instance separately.

I'm using a ElasticBeanstalk, which offers the describeInstancesHealth endpoint. Alternatively you can use ec2's describeInstanceStatus endpoint.

Then if you detect a bad instance, ec2's rebootInstances let's you reboot one or more instances.

Outdate

Answer 6

You can automatically do this via a CloudWatch Alarm. When the alarm is triggered because an instance is dead, one of the actions you can then have the alarm take is to reboot the instance. No need to spend time making a lambda or having a micro instance handle this for you.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/UsingAlarmActions.html#AddingRebootActions

Bonni