How to stop/start an EC2 instance on a schedule within an autoscale group without terminating the instance?

Question

If you have have an autoscaling group, how do you go about start/stopping the instances (not launch/terminate) ? The default behaviour is only for launching and terminating instances, but since we also get cost savings for just stopping an instance, we would rather allow the instance to persist.

For our purposes, we have a QA environment and we want to start/stop several autoscaling groups daily to save money when the development team leaves the office for evenings and weekends.

So how to accomplish this?

NOTE:

I will be following up with my own detailed answer shortly.

Holna

I would suggest that one would be wise to actually implement their application such that terminating and getting a new instance from scratch is not problematic to operation of the system. – Airship

Agreed. But sometimes in a test environment you want to maintain the state of the instance, as developers have been running tests on it, logs are created, and in my case, I'm a bit lazy and I don't want to script the secondary ENI attachment at this time for instances living in my AS groups. – Holna

I'm a step further and I'm having problems, Please have a look at my question AWS Autoscaling Group EC2 instances go down during cron jobs – Constituent

Answer 1

First and foremost, the AWS autoscaling group is a container for multiple instances that are based on a launch configuration. If we are able to disable the processes that trigger up/down scaling, we are back to the container that just holds instances.

In order to disable these processes, we need to use the suspend-processes command from AWS-CLI. For this example, I will use powershell, but its just as easy to write in bash:

# suspend HealthCheck and ReplaceUnhealthy processes, you may find another combination works better for you.
$asGroup = "nameOfYourAutoScalingGroup" ;
aws autoscaling suspend-processes `
    --auto-scaling-group-name $asGroup `
    --scaling-processes HealthCheck ReplaceUnhealthy ;

# verify the change
awsp autoscaling describe-auto-scaling-groups `
    --auto-scaling-group-name $asGroup ;

For my purposes, I wanted the instances to be online between 7am and 7pm to reduce costs. These instances are used between our development and QA teams, and they prefer to keep the state of the server from day to day.

NOTE: in the circumstance where the EC2 instance becomes corrupt or accidentally terminated, our team is fine with rollbacking back to the latest AMI (they really just want the logs to persist, but if they are lost, it isn't the end of the world)

Next we'll require a script to start/stop the servers, here I have it as 2 scripts, but you can easily optimize it into one script and pass an argument in:

# in our case, we want to perform this to all autoscaling groups
# you'll need Powershell 3.0+ in order to use ConvertFrom-Json
$asGroups = aws autoscaling describe-auto-scaling-groups --query 'AutoScalingGroups[*].{Name:AutoScalingGroupName,Instances:Instances[*].InstanceId}' ;
$asGroups = "{ asGroups: $asGroups }" | ConvertFrom-Json ;

# foreach autoscaling group, go through each instance and start
foreach ($asGroup in $($asGroups.asGroups)) {

    echo "AS: $($asGroup.Name)" ;
    foreach ($instance in $asGroup.instances) {
        echo "starting instance: $instance";
        aws ec2 start-instances `
            --instance-ids $instance ;
    }
}

# in our case, we want to perform this to all autoscaling groups
# you'll need Powershell 3.0+ in order to use ConvertFrom-Json
$asGroups = awsp autoscaling describe-auto-scaling-groups --query 'AutoScalingGroups[*].{Name:AutoScalingGroupName,Instances:Instances[*].InstanceId}' ;
$asGroups = "{ asGroups: $asGroups }" | ConvertFrom-Json ;

# foreach autoscaling group, go through each instance and stop
foreach ($asGroup in $($asGroups.asGroups)) {

    echo "AS: $($asGroup.Name)" ;
    foreach ($instance in $asGroup.instances) {
        echo "stopping instance: $instance";
        awsp ec2 stop-instances `
            --instance-ids $instance ;
    }
}

The last step would be to add it to the Scheduled Tasks on a control server (I'm currently just using my desktop which never turns off). Attached is an example of the exported Schedule Task, runs weekly Mon, Tue, Wed, Thurs, Friday at 7am.

<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.2" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
  <RegistrationInfo>
    <Date>2014-08-22T13:13:02.2103946</Date>
    <Author>localhost\Administrator</Author>
  </RegistrationInfo>
  <Triggers>
    <CalendarTrigger>
      <StartBoundary>2014-08-22T07:00:00</StartBoundary>
      <Enabled>true</Enabled>
      <ScheduleByWeek>
        <DaysOfWeek>
          <Monday />
          <Tuesday />
          <Wednesday />
          <Thursday />
          <Friday />
        </DaysOfWeek>
        <WeeksInterval>1</WeeksInterval>
      </ScheduleByWeek>
    </CalendarTrigger>
  </Triggers>
  <Principals>
    <Principal id="Author">
      <UserId>localhost\Administrator</UserId>
      <LogonType>InteractiveToken</LogonType>
      <RunLevel>LeastPrivilege</RunLevel>
    </Principal>
  </Principals>
  <Settings>
    <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
    <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>
    <StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
    <AllowHardTerminate>true</AllowHardTerminate>
    <StartWhenAvailable>false</StartWhenAvailable>
    <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
    <IdleSettings>
      <StopOnIdleEnd>true</StopOnIdleEnd>
      <RestartOnIdle>false</RestartOnIdle>
    </IdleSettings>
    <AllowStartOnDemand>true</AllowStartOnDemand>
    <Enabled>true</Enabled>
    <Hidden>false</Hidden>
    <RunOnlyIfIdle>false</RunOnlyIfIdle>
    <WakeToRun>false</WakeToRun>
    <ExecutionTimeLimit>P3D</ExecutionTimeLimit>
    <Priority>7</Priority>
  </Settings>
  <Actions Context="Author">
    <Exec>
      <Command>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe</Command>
      <Arguments>-ExecutionPolicy ByPass c:\tasks\AWS-Autoscaling-EC2-Start-Morning.ps1</Arguments>
      <WorkingDirectory>c:\tasks</WorkingDirectory>
    </Exec>
  </Actions>
</Task>

You'll need to make the "Stop" version of this task to stop servers at 7pm. Just change your start boundary to <StartBoundary>2014-08-22T19:00:00</StartBoundary> and update <Arguments>-ExecutionPolicy ByPass c:\tasks\AWS-Autoscaling-EC2-Start-Morning.ps1</Arguments> to the correct ps1.

Holna

Answer 2

You may consider to run your script using AWS Data Pipeline.
Use this script to retrieve instance ID as well as availability zone and region, etc

Choose Create New Pipeline and enter the following information:

Name: for example, "Start EC2 instances" and "Stop EC2 instances".
Description: Provide relevant details about the pipeline as needed.
Source: Choose Build using template and choose the template Run AWS CLI command.
AWS CLI command: This is where you put your script to specify what the pipeline does.

Configure each pipeline with appropriate scheduling information.

Run: Choose on activation to run the pipeline as an on-demand pipeline.  
Run every: Enter a period for every pipeline run.
Starting: Enter a time and date for the pipeline to start.
Ending: Enter a time and date for the pipeline to end.

Set the following options for implementing appropriate security access:

IAM Roles: Choose Custom
Pipeline Role: DataPipelineDefaultRole
EC2 Instance Role: DataPipelineDefaultResourceRole

Note that Data Pipeline creates the necessary IAM Roles for you.
For more information, see AWS Data Pipeline Pricing.

See also other option using AWS Lambda.

Bobbiebobbin