How to launch EC2 instance with Boto, specifying size of EBS?

Question

I'm using boto/python to launch a new EC2 instance that boots from an EBS volume. At the time I launch the instance, I'd like to override the default size of the booting EBS volume.

I found no boto methods or parameters that might fit into my launch code:

ec2 = boto.connect_ec2( ACCESS_KEY, SECRET_KEY, region=region )

reservation = ec2.run_instances( image_id=AMI_ID, 
                                 key_name=EC2_KEY_HANDLE, 
                                 instance_type=INSTANCE_TYPE,
                                 security_groups = [ SECGROUP_HANDLE, ] )

This web page shows how to increase the size of a running EC2-instance's EBS volume using command-line tools, but I'd like to use boto at the time the EC2 instance is specified:

Reserve

Answer 1

You have to create a block device mapping first:

dev_sda1 = boto.ec2.blockdevicemapping.EBSBlockDeviceType()
dev_sda1.size = 50 # size in Gigabytes
bdm = boto.ec2.blockdevicemapping.BlockDeviceMapping()
bdm['/dev/sda1'] = dev_sda1 

After this you can give the block device map in your run_instances call:

reservation = ec2.run_instances( image_id=AMI_ID, 
                                 key_name=EC2_KEY_HANDLE, 
                                 instance_type=INSTANCE_TYPE,
                                 security_groups = [ SECGROUP_HANDLE, ],
                                 block_device_mappings = [bdm])

Unfortunately this is not really well documented, but the example can be found in the source code.

Hippy

This works! Thank you! One caution to other noobs like me: I thought I would be clever and change /dev/sda1 to /dev/xvda1 because, on the Ubuntu instances that I run, that is the name of the boot volume, according to the df -h command. For whatever reason, that failed with the error message: "Invalid device name /dev/xvda1" So I changed it back to /dev/sda1 and all went well. – Reserve

Wouldn't a default amazon linux 8GB ami still have a partition table of 8GB size which would have to be extended once booted for the first time? – Ferdie

You are basically right. I don't know what the Amazon AMIs do in this case, the Ubuntu AMIs automatically expand the partition on first boot. – Hippy

A late comment, but perhaps useful. You may use the value of 'block_device_mapping' from the boto.ec2.image.Image object, modify it, and use it directly as the block_device_map argument to run_instances() of boto.ec2.image.Image.run() – Subsequent

Important note: In Boto, delete_on_termination=False by default. However, in the Web Console, the "Delete on Termination" check box is checked by default. – Monitorial

In newer versions of Boto, the param block_device_mappings in the run_instances() method has been renamed to block_device_map and should be of type BlockDeviceMapping rather than an array of such. – Barite

In my case block_device_mappings had to be changed to # block_device_map= bdm – Braynard

Answer 2

You can also use CloudFormation, which is used to document and automate your environment. You can check the template for the ESB definition at: https://s3.amazonaws.com/cloudformation-templates-us-east-1/EC2WithEBSSample.template

 "Resources" : {
    "Ec2Instance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "AvailabilityZone" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "TestAz" ]},
        "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
        "KeyName" : { "Ref" : "KeyName" },
        "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},
        "Volumes" : [ 
          { "VolumeId" : { "Ref" : "NewVolume" },
            "Device" : "/dev/sdk"
          }
        ]
      }
    },

    ...

    "NewVolume" : {
      "Type" : "AWS::EC2::Volume",
      "Properties" : {
        "Size" : "100",
        "AvailabilityZone" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "TestAz" ]}
      }
    }

You can then use Boto CloudFormation API to deploy your environment.

Gulley

Thanks Guy. cloudformation is looking pretty interesting, will take a closer look at this finally – Ferdie

Answer 3

Here is a version of the code using the boto3 "everything is a resource" approach. Note also how to avoid hard-coded disk names:

import boto3

ec2 = boto3.resource('ec2')


def ec2_one_by_key_and_value(collection: str, key: str, value: str):
    handler = getattr(ec2, collection)
    return list(handler.filter(Filters=[{'Name': key, 'Values': [value]}]))[0]


image = ec2_one_by_key_and_value('images', 'name', 'ubuntu/images/hvm-ssd/...')
root_disk = None
for block_device in image.block_device_mappings:
    if block_device['DeviceName'] == image.root_device_name:
        root_disk = block_device
        assert root_disk['Ebs']
        root_disk['Ebs']['VolumeSize'] = 16 # New disk 
        break
assert root_disk
instances = ec2.create_instances(MinCount=1, ..., ImageId=image.id,
                                 BlockDeviceMappings=image.block_device_mappings)

Pyrostat