Login/Register
Retrieve correct Amazon attached EBS device from instance metadata endpoint
Asked Answered
amazon-web-servicesamazon-ec2block-device
D

3

16

EDIT and TL;DR:

ubuntu@ip-172-31-19-77:~/.aws$ aws ec2 describe-instances | jq . | grep -i device
          "BlockDeviceMappings": [],
          "RootDeviceType": "ebs",
          "RootDeviceName": "/dev/sda1",
                "DeviceIndex": 0,
          "BlockDeviceMappings": [
              "DeviceName": "/dev/sda1",
              "DeviceName": "/dev/xvdb",
          "RootDeviceType": "ebs",
          "RootDeviceName": "/dev/sda1",
                "DeviceIndex": 0,
          "BlockDeviceMappings": [
              "DeviceName": "/dev/sda1",
          "RootDeviceType": "ebs",
          "RootDeviceName": "/dev/sda1",
                "DeviceIndex": 0,
          "BlockDeviceMappings": [
              "DeviceName": "/dev/xvda",
          "RootDeviceType": "ebs",
          "RootDeviceName": "/dev/xvda",
                "DeviceIndex": 0,
          "BlockDeviceMappings": [
              "DeviceName": "/dev/sda1",
              "DeviceName": "/dev/sdf",
          "RootDeviceType": "ebs",
          "RootDeviceName": "/dev/sda1",
ubuntu@ip-172-31-19-77:~/.aws$ aws ec2 describe-volumes | jq . | grep -i device
          "Device": "/dev/sda1"
          "Device": "/dev/sdf"
          "Device": "/dev/xvda"
          "Device": "/dev/sda1"
          "Device": "/dev/xvdb"
          "Device": "/dev/sda1"
ubuntu@ip-172-31-19-77:~/.aws$ ls /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
ubuntu@ip-172-31-19-77:~/.aws$ ls /dev/xv*
ls: cannot access '/dev/xv*': No such file or directory
ubuntu@ip-172-31-19-77:~/.aws$ lsblk
NAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   1G  0 disk
nvme1n1     259:1    0   8G  0 disk
└─nvme1n1p1 259:2    0   8G  0 part /

Explanation:

I would like to know in advance which block device a particular AWS instance is supposed to have post-attachment.

Unfortunately, the instance metadata endpoint for block devices introduced back in 2007 does not seem to work reliably anymore?

Here's an example t2.medium instance I'm running with latest Ubuntu (17.10) AMI at the time of writing this (not Amazon Linux):

enter image description here

Not only the EBS modules show up in the console, but seem to be properly attached:

enter image description here

According to the official AWS docs on block-device-mappings:

block-device-mapping/ebsN: The virtual devices associated with Amazon EBS volumes, if any are present. Amazon EBS volumes are only available in metadata if they were present at launch time or when the instance was last started. The N indicates the index of the Amazon EBS volume (such as ebs1 or ebs2). 2007-12-15

But unfortunately there are no such endpoints present after the instance is fully up and running and all EBS volumes are attached:

$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/
ami
ephemeral0
ephemeral1
$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ami
/dev/sda1
$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ebs1
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ebs2
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

On the other hand, the operating system (under XEN device scheme), exposes the volumes via xvdXX devices:

$ ls /dev/xvd*
/dev/xvda  /dev/xvda1  /dev/xvdf  /dev/xvdg

So this leaves me with a poor solution of just assuming which block device is going to be depending on which instance I'm running, which predictably breaks once, for instance, Amazon introduces new iron with completely different underlying block device naming schemes such as /dev/nvme0n1p1:

https://twitter.com/braincode/status/968005482102190080

Here's what happens on freshly instantiated M5 instance with an EBS volume attached:

$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/
ami
ebs2
root

$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ebs2
sdf
  1. Why is only ebs2 listed and not ebs1?
  2. sdf?

There's not a single sd* on m5 instance's operating system:

$ ls /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory

What am I doing wrong? Is this a known bug? How do people/boto/cloudinit modules handle that?

Dougall answered 18/4, 2018 at 4:9 Comment(0)
U
5

Getting and tagging the EBS RootVolume from running EC2instance: this is how you can achieve it:

Instance_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
Instance_Name=$(aws ec2 describe-instances --instance-ids ${Instance_ID} --query 'Reservations[0].Instances[0].Tags[?Key==`Name`].Value' --output text)
RootDeviceName=$(aws ec2 describe-instances --instance-ids ${Instance_ID} --query 'Reservations[0].Instances[0].RootDeviceName' --output text)
RootVolumeId=$(aws ec2 describe-instances --instance-ids ${Instance_ID} --query "Reservations[0].Instances[0].BlockDeviceMappings[?DeviceName==\`${RootDeviceName}\`].Ebs.VolumeId" --output text)

if you want to set a RootVolume Tag (which isn't supported by CloudFormation yet (2019-03-14)):

aws ec2 create-tags --resources ${RootVolumeId} --tags Key=Name,Value=RootVolume-${Instance_Name}

You need the following IAM Permissions for that:

"ec2:DescribeInstances"
"ec2:CreateTags"

This is how it works:

just look at the (shortened) ec2 json response:

aws ec2 describe-instances --instance-ids ${Instance_ID}

{
    "Reservations": [
        {
            "Instances": [
                {
                    "InstanceId": "i-XXXXX", 
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/xvda", 
                            "Ebs": {
                                "Status": "attached", 
                                "DeleteOnTermination": true, 
                                "VolumeId": "vol-XXXXX", 
                                "AttachTime": "2019-02-27T07:56:07.000Z"
                            }
                        }, 
                        {
                            "DeviceName": "/dev/sdm", 
                            "Ebs": {
                                "Status": "attached", 
                                "DeleteOnTermination": false, 
                                "VolumeId": "vol-XXXXX", 
                                "AttachTime": "2019-02-27T07:58:02.000Z"
                            }
                        }
                    ], 
                    "RootDeviceType": "ebs", 
                    "RootDeviceName": "/dev/xvda", 
                }
            ], 
            "ReservationId": "r-XXXXX", 
            "RequesterId": "XXXXX", 
            "Groups": [], 
            "OwnerId": "XXXXX"
        }
    ]
}

As you can see: every instance has BlockDeviceMappings and a RootDeviceName property.
1) You can select the current as DeviceName as Value from RootDeviceName property.
2) With that information you can parse your BlockDeviceMappings by matching your selected DeviceName and finally get the VolumeID.

Thats it.

Unitarian answered 14/3, 2019 at 9:5 Comment(2)
I've been told by a colleague recently that AWS fixed that at last. Can you include the output of the commands for completeness? Happy to accept your answer after verifying this is the case ;) – Dougall
I clearified it in the original post. The commands doesn't provide output. Just try yourself. – Unitarian
I
4

The metadata service shows the block device mapping at instance launch and is immutable. You can trust the value of the root device and any instance store volumes to be accurate for the life of the instance. The NVMe block devices do not need a mapping. Any other block devices under /dev/xvd* will be currently attached EBS volumes.

If you need to know which volume corresponds to each device file, the only accurate method is to call ec2.DescribeVolumes with the attachment.instance-id filter. Your instance will need an appropriate instance profile to allow it to make that call. You also need to be careful to not call it too often and implement retries with exponential backoff and jitter to handle throttling.

Impudicity answered 23/4, 2018 at 14:56 Comment(2)
No, you cannot trust them. In a m5 instance, having an attached EBS volume with device name nvme0n1 the metadata endpoint shows it like sdf, I will update the question to reflect this breakage from AWS. – Dougall
Here is some more evidence that you cannot do that (see point 6): github.com/HumanCellAtlas/upload-service/tree/master/amis – Dougall
E
1

All information is not available in the EC2 meta, but you can grab all required information base on EC2 Instance ID that is available in Ec2 meta.

Get instance ID

Instance_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

Base on Instance ID, Get Attached block devices

Instance_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 describe-instances --instance-ids ${Instance_ID} --query "Reservations[].Instances[].BlockDeviceMappings"

output

[
    [
        {
            "DeviceName": "/dev/xvda", 
            "Ebs": {
                "Status": "attached", 
                "DeleteOnTermination": true, 
                "VolumeId": "vol-12345adv ", 
                "AttachTime": "2020-05-11T07:01:58.000Z"
            }
        }, 
        {
            "DeviceName": "/dev/xvdcz", 
            "Ebs": {
                "Status": "attached", 
                "DeleteOnTermination": true, 
                "VolumeId": "vol-sdfsdfs", 
                "AttachTime": "2020-05-11T07:01:58.000Z"
            }
        }
    ]
]
Ehudd answered 9/7, 2020 at 9:25 Comment(2)
Would you mind running those commands on an M5/M6 instance type? – Dougall
I Checked and I do not think it's instance dependent command its aws available end point – Ehudd

© 2022 - 2024 — McMap. All rights reserved.