Using IMDS (v2) with token inside docker on EC2 or ECS

Question

I'd like to use IMDSv2 inside a container running on an EC2 instance.

I want to use the tokens because they are required in my metadata options:

metadata_options {
  http_tokens   = "required"
  http_endpoint = "enabled"
}

Calling the API from the EC2 instance returns my token as expected.

curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"

However, if I try to call it from a docker container:

docker run -it curlimages/curl sh
/ $ curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
curl: (56) Recv failure: Connection reset by peer

I just have a timeout.

According to this answer, it should work out of the box, but it's not. If I add a --network=host flag, it works, but that's not a solution for me.

Thanks

Vitus

Answer 1

I order to access IMDSv2 metadata from a docker container, you must increase the hop limit for IMDSv2 in the instance metadata configuration. From the aws docs:

In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop. To avoid the process of falling back to IMDSv1 and the resultant delay, in a container environment we recommend that you set the hop limit to 2

To change the hop limit, you can use modify-instance-metadata-options in awscli:

aws ec2 modify-instance-metadata-options \
    --instance-id <instance_id> \
    --http-put-response-hop-limit 2 \
    --http-endpoint enabled

Bassarisk

Thank you so much for this great answer ! I was searching in the ECS doc, my bad ! Is putting a high value for http-put-response-hop-limit a bad practice, or it doesn't really matter ? – Vitus

@Vitus Not sure, I mostly run in EKS and have never needed to set higher than 2. – Bassarisk

@Vitus The hop limit is set to 1 because the packet isn't supposed to leave localhost, so in this case setting it to 2 is necessary because it has to go from localhost to docker which is 2 hops. You don't want it higher than that though. More details in this blog post: aws.amazon.com/blogs/security/… – Crosscurrent

For anyone using ElasticBeanstalk see this GitHub issue for more details: github.com/aws/elastic-beanstalk-roadmap/issues/242 – Crosscurrent

Answer 2

Here the same issue. Containers has eventual errors log errors like this:

error parsing user Error [CredentialsError]: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

Is related to missing get EC2 metadata and containers crash due missing credentials. The error is reproduced when instance metadata IMDSv2 is changed from "optional" to "required" and container is recreated. This change was done without consider any additional settings change:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html#Considerations

If you have awsvpc networking mode you can bypass with the default value HttpPutResponseHopLimit = "1"

"MetadataOptions": {
"State": "applied",
"HttpTokens": "optional",
"HttpPutResponseHopLimit": "1",
"HttpEndpoint": "enabled"
},

But if you bridge networking mode , then you need set minimum of "2" hops in order the containers could to reach out metadata.

"MetadataOptions": {
"State": "applied",
"HttpTokens": "optional",
"HttpPutResponseHopLimit": "2",
"HttpEndpoint": "enabled"
},

You can use modify-instance-metadata-options in awscli:

aws ec2 modify-instance-metadata-options \
    --instance-id <instance_id> \
    --http-put-response-hop-limit 2 \
    --http-endpoint enabled

Or from Terraform, from resource "aws_launch_template" https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template

with followwing metadata options:

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 2
    instance_metadata_tags      = "enabled"
  }

Now from your EC2 or inside the containers, Token should be receive a response:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2ata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/

/app_start # TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2ata-token: $TOKEN" -
v http://169.254.169.254/latest/meta-data/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0   6784      0 --:--:-- --:--:-- --:--:--  8000
* processing: http://169.254.169.254/latest/meta-data/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80
> GET /latest/meta-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/8.2.1
> Accept: */*
> X-aws-ec2ata-token: xxxxxxx
>
< HTTP/1.1 401 Unauthorized
< Content-Length: 0
< Date: Thu, 14 Sep 2023 11:51:42 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
<
* Closing connection

Kerb

Answer 3

In case of in does not really work, you can try to increase the hop limit value.

Our context is: RKE2 + cilium on EC2 instances.

We have increase the hop limit from 2 to 3 and it works.

With hop limit=2

curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
^C
bash-4.2# curl http://169.254.169.254/latest/meta-data/
bash-4.2# curl http://169.254.169.254/latest/meta-data/ -vv
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Content-Length: 0
< Date: Thu, 06 Apr 2023 09:02:51 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
<
* Closing connection 0

After increased hop-limit=3

curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" -vv
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.87.0
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Content-Length: 56
< Date: Thu, 06 Apr 2023 09:14:54 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
<
* Closing connection 0
AQAEAPrjYxOT2_9q00Flibi5iB-KbE..redacted

Penton