Is it secure to store EC2 User-Data shell scripts in a private S3 bucket?

Question

I have an EC2 ASG on AWS and I'm interested in storing the shell script that's used to instantiate any given instance in an S3 bucket and have it downloaded and run upon instantiation, but it all feels a little rickety even though I'm using an IAM Instance Role, transferring via HTTPS, and encrypting the script itself while at rest in the S3 bucket using KMS using S3 Server Side Encryption (because the KMS method was throwing an 'Unknown' error).

The Setup

• Created an IAM Instance Role that gets assigned to any instance in my ASG upon instantiation, resulting in my AWS creds being baked into the instance as ENV vars
• Uploaded and encrypted my Instance-Init.sh script to S3 resulting in a private endpoint like so : https://s3.amazonaws.com/super-secret-bucket/Instance-Init.sh
  
  

In The User-Data Field

I input the following into the User Data field when creating the Launch Configuration I want my ASG to use:

#!/bin/bash

apt-get update
apt-get -y install python-pip
apt-get -y install awscli
cd /home/ubuntu
aws s3 cp s3://super-secret-bucket/Instance-Init.sh . --region us-east-1
chmod +x Instance-Init.sh
. Instance-Init.sh
shred -u -z -n 27 Instance-Init.sh

The above does the following:

• Updates package lists
• Installs Python (required to run aws-cli)
• Installs aws-cli
• Changes to the /home/ubuntu user directory
• Uses the aws-cli to download the Instance-Init.sh file from S3. Due to the IAM Role assigned to my instance, my AWS creds are automagically discovered by aws-cli. The IAM Role also grants my instance the permissions necessary to decrypt the file.
• Makes it executable
• Runs the script
• Deletes the script after it's completed.
  
  

The Instance-Init.sh Script

The script itself will do stuff like setting env vars and docker run the containers that I need deployed on my instance. Kinda like so:

#!/bin/bash

export MONGO_USER='MyMongoUserName'
export MONGO_PASS='Top-Secret-Dont-Tell-Anyone'

docker login -u <username> -p <password> -e <email>
docker run - e MONGO_USER=${MONGO_USER} -e MONGO_PASS=${MONGO_PASS} --name MyContainerName quay.io/myQuayNameSpace/MyAppName:latest

Very Handy

This creates a very handy way to update User-Data scripts without the need to create a new Launch Config every time you need to make a minor change. And it does a great job of getting env vars out of your codebase and into a narrow, controllable space (the Instance-Init.sh script itself).

But it all feels a little insecure. The idea of putting my master DB creds into a file on S3 is unsettling to say the least.

The Questions

1. Is this a common practice or am I dreaming up a bad idea here?
2. Does the fact that the file is downloaded and stored (albeit briefly) on the fresh instance constitute a vulnerability at all?
3. Is there a better method for deleting the file in a more secure way?
4. Does it even matter whether the file is deleted after it's run? Considering the secrets are being transferred to env vars it almost seems redundant to delete the Instance-Init.sh file.
5. Is there something that I'm missing in my nascent days of ops?
   
   

Thanks for any help in advance.

Archegonium

I'm not of the exact scope of the site but this might fit better on security.stackexchange.com. – Jetpropelled

I agree that this should go to security.stackexchange.com. Needless to say, tough, putting any credentials into any plaintext file anywhere at all is a BIG security risk. Never do that. Ever. If it's at all possible, have it ask for the credentials from user-input through terminal or figure out a different way to do it. – Croton

It might be a 'better fit' for InfoSec, but I'd say the popularity so far means it should stay here. The Vox Populi has voted with their votes! – Archegonium

On the topic of the question itself. You're not wrong, @nln, but the creds aren't being stored in plaintext, they're being encrypted using S3 Server Side Encryption. AWS offers this encryption and from what I understand each object is encrypted with its own key, which itself is encrypted but he AWS master key, which they are responsible for. The object keys are rotated (don't remember how often) so, to be honest, they've got a better process in-place than I could roll myself. – Archegonium

Creds via cli input aren't possible because any given instance that needs the creds would be part of an auto-scaling group that has no human interaction when it comes to launching. – Archegonium

The thinking about the fit was also that people on security are more likely to understand the scenario in question and be able to better help you model what threats you are actually concerned with and what the attack vectors are. – Jetpropelled

I'd personally use a cloudformation script with parameters to set up all the components, which can then inject the credentials as variables into your user data as part of the launch configuration. Then, when you have to update the userdata bit, just update the cloudformation template, and AWS will handle the updates to your stack. If that means, the Launch Config was changed, you can scale up and down, and end up with a set of updated instances. Having said that, your way is quite common, but you have to make sure to review bucket policies and IAM permissions very carefully – Herzog

Answer 1

What you are describing is almost exactly what we are using to instantiate Docker containers from our registry (we now use v2 self-hosted/private, s3-backed docker-registry instead of Quay) into production. FWIW, I had the same "this feels rickety" feeling that you describe when first treading this path, but after almost a year now of doing it -- and compared to the alternative of storing this sensitive configuration data in a repo or baked into the image -- I'm confident it's one of the better ways of handling this data. Now, that being said, we are currently looking at using Hashicorp's new Vault software for deploying configuration secrets to replace this "shared" encrypted secret shell script container (say that five times fast). We are thinking that Vault will be the equivalent of outsourcing crypto to the open source community (where it belongs), but for configuration storage.

In fewer words, we haven't run across many problems with a very similar situation we've been using for about a year, but we are now looking at using an external open source project (Hashicorp's Vault) to replace our homegrown method. Good luck!

Homogeneity

Yup. That's the ticket. Thanks @L0j1k, that's pretty much exactly the answer I was looking for. It just feels wrong, eh? Anyhow, great to know that I'm not venturing out into totally unknown territory. Vault looks like a tricked-out etcd, very, very cool. I love the Dynamic Secrets concept. Gonna go read their site now. Cheers. – Archegonium

Right on. For me personally, I knew we were on the right track the minute I saw Hashicorp's Vault project, because it's mechanically very similar to the way you and I are solving this problem. Validation, if you will. :) – Homogeneity

Yup. My thinking of creds management has changed drastically. I used to think of creds as stand-alone objects that floated from user-to-user. Now I see them as a byproduct of a carefully engineered Secrets Layer. Which reduces the attack surface for the creds greatly. I've managed to get them all in one spot via this S3 hack, but I was unsure of how to make them dynamic. Vault seems to solve this problem very nicely. – Archegonium

This is a totally legitimate and common practice! I think having the IAM role permissions tightly scoped down is a great idea, and I would still go ahead and shred the script even after execution. I have seen some people who go to great lengths to generate a message in an SQS queue, which then triggers a signed URL in S3, which they can then download as part of bootstrapping, but that seems less reliable. My only suggested changes in your user-data script would be to have pip and the awscli installed in the AMI already, and I think the pip installation of the awscli is fresher than apt-get. – Slemmer

Answer 2

An alternative to Vault is to use credstash, which leverages AWS KMS and DynamoDB to achieve a similar goal.

I actually use credstash to dynamically import sensitive configuration data at container startup via a simple entrypoint script - this way the sensitive data is not exposed via docker inspect or in docker logs etc.

Here's a sample entrypoint script (for a Python application) - the beauty here is you can still pass in credentials via environment variables for non-AWS/dev environments.

#!/bin/bash
set -e

# Activate virtual environment
. /app/venv/bin/activate

# Pull sensitive credentials from AWS credstash if CREDENTIAL_STORE is set with a little help from jq
# AWS_DEFAULT_REGION must also be set
# Note values are Base64 encoded in this example
if [[ -n $CREDENTIAL_STORE ]]; then
  items=$(credstash -t $CREDENTIAL_STORE getall -f json | jq 'to_entries | .[]' -r)
  keys=$(echo $items | jq .key -r)
  for key in $keys
  do
    export $key=$(echo $items | jq 'select(.key=="'$key'") | .value' -r | base64 --decode)
  done
fi

exec $@

Lambdoid

Very cool. Thanks for the tip, @mixja. Love how it "hides" the creds from docker inspect/logs. – Archegonium