AWS Elastic Beanstalk and Secret Manager

Question

Does anyone know is it possible to pass a secret value as an environment variable in elastic beanstalk? The alternative obviously is to use the sdk in our codebase but I want to explore the environment variable approach first

Cheers Damien

Sachiko

Answer 1

Unfortunately, EB doesn't support secrets at this point, this might be added down the road. You can use them in your environment variables as the documentation suggests but they will appear in plain text in the console. Another, and IMO better, approach would be to use ebextensions, and use AWS CLI commands to grab secrets from the secrets manager, which needs some set up (e.g. having AWS CLI installed and having your secrets stored in SM). You can set these as environment variables in the same eb configuration. Hope this helps!

Bateman

To complement this, you could create your secret manager resource directly in EB, in case you want a lifecycle of the secret resource to be coupled with the life of your environment. – Disused

What do you mean when saying "EB doesn't support secrets"? Because when I deploy the application on EB (.net core), the secrets are all null. Locally instead, they work – Leaves

@Marcin, would love you to elaborate on "create secret manager resource directly in EB" – Secrest

@MarkNadig I would suggest making new question, specific to your use-case and difficulties. – Disused

Answer 2

Per @Ali's answer, it is not built-in at this point. However, it is relatively easy to use .ebextensions and the AWS cli. Here is an example that extracts a secret to a file, according to an MY_ENV environment variable. This value could then be set to an environment variable, but keep in mind environment variables are specific to the shell. You'd need to pass them in to anything you are launching.

  10-extract-htpasswd:
    env:
      MY_ENV: 
        "Fn::GetOptionSetting":
          Namespace: "aws:elasticbeanstalk:application:environment"
          OptionName: MY_ENV
    command: |
      aws secretsmanager get-secret-value --secret-id myproj/$MY_ENV/htpasswd --region=us-east-1 --query=SecretString --output text > /etc/nginx/.htpasswd
      chmod o-rwx /etc/nginx/.htpasswd
      chgrp nginx /etc/nginx/.htpasswd

This also requires giving the EB service role IAM permissions to the secrets. i.e. A policy like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "xxxxxxxxxx",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxx:secret:myproj*"
        }
    ]
}

Giselegisella

Answer 3

As above answers mention, there is still no built-in solution if you want to do this in Elastic Beanstalk. However a work around solution is to use "platform hook". Unfortunately it is poorly documented at this point.

To store your secret, best solution is to create a custom secret in AWS-Secret-Manager. In secret manager you can create a new secret by clicking "Store a new secret", then selecting "Other type of secret" and entering your secret key/value (see ). At the next step you need to provide a Secret Name (say "your_secret_name") and you can leave everything else to their default settings.

Then, you need to allow Elastic Beanstalk to get this secret. You can do it by creating a new IAM policy, for instance with this content:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "Getsecretvalue",
        "Effect": "Allow",
        "Action": [
            "secretsmanager:GetResourcePolicy",
            "secretsmanager:GetSecretValue",
            "secretsmanager:DescribeSecret",
            "secretsmanager:ListSecretVersionIds"
        ],
        "Resource": "your-secret-arn"
    }
]}

You need to replace "your-secret-arn" with your secret ARN which you can get on AWS-secret-manager interface. Then, you need to add the policy you created to EB roles (it should be either "aws-elasticbeanstalk-ec2-role" or "aws-elasticbeanstalk-service-role").

Finally you need to add a hook file in your application. From the root of your application, add file ".platform/hooks/prebuild/your_hook.sh". Content of your file can be something like this:

#!/bin/sh
export your_secret_key=$(aws secretsmanager get-secret-value --secret-id your-secret-name --region us-east-1 | jq -r '.SecretString' | jq -r '.your_secret_key')

touch .env
{
  printf "SECRET_KEY=%s\n" "$your_secret_key"
  # printf whatever other variable you want to pass
} >> .env

Obviously you need to replace "your_secret_name" and the other variable by your own values and set the region to the region where your secret is stored (if it is not us-east-1). And don't forget to make it executable ("chmod +x your_hook.sh").

This assumes that your application can load its env from a .env file (which works fine with docker / docker-compose for example).

Another option is to store the variable in an ".ebextensions" config file but unfortunately it doesn't seem to work with the new Amazon Linux 2 platform. What's more you should not store sensitive information such as credentials directly in your application build. Builds of the application can be accessed by anyone with Elastic Beanstalk Read Access and they are also store unencrypted on S3.

With the hook approach, the secret is only stored locally on your Elastic Beanstalk underlying EC2 instances, and you can (should!) restrict direct SSH access to them.

Paniculate

Thanks. I wanted to pass datadog api key securely through aws secret manager (for datadog side car container) in elastic beanstalk env (Platform : Docker , Branch : 64bit Amazon Linux2) docker-compose.yml file . Following this , I created shell script & placed it at both places (.platform/confighooks/prebuild/test.sh) & (.platform/hooks/prebuild/test.sh) fetching DD_API_KEY from aws secret manager & writing it to .env file. Refer docs.aws.amazon.com/elasticbeanstalk/latest/dg/… – Katherinkatherina

PS : config stage hooks .platform/confighooks/prebuild gets executed when there are eb config changes , which doesn't require any deployment, while deploy stage hooks .platform/hooks/prebuild gets executed post config stage , mostly in case of uploading new app version , changing few eb configs which requires deployment. so .platform/confighooks/prebuild makes sure that DD_API_KEY is still available in case elastic beanstalk gets updated with config changes only – Katherinkatherina

For me, I had to attach the IAM policy to the aws-elasticbeanstalk-ec2-role to get the command to work (attaching it to the aws-elasticbeanstalk-service-role didn't let the command work). – Chaing

Note that if the value of your secret is just a simple string, the your-secret-name and your_secret_key will be the same in the aws secretsmanager command. – Chaing

Answer 4

I'm just adding to @kaliatech's answer because while very helpful, it had a few gaps that left me unable to get this working for a few days. Basically you need to add a config file to the .ebextensions directory of your EB app, which uses a container_commands section retrieve your secret (in JSON format) and output it as a .env. file into the /var/app/current directory of the EC2 instances where your app's code lives:

# .ebextensions/setup-env.config
container_commands:
  01-extract-env:
    env:
      AWS_SECRET_ID:
        "Fn::GetOptionSetting":
          Namespace: "aws:elasticbeanstalk:application:environment"
          OptionName: AWS_SECRET_ID
      AWS_REGION: {"Ref" : "AWS::Region"}
      ENVFILE: .env

    command: >
        aws secretsmanager get-secret-value --secret-id $AWS_SECRET_ID --region $AWS_REGION |
        jq -r '.SecretString' |
        jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]' > $ENVFILE

Note: this assumes the AWS_SECRET_ID is configured in the app environment, but it can easily be hardcoded here as well.

All the utils needed for this script to work are already baked into the EC2 Linux image, but you'll need to grant permissions to the IamInstanceProfile role (usually named aws-elasticbeanstalk-ec2-role) which is assumed by EC2 to allow it access SecretManager:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SecretManagerAccess",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:ap-southeast-2:xxxxxxxxxxxx:secret:my-secret-name*"
        }
    ]
}

Finally, to debug any issues encountered during EC2 instance bootstrap, download the EB logs and check the EC2 log files at /var/log/cfn-init.log and /var/log/cfn-init-cmd.log.

Mycenaean

Its working but theres a little issue, if you then update the environment variables through the elastic beanstalk gui, it will do a deploy but (I believe) won't run the .ebextensions files, so the .env file isn't created – Unlatch

@FranciscoTrillo I'm no expert on EB configuration changes but according to the documentation the behaviour varies depending on your update policy and the type of configuration you're changing. Some changes can be applied to running instances, whilst others will use either rolling, immutable or disabled updates whereby instances are stopped and restarted using various resilience strategies. If instances are stopped and started I would expect the .ebextensions scripts to most certainly run. See docs.aws.amazon.com/elasticbeanstalk/latest/dg/… – Mycenaean

yes, they've changed the way it works since amazon linux 2. I've do it modifying your script a bit to make it a bash script executable. If you want to run it after environment variables are changed, you have to put it on a folder in .platform/confighooks/postdeploy – Unlatch

Interesting, that matches what @Paniculate said in their answer. I'm running a "64bit Amazon Linux 2 v5.5.0 running Node.js 14" solution stack so would probably experience the same issues if I ever did a configuration change. Thanks for the warning. 👍 – Mycenaean

Thanks. created shell script & placed it at both places (.platform/confighooks/prebuild/test.sh) & (.platform/hooks/prebuild/test.sh) just to make sure that environment variable is available in both config & deploy stages . Note : config stage hooks `.platform/confighooks/prebuild` gets executed when there are eb config changes , which doesn't require any deployment, while deploy stage hooks `.platform/hooks/prebuild` gets executed post config stage , mostly in case of uploading new app version , changing few eb configs which requires deployment – Katherinkatherina

Answer 5

This answer only applies if you're using code pipeline.

I think you can add a secret in the environment variables section now

Bink

Could you provide more context to this? This is not what my interface looks like for configuring environment variables in elastic beanstalk – Discourage

Hi @johnchas If I go to the env variables section of my Code Pipeline this is what I see ^ . Updated the post above with a better screenshot. Do you see something else? – Bink

This response seems to be only relevant to code pipeline and not elastic beanstalk. I would recommend revising to indicate that the interface you are showing here is not elastic beanstalk – Discourage

Answer 6

If you use AWS CodeBuild use pre_build, add the following commands in your project's buildspec.yml to retrieve your environment variables from AWS Secrets Manager use sed to do some substituting/formatting and append them to .ebextensions/options.config's aws:elasticbeanstalk:application:environment namespace:

phases:
  pre_build:
    commands:
      - secret=$(aws secretsmanager get-secret-value --secret-id foo-123 --region=bar-xyz --query=SecretString --output text)
      - regex=$(cat ./sed_substitute)
      - echo $secret | sed "${regex}" >> .ebextensions/options.config

Bit of a hack but the sed_substitute used in the commands above used to get the correct indentation/formatting that .ebextensions/options.config demands was:

s/",/\n /g; s/":/": /g; s/{"/ /g; s/"}//g; s/"//g;

Several

With this solution the secret will be stored in clear in the ".ebextensions/options.config file". Anyone that can download the application image (if they has either a read access to Elastic Beanstalk Console or to S3 buckets where images are stored) can get the secret. Thus it is highly unsecured and not really better than simply setting the secret directly in the Elastic Beanstalk Console. – Paniculate

Sure that's a problem, IF, you've left those key services unsecured - if that's the scenario anything is accesible. We're looking for a 'secure' way of setting environment variables programatically, without, having to manully set them via EB Console. Until EB supports Secrets directly we've no other option AFAIK. – Several