How to retrieve Secret Manager data in buildspec.yaml

Question

Im working on creating the CodeBuild which is integrated with SonarQube, So I pass values and sonar credentials directly in my Buildspec.yaml

Instead of Hardcoding directly, I tried to retrieve using the below command from SecretManager as it is mentioned in the below link. But it is not getting the correct values. it throws an error.

Command : '{{resolve:secretsmanager:MyRDSSecret:SecretString:username}}'

Link: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager

Error [ERROR] SonarQube server [{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:********:secret:**********:SecretString:SonarURL}}] can not be reached

How I used echo '{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:***:secret:**************:SecretString:*******}}'

Note: All the * inside my commard are the secretname and secreturl

Pusan

Answer 1

If you wish to retrieve secrets in your buildspec file, I would recommend to use Systems Manager Parameter Store which is natively integrated with CodeBuild. Systems Manager is a service in itself, search it from the AWS Console homepage, then Paramater Store is in the bottom left of the Systems Manager Console page.

Lets assume you want to include Access Key and Secret Key in buildspec.yml file:
- Create AccessKey/SecretKey pair for a IAM User
- Save the above keys in an SSM parameter store as secure string (e.g. '/CodeBuild/AWS_ACCESS_KEY_ID' and '/CodeBuild/AWS_SECRET_ACCESS_KEY')
- Export the two values in your build environment using the following buildspec directive(s):

version: 0.2
env:
    parameter-store:
        AWS_ACCESS_KEY_ID_PARAM: /CodeBuild/AWS_ACCESS_KEY_ID
        AWS_SECRET_ACCESS_KEY_PARAM: /CodeBuild/AWS_SECRET_ACCESS_KEY

phases:
    build:
        commands:
            - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_PARAM
            - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_PARAM
            # Your Ansible commands below
            - ansible-playbook -i hosts ec2-key.yml 

[1] Build Specification Reference for CodeBuild - Build Spec Syntax - https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax

Petiolule

Thanks for this quick turn around and help me in getting some idea on Parameter Store. – Pusan

Can you give a sample on how can it be used in CodeBuild & CodePipeline? for buildspec.yaml parameter-store: CUSTOM_PARAM: /aws/reference/secretsmanager/DBUser1 Eg: I have a Secret named "SajivSecret" which has few key and value pair. In which one Key named "DBUser1" and Value as some "***". I want to retrive this specific key in my buildspec.yaml file My Question is, Will it look for the Secret Named DBUSer1 or Key as DBUser1 in all the secrets in Secret Manager – Pusan

How to retrieve secrets specific to my Secret in Secret Manager in buildspec.yaml under parameter-store – Pusan

@SajivSriraam It will look for the Secret Named DBUSer1 - "When you retrieve a Secrets Manager secret from Parameter Store, the parameter name must begin with the following reserved path: /aws/reference/secretsmanager/secret_ID_in_Secrets_Manager" - docs.aws.amazon.com/systems-manager/latest/userguide/… – Unceasing

@Unceasing - Where can i find the secret ID in secret Manager. I could see SecretKey Name, Secret Name, Secret ARN – Pusan

@SajivSriraam secret id is the secret name – Unceasing

okay.. then how to retrive the particular Secret that is stored inside the Secret? – Pusan

aws secretsmanager get-secret-value --secret-id, then pass in the name of the secret – Unceasing

Answer 2

CodeBuild just launched this today - https://aws.amazon.com/about-aws/whats-new/2019/11/aws-codebuild-adds-support-for-aws-secrets-manager/

Unceasing

This should be the accepted answer. It is the most direct solution to op's question. – Grous

Yes, this is the answer. docs.aws.amazon.com/codebuild/latest/userguide/… In the following example, TestSecret is the name of the key-value pair stored in Secrets Manager. The key for TestSecret is MY_SECRET_VAR. You access the variable during the build using the LOCAL_SECRET_VAR name. env: secrets-manager: LOCAL_SECRET_VAR: "TestSecret:MY_SECRET_VAR" – Kiowa

Answer 3

The dynamic reference syntax you are trying to use only works with the Cloud Formation (CFN) service. In some cases, CFN restricts where these dynamic references to secrets will expand. Specifically, they do not expand in places where the secrets might be visible in the console, such as in EC2 metadata.

If you are trying to setup Code Build via CFN, this may be what you are seeing. However, as shariqmaws mentioned, you can use parameter store and either store your secret there or use parameter store as a pass through to secrets manager (in case you want to use secrets manager to rotate your secrets or for other reasons).

Conga

Can you give a sample on how can it be used in CodeBuild & CodePipeline? for buildspec.yaml parameter-store: CUSTOM_PARAM: /aws/reference/secretsmanager/DBUser1 Eg: I have a Secret named "SajivSecret" which has few key and value pair. In which one Key named "DBUser1" and Value as some "***". I want to retrive this specific key in my buildspec.yaml file My Question is, Will it look for the Secret Named DBUSer1 or Key as DBUser1 in all the secrets in Secret Manager – Pusan

Answer 4

version: 0.2

env:
  parameter-store:
    AWS_ACCESS_KEY_ID         : /terraform-cicd/AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY     : /terraform-cicd/AWS_SECRET_ACCESS_KEY
    AWS_CODECOMMIT_SSH_ID     : /terraform-cicd/AWS_CODECOMMIT_SSH_ID
  secrets-manager:  
    AWS_CODECOMMIT_SSH_PRIVATE: /terraform-cicd/AWS_CODECOMMIT_SSH_PRIVATE

Kerbela