Parsing secrets from AWS secrets manager using AWS cli

Question

I am retrieving secrets I have stored in AWS secrets manager with the AWS cli like this:

aws secretsmanager get-secret-value --secret-id secrets

Which returns

arn:aws:secretsmanager<ID>:secret:my_secrets <number> my_secrets {"API_KEY":"ABCDEFGHI"}       <UUID string>
VERSIONSTAGES   AWSCURRENT

Does anyone know how I only get the secret ("API_KEY": "ABCDEFGHI")? I need to move these secrets to my register-task-definition environment variables. The best way would be to store them in a file and delete it after us or store them in variable. It is running on a linux machine.

Irenairene

Answer 1

Use the --query option of the CLI to extract just the secret.

aws secretsmanager get-secret-value --secret-id secrets --query SecretString --output text

Melon

Is this no longer working? every secret is null for me – Yonita

Ah SecretString selectes the entire secret string, it is not used to match a secret key – Yonita

@caleb Just running the command resulted in that error or w -query did it work? – Otten

Came to a wrong place but found an answer (--output text) :) – Lashing

This answer does not address the problem by the OP regarding parsing the SecretString. – Vasti

Answer 2

aws secretsmanager get-secret-value --secret-id secrets| jq --raw-output '.SecretString' | jq -r .API_KEY

using jq you can print.

Estelleesten

jq is not a standard util – Namhoi

And if your secret has a dash, you can use: aws secretsmanager get-secret-value --secret-id my-secret --query SecretString --output text | jq '."my-secret"' – Occult

Correction, to remove quotes in addition to making it work for a secret with a dash: aws secretsmanager get-secret-value --secret-id my-secret --query SecretString --output text | jq -r '."my-secret"' – Occult

Answer 3

Small addition to helloV answer. You can add the output parameter text to remove the quotes.

aws secretsmanager get-secret-value \
   --secret-id secrets \
   --query SecretString \
   --output text

Resnick

Answer 4

If your secret will only have one key/pair value, and you only want the value to be printed out, and you don't want to rely on your system having jq installed first, you can do:

aws secretsmanager get-secret-value --secret-id secrets --query SecretString --output text | cut -d: -f2 | tr -d \"}

Bricebriceno

you're missing a "\" before the last }, like tr -d \"\} – Bottomless

@Bottomless not needed on GNU/Bash – Bricebriceno

ah yeah you might be right, I was using ZSH on mac – Bottomless

a good answer which doesn't require jq – Tripletail

Answer 5

So I faced a bit of trouble in extracting what I needed, the value for my two variables that I stored in SecretsManager. So here is what worked for me.

NOTE: It's an example from the AWS SecretsManager doc.

I ran this

aws secretsmanager get-secret-value --secret-id MyTestDatabaseSecret --version-stage AWSPREVIOUS

The response of this query is:

{
  "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyTestDatabaseSecret-a1b2c3",
  "Name": "MyTestDatabaseSecret",
  "VersionId": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE",
  "SecretString": "{\n  \"username\":\"david\",\n  \"password\":\"BnQw&XDWgaEeT9XGTT29\"\n}\n",
  "VersionStages": [
    "AWSPREVIOUS"
  ],
  "CreatedDate": 1523477145.713
}

Now I want to get the value of username or password to be precise

aws secretsmanager get-secret-value --secret-id MyTestDatabaseSecret --version-stage AWSPREVIOUS | jq --raw-output .SecretString | jq -r ."password"

Output

BnQw&XDWgaEeT9XGTT29

Wulfe

I want to list all secrets and then get all values and run grep against it – Sanjuana

> aws secretsmanager get-secret-value --secret-id * is not working for me – Sanjuana

what is the error message? Do you have access rights to all? Are you passing the version number in your query? Instead of using * wild character, can you make a list of secret-id and then try to run a loop. First, try to run it for 10 odd secrets and then check to scale it up. – Wulfe

github.com/ashishkarpe/scripts_aws_cli/blob/main/… have written script which worked for me thanks – Sanjuana

Answer 6

All answers working but require 3rd party integration ( mainly jq ). the following bash command grabs the relevant Value without any other 3rd party solution -

SECRET_ARN=arn:aws:secretsmanager:eu-west-1:123456:secret:/test
SECRET_KEY=DB_PASSWORD
aws secretsmanager get-secret-value \ 
  --secret-id $SECRET_ARN \ 
  --query SecretString \
  --output text | grep -o '"$SECRET_KEY":"[^"]*' |  grep -o '[^"]*$'

Sherrie

The answered marked as the correct one does not require jq: aws secretsmanager get-secret-value --secret-id secrets --query SecretString --output text – Irenairene

The marked answer gets all secret values. my example grabs a specific value based on a secret key. – Sherrie

This works great for extracting the value of a single key in a simple key/value pair JSON object stored in Secrets Manager. Thank you @AmitBaranes – Zymogen

Answer 7

When you have multiple secret and you get json return, you can use get the exact value of password by using

aws secretsmanager get-secret-value --secret-id <secret_bucket_name> | jq --raw-output '.SecretString' | jq -r .key_for_password

Moorings

jq is json utility for shell command. it helps to parse the Json and pull the attribute from the file. – Moorings

this is a copy-paste of this answer https://mcmap.net/q/268115/-parsing-secrets-from-aws-secrets-manager-using-aws-cli – Droppings

In my case, The password has '&' symbol and is converted to '\u0026'. Is there any workaround for this? – Cinder

Answer 8

Lots of answers here depend on jq. If you don't want to install any other dependencies, you can use a python3 one-liner:

aws secretsmanager get-secret-value \
   --output text \
   --query SecretString \
   --secret-id my-secret-name \
| python3 -c 'import json, sys; print(json.load(sys.stdin)["my-secret-key"])'

Based on helloV's answer.

Cribbing

Thank you for this. You can also assign this to a variable this way: SECRET_ACCESS_KEY=$(aws secretsmanager get-secret-value \ --output text \ --query SecretString \ --secret-id my-secret-name \ | python3 -c 'import json, sys; print(json.load(sys.stdin)["my-secret-key"])') – Levorotatory

Answer 9

PowerShell solution without Jq

$a = aws secretsmanager get-secret-value --region <region> --secret-id <secret-name>  | ConvertFrom-Json 

$a all json converted to objects type

Output

ARN           : xxxxxx
Name          : postgxxx
VersionId     : fxxxx-xx-x-xx
SecretString  : {"key":"value","key2":"value"}
VersionStages : {xxxxx}
CreatedDate   : xxxxx.xx

---

$b = $a.SecretString | ConvertFrom-Json

Output

key : value
key2 : value

---

$b.key

**Output** 
value

Quark

Answer 10

Script to List all available AWS secrets to a /tmp/name.text and find specific secret values from it

Note needs AWS CLI configure to run this script successfully

#!/bin/bash

aws secretsmanager list-secrets | grep  "Name" | awk '{print $2}' | tr -d '"' | sed 's/,/ /g' > /tmp/name.text

for line in `cat /tmp/name.text`
do

echo $line >> /tmp/secrets-values.txt

aws secretsmanager get-secret-value --secret-id "$line" | grep "XYZ" >>  /tmp/secrets-values.txt
done

Sanjuana

Did it work? You tested. If yes, then great. :) – Wulfe

yes I tested it and it worked – Sanjuana

Answer 11

In the vein of "... without jq" answers, here's one for node users. (requires modern bash and nodejs, could easily be rewritten to just use sh by using an echo | instead of the cleaner <<<)

SECRET_ARN="..."
REGION=us-east-1

SECRET_BLOB=$(aws secretsmanager get-secret-value --region="$REGION" --output=text --query SecretString --secret-id "$SECRET_ARN")

MY_VALUE=$(node -pe 'JSON.parse(require("fs").readFileSync("/dev/stdin").toString()).myKey' <<< "$SECRET_BLOB")

MY_OTHER_VALUE=$(node -pe 'JSON.parse(require("fs").readFileSync("/dev/stdin").toString()).myOtherKey' <<< "$SECRET_BLOB")

If you need to pull multiple values from the secret, you'll want to cache the json blob in an env var. If you only need a single value though:

MY_VALUE=$(aws secretsmanager get-secret-value --region="$REGION" --output=text --query SecretString --secret-id "$SECRET_ARN" | node -pe 'JSON.parse(require("fs").readFileSync("/dev/stdin").toString()).myKey' <<< "$SECRET_BLOB")

Rolanderolando

Answer 12

I see many JQ examples but Powershell has a pretty awesome integration with AWS. This is the way I do it in Powershell:

Your JSON value

{"API_KEY":"ABCDEFGHI"}

$aws_secret = Get-SECSecretValue -SecretId my_secrets
$mysecret = $aws_secret.SecretString | ConvertFrom-Json
$myapikey = $mysecret.API_KEY
$newsecret = ConvertTo-SecureString -String $myapikey -AsPlainText -Force

The value from the secret manager is a JSON which Powershell can natively convert into a type of array that you can reference. I convert it back into a secure string under the assumption its a secret and you want to pass it in. The code above should work for you. Let me know if you run into any issues with the code I provided.

Unbecoming

Answer 13

One liner to list all values in SecretString using PowerShell.

(aws secretsmanager get-secret-value --secret-id secretId | ConvertFrom-Json).SecretString | ConvertFrom-Json

Retardant

Answer 14

If you have vector by DataDog installed and do not want to use jq, the VectorRemapLanguage (vrl) is able to do the work. In this bash function we use map_values to replace all JSON fields by there parsed value:

function vq() {
  local data=$1
  local query=$2
  data=${data//$'\n'/}  # Remove newlines
  r=$(($HOME/vector/bin/vector vrl | tail -n 2) << EOF
. = map_values($data) -> |value| {
  struct, err = parse_json(value)
  if err != null {value} else {struct}
}
.$query
EOF
)
  printf '%s' "${r//\"/}"
}

Then retrieve the AWS secret with

data=$(aws secretsmanager get-secret-value --secret-id MyName)
vq "$data" SecretString.API_KEY
# -> ABCDEFGHI

Vasti

Answer 15

Use this to get just the value of the secret key. Make sure to fill in your secert ID and the key of the secret:

aws secretsmanager get-secret-value --secret-id <yourSecretID> | jq '.SecretString' | tail -c +2 | head -c -2 | tr -d '\' | jq .<YourSecretKey>

Pyramid