Using yq in for loop bash

Question

I have a yaml array like below,

identitymappings:
- arn: "arn:aws:iam::12345567:role/AdmRole"
  group: "system:masters"
  user: "user1"
- arn: "arn:aws:iam::12345567:role/TestRole"
  group: "system:masters"
  user: "user2"

I am trying to parse this yaml in a bash script using for loop and yq.

 for identityMapping in $(yq read test.yaml "identitymappings[*]"); do
      roleArn=$identityMapping["arn"]
      group=$identityMapping.group
      user=$identityMapping.user
done

But I am not getting the expected results like not able to fetch the values of roleArn,group,user. Please let me know how to fix this.

Deluca

This is very closely related to iterate over json with jq. – Strickler

Answer 1

The way I would do it is:

# load array into a bash array
# need to output each entry as a single line
readarray identityMappings < <(yq e -o=j -I=0 '.identitymappings[]' test.yml )

for identityMapping in "${identityMappings[@]}"; do
    # identity mapping is a yaml snippet representing a single entry
    roleArn=$(echo "$identityMapping" | yq e '.arn' -)
    echo "roleArn: $roleArn"
done

output:

roleArn: arn:aws:iam::12345567:role/AdmRole
roleArn: arn:aws:iam::12345567:role/TestRole

Disclaimer: I wrote yq

Monique

note that the latest yq now also has a '@tsv' operator (so you can do it the way @charles-duffy posted without needing jq). – Monique

The disclaimer though xD. – Hemicycle

yq e .identitymappings[] file.yml – Tronna

Answer 2

I wasn't able to comment on Charles Duffy's proper answer, but this works for yq v4 without the use of jq...

while IFS=$'\t' read -r roleArn group user _; do
  echo "Role:  $roleArn"
  echo "Group: $group"
  echo "User:  $user"
done < <(yq e '.identitymappings[] | [.arn, .group, .user] | @tsv' test.yaml)

Afternoon

Good to know; BTW, TSV values might be escaped (with \t \\ \n etc...); you can unescape them with for example printf -v roleArn %b "$roleArn" – Olvera

Answer 3

There is an improvement of @Rad4's answer that worked for me.

You can neatly loop through using latest yq and jq via:

for im in $(yq eval -o=j test.yaml | jq -cr '.identitymappings[]'); do
      arn=$(echo $im | jq -r '.arn' -)
      group=$(echo $im | jq -r '.group' -)
      user=$(echo $im | jq -r '.user' -)
      echo $arn $group $user
done

This loops through valid online jsons, which makes jq still work inside the loop.

Cartier

Answer 4

1. Get identitymappings length
2. Using index to access element of identitymappings

array_length=`yq e ". identitymappings | length - 1" test.yaml`

if [ $array_length -le 0 ] ; then
  exit
fi

for element_index in `seq 0 $array_length`;do
    arn=`yq e ".identitymappings[$element_index]. arn" test.yml`
    group=`yq e ".identitymappings[$element_index]. group" test.yml`
    user=`yq e ".identitymappings[$element_index]. user" test.yml`
done

1. How to get length of array in yq?
   • https://mikefarah.gitbook.io/yq/operators/length

Cad

I find this approach the most straight forward. It allows to utilise yq for parsing data, instead of worrying about the outputs of yq and reading them in bash. The only thing I would change is replace backticks with $() – Rudimentary

Answer 5

The easiest way to read from jq or yq into bash is to use a BashFAQ #1 while read loop to handle line-oriented data; in the below, we use @tsv to generate line-oriented output:

while IFS=$'\t' read -r roleArn group user _; do
  echo "Role:  $roleArn"
  echo "Group: $group"
  echo "User:  $user"
done < <(yq -j read test.yaml \
         | jq -r '.identitymappings[] | [.arn, .group, .user] | @tsv')

Note that if you were using the Python yq rather than the Go one, you could remove the yq -j read and just use yq -r '...' in place of jq -r '...'.

Strickler

Answer 6

The answer by @mike.f is a good one. However, it does not work on OSX machines, because readarray is not an available command. You can read more about this here.

Here is the equivalent that would work on a mac:

# load array into a bash array
# need to output each entry as a single line
identitymappings=( $(yq e -o=j -I=0 '.identitymappings[]' test.yml ) )

for identityMapping in "${identityMappings[@]}"; do
    # identity mapping is a yaml snippet representing a single entry
    roleArn=$(echo "$identityMapping" | yq e '.arn' -)
    echo "roleArn: $roleArn"
done

Alterable

Answer 7

My solution, using yq map function:

  yq e '.identitymappings | map([.arn, .group, .user] | join(" ")) | .[]' file.yaml \
| while read -r arn group user; do
    echo $arn
  done

Ornithic

Answer 8

I figured out..

for identityMapping in $(yq read test.yaml -j "identitymappings[*]"); do
      echo $identityMapping
      roleArn= echo $identityMapping | jq -r '.arn'
      echo $roleArn
      group= echo $identityMapping | jq -r '.group'
      echo $group
      user= echo $identityMapping | jq -r '.user'
      echo $user

Deluca

No. This isn't assigning values to roleArn, group or user at all. Your output isn't written from the echos, it's written by the jq commands. – Strickler

That's because roleArn= echo $identityMapping | jq -r '.arn' doesn't assign to roleArn, it's just setting roleArn as an environment variable with an empty value during the execution of echo. – Strickler

...see How do I set a variable to the output of a command in bash? – Strickler