yq v4: print all key value pairs with full key path

Question

I'm trying to determine the correct syntax for using yq to print all key/value pairs from a given yaml input using yq v4 - with the desired output having the full key "path". This was possible using v3 such as this:

$ cat << EOF | yq r -p pv - '**'
> a:
>   b: foo
>   c: bar
> EOF
a.b: foo
a.c: bar

but I'm having difficulty wrapping my head around the new syntax.

Any help is greatly appreciated.

Lyre

Answer 1

$ cat << EOF | yq e '.. | select(. == "*") | {(path | join(".")): .} ' -
> a:
>   b: foo
>   c: bar
> EOF
a.b: foo
a.c: bar

What does this do? Let's go over it:

• .. recursively select all values
• select(. == "*") filter for scalar values (i.e. filter out the value of a)
• (path | join(".")) gets the path as array and joins the elements with .
• {…: .} create a mapping, having the joined paths as keys and their values as values

Edit: to get sequence indexes in square brackets ([0] etc), do

$ cat << EOF | yq e '.. | select(. == "*") | {(path | . as $x | (.[] | select((. | tag) == "!!int") |= (["[", ., "]"] | join(""))) | $x | join(".") | sub(".\[", "[")): .} ' -

This seems like there should be a simpler way to do it, but I don't know yq well enough to figure it out.

Git

Hi flyx, Thank you for this.. if I have an list of values, it yields me something like this - a.b.0.name: testname a.b.0.state: teststate ......is it possible to get it out in standard yaml query path like this - a.b[0].name: testname a.b[0].state: teststate – Ecosphere

@PonsonThankavel I updated the answer with code that does what you want. btw, you'll find no mention of „yaml query path“ in the YAML spec. I think this form is mainly used in conjunction with Spring boot, where this is the way to transform YAML into Java properties, but there's nothing standard about it. – Git

Thank you very much... The output now has the array element notation now but it still contains a dot (.) character before the array index (like a.b.[0].state). is it possible to get rid of that dot character (like a.b[0].state).. Tried to tweak your expressions but couldn't figure this out... – Ecosphere

tweaked the command like this and made it work along with sed... $ cat << EOF | yq e '.. | select(. == "*") | {(path | . as $x | (.[] | select((. | tag) == "!!int") |= (["||||[", ., "]"] | join(""))) | $x | join(".")): .} ' - | sed -e 's/.||||\[/\[/g' ..... Thank you flyx... you are awesome... – Ecosphere

@PonsonThankavel Ah yes, I missed the dot. I updated the command to replace any .[ with [, that should do it. Basically what your sed command does. – Git

works very well... Thank you again flyx.... – Ecosphere

Tried it with the most recent mikefarah.gitbook.io/ - it works flawlessly. – Veldaveleda

I appreciate the breakdown of the yq query and the explanation for each step. The version with indices in square backets didn't work, however. The output was the same as the other version. (Using yq (https://github.com/mikefarah/yq/) version 4.27.2.) – Patchwork

I was expecting a bug if properties b: and c: have the same value, but somehow piping to path sorts it out. Anyone know how path identifies which key a given value belongs to? – Notional

@Notional That's because yq works on nodes. A node may be a scalar value in which case it has textual content, but it is always uniquely identified inside the YAML structure you're operating on. – Git

@Mr.LanceESloan I found the same issue, that the indices were not bracketed. I was able to address that with this version: .. | select(. == "*") | . as $x | { ( [ path | ( .[] | select((. | tag) == "!!int") |= (["[", ., "]"] | join("")) ) ] | join(".") | sub(".\[", "[") ) : $x } – Ibbison