Merging two yaml documents while concatenating arrays

Question

I want to merge two yaml documents with the result containing

• all mapped values (with the last one taking precedence)
• concatenated arrays

e.g. given this file

# file1.yml
animals:
  - elephant
    donkey
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
strange: true

and this file

#file2.yml
animals:
  - monkey
    pelican
flavours:
  sweet:
    - vanilla
strange: false

the result should contain all the nodes, with merged arrays and values that are not arrays from the last file

#result.yml
animals:
  - elephant
    donkey
    monkey
    pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
      vanilla
strange: false

Can yq do this, maybe?

Sphacelus

My experience with yq is that it is quite quick to abort with strange errors once you do something complex. This answer has usable code that solves the problem in Go if that helps you. – Ferriter

Which version of yq are you using? The Go or the Python version? – Mahan

I tried using the latest Go version. I have now got it running using v3.4.1, which may be less powerful, but also a lot less complex. Any pointers as to how to get it to work with 4.x are appreciated. – Sphacelus

Answer 1

Merging YAML files with yq, concatenating arrays.

Assumption

In both file1.yml and file2.yml, the animals array contains a single index with a multiline string.

elephant
donkey

for file1.yml, and

monkey
pelican

for file2.yml.

Since you asked about concatenated array, I am assuming file1.yml, file2.yml and results.yml should be like this:

# file1.yml
animals:
  - elephant
  - donkey
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
strange: true

---

# file2.yml
animals:
  - monkey
  - pelican
flavours:
  sweet:
    - vanilla
strange: false

---

# result.yml
animals:
  - elephant
  - donkey
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
    - vanilla
strange: false

Example

With yq 4.x, you can use the ireduce operator to merge two or more files:

https://mikefarah.gitbook.io/yq/operators/reduce#merge-all-yaml-files-together

$ yq4 eval-all '. as $item ireduce ({}; . * $item)' file1.yml file2.yml
animals:
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - vanilla
strange: false

Both files have been merged, but duplicated keys and arrays have been overwritten by the latest file.

To appends arrays instead of overriding them, simply add a + after the merge operator (*).

https://mikefarah.gitbook.io/yq/operators/multiply-merge#merge-appending-arrays

$ yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file1.yml file2.yml
animals:
  - elephant
  - donkey
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
    - vanilla
strange: false

Swanson

I am having the similar requirement, how to merge and save into another file ? is it possible to write this content into another file ? – Corned

@DnyaneshwarJadhav. With yq ou can modify a file --inplace. When multiple files are used, only the first one will be modified. Note that you can always redirect the output to another file: yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file1.yml file2.yml > anotherFile.yml – Swanson

In terms of duplicate variable entries (like strange: false vs strange: true) the right most file wins. yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file1.yml file2.yml > anotherFile.yml --> anotherFile.yml having strange: false, but yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file2.yml file1.yml > anotherFile.yml --> anotherFile.yml would have strange: true – Undersigned

Thank you, @jpmorin! yq eval-all '. as $item ireduce ({}; . *+ $item)' merges nicely ~/.kube K8s configs. – Supernatural

What about multiple files? fille.yml file2.yml and file3.yml. – Mannered

In yq v4.44.3 It does not work as expected when the arrays elements are objects, it does not seem to be a difference between using * or *d (deep), and *+ will just append items, creating duplicates. So running the same expression more than once will just add data, will concatenate, but will not overwrite existing objects – Seko