How to merge multiple json files in a directory with jq or any tool?

Question

I am trying to merge all json files in a directory (can be many).

I know that if there are only 2 files, I can use jq -s . file1.json file2.json. But when I tried to use the a wild card like jq -s . file*.json, it fails. Any idea?

Papst

How does it fail? Any error messages? – Methylal

Mergine many json file into one – Scrim

Answer 1

Try this:

jq -s add *.json > merged.json

Seringapatam

Answer 2

The answer Here is the jq answer... for more see the explanation below:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

Step by step explanation

To merge two json files using jq use this command:

jq -s ".[0] * .[1]" file1.json file2.json

Now to merge three json files:

jq -s ".[0] * .[1] * .[2]" file1.json file2.json file3.json

Let's follow the same logic and generalise for n files using wildcards. The difficult part is to generate the string .[0] * .[1] ... .[n-1].

First let's find n. n is the number of json files in the folder. so:

ls *.json | wc -l

now let's write a loop that will build the string ".[1] * .. * .[a] * .[a+1] * .. * .[n-2] *". The kernel of this string is .[a] * so:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done;

ok and now let's add the last part .[n-1]:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";

now let's pass that string to the jq command... and let's parameterise *.json so that we can replace it with whatever we want with ease:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

et voilà!

Geisel

Answer 3

My JSON files contain JSON object. jq -s add *.json did not work, but the following works

jq -s '.' *.json > merged.json

Buffer

Answer 4

From what I've seen, jq -s add *.json merges objects only at the top level (i.e. not recursively), and jq -s '.' *.json creates an array with each element being one of the JSON files.

The command Ouss wrote just hung for me, so I don't know about that.

I ended up using the following, which simple enough and worked for me to perform a recursive merge of multiple JSON files:

jq -s 'reduce .[] as $obj ({}; . * $obj)' *.json

Intend

Answer 5

While waiting a jq's solution, let me address the question "... or any tool?"

Different people mean different things by "merging", thus assuming here a plain recursive JSON merging, then with jtc (another tool) the solution would be like this (for an arbitrary number of files):

• read all the JSONs into one super JSON array
• merge into the 1st child (the first read JSON) the rest of read JSONs
• output the resulting 1st JSON child in the array

jtc -J / -w[0] -mi[1:] / -w[0] *.json 

to illustrate the solution, here's how it works with couple arbitrary JSON files:

bash $ jtc -tc file1.json 
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": { "seconds": 1400 }
   },
   "total": { "seconds": 1490 }
}
bash $ jtc -tc file2.json 
{
   "days": {
      "2020-05-20": { "seconds": 95 },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": { "seconds": 295 }
}
bash $ jtc -J / -w[0] -mi[1:] / -w[0] -tc file*.json
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": {
         "seconds": [ 1400, 95 ]
      },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": {
      "seconds": [ 1490, 295 ]
   }
}
bash $ 

-tc flag is used only to display JSONs here in a compact format

PS. It so happens that I'm the creator of jtc tool - unix JSON processing utility.

Damali

I am sorry that I did not post my question clearly. The jsons are all with the same format, just with different data. I need to run the tool on windows as our servers are all windows. – Papst

Does jtc have a windows version? I could not find it so far. – Papst

The commands of some of the "answers" from the various threads here in stackoverflow or stackexchange do not work. For example, I used command jq -s '{ attributes: map(.attributes[0]) }' *.json. But jq throws an error "This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information." – Papst

If I used the simple command "jq.exe -s . *.json", I get exactly the same error thrown by jq. Not sure if jq supports wildcard on windows platform? If I only passed 2 files, jq does work properly. – Papst

No, there's no native version of jtc for widows yet. But it's easily compilable under cygwin environment – Damali

wildcard (*) is not directly supported by end application, it's the shell's job to expand it into the list of files and supply (the list) to the end application – Damali

> I get exactly the same error thrown by jq - why don't you post the error that you're getting? it would be easier then to help you – Damali

Here is the error: D:\work\json>jq.exe -s . *.json Assertion failed! Program: D:\work\json\jq.exe File: src/main.c, Line 256 Expression: wargc == argc This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. – Papst

here's the answer to that error: #53388311 – Damali

Thanks, Dmitry. I will try out that cmd script. – Papst

Just tested the cmd. It runs but the json outputed is in bad format. Looks like the cmd simply appends the data in the json files together which obviously is no good. – Papst

I don't know how to show the json test files in the site as there is a character limitation – Papst

I ran the cmd as "dir /b *.json | list2args.cmd jq ." as suggested by the thread – Papst

I'd suggest you to file it as a separate question. Someone with the access to windows should be able to help you. Because now it becomes not a JSON query but rather a OS specific question. – Damali