Convert bash array to json array and insert to file using jq

Question

Given a bash array, how to convert it to a JSON array in order to output to a file with jq?

Additionnally: is there a way to keep the server_nohup array unchanged instead of re-writing the whole json file each time?

newArray=(100 200 300)
jq -n --arg newArray $newArray '{
    client_nohup: [ 
        $newArray
    ],
    server_nohup: [

    ]
}' > $projectDir/.watch.json

Current output:

{
"client_nohup": [
    "100"
],
"server_nohup": []
}

Desired output:

{
"client_nohup": [
    100,
    200,
    300
],
"server_nohup": []
}

Adenitis

Answer 1

Given a bash array, how to convert it to a JSON array

Assuming your jq supports the --args command-line option, you can use it in conjunction with the built-in $ARGS pseudo-variable, as illustrated by this example:

$ declare -a output=($'abc\ndef' "x y z")
$ jq -c -n '$ARGS.positional' --args "${output[@]}"
["abc\ndef","x y z"]

Here, all the values of the bash array are converted to JSON strings. Notice that the embedded new-line in the first value of the bash array is correctly handled.

Additionally ...

(1) If all the values in newArray are valid as JSON values without spaces, then you could get away with piping the values as a stream, e.g.

newArray=(100 200 300)
echo "${newArray[@]}" |
  jq -s '{client_nohup: ., server_nohup: []}'

(2) Now let's suppose you merely wish to update the "nohup" object in a file, say nohup.json:

{ "client_nohup": [], "server_nohup": [ "keep me" ] }

Since you are using bash, you can then write:

echo "${newArray[@]}" |
  jq -s --argjson nohup "$(cat nohup.json)" '
    . as $newArray | $nohup | .client_nohup = $newArray
  '

Output

(1)

{
  "client_nohup": [
    100,
    200,
    300
   ],
  "server_nohup": []
}

(2)

{
  "client_nohup": [
    100,
    200,
    300
  ],
  "server_nohup": [
    "keep me"
  ]
}

Other cases

Where there's a will, there's a jq way :-)

See for example the accepted answer at How to format a bash array as a JSON array (though this is not a completely generic solution).

Further thoughts

You might wish to convert numeric strings to JSON numbers, e.g. using the jq idiom: (tonumber? // .).

You might also wish to exploit the fact that bash strings cannot contain NUL characters.

Catenane

Thanks for a thorough answer, but does case 1 need the value for key client_nohup to be an array nested in another array? – Coan

It seems I copied-and-pasted the wrong output. Fixed. – Catenane

This would work for simple values, but in general will fail because the array elements could contain newlines or other invalid JSON characters. – Eardrum

@Eardrum -please note that your point was made in the answer, both at the end and in the final section, which points to techniques which can be applied in other cases. – Catenane

The answer in the FAQ fails for the array x=(1 $'a\nb' 2). – Eardrum

@Eardrum - Please read the FAQ more carefully. In particular, your case is covered using NUL. – Catenane

No, it isn't. The problem is that $'a\n'b is not a valid JSON value (it contains a newline character. If you are sure that each element of the array is already valid JSON, there would be no problem with using a newline to separate the values. – Eardrum

Please read my counterexample more closely; I am using $'a\nb' to include a literal newline, not 'a\nb' which contains a digraph that can be interpreted as a newline. -R does not help. My point is, indeed, that you cannot encode an arbitrary bash array, only an array of JSON values. – Eardrum

Ok, my apologies, I did gloss over "are valid as JSON values" in the first sentence. – Eardrum

And my apologies - the FAQ example should also have included the -s option in addition to -R. Fixed. – Catenane

@Catenane is there any way I could maintain the format instead of becoming a string line. Now I have array=(“xxx”,”yyy”), I want it to insert to file like [xxx,yyy] not simply string xxx yyy – Less

@NicHuang - Sorry, I do not understand the question. If your question isn't the same as the one here, why not ask yours as a top-level SO question? – Catenane

Answer 2

In general, the only truly safe way to do this is with multiple invocations of jq, adding each element to the output of the previous command.

arr='[]'  # Empty JSON array
for x in "${newArray[@]}"; do
  arr=$(jq -n --arg x "$x" --argjson arr "$arr" '$arr + [$x]')
done

This ensures that each element x of your bash array is properly encoded prior to adding it to the JSON array.

This is complicated, though, by the fact that bash doesn't not distinguish between numbers and strings. This encodes your array as ["100", "200", "300"], not [100, 200, 300]. In the end, you need to have some awareness of what your array contains, and preprocess it accordingly.

Eardrum

It is not the case that ‘the only truly safe way to do this is with multiple invocations of jq’. The jq FAQ shows how this can be avoided using NUL. Search for NUL in the FAQ github.com/stedolan/jq/wiki/FAQ – Catenane

One can get around the number/string problem using tonumber? // ., though of course that would convert quoted numbers to numbers ... – Catenane

@Catenane The problem here isn't the separator, but that bash strings aren't necessarily valid JSON strings. It's broken for the same reason that a hard-coded value like $'"foo\bar"' isn't valid JSOn. – Eardrum

As mentioned elsewhere, you have evidently overlooked the use of the -R option. Of course this forces everything to be a JSON string, and thus your point about the distinction between numbers and strings being lost in a "generic" solution is perfectly valid.0. – Catenane