jq to replace text directly on file (like sed -i)

Question

I have a json file that needs to be updated on a certain condition.

Sample json

{
   "Actions" : [
      {
         "value" : "1",
         "properties" : {
            "name" : "abc",
            "age" : "2",
            "other ": "test1"
          }
      },
      {
         "value" : "2",
         "properties" : {
            "name" : "def",
            "age" : "3",
            "other" : "test2"
          }
      }
   ]
}

I am writing a script that makes use of Jq to match a value and update, as shown below

cat sample.json |  jq '.Actions[] | select (.properties.age == "3") .properties.other = "no-test"'

Output (printed to terminal)

{
  "value": "1",
  "properties": {
    "name": "abc",
    "age": "2",
    "other ": "test1"
  }
}
{
  "value": "2",
  "properties": {
    "name": "def",
    "age": "3",
    "other": "no-test"
  }
}

While this command makes the needed change, it outputs the entire json on the terminal and does not make change to the file itself.

Please advise if there is an option to have jq make changes on the file directly (similar to sed -i).

Bah

For a number of general solutions to "how do I change a file in-place" see also #6697342 – Merrie

FWIW, there is a feature request open here: github.com/stedolan/jq/issues/105 – Bula

Answer 1

This post addresses the question about the absence of the equivalent of sed's "-i" option, and in particular the situation described:

I have a bunch of files and writing each one to a separate file wouldn't be easy.

There are several options, at least if you are working in a Mac or Linux or similar environment. Their pros and cons are discussed at http://backreference.org/2011/01/29/in-place-editing-of-files/ so I'll focus on just three techniques:

One is simply to use "&&" along the lines of:

jq ... INPUT > INPUT.tmp && mv INPUT.tmp INPUT

Another is to use the sponge utility (part of GNU moreutils):

jq ... INPUT | sponge INPUT

The third option might be useful if it is advantageous to avoid updating a file if there are no changes to it. Here is a script which illustrates such a function:

#!/bin/bash

function maybeupdate {
    local f="$1"
    cmp -s "$f" "$f.tmp"
    if [ $? = 0 ] ; then
      /bin/rm "$f.tmp"
    else
      /bin/mv "$f.tmp" "$f"
    fi
}

for f
do
    jq . "$f" > "$f.tmp"
    maybeupdate "$f"
done

Nihil

If the document isn't too big for the command line, a file can be avoided: json="$( jq ... file.json )" plus printf '%s\n' "$json" >file.json – Micromillimeter

Answer 2

instead of sponge :

cat <<< $(jq 'QUERY' sample.json) > sample.json

Weintraub

Is cat really able to replace sponge? Is this guaranteed to always work? – Ulster

This isn't working for me on ubuntu 18.04 with jq 1.5.1. Sample.json is empty after running command. – Braid

Yeah this is nice but probably best to not overwrite the source file. It will be empty if there was an issue and stdout shows nothing. This is great when you need to copy+modify to somewhere else. – Meiny

This worked great for me but how to write formatted (pretty) json? This one writes in one single line. – Addressograph

This worked fine for me on Ubuntu 18.04 - my example command cat <<< $(jq '.name = "value"' configuration.json) > configuration.json – Lowgrade

This results in a blank file on RHEL7 – Felten

I must admit I was suspicious at first, but apparently here strings are evaluated before launching a pipeline. Tested on Arch Linux, bash-5.1.8 and Alpine Linux, bash-5.0.11. Try this echo '{"a": 1}' > a.json && cat <<< $(jq .b=2 a.json) > a.json && jq . a.json. – Thrive

yes, $() must be resolved by bash prior to invocation of a command because otherwise there's no way to invoke the command with its positional parameters. In this case, the substitution is being invoked to generate stdin and technically could be invoked after the command starts, but quite often $() is used for positional parameters. – Rodina

This solution has a race between jq reading the file and shell truncating it at the same time. Even if it worked for you, it cannot be guaranteed to always work. – Cammiecammy

This indeed does not solve the issue, you'll also see a shellcheck warning – Glendaglenden

Oh, that's pretty. "Works for me". +1 and thank you @Thrive – Rufusrug

For those of you with an empty file afterward, ensure you escaped your replacements according to the type inferred. For example, you may need both single and double quotes: jq .b='"2.3.4"' is very different from jq .b=2.3.4, and results in an empty file. If @Thrive 's example works for you then you're good to go. – Rufusrug

@BrianChrisman What does it have to do with positional parameters? cat <<< $(echo a b c) doesn't try to open files a, b or c. What do you mean by "resolved"? How can $() that follows >>> be used for positional parameters? | There indeed might be a race condition here. That might go unnoticed in some (or maybe most of the) cases. – Thrive

@Thrive not a lot, but for example: echo -e 'a\nb\nc\nd' > a; echo "$(grep -v b a)" > a; wc -l a outputs '3'. The parameter to echo, the grep, is being evaluated before file 'a' gets truncated for output. It's a hacky way of doing something like 'sed -i' with lots of caveats. – Rodina

Answer 3

You ran into two issues:

• This is a common problem for text processing, not solved in the base Linux distribution.
• jq did not write special code to overcome this problem.

One good solution:

• Install moreutils using brew install moreutils or your favorite package manager. This contains the handy program sponge, for just this purpose.
• Use cat myfile | jq blahblahblah | sponge myfile. That is, run jq, capturing the standard out, when jq has finished, then write the standard output over myfile (the input file).

Prologize

Answer 4

I use yq, For advanced users this -i (in-place update) is needed, hope be added to jq

yq -iP '.Email.Port=3030' config.json -o json

• -i in place update
• -P pretty print
• -o output should be json

---

yq --version
yq (https://github.com/mikefarah/yq/) version 4.21.1

Arterial

Answer 5

You'll want to update the action objects without changing the context. By having the pipe there, you're changing the context to each individual action. You can control that with some parentheses.

$ jq --arg age "3" \
'(.Actions[] | select(.properties.age == $age).properties.other) = "no-test"' sample.json

This should yield:

{
  "Actions": [
    {
      "value": "1",
      "properties": {
        "name": "abc",
        "age": "2",
        "other ": "test1"
      }
    },
    {
      "value": "2",
      "properties": {
        "name": "def",
        "age": "3",
        "other": "no-test"
      }
    }
  ]
}

You can redirect the results to a file to replace the input file. It won't do in-place updates to a file as sed does.

Copybook

Thanks Jeff, this is super helpful. What tool would you recommend for making conditional json changes, directly to the file? I have a bunch of files and writing each one to a separate file wouldn't be easy. Thanks again. – Bah

If you need to do it in the command line, jq is great. You can do quite a lot with it. If you need to do more complex updates with more control, I'd just write a script to do the updates using your favorite scripting/programming language. – Copybook

Answer 6

use tee command

➜ cat config.json|jq '.Actions[] | select (.properties.age == "3") .properties.other = "no-test"'|tee config.json
{
  "value": "1",
  "properties": {
    "name": "abc",
    "age": "2",
    "other ": "test1"
  }
}
{
  "value": "2",
  "properties": {
    "name": "def",
    "age": "3",
    "other": "no-test"
  }
}

➜ cat config.json
{
  "value": "1",
  "properties": {
    "name": "abc",
    "age": "2",
    "other ": "test1"
  }
}
{
  "value": "2",
  "properties": {
    "name": "def",
    "age": "3",
    "other": "no-test"
  }
}

Petry

If you get this command wrong, you end up with an empty config.json file – Plausible

Answer 7

Using my answer to a duplicate question

Assignment prints the whole object with the assignment executed so you could assign a new value to .Actions of the modified Actions array

.Actions=([.Actions[] | if .properties.age == "3" then .properties.other = "no-test" else . end])

I used an if statement but we can use your code to do the same thing

.Actions=[.Actions[] | select (.properties.age == "3").properties.other = "no-test"]

The above will output the entire json with .Actions edited. jq does not had sed -i like functionality, but all you need to do is pipe it back into a sponge to the file with | sponge

 jq '.Actions=([.Actions[] | if .properties.age == "3" then .properties.other = "no-test" else . end])' sample.json | sponge sample.json

Insurer

Piping output to input along the lines of `CMD < FILE > FILE' or equivalent is generally severely deprecated as explained for example at #3055505 There are many good alternatives so please adjust your response accordingly. – Nihil

Answer 8

It's possible to do something like:

echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' /etc/docker/daemon.json)" > /etc/docker/daemon.json

So it gets text in sub-shell using jq and echoes it to file in 'main' shell.

Note: The main idea here is to illustrate how it can be achieved without additional tools like sponge or so. Instead of echo you can use any command which can write to stdout e.g. printf '%s' "$(jq ... file)" > file.

P.S Issue in jq project is still open: https://github.com/stedolan/jq/issues/105

Verbiage

will remove \ from content like this {"transform": {"^.+\\.tsx?$": "ts-jest"}} -> {"transform": {"^.+\.tsx?$": "ts-jest"}} – Thromboplastin

As I said it's one of possible ways to do that, sorry I didn't solve your issue but have you tried to use printf instead of echo? – Verbiage

Answer 9

The simplest way of accomplishing it is to load the file into a variable first and then send it into jq.

content=$(cat sample.json) && 
jq '<your jq script>' <<<$content >sample.json

Cammiecammy

Answer 10

This bash (probably sh compatible) function jqi will take care of everything.

Usage: jqi [-i] <filename> [jq options] <jq filter>

e.g.:

fix-node-sass() 
{ 
    jqi -i package.json '.resolutions += {"node-sass": "6.0.1"}' \
                  '| .devDependencies += {"node-sass": "6.0.1"}'

}

Much like sed or perl, specify -i as the leading argument to force rewriting of the original file. If -i is not specified, it will be a "dry run" and output will go to stdout.

If for some arcane reason you want to do something weird like:

cat in.json | jq -i - > out.json

Then out.json will hold either the result, or the original contents of in.json on error -- i.e., out.json should be valid json.

Note: an output of less than 7 characters (e.g. null) is considered an error, and will not overwrite. You can disable this safety feature if you wish.

jqi () 
{ 
    local filename=$1;
    shift;
    local inplace=;
    local stdin=;
    if [[ $filename == "-i" ]]; then
        echo "jqi: in-place editing enabled" 1>&2;
        inplace=y;
        filename=$1;
        shift;
    fi;
    if [[ $filename == "-" ]]; then
        echo "jqi: reading/writing from stdin/stdout" 1>&2;
        if [ -n "$inplace" ]; then
            stdin=y;
            inplace=;
        fi;
        filename="/dev/stdin";
    fi;
    local tempname="$( mktemp --directory --suffix __jq )/$( dirname "$filename" ).$$.json";
    local timestamp="${tempname%json}timestamp";
    local -i error=0;
    cat "$filename" > "$tempname";
    touch "$timestamp";
    while :; do
        if jq "${*}" "$filename" > "$tempname"; then
            if test "$tempname" -nt "$timestamp"; then
                local ls_output=($( ls -Lon "$tempname" ));
                filesize=${ls_output[3]};
                if [[ $filesize -lt 7 ]]; then
                    echo "jqi: read only $filesize bytes, not overwriting" 1>&2;
                    error=1;
                    break;
                fi;
                if [ -n "$inplace" ]; then
                    cat "$tempname" > "$filename";
                else
                    echo "jqi: output from dry run" 1>&2;
                    cat "$tempname";
                fi;
                error=0;
                break;
            else
                echo "jqi: output not newer, not overwriting" 1>&2;
                error=1;
                break;
            fi;
        else
            echo "jqi: jq error, not overwriting" 1>&2;
            error=1;
            break;
        fi;
    done;
    if [ -n "$stdin" ] && [ $error -eq 1 ]; then
        echo "jqi: output original to stdout" 1>&2;
        cat "$filename";
    fi;
    rm "$tempname" "$timestamp";
    rmdir "$( dirname "$tempname" )"
}

Litharge

Answer 11

in one line : cat file.json | jq '.' | tee file.json.json >/dev/null

Crat

Remember that Stack Overflow isn't just intended to solve the immediate problem, but also to help future readers find solutions to similar problems, which requires understanding the underlying code. This is especially important for members of our community who are beginners, and not familiar with the syntax. Given that, can you edit your answer to include an explanation of what you're doing and why you believe it is the best approach? – Rattrap