How to convert a json response into yaml in bash

Question

I read data from a json file with jq. I wanna append the results into a yaml file, but I dont get it working. I am quite new to shell programming. My goal is to append that "users" to an existing "users"-Array in a yaml file.

This is my json file:

#$DEFAULTS_FILE

{"users":
  [
    {"name":"pi",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "groups":"users,docker,video",
      "plain_text_passwd":"pi",
      "lock_passwd":"false",
      "ssh_pwauth":"true",
      "chpasswd": {"expire": false}
    },
    {"name":"admin",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "primary-group": "users",
      "groups":"users,docker,adm,dialout,audio,plugdev,netdev,video",
      "ssh-import-id":"None",
      "plain_text_passwd":"pi",
      "lock_passwd":"true",
      "ssh_pwauth":"true",
      "chpasswd": "{expire: false}",
      "ssh-authorized-keys": ["ssh-rsa abcdefg1234567890 [email protected]"]
    }
  ]
  }

I filter it with that:

cat $DEFAULTS_FILE | jq .users

I have no clue how to convert that json into a yaml.

My expected result should be:

users:
  - name:                pi
    gecos:               "Hypriot Pirate"
    sudo:                ALL=(ALL) NOPASSWD:ALL
    shell:               /bin/bash
    groups:              users,docker,video
    plain_text_passwd:   pi
    lock_passwd:         false
    ssh_pwauth:          true
    chpasswd: { expire:  false }
  - name:                admin
    primary-group:       users
    shell:               /bin/bash
    sudo:                ALL=(ALL) NOPASSWD:ALL
    groups:              users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:       None

I tried to use a second tool called yq which is similar to jq and can write yaml files. But I have no positive progress.

EDIT

I know that I can add content to the yaml with that:

yq w -i "my.yml" "users[+]" "some content"

But I dont know how to merge my json into that.

Any help or hint would be nice, thank you in advance...

Dichromate

You can use python/perl which have in-built YAML and JSON modules - commandlinefu.com/commands/view/12218/convert-yaml-to-json – Receiver

That's exact the opposite direction – Dichromate

Bash itself is not a good platform for this. Find an existing tool, or write one in e.g. Python. Depending on your preferred language, it should not be hard to find an existing question on Stack Overflow with suggestions; here's a search for Python – Prolamine

Please follow the minimal reproducible example guidelines. In particular, a sample my.yml (i.e. input) would be helpful. – Vanessa

yq r input.json should do the job. – Chromoplast

Thank you all, I followed @Receiver `s approach to write a 'lil script in Ruby. and @Jeff Mercado. I don't know why this is quoted, but this is exactly what the program spit. – Dichromate

as @Chromoplast pointed out yq r works (as JSON is a subset of YAML); the output should be pretty-printed, though to make it "look more like YAML": yq -P r input.json; see also mikefarah.gitbook.io/yq/usage/convert#json-to-yaml – Soffit

YAML spec says: "every JSON file is also a valid YAML file" – Moira

Answer 1

yq a yaml wrapper for jq

With yq 4.18.1+

cat "$DEFAULTS_FILE" | yq -P    # or yq --prettyPrint

See: https://mikefarah.gitbook.io/yq/#notice-for-v4.x-versions-prior-to-4.18.1

With yq version 4.8.0:

cat $DEFAULTS_FILE | yq e -P -

• e or eval handles file separately. ea or eval-all will merge files first.
• -P or --prettyPrint YAML output
• - from STDIN

Note: you can go the other way (yaml to json) too yq e -j file.yaml

With yq version 3.3.2:

cat $DEFAULTS_FILE | yq r -P -

• r read
• -P --prettyPrint
• - from STDIN

Infusible

You're a mad lad! Thanks mate! – Gustavo

This answer needs an edit. Your yq example is an implementation of mikefarah/yq, which is NOT a wrapper over jq. See my answer – Receiver

If using mikefarah/yq V4, you need to use the following command: yq eval '.. style= ""' sample.json or cat sample.json | yq eval '.. style= ""' -. See mikefarah.gitbook.io/yq/usage/convert – Apia

Actually, all you need is cat $DEFAULTS_FILE | yq -y – Coadjutrix

With mikefarah/yq version 4, yq eval -P is the correct syntax. – Distichous

With mikefarah/yq version 4.15, yq e -P and yq eval -P both works fine. – Becharm

Tangentially, you want to avoid the useless use of cat – Prolamine

Answer 2

function yaml_validate {
  python -c 'import sys, yaml, json; yaml.safe_load(sys.stdin.read())'
}

function yaml2json {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read())))'
}

function yaml2json_pretty {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read()), indent=2, sort_keys=False))'
}

function json_validate {
  python -c 'import sys, yaml, json; json.loads(sys.stdin.read())'
}

function json2yaml {
  python -c 'import sys, yaml, json; print(yaml.dump(json.loads(sys.stdin.read())))'
}

More Bash tricks at http://github.com/frgomes/bash-scripts

Homoeo

Possibly superfluous in light of yq and jq, but very nicely done, and probably has some good applications. – Ardys

Ha! As it happens, I ran into errors installing yq just now in a gpc ai notebook. so, I guess these will come in very handy! – Ardys

perhaps the best answer,as it does not make you install yq – Railey

Answer 3

I'm not sure what rules you're using to get to your expected result. It seems like you're randomly applying different rules to how the values are being converted.

As I understand it, scalar values are just output as is (with potential encoding), objects are output as key/value pairs, and array objects are output with a - for every item. The indentation associates what's part of what.

So based on those rules if you're going to use jq:

def yamlify:
    (objects | to_entries[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify)"
        else
            "\(.key):\t\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;

Then to use it, add it to your .jq file and use it:

$ jq -r yamlify input.json
users:
  - name:       pi
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    groups:     users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth: true
    chpasswd:
        expire: false
  - name:       admin
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    primary-group:      users
    groups:     users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:      None
    plain_text_passwd:  pi
    lock_passwd:        true
    ssh_pwauth: true
    chpasswd:   {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 [email protected]

---

Here's another variation that aligns the values

def yamlify2:
    (objects | to_entries | (map(.key | length) | max + 2) as $w |
        .[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify2)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify2)"
        else
            "\(.key):\(" " * (.key | $w - length))\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify2] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;

$ jq -r yamlify2 input.json
users:
  - name:               pi
    gecos:              Hypriot Pirate
    sudo:               ALL=(ALL) NOPASSWD:ALL
    shell:              /bin/bash
    groups:             users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth:         true
    chpasswd:
        expire:  false
  - name:                 admin
    gecos:                Hypriot Pirate
    sudo:                 ALL=(ALL) NOPASSWD:ALL
    shell:                /bin/bash
    primary-group:        users
    groups:               users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:        None
    plain_text_passwd:    pi
    lock_passwd:          true
    ssh_pwauth:           true
    chpasswd:             {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 [email protected]

Convulsion

really cool. love it, unfortunately, it breaks for multiline string values.. – Newhouse

Based on the code of @Jeff Mercado I added some code to support multiline strings and escaping of single quotes. See my answer "Solution in jq (without other tools)". – Gravamen

Answer 4

yq eval -P

with mikefarah/yq version 4.0 (released December 2020), installable via most Unix-like OS package managers: via Homebrew for macOS (brew install yq), Debian with apt (apt install yq), Alpine with apk (apk add yq), etc.

See Working with JSON.

To read in json, just pass in a json file instead of yaml, it will just work - as json is a subset of yaml. However, you will probably want to use the Style Operator or --prettyPrint/-P flag to make look more like an idiomatic yaml document.

Distichous

Answer 5

I suggest using yq with -y option

$ pip3 install yq # requires jq

$ cat in.json | yq -y
users:
  - name: pi
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    groups: users,docker,video
    plain_text_passwd: pi
    lock_passwd: 'false'
    ssh_pwauth: 'true'
    chpasswd:
      expire: false
  - name: admin
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    primary-group: users
    groups: users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id: None
    plain_text_passwd: pi
    lock_passwd: 'true'
    ssh_pwauth: 'true'
    chpasswd: '{expire: false}'
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 [email protected]

Pattypatulous

Answer 6

I've used ruby to write my json content into yaml.

As for your example, it can be achieved like this:

cat $DEFAULTS_FILE | jq .users | ruby -ryaml -rjson -e 'puts YAML.dump(JSON.parse(STDIN.read))' > my.yml

Agulhas

While I can't stand ruby, my machine already had it installed. I was able to create an alias and pipe it to this alias. Thank you, this has been very helpful! – Literatim

Exact same sentiment here. – Speculate

Answer 7

Another oneliner:

python -c 'import yaml, sys; print(yaml.dump(yaml.load(open(sys.argv[1])), default_flow_style=False))' input.json

(exploiting the fact that valid json is also valid yaml)

And yaml to json:

python -c 'import yaml, json, sys; print(json.dumps(yaml.load(open(sys.argv[1])), indent=2))' input.yaml

Charil

Your yaml-to-json oneliner fails with found character '\t' that cannot start any token; it's the same error message yq reports, which must work similarly. Richard Gomes' one-liner works fine. – Haldas

Answer 8

Solution in jq (without other tools)

Based on the code of @Jeff Mercado in this post I added support for multiline strings and escaping for single quotes.

# purpose: converts Json to Yaml
# remarks:
#   You can use 'yq -y' to convert json to yaml, but ...
#     * this function can be used several times within a single jq program
#     * this function may be faster than using yq
#     * maybe yq is not available in your environment
#
# input: any Json
# output: json converted to yaml
def toYaml:
   def handleMultilineString($level):
      reduce ([match("\n+"; "g")]                       # find groups of '\n'
              | sort_by(-.offset))[] as $match
             (.; .[0:$match.offset + $match.length] +
                 "\n\("    " * $level)" +               # add one extra '\n' for every group of '\n's. Add indention for each new line
                 .[$match.offset + $match.length:]);

   def toYamlString($level):
      if type == "string"
      then handleMultilineString($level)
           | sub("'"; "''"; "g")           # escape single quotes
           | "'\(.)'"                      # wrap in single quotes
      else .
      end;

   def _toYaml($level):
      (objects | to_entries[] |
          if (.value | type) == "array" then
              "\(.key):", (.value | _toYaml($level))
          elif (.value | type) == "object" then
              "\(.key):", "\("    ")\(.value | _toYaml($level))"
          else
              "\(.key): \(.value | toYamlString($level))"
          end
      )
      // (arrays | select(length > 0)[] | [_toYaml($level)] |
          "  - \(.[0])", "\("    ")\(.[1:][])"
      )
      // .;

   _toYaml(1);

Example usage

File 'containsMultilineStrings.json'

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

jq -r 'toYaml' < containsMultilineStrings.json

response:
    code: 200
    message: 'greeting

    that''s all folks



    '

jq -r 'toYaml' containsMultilineStrings.json | yq (roundtrip)

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

Test

You can test the correctness of the function toYaml by converting json to yaml and than back to json using yq.

FILE='containsMultilineStrings.json'; diff <(cat "$FILE") <(jq -r 'toYaml' $FILE | yq)

Performance

A quick benchmark shows a reduced runtime of the function toYaml compared to the use of yq. On my computer, I measured:

time for i in {1..100}; do yq -y > /dev/null < containsMultilineStrings.json; done

8.4 sec

time for i in {1..100}; do jq -r 'toYaml' > /dev/null containsMultilineStrings.json; done

3.4 sec

Gravamen

Answer 9

Another option is to use gojq. It is a port of jq with support for reading and writing yaml. It can be installed via GitHub releases, homebrew, and zero install. The command for your question would be:

cat test.json | gojq --yaml-output > test.yaml

Cerebration

Answer 10

You can use the awesome library yq

# install yq on macOS
brew install yq

Then run the following command

cat file.json | yq -P > file.yml

You can also convert and save a JSON Response into a YAML file, for example:

curl https://raw.githubusercontent.com/typicode/demo/master/db.json | yq -P > example.yml

Demonology