GitHub Actions: How to pass toJSON() result to shell commands

Question

So, I'm working with Github Actions on end-to-end testing. The setup I'm looking at is having one job retrieve a list of urls to be tested, and my second job creates a matrix with that list and tests them all. My problem here is that when I actually run my testing script, it has to be done from the command line, because I'm using Playwright. Therefore I can't use my matrix object directly; I have to output it to a JSON file. The problem is that toJSON creates invalid pretty-printed JSON when I output it to my file, which breaks my script. Here's my code:

name: <name>

on:
    push:
    workflow_dispatch:

jobs:
    fetch_strategic_urls:
        runs-on: ubuntu-latest
        outputs:
            urls: ${{ steps.req-urls.outputs.urls }}
        steps:
            - name: Request Urls
              id: req-urls
              run: |
                  export RESPONSE=$(
                    curl -X GET -H "Accept: application/json" <api-endpoint>)
                  echo "::set-output name=urls::$RESPONSE"
    run_tests:
        runs-on: ubuntu-latest
        strategy:
            matrix:
                url: ${{needs.fetch_strategic_urls.outputs.urls}}
        needs: fetch_strategic_urls
        steps:
            - ...
            - ...
            - run: |
                  ls
                  echo '${{ toJSON(matrix.url) }}' >> props.json
                  cat props.json
                  npm test
              working-directory: E2E.Tests

No matter which configuration of echo ${{matrix.url}} >> props.json I've tried (cat <<'EOF' > props.json ${{matrix.url}}, adding and removing quotes), it always produced JSON files that have no quotes, i.e.: { url: string } instead of {"url": "string"}, which is invalid. This is obviously pretty breaking behavior. I've seen a lot of people online recommending jq, but I don't see how I would use it in this case, since I doubt jq can parse a GitHub-type JSON object, which is necessary for me to use when sharding my jobs. Any help is greatly appreciated!

Redon

Please don't just paste your entire YAML, but create a minimal reproducer – Amoral

Answer 1

It's not easy to put a JSON doc directly in the command line. You can use env vars.

- shell: bash
  env:
    JSON_DOC: ${{ toJSON(some.var) }}
  run: |
    printf '%s\n' "$JSON_DOC" > foo.json
    cat foo.json
    ...

Faustina

This works best if you have a small json. Once JSON gets big, Github fails with Error: An error occurred trying to start process '/usr/bin/bash' with working directory '/runner/_work/repo/repo'. Argument list too long – Indopacific

@ShinebayarG for big json doc you may use awk, python, ... – Faustina

Answer 2

Github actions documenation has a solution here:

name: Context testing
on: push

jobs:
  dump_contexts_to_log:
    runs-on: ubuntu-latest
    steps:
      - name: Dump GitHub context
        id: github_context_step
        run: echo '${{ toJSON(github) }}'

From https://docs.github.com/en/actions/learn-github-actions/contexts#example-printing-context-information-to-the-log

Trull

this does not work if toJSON() result includes ' chars. – Macswan

Answer 3

This isn't an answer, or even an attempt, more a place to post about a related GH actions oddity.

I thought I had the above solved with:

- name: step-name
  run: | # shell
    echo "${{ toJson(github) }}"

which, worked fine, until I merged to main, the default branch, at which point it generates this error:

##[debug]/usr/bin/bash -e /home/runner/work/_temp/11b0a2ad-c8d6-4251-8b4a-6ab18d3de45f.sh
/home/runner/work/_temp/11b0a2ad-c8d6-4251-8b4a-6ab18d3de45f.sh: line 58: syntax error near unexpected token `('
Error: Process completed with exit code 2.

Despite that offending statment ( ${{ toJson(github) }} ) being no where near line 58.

Note that one can't even have this as a comment:

# echo ${{ toJson(github) }}

as that still fails the job, apparently with something in the github object, that is only present in the default branch.

All I can think of is that the returned string from toJSON() isn't always correctly escaped, so it is breaking the YAML parser's ability to extract a legal shell script from the action yaml.

Scup

Answer 4

You can use echo with single quotes:

- name: Print inputs
  run: |
  echo 'Inputs: ${{ toJson(inputs) }}'

Watchcase

Answer 5

I've found that the output of toJSON() is parseable as YAML.

jobs:
  job1:
    runs-on: ubuntu-latest

    outputs:
      my_output: ${{ steps.out1.outputs.my_output }}
      output_2:  ${{ steps.out2.outputs.output_2  }}

    steps:
      - name: output one line
        id: out1
        run: echo 'my_output="Hello, world!"' >> $GITHUB_OUTPUT

      - name: output two line
        id: out2
        run: |
          echo 'output_2="Hello, chicago!"' >> $GITHUB_OUTPUT

     
  job2:
    needs: job1
    runs-on: ubuntu-latest

    steps:
      - name: get all outputs from prior job
        run: |
          echo "The outputs are ${{ toJSON(needs.job1.outputs) }}"
          echo
          jobjson=$(echo "${{ toJSON(needs.job1.outputs) }}" )
          echo $jobjson | yq
          echo
          echo "my_output is $(echo $jobjson | yq .my_output)"

In the console:

| The outputs are {
|   my_output: "Hello, world!",
|   output_2: "Hello, chicago!"
| }
| 
| {my_output: "Hello, world!", output_2: "Hello, chicago!"}
| 
| my_output is Hello, world!

Collectivity

even this solution doesn't work if your JSON has single quotes. What an awesome platform where you don't have absolutely any way of printing an arbitrary json to the console or file – Siberson