How might I run Pandoc 'convert all files in Dir' command in Github Actions

Question

I would like to setup a github action that runs this command from pandoc FAQ on a repo when its pushed to master. Our objective is to convert all md files in our repo from md to another format using the pandoc docker container.

here is where I got so far. In the first example i do not declare an entrypoint and i get the error "/usr/local/bin/docker-entrypoint.sh: exec: line 11: for: not found."

name: Advanced Usage

on:
  push:
    branches:
      - master

jobs:
  convert_via_pandoc:
    runs-on: ubuntu-18.04
    steps:
      - name: convert md to rtf
        uses: docker://pandoc/latex:2.9
        with:
          args: |
            for f in *.md; do pandoc "$f" -s -o "${f%.md}.rtf"; done

In the second example we declare entrypoint: /bin/sh and the result is error "/bin/sh: can't open 'for': No such file or directory"

name: Advanced Usage

on:
  push:
    branches:
      - master

jobs:
  convert_via_pandoc:
    runs-on: ubuntu-18.04
    steps:
      - name: convert md to rtf
        uses: docker://pandoc/latex:2.9
        with:
          entrypoint: /bin/sh
          args: |
            for f in *.md; do pandoc "$f" -s -o "${f%.md}.rtf"; done

I am a total noob to git actions and not a technical person so my guess is this is easy idea for the SO community. just trying some simple workflow automation. any explicit and beginner feedback is appreciated. thanks - allen

Alcoholize

Worth noting, SO is the only reason I was able to get this far, if in fact I am on the right track at all! – Alcoholize

maybe you need to quote it like "for f in...." and remove the "|" .... so that it becomes one argument? see github.com/pandoc/dockerfiles#github-actions – Dwanadwane

thanks @mb21, I tried that today and same message. my guess is that i am missing something more than just the syntax, or several somethings prob. – Alcoholize

Answer 1

I needed to do a recursive conversion of md files to make a downloadable pack, so this answer extends beyond the OP's goal.

This github action will:

1. Make the output directory (mkdir output)
2. Recurse through the folders, create similarly named folders in an output directory (for d in */; do mkdir output/$d; done)
3. Find all md files recursively (find ./ -iname '*.md' -type f) and execute a pandoc command (-exec sh -c 'pandoc ${0} -o output/${0%.md}.docx' {} \;)

Note that you have to be careful with double and single quote marks when converting from stuff that works in terminal to things that get correctly transformed into a single docker command as part of the github action.

First iteration

jobs:
  convert_via_pandoc:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2
      - name: convert md to docx
        uses: docker://pandoc/latex:2.9
        with:
          entrypoint: /bin/sh
          args: -c "mkdir output;for d in */; do mkdir output/$d; done;find ./ -iname '*.md' -type f -exec sh -c 'pandoc ${0} -o output/${0%.md}.docx' {} \;"
      - uses: actions/upload-artifact@master
        with:
          name: output
          path: output

This solution was developed using @anemyte's info and this SO post on recursive conversion

Second iteration from @caleb

name: Generate Word docs
on: push

jobs:
  convert_via_pandoc:
    runs-on: ubuntu-20.04
    container:
      image: docker://pandoc/latex:2.9
      options: --entrypoint=sh
    steps:
      - uses: actions/checkout@v2
      - name: prepare output directories
        run: |
          for d in */; do
            mkdir -p output/$d
          done
      - name: convert md to docx
        run: |
          find ./ -iname '*.md' -type f -exec sh -c 'pandoc ${0} -o output/${0%.md}.docx' {} \;
      - uses: actions/upload-artifact@master
        with:
          name: output
          path: output

Crinose

This is way better than if edited my answer with your solution. :) – Prepossessing

There is another way to handle this as well setting the top level container to the Pandoc one so that any/all steps with simple run: parameters get run inside the Pandoc docker container. See my comments or your PR here. – Marla

Answer 2

You can make your life easier if you do this with just shell:

name: Advanced Usage

on:
  push:
    branches:
      - master

jobs:
  convert_via_pandoc:
    runs-on: ubuntu-18.04
    steps:
      - name: convert md to rtf
        run: |
          docker run -v $(pwd):/data -w /data pandoc/latex:2.9 sh -c 'for f in *.md; do pandoc "$f" -s -o "${f%.md}.rtf"; done'

The -v key mounts current working directory to /data inside the container. The -w key makes /data a working directory. Everything else you wrote yourself.

The problem you faced is that your args is been interpreted as a sequence of arguments. Docker accepts entrypoint and cmd (args in this case) arguments either as a string or as an array of strings. If it is a string it is parsed to create an array of elements. for became first element of that array and as the first element is an executable it tried to execute for but failed.

Unfortunately, it turned out that the action does not support an array of elements at this moment. Check @steph-locke's answer for a solution with correct args as a string for the action.

Prepossessing

I couldn't get these to work on gh actions for me, but taking the info on the args etc, this did work: gist.github.com/stephlocke/7eafab0669b6568d423599b89eff9d43 – Crinose

Happy for you to incorporate the contents of the gist into your answer so I can award you the bounty :) – Crinose

This gist modifies your args and entrypoints example - at least on my runthrough of getting this to work first, I found that gh didn't like an array of values for these – Crinose

Thanks for the feedback. I removed that part from the answer. – Prepossessing