How to split a command over multiple lines in appveyor.yml

Question

I would like to split a long build command across multiple lines in my appveyor.yml file, however I cannot get it to unwrap, so the build is failing when the first FOR command gets cut off and returns an error. I am not sure how to correctly split the lines in the .yml file so that they are reassembled inside Appveyor. How can this be done?

Here is a simplified version:

build_script:
- cmd: >-
    @echo off
    FOR %%P IN (x86,x64) DO ( ^
      FOR %%C IN (Debug,Release) DO ( ^
        msbuild ^
          /p:Configuration=%%C ^
          /p:Platform=%%P ... ^
        || EXIT 1 ^
      ) ^
    )

I want it to appear in AppVeyor as this:

@echo off
FOR %%P IN (x86,x64) DO ( FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 ) )

Extra spaces are unimportant, the important bit is that the line beginning with FOR until the final ) appears on the same line.

Note that in theory it would also be acceptable for Appveyor to see this:

@echo off
FOR %%P IN (x86,x64) DO ( ^
  FOR %%C IN (Debug,Release) DO ( ^
    msbuild ^
      /p:Configuration=%%C ^
      /p:Platform=%%P ... ^
    || EXIT 1 ^
  ) ^
)

As the Windows cmd.exe interpreter would then see the continuation markers (^) at the end of each line and treat them as one big long command, except that Appveyor does not appear to recognise the ^ marker so it sends each line to cmd.exe one at a time, instead of sending the whole multi-line block together.

This means the first option looks like the only viable solution, where the YAML is constructed such that the FOR line and everything after it is combined into a single line.

I have tried:

• Single spacing with no extra characters at the end of each line. According to this guide, single-spaced YML lines are supposed to be unwrapped into a single line, but this does not happen with Appveyor.
• Double-spaced lines with no extra characters at the end of each line. This is supposed to make each line a separate command, and indeed they are, as the first FOR command fails with error 255 because it is incomplete (only the FOR line is present and not the rest of the loop.)
• Double-spaced lines terminated with ^. Appveyor only runs each line one at a time, so I get an error 255 on the first incomplete FOR command.
• Single-spaced lines terminated with ^ as shown above. Same issue as double-spaced lines, error 255 from an incomplete FOR command.
• Ending each line with && ^ does actually work when running separate commands (e.g. multiple msbuild statements), but this won't work with FOR loops because you can't have && without a command preceding it.

Is there a trick to splitting a single cmd command over multiple lines in appveyor.yml?

Spithead

Answer 1

How to split a command over multiple lines in appveyor.yml?

Here are some some syntax examples for batch, cmd, ps.

I hope these examples save you some time...

Syntax Examples

Batch

# please note the & at EOL in the next example
install:
    # Install VULKAN_SDK 
    - if not exist %VULKAN_SDK% (
       curl -L --silent --show-error --output Vulkan_SDK_Installer.exe https://sdk.lunarg.com/sdk/download/%VULKAN_VERSION%/windows/VulkanSDK-%VULKAN_VERSION%-Installer.exe?Human=true &
       Vulkan_SDK_Installer.exe /S
    )
    - dir %VULKAN_SDK%

before_build:
  - |-
    set MINGW32_ARCH=i686-w64-mingw32
  - if exist %PREFIX% set NEEDDEPENDS=rem

  # Depends
  - |-
    %NEEDDEPENDS% mkdir %PREFIX%\include\SDL2
    %NEEDDEPENDS% mkdir %PREFIX%\lib
    %NEEDDEPENDS% cd %TEMP%
    %NEEDDEPENDS% appveyor DownloadFile https://sourceforge.net/projects/gnuwin32/files/gettext/0.14.4/gettext-0.14.4-lib.zip
    %NEEDDEPENDS% mkdir gettext-0.14.4-lib
    %NEEDDEPENDS% move gettext-0.14.4-lib.zip gettext-0.14.4-lib
    %NEEDDEPENDS% cd gettext-0.14.4-lib
    %NEEDDEPENDS% 7z x gettext-0.14.4-lib.zip > nul
    %NEEDDEPENDS% copy include\* %PREFIX%\include > nul
    %NEEDDEPENDS% copy lib\* %PREFIX%\lib > nul
    %NEEDDEPENDS% cd ..

---

deploy_script:
  # if tagged commit, build/upload wheel
  - IF "%APPVEYOR_REPO_TAG%"=="true" IF NOT "%TESTENV%"=="check" (
      pip install twine &&
      python setup.py register &&
      twine upload -u %PYPI_USER% -p %PYPI_PASS% dist/*
    )

CMD

before_build:
    - cmd: >-     

        mkdir build

        cd .\build

        set OpenBLAS_HOME=%APPVEYOR_BUILD_FOLDER%/%MXNET_OPENBLAS_DIR%

        set OpenCV_DIR=%APPVEYOR_BUILD_FOLDER%/%MXNET_OPENCV_DIR%/build

        cmake .. -DOPENCV_DIR=%OpenCV_DIR% -DUSE_CUDA=0 -DUSE_CUDNN=0 -DUSE_NVRTC=0 -DUSE_OPENCV=1 -DUSE_OPENMP=1 -DUSE_BLAS=open -DUSE_DIST_KVSTORE=0 -G "Visual Studio 12 2013 Win64"

PS

install:
    - ps: >-

        git submodule init

        git submodule update

        if (!(Test-Path ${env:MXNET_OPENBLAS_FILE})) {

            echo "Downloading openblas from ${env:MXNET_OPENBLAS_PKG} ..."

            appveyor DownloadFile "${env:MXNET_OPENBLAS_PKG}" -FileName ${env:MXNET_OPENBLAS_FILE} -Timeout 1200000
        }

---

install:
      - ps: |
          Add-Type -AssemblyName System.IO.Compression.FileSystem
          if (!(Test-Path -Path "C:\maven" )) {
            (new-object System.Net.WebClient).DownloadFile('https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.3.9/apache-maven-3.3.9-bin.zip', 'C:\maven-bin.zip')
            [System.IO.Compression.ZipFile]::ExtractToDirectory("C:\maven-bin.zip", "C:\maven")
          }

---

on_success:
  - ps: |
  if ($true)
  {
    Write-Host "Success"
  }

Jacqulynjactation

Thanks for the samples. Which one of these reassembles multiple .yml lines back into a single-line command inside Appveyor? – Spithead

Careful when using additional parentheses inside an IF condition (), on any command. It messes Batch parsing and gives an error. I solved that by removing the parens. setlocal enabledelayedexpansion might help here too, but I haven't tried it. – Underplay

Answer 2

Use double-quotes:

build_script:
- cmd: "
    @echo off
    FOR %%P IN (x86,x64) DO (
      FOR %%C IN (Debug,Release) DO (
        msbuild
          /p:Configuration=%%C
          /p:Platform=%%P ...
        || EXIT 1
      )
    )"

You can check yaml ref, and more precisely this example.

I was confronted to the same issue using appveyor (what a strange limitation !). Using double-quotes allowed me to leverage YAML subtlety to :

• keep on my side an easy to read/write multi-line expression,
• have a single-line value effectively stored application wise.

You can have a look where I use it and see how the build passes.

For other readers, please note, that the line will be collapsed, and as a consequence, need to be written with a special syntax to support that... you can't omit the || in the expression given by the OP for instance.

Symbolics

Answer 3

CMD commands are always broken into separate lines and run one-by-one by wrapping into .cmd files. Place your code to build.cmd, commit to repo and then call as:

build_script:
- build.cmd

Phocaea

So no way to have the commands on separate lines in appveyor.yml but have them unwrapped so that they are executed on a single line within Appveyor? I would much prefer to keep all the Appveyor-related content in a single file. – Spithead

Nope, sorry. Currently, the only way is putting that code into a batch file. – Phocaea

SO each line is placed into a separate .cmd file? – Lorna

It is possible to execute multi-line PowerShell scripts (using - ps: |), so you convert your CMD script to PS and use that instead. – Lorna

Answer 4

There's another prompt to consider other then batch and PS. msys2/mingw32/mingw64 are installed on appveyor VMs C:\msys64\usr\bin\bash, in addition to the "Git for Windows" version of mingw64.

Unfortunately, getting a multiline command in bash to work in appveyor was less then simple:

Example:

A piece of code like this:

cd /tmp
for server in $(grep '^Server' /etc/pacman.d/mirrorlist.msys | awk '{print $3}' | shuf | arch=x86_64 envsubst); do
  echo Trying ${server}
  curl --connect-timeout 10 -LO ${server}msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz && break
done"

becomes

  - >-
    C:\msys64\usr\bin\bash -lc "
    cd /tmp;
    for server in $(grep $'\x5eServer' /etc/pacman.d/mirrorlist.msys | awk '{print $3}' | shuf | arch=x86_64 /usr/bin/envsubst); do
    :;  echo Trying ${server};
    :;  curl --connect-timeout 10 -LO ${server}msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz && break;
    done"

Why is it so different?

A number of problems happen with a multiline bash string in batch in appveyor:

1. A multiline string in batch has to end in a ^. However doing this

# Literal newlines were not working
- |
  bash -c "echo Not ^
           echo good"

# Double quotes become a literal quote
# Equivalent: echo This is""echo now dangling
#             and "" is an escaped "
- |
  bash -c "echo This is"^
          "echo now dangling"

# Double double quote cancel out, but you're back
# to a literal newline, not working
- |
  bash -c "echo This does not"^
          """echo work either"

# The `|` notation does not seem to be useful here
# It just results in a string that won't run in appveyor 
# as we desired, when using literal newlines.

2. Folding needs to all be on the same indent, so no extra indenting allowed

# > vs >- vs >+ isn't important, it just strips extra newlines at the end
- >
  bash -c "echo This
  echo works"

# Indent means literal newline again
- >
  bash -c "echo Does not
           echo work"

# Says "This echo says", not "This" and "says"
- >-
  bash -c "
  echo This
  echo says"

# Basically: bash -c "echo This; echo works"
- >-
  bash -c "
  echo This;
  echo works"

3. In order to have proper indentation, we need to trick yaml with some dummy characters

# You cannot have ; after keywords then, do, etc...
- >-
  bash -c "
  if [ 1 == 1 ]; then
  ;  echo No;
  ;  echo good;
  fi

# Add in a dummy "true" + semicolon, as a no-op
- >-
  bash -c "
  if [ 1 == 1 ]; then
  :;  echo This works;
  fi

# Or if you prefer this style
- >-
  bash -c "if [ 1 == 1 ]; then
  :;         echo ""This also works"";
  :;       fi

4. The infamous caret. Apparently something expands ^ to ^^, and there was no good way around it

# echoes ^^
bash -c "echo ^"
bash -c "echo \^"
bash -c "echo \\^"

# echoes ^^^^ 
bash -c "echo ^^"

# Uses a hex notation to get around the issue
bash -c "echo $'\x5e'"

5. You have to be careful with some command in mingw64, as they introduce \r\n when in bash you only need a \n

# This is really using /mingw64/bin/envsubst in MINGW64 mode
bash -c "for x in $(echo $'${PWD}\n${OLDPWD}' | envsubst); do
         :;  echo ""${x}"" | xxd;
         done"

# Either need to have MSYSTEM set to MSYS, or
bash -c "for x in $(echo $'${PWD}\n${OLDPWD}' | /usr/bin/envsubst); do
         :;  echo ""${x}"" | xxd;
         done"

# or, as a last resort, use dos2unix
bash -c "for x in $(echo $'${PWD}\n${OLDPWD}' | envsubst | dos2unix); do
         :;  echo ""${x}"" | xxd;
         done"

Vanhook

Answer 5

If you know what appveyor expects (I don't), let's assume:

@echo off
FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
)

Then it is easy to generate the appropriate YAML by dumping it e.g. from Python:

import sys
import ruamel.yaml

appveyor_str = """\
@echo off
FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
)
"""

data = dict(build_script=[dict(cmd=appveyor_str)])

ruamel.yaml.round_trip_dump(data, sys.stdout)

gives you:

build_script:
- cmd: "@echo off\nFOR %%P IN (x86,x64) DO (\n    FOR %%C IN (Debug,Release) DO (\
    \ msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )\n)\n"

(there are no spaces before any of the newlines in the above example)

Using the folded block style scalar (with >) gives you very little control over the folding of the scalar as you experienced. It is also impossible to escape sequences in a folded (or literal) block style scalar.

If your multiline string doesn't need escapes you can try to dump as a block style scalar:

import sys
import ruamel.yaml

appveyor_str = """\
@echo off
FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
)
"""

data = dict(build_script=[dict(cmd=ruamel.yaml.scalarstring.PreservedScalarString(appveyor_str))])

ruamel.yaml.round_trip_dump(data, sys.stdout)

which gives:

build_script:
- cmd: |
    @echo off
    FOR %%P IN (x86,x64) DO (
        FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 )
    )

(i.e. exactly what you put in)

If you right align everything, which is not as readable as you wish, and double newline after the first you get the required output:

import sys
import ruamel.yaml

yaml_str = """\
build_script:
- cmd: >-
    @echo off

    FOR %%P IN (x86,x64) DO (
    FOR %%C IN (Debug,Release) DO (
    msbuild
    /p:Configuration=%%C
    /p:Platform=%%P ...
    || EXIT 1
    )
    )
"""

data = ruamel.yaml.load(yaml_str)

print(data['build_script'][0]['cmd'])

gives:

@echo off
FOR %%P IN (x86,x64) DO ( FOR %%C IN (Debug,Release) DO ( msbuild /p:Configuration=%%C /p:Platform=%%P ... || EXIT 1 ) )

But you cannot indent (from the details of the folded block style scalar):

Lines starting with white space characters (more-indented lines) are not folded.

Fainthearted

Unfortunately your output still mostly crams everything onto the same line, which is what I'm trying to avoid as it's difficult to read. I can put everything onto the one line manually in the .yml file but it's not at all easy to read, so I'm trying to figure out how to have real newlines in the .yml file. Appveyor expects everything on the same line (a newline terminates the command). – Spithead

@Spithead I updated my answer with how to dump your appveyor string as literal block style scalar. If that is not working for you, can you please update your question to include the string as it should go into appveyor (especially wrt newlines and leading/trailing blanks) – Fainthearted

Thanks for the update. I've updated the question to show what I'm after - the issue is that from FOR to the last ) must all appear on the same line after the YAML is read, but in the .yml file I want them on different lines. – Spithead

I don't think you can get that with indenting, that confuses the unfolding. If appveyor strips your ^ you'll have to stick with left aligned code (see my update). – Fainthearted