Batch character escaping

Question

I'm fairly proficient at writing Batch scripts for Windows, but even after all these years how to correctly escape characters puzzles me. It's especially difficult when trying figure out the correct way to escape a regular expression for use with sed. Is there any tool that can help me? Perhaps something that allows me to paste in a "normal" string and it spits out the correctly escaped version of that string?

Update: I'm reluctant to give an example because I'm not looking for an answer on how to escape one specific string. I'm also not looking for a solution that will work for one specific app. I'm looking for a tool that will help me get the escape syntax correct for every string I ever need to escape no matter what tool might be consuming it from the command line.

That being said the regex I really want is

(^.*)(Form Product=")([^"]*") FormType="[^"]*" FormID="([0-9][0-9]*)".*$

Take that true regex (i.e. unescaped as far as BATCH is concerned) and wrap it in some sed syntax such as ssed "s@ --- Insert escaped regex here --- @http://psph/\1/\2@g" "%~1" and finally escape it... Again, is there any tool that can assist in escaping any string for use on the BATCH command line?

p.s. There are so many exceptions to BATCH's escaping syntax that I'll even settle for a good cheat sheet.

Barnabe

@Pacerier I have been writing batch scripts in DOS, etc since 1986 (and assembler, C, C++, etc), and then moved to unix, and am now forced for a moment to come back. The question asked by the OP is not an oxymoron; the answer below is proof thereof. Batch was always a dark art. – Candida

To truly understand batch escaping, you must absorb and understand phases 1, 2, 4, 5, 5.3, and 6 of the cmd.exe batch parser, described at https://mcmap.net/q/13959/-how-does-the-windows-command-interpreter-cmd-exe-parse-scripts. And for good measure you should also understand the slight differences that apply to command line mode. And that only explains the parsing of cmd.exe itself, and all of its internal commands. Every external command may have additional escape rules that are totally different. – Worrywart

Closely related: using batch echo with special characters – Shanon

Answer 1

As dbhenham points out in this comment, a (MUCH) more detailed answer can be found in portions of this answer (originally by another user jeb and significantly edited and updated by dbhenham since) on a related but much more general question:

• parsing - How does the Windows Command Interpreter (CMD.EXE) parse scripts? - Stack Overflow

Note that, per dbhenham, this answer is:

incorrect, misleading, and incomplete

I think this answer is still good enough, for almost all cases, but a careful reading of the above answer might be warranted depending on one's exact character escaping needs and the limitations of this answer.

The remaining has been adapted with permission of the author from the page Batch files - Escape Characters on Rob van der Woude's Scripting Pages site.

TLDR

Windows (and DOS) batch file character escaping is complicated:

Much like the universe, if anyone ever does fully come to understand Batch then the language will instantly be replaced by an infinitely weirder and more complex version of itself. This has obviously happened at least once before ;)

Percent Sign %

% can be escaped as %% – "May not always be required [to be escaped] in doublequoted strings, just try"

Generally, Use a Caret ^

These characters "may not always be required [to be escaped] in doublequoted strings, but it won't hurt":

• ^
• &
• <
• >
• |

Example: echo a ^> b to print a > b on screen

' is "required [to be escaped] only in the FOR /F "subject" (i.e. between the parenthesis), unless backq is used"

` is "required [to be escaped] only in the FOR /F "subject" (i.e. between the parenthesis), if backq is used"

These characters are "required [to be escaped] only in the FOR /F "subject" (i.e. between the parenthesis), even in doublequoted strings":

• ,
• ;
• =
• (
• )

Double Escape Exclamation Points when Using Delayed Variable Expansion

! must be escaped ^^! when delayed variable expansion is active.

Double Double-Quotes in find Search Patterns

" → ""

Use a Backslash in findstr Regex Patterns

• \
• [
• ]
• "
• .
• *
• ?

Also

Rob commented further on this question (via email correspondence with myself):

As for the answer, I'm afraid the chaos is even worse than the original poster realizes: requirements for escaping parentheses also depend on the string being inside a code block or not!

I guess an automated tool could just insert a caret before every character, then doubling all percent signs - and it would still fail if the string is doublequoted!

Further, individual programs are responsible for parsing their command line arguments so some of the escaping required for, e.g. for sed or ssed, may be due to the specific programs called in the batch scripts.

Lowbred

I have always found Rob Vanderwoude's web site to be an excellent reference for learning BATCH and have read many sections of it, but somehow I never read that section. Thanks for pointing it out. – Barnabe

Rob's explanation beats ss64.com hands down. – Mhd

What about spaces (say, in file paths that are used as parameters for calling a batch script (from another batch script))? – Dualpurpose

@PeterMortensen You don't have to escape spaces but you do have to quote strings that contain them (as otherwise they're interpreted as separate strings). – Lowbred

That is problematic if it is folder paths (say, partial folder paths) that are going to be combined with other items in the called script - the quotes are not stripped. I ended up using environment variables instead to pass the necessary information. That is another reason for using PowerShell instead, but there is the learning curve (the investment to be made). – Dualpurpose

I will post a question about this if I can't find a duplicate (but surely it must exist 7 years after Stack Overflow was lauched?). – Dualpurpose

@PeterMortensen – This answer seems to cover what you're trying to do. – Lowbred

Though this is a good start, there are lots of incorrect, misleading, and incomplete statements in this answer (and Rob Van der Woud's site). For a true understanding, see https://mcmap.net/q/13959/-how-does-the-windows-command-interpreter-cmd-exe-parse-scripts. – Worrywart

@Worrywart That's a great answer. Obviously Rob triggered the 'instance replacement' property. Hopefully that other answer will remain incomplete enough that it avoids also becoming "incorrect, misleading, and incomplete" like this one has. – Lowbred

Answer 2

The escape character for batch is the caret (^). If you want to include any of the pipeline characters in your script you need to prefix the character with the caret:

:: Won't work:
@echo Syntax: MyCommand > [file]

:: Will work:
@echo Syntax: MyCommand ^> [file]

Rancidity

Yes, the caret is my goto escape character, but it's not the only escape character in BATCH. Sometimes the escape character is backslash, sometimes it is percent, other times it is double-double quote. Not confused yet? There's more. Sometimes the escape character(s) is double percent. Once in a while it's a combination of caret and percent. See why I'm looking for a helper tool? – Barnabe

Can you please give some examples of where you use something other than the caret to escape characters in Windows BATCH? – Rancidity

Here's one Patrick... in this line I use both ^ and '%%' echo if %%errorlevel%% neq 0 echo Problem moving [filename].txt ^>^> Log.txt >> some.bat There I'm writing error checking and reporting into one batch file from another. The ^ in front of %errorlevel% won't work, you need the %%errorlevel%% for it to come out correctly in the new batch. – Robbyrobbyn

@PatrickCuff, What other "pipeline characters" are there besides >, <, &, and |? – Mhd

Answer 3

You could simply use an external file as input for sed.

Or using strings directly in batch, it's a good idea to use the delayed expansion.

setlocal DisableDelayedExpansion
set "regEx=s/^#*$/""/g"
setlocal EnableDelayedExpansion
sed !regEx! file.txt

EDIT: How to use unmodified strings with a batch

This uses findstr to get the string directly from the batch and return it into a result-variable.
So you can use the sed-string as is.

@echo off
setlocal
REM SedString1#(^.*)(Form Product=")([^"]*") FormType="[^"]*" FormID="([0-9][0-9]*)".*$

call :GetSEDString result SedString1
setLocal EnableDelayedExpansion
echo the sedString is !result!
sed !result!
goto :eof

:GetSEDString <resultVar> <searchName>
:: Search the own batch file for <searchName> in a line with "REM <searchName>#"
:: Return all after the "#" without any modification
setLocal DisableDelayedExpansion
for /f "usebackq tokens=* delims=" %%G in (`findstr /n /c:"REM %~2#" "%~f0"`) do (
    set "str=%%G"
)
setLocal EnableDelayedExpansion
set "str=!str:*#=!"

for /F "delims=" %%A in ("!str!") DO (
  endlocal
  endlocal
  set "%~1=%%A"
  goto :eof
)

goto :eof

Hamforrd

Answer 4

A simple solution to preserve all command line arguments is to use %*: it returns the whole command line starting at the first command line argument (in Windows NT 4, %* also includes all leading spaces) and excluding any output redirection.

For example, given this test.bat:

@echo off
echo Parameters are [%*] end params

if you run: test.bat qwe rt ":' *" ? (=)

you get: Parameters are [qwe rt ":' *" ? (=)] end params

Daimyo

I noticed that, after over 4 years, this answer still has zero upvotes (as of this writing). Can you provide an example to explain exactly what you mean? – Toddy