TL;DR
If you just want a solution for Powershell 5, see:
ConvertTo-ArgvQuoteForPoSh.ps: Powershell V5 (and C# Code) to allow escaping native command arguments
The Question I will try to answer
..., it appears PowerShell is stripping double quotes from command
line arguments, even when properly escaped.
PS C:\Documents and Settings\Nick> echo.exe '"hello"'
hello
PS C:\Documents and Settings\Nick> echo.exe '\"hello\"'
"hello"
Notice that the double quotes are there when passed to PowerShell's
echo cmdlet, but when passed as an argument to echo.exe, the double
quotes are stripped unless escaped with a backslash (even though
PowerShell's escape character is a backtick, not a backslash).
This seems like a bug to me. If I am passing the correct escaped
strings to PowerShell, then PowerShell should take care of whatever
escaping may be necessary for however it invokes the command.
What is going on here?
The Non-Powershell Background
The fact that you need to escape the quotes with backslashes \ has nothing to to with powershell, but with the CommandLineToArgvW function that is used by all msvcrt and C# programs to build the argv array from the single-string command line that the Windows process gets passed.
The details are explained at Everyone quotes command line arguments the wrong way and it basically boils down to the fact that this function historically has very uninutitive escaping rules:
- 2n backslashes followed by a quotation mark produce n backslashes followed by begin/end quote. This does not become part of the parsed
argument, but toggles the "in quotes" mode.
- (2n) + 1 backslashes followed by a quotation mark again produce n backslashes followed by a quotation mark literal ("). This does not
toggle the "in quotes" mode.
- n backslashes not followed by a quotation mark simply produce n backslashes.
leading to the described generic escaping function (shortquote of the logic here):
CommandLine.push_back (L'"');
for (auto It = Argument.begin () ; ; ++It) {
unsigned NumberBackslashes = 0;
while (It != Argument.end () && *It == L'\\') {
++It;
++NumberBackslashes;
}
if (It == Argument.end ()) {
// Escape all backslashes, but let the terminating
// double quotation mark we add below be interpreted
// as a metacharacter.
CommandLine.append (NumberBackslashes * 2, L'\\');
break;
} else if (*It == L'"') {
// Escape all backslashes and the following
// double quotation mark.
CommandLine.append (NumberBackslashes * 2 + 1, L'\\');
CommandLine.push_back (*It);
} else {
// Backslashes aren't special here.
CommandLine.append (NumberBackslashes, L'\\');
CommandLine.push_back (*It);
}
}
CommandLine.push_back (L'"');
The Powershell specifics
Now, up to Powershell 5 (including PoSh 5.1.18362.145 on Win10/1909) PoSh knows basically diddly about these rules, nor should it arguably, because these rules are not really general, because any executable you call could, in theory, use some other means to interpret the passed command line.
Which leads us to -
The Powershell Quoting Rules
What PoSh does do however is try to figure out whether the strings you pass it as arguments to the native commands need to be quoted because they contain whitespace.
PoSh - in contrast to cmd.exe - does a lot more parsing on the command you hand it, since it has to resolve variables and knows about multiple arguments.
So, given a command like
$firs = 'whaddyaknow'
$secnd = 'it may have spaces'
$third = 'it may also have "quotes" and other \" weird \\ stuff'
EchoArgs.exe $firs $secnd $third
Powershell has to take a stance on how to create the single string CommandLine for the Win32 CreateProcess (or rather the C# Process.Start) call it will evetually have to do.
The approach Powershell takes is weird and got more complicated in PoSh V7 , and as far as I can follow, it's got to do how powershell treats unbalanced quotes in unquoted string. The long stories short is this:
Powershell will auto-quote (enclose in <">) a single argument
string, if it contains spaces and the spaces don't mix with an
uneven number of (unsescaped) double quotes.
The specific quoting rules of PoSh V5 make it impossible to pass a certain category of string as single argument to a child process.
PoSh V7 fixed this, so that as long as all quotes are \" escaped -- which they need to be anyway to get them through CommandLineToArgvW -- we can pass any aribtrary string from PoSh to a child executable that uses CommandLineToArgvW.
Here's the rules as C# code as extracted from the PoSh github repo for a tool class of ours:
PoSh Quoting Rules V5
public static bool NeedQuotesPoshV5(string arg)
{
// bool needQuotes = false;
int quoteCount = 0;
for (int i = 0; i < arg.Length; i++)
{
if (arg[i] == '"')
{
quoteCount += 1;
}
else if (char.IsWhiteSpace(arg[i]) && (quoteCount % 2 == 0))
{
// needQuotes = true;
return true;
}
}
return false;
}
PoSh Quoting Rules V7
internal static bool NeedQuotesPoshV7(string arg)
{
bool followingBackslash = false;
// bool needQuotes = false;
int quoteCount = 0;
for (int i = 0; i < arg.Length; i++)
{
if (arg[i] == '"' && !followingBackslash)
{
quoteCount += 1;
}
else if (char.IsWhiteSpace(arg[i]) && (quoteCount % 2 == 0))
{
// needQuotes = true;
return true;
}
followingBackslash = arg[i] == '\\';
}
// return needQuotes;
return false;
}
Oh yeah, and they also added in a half baked attempt to correctly escape the and of the quoted string in V7:
if (NeedQuotes(arg))
{
_arguments.Append('"');
// need to escape all trailing backslashes so the native command receives it correctly
// according to http://www.daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULESDOC
_arguments.Append(arg);
for (int i = arg.Length - 1; i >= 0 && arg[i] == '\\'; i--)
{
_arguments.Append('\\');
}
_arguments.Append('"');
The Powershell Situation
Input to EchoArgs | Output V5 (powershell.exe) | Output V7 (pwsh.exe)
===================================================================================
EchoArgs.exe 'abc def' | Arg 0 is <abc def> | Arg 0 is <abc def>
------------------------------|-----------------------------|---------------------------
EchoArgs.exe '\"nospace\"' | Arg 0 is <"nospace"> | Arg 0 is <"nospace">
------------------------------|-----------------------------|---------------------------
EchoArgs.exe '"\"nospace\""' | Arg 0 is <"nospace"> | Arg 0 is <"nospace">
------------------------------|-----------------------------|---------------------------
EchoArgs.exe 'a\"bc def' | Arg 0 is <a"bc> | Arg 0 is <a"bc def>
| Arg 1 is <def> |
------------------------------|-----------------------------|---------------------------
...
I'm snipping further examples here for time reasons. They shouldn't add overmuch to the answer anyways.
The Powershell Solution
To pass arbitrary Strings from Powershell to a native command using CommandLineToArgvW, we have to:
- properly escape all quotes and Backslashes in the source argument
- This means recognizing the special string-end handling for backslashes that V7 has. (This part is not implemented in the code below.)
- and determine whether powershell will auto-quote our escaped string and if it won't auto-quote it, quote it ourselves.
- and make sure that the string we quoted ourselves then doesn't get auto-quoted by powershell: This is what breaks V5.
Powershell V5 Source code for correctly escaping all arguments to any native command
I've put the full code on Gist, as it got too long to include here: ConvertTo-ArgvQuoteForPoSh.ps: Powershell V5 (and C# Code) to allow escaping native command arguments
- Note that this code tries it's best, but for some strings with quotes in the payload and V5 you simply must add in leading space to the arguments you pass. (See code for logic details).
echois an alias forWrite-Output. – Michaelmascmake -G "Ninja" ../... One workaround is to invoke the external commandcmakethrough CMD:"cmake -G"Ninja" ../.. " | cmd(note thatInvoke-Expressionis evil) – Ophir$argList = $argumentList | ForEach-Object { "`"$_`"" } ; & $cmake.Fullname $argList– Schnitzel&will do the correct thing, ie, never quote, just pass an arrays of strings, it does the correct thing always. – Schnitzel