bash variable expansion ${var:+"..."} in here-document removing double quotes?

Question

I'm trying to understand why Bash removes double quotes (but not single quotes) when doing variable expansion with ${parameter:+word} (Use Alternate Value), in a here-document, for example:

% var=1
% cat <<EOF
> ${var:+"Hi there"}
> ${var:+'Bye'}
> EOF
Hi there
'Bye'

According to the manual, the "word" after :+ is processed with tilde expansion, parameter expansion, command substitution, and arithmetic expansion. None of these should do anything.

What am I missing? How can I get double quotes in the expansion?

Saporific

Seems like a bug or implementation leaking. It's impossible to use a double quote as an alternate value without assigning it to a variable: dq='"' ; ... ${var:+$dq} – Er

@choroba: ${var:+$(echo '"')} But I basically agree. There is something quite strange in the parsing of word in quoted parameter expansions (parameter expansions in here docs are expanded as though quoted, according to the manual, and that seems to be the case.) – Promising

The almost-equivalent of a here document can be seen as an echo "...", and in this case, the " are removed just by typing echo "${var:+"Hi there"}" on the command prompt. They come back when using \" but the HERE document is not consistent and produces the \" on output instead of ". Something is clearly not correct. – Crossarm

Answer 1

tl;dr

$ var=1; cat <<EOF
"${var:+Hi there}"
${var:+$(printf %s '"Hi there"')}
EOF

"Hi there"
"Hi there"

The above demonstrates two pragmatic workarounds to include double quotes in the alternative value.
The embedded $(...) approach is more cumbersome, but more flexible: it allows inclusion of embedded double quotes and also gives you control over whether the value should be expanded or not.

---

Jens' helpful answer and Patryk Obara's helpful answer both shed light on and further demonstrate the problem.

Note that the problematic behavior equally applies to:

• (as noted in the other answers) regular double-quoted strings (e.g., echo "${var:+"Hi there"}"; for the 1st workaround, you'd have to \-quote surrounding " instances; e.g., echo "\"${var:+Hi there}\""; curiously, as Gunstick points out in a comment on the question, using \" in the alternative value to produce " in the output does work in double-quoted strings - e.g., echo "${var:+\"Hi th\"ere\"}" - unlike in unquoted here-docs.)

• related expansions ${var+...}, ${var-...} / ${var:-...}, and ${var=...} / ${var:=...}

• Also, there's a related oddity with respect to \-handling inside double-quoted alternative values inside a double-quoted string / unquoted here-doc: bash and ksh unexpectedly remove embedded \ instances; e.g.,
  echo "${nosuch:-"a\b"}" unexpectedly yields ab, even though echo "a\b" in isolation yields a\b - see this question.

I have no explanation for the behavior^[1] , but I can offer pragmatic solutions that work with all major POSIX-compatible shells (dash, bash, ksh, zsh):

Note that " instances are never needed for syntactic reasons inside the alternative value: The alternative value is implicitly treated like a double-quoted string: no tilde expansion, no word-splitting, and no globbing take place, but parameter expansions, arithmetic expansions and command substitutions are performed.

^{Note that in parameter expansions involving substitution or prefix/suffix-removal, quotes do have syntactic meaning; e.g.: echo "${BASH#*"bin"}" or echo "${BASH#*'bin'}" - curiously, dash doesn't support single quotes, though.}

• If you want to surround the entire alternative value with quotes, and it has no embedded quotes and you want it expanded,
  quote the entire expansion, which bypasses the problem of " removal from the alternative value:

  # Double quotes
  $ var=1; cat <<EOF
  "${var:+The closest * is far from   $HOME}"
  EOF
  "The closest * is far from   /Users/jdoe"
  

  # Single quotes - but note that the alternative value is STILL EXPANDED,
  # because of the overall context of the unquoted here-doc.
  var=1; cat <<EOF
  '${var:+The closest * is far from   $HOME}'
  EOF
  'The closest * is far from   /Users/jdoe'
  

• For embedded quotes, or to prevent expansion of the alternative value,
  use an embedded command substitution (unquoted, although it'll behave as if it were quoted):

  # Expanded value with embedded quotes.
  var=1; cat <<EOF
  ${var:+$(printf %s "We got 3\" of snow at   $HOME")}
  EOF
  We got 3" of snow at   /Users/jdoe
  

  # Literal value with embedded quotes.
  var=1; cat <<EOF
  ${var:+$(printf %s 'We got 3" of snow at   $HOME')}
  EOF
  We got 3" of snow at   $HOME
  

These two approaches can be combined as needed.

---

^[1] In effect, the alternative value:

• behaves like an implicitly double-quoted string,
• ' instances, as in regular double-quoted strings, are treated as literals.

Given the above,

• it would make sense to treat embedded " instances as literals too, and simply pass them through, just like the ' instances.
  Instead, sadly, they are removed, and if you try to escape them as \", the \ is retained too (inside unquoted here-documents, but curiously not inside double-quoted strings), except in ksh - the laudable exception -, where the \ instances are removed. In zsh, curiously, trying to use \" breaks the expansion altogether (as do unbalanced unescaped ones in all shells).

  • More specifically, the double quotes have no syntactic function in the alternative value, but they are parsed as if they did: quote removal is applied, and you can't use (unbalanced) " instances in the interior without \"-escaping them (which, as stated, is useless, because the \ instances are retained).

Given the implicit double-quoted-string semantics, literal $ instances must either be \$-escaped, or a command substitution must be used to embed a single-quoted string ($(printf %s '...')).

Inflict

I think embedded " ARE treated as literals, and then removed away when bash performs value evaluation. But that makes me curious, why prefix/suffix removal does not behave consistently. I think I'll need to look into bash source code to untangle, what is really happening... – Trial

@PatrykObara: That's precisely the inconsistency: the double quotes have no syntactic function in the alternative value, but they are parsed as if they did: quote removal is applied, as you describe, and you can't use " instances in the interior without \"-escaping them (which is useless, because the \ is retained). – Inflict

@PatrykObara: The difference may be unexpected, but in the context of prefix/suffix removal and string replacement quoting does serve a useful function: to distinguish pattern metacharacters (e.g., *) from literals (e.g., "*"), so I wouldn't call that an inconsistency. – Inflict

Answer 2

The behavior looks deliberate--it is consistent across all Bourne shells I tried (e.g. ksh93 and zsh behave the same way).

The behavior is equivalent to treating the here-doc as double-quoted for these special expansions only. In other words, you get the same result for

$ echo "${var:+"hi there"}"
hi there
$ echo "${var:+'Bye'}"
'Bye'

There is only a very faint hint in the POSIX spec I found that something special happens for double quoted words in parameter expansions. This is from the informative "Examples" section of Parameter Expansion:

The double-quoting of patterns is different depending on where the double-quotes are placed.

"${x#*}"
The <asterisk> is a pattern character.
${x#"*"}
The literal <asterisk> is quoted and not special.

I would read the last line as suggesting that quote removal for double quotes applies to the word. This example would not make sense for single quotes, and by omission, there's no quote removal for single quotes.

Update

I tried the FreeBSD /bin/sh, which is derived from an Almquist Shell. This shell outputs single and double quotes. So the behavior is no longer consistent across all shells, only across most shells I tried.

As for getting double quotes in the expansion of the word after :+, my take is

$ var=1
$ q='"'
$ cat <<EOF
${var:+${q}hi there$q}
EOF
"hi there"

Afterguard

Answer 3

$ cat <<EOF
${var:+bare alt value is string already}
${var:+'and these are quotes within string'}
${var:+"these are double quotes within string"}
${var:+"which are removed during substitution"}
"${var:+but you can simply not substitute them away ;)}"
EOF
bare alt value is string already
'and these are quotes within string'
these are double quotes within string
which are removed during substitution
"but you can simply not substitute them away ;)"

Note, that here-document is not needed to reproduce this:

$ echo "${var:+'foo'}"
'foo'

Trial