How to escape history expansion exclamation mark ! inside a double quoted string?

Question

EDIT: the command substitution is not necessary for the surprising behavior, although it is the most common use case. The same question applies to just echo "'!b'"

b=a

# Enable history substitution.
# This option is on by default on interactive shells.
set -H

echo '!b'
# Output: '!b'
# OK. Escaped by single quotes.

echo $(echo '!b')
# Output: '!b'
# OK. Escaped by single quotes.

echo "$(echo '$b')"
# Output: '$b'
# OK. Escaped by single quotes.

echo "$(echo '!b')"
# Output: history expands
# BAD!! WHY??

• In the last example, what is the best way to escape the !?
• Why was it not escaped even if I used single quotes, while echo "$(echo '$b')" was? What is the difference between ! and $?
• Why was does echo $(echo '!b') (no quotes) work? (pointed by @MBlanc).

I would prefer to do this without:

• set +H as I would need set -H afterwards to maintain shell state

• backslash escapes, because I need one for every ! and it has to be outside the quotes:

  echo "$(echo '\!a')"
  # '\!a'.
  # No good.
  
  echo "$(echo 'a '\!' b '\!' c')"
  # a ! b ! c
  # Good, but verbose.
  

• echo $(echo '!b') (no quotes), because the command could return spaces.

Version:

bash --version | head -n1
# GNU bash, version 4.2.25(1)-release (i686-pc-linux-gnu)

Sukhum

echo "$(echo \!b)" works for me under Bash-3.2 – Heavenly

It also works for me. I would prefer a quoting solution so I don't have to escape every ! in the string, but it works. Please add your comment to an answer. There is still the why question. – Sukhum

Removing the outermost quotes also prevents substitution. I'd go with set +H myself if I were writing a script. I hope a bash expert sees this and enlightens us. Good luck! :) – Heavenly

True! I'm even more confused now. – Sukhum

It's equivalent to echo "!b"... what did you expect? – Noiseless

@KarolyHorvath I also thought of this reasoning, but why does echo "$(echo -e '! \bb')" or echo "$(echo -e "! \bb")" work? Don't both of these lead to echo "!b"? – Hype

@Hype those work because the space after ! prevents ! expansion. Thats clear in the man pages. – Sukhum

@cirosantilli I was referring to the result of command substitution. – Hype

"Several characters inhibit history expansion if found immediately following the history expansion character, even if it is unquoted: space, tab, newline, carriage return, and =." – Noiseless

@KarolyHorvath but why echo "$(echo '$b')" gives $b and not a? (it expands "only once", while the ! command expands twice) – Sukhum

@Hype Can you provide an example without -e? With -e, \b is backspace, and that makes by brain twist even more. Without the -e the result is different. – Sukhum

@cirosantilli -e enables interpretation of backslash escapes; that would explain different results with and without it. – Hype

@close votes, what is the reason to close? – Sukhum

@cirosantilli, if this is a script, history expansion is off by default, so you don't need to worry about this at all. If this is not a script, the issue belongs on SuperUser, not StackOverflow, so it's off-topic. – Deliciadelicious

@cirosantilli ...by the way, I strongly suggest turning off history expansion for your interactive shells: set -H in your bashrc. It's a feature that hails from a day when the readline library didn't exist yet and command-line editing was far more cumbersome. – Deliciadelicious

@CharlesDuffy I agree. – Sukhum

@CharlesDuffy: Good advice, but you meant set +H to turn history expansion off - set -H turns it on. – Resolvable

Related: Difference between single and double quotes in Bash. – Dais

See also https://mcmap.net/q/12112/-echo-quot-quot-fails-quot-event-not-found-quot – Reconciliatory

Also see Bash Pitfalls #23 (echo "Hello World!"). – Vale

Answer 1

In your last example,

echo "$(echo '!b')"

the exclamation point is not single-quoted. Because history expansion occurs so early in the parsing process, the single quotes are just part of the double-quoted string; the parser hasn't recognized the command substitution yet to establish a new context where the single quotes would be quoting operators.

To fix, you'll have to temporarily turn off history expansion:

set +H
echo "$(echo '!b')"
set -H

Chancery

Thanks for you answer! What are the names of the relevant parsing stages? Where can I get that info on the man page? – Sukhum

This does even happen when a password is read from command line. Thank you! – Satiated

@CiroSantilli Unfortunately, the exact order of operations is scattered throughout the man page. Under HISTORY EXPANSION, it is mentioned that history expansion occurs immediately after the line is read, but before any word-splitting occurs. This implies that !b would be replaced by the appropriate command prior to any other structures in the line being recognized. – Chancery

Great answer; strictly speaking, ! parsing doesn't even respect double-quotes per se, and considers any run of non-whitespace characters following ! the history-expansion argument, including "; for instance, echo foo!" - despite imbalanced double-quotes - is considered a valid command, as is echo foo!bar"baz. – Resolvable

The quoting of ! raises some troubling issues for software that generates text for interpretation by bash. With the exception of that one character, the quoting rules are simple. Protecting against exclamation point injection into a shell with history and history expansion enabled is going to be … an interesting challenge. – Monegasque

This is a bug in bash 4.1, fixed in 4.2 (released five years ago, 2011). – Neary

@Neary What bug are you referring to? echo "$(echo '!b')" still subjects !b to history expansion in 4.3. (It does appear changed in 4.4, though.) – Chancery

@Chancery Hunh. I got here from https://mcmap.net/q/57454/-sed-replace-last-line-matching-pattern and with the command there, sed "$(sed -n /a/= file | sed '$!d;s/$/ s,a,c,/' )" file , history expansion doesn't take place since 4.2 (I just finished running a bisect to find it). I didn't think to check this one. It still does, as you say. – Neary

Might have been fixed in 4.2, reintroduced in 4.3, and fixed again in 4.4 :) – Chancery

actually, that's possible, because my "it still does" was done with the git bisect old shell. on 4.4.5 I get !b. Hang on, I'll find where this was fixed. I wish they'd just used git so I could just git log --grep for it. – Neary

git://git.sv.gnu.org/bash.git – Chancery

Let us continue this discussion in chat. – Neary

This is a bug, fixed in bash 4.4 – Neary

@Neary That's probably worth posting as a new answer, especially if you can provide an explanation for why it was considered a bug and what change was made to protect the history expansion. – Chancery

Is not fixed on GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu) version! – Tattered

@MarceloFilho No, it is not, as discussed in the previous comments and leading to jthill's answer. – Chancery

Answer 2

This was repeatedly reported as a bug, most recently against bash 4.3 in 2014, for behavior going back to bash 3.

There was some discussion whether this constituted a bug or expected but perhaps undesirable behavior; it seems the consensus has been that, however you want to characterize the behavior, it shouldn't be allowed to continue.

It's fixed in bash 4.4, echo "$(echo '!b')" doesn't expand, echo "'!b'" does, which I regard as proper behavior because the single quotes are shell syntax markers in the first example and not in the second.

Neary

Answer 3

If History Expansion is enabled, you can only echo the ! character if it is put in single quotes, escaped or if followed by a whitespace character, carriage return, or =.

From man bash:

   Only backslash (\) and single quotes can  quote  the  history
   expansion character.

   Several  characters inhibit history expansion if found immediately fol-
   lowing the history expansion character, even if it is unquoted:  space,
   tab,  newline,  carriage return, and =.

I believe the key word here is “Only”. The examples provided in the question only consider the outer most quotes being double quotes.

Aster

Thank you John B, but as implied in the question I knew about the single quotes, and as stated in the comments I also knew about the characters that come after it. The main question is: why does the inner single quote does not work? – Sukhum

but then why echo "$(echo '$b')" does not expand (gives $b) while echo "$(echo '!b')" does (does history expansion)? – Sukhum

@cirosantilli I think that happens because Bash treats ! differently than word splitting when it comes to quoting. As the man page specificity of quoting implies, it is much more strict with the ! character. – Aster

Do you mean: "If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed"? That is the only specific thing about quoting and !. But it twists my brain. On the above examples the backslash was removed at some point. To actually understand this I would need a breakdown of the relevant manpage line + step by step of the expansions done. – Sukhum

Answer 4

Sometimes you need to make a small addition to a big command pipe

The OP's "Good, but verbose" example is actually pretty awesome for many cases.

Please forgive the contrived example. The whole reason I need such a solution is that I have a lot of distracting, nested code. But, it boils down to: I must do a !d in sed within a double quoted bash command expansion.

This works

$ ifconfig | sed '/inet/!d'
inet 127.0.0.1 netmask 0xff000000
…

This does not

$ echo "$(ifconfig | sed '/inet/!d')"
-bash: !d': event not found

This is a simplest compromise

$ echo "$(ifconfig | sed '/inet/'\!'d')"
inet 127.0.0.1 netmask 0xff000000
…

Using the compromise allows me to insert a few characters into the existing code and produce a Pull Request that anyone can understand… even though resulting code is more difficult to understand. If I did a complete refactor, the code reviewers would have a much more challenging time verifying it. And of course this bash has no unit tests.

Aphra

Answer 5

Close the double quote, put the ! in single quotes ' immediately after (without a space before or after it) and then open the double quote again:

$ echo "this is "'!'"how you do it"
this is !how you do it

Meliorate