sed substitution with Bash variables

Question

I am trying to change the values in a text file using sed in a Bash script with the line,

sed 's/draw($prev_number;n_)/draw($number;n_)/g' file.txt > tmp

This will be in a for loop. Why is it not working?

Washedout

Answer 1

Variables inside ' don't get substituted in Bash. To get string substitution (or interpolation, if you're familiar with Perl) you would need to change it to use double quotes " instead of the single quotes:

# Enclose the entire expression in double quotes
$ sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp

# Or, concatenate strings with only variables inside double quotes
# This would restrict expansion to the relevant portion
# and prevent accidental expansion for !, backticks, etc.
$ sed 's/draw('"$prev_number"';n_)/draw('"$number"';n_)/g' file.txt > tmp

# A variable cannot contain arbitrary characters
# See link in the further reading section for details
$ a='foo
bar'
$ echo 'baz' | sed 's/baz/'"$a"'/g'
sed: -e expression #1, char 9: unterminated `s' command

Further Reading:

• Difference between single and double quotes in Bash
• Is it possible to escape regex metacharacters reliably with sed
• Using different delimiters for sed substitute command
• Unless you need it in a different file you can use the -i flag to change the file in place

Atonality

Maybe also mention that if the variable contains slashes, you need to use a different delimiter which doesn't occur in the string (or backslash-escape every occurrence of the delimiter). The slash is a common default for s but you can use any punctuation character; sed "s%draw($prev_number;n_)%draw($number;n_)%g" file.txt or s#from#to#g or s_from_to_g etc. – Tactician

it doesn't work if the variable contains new-lines ("\n") – Sundin

but it did once escaped (for my use cases) also slashes: changelogEscaped=$(sed -e 's%\/%\\/%g; $!a\'$'\n''\\n' <<<"$changelog" | tr -d '\n') Thanks for the link above. – Sundin

Answer 2

Variables within single quotes are not expanded, but within double quotes they are. Use double quotes in this case.

sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp

You could also make it work with eval, but don’t do that!!

Refutative

Thanks that worked. Would you mind explaining what you mean by expanded? My intuition tells me that it should be something like, variable substituted. Also why not use eval? – Washedout

Answer 3

This may help:

sed "s/draw($prev_number;n_)/draw($number;n_)/g" 

Howardhowarth

Answer 4

You can use variables like below. Like here, I wanted to replace hostname i.e., a system variable in the file. I am looking for string look.me and replacing that whole line with look.me=<system_name>

sed -i "s/.*look.me.*/look.me=`hostname`/"

You can also store your system value in another variable and can use that variable for substitution.

host_var=`hostname`

sed -i "s/.*look.me.*/look.me=$host_var/"

---

Input file:

look.me=demonic

Output of file (assuming my system name is prod-cfm-frontend-1-usa-central-1):

look.me=prod-cfm-frontend-1-usa-central-1

Asiaasian

Answer 5

I needed to input github tags from my release within github actions. So that on release it will automatically package up and push code to artifactory.

Here is how I did it. :)

  - name: Invoke build
    run: |
      # Gets the Tag number from the release
      TAGNUMBER=$(echo $GITHUB_REF | cut -d / -f 3)
      
      # Setups a string to be used by sed
      FINDANDREPLACE='s/${GITHUBACTIONSTAG}/'$(echo $TAGNUMBER)/
      
      # Updates the setup.cfg file within version number
      sed -i $FINDANDREPLACE setup.cfg
      
      # Installs prerequisites and pushes 
      pip install -r requirements-dev.txt
      invoke build

Retrospectively I wish I did this in python with tests. However it was fun todo some bash.

Harping

Answer 6

Another variant, using printf:

SED_EXPR="$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)"

sed "${SED_EXPR}" file.txt

or in one line:

sed "$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)" file.txt

Using printf to build the replacement expression should be safe against all kinds of weird things, which is why I like this variant.

Therapeutics