Environment variable substitution in sed

Question

If I run these commands from a script:

#my.sh
PWD=bla
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
xxx
bla

it is fine.

But, if I run:

#my.sh
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
$ sed: -e expression #1, char 8: Unknown option to `s' 

I read in tutorials that to substitute environment variables from shell you need to stop, and 'out quote' the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before.

How can I get sed to recognize a $var as an environment variable as it is defined in the shell?

Reformism

$PWD contains a / which is ending the substitute command. – Tonettetoney

@derobert: tnx. One of the solutions addresses this ... – Reformism

Use set -x in the shell to get the shell to echo each command just before it executes them. This can clear up a lot of confusion. (Also, I often use set -u to make de-referencing unset variables a hard error. (See set -e too.)) – Carolynncarolynne

I was hoping to find a way for sed to handle the environment variables as not to leak the values into the process table, seems like sed is the wrong tool for installing secrets according to all the answers in this thread – Komarek

Answer 1

Your two examples look identical, which makes problems hard to diagnose. Potential problems:

1. You may need double quotes, as in sed 's/xxx/'"$PWD"'/'

2. $PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.

To nail both issues at once, perhaps

sed 's@xxx@'"$PWD"'@'

Radborne

but, then what character can i use i know for sure will not appear in a path name ? – Reformism

You can have several candidates like @#%! and check with a case expression to find if $PWD has them. E.g., case "$PWD" of @) ;; *) delim="@" ;; esac; repeat until $delim is not empty. – Radborne

There's another alternative instead of using double quotes. See my answer below. – Salesroom

You can use another delimiter for sed. You dont need to use / you can use , as well if your environment variable is an url. – Sejm

What if the string contains a \ followed by an n - how to stop sed from converting that into a single newline character? – Gilli

Answer 2

In addition to Norman Ramsey's answer, I'd like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).

So if you want to search for 'foo' and replace it with the content of $BAR, you can enclose the sed command in double-quotes.

sed 's/foo/$BAR/g'
sed "s/foo/$BAR/g"

In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.

Curtcurtail

This is cleaner than messing with double quotes, single quotes etc. – Traditional

this is what I had to use to get the environment variable to expand correctly in this command: sed -i "s/127.0.0.1/127.0.0.1 localhost $HOSTNAME/" hosts – Aphis

This is neat but it is not working when you want to do some more complex substitutions, such as "2,$s/^/$foo/" as $s gets interpreted as a variable too and it should not. – Impudicity

This alone didn't work as I still needed a different delimiter for paths like the other answers suggest. – Anderegg

@mjp, what you say is correct but I think one may write it as "2,$ s/^/$foo/" – Calculated

Answer 3

Another easy alternative:

Since $PWD will usually contain a slash /, use | instead of / for the sed statement:

sed -e "s|xxx|$PWD|"

Salesroom

You said "an alternative to using double quotes" and yet your example uses double quotes? – Curtcurtail

I guess the point is it doesn't use double-quotes directly around $PWD...? – Aftersensation

Nice clean pattern but the -e (--expression) is not needed here, since this example is not stringing several expressions together. – Jonell

This has been better using | than / as some of the value in my variable were having special characters like preview:1.2.7-2208.190204 – Alys

Answer 4

You can use other characters besides "/" in substitution:

sed "s#$1#$2#g" -i FILE

Othilia

In my specific case, $2 was a file path, so sed was barfing due to interpreting the / in the contents of $2. This was exactly what I needed to get past it. Thanks fora great tip! – Redvers

Answer 5

一. bad way: change delimiter

sed 's/xxx/'"$PWD"'/'
sed 's:xxx:'"$PWD"':'
sed 's@xxx@'"$PWD"'@'

maybe those not the final answer,

you can not known what character will occur in $PWD, / : OR @.
if delimiter char in $PWD, they will break the expression

the good way is replace(escape) the special character in $PWD.

二. good way: escape delimiter

for example: try to replace URL as $url (has : / in content)

x.com:80/aa/bb/aa.js

in string $tmp

<a href="URL">URL</a>

A. use / as delimiter

escape / as \/ in var (before use in sed expression)

## step 1: try escape
echo ${url//\//\\/}
x.com:80\/aa\/bb\/aa.js   #escape fine

echo ${url//\//\/}
x.com:80/aa/bb/aa.js      #escape not success

echo "${url//\//\/}"
x.com:80\/aa\/bb\/aa.js   #escape fine, notice `"`


## step 2: do sed
echo $tmp | sed "s/URL/${url//\//\\/}/"
<a href="x.com:80/aa/bb/aa.js">URL</a>

echo $tmp | sed "s/URL/${url//\//\/}/"
<a href="x.com:80/aa/bb/aa.js">URL</a>

OR

B. use : as delimiter (more readable than /)

escape : as \: in var (before use in sed expression)

## step 1: try escape
echo ${url//:/\:}
x.com:80/aa/bb/aa.js     #escape not success

echo "${url//:/\:}"
x.com\:80/aa/bb/aa.js    #escape fine, notice `"`


## step 2: do sed
echo $tmp | sed "s:URL:${url//:/\:}:g"
<a href="x.com:80/aa/bb/aa.js">x.com:80/aa/bb/aa.js</a>

Hydrophyte

Answer 6

With your question edit, I see your problem. Let's say the current directory is /home/yourname ... in this case, your command below:

sed 's/xxx/'$PWD'/'

will be expanded to

sed `s/xxx//home/yourname//

which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.

Devilment

but PWD is defined by the shell ... if i go echo $PWD i get the pwd – Reformism

So, how do you escape the environment variables? – Affinity

The selected answer describes a workaround ... don't use slash for the delimiter – Devilment

Which will become a problem as soon as your current working directory contains that other character. – Clothespress

Answer 7

Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution (s) command. So, instead of s/pattern/'$mypath'/ being expanded to s/pattern//my/path/, which will of course confuse the s command, use s!pattern!'$mypath'!, which will be expanded to s!pattern!/my/path!. I’ve used the bang (!) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.

Dahna

Answer 8

VAR=8675309
echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\
sed 's/:[0-9]*:/:'$VAR':/1' 

where VAR contains what you want to replace the field with

Noyade

Answer 9

Dealing with VARIABLES within sed

[root@gislab00207 ldom]# echo domainname: None > /tmp/1.txt

[root@gislab00207 ldom]# cat /tmp/1.txt

domainname: None

[root@gislab00207 ldom]# echo ${DOMAIN_NAME}

dcsw-79-98vm.us.oracle.com

[root@gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: ${DOMAIN_NAME}/g'

 --- Below is the result -- very funny.

domainname: ${DOMAIN_NAME}

 --- You need to single quote your variable like this ... 

[root@gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '${DOMAIN_NAME}'/g'


--- The right result is below 

domainname: dcsw-79-98vm.us.oracle.com

Ocampo

I was struggling to get this working in Azure DevOps YAML pipelines. This comment helped me to successfully use this trick to tokenize some configuration files. – Clercq

Answer 10

If your replacement string may contain other sed control characters, then a two-step substitution (first escaping the replacement string) may be what you want:

PWD='/a\1&b$_' # these are problematic for sed
PWD_ESC=$(printf '%s\n' "$PWD" | sed -e 's/[\/&]/\\&/g')
echo 'xxx' | sed "s/xxx/$PWD_ESC/" # now this works as expected

Churchwoman

Answer 11

I had similar problem, I had a list and I have to build a SQL script based on template (that contained @INPUT@ as element to replace):

for i in LIST 
do
    awk "sub(/\@INPUT\@/,\"${i}\");" template.sql >> output
done

Parkinson

Answer 12

for me to replace some text against the value of an environment variable in a file with sed works only with quota as the following:

sed -i 's/original_value/'"$MY_ENVIRNONMENT_VARIABLE"'/g' myfile.txt

BUT when the value of MY_ENVIRONMENT_VARIABLE contains a URL (ie https://andreas.gr) then the above was not working. THEN use different delimiter:

sed -i "s|original_value|$MY_ENVIRNONMENT_VARIABLE|g" myfile.txt

Wendellwendi