Replace one substring for another string in shell script

Question

I have "I love Suzi and Marry" and I want to change "Suzi" to "Sara".

firstString="I love Suzi and Marry"
secondString="Sara"

Desired result:

firstString="I love Sara and Marry"

Bundy

For the record, this may have been a fine question once upon a time, but since many years now, Stack Overflow does not encourage "give me code" type of questions. Please don't take this as a good example of how to ask questions here. – Stucker

@Stucker what exactly is the purpose of Stack Overflow if people aren't allowed to ask for code examples? – Layamon

Most questions of this type are too broad, and these days also duplicates. Probably review the help center and in particular How to ask as well as the guidance for providing a minimal reproducible example. Perhaps see also the Stack Overflow homework FAQ which expands on a related topic. – Stucker

@Stucker You mean, those questions tend to be too specific, asking only for a special case applied to their specific situation instead of asking for a general class of equal problems. If stack overflow would be able to group related questions in classes, it would improve usability. It sems, after decades we still need a website that lists for every mental concept or generic operation how to achieve it with any selected programming technology (code examples, patterns or templates exactly). Documentations typically do not provide these mappings and we need to search all over the web. – Rouleau

Answer 1

To replace the first occurrence of a pattern with a given string, use ${parameter/pattern/string}:

#!/bin/bash
firstString="I love Suzi and Marry"
secondString="Sara"
echo "${firstString/Suzi/"$secondString"}"    
# prints 'I love Sara and Marry'

To replace all occurrences, use ${parameter//pattern/string}:

message='The secret code is 12345'
echo "${message//[0-9]/X}"           
# prints 'The secret code is XXXXX'

(This is documented in the Bash Reference Manual, §3.5.3 "Shell Parameter Expansion".)

Note that this feature is not specified by POSIX — it's a Bash extension — so not all Unix shells implement it. For the relevant POSIX documentation, see The Open Group Technical Standard Base Specifications, Issue 7, the Shell & Utilities volume, §2.6.2 "Parameter Expansion".

Bridegroom

@Bridegroom how do I write this statement with a or condition. Just if I want to replace Suzi or Marry with new string. – Diapedesis

@Priyatham51: There's no built-in feature for that. Just replace one, then the other. – Bridegroom

@Bridegroom : Thank you for your reply . I already did that yesterday, and just wanted to check with you if there is an option so that I don't have to write two lines of code. :) – Diapedesis

Will this work for replacing "\n" (newline) with "<br />" (html break)? $STRING="${STRING/\n/<br />}" – Mutton

@Bu: No, because \n in that context would represent itself, not a newline. I don't have Bash handy right now to test, but you should be able to write something like, $STRING="${STRING/$'\n'/<br />}". (Though you probably want STRING// -- replace-all -- instead of just STRING/.) – Bridegroom

And if original string contains "/home/user/dir/" and I want to replace "/home/usr" by "/home" (getting the result "/home/dir/") ? how can I escape the meaning of "/" itself? – Baptista

@DenioMariz: foo=/home/user/dir, followed by echo "${foo/\/home\/user//home}", will print /home/dir. – Bridegroom

To be clear, since this confused me for a bit, the first part has to be a variable reference. You can't do echo ${my string foo/foo/bar}. You'd need input="my string foo"; echo ${input/foo/bar} – Sternway

@Bridegroom Mary is usually spelled with a single r :D – Biopsy

@JobJob: Who cares how it's "usually" spelled? The OP wrote Marry, end of story. – Bridegroom

@Bridegroom Oh shit - so they did! My apologies, should've commented on the OP. To be clear - was just having a bit of a laugh :D – Biopsy

@DenioMariz I have foo = "$(pwd)" and bar="/test/" how can I use now echo "${foo/bar/profi/}" – Pentheam

NB: If the replacement string contains slashes, just escape them. path=/home/me/docs; echo "${path/me\/docs/you\/docs}" – Chicanery

How do I replace X with Y and A with B also. Would it be like ${${I like Suzi and Marry/Suzi/Sara}/like/love}, ${I like Suzi and Marry/Suzi/Sara/like/love}, or something else? – Herthahertz

@facepalm42: That was asked and answered in the first two comments to this answer. – Bridegroom

Where are parameters like the global selector // explained? And which characters do I need to escape in match and replace ? – Pester

@mgutt: This answer links to the relevant documentation. The documentation is not perfect -- for example, instead of mentioning // directly, it mentions what happens "If pattern begins with ‘/’" (which isn't even accurate, since the rest of that sentence assumes that the extra / is not actually part of the pattern) -- but I don't know of any better source. – Bridegroom

Why am I getting error about Bad substitution? I copied the entire code. – Indurate

@Čamo: I'm not a psychic, so unless you post details about what you're doing, I won't be able to tell you what you're doing wrong. – Bridegroom

The reason was that I was not in bash. I called it via sh .... and it should be bash .... – Indurate

@Bridegroom I've done multiple tests on Bash 4.4 and you can't do two replacements in a row. It doesn't work seemingly because it executes the second statement before the first one has stored the result in the variable needed in the second replacement. If there is a way to force the first statement to store its result in the variable before the second one executes please tell me. – Nuremberg

@PHPGuru: You must be doing something interestingly different from what I'm picturing; but I can't guess what. Maybe post a question with the details? (If you do post a question, feel free to comment here linking to it. I'm curious. :-) ) – Bridegroom

my_array="${my_array[@]//*\/tmp\/*/}" my_array="${my_array[@]//*\/cache\/*/}" – Nuremberg

@PHPGuru: OK, I see what you're doing wrong. If you post a question and comment here with a link to it, I'll post an answer (unless someone beats me to it). – Bridegroom

Sure #62855932 – Nuremberg

Can I make the pattern case-insensitive? – Subsume

this solution is giving me bad substitution. echo "${firstString/Suzi/$secondString}" – Kenay

I also get a bad substitution, my command has /, spaces, symbols like %,+ – Le

Answer 2

This can be done entirely with Bash string manipulation:

first="I love Suzy and Mary"
second="Sara"
first=${first/Suzy/$second}

That will replace only the first occurrence; to replace them all, double the first slash:

first="Suzy, Suzy, Suzy"
second="Sara"
first=${first//Suzy/$second}
# first is now "Sara, Sara, Sara"

Radioactivate

It appears that they both answered in the exact same minute :O – Joni

What if first or second contain special characters, like /, $, {, }, <backslash>, ., +, (, ), *, etc.? (Problems with formatting of backslash in this comment.) Perhaps address that in the answer? – Dukes

As to wildcards this worked for me r_getfilter="${r_getfilter/../.*}" to replace .. with .* (a grep "match anything" string). Using '.*' did not go well, the single quotes ended up in the substitution. – Fellmonger

Answer 3

For Dash all previous posts aren't working

The POSIX sh compatible solution is:

result=$(echo "$firstString" | sed "s/Suzi/$secondString/")

This will replace the first occurrence on each line of input. Add a /g flag to replace all occurrences:

result=$(echo "$firstString" | sed "s/Suzi/$secondString/g")

Gillett

I got this: $ echo $(echo $firstString | sed 's/Suzi/$secondString/g') I love $secondString and Marry – Reynolds

@Qstnr_La use double quotes for variable substitution: result=$(echo $firstString | sed "s/Suzi/$secondString/g") – Palace

Plus 1 for showing how to output to a variable as well. Thanks! – Sommerville

I fixed the single quotes and also added the missing quotes around the echo argument. It deceptively works without quoting with simple strings, but easily breaks on any nontrivial input string (irregular spacing, shell metacharacters, etc). – Stucker

In sh (AWS Codebuild / Ubuntu sh) I found that I need a single slash at the end, not a double. I'm going to edit the comment as the comments above also show a single slash. – Oriental

One should keep in mind that search string (in this case Suzi) is a regular expression, so some characters have special meaning (try matching a dot .) – Unfledged

Answer 4

Try this:

sed "s/Suzi/$secondString/g" <<< "$firstString"

The three greater-than signs create a here string.

Backstay

You don't actually need Sed for this; Bash supports this sort of replacement natively. – Bridegroom

I guess this is tagged "bash" but came here because needed something simple for another shell. This is a nice succinct alternative to what wiki.ubuntu.com/… made it look like I'd need. – Partheniaparthenocarpy

This works great for ash/dash or any other POSIX sh. – Phototaxis

I get error sed: -e expression #1, char 9: unknown option to `s – Entresol

@Bridegroom Actually, I found this useful for cases where the haystack string is big. With the other option I was getting zsh: command too long. – Dietetic

First answer that works with stdin, great for pipelining strings. – Quaint

@Bridegroom this answer works with pipes. Like my_command | sed ... | ... – Quaggy

@NamGVU, or anyone else getting that error, it usually means that one of the strings in your search or replace section has the same character as your delimiters. For example, it's common to use / as a delimiter, but if your strings contain *nix directories, then you'll see that error since they also have the / character. – Allowedly

@YoshuaWuyts The <<< "here string" syntax is not POSIX. – Stucker

Answer 5

It's better to use Bash than sed if strings have regular expression characters.

echo ${first_string/Suzi/$second_string}

It's portable to Windows and works with at least as old as Bash 3.1.

To show you don't need to worry much about escaping, let's turn this:

/home/name/foo/bar

Into this:

~/foo/bar

But only if /home/name is in the beginning. We don't need sed!

Given that Bash gives us magic variables $PWD and $HOME, we can:

echo "${PWD/#$HOME/\~}"

---

Thanks for Mark Haferkamp in the comments for the note on quoting/escaping ~.*

Note how the variable $HOME contains slashes, but this didn't break anything.

Further reading: Advanced Bash-Scripting Guide.
If using sed is a must, be sure to escape every character.

Oscillatory

This answer stopped me from using sed with the pwd command to avoid defining a new variable each time my custom $PS1 runs. Does Bash provide a more general way than magic variables to use the output of a command for string replacement? As for your code, I had to escape the ~ to keep Bash from expanding it into $HOME. Also, what does the # in your command do? – Nonresident

@MarkHaferkamp See this from the "further reading recommended" link. About "escaping the ~": notice how I quoted stuff. Remember to always quote stuff! And this doesn't just work for magic variables: any variable is capable of substitutions, getting string length, and more, within bash. Congrats on trying to your $PS1 fast: you may also be interested in $PROMPT_COMMAND if you are more comfortable in another programming language and want to code a compiled prompt. – Oscillatory

The "further reading" link explains the "#". On Bash 4.3.30, echo "${PWD/#$HOME/~}" doesn't replace my $HOME with ~. Replacing ~ with \~ or '~' works. Any of these work on Bash 4.2.53 on another distro. Can you please update your post to quote or escape the ~ for better compatibility? What I meant by my "magic variables" question was: Can I use Bash's variable substitution on, e.g., the output of uname without saving it as a variable first? As for my personal $PROMPT_COMMAND, it's complicated. – Nonresident

@MarkHaferkamp Whoa, you're totally right, my bad. Will update the answer now. – Oscillatory

It seems that I was wrong.... Actually rebooting between Chakra Linux and openSUSE (as opposed to chrooting) shows that ${PWD/#$HOME/~} works in Bash 4.2.53 on SUSE and ${PWD/#$HOME/\~} works in Bash 4.3.30 on Chakra, but not vice versa. A workaround that works on both distros is to replace ~ with $(echo '~'), as in the verbose ${PWD/#$HOME/$(echo '~')}. Edit: I just realized that $'~' might be better. I'm testing it now. – Nonresident

Please ignore the edit in my previous comment. $'~' works great in an echo command, but caused me nothing but frustration when I tried to use it in $PS1 to get a literal ~. $(echo '~') seems to be the best workaround. – Nonresident

@MarkHaferkamp Bash and its obscure pitfalls... :P – Oscillatory

this will break if, for example, your name was "abc" and you were in /home/abcd. should search for $HOME/ but you'll have to escape the slash – Gisele

Answer 6

echo [string] | sed "s|[original]|[target]|g"

• "s" means "substitute"
• "g" means "global, all matching occurrences"

Tellurate

The g flag is hugely misunderstood. Without it, sed will replace the first occurrence on each line but if you don't expect multiple occurrences per line, you don't need g. (Frequently you see it in expressions where there could only ever be a single match per line, like s/.*everything.*/all of it/g where obviously you are matching the entire line in the first place, so there is no way you could match the regex several times). – Stucker

this works perfectly when replacing paths with "/" – Kira

Answer 7

If tomorrow you decide you don't love Marry either she can be replaced as well:

today=$(</tmp/lovers.txt)
tomorrow="${today//Suzi/Sara}"
echo "${tomorrow//Marry/Jesica}" > /tmp/lovers.txt

There must be 50 ways to leave your lover.

Bernini

Answer 8

Since I can't add a comment. @ruaka To make the example more readable write it like this

full_string="I love Suzy and Mary"
search_string="Suzy"
replace_string="Sara"
my_string=${full_string/$search_string/$replace_string}
or
my_string=${full_string/Suzy/Sarah}

Kegler

Till I came to your example I had understood the order the other way round. This helped clarify what is happening – Gesticulative

the replacement order much clear to understand, thanks – Diplosis

Answer 9

Pure POSIX shell method, which unlike Roman Kazanovskyi's sed-based answer needs no external tools, just the shell's own native parameter expansions. Note that long file names are minimized so the code fits better on one line:

f="I love Suzi and Marry"
s=Sara
t=Suzi
[ "${f%$t*}" != "$f" ] && f="${f%$t*}$s${f#*$t}"
echo "$f"

Output:

I love Sara and Marry

---

How it works:

• Remove Smallest Suffix Pattern. "${f%$t*}" returns "I love" if the suffix $t "Suzi*" is in $f "I love Suzi and Marry".

• But if t=Zelda, then "${f%$t*}" deletes nothing, and returns the whole string "I love Suzi and Marry".

• This is used to test if $t is in $f with [ "${f%$t*}" != "$f" ] which will evaluate to true if the $f string contains "Suzi*" and false if not.

• If the test returns true, construct the desired string using Remove Smallest Suffix Pattern ${f%$t*} "I love" and Remove Smallest Prefix Pattern ${f#*$t} "and Marry", with the 2nd string $s "Sara" in between.

Nogood

it's worth noting that this depends on t occurring only once in f but it can be modified with a loop and one longest match to replace multiple occurrences – Stephanistephania

The only thing I'd add after all this time is to better quote the variables to protect their expanded values. [ "${f%"$t"*}" != "$f" ] && f="${f%"$t"*}$s${f#*"$t"}"] – Branchiopod

Answer 10

Using AWK:

firstString="I love Suzi and Marry"
echo "$firstString" | awk '{gsub("Suzi","Sara"); print}'

Roberts

That trick is the way to go if what you're trying to split isn't actually in a variable but in a text file. Thanks, @Payam! – Shabby

Answer 11

Pattern to substitute the first occurrence with special charters:

${parameter/pattern/string}

Pattern to substitute all occurrence with special charters:

${parameter//pattern/string}

firstString="I love //Suzi// and Marry"
secondString="Sara"
firstString="${firstString/\/\/Suzi\/\//"$secondString"}"
echo $firstString

It will print: I love Sara and Marry

Putrefy

Answer 12

I think this is the cleanest form for your use case:

firstString="${firstString//Suzi/$secondString}"

Foundry

Answer 13

Try this:

ls *.ext | awk '{print "mv "$1" "$1".newext"}' | sed "s/.ext.newext/.newext/" | parallel {}

Dormancy

Answer 14

based on proposed above awk solution, I would extend it to use awk-variables. This will allow passing a text containing special chars..

aString="I love _p1_ very much!"
aVar="complicated \" text \' with \. special ) chars"
awk -v p1="$aVar" '{gsub("_p1_",p1); print}' <<< $aString

produces:

I love complicated " text ' with . special ) chars very much

it would be uneasy to implement this case with sed -e or bash substitutions.

Clishmaclaver

Answer 15

As python now builtin available in linux, I would suggest this py string replace str.replace()

firstString="I love Suzi and Marry"
secondString="Sara"

secondString=`python3 -c "s='$firstString'.replace('Suzi', 'Sara'); print(s)" `
echo $secondString

Entresol

Answer 16

The only way I found is store the string in a file, use sed then store the file content in a var :

echo "I love Suzy" > tmp.txt
sed -i "s/Suzy/Sarah/" tmp.txt
set res=`cat tmp.txt`
echo $res
rm tmp.txt

I don't know which kind of shell I am using (only thing I found is sh-4.2 if I type 'sh') but all classic syntax fails, like the simple test=${test2}. It fails 2 times : at the assignment (must use set) and at the ${}.

Mankind

Sounds like you are using csh, but even that supported pipes. sh-4.2 looks like Bash running in POSIX compatibility mode, but that would not require (or understand) the set syntax. There was a time when Csh was quite popular in spite of its obvious flaws, but that was 30+ years ago; probably join us in abandoning it. – Stucker

Answer 17

Using sed we can do it easily

sed -i "s+$value_to_be_replaced+$with_variable1 "some character" $with_variable2+g" $file_name

Lourielouse

Not my downvote, but the broken nested quoting and the lack of explanation (as well as the inadequate formatting, but I fixed that) are all problematic. – Stucker