How to insert strings containing slashes with sed? [duplicate]

Question

I have a Visual Studio project, which is developed locally. Code files have to be deployed to a remote server. The only problem is the URLs they contain, which are hard-coded.

The project contains URLs such as ?page=one. For the link to be valid on the server, it must be /page/one .

I've decided to replace all URLs in my code files with sed before deployment, but I'm stuck on slashes.

I know this is not a pretty solution, but it's simple and would save me a lot of time. The total number of strings I have to replace is fewer than 10. A total number of files which have to be checked is ~30.

An example describing my situation is below:

The command I'm using:

sed -f replace.txt < a.txt > b.txt

replace.txt which contains all the strings:

s/?page=one&/pageone/g
s/?page=two&/pagetwo/g
s/?page=three&/pagethree/g

a.txt:

?page=one&
?page=two&
?page=three&

Content of b.txt after I run my sed command:

pageone
pagetwo
pagethree

What I want b.txt to contain:

/page/one
/page/two
/page/three

Resuscitator

Answer 1

The easiest way would be to use a different delimiter in your search/replace lines, e.g.:

s:?page=one&:pageone:g

You can use any character as a delimiter that's not part of either string. Or, you could escape it with a backslash:

s/\//foo/

Which would replace / with foo. You'd want to use the escaped backslash in cases where you don't know what characters might occur in the replacement strings (if they are shell variables, for example).

Claudieclaudina

> Or, you could escape it with a backslash. An example of that would be more useful, since you don't always know what characters are in a string to be able to choose something different. eg, this: echo / | sed s/\//a/g does not work: sed: -e expression #1, char 5: unknown option to `s' – Honeybunch

Could you add one then? Thanks :) I found surrounding in double quotes seems to work: echo / | sed "s/\//a/g" – Honeybunch

@MaxWaterman it's standard operating procedure when using sed that the regex command is put in double quotes. I didn't use them in my answer because I wasn't showing the whole sed command line but just the sed regex command string as the OP had done. If you put it in a file, as the OP did, you don't need the quotes. – Claudieclaudina

Yeah, fair enough (though perhaps it could be mentioned). That example helps. I have been finding I need to put in lots and lots of backslashes sometimes...and it gets really confusing. eg -e "s/'/\\\\\\\&/g" I think the text is wrong, though: "Which would replace \ with foo" - should be "Which would replace / with foo", no? – Honeybunch

@MaxWaterman thanks for catching that on \ vs. /. Fixed it. If you have a sed command in a shell script, then more backslashes may be necessary (each backslash needs to be backslashed again). – Claudieclaudina

Not that this applies only to subsitute (s) you can use only / when inserting a line before or after a match. You're free to use as much /'s after the second / (in the addition leg): `sed -i '/^.*my.match.pattern.*$/i \ <logger name="com.ing.cea.measurement.MeasurementAPI" level="DEBUG"/>' logback.xml – Gonfanon

@Tom Anderson's solution is far more elegant, which allows your source code to be readable (with slashes). – Carditis

@AbelWenning Tom Anderson's solution is basically the same as mine. He just chose # instead of : as his delimeter as preference and indicated that he preferred #. My point was that you can pick anything you want that's not in the search or replace string. I was not trying to cite : as a particular favored choice. – Claudieclaudina

@Claudieclaudina I should have specified that the problem with a colon (:) is that it is common in things like URLs and JSON, whereas octothorpe (#) is uncommon in syntaxes. – Carditis

@AbelWenning yes, it is, I would agree. The point of my answer, though, wasn't to call out the colon as the best universal alternative. It was just showing that an alternative was allowed by the syntax. The colon just happened to be the example I whimsically chose. I suppose if the OP's examples had "http://" prefix my whim probably would have taken me another direction. :) – Claudieclaudina

Answer 2

The s command can use any character as a delimiter; whatever character comes after the s is used. I was brought up to use a #. Like so:

s#?page=one&#/page/one#g

Hereunto

The man page for the BSD sed on OS X says of the s command: Substitute the replacement string for the first instance of the regular expression in the pattern space. Any character other than backslash or newline can be used instead of a slash to delimit the RE and the replacement. I would bet money that the man page for GNU sed says something similar. – Hereunto

The current accepted answer is basically the same as this one, and was posted a minute earlier! – Hereunto

Answer 3

A very useful but lesser-known fact about sed is that the familiar s/foo/bar/ command can use any punctuation, not only slashes. A common alternative is s@foo@bar@, from which it becomes obvious how to solve your problem.

Outlet

Genius advice when you want to substitute for forward slashes. Thanks! – Pilsudski

Answer 4

add \ before special characters:

s/\?page=one&/page\/one\//g

etc.

Orlon

I may have missed something, but I've tried this and it doesn't seem to work. It did seem the obvious thing to try, but assuming I'm right and it indeed doesn't work, why post it? – Leprose

@Leprose (and anyone else who gets here) -- the 's' at the beginning is required. s/foo\/bar/foo_bar/ will work, but /foo\/bar/foo_bar/ won't. – Clachan

Answer 5

In a system I am developing, the string to be replaced by sed is input text from a user which is stored in a variable and passed to sed.

As noted earlier on this post, if the string contained within the sed command block contains the actual delimiter used by sed - then sed terminates on syntax error. Consider the following example:

This works:

$ VALUE=12345
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345

This breaks:

$ VALUE=12345/6
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
sed: -e expression #1, char 21: unknown option to `s'

Replacing the default delimiter is not a robust solution in my case as I did not want to limit the user from entering specific characters used by sed as the delimiter (e.g. "/").

However, escaping any occurrences of the delimiter in the input string would solve the problem. Consider the below solution of systematically escaping the delimiter character in the input string before having it parsed by sed. Such escaping can be implemented as a replacement using sed itself, this replacement is safe even if the input string contains the delimiter - this is since the input string is not part of the sed command block:

$ VALUE=$(echo ${VALUE} | sed -e "s#/#\\\/#g")
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345/6

I have converted this to a function to be used by various scripts:

escapeForwardSlashes() {

     # Validate parameters
     if [ -z "$1" ]
     then
             echo -e "Error - no parameter specified!"
             return 1
     fi

     # Perform replacement
     echo ${1} | sed -e "s#/#\\\/#g"
     return 0
}

Easel

The take away from your answer for me, was that if the VALUE you're using to replace DEF_VALUE, has forward slashes in it, then you have to escape them with 3 backslashes for sed to work e.g. VALUE="01\\\/01\\\/2018" – Trottier

Answer 6

this line should work for your 3 examples:

sed -r 's#\?(page)=([^&]*)&#/\1/\2#g' a.txt

• I used -r to save some escaping .
• the line should be generic for your one, two three case. you don't have to do the sub 3 times

test with your example (a.txt):

kent$  echo "?page=one&
?page=two&
?page=three&"|sed -r 's#\?(page)=([^&]*)&#/\1/\2#g'
/page/one
/page/two
/page/three

Feudalize

Answer 7

replace.txt should be

s/?page=/\/page\//g
s/&//g

Billi

Answer 8

please see this article http://netjunky.net/sed-replace-path-with-slash-separators/

Just using | instead of /

Perigordian

Answer 9

Great answer from Anonymous. \ solved my problem when I tried to escape quotes in HTML strings.

So if you use sed to return some HTML templates (on a server), use double backslash instead of single:

var htmlTemplate = "<div style=\\"color:green;\\"></div>";

Inhumation

Answer 10

A simplier alternative is using AWK as on this answer:

awk '$0="prefix"$0' file > new_file

Transportation

Answer 11

You may use an alternative regex delimiter as a search pattern by backs lashing it:

sed '\,{some_path},d'

For the s command:

sed 's,{some_path},{other_path},'

Petitionary