sed in-place flag that works both on Mac (BSD) and Linux

Question

Is there an invocation of sed todo in-place editing without backups that works both on Linux and Mac? While the BSD sed shipped with OS X seems to need sed -i '' …, the GNU sed Linux distributions usually come with interprets the quotes as empty input file name (instead of the backup extension), and needs sed -i … instead.

Is there any command line syntax which works with both flavors, so I can use the same script on both systems?

Makassar

Is perl not an option? Must you use sed? – Nichani

Maybe install the GNU version and use that! topbug.net/blog/2013/04/14/…? I'd try that first. Then again, that kind of sucks too. – Undulatory

@dimadima: Might be interesting to some other people browsing this question who have personal scripts that break on their OS X machine. In my case, though, I needed it for the build system of an open source project, where telling your user to install GNU sed first would have defeated the original purpose of this exercise (patch a few files in a "works everywhere" fashion). – Makassar

@klickverbot yeah, makes sense. I first added the comment as an answer, and then deleted it, realizing it wasn't an answer to your question :). – Undulatory

Cross reference: How to achieve portability with sed -i (in-place editing)? on the Unix & Linux Stack Exchange. – Ventilation

In particular, this answer: unix.stackexchange.com/a/92896/43390 does not use -i, and instead writes to a temporary file that it then && mv to the source file. – Ratite

I have a Linux/macOS/BSD solution to this in my answer https://mcmap.net/q/56693/-how-can-i-check-the-version-of-sed-in-os-x – Ambsace

Answer 1

If you really want to just use sed -i the 'easy' way, the following DOES work on both GNU and BSD/Mac sed:

sed -i.bak 's/foo/bar/' filename

Note the lack of space and the dot.

Proof:

# GNU sed
% sed --version | head -1
GNU sed version 4.2.1
% echo 'foo' > file
% sed -i.bak 's/foo/bar/' ./file
% ls
file  file.bak
% cat ./file
bar

# BSD sed
% sed --version 2>&1 | head -1
sed: illegal option -- -
% echo 'foo' > file
% sed -i.bak 's/foo/bar/' ./file
% ls
file  file.bak
% cat ./file
bar

Obviously you could then just delete the .bak files.

Zandrazandt

This is the way to go. Add a few characters, and it's portable. Deleting the backup file is trivial (you already have the filename when invoking sed) – Cytogenetics

This worked on mac, but on ubuntu 16.04 it overwrite the original file to 0 bytes and creates the the .bak file also with 0 bytes ? – Earflap

Of note, on macOS Sierra (and possibly earlier, I don't have a machine handy), sed does not accept a positional command argument when invoked with -i — it must be supplied with another flag, -e. Thus, the command becomes: sed -i.bak -e 's/foo/bar/' filename – Artificial

This creates backups, while the OP specifically asks for in-place editing. – Stalinist

So the full solution is: sed -i.bak 's/foo/bar/' file && rm file.bak – Springhalt

Note that the dot doesn’t really matter. You could say sed -ibak 's/foo/bar/' kinefile and it would create kinefilebak rather than kinefile.bak. As long as you rm $1bak instead of rm $1.bak, it works fine. (Granted, xyz.bak files jump out at you a little bit more than xyzbak files in an ls listing, *.bak “feels” more natural than *bak, and, if the filename is in $f, then you have to say "${f}bak" rather than the simpler "$f.bak".) – Oligopsony

@Marc-AndréLafortune That's true but the sed that comes with macOS doesn't have that option so what do you expect? One has to do it the way given and then move the file. Of course sed -i at the lower level also doesn't do an in place edit. How could it? – Authentic

@Authentic take a look at the answer from dnadlinger below... – Stalinist

Answer 2

This works with GNU sed, but not on OS X:

sed -i -e 's/foo/bar/' target.file
sed -i'' -e 's/foo/bar/' target.file

This works on OS X, but not with GNU sed:

sed -i '' -e 's/foo/bar/' target.file

On OS X you

• can't use sed -i -e since the extension of the backup file would be set to -e
• can't use sed -i'' -e for the same reasons—it needs a space between -i and ''.

Makassar

@AlexanderMills Tells sed that the next argument is a command - could also be skipped here. – Hagbut

Note that -i and -i'' are identical at shell parsing time; sed can’t behave differently on those two first invocations, because it gets the exact same arguments. – Antioch

Answer 3

When on OSX, I always install GNU sed version via Homebrew, to avoid problems in scripts, because most scripts were written for GNU sed versions.

brew install gnu-sed --with-default-names

Then your BSD sed will be replaced by GNU sed.

Alternatively, you can install without default-names, but then:

• Change your PATH as instructed after installing gnu-sed
• Do check in your scripts to chose between gsed or sed depending on your system

Bucksaw

the --with-default-names was removed from homebrew-core, more info in this answer. when installing gnu-sed now, the installation instructions specify that you need to add gnubin to your PATH: PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH" – Stocky

The page for the homebrew package -- formulae.brew.sh/formula/coreutils -- recommends adding this to your bashrc if you want them in your path before Mac tools of the same name: PATH="$(brew --prefix)/opt/coreutils/libexec/gnubin:$PATH" – Fourwheeler

You can also add alias sed=gsed in your RC files 🙂. – Togs

Answer 4

As Noufal Ibrahim asks, why can't you use Perl? Any Mac will have Perl, and there are very few Linux or BSD distributions that don't include some version of Perl in the base system. One of the only environments that might actually lack Perl would be BusyBox (which works like GNU/Linux for -i, except that no backup extension can be specified).

As ismail recommends,

Since perl is available everywhere I just do perl -pi -e s,foo,bar,g target.file

and this seems like a better solution in almost any case than scripts, aliases, or other workarounds to deal with the fundamental incompatibility of sed -i between GNU/Linux and BSD/Mac.

Tutor

I love this solution. perl is part of the Linux Standard Base specification since May 1, 2009. refspecs.linuxfoundation.org/LSB_4.0.0/LSB-Languages/… – Cassidy

This is easily the best solution for portability – Volk

Answer 5

Here's another version that works on Linux and macOS without using eval and without having to delete backup files. It uses Bash arrays for storing the sed parameters, which is cleaner than using eval:

# Default case for Linux sed, just use "-i"
sedi=(-i)
case "$(uname)" in
  # For macOS, use two parameters
  Darwin*) sedi=(-i "")
esac

# Expand the parameters in the actual call to "sed"
sed "${sedi[@]}" -e 's/foo/bar/' target.file

This does not create a backup file, neither a file with appended quotes.

Pacify

this is the correct answer. I would simplify it a bit, but the idea works perfectly sedi=(-i) && [ "$(uname)" == "Darwin" ] && sedi=(-i '') sed "${sedi[@]}" -e 's/foo/bar/' target.file – Gus

Good stuff! I prefer the longer version for readability, though. – Pacify

This a best way for me to be compatible with different system. – Must

This fails if you’ve installed a GNU sed on the system (e.g. using MacPorts or Brew). – Akanke

Answer 6

Answer: No.

The originally accepted answer actually doesn't do what is requested (as noted in the comments). (I found this answer when looking for the reason a file-e was appearing "randomly" in my directories.)

There is apparently no way of getting sed -i to work consistently on both MacOS and Linuces.

My recommendation, for what it is worth, is not to update-in-place with sed (which has complex failure modes), but to generate new files and rename them afterwards. In other words: avoid -i.

Singband

Answer 7

There is no way to have it working.

One way is to use a temporary file like:

TMP_FILE=`mktemp /tmp/config.XXXXXXXXXX`
sed -e "s/abc/def/" some/file > $TMP_FILE
mv $TMP_FILE some/file

This works on both

Caldeira

Is there a reason that SOME_FILE="$(sed -s "s/abc/def/" some/file)"; echo "$SOME_FILE" > some/file; won't work instead – Sheers

Looks like you would be limited by the maximum size of a bash variable, no ? Not sure it will work with GBs files. – Caldeira

If you are creating a temporary file, why not just give an extension to create a backup file (which you can then remove) as @Zandrazandt suggests below? – Tutor

Answer 8

The -i option is not part of POSIX Sed. A more portable method would be to use Vim in Ex mode:

ex -sc '%s/alfa/bravo/|x' file

1. % select all lines

2. s replace

3. x save and close

Randyranee

This is the correct answer. A key feature of POSIX is portability. – Buckjump

Answer 9

Steve Powell's answer is quite correct, consulting the MAN page for sed on OSX and Linux (Ubuntu 12.04) highlights the in-compatibility within 'in-place' sed usage across the two operating systems.

JFYI, there should be no space between the -i and any quotes (which denote an empty file extension) using the Linux version of sed, thus

sed Linux Man Page

#Linux
sed -i"" 

and

sed OSX Man page

#OSX (notice the space after the '-i' argument)
sed -i "" 

I got round this in a script by using an alias'd command and the OS-name output of 'uname' within a bash 'if'. Trying to store OS-dependant command strings in variables was hit and miss when interpreting the quotes. The use of 'shopt -s expand_aliases' is necessary in order to expand/use the aliases defined within your script. shopt's usage is dealt with here.

Checkerberry

Answer 10

Portable script for both GNU systems and OSX:

if [[ $(uname) == "Darwin" ]]; then
    SP=" " # Needed for portability with sed
fi

sed -i${SP}'' -e "s/foo/bar/g" -e "s/ping/pong/g" foobar.txt

Defoe

There's no reason to put quotes around "Darwin" -- it can't parse to anything but itself. Whereas if it weren't for [[ ]]'s special semantics, you would need quotes around $(uname) to ensure correct behavior with all possible return values. – Epifaniaepifano

Answer 11

I ran into this problem. The only quick solution was to replace the sed in mac to the gnu version:

brew install gnu-sed

Jacky

This duplicates an existing answer with much more detail from 2015. – Chaille

Answer 12

If you need to do sed in-place in a bash script, and you do NOT want the in-place to result with .bkp files, and you have a way to detect the os (say, using ostype.sh), -- then the following hack with the bash shell built-in eval should work:

OSTYPE="$(bash ostype.sh)"

cat > myfile.txt <<"EOF"
1111
2222
EOF

if [ "$OSTYPE" == "osx" ]; then
  ISED='-i ""'
else # $OSTYPE == linux64
  ISED='-i""'
fi

eval sed $ISED 's/2222/bbbb/g' myfile.txt
ls 
# GNU and OSX: still only myfile.txt there

cat myfile.txt
# GNU and OSX: both print:
# 1111
# bbbb

# NOTE: 
# if you just use `sed $ISED 's/2222/bbbb/g' myfile.txt` without `eval`,
# then you will get a backup file with quotations in the file name, 
# - that is, `myfile.txt""`

Bummalo

Answer 13

The problem is that sed is a stream editor, therefore in-place editing is a non-POSIX extension and everybody may implement it differently. That means for in-place editing you should use ed for best portability. E.g.

ed -s foobar.txt <<<$',s/foo/bar/g\nw'

Also see https://wiki.bash-hackers.org/howto/edit-ed.

Brunt

Answer 14

You can use sponge. Sponge is an old unix program, found in moreutils package (both in ubuntu and probably debian, and in homebrew in mac).

It will buffer all the content from the pipe, wait until the pipe is close (probably meaning that the input file is already close) and then overwrite:

From the man page:

Synopsis

sed '...' file | grep '...' | sponge file

Michigan

Nice approach – unfortunately doesn't really help in the original situation as the reason for resorting to sed was to have something to use in a cross-platform helper script used on client machines, where you can't depend on anything else but the system tools being installed. Might be helpful to somebody else, though. – Makassar

Answer 15

The following works for me on Linux and OS X:

sed -i' ' <expr> <file>

e.g. for a file f containing aaabbaaba

sed -i' ' 's/b/c/g' f

yields aaaccaaca on both Linux and Mac. Note there is a quoted string containing a space, with no space between the -i and the string. Single or double quotes both work.

On Linux I am using bash version 4.3.11 under Ubuntu 14.04.4 and on the Mac version 3.2.57 under OS X 10.11.4 El Capitan (Darwin 15.4.0).

Hyperaemia

Wow, I'm glad I didn't listen to everyone above saying it was impossible and kept reading! This really just works. Thanks! – Fissile

Doesn't work for me on Mac OS... new file with space appended to the end of the filename is created – Pashto

Doesn't work for me on Mac either (/usr/bin/sed) - I now have an extra backup file with a space appended to the file name :-( – Reentry

Try taking the space out of the single quotes, it works for me. – Yuji