sed -i command for in-place editing to work with both GNU sed and BSD/OSX

Question

I've got a makefile (developed for gmake on Linux) that I'm attempting to port to MacOS, but it seems like sed doesn't want to cooperate. What I do is use GCC to autogenerate dependency files, and then tweak them a bit using sed. The relevant portion of the makefile:

$(OBJ_DIR)/%.d: $(SRC_DIR)/%.cpp
  $(CPPC) -MM -MD $< -o $@
  sed -i 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $@

While this runs with no trouble under GNU/Linux, I get errors like the following when attempting to build on MacOS:

sed: 1: "test/obj/equipmentConta ...": undefined label 'est/obj/equipmentContainer_utest.d'
sed: 1: "test/obj/dice_utest.d": undefined label 'est/obj/dice_utest.d'
sed: 1: "test/obj/color-string_u ...": undefined label 'est/obj/color-string_utest.d'

It would seem like sed is chopping off a character, but I can't see the solution.

Incoherence

May be, BashX project can help to you with this kind of problems. – Fulford

Answer 1

OS X sed handles the -i argument differently to the Linux version.

You can generate a command that might "work" for both by adding -e in this way:

#      vv
sed -i -e 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $@

OS X sed -i interprets the next thing after the -i as a file extension for a backup copy of the in-place edit. (The Linux version only does this if there is no space between the -i and the extension.) Obviously a side affect of using this is that you will get a backup file with -e as an extension, which you may not want. Please refer to other answers to this question for more details, and cleaner approaches that can be used instead.

The behaviour you see is because OS X sed consumes the s||| as the extension (!) then interprets the next argument as a command - in this case it begins with t, which sed recognizes as a branch-to-label command expecting the target label as an argument - hence the error you see.

If you create a file test you can reproduce the error:

$ sed -i 's|x|y|' test
sed: 1: "test": undefined label 'est'

Awed

I think if POSIXLY_CORRECT or (older) POSIX_ME_HARDER is defined in the environment, the same behavior can be expected. – Calculus

The proper fix for OSX is to have an empty argument to -i but then again this is not compatible with Linux sed. )-: Maybe use Perl instead? perl -pi -e 's|x|y|g' file – Corody

@TimPost: -i is not a POSIX-compliant option, so these environment variables don't apply. – Pirali

Answer 2

Actually, doing

sed -i -e "s/blah/blah/" files

doesn't do what you expect in MacOS either. Instead it creates backup files with -e extension.

The proper command for MacOS is

sed -i "" -e "s/blah/blah/" files

On Linux, remove the space between -i and "" (see related answer)

sed -i"" -e "s/blah/blah/" files

Frondescence

Right but your answer doesn't work for Linux which is the main issue: needs to work for both. Using -i -e it's still possible to delete -e extension files. – Cammycamomile

I suppose for generous definitions of 'working' that is the case. It's a clever hack, but it produces extra files. Urkle's answer is what I was looking for. – Poon

WARNING if you try to find all files with the -e file extension and delete them with something like find . -type f | grep "-e$" | xargs rm you will delete all your files. Need to use "\-e" and even then run it without the pipe to xargs rm first. – Taritariff

@JamesRobinson, just use find . -name "*-e" -delete. Always run without the -delete flag first though just to confirm what you're about to delete. – Ruthieruthless

It appears that the space in -i "" needs to be removed to make it work also on Linux: see https://mcmap.net/q/56948/-sed-command-with-i-option-failing-on-mac-but-works-on-linux – Mcnelly

This solution works on OS X, but not on Linux because of the space after the -i. – Weighting

sed -i"" -e "s/blah/blah/" files commands helps me and it works fine on GitLab-CI – Contrariety

Answer 3

The currently accepted answer is flawed in two very important ways.

1. With BSD sed (the OSX version), the -e option is interpreted as a file extension and therefore creates a backup file with a -e extension.

2. Testing for the darwin kernel as suggested is not a reliable approach to a cross platform solution since GNU or BSD sed could be present on any number of systems.

A much more reliable test would be to simply test for the --version option which is only found in the GNU version of sed.

sed --version >/dev/null 2>&1

Once the correct version of sed is determined, we can then execute the command in its proper syntax.

GNU sed syntax for -i option:

sed -i -- "$@"

BSD sed syntax for -i option:

sed -i "" "$@"

Finally put it all together in a cross platform function to execute an in place edit sed commend:

sedi () {
    sed --version >/dev/null 2>&1 && sed -i -- "$@" || sed -i "" "$@"
}

Example usage:

sedi 's/old/new/g' 'some_file.txt'

This solution has been tested on OSX, Ubuntu, Freebsd, Cygwin, CentOS, Red Hat Enterprise, & Msys.

Ils

If you'd like to combine this with find, see this thread – Anhanhalt

excellent working on mac el capitan, and ubuntu 17 desktop & server – Kilimanjaro

Not working with sed (GNU sed) 4.7 in Ubuntu 20.04.1 docker image. Removing -- works. – Mycology

Answer 4

martin clayton's helpful answer provides a good explanation of the problem^[1], but his solution - as he states - has a potentially unwanted side effect.

Here are side-effect-free solutions:

Caveat: Solving the -i syntax problem alone, as below, may not be enough, because there are many other differences between GNU sed and BSD/macOS sed (for a comprehensive discussion, see this answer of mine).

---

Workaround with -i: Create a backup file temporarily, then clean it up:

With a non-empty suffix (backup-file filename extension) option-argument (a value that is not the empty string), you can use -i in a way that works with both BSD/macOS sed and GNU sed, by directly appending the suffix to the -i option.

This can be utilized to create a backup file temporarily that you can clean up right away:

sed -i.bak 's/foo/bar/' file && rm file.bak

Obviously, if you do want to keep the backup, simply omit the && rm file.bak part.

---

Workaround that is POSIX-compliant, using a temporary file and mv:

If only a single file is to be edited in-place, the -i option can be bypassed to avoid the incompatibility.

If you restrict your sed script and other options to POSIX-compliant features, the following is a fully portable solution (note that -i is not POSIX-compliant).

sed 's/foo/bar' file > /tmp/file.$$ && mv /tmp/file.$$ file

• This command simply writes the modifications to a temporary file and, if the sed command succeeds (&&), replaces the original file with the temporary one.

  • If you do want to keep the original file as a backup, add another mv command that renames the original first.

• Caveat: Fundamentally, this is what -i does too, except that it tries to preserve permissions and extended attributes (macOS) of the original file; however, if the original file is a symlink, both this solution and -i will replace the symlink with a regular file.
  ^{See the bottom half of this answer of mine for details on how -i works.}

---

^{[1] For a more in-depth explanation, see this answer of mine.}

Pirali

The 'workaround' here is by far the best solution (simplest/most-lightweight/most-posixly). This question is about a slightly different problem, but includes good discussion of the same -i issue. – Sublimate

Thanks, @NormanGray. Within the given constraints (1 input file only, no preservation of permissions, creation date, extended attributes), that's true. – Pirali

Workaround 2 is by far the simplest. It's also the safest — the in place edit is subject to failure and you've lost the original. – Fellatio

@JonathanLeffler: I agree that the sed -i.bak ... file && rm file.bak solution is the simplest and best addresses the original question, so I've moved it to now show as the 1st workaround (and I've removed the numbering to avoid confusion). However, with the POSIX-compliant workaround, given the use of && to only replace the original if sed reports success, what problems do you see? I suppose a lack of permissions to replace the original could result in mv failing, which would be unpleasant, but you wouldn't lose the original. – Pirali

Agreed that the POSIX-compliant solution also works well — and works on platforms like Solaris, AIX or HP-UX where the local sed (probably) doesn't support -i at all. So, of the solutions actually using -i, the sed -i.bak …"$file" && rm -f "$file.bak" solution is best, and the solution not using -i at all also has merits. (All the solutions become problematic if the original file is itself a symlink, or if it has multiple links. And permissions/ownership can be a problem when you move/remove files, but that probably applies to the solutions using -i too.) – Fellatio

Answer 5

This isn't quite an answer to the question, but one can get linux-equivalent behavior through

brew install gnu-sed

# Add to .bashrc / .zshrc
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"

(previously there was a --with-default-names option to brew install gnu-sed but that has recently been removed)

Pressurize

This is definitely the quickest solution for me. – Reahard

brew install gnu-sed will install sed as gsed, so you don't need to think about the previous scripts written with OSX's sed. – Scallion

I did try with updated --with-default-names, only noticed it after opening a new term window, and with sed --version, etc. – Jenine

Perhaps hash -r after install will help :) – Pressurize

This isn't a workable solution if it is for the general populace. Programmers may be OK with having to install software to get stuff working, but general users won't. Since the context of the question is programming and makefiles (broken makefiles IMO, but that's a different discussion), this is potentially a solution. – Fellatio

@JonathanLeffler reasonable point, though this is a site for programmers and I doubt non-programmers are going to be dealing with sed – Pressurize

Answer 6

I came across this issue as well and thought of the following solution:

darwin=false;
case "`uname`" in
  Darwin*) darwin=true ;;
esac

if $darwin; then
  sedi="/usr/bin/sed -i ''"
else
  sedi="sed -i"
fi

$sedi 's/foo/bar/' /home/foobar/bar

Works for me ;-), YMMV

I work in a multi-OS team where ppl build on Windows, Linux and OS X. Some OS X users complained because they got another error - they had the GNU port of sed installed so I had to specify the full path.

Tempting

That creates a bar'' backup file for me. But not when writing out the command directly in terminal. Can't really figure out why. – Navar

I had to use eval. eval "$sedi 's/foo/bar/' /home/foobar/bar". This was only tested on os x. – Navar

Take a look at my answer below, I've fixed the problem with the " being generated, better than the eval solution. – Embolism

Answer 7

I've corrected the solution posted by @thecarpy:

Here's a proper cross-platform solution for sed -i:

sedi() {
  case $(uname) in
    Darwin*) sedi=('-i' '') ;;
    *) sedi='-i' ;;
  esac

  LC_ALL=C sed "${sedi[@]}" "$@"
}

Embolism

@masimplo almost the same as sed -i on linux, most things are supported, you'd have to dig into the manuals to see where they differ – Embolism

Promising, but here's no reason to use LC_ALL=C unless you explicitly need to ignore the current locale's character encoding and treat each byte as a character. – Pirali

And while using a bash array to pass dynamically constructed arguments, as you do, is generally more robust, there's nothing actually wrong with @thecarpy's answer (except that it would be better to test what implementation the sed binary is rather than to test the host platform). – Pirali

Answer 8

I avoid using sed -i when writing scripts and i came up with simple solution:

printf '%s' "$(sed 's/foo/bar' file)" > file

much compatible and is POSIX-compliant. It is doing pretty much the same as sed -i, but this one does not create temp files, it directly redirect the changes to file.

As a noob idk what's the cons of doing this, the only matters is "It works"

Elasticize