sed command with -i option (in-place editing) works fine on Ubuntu but not Mac [duplicate]

Question

I know nothing about Sed but need this command (which works fine on Ubuntu) to work on a Mac OSX:

sed -i "/ $domain .*#drupalpro/d" /etc/hosts

I'm getting:

sed: 1: "/etc/hosts": extra characters at the end of h command

Hylton

@AnthonySottile: The suggested duplicate is definitely closely related, except that - unlike this question - it specifically asks for a solution that works with both the BSD/macOS and GNU implementations of sed. Thus, people interested in a BSD/macOS-only solution may find more focused answers here. – Belanger

@Belanger yeah the question is slightly different, but the answers on both are nearly identical -- imo not worth having two questions shrugs – Broadside

@AnthonySottile: It was just a suggestion - either way is probably fine - let's let future close-voters decide. – Belanger

May be, BashX project can help to you with this kind of problems. – Tachistoscope

Answer 1

Ubuntu ships with GNU sed, where the suffix for the -i option is optional. OS X ships with BSD sed, where the suffix is mandatory. Try sed -i ''

Homegrown

So just follow the sed -i with single quotes, sed -i '1i export PATH="$HOME/.composer/vendor/bin:$PATH"' $HOME/.bashrc Still does not work for me – Auten

@Auten the point is that you need a separate set of quotes before you start the command, so something like sed -i '' '1i export PATH="$HOME/.composer/vendor/bin:$PATH"' $HOME/.bashrc – Homegrown

If you need the same script to work on both macOS (OS X, BSD) and Linux et al, then you must use a backup suffix (e.g. .bak) and you must attach it to the -i option — sed -i.bak …. If you don't do this, the script will not operate correctly on one of the two platforms. GNU sed does not like the empty argument that the BSD sed requires to edit in place; you can't attach the empty argument to -i because it is empty and indistinguishable from -i. Also, thanks to contradictory designs, the -i option won't be standardized as -i by POSIX; it can't reconcile the irreconcileable. – Charlenecharleroi

Oh, and if you don't want the backup, run rm -f /etc/hosts.bak after running the sed. That's simpler than trying to work out which variant of sed you have installed or other gizmos I've seen suggested. I hope you keep the file under version control (or otherwise backed up) so that you can recover if there's a bug in your sed script. Of course, the .bak file gives you that protection. – Charlenecharleroi

May be, BashX project can help to you with this kind of problems. – Tachistoscope

Answer 2

To complement microtherion's helpful, to-the-point answer:

• with a portable solution
• with background information

tl;dr:

The equivalent of this GNU sed (standard on most Linux distros) command:

sed -i    's/foo/bar/' file

is this BSD/macOS sed command:

sed -i '' 's/foo/bar/' file  # Note the '' as a *separate argument*

With BSD/macOS sed, the following commands do not work at all or not as intended:

sed -i    's/foo/bar/' file  # Breaks; script is misinterpreted as backup-file suffix
sed -i''  's/foo/bar/' file  # Ditto
sed -i -e 's/foo/bar/' file  # -e is misinterpreted as backup-file suffix

For a discussion of all the differences between GNU sed and BSD/macOS sed, see this answer of mine.

Portable approach:

^{Note: Portable here means that the command works with both implementations discussed. It is not portable in a POSIX sense, because the -i option is not POSIX-compliant.}

# Works with both GNU and BSD/macOS Sed, due to a *non-empty* option-argument:
# Create a backup file *temporarily* and remove it on success.
sed -i.bak 's/foo/bar/' file && rm file.bak

For an explanation, see below; for alternative solutions, including a POSIX-compliant one, see this related answer of mine.

---

Background information

In GNU sed (standard on most Linux distros) and BSD/macOS sed, the -i option, which performs in-place updating^[1] of its input files, accepts an option-argument that specifies what suffix (filename extension) to use for the backup file of the file being updated.

E.g., in both implementations, the following keeps the original file, file, as backup file file.bak:

sed -i.bak 's/foo/bar/' file  # Keep original as 'file.bak'; NO SPACE between -i and .bak

Even though with GNU sed the suffix argument is optional, whereas with BSD/macOS sed it is mandatory, the above syntax works with both implementations, because directly adjoining the option-argument (.bak) to the option (-i) - -i.bak, as opposed to -i .bak - works both as an optional and a mandatory option-argument:

• Syntax -i.bak is the only form that works for an optional option-argument.
• Syntax -i.bak also works as a mandatory option-argument, as an alternative to -i .bak, i.e., specifying the option and its argument separately.

Not specifying a suffix - which is often the case - means that no backup file should be kept, and that is where the incompatibility arises:

• With GNU sed, not specifying a suffix means simply using -i by itself.

• With BSD/macOS sed, not specifying a suffix means specifying the empty string as the - mandatory - suffix, and for technical reasons, the empty string can only be passed as a separate argument: that is, -i '' not -i''.

-i'' doesn't work, because to sed it is indistinguishable from just -i, because the shell effectively removes the empty quotes (it concatenates -i and '' and removes quotes with syntactical function), and passes just -i in both cases.

With (effectively) just -i specified, it is the next argument that is interpreted as the option-argument:

sed -i 's/foo/bar/' file # BREAKS with BSD/macOS Sed

's/foo/bar/' - intended to the the Sed script (command) - is now interpreted as the suffix, and the word file is interpreted as the script.
Interpreting such a word as script then leads to obscure error message such as
sed: 1: "file": invalid command code f,
because the f is interpreted as a Sed command (function).

Similarly, with:

sed -i -e 's/foo/bar/' file # CREATES BACKUP FILE 'file-e'

-e is interpreted as the suffix argument, and NOT as Sed's -e option (which can be used to specify multiple commands, if needed).
As as result, instead of keeping NO backup, you get a backup file with suffix -e.

That this command doesn't work as intended is less obvious, because the in-place updating does succeed, given that the syntax requirement of the suffix argument is satisfied by the -e argument.

That the accidental creation of these backup files easily goes unnoticed is the likeliest explanation for Crt's incorrect answer and this incorrect answer to a similar question having received so many up-votes (as of this writing).

---

^{[1] Strictly speaking, a temporary file is created behind the scenes that then replaces the original file; this approach can be problematic: see the bottom half of this answer of mine.}

Belanger

This is the correct and most informative answer to the question. Not sure who downvoted it but no reasons have been expressed for the downvote while I have many reasons to accept this over the other answers. – Fief

I'm not an expert here, but, to keep POSIX compliance, would mv file file.bak && sed 's/foo/bar/' file.bak > file && rm file.bak work? – Alongside

@CássioRenan: In principle, yes, but sed 's/foo/bar' file > file.bak && mv file.bak file (as shown in one linked answer) is simpler. If -i is available and portability is not a concern, its use is preferable, because it makes some effort to preserve the original file's attributes, though it notably still destroys a symlink - see the 2nd half of another linked answer. The POSIX-compliant approach not only also has the symlink problem, but simply creates a new file with the default permissions, ... – Belanger

Answer 3

man is your friend.

OS X

 -i extension
         Edit files in-place, saving backups with the specified extension.
         If a zero-length extension is given, no backup will be saved.  It
         is not recommended to give a zero-length extension when in-place
         editing files, as you risk corruption or partial content in situ-
         ations where disk space is exhausted, etc.

Svetlanasvoboda

Answer 4

In OS X, you can use the GNU version of sed: gsed.

# if using brew
brew install gnu-sed

#if using ports
sudo port install gsed

Then, if your script should be portable, depending on your OS you can define which command to use.

SED=sed
unamestr=`uname`
if [[ "$unamestr" == "Darwin" ]] ; then
    SED=gsed
    type $SED >/dev/null 2>&1 || {
        echo >&2 "$SED it's not installed. Try: brew install gnu-sed" ;
        exit 1;
    }
fi
# here your sed command, e.g.:
$SED -i "/ $domain .*#drupalpro/d" /etc/hosts

Eileneeilis

This is the one that saved me. I upgraded my install and added /usr/local/opt/gnu-sed/libexec/gnubin to my path. I'm on OSX 10.12.6. – Benign

this should be accepted solution – Inspection

Nice handling of dependency on gnu SED, very helpful! – Extensometer