Save modifications in place with awk

Question

I am learning awk and I would like to know if there is an option to write changes to file, similar to sed where I would use -i option to save modifications to a file.

I do understand that I could use redirection to write changes. However is there an option in awk to do that?

Backing

In case anyone wants to have inplace save with NON GNU awk could use following link too #59243604 fyi please. – Riva

Everyone should read backreference.org/2011/01/29/in-place-editing-of-files which addresses the topic of "inplace" editing in general, not just using awk. – Hudgins

Answer 1

In GNU Awk 4.1.0 (released 2013) and later, it has the option of "inplace" file editing:

[...] The "inplace" extension, built using the new facility, can be used to simulate the GNU "sed -i" feature. [...]

Example usage:

$ gawk -i inplace '{ gsub(/foo/, "bar") }; { print }' file1 file2 file3

To keep the backup:

$ gawk -i inplace -v INPLACE_SUFFIX=.bak '{ gsub(/foo/, "bar") }
> { print }' file1 file2 file3

Bloxberg

Looks like the option may have been removed? With 4.1.3, I have "-i includefile --include=includefile" – Insignificance

@Keith I had the same question. I just tried it and it works on my 4.1.3. inplace is actually a library included with gawk according to iiSeymour's answer, so inplace is something that can be included as an includefile. – Poole

An important caveat here: the 'seen' array will fill up with duplicate lines from ALL the files included in the command. So if each file has e.g. a common header, that will be removed in every file after the first one. If you instead want to treat each file independently, you'll need to do something like for f in *.txt; do gawk -i inplace '!seen[$0]++' "$f"; done – Edelstein

Note: Since GNU Awk 5.x, the INPLACE_SUFFIX is now a namespaced variable called inplace::suffix. The extension will fall back to the old variable, but not in versions 5.0.0 and 5.0.1 (5.0.1 ships with Ubuntu 20.04 LTS), only since 5.1.0. But you can simply pass both variables to avoid any problems. – Allix

@IngoKarkat Good review, you're right! Your note was useful for me, thanks! – Halimeda

Answer 2

Unless you have GNU awk 4.1.0 or later...

You won't have such an option as sed's -i option so instead do:

$ awk '{print $0}' file > tmp && mv tmp file

Note: the -i is not magic, it is also creating a temporary file sed just handles it for you.

---

As of GNU awk 4.1.0...

GNU awk added this functionality in version 4.1.0 (released 10/05/2013). It is not as straight forwards as just giving the -i option as described in the released notes:

The new -i option (from xgawk) is used for loading awk library files. This differs from -f in that the first non-option argument is treated as a script.

You need to use the bundled inplace.awk include file to invoke the extension properly like so:

$ cat file
123 abc
456 def
789 hij

$ gawk -i inplace '{print $1}' file

$ cat file
123
456
789

The variable INPLACE_SUFFIX can be used to specify the extension for a backup file:

$ gawk -i inplace -v INPLACE_SUFFIX=.bak '{print $1}' file

$ cat file
123
456
789

$ cat file.bak
123 abc
456 def
789 hij

I am happy this feature has been added but to me, the implementation isn't very awkish as the power comes from the conciseness of the language and -i inplace is 8 characters too long i.m.o.

Here is a link to the manual for the official word.

Macao

Shouldn't your 'first' example be more like: awk '{ gsub(/foo/, "bar" ) } ; { print $0 }' file > tmp.txt && mv -v tmp.txt file ? – Perreault

To my surprise, as of April 2019, still at gawk 4.0.2. Don't let anyone tell you such and such version will be available. – Wireworm

Litte shorter awk '{print $0}' file | sponge file using sponge from moreutils. – Clayborn

Note that mv tmp file is not in general sufficient - the file perms may have changed and must be fixed. -i inplace doesn't have this problem. – Mogul

Answer 3

just a little hack that works

echo "$(awk '{awk code}' file)" > file

Microclimatology

Works like a charm! But is it possible to save awk command into variable and just use it in your nifty trick? – Outhaul

The -i inplace method breaks hardlinks, this hack follows hardlinks♥♥ – Ozell

Answer 4

An alternative is to use sponge:

awk '{print $0}' your_file | sponge your_file

Where you replace '{print $0}' by your awk script and your_file by the name of the file you want to edit in place.

sponge absorbs entirely the input before saving it to the file.

Snoop

How standard/portable is sponge? – Demodulate

sponge is part of moreutils. So it won't be present by default in most systems. But looks like at least sponge itself is portable enough and can be run almost everywhere. – Swine

The downside of this solution compared to tee-based is that sponge will read everything to RAM before writing down, hence it will freeze on large files. – Swine

Answer 5

@sudo_O has the right answer.

This can't work:

someprocess < file > file

The shell performs the redirections before handing control over to someprocess (redirections). The > redirection will truncate the file to zero size (redirecting output). Therefore, by the time someprocess gets launched and wants to read from the file, there is no data for it to read.

Sleety

Answer 6

following won't work

echo $(awk '{awk code}' file) > file

this should work

echo "$(awk '{awk code}' file)" > file

Lir

echo with -n option to avoid trailing newline. – Gabelle

Answer 7

In case you want an awk-only solution without creating a temporary file and usable with version!=(gawk 4.1.0):

awk '{a[b++]=$0} END {for(c=0;c<=b;c++)print a[c]>ARGV[1]}' file

Oosphere

But does this buffer the entire file to memory? Consider a 20GB file. – Macbeth