Add a newline only if it doesn't exist

Question

I want to add a newline at the end of a file only if it doesn't exist. This is to prevent multiple newlines at the end of the file.

I'm hoping to use sed. Here are the issues I'm having with my current code:

sed -i -e '/^$/d;$G' /inputfile

echo file1
name1
name2

echo file2
name3
name4
(newline)

when I run my code on to the files;

echo file1
name1
name2
(newline)

echo file2
name3
name4

it adds a newline if it doesn't have one but removes it if it exists... this puzzles me.

Bergius

this worked for me: unix.se/31955/11438 – Clamorous

This post was merged with How to fix “No newline at end of file” warning for lots of files?. But for working with A lot of files, the command tee -a should be used. See How to add newline character at the end of many files – Mime

Answer 1

Since it removes newline if it's not there, you could simply use:

echo "" >> file;  sed -ie '/^$/d;$G' file; sed -ie '/^$/d;$G' file

Adds a newline and removes everything then adds newline. Not the elegant way, but certainly works :)

Rident

echo "" >> file; adds a newline; sed -ie '/^$/d;$G' file; removes a newline? and apply the same code again? EDIT: it certainly works, but i think there must be a neater way.. :P – Bergius

Answer 2

sed

GNU:

sed -i '$a\' *.txt

OS X:

sed -i '' '$a\' *.txt

$ addresses the last line. a\ is the append function.

OS X's sed

sed -i '' -n p *.txt

-n disables printing and p prints the pattern space. p adds a missing newline in OS X's sed but not in GNU sed, so this doesn't work with GNU sed.

awk

awk 1

1 (the number one) can be replaced with anything that evaluates to true. Modifying a file in place:

{ rm file;awk 1 >file; }<file

bash

[[ $(tail -c1 file) && -f file ]]&&echo ''>>file

Trailing newlines are removed from the result of the command substitution, so $(tail -c1 file) is empty only if file ends with a linefeed or is empty. -f file is false if file is empty. [[ $x ]] is equivalent to [[ -n $x ]] in bash.

Preterhuman

points for neatness! the $\a formulation didn't work (on a mac), but the -n p one did... how does it work though? – Derangement

hm, actually, trying them out on ubuntu bash, both return an error sed: can't read p: No such file or directory – Derangement

I edited the answer. It was supposed to be sed '$a\'. Apparently -i '' doesn't work with GNU sed. – Preterhuman

sed -i '$a\' <files> worked perfectly for me on Ubuntu. Thanks! – Dusty

The GNU solution works just fine on OS X El Capitan (10.11). – Buxtehude

It's possible to use find to apply this to all files in a directory (and subdirectories). Ex : find . -type f -print0 | xargs -0 sed -i '' -e '$a\' – Pluvial

The bash answer is very helpful to me. however one issue I run into is that if I place it in a bash script, it produces a non-zero return code if the blank line exists which is throwing errors if used in ansible. – Six

for Bash to avoid failure on non-zero returns (for example, by set -e), do: [[ ! -f file || -z $(tail -c1 file) ]] || echo >>file (If file doesn't exist, then don't add a leading newline OR if the last char of file is a newline, then don't add a newline; otherwise, add a newline) – Waits

the globbing doesn't work for me on os x, i can only do it for each file individually – Rocca

neither sed -i '' '$a\' nor sed -i '' -n p are working for me on os x 13.4 stock sed. – Neisse

Answer 3

Rather than processing the whole file with sed just to add a newline at the end, just check the last character and if it's not a newline, append one. Testing for newline is slightly interesting, since the shell will generally trim them from the end of strings, so I append "x" to protect it:

if [ "$(tail -c1 "$inputfile"; echo x)" != $'\nx' ]; then
    echo "" >>"$inputfile"
fi

Note that this will append newline to empty files, which might not be what you want. If you want to leave empty files alone, add another test:

if [ -s "$inputfile" ] && [ "$(tail -c1 "$inputfile"; echo x)" != $'\nx' ]; then
    echo "" >>"$inputfile"
fi

Carlsen

nice and clean in that it only requires tail and echo, rather than sed or awk. – Indogermanic

Answer 4

Converted Norman's answer to a split one-liner for convenience.

for i in * ; do  echo $i; \
 if diff /dev/null "$i" | tail -1 | \
  grep '^\\ No newline' > /dev/null; then echo >> "$i"; \
 fi; done

Replace * with whatever file pattern you want, eg *.c

And another to just tell you which files are broken:

for i in * ; do \
 if diff /dev/null "$i" | tail -1 | \
  grep '^\\ No newline' > /dev/null; then  echo $i; \
 fi; done

Eadie

If you want it to go recursively, you can swap * with $(find . -type f) or $(find <dirname> -type f -name <filepattern>) – Sybilla

Answer 5

A simple fix for files that are "missing" newline at end of file is simply sed; the following fixes the file "in-place" (using the "-i" option):

find . -type f -exec sed -i -e '$a\' {} \; -print 

Explanation:

• find all files (-type f),
• run sed,
• change the files in-place (-i),
• given the following (-e) script/expression, which matches the end of the file ($),
  • and perform the "append" action (a\),
  • but don't actually specify any text to append (nothing after the \) which is going to add a newline to the end of the file, but only if it's missing.
• Prints all files found (fixed or not), which is probably unnecessary.

Main caveat is that sed features vary across platforms, so -i and -e may or may not be supported / the same; e.g. older Unix, or MacOS oddities may require slightly different syntax.

To only operate on filename(s) matching specific suffix(es), just add find path/to/dir -type f \( -name \*.C -o -name \*.h -o -name \*.java \) -exec ...

Redeem

Warning: Do not do this at the root of a Git repo without adding a -prune or -not -name to the find command to omit the .git/ directory. Otherwise, you'll corrupt it. – Algebra

True, and generally speaking, don't run sed on any binary file. Since .git is a dot file that is (typically) only found at the root dir, a simple solution is to change find . -type f... to find * -type f .... But in any case, if there's any chance of binary files, either tailor the find as necessary, or don't use find at all. – Redeem

Answer 6

If you have access to Unix tools, you can run diff to find out which files lack a newline and then append it:

#!/bin/sh
for i
do
  if diff /dev/null "$i" | tail -1 | grep '^\\ No newline' > /dev/null
  then 
    echo >> "$i"
  fi
done

I'm relying on diff to produce the message with a \ in the first column, tail to give me the last line of diff's output, and grep to tell me if the last line is the message I'm looking for. If all that works, then the echo produces a newline and the >> appends it to the file "$i". The quotes around "$i" make sure things still work if the filename has spaces in it.

Pahl

Not bad, but grep returns a localized message, like "\Brak znaku nowej linii (etc.)". Besides, the diff outputs the whole file. I'd use tail -1 $f | grep '\n' for the condition (works on my box). – Powell

@TomaszGandor : 'tail -1 filename | grep '\n' seems to always return a false result on my mac regardless of whether there is a trailing newline or not.. – Stuck

Answer 7

OK, after complaining in the comments, there is my better solution. First, you want to know, which files are missing newlines:

find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -v 0a$" ';' -print

Not super fast (calling a couple of processes for each file), but it's OK for practical use.

Now, when you have it, you may as well add the newline, with another -exec:

find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -v 0a$" ';' -exec sh -c "echo >> {}" ';'

Possible gotchas:

• if filenames are bad, e.g. they have spaces, you may need tail -1 \"{}\". Or does find do it right?

• you may want to add more filtering to find, like -name \*py, or the like.

• think about possible DOS/Unix newlines mess before use (fix that first).

EDIT:

If you don't like the output from these commands (echoing some hex), add -q to grep:

find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -q -v 0a$" ';' -print
find -type f -exec sh -c "tail -1 {} | xxd -p | tail -1 | grep -q -v 0a$" ';' -exec sh -c "echo >> {}" ';'

Powell

This is huge overkill. – Bilateral

Answer 8

tail -c1 file | read -r _ || echo >> file

gets the last character of the file pipes it into read, which will exit with a nonzero exit code if it encounters EOF before newline (so, if the last character of the file isn't a newline). If read exits nonzero, then append a newline onto the file using echo (if read exits 0, that satisfies the ||, so the echo command isn't run).

From http://backreference.org/2010/05/23/sanitizing-files-with-no-trailing-newline/.

Seeger

This is the only solution that worked for me in ansible – Happily

Answer 9

Try ex-way:

ex -s +"bufdo wq" *.c

And recursively (with a new globbing option enabled):

ex -s +"bufdo wq" **/*.c

This is equivalent to vi -es. Change *.c to extension of your interest.

The ex/vi would automatically append newline on save if it's not present.

Lasonde

Answer 10

Using Bash only

You can use Command Substitution (remove trailing newlines) with Here Strings (appends newline):

   Command Substitution
       Command substitution allows the output of a command to replace the command  name.   There  are  two
       forms:

          $(command)
       or
          `command`

       Bash  performs  the expansion by executing command in a subshell environment and replacing the com-
       mand substitution with the standard output of the command,  with  any  trailing  newlines  deleted.
       Embedded newlines are not deleted, but they may be removed during word splitting.  The command sub-
       stitution $(cat file) can be replaced by the equivalent but faster $(< file).



   Here Strings
       A variant of here documents, the format is:

          [n]<<<word

       The word undergoes brace expansion, tilde expansion, parameter and variable expansion, command sub-
       stitution,  arithmetic expansion, and quote removal.  Pathname expansion and word splitting are not
       performed.  The result is supplied as a single string, with a newline appended, to the  command  on
       its standard input (or file descriptor n if n is specified).

Here's how it works:

cat <<<"$(<inputfile)"

Output to file:

cat <<<"$(<inputfile)" >outputfile

If you need inputfile and outputfile to be the same file name, you have a couple options - use sponge command, save to temporary variable with more command substitution, or save to temporary file.

---

Using Sed

Others have suggested using

sed '$a\' inputfile

which appends nothing to the last line. This is fine, but I think

sed '$q' inputfile

is a bit clearer, because it quits on the last line. Or you can do

sed -n 'p'

which uses -n to suppress output, but prints it back out with p.

In any of these cases, sed will fix up the line and add a newline, at least for GNU and BSD sed. However, I'm not sure if this functionality is defined by POSIX. A version of sed might just skip your line without a newline since a line is defined as

A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

Veinstone

Answer 11

Using awk :

awk '/^$/{f=1}END{ if (!f) {print "\r"}}1' inputfile

Match blank line ^$(just like you did) and set up a flag. If flag is not set at the end, place newline character.

Note: that \r is in OS X. Use \n for other.

Janelljanella

Answer 12

Try using vi or ex:

ex -scwq foo.txt

or for multiple files:

vi -es +"bufdo wq" *.txt
ex -s +"bufdo wq" *.txt

which automatically adds EOL at EOF on file save if it's missing.

To apply for certain files recursively, use a new globbing option (**) such as **/*.txt (enable by shopt -s globstar).

Lasonde

Answer 13

I'm surprised nobody has mentioned that many simple text-processing tools like Awk will add a newline as a side effect. Here is a simple loop which will overwrite a file only if a newline was actually added.

for f in *; do
    awk 1 "$f" >tmp
    cmp -s tmp "$f" || mv tmp "$f"
done
rm -f tmp

(The temporary file is obviously a bit of a wart.)

IDEone demo: http://ideone.com/HpRHcx

Bilateral

Answer 14

find -type f | while read f; do [[ `tail -c1 "$f"` ]] && echo >> "$f"; done

I'm using find instead of for f in * as it is recursive and the question was about "huge number of source files".

I'm using while read instead of find -exec or xargs for performance reasons, it saves spawning shell process every time.

I'm taking advantage of the fact that backtick operator is returning output of command "with any trailing newlines deleted" man bash, so for properly terminated files backtick will be empty and echo will be skipped.

The find | read couple will fail on filenames that contain newlines, but it's easy to fix if required:

find -type f -print0 | while read -d $'\0' f; do [[ `tail -c1 "$f"` ]] && echo >> "$f"; done

Accentuation

Answer 15

Below is my bash script solution. It first checks that the file is a text file. Then, if it's a text file, it uses tail and od (octal dump) to see if the last character is the newline character. If it isn't, then it appends a newline using echo:

item="$1"

if file "$item" | egrep '\btext\b' > /dev/null
then
    if ! tail -c 1 "$item" | od -b -A n | egrep '\b012\b' > /dev/null
    then
        echo "(appending final newline to ${item})"
        echo >> "$item"
    fi
fi

Stuck

Answer 16

In case it is of use, something like this usually works for me:

printf "%s\n" "$(cat file_that_MIGHT_need_a_NL.txt)"

It is not the most elegant solution, but it allows me to use tools like sed, grep, etc. instead of just cat in there.

Of course any valid variable or string should work also.

YMMV

Mohican

Answer 17

Since dos2unix 7.5.0 you can add a newline to the last line, only if it is not already there, by using the -e or --add-eol option:

dos2unix -e file.txt

To check if the last line has a newline type

dos2unix -e -ih file.txt

It will print the type of the newline (dos/unix/mac) or noeol if there isn't one.

Amritsar

Answer 18

Due To command localization Tim and Norman answer Shall be improved using 'LANG=C' prefix to have a chance to match 'No newline' pattern with every system having any regional parameters

This ensures an ending empty line to every file put on the command line of this script :

 #!/bin/sh -f
 for i in $* ; do  echo $i; \
 if LANG=C diff /dev/null "$i" | tail -1 | \
  grep '^\\ No newline' > /dev/null; then echo >> "$i"; \
 fi; done

And this script detects files lacking of it :

 #!/bin/sh -f
 for i in $* ; do \
 if LANG=C diff /dev/null "$i" | tail -1 | \
  grep '^\\ No newline' > /dev/null; then  echo $i; \
 fi; done

Hilliard

Answer 19

I solved this task by using dos2unix (or counterparts) with the --newline flag. The advantage is that these tools detect binary files on their own. I like the solution with tail -c1 but filtering binary files beforehand has been really slow for me.

dos2unix --newline my_file.txt

Eventually I wrote a script that searched my project directory, converted all files to LF (dos2unix) except *.cmd files (CRLF, unix2dos) and used the flag to get the newlines right with one call.

Ethelynethene

I've tested the command, but somehow it doesn't work (at least on OS X). Testing: printf foo > foo.txt && dos2unix --newline foo.txt && wc foo.txt which gives me 0 1 3 which is not correct. – Lasonde

I can confirm that this doesn't work on Debian Linux. I recall doing this successfully in Cygwin back then. – Gurias

Answer 20

After finding the tool do this job with no luck. I decide to write my own

This is my python script to do that job

It only append (\r\n) to file not contains (\n) at the end of file

https://github.com/tranhuanltv/append_newline

Usage: append_newline.py .c ./projects ./result_dir

Make Pull Requests if you want to

Obla

This is very questionable - the seek to -1 from END is OK, but you could easily mix Unix and DOS newlines this way... – Powell

Answer 21

pcregrep --recursive --exclude-dir=.git \
  --files-without-match --multiline '\n\z' . |
  while read k ; do echo >> "$k"; done

There are several steps involved here:

1. Recursively find files
2. Detect which files lack a trailing new line
3. Loop over each of those files
4. Append the newline

Step 1 is traditionally done with find (following the Unix tradition of "each tool doing one thing and doing it well"), but since pcregrep has builtin support, I'm comfortable using it. I'm careful to avoid messing around with the .git folder.

Step 2 is done with a multiline regular expression matching files that do have a final newline, and printing the names of files that don't match.

Step 3 is done with a while/read loop rather than a for/in, since the latter fails for filenames with spaces and for extremely long lists of files.

Step 4 is a simple echo, following @norman-ramsey's approach.

h/t @anthony-bush https://mcmap.net/q/271377/-how-do-i-find-files-that-do-not-end-with-a-newline-linefeed for the pcregrep suggestion.

Enlistee

Answer 22

An elegant solution exists using standard shell commands:

tail -c 1 file.txt | read || echo >> file.txt    

1. tail Outputs the last byte of file
2. read Reads a line into a variable. With no variable specified, does nothing, but if an EOF occurs before a newline, exits with code 1.
3. echo Runs only if read fails (i.e. if the last character was not a newline), and appends a newline to file.txt

Porcine

With no variable, read will populate $REPLY variable, by default. If you really want to ignore read input, you could use _ variable: read _. – Mime