Why should text files end with a newline?

Question

I assume everyone here is familiar with the adage that all text files should end with a newline. I've known of this "rule" for years but I've always wondered — why?

Copybook

just a nitpick. it's not a "new line" at the end of the file. It's a "line break" at the end of the last line. Also, see the best answer on a related question: #16223030 – Sollars

Just to nitpick some more, he didn't actually write “new line”, he wrote “newline”, which is correct. – Prud

not familiar, but wondering I am indeed because the number of cases where that superfluous newline is actually breaking things is a little too high to my tastes – Goodfornothing

I'd not heard this adage until today. I was aware the POSIX requires it because lots of old tools assume it, but I generally live in a non-POSIX world where most tools don't care. – Jarrell

I'm currently using Node.js streams to parse plain-text data line-by-line, and the lack of terminal line-break is annoying, as I have to add extra logic for when the input side of the stream is finished/closed in order to ensure that the last line gets processed. – Cryostat

The way Unix regards its general behavior at the end of files is as follows: \n characters don't start lines; instead, they end them. So, \n is a line terminator, not a line separator. The first line (like all lines) needs no \n to start it. The last line (like all lines) needs an \n to end it. An \n at the end of the file doesn't create an additional line. Sometimes, however, text editors will add a visible blank line there. Even emacs does so, optionally. – Mauk

@Prud I suppose this whole confusion is verbal. To avoid it, the C language and Unix developers instead could have called the ASCII linefeed character EOL or endOfLine. Following that idea, \l would be its natural abbreviation (\l is unused in C). However, visibly, this lower-case letter L doesn't differ much from the numeral "1" (unfortunately). And \1 is used in regular expressions. – Mauk

@Mauk Well, yeah. But why are you bringing this up now? :P – Prud

@Prud I bring this up to help people. If people remembered "end of line character" as its name, they wouldn't question why the last line of a file has one. Since the developers grabbed the next best thing (calling it "new line character"—which is a slight misnomer), naturally they wonder why it doesn't perform its apparent function and create a new line there. – Mauk

@Mauk I absolutely agree with you that the name is kind of terrible in regards to what it's actually there for. ☺ – Prud

@Mauk Of course you could say that that's arguing semantics too. Certainly '\n' in C is newline. But then again in ascii(7) it has: 012 10 0A LF '\n' (new line). So is it newline, line feed or something else? Well that depends on whom you ask. I know what you're saying though: if main() ends with: printf("\nTest\n"); then the line under 'Test' will be the shell prompt. It's subtle but still worth mentioning in my mind. (Of course putchar('\n'); will just show a blank line above the prompt.) – Soap

I am wondering if anyone noticed that the definition of term "Line" in POSIX standard (pubs.opengroup.org/onlinepubs/9699919799/basedefs/…) doesn't mention term "file". It is obvious to me that there is no reason to treat any byte (even 0x0a) in a "file" as "terminating <newline> character" of the "line". This is just a convenience people got used to when dealing with files. The POSIX "line" is just a part of text stream which can be "piped" in or out of a process. It is not something that "text" "file" should consist of. – Percaline

POSIX distinguishes between a File and a Text File, with the latter being defined as “A file that contains characters organized into zero or more lines” (emphasis mine). As a Line by definition is always terminated by a <newline> character, a pedantic argument can be made that files not ending in \n are not really text files. – Daydream

How much engineering time has been wasted adding newlines to the end of files? How about we just stop using garbage tools that can't handle files that don't have line breaks at the end? – Delfeena

One might also suggest we stop using tools that allow users to produce that which are objectively not "standard" text files, despite their claims to operate exclusively on such files. vim/emacs/edit/&c. have all included newlines after every line by default for decades (though this can be disabled when/if needed). Personally, I have yet to encounter the tool that truly cannot gracefully handle both cases. read(1) (as in while read line || [ -n "$line" ]; do) is the only case I know where the difference matters, and even then, not w/o good reason: it allows for keying & lossless I/O. – Daydream

Answer 1

Because that’s how the POSIX standard defines a line:

3.206 Line

A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

Therefore, “lines” not ending in a newline character aren't considered actual lines. That's why some programs have problems processing the last line of a file if it isn't newline terminated.

The advantage of following this convention is that all POSIX tools expect and use it. For instance, when concatenating files with cat, a file terminated by newline (a.txt and c.txt below) will have a different effect than one without (b.txt):

$ more a.txt
foo

$ more b.txt
bar
$ more c.txt
baz

$ cat {a,b,c}.txt
foo
barbaz

We follow this rule for consistency. Doing otherwise would incur extra work when dealing with the default POSIX tools.

---

Think about it differently: If lines aren’t terminated by newline, making commands such as cat useful is much harder: how do you make a command to concatenate files such that

1. it puts each file’s start on a new line, which is what you want 95% of the time; but
2. it allows merging the last and first line of two files, as in the example above between b.txt and c.txt?

Of course this is solvable but you need to make the usage of cat more complex (by adding positional command line arguments, e.g. cat a.txt --no-newline b.txt c.txt), and now the command rather than each individual file controls how it is pasted together with other files. This is almost certainly not convenient.

… Or you need to introduce a special sentinel character to mark a line that is supposed to be continued rather than terminated. Well, now you’re stuck with the same situation as on POSIX, except inverted (line continuation rather than line termination character).

---

_{Now, on non POSIX compliant systems (nowadays that’s mostly Windows), the point is moot: files don’t generally end with a newline, and the (informal) definition of a line might for instance be “text that is separated by newlines” (note the emphasis). This is entirely valid. However, for structured data (e.g. programming code) it makes parsing minimally more complicated: it generally means that parsers have to be rewritten. And if a parser was originally written with the POSIX definition in mind, then it might be easier to modify the token stream rather than the parser — in other words, add an “artificial newline” token to the end of the input.}

Whencesoever

Although now quite impractical to rectify, clearly POSIX made a mistake when defining the line -- as evidence by the number of questions regarding this issue. A line should have been defined as zero or more characters terminated by <eol>, <eof>, or <eol><eof>. Parser complexity is not a valid concern. Complexity, wherever possible, should be moved from the programmers head and into the library. – Depravity

@DougCoburn This answer used to have an exhaustive, technical discussion explaining why this is wrong, and why POSIX did the right thing. Unfortunately these comments were apparently recently deleted by an overzealous moderator. Briefly, it’s not about parsing complexity; rather, your definition makes it much harder to author tools such as cat in a way that’s both useful and consistent. – Whencesoever

Fair enough -- That's unfortunate that they were removed. I would be interested in knowing the reason why this is still considered the "right" thing. – Depravity

@KonradRudolph I don't see anything in the edit history. Shouldn't it be in the edit history? Or was it before there was history? It was back in 09... – Clambake

@adjenks Not in the answer but in the comments. – Whencesoever

Oh I see, "This comment was edited 3 times."... But you can't view history. – Clambake

@adjenks There previously used to be ~20 comments under this answer discussing this. They were deleted, not edited. Anyway, I’ve just edited the answer to add an explanation of why POSIX’ newline definition is more practical, and how an alternative definition would have to look like to be equally convenient (the one suggested by Doug is insufficient). – Whencesoever

The solution to the concatenation problem is to not treat files as if they're all in the same encoding. If we're talking about source code files, it would pretty much never make sense to concatenate them such that the last line of the first file, and the first line of the second file, are merged into one line. Context matters. Trying to shoehorn everything such that it fits your 5% use case is almost certainly a bad call. Unix and linux hasn't shied away from command line flags before, why shy away from using a -r (for raw) or something with cat? I think POSIX did in fact make a mistake here – Delfeena

Software should handle edge cases properly instead of mandating people to comply with such stupid rules (there are so many stupid things wasting develops' life). – Bigham

@Bigham The POSIX rule is all about reducing edge cases. And it does so beautifully. I’m actually somewhat at a loss how people fail to understand this: It’s the simplest possible, self-consistent definition of a line. – Whencesoever

@BT I think you’re assuming that my example of a more convenient workflow is the reason behind the decision. It’s not, it’s just a consequence. The reason is that the POSIX rule is the rule that’s simplest, and which makes handling lines in a parser the easiest. The only reason we’re even having the debate is that Windows does it differently, and that, as a consequence, there are numerous tools which fail on POSIX files. If everybody did POSIX, there wouldn’t be any problem. Yet people complain about POSIX, not about Windows. – Whencesoever

@KonradRudolph If a file is ended, of course the last line is ended. IMO this is everyone's expectation. The file is ended, but you expect the line to be continued -- this is strange. Anyway, I can't change the behavior of sed, wc etc. – Bigham

@Bigham You don’t need to change the behaviour of any of these tools, they already behave correctly and consistently when used on any file produced by such tools, or otherwise adhering to POSIX guidelines. How would you even create a text file which doesn’t? That’s already not trivial on GNU/BSD/POSIX systems. – Whencesoever

This isn't about windows, don't build a straw man. Literally no one except you is talking about windows here. I agree forcing every file to end in a newline makes parsing things by line (insignificantly) easier. I also agree that doing that makes concatenating files that may represent chunks not split by line, (insignificantly) easier. What I don't agree with is that making parsing slightly easier is worth forcing almost everyone to add otherwise usually-unnecessary newlines at the end of their file. Its also no excuse to write programs that keel over for a file without an ending newline. – Delfeena

@BT I’m only referring to Windows to point out the cases where POSIX rules don’t make sense (in other words, I was throwing you a bone). I’m more than happy never to mention it in this discussion again. But then your claim makes even less sense: on POSIX platforms it simply makes no sense to discuss text files with different line ending conventions, because there’s no reason to produce them. What’s the advantage? There is literally none. — In summary, I really don’t understand the hatred this answer (or the POSIX rule) are engendering. To be frank, it’s completely irrational. – Whencesoever

I disagree, it's not the best way. You should just add '\n''s before concatenating another file. – Nahama

@Nahama Of course it’s possible. But I fail to see how that would be of any advantage whatsoever. – Whencesoever

It's a small requirement, but when you multiply that small requirement by dozens of tools and editors, it becomes a big requirement. – Nahama

@Nahama Not putting newline is also a requirement, and is in no way cheaper or easier. The requirement, in either case, is “adhere to the standard”. Your claim that omitting newlines makes anything easier is a fallacy. – Whencesoever

@KonradRudolph False. Not putting newline is not a requirement: the user can always either put one or not, the choice is up to them. They've never been constrained in this manner. Some of the tools, however - namely, the older Unix ones - require a newline. So no, the former is not "adhere to the standard", it's "make your own personal/group standard", whereas the latter is, "this is how we're all going to do it, now comply with our arbitrary standard". It's not about being easier - nor did I say it was - I said it was better. Not standardizing on unnecessary requirements is better. – Nahama

@Nahama Of course there’s such a requirement: Your rule must be internally consistent (otherwise a tool doesn’t know whether a terminal newline signifies an extra, empty line or not). And “users” don’t come into this. Tools need to produce consistent files. Tools could agree to treat the newline character as line separator rather than line terminator (which is what you suggest, and what many Windows tools do) but — contrary to what you’ve said — this doesn’t make anything better, it just makes it different. If we needed no standard, as you claim, we wouldn’t be discussing this. – Whencesoever

I really don't understand all the vitriol toward POSIX here. It's like arguing whether array indices should start at 0 or 1. 0 provides technical advantages, 1 is more familiar, but either one can work as long as it's consistent. Editors for POSIX still handle it for you, the same way they handle LF vs CR/LF. Side note: Other technical advantages of the POSIX way are 1) It lets you do things like split a text file into 4KB chunks without worrying about splits happening mid-line, and 2) (the big one) it allows tools like cat to also work on binary files with no special considerations. – Antonetta

The Unix line terminator convention is one instance of the Unix philosophy: "Make data complicated when required, not the program". By specifying the convention that lines must ends with newlines, you simplify the implementation of all programs and many POSIX APIs. This simplicity means that sometimes things are not perfect (particularly when dealing with files that don't follow convention), but when you stack together many tools into a complex pipeline, it is a lot easier to reason about why things doesn't work as expected when the tools themselves are aren't being too smart for its own good. – Grille

Years ago, I had a strange syntax error in the slash-star comment header at the top of a source file. The comment was correct; drove me crazy. Then, I realized that that file and all the others were concatenated before heading to the compiler. Surprise, the file before ended with a slash-slash comment, and no newline char, so the initial slash-star of the next file was commented out. – Cultivate

@DougCoburn This issue has nothing to do with the actual POSIX standard and instead everything to do with the fact that tools exist that allow users to easily and unknowingly break the standard. An editor that allows you to see, remove, and save a file without the terminating newline is a broken editor. You're blaming the standard when the problem is programs not following the standard. Whether the standard is good or bad is irrelevant. Text editors should be expected to always create standard-compliant files without the user needing to know about the standard. – Belovo

@Belovo I agree. I like how github, for example, warns when you do not end a file with a newline. When I first commented, I had not considered the case of splitting large text (ie log) files at size boundaries rather than newline boundaries. The POSIX definition makes sense to me now. – Depravity

@JShorthouse: What term should one use for a file that contains or ends with a partial line, with the intention that concatenating its contents with some other data will cause the first line of the latter file to be joined with the trailing characters of the first one? What sort of program should be used to edit such files? – Beckerman

@Beckerman a file that doesn't adhere to the POSIX standard. The fact that all editors will still open files that break the standard is irrelevant. I could split the binary data of a jpg file in half with the intention of joining them together later. The fact that two halves join together to make a valid file doesn't mean that the halves by themselves are valid. – Belovo

@JShorthouse: The file may not adhere to the POSIX standard for a "text" file, but POSIX does not require that all files be text files. Being able to use text-file editing tools to view and edit such files seems more helpful than having to use special-purpose tools, though I'd certainly agree that text editors could probably do a better job distinguishing between the use cases of editing text files and editing text-ish files. – Beckerman

For me, it's solving small problem by introducing a big one and quoting POSIX is not a valid answer, because it moves the question just to "Why is the end of file unused when determining end of file instead of using end of line for it?" There is way neater solution to run some tools if there is specialized POSIX environment needing the end of line for last line and changing the standard appropriately. In the current state, there is reality and some POSIX standard being in a parallel one. – Excavator

@LukasSalich I invite you to set a reminder in, say, five years. Come back to the question then and revisit your stance. If you’ve used Unix shells in all that time, I’m pretty sure you’ll have changed your opinion. At the moment there’s no arguing with you because your commend isn’t really cogent and makes no sense. – Whencesoever

@Bigham "If a file is ended, of course the last line is ended." That reduces flexibility; with that rule you can no longer have files with line fragments that can be concatenated to create a single line. Instead, you need to build a special tool to join lines and whether or not a file contains a line fragment must be stored separately outside the file. – Drobman

@DougCoburn "Parser complexity is not a valid concern." It absolutely is a concern in Unix's (and thus POSIX's) "worse is better" approach, as contrasted with the "MIT approach" used in some other operating systems. WiB has its good and bad points, as does MIT, but it would be inarguably worse to use WiB for some things and MIT for others as that would leave you with the disadvantages of both. – Drobman

@Drobman As a user of text file, one would never worry about your issue about concat, sed, wc etc. As a developr, you could not presume any text file has an endline at the end of file. You have to check it (when necessary). – Bigham

@Bigham Your comment seems to have nothing to do with the comment you are replying to. Instead you seem to be rehashing a now three year old basic misunderstanding that has been explained to you repeatedly. – Whencesoever

In the POSIX manual for cat the 'INPUT FILES' section says "The input files can be any file type." The cat example in this answer is informative, but the best conclusion might simply be that cat isn't always the right tool for the job. – Kemeny

Answer 2

Each line should be terminated in a newline character, including the last one. Some programs have problems processing the last line of a file if it isn't newline terminated.

GCC warns about it not because it can't process the file, but because it has to as part of the standard.

The C language standard says A source file that is not empty shall end in a new-line character, which shall not be immediately preceded by a backslash character.

Since this is a "shall" clause, we must emit a diagnostic message for a violation of this rule.

This is in section 2.1.1.2 of the ANSI C 1989 standard. Section 5.1.1.2 of the ISO C 1999 standard (and probably also the ISO C 1990 standard).

Reference: The GCC/GNU mail archive.

Willenewillet

please write good programs then that either allow to insert that newline where needed while processing or are able to properly handle "missing" ones... which are, in fact, not missing – Goodfornothing

@BilltheLizard, What are some examples of "Some programs have problems processing the last line of a file if it isn't newline terminated"? – Exhibitive

@Exhibitive wc -l won't count the last line of a file if it isn't newline terminated. Also, cat will join the last line of a file with the first line of the next file into one if the last line of the first file isn't newline terminated. Pretty much any program that's looking for newlines as a delimiter has the potential to mess this up. – Willenewillet

@BilltheLizard, I mean wc has already been mentioned.... – Exhibitive

@BilltheLizard, My bad, to clarify: what are some examples of programs that have problems processing the last line of a file if it isn't newline terminated (besides those that have already been mass-mentioned on the thread like cat and wc)? – Exhibitive

Visual Studio Resource Compiler (rc) chokes if the last line isn't terminated with a line terminator. – Jarrell

@BilltheLizard: So, is this program invokes UB when compiled as a C++98 & C++03 code. See the program: ideone.com/jswwf9 – Poppied

The C++14 standard says something slightly different: "A source file that is not empty and that does not end in a new-line character ... shall be processed as if an additional new-line character were appended to the file" [emphasis added]. This seems redundant since phase 1 of translation requires the insertion of "new-line characters for end-of-line indicators" [lex.phases]. – Jarrell

I think the question is not gcc related.... C standard talks about properly line ending files, but from a source code perspective. The compiler is allowed to arrange source code characters to deal with files lacking the last line ending. The purpose of forcing a line end in the last line of a file (logical) is for parsing purposes (two tokens could be appended if no separator is in between) – Seymourseys

Answer 3

This answer is an attempt at a technical answer rather than opinion.

If we want to be POSIX purists, we define a line as:

A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

Source: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206

An incomplete line as:

A sequence of one or more non- <newline> characters at the end of the file.

Source: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_195

A text file as:

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2008 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

Source: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_397

A string as:

A contiguous sequence of bytes terminated by and including the first null byte.

Source: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_396

From this then, we can derive that the only time we will potentially encounter any type of issues are if we deal with the concept of a line of a file or a file as a text file (being that a text file is an organization of zero or more lines, and a line we know must terminate with a <newline>).

Case in point: wc -l filename.

From the wc's manual we read:

A line is defined as a string of characters delimited by a <newline> character.

What are the implications to JavaScript, HTML, and CSS files then being that they are text files?

In browsers, modern IDEs, and other front-end applications there are no issues with skipping EOL at EOF. The applications will parse the files properly. It has to since not all Operating Systems conform to the POSIX standard, so it would be impractical for non-OS tools (e.g. browsers) to handle files according to the POSIX standard (or any OS-level standard).

As a result, we can be relatively confident that EOL at EOF will have virtually no negative impact at the application level - regardless if it is running on a UNIX OS.

At this point we can confidently say that skipping EOL at EOF is safe when dealing with JS, HTML, CSS on the client-side. Actually, we can state that minifying any one of these files, containing no <newline> is safe.

We can take this one step further and say that as far as NodeJS is concerned it too cannot adhere to the POSIX standard being that it can run in non-POSIX compliant environments.

What are we left with then? System level tooling.

This means the only issues that may arise are with tools that make an effort to adhere their functionality to the semantics of POSIX (e.g. definition of a line as shown in wc).

Even so, not all shells will automatically adhere to POSIX. Bash for example does not default to POSIX behavior. There is a switch to enable it: POSIXLY_CORRECT.

Food for thought on the value of EOL being <newline>: https://www.rfc-editor.org/old/EOLstory.txt

Staying on the tooling track, for all practical intents and purposes, let's consider this:

Let's work with a file that has no EOL. As of this writing the file in this example is a minified JavaScript with no EOL.

curl http://cdnjs.cloudflare.com/ajax/libs/AniJS/0.5.0/anijs-min.js -o x.js
curl http://cdnjs.cloudflare.com/ajax/libs/AniJS/0.5.0/anijs-min.js -o y.js

$ cat x.js y.js > z.js

-rw-r--r--  1 milanadamovsky   7905 Aug 14 23:17 x.js
-rw-r--r--  1 milanadamovsky   7905 Aug 14 23:17 y.js
-rw-r--r--  1 milanadamovsky  15810 Aug 14 23:18 z.js

Notice the cat file size is exactly the sum of its individual parts. If the concatenation of JavaScript files is a concern for JS files, the more appropriate concern would be to start each JavaScript file with a semi-colon.

As someone else mentioned in this thread: what if you want to cat two files whose output becomes just one line instead of two? In other words, cat does what it's supposed to do.

The man of cat only mentions reading input up to EOF, not <newline>. Note that the -n switch of cat will also print out a non- <newline> terminated line (or incomplete line) as a line - being that the count starts at 1 (according to the man.)

-n Number the output lines, starting at 1.

Now that we understand how POSIX defines a line , this behavior becomes ambiguous, or really, non-compliant.

Understanding a given tool's purpose and compliance will help in determining how critical it is to end files with an EOL. In C, C++, Java (JARs), etc... some standards will dictate a newline for validity - no such standard exists for JS, HTML, CSS.

For example, instead of using wc -l filename one could do awk '{x++}END{ print x}' filename , and rest assured that the task's success is not jeopardized by a file we may want to process that we did not write (e.g. a third party library such as the minified JS we curld) - unless our intent was truly to count lines in the POSIX compliant sense.

Conclusion

There will be very few real life use cases where skipping EOL at EOF for certain text files such as JS, HTML, and CSS will have a negative impact - if at all. If we rely on <newline> being present, we are restricting the reliability of our tooling only to the files that we author and open ourselves up to potential errors introduced by third party files.

Moral of the story: Engineer tooling that does not have the weakness of relying on EOL at EOF.

Feel free to post use cases as they apply to JS, HTML and CSS where we can examine how skipping EOL has an adverse effect.

Schweiz

POSIX is not tagged in the question... wat about MVS/OS line endings? or MS-DOS line endings? By the way, all known posix systems allow text files without a final line ending (no case found of a posix compliant claiming system on which "text file" has special treatment in the kernel to insert a proper newline in case it doesn't have it) – Seymourseys

"There will be very few real life use cases where skipping...". Not true. In real life I review code every day, and it's a waste of time dealing with useless merge diffs caused by files missing the trailing newline. For consistency, every line (even the last line in the file) should be properly terminated. – Clearheaded

POSIX's definition of "text line" is relevant as a term they can refer to within the standard e.g. wc -l will not count the last incomplete line. But whether it aspire to regulate semantics or not, doesn't matter much; a very important quality of UNIX I/O is that it's not record-oriented, it's just a flat sequence of 8-bit octets — any meaning is assigned by individual programs! Most core utilities respect that, and try not to choke on any input. For example diff is line-oriented, but it had to invent \ No newline at end of file notation in cases that's the only difference. – Diamond

Answer 4

It may be related to the difference between:

• text file (each line is supposed to end in an end-of-line)
• binary file (there are no true "lines" to speak of, and the length of the file must be preserved)

If each line does end in an end-of-line, this avoids, for instance, that concatenating two text files would make the last line of the first run into the first line of the second.

Plus, an editor can check at load whether the file ends in an end-of-line, saves it in its local option 'eol', and uses that when writing the file.

A few years back (2005), many editors (ZDE, Eclipse, Scite, ...) did "forget" that final EOL, which was not very appreciated.
Not only that, but they interpreted that final EOL incorrectly, as 'start a new line', and actually start to display another line as if it already existed.
This was very visible with a 'proper' text file with a well-behaved text editor like vim, compared to opening it in one of the above editors. It displayed an extra line below the real last line of the file. You see something like this:

1 first line
2 middle line
3 last line
4

Swum

+1. I've found this SO question while experiencing this very issue. It is very annoying of Eclipse to show this "fake" last line, and If I remove it, then git (and all other unix tools that expect EOL) complains. Also, note that this is not only in 2005: Eclipse 4.2 Juno still has this issue. – Cabretta

@MestreLion, Continuation at #730192 – Exhibitive

Answer 5

Some tools expect this. For example, wc expects this:

$ echo -n "Line not ending in a new line" | wc -l
0
$ echo "Line ending with a new line" | wc -l
1

Pulchritude

I would not say "some", I say most tools expect that for text files, if not all. cat, git, diff, wc, grep, sed... the list is huge – Cabretta

Maybe one could say that wc doesn't expect this, as much as it is simply working within the POSIX definition of a "line" as opposed to most people's intuitive understanding of "line". – Babbie

@Babbie The intuitive definition would be for wc -l to print 1 in both cases, but some people might say the second case should print 2. – Pulchritude

@Pulchritude If you think of \n as a line terminator, rather than as a line separator, as POSIX/UNIX does, then expecting the second case to print 2 is absolutely crazy. – Mccomas

Answer 6

A separate use case: commit hygiene, when your text file is version controlled.

If content is added to the end of the file, then the line that was previously the last line will have been edited to include a newline character. This means that blameing the file to find out when that line was last edited will show the newline addition, not the commit before that you actually wanted to see.

(The example is specific to git, but the same approach applies to other version control systems too.)

Sugarplum

Answer 7

Why should text files end with a newline?

Because that's the sanest choice to make.

Take a file with the following content,

one\n
two\n
three

where \n means a newline character, which on Windows is \r\n, a return character followed by line feed, because it's so cool, right?

How many lines does this file have? Windows says 3, we say 3, POSIX (Linux) says that the file is crippled because there should be a \n at the end of it.

Regardless, what would you say its last line is? I guess anybody agrees that three is the last line of the file, but POSIX says that's a crippled line.

And what is its second line? Oh, here we have the first strong separation:

• Windows says two because a file is "lines separated by newlines" (wth?);
• POSIX says two\n, adding that that's a true, honest line.

What's the consequence of Windows choice, then? Simple:

You cannot say that a file is made up of lines

Why? Try to take the last line from the previous file and replicate it a few times... What you get? This:

one\n
two\n
threethreethreethree

Try, instead, to swap second and third line... And you get this:

one\n
threetwo\n

Therefore

You must say that a text file is an alternation of lines and \ns, which starts with a line and ends with a line

which is quite a mouthful, right?

And you want another strange consequence?

You must accept that an empty file (0 bytes, really 0 bits) is a one-line file, magically, always because they are cool at Microsoft

Which is quite a crazyness, don't you think?

What is the consequence of POSIX choice?

That the file on the top is just a bit crippled, and we need some hack to deal with it.

Being serious

I'm being provocative, in the preceding text, for the reason that dealing with text files lacking the \n at the end forces you to treat them with ad-hoc ticks/hacks. You always need an if/else somewhere to make things work, where the branch dealing with the crippled line only deals with the crippled line, all the other lines taking the other branch. It's a bit racist, no?

My conclusion

I'm in favour of POSIX definition of a line for the following reasons:

• A file is naturally conceived as a sequence of lines
• A line shouldn't be one thing or another depending on where it is in the file
• An empty file is not a one-line file, come on!
• You should not be forced to make hacks in your code

---

And yes, Windows does encourage you to omit the trailing \r\n. If you want a two lines file below, you have to omit the trailing \r\n otherwise text editors will show it as a 3-lines file:

Mortgage

your answer makes me disagree with the posix choice. it unnecessarily introduces invalid file states, and it makes the meaning of "newline" incorrect. "newline" should be called "line marker" instead, being the only thing that turns text content into lines and without which the content is (for some reason) meaningless. – Burne

@DaveCousineau, sorry I don't understand. Why is line marker a better choice? Is it an accurate name? What in its name, for instance, implies that it should be the trailing character of the line? Line terminator is probably a better choice. But anyway, try to substitute newline character with \n, and you'll see how your comment changes: it makes the meaning of \n incorrect. \n should be called "line marker" instead. I'd answer call it as you want, even line drug dealer, it will still be the same thing. – Mortgage

What makes a newline character, line terminator, line feed, nuova riga, a capo, or whatever you want to call it, is not its name, but its role in the POSIX definition of what a line or text file is. – Mortgage

the implication of "newline" is that it makes a new line, which apparently it does not. rather it makes the current non-line into a line. "line terminator" does work. but still I don't think I agree with having unnecessary invalid states. there's no reason why "text\ntext" should not be decipherable text. – Burne

I'm not sure if it matters, but we don't have to call a 0-byte file a 1 line file if we say a "line" has to have at least one character (ie that the empty string is not a line). maybe I'm missing something though. (the context of my comments is that I have recently forced myself to start adding a "line terminator" to the end of my text files, despite my instincts on the matter. I'm still undecided, but reading your answer makes me NOT want to add them.) – Burne

@DaveCousineau, (To the implication of comment) Indeed all tools (that I know of) can deal with that. The fact that echo -ne 'ciao' | wc -l gives 0 doesn't mean that the file cannot be deciphered, but just that it doesn't have a line based on the definition. If you echo -ne 'ciao' | cat, it'll print it indeed, in a way that is influenced by the lack of trailing \n. (To I'm not sure comment) Well, it's Windows that defines the \n character to be the line separator, implying that an empty file is a one line file (just like any file which has all non-\n characters). – Mortgage

The motivation is correct, but I'm not aware of spefically Windows encouraging omitting final \r\n or semantically treating it as "separator". Most windows files do end with a newline. At the OS interface level AFAIK both systems treat files as sequence of bytes; C programs on windows internally support the fiction of "text mode" translating \r\n <-> \n. Other than that, the C stdlib on both is the practically same, and the conventions very similar. – Diamond

@BeniCherniavsky-Paskin, Most windows files do end with a newline can be rewritten as Most windows files are shown with an empty line at the end in text editors. And since perfectionist people don't like trailing empty lines, that's actually an encouragement to omit the final \r\n. – Mortgage

Interesting point. But so do many unix editors e.g. emacs, gedit, micro. FWIW I consider myself a perfectionist that does like a trailing empty line in an editor, but that's because I understand the byte-level meaning of trailing \n ;-) I also like trailing commas in multi-line arrays. – Diamond

But so do many unix editors. Not Vim. Nor other utilities, such as cat. And not even some "unconvetional" editors like sed. I'd be curious to know what the byte-level meaning that I don't understand is. – Mortgage

I just meant that I don't think of it as "extra empty line" but understand (in those editors) it means the last line has the proper \n byte. I can position cursor before or after \n—but only if it's there. BTW, this correlates to editors letting you move from end of line to start of next line using ←/→ keys. emacs, gedit, micro all do! vim doesn't (ootb, I'm sure is configurable). P.S. need micro --eofnewline=false to allow saving without trailing \n. – Diamond

Answer 8

Basically there are many programs which will not process files correctly if they don't get the final EOL EOF.

GCC warns you about this because it's expected as part of the C standard. (section 5.1.1.2 apparently)

"No newline at end of file" compiler warning

Cosher

GCC isn't incapable of processing the file, it has to give the warning as part of the C standard. – Willenewillet

IIRC, MSVC 2005 complained about C files which ended with incomplete lines and possibly refused to compile them. – Cryostat

Answer 9

I've wondered this myself for years. But i came across a good reason today.

Imagine a file with a record on every line (ex: a CSV file). And that the computer was writing records at the end of the file. But it suddenly crashed. Gee was the last line complete? (not a nice situation)

But if we always terminate the last line, then we would know (simply check if the last line is terminated). Otherwise we would probably have to discard the last line every time, just to be safe.

Woodsman

I agree, I always think it is a poor's man "checksum" that says that when the end of line is missing it indicates that the file is probably truncated. Of course it is not a guarantee the other way around. At least for text files; for binary files I don't know if it is a valid convention. – Quenby

@Quenby that's a good description. that's how i'm using it. yeah, this of course doesn't work for binary files – Woodsman

This is actually a terrible reason. Your filesystem should be used to handle this. Modern file systems are journaled, which is a far far far better way of recognizing if a file write completed as it works for binary files as well as text files, and has an actual historical record of the last attemtped write. – Delfeena

@BT that depends on what you mean with "a file write". when i encountered this (repeatedly), the last line was not fully written and it was on a journaling file system – Woodsman

@BT not all files are written directly to a local filesystem. Streams, objects stores, etc. do exist. The same people complaining about having to use a line break on a last line – typically for rather superficial reasons – will happily use a trailing comma on array entries or function arguments on programming languages that support them. – Auriga

@Auriga thanks for objecting to my objection with information irrelevant to symbiont's answer, which is specific to filesystems. Find a better place to inject your opinion and I'd discuss it with you. – Delfeena

@BT You're quite a surly chap, eh? Quite relevant, You literally wrote that it's the filesystem's responsibility to know whether a file is complete or not, but a terminating line, as symbiont explains, makes it clear whether you at least have a complete record. There are plenty of circumstances where one can encounter an incomplete file, which to add merit to a terminating character. Your personal vendetta against them does not at all detract from the sensible option to terminate records with them. Yours is a bogus reason that only applies to a specific write destination. – Auriga

What if it crashes right after it wrote a newline that is not at the end of file? How would you know if the newline belongs to the last line? – Valuator

@КонстантинВан I mean that we are only appending to the end of the file. So we don't write any text, before the preceding text is written. So we don't go back and change text (we only append). So the missing newline can only occur at the end of the file. Only the last line can be incomplete – Woodsman

Answer 10

This originates from the very early days when simple terminals were used. The newline char was used to trigger a 'flush' of the transferred data.

Today, the newline char isn't required anymore. Sure, many apps still have problems if the newline isn't there, but I'd consider that a bug in those apps.

If however you have a text file format where you require the newline, you get simple data verification very cheap: if the file ends with a line that has no newline at the end, you know the file is broken. With only one extra byte for each line, you can detect broken files with high accuracy and almost no CPU time.

Monometallism

nowadays the newline at EOF for text files may not be a requirement, but it is a useful convention that makes most unix tools work together with consistent results. It's not a bug at all. – Cabretta

Lots of us don't use Unix tools at all, and we don't care. – Quick

It's not just unix tools, any tool will work better and/or be coded more simply if it can assume sensible file formats. – Byre

@Sam Watkins Agree having simple well defined formats is good. Yet code still needs to verity, and not assume, the data is format compliant. – Thousandth

@Cabretta This is a useless legacy from a set of bad tools compliant to stupid standards. These artifacts of extremist programming(i.e. everything's file! everything should talk plain text!) didn't die soon after their invention because they were the only available tools of the kind at a certain moment of history. C was superseded by C++, it's not a part of POSIX, it requires no EOL at EOF, and its usage is (obviously) discouraged by *nix luddists. – Lumbard

@Lumbard Actually, data formats and have become more and more text-based over the years. XML/HTML, JSON, YAML, as well as protocols like HTTP, RPC, SOAP, REST. Those are not legacy, and having a solid convention on how tools should handle lines is neither useless nor stupid. – Cabretta

@Cabretta Programming gets less and less science-based. More and more uneducated people start calling themselves programmers. Big-big companies tend to create more technical debt in the industry to kill small business. That's how these protocols were created. Having a solid convention "nobody cares" is neither useless nor stupid. Having a convention "we must end every file with an invisible character" is. – Lumbard

The point here really is just that you "end with a newline"rs are forcing standards that are unnecessary, even if they have arguments behind them. Some of us just don't want unnecessary conventions. – Nahama

Answer 11

In addition to the above practical reasons, it wouldn't surprise me if the originators of Unix (Thompson, Ritchie, et al.) or their Multics predecessors realized that there is a theoretical reason to use line terminators rather than line separators: With line terminators, you can encode all possible files of lines. With line separators, there's no difference between a file of zero lines and a file containing a single empty line; both of them are encoded as a file containing zero characters.

So, the reasons are:

1. Because that's the way POSIX defines it.
2. Because some tools expect it or "misbehave" without it. For example, wc -l will not count a final "line" if it doesn't end with a newline.
3. Because it's simple and convenient. On Unix, cat just works and it works without complication. It just copies the bytes of each file, without any need for interpretation. I don't think there's a DOS equivalent to cat. Using copy a+b c will end up merging the last line of file a with the first line of file b.
4. Because a file (or stream) of zero lines can be distinguished from a file of one empty line.

Frazer

Answer 12

Presumably simply that some parsing code expected it to be there.

I'm not sure I would consider it a "rule", and it certainly isn't something I adhere to religiously. Most sensible code will know how to parse text (including encodings) line-by-line (any choice of line endings), with-or-without a newline on the last line.

Indeed - if you end with a new line: is there (in theory) an empty final line between the EOL and the EOF? One to ponder...

Gwendolyn

It's not a rule, it's a convention: a line is something that ends with an end-of-line. So no, there is no "empty final line" between EOL and EOF. – Cabretta

@MestreLion: But the character in question is not named "end-of-line", it's named "newline" and/or "linefeed". A line separator, not a line terminator. And the result IS a final empty line. – Rudin

No (sane) tool would count the last EOL (CR, LF, etc) of a file as an additional, empty line. And all POSIX tools will not count the last characters of a file as a line if there is no ending EOL. Regardless of the EOL character name being "line feed" or "carriage return" (there's no character named "newline"), for all practical puposes sensible tools treat it as a line terminator, not as a line separator. – Cabretta

@MestreLion, Are you sure "line terminator" is sane? Grab a few non-programmers and do a quick survey. You'll quickly realize the concept of lines is closer to the concept of "line separators". The concept of "line terminator" is just weird. – Exhibitive

@Cabretta if no "sane" tool would count the last EOL as creating a new empty line, how does a user get to the next line to add content to it? I guess in your view, there is always an extra EOL, even in a completely "empty" file? – Burne

@Sahuagin: This is not my view, this is how the POSIX Standard defines a line. An empty file with 0 bytes has 0 lines, hence no EOL, and a file to be considered as having just a single, blank line, it does require an EOL. Also note this is only relevant if you want to count the lines on a file, as obviously any editor will let you "get" to the next (or the first) line regardless if there is already an EOL there. – Cabretta

@Cabretta so, the user can browse to a line that doesn't exist, and there can be data within the file that does not have a line to exist on. I understand this being a standard, but your implication was that not doing things this way was "insane". really it just seems to make a lot of situations invalid that could just as easily have been valid, allowing applications that process files to behave poorly when given an unnecessarily "invalid" text file. – Burne

@Sahuagin: maybe "sane" was a harsh word, but my point is: if there is a convention formally defined by a standard, there's no reason not to adopt it. And, using the robustness principle "be conservative in what you send, be liberal in what you accept", tools that process text (parsers, compilers, filters) should be able to handle both cases whenever possible, but text editors should try to create files with the terminating EOL for each line, including the last one. – Cabretta

Answer 13

There's also a practical programming issue with files lacking newlines at the end: The read Bash built-in (I don't know about other read implementations) doesn't work as expected:

printf $'foo\nbar' | while read line
do
    echo $line
done

This prints only foo! The reason is that when read encounters the last line, it writes the contents to $line but returns exit code 1 because it reached EOF. This breaks the while loop, so we never reach the echo $line part. If you want to handle this situation, you have to do the following:

while read line || [ -n "${line-}" ]
do
    echo $line
done < <(printf $'foo\nbar')

That is, do the echo if the read failed because of a non-empty line at end of file. Naturally, in this case there will be one extra newline in the output which was not in the input.

Sherise

Answer 14

Why should (text) files end with a newline?

As well expressed by many, because:

1. Many programs do not behave well, or fail without it.

2. Even programs that well handle a file lack an ending '\n', the tool's functionality may not meet the user's expectations - which can be unclear in this corner case.

3. Programs rarely disallow final '\n' (I do not know of any).

---

Yet this begs the next question:

What should code do about text files without a newline?

1. Most important - Do not write code that assumes a text file ends with a newline. Assuming a file conforms to a format leads to data corruption, hacker attacks and crashes. Example:

   // Bad code
   while (fgets(buf, sizeof buf, instream)) {
     // What happens if there is no \n, buf[] is truncated leading to who knows what
     buf[strlen(buf) - 1] = '\0';  // attempt to rid trailing \n
     ...
   }
   

2. If the final trailing '\n' is needed, alert the user to its absence and the action taken. IOWs, validate the file's format. Note: This may include a limit to the maximum line length, character encoding, etc.

3. Define clearly, document, the code's handling of a missing final '\n'.

4. Do not, as possible, generate a file the lacks the ending '\n'.

Thousandth

Answer 15

It's very late here but I just faced one bug in file processing and that came because the files were not ending with empty newline. We were processing text files with sed and sed was omitting the last line from output which was causing invalid json structure and sending rest of the process to fail state.

All we were doing was:

There is one sample file say: foo.txt with some json content inside it.

[{
    someProp: value
},
{
    someProp: value
}] <-- No newline here

The file was created in widows machine and window scripts were processing that file using PowerShell commands. All good.

When we processed same file using sed command sed 's|value|newValue|g' foo.txt > foo.txt.tmp

The newly generated file was

[{
    someProp: value
},
{
    someProp: value

and boom, it failed the rest of the processes because of the invalid JSON.

So it's always a good practice to end your file with empty new line.

Canine

echo -n foo | sed '{}' works fine with sed (GNU sed) 4.4 – Siderolite

end your file with empty new line? What is an empty new line? – Mortgage

Answer 16

I was always under the impression the rule came from the days when parsing a file without an ending newline was difficult. That is, you would end up writing code where an end of line was defined by the EOL character or EOF. It was just simpler to assume a line ended with EOL.

However I believe the rule is derived from C compilers requiring the newline. And as pointed out on “No newline at end of file” compiler warning, #include will not add a newline.

Djambi

Answer 17

Imagine that the file is being processed while the file is still being generated by another process.

It might have to do with that? A flag that indicates that the file is ready to be processed.

Norton

Answer 18

I personally like new lines at the end of source code files.

It may have its origin with Linux or all UNIX systems for that matter. I remember there compilation errors (gcc if I'm not mistaken) because source code files did not end with an empty new line. Why was it made this way one is left to wonder.

Gastrula

Answer 19

IMHO, it's a matter of personal style and opinion.

In olden days, I didn't put that newline. A character saved means more speed through that 14.4K modem.

Later, I put that newline so that it's easier to select the final line using shift+downarrow.

Finable