Why do gedit and vim hide the final newline from the user?

Question

Let's assume we have two text documents:

1. Our first file contains "hi" as text.
2. Our second file contains "hi" as text.

When we open these two files in gedit, vi, or vim, the two files are visually identical in every way.

However, when we run xxd on the files, we get the following:

1. Hex content of our first file reads: 6869
2. Hex content of our second file reads: 6869 0a

Aha! There's an invisible newline. In vim, if we were paying close enough attention to the status bar and happen to understand what [noeol] means, then we might pick up on this, but in gedit, the two files open exactly the same!

In a small survey, when I asked people to distinguish between the two files using only gedit or vim, they failed 100% of the time. When I asked them to accomplish the same task using leafpad or emacs, they succeeded 100% of the time.

I understand that vi and gedit want to add a newline to every file they create (and I acknowledge that there are probably advantages to this). What I don't understand is why gedit and vim think visually hiding this newline from their users is beneficial? Especially when this kind of behavior is potentially extremely destructive...

(Take for instance the two C-programmers who see the contents of these two files the same in their vi/gedit text editor, and then assuming what they see is what they get, go on to write the contents into the array char greeting[2]. The first programmer writing the first file - although a bit sloppy with his code - goes on to fame and fortune, but the second programmer writing the second file, dies in miserable poverty, confused and bewildered by this invisible (and preventable) stack overflow.)

So pray tell, what are the advantages of having text editors like vim and gedit add invisible newlines at the end of each document they create, and then proceed to hide those newlines from the user, so that the true contents of these files are only visibly detectable using other text editors?

Permenter

Naturally, Emacs will add a final newline if you want it to with variable `require-final-newline'. I think there is a mode-specific way to do it too. It does seem rather presumptuous of a general-purpose editor to add bytes to a file you didn't ask it to, but then if Vim is only intended to be used on Unix systems for writing particular types of files, then perhaps it's not fair to call it a general-purpose editor. – Corkboard

Answer 1

That you perceive the last line as "visually hidden" stems from the opposite viewpoints.

Many people (especially with a Unix background) argue that text files should always end with a newline (e.g. see Why should text files end with a newline?), and that any text editor that allows to create files without it is fundamentally broken. From that perspective, that there is no additional empty line in Vim is only consistent with this world view. The file appears in the editor as it would when it is cated into the terminal.

From Vim's point of view, a file with a missing newline is heretical and defective; that's why it is indicated with the [noeol] message on load (and it is quite hard to edit the file while keeping the missing newline; though I've written the PreserveNoEOL plugin to aid with that).

Giselagiselbert

Coming from a Windows background, your answer helps clarify unix text editor/text file expectations, thank you. However I don't think "visually hidden" can be dismissed as merely a viewpoint (i.e. if two different text files display identically to the eye of a user, then there is something - objectively speaking - "visually hidden"). If vi displayed an unmodifiable blank line at the end of every file it creates, it would meet your unix text file/text editor expectations, while also providing a contrasting visual representation for the user who encounters text files with no final newline. – Permenter

@schulwitz, why should Vim display a line that is not there as if it was? What about fixing all the wrong interpretations of the EOL character instead? And the user can perfectly see if the last line doesn't end with an EOL with [noeol] in the status line. – Rabi

Thanks. From the "Unix perspective", these two files are practically identical (and they will be after the first :write command). In that point of view, it's simply not worthwhile to care about the differences. You're right though that this causes friction when interacting with the "other world". – Giselagiselbert

@Rabi "the user can perfectly see if the last line doesn't end with an EOL with [noeol] in the status line." Exactly my point, if the only thing that visually distinguishes one file from another is to be found in the status bar, your visual design has failed to convey the appropriate meaning to the user. – Permenter

I don't agree. The last line of a text file is supposed to end with an EOL character: it's normal and it makes sense. What doesn't make sense is displaying an empty line after the very last character of the file and Vim correctly avoids doing that. The presence of an EOL is normal and expected so there's nothing special to display! When it encounters a file without an EOL, Vim helpfully warns you. How would you indicate that there's no EOL? A symbol at the end of the last line? What if that last line is #4325 and you can't see it? The statusline is obviously the only sensible solution. – Rabi

@romaini The only reason you wouldn't be able to see line #4325 is if you weren't trying to look at it (i.e. if we weren't using vi's visual interface to see the eol), so there's no reason to worry about it. And btw, I'm not against [noeol] in the status bar, but by no means is it the only sensible solution, just one of them. Your suggestion of a symbol would work nicely, a red ~ (to contrast the trailing blue ones) would do the trick nicely IMO. – Permenter

Answer 2

I believe this is a convention, that is carried over from vi. One advantage of doing so, is that when concatenating files, you can still distinguish where each file ended.

Some more context is provided in this thread on the vim_use mailinglist

Untraveled

I understand (and acknowledge) that there are advantages to an ending newline. What I don't understand is why this is visually hidden from the user? (i.e. In vim, the final newline has no visual representation, instead a ~ appears, as if there's nothing left in the file. However, literally every other newline in vim has a blank line to indicate its presence, why not the final one?) – Permenter

@schulwitz, that empty line is not displayed because it isn't there. The EOL character is usually interpreted in two ways: "anything after me must be considered to be on a new line" and "after me comes a new line". Most GUI text editors/IDEs, with or without a UNIX background, follow the first one (the wrong one): what comes after EOL is considered to be a line and the editor displays it as such, even if the logical definition of a "line" is not met. Most UNIX text editors follow the second: because there's nothing at all after the last EOL there's nothing to show. – Rabi

I think that is just a different view of concepts. In the Unix World, the final newline is considered a line terminator while in the Windows World one seems to consider it a line separator Therefore looking from the line terminator view point, there simply is no final new line that is hidden from the user. – Untraveled

I like your response, Christian, which I would rephrase as: In Vim's conceptual view of the EOL character, when there is no EOL at the end of the file, then there is one incomplete line only, because it has no EOL. With the EOL, there is still only one line, and it is properly terminated. In other editors' view, when there is no EOL at the end of the file, then there is one line in the file and there is nothing special about that line. When there is an EOL at the end of a file, there are two lines separated by the EOL and the last line is empty. – Merrymerryandrew

Answer 3

You can edit the Gedit setting that hide the newline (ensure-trailing-newline) and make it visible using the following command:

gsettings set org.gnome.gedit.preferences.editor ensure-trailing-newline false

Kerplunk