Convert DOS/Windows line endings to Linux line endings in Vim

Question

If I open files I created in Windows, the lines all end with ^M.
How do I delete these characters all at once?

Jacobi

If you do a hexdump -C badfile and see 0x0d 0x0a "\r\n" that is your problem. – Whitt

Worth comparing with - What is ^M and how do I get rid of it? – Headman

Answer 1

dos2unix is a commandline utility that will do this.

In Vim, :%s/^M//g will if you use Ctrl-v Ctrl-m to input the ^M (On Windows, use Ctrl-q Ctrl-m instead).

Vim may not show the ^M characters if the file is loaded with a dos (Windows) file format. In this case you can :set ff=unix and then save the file with :w or :w! and Vim will reformat the file as it saves it for you.

There is documentation on the fileformat setting, and the Vim wiki has a comprehensive page on line ending conversions.

Alternately, if you move files back and forth a lot, you might not want to convert them, but rather to do :set ff=dos, so Vim will know it's a DOS file and use DOS conventions for line endings.

Inject

That was it for me thank you pal :> Bash wouldn't launch the script because of line endings. Stupid notepad really... – Mayhew

:%s/^M//g should be :%s/\r//g, because ^M just means "match capital "M" at the beginning of the line". – Efficacy

Not if you do as the answer says and 'use ctrl-v ctrl-m to input the ^M'. – Inject

crtl-v is no good, on windows it pastes clipboard contents to the command line. Solution :%s/\r//g worked for me, cheers @Efficacy – Knawel

I got confused when I tried this, because it seemed to take ages to do the conversion, when in fact it was just waiting for user input. – Barocchio

@ropata What you want on Windows is ctrl-q. – Fini

dos2unix might be the better solution but only if it's present on the system. I don't think we should lead the answer with it. set ff=unix seems the best answer. – Barocchio

I must be missing something, because set ff=unix does nothing. Maybe it converts the file, but all of the ^M characters are still there. – Festival

If you want to ensure you keep the new lines its best to actually do this :%s/\r/\r/g unix/vim will put in the right newline char with the replace \r – Helpmeet

both ^M (Ctrl+V Ctrl+M) and \r are not found in my file but I definitely see them in diff (VIM - Vi IMproved 7.4) – Forenamed

:%s/\r//g or :%s/^M//g does not work. :set ff=unix is work as desired. But It formats whole file, Not only those lines which contain ^m – Moa

There is good source for change or convert to and from Windows to UNIX. – Moa

@Festival - :set ff=unix then you :wq the file to write out the new format. I just did this two minutes ago to clean a file I got from a developer. – Banian

Use the vim set command to show the file format :set ff?. The command returns fileformat=<dos/unix/mac> to indect current file format. – Orella

when you run :set ff=unix you still need to write the changes to the file with :w or :x – Snot

@Barocchio I agree that :set ff=unix is the best, but although I now have edit privileges, that seems like an edit to leave to the original answer's discretion. – Feathercut

Answer 2

Change the line endings in the view:

:e ++ff=dos
:e ++ff=mac
:e ++ff=unix

This can also be used as saving operation (:w alone will not save using the line endings you see on screen):

:w ++ff=dos
:w ++ff=mac
:w ++ff=unix

And you can use it from the command-line:

for file in *.cpp
do 
    vi +':w ++ff=unix' +':q' "$file"
done

Yahiya

Thank you very much. I tried vi +':wq ++ff=unix' <filepath> for a lazy one like me. – Flan

This should be the most upvoted answer. :w +ff=unix is so much nicer than most of the other stuff written here, and the bash script is a nice bonus. – Juliusjullundur

The :e commands don't appear to do anything on my Windows vim install. My view still shows the ^M codes. – Weber

I read the :e ++ff=unix everywhere but the simplest for me that worked is :w ++ff=unix then quit & reopen file. – Mansuetude

@OlivierPons but :e does the same thing as reopening the file? How is adding more unnecessary steps to your solution making it simpler? – Greaten

Well changing the line ending in the view does not do what he asked, which is removing the ^M. – Nidify

Answer 3

I typically use

:%s/\r/\r/g

which seems a little odd, but works because of the way that Vim matches linefeeds. I also find it easier to remember :)

Slightly

This works consistently across platforms. The best answer here. – Agnesse

I've never had any problem with :set ff=unix before, but the file I opened today must have been particularly weird. Vim said it was already fileformat=unix but all the line endings were ^M. This solution worked for me. – Caricature

This solution adds unwanted extra lines for me, doubling the number of lines in the file. :%s/\r//g instead works for me. – Greaten

Victor, your files likely have \r\n endings. the \r isn't read as a newline but the \n is. In the files I'm running into are \r and you have to add a newline character. – Halvorsen

astyle, dos2unix, vim fileformat, notepad, ultra-edit, etc. all didn't work; this did! Previous to trying this I had to open the file in Wordpad, paste into notepad and then paste into ultraedit before I could go back to VIM. Someone put varying types of linefeed types and EOF in files which then made it hard to find where function calls were in sources. grep returned junk for the file with mixed linefeed types and EOF. Disappointed that other methods didn't work. Thanks for a simple solution! – Willette

I tried the other solutions here, but this is the only one that worked for me. I had imported a CSV file into Excel (Mac OS X), and after transforming it a bit, exported it as a CSV. It ended up being a one-line file with ^M separating what were the original lines. – Haematoxylin

I’m not vim-ish enough to understand why, but this solution works for me in circumstances where the accepted answer failed – Fanfare

When you do a hexdump -C badfile you can see the hex sequence 0x0d 0x0a which is your problem. – Whitt

@VictorZamanian's :%s/\r//g is the only general-purpose solution – especially for mixed-mode files containing a heterogeneous admixture of both DOS- and UNIX-style newlines. The canonical solutions (e.g., :set ff=unix, :e ++ff=unix) assume every line of the current buffer ends in the same newline style. Sometimes they do; sometimes they don't. Cue sadface. – Suited

@VictorZamanian's, this method should help solve the problem with adding new lines. – Greengrocery

@VictorS. ? I didn't have a problem adding newlines. I got extra newlines I didn't want. And if you read my comment to the end you'll see I had a solution already. Thanks though. – Greaten

Answer 4

I prefer to use the following command:

:set fileformat=unix

You can also use mac or dos to respectively convert your file to Mac or MS-DOS/Windows file convention. And it does nothing if the file is already in the correct format.

For more information, see the Vim help:

:help fileformat

Purifoy

This command doesn't appear to do anything on my Windows vim. My view still has ^M chars in it. – Weber

Adding set fileformat=unix to my .vimrc file worked. – Loomis

Answer 5

:set fileformat=unix to convert from DOS to Unix.

Oster

This actually resolved the issue for me. I wasn't able to find those characters while searching. – Spank

Answer 6

In VIM:

:e ++ff=dos | set ff=unix | w!

In shell with VIM:

vim some_file.txt +'e ++ff=dos | set ff=unix | wq!'

e ++ff=dos - force open file in dos format.

set ff=unix - convert file to unix format.

Greengrocery

Thank you. I managed to convert a file to the wrong format and couldn't use Git to undo it. This worked perfectly. – Hospitium

Answer 7

:%s/\r\+//g

In Vim, that strips all carriage returns, and leaves only newlines.

Opt

For some reason above didn't work for me under windows gvim. But when changed to :%s/\r//g it worked like a charm. – Cocklebur

@Cocklebur :%s/\r\+$//g – Bushhammer

Answer 8

From: File format

[Esc] :%s/\r$//

Amphiarthrosis

+1 for pointing to the official doc site. For anyone using the above link, see the section "Converting the current file" on that page. – Hunter

This + vim -b <filename> worked like a charm, thanks, +1 – Subscapular

The link is half-broken. It redirects to another page. – Abib

Answer 9

tr -d '\15\32' < winfile.txt > unixfile.txt

(See: Convert between Unix and Windows text files)

Danged

That seems to be a more universal answer; it works for me on FreeBSD out of the box (on that machine, vi seems to be truly ancient version...). Yes, vim is also installed on that machine; I should have tried that...whatever. – Elongate

Answer 10

To run directly in a Linux console:

vim file.txt +"set ff=unix" +wq

Carvel

Answer 11

Convert directory of files from DOS to Unix

Using command line and sed, find all files in current directory with the extension ".ext" and remove all "^M"

@ https://gist.github.com/sparkida/7773170

find $(pwd) -type f -name "*.ext" | while read file; do sed -e 's/^M//g' -i "$file"; done;

Also, as mentioned in a previous answer, ^M = Ctrl+V + Ctrl+M (don't just type the caret "^" symbol and M).

Olen

Answer 12

dos2unix can directly modify the file contents.

You can directly use it on the file, without any need for temporary file redirection.

dos2unix input.txt input.txt

The above uses the assumed US keyboard. Use the -437 option to use the UK keyboard.

dos2unix -437 input.txt input.txt

Mcmanus

Answer 13

The following steps can convert the file format for DOS to Unix:

:e ++ff=dos     Edit file again, using dos file format ('fileformats' is ignored).[A 1]
:setlocal ff=unix     This buffer will use LF-only line endings when written.[A 2]
:w     Write buffer using Unix (LF-only) line endings.

Reference: File format

Majolica

The link is half-broken. It redirects to another page. – Abib

Answer 14

I found a very easy way: Open the file with nano: nano file.txt

Press Ctrl + O to save, but before pressing Enter, press: Alt+D to toggle between DOS and Unix/Linux line-endings, or: Alt+M to toggle between Mac and Unix/Linux line-endings, and then press Enter to save and Ctrl+X to quit.

Carton

done this on a linux server, the file had ^M endings. save as dos to keep, save as linux to remove. can check using cat -v – Sandell

This question is about vim, not nano. – Weber

Answer 15

With the following command:

:%s/^M$//g

To get the ^M to appear, type CtrlV and then CtrlM. CtrlV tells Vim to take the next character entered literally.

Unicycle

Answer 16

The comment about getting the ^M to appear is what worked for me. Merely typing "^M" in my vi got nothing (not found). The CTRL+V CTRL+M sequence did it perfectly though.

My working substitution command was

:%s/Ctrl-V Ctrl-M/\r/g

and it looked like this on my screen:

:%s/^M/\r/g

Nightshirt

Answer 17

:g/Ctrl-v Ctrl-m/s///

CtrlM is the character \r, or carriage return, which DOS line endings add. CtrlV tells Vim to insert a literal CtrlM character at the command line.

Taken as a whole, this command replaces all \r with nothing, removing them from the ends of lines.

Labdanum

Elaborated in Dannid's answer. – Abib

Answer 18

You can use the following command:
:%s/^V^M//g
where the '^' means use CTRL key.

Snakemouth

Answer 19

You can use:

vim somefile.txt +"%s/\r/\r/g" +wq

Or the dos2unix utility.

Ryurik

Answer 20

In Vim, type:

:w !dos2unix %

This will pipe the contents of your current buffer to the dos2unix command and write the results over the current contents. Vim will ask to reload the file after.

Stepdame

Answer 21

The below command is used for reformating all .sh file in the current directory. I tested it on my Fedora OS.

for file in *.sh; do awk '{ sub("\r$", ""); print }' $file >luxubutmp; cp -f luxubutmp $file; rm -f luxubutmp ;done

Kellene

Answer 22

From Wikia:

%s/\r\+$//g

That will find all carriage return signs (one and more reps) up to the end of line and delete, so just \n will stay at EOL.

Apoloniaapolune

Answer 23

This is my way. I opened a file in DOS EOL and when I save the file, that will automatically convert to Unix EOL:

autocmd BufWrite * :set ff=unix

Demonstration

Answer 24

I wanted newlines in place of the ^M's. Perl to the rescue:

perl -pi.bak -e 's/\x0d/\n/g' excel_created.txt

Or to write to stdout:

perl -p -e 's/\x0d/\n/g' < excel_created.txt

Norway

Answer 25

I knew I'd seen this somewhere. Here is the FreeBSD login tip:

Do you need to remove all those ^M characters from a DOS file? Try

tr -d \\r < dosfile > newfile
    -- Originally by Dru <[email protected]>

Predigestion

Answer 26

This is a little more than you asked for but:

nmap <C-d> :call range(line('w0'),line('w$'))->map({_,v-> getline(v)})->map({_,v->trim(v,join(map(range(1,0x1F)+[0xa0],{n->n->nr2char()}),''),2)})->map({k,v->setline(k+1,v)})<CR>

Run this and :set ff=unix|dos and no more need for unix2dos.

• the single arg form of trim() has the same default mask above, plus 0X20 (an actual space) instead of 0x1F
• that default mask clears out all non-printing chars including non-breaking spaces [0xa0] that are hard to find
• create a list of lines from the range of lines
• map that list to the trim function with using the same mask code as the source, less spaces
• map that again to setline to replace the lines.
• all :set fileformat= does at this point is choose which eol to save it with, dos or unix
• it should be pretty easy to change the range of characters above if you want to eliminate or add some

Cabal

Answer 27

To delete these DOS/Windows line endings characters all at once, regardless of where they occur in a line (this is not a good idea if two lines were separated only by a CR because the command joins the lines together):

:%s/\r//g

Reference: File format -> 6.Removing unwanted CR or LF characters.

But, You could choose to convert these DOS/Windows line endings into Unix.

That means, to convert the current file from any mixture of CRLF/LF-only line endings, so all lines end with LF only:

:update              Save any changes. 
:e ++ff=dos          Edit file again, using dos file format ('fileformats' is ignored).
:setlocal ff=unix    This buffer will use LF-only line endings when written.
:w                   Write buffer using unix (LF-only) line endings. 

Reference: File format -> 3.Converting the current file.

In the above, replacing :set ff=unix with :set ff=mac would write the file with mac (CR-only) line endings. Or, if it was a mac file to start with, you would use :e ++ff=mac to read the file correctly, so you could convert the line endings to unix or dos.

More reference:

1.Configure line separators -> Change line separators for the current file(IntelliJ IDEA).

2.Configure line separators -> Configure line separators for new files(IntelliJ IDEA).

3.Vim Tips Wiki -> File format.

Frawley

Answer 28

Usually there is a dos2unix command you can use for this. Just make sure you read the manual as the GNU and BSD versions differ on how they deal with the arguments.

BSD version:

dos2unix $FILENAME $FILENAME_OUT
mv $FILENAME_OUT $FILENAME

GNU version:

dos2unix $FILENAME

Alternatively, you can create your own dos2unix with any of the proposed answers here, for example:

function dos2unix(){
    [ "${!}" ] && [ -f "{$1}" ] || return 1;

    { echo ':set ff=unix';
      echo ':wq';
    } | vim "${1}";
}

Woodrum

Answer 29

If you create a file in Notepad or Notepad++ in Windows, bring it to Linux, and open it by Vim, you will see ^M at the end of each line. To remove this,

At your Linux terminal, type

dos2unix filename.ext

This will do the required magic.

Waitress

dos2unix is not installed by default on e.g. Ubuntu 20.04 (Focal Fossa). – Abib

Much older answers already cover this solution in detail. – Maximinamaximize