Make 'git diff' ignore ^M

Question

In a project where some of the files contain ^M as newline separators, diffing these files is apparently impossible, since git diff sees the entire file as just a single line.

How does one git diff when comparing the current and previous versions of a source code file?

Is there an option like "treat ^M as newline when diffing" ?

prompt> git-diff "HEAD^" -- MyFile.as 
diff --git a/myproject/MyFile.as b/myproject/MyFile.as
index be78321..a393ba3 100644
--- a/myproject/MyFile.cpp
+++ b/myproject/MyFile.cpp
@@ -1 +1 @@
-<U+FEFF>import flash.events.MouseEvent;^Mimport mx.controls.*;^Mimport mx.utils.Delegate
\ No newline at end of file
+<U+FEFF>import flash.events.MouseEvent;^Mimport mx.controls.*;^Mimport mx.utils.Delegate
\ No newline at end of file
prompt>

---

UPDATE:

I have written a Ruby script that checks out the latest 10 revisions and converts CR to LF.

require 'fileutils'

if ARGV.size != 3
  puts "a git-path must be provided"
  puts "a filename must be provided"
  puts "a result-dir must be provided"
  puts "example:"
  puts "ruby gitcrdiff.rb project/dir1/dir2/dir3/ SomeFile.cpp tmp_somefile"
  exit(1)
end

gitpath = ARGV[0]
filename = ARGV[1]
resultdir = ARGV[2]

unless FileTest.exist?(".git")
  puts "this command must be run in the same dir as where .git resides"
  exit(1)
end

if FileTest.exist?(resultdir)
  puts "the result dir must not exist"
  exit(1)
end
FileUtils.mkdir(resultdir)

10.times do |i|
  revision = "^" * i
  cmd = "git show HEAD#{revision}:#{gitpath}#{filename} | tr '\\r' '\\n' > #{resultdir}/#{filename}_rev#{i}"
  puts cmd 
  system cmd
end

Barefoot

you may have wanted to git diff -b - I showed this in https://mcmap.net/q/63505/-make-39-git-diff-39-ignore-m – Tumbleweed

With Git 2.16 (Q1 2018), you will have git diff --ignore-cr-at-eol. See my answer below. – Desulphurize

@JasonPyeron and for future Googlers: I had to look up that git diff -b is identical to git diff --ignore-space-change. – Benoit

Related: on Linux you can use the command-line utility dos2unix (and unix2dos) to convert between line-endings. – Curet

git config core.whitespace cr-at-eol alesub answer below should be the accepted answer. None of these other fixes work on latest MacOS. Other fixes do seem to work on older Mac OSes. – Thievish

Answer 1

GitHub suggests that you should make sure to only use \n as a newline character in git-handled repos. There's an option to auto-convert:

$ git config --global core.autocrlf true

Of course, this is said to convert crlf to lf, while you want to convert cr to lf. I hope this still works …

And then convert your files:

# Remove everything from the index
$ git rm --cached -r .

# Re-add all the deleted files to the index
# You should get lots of messages like: "warning: CRLF will be replaced by LF in <file>."
$ git diff --cached --name-only -z | xargs -0 git add

# Commit
$ git commit -m "Fix CRLF"

core.autocrlf is described on the man page.

Zlatoust

ouch.. will I have to do this for every revision of that file? – Barefoot

No, of course not, once the setting is there, it will silently convert upon commit. If everything works the way I think it does, that is … – Zlatoust

The problem is that I already have some files in the repository that have CRLF endings and others that doesn't. I suspect that Adobe Flash adds CRLF even though I'm using the Mac version. I need to compare against older revisions of these files. Converting line endings starting from now on does not solve the problem with older revisions :-/ – Barefoot

Well … you can write a script and change all the past commits :) It's not pretty, but you can re-write the history of your project. (At the risk of ruining your repo, that is) – Zlatoust

yeah, that is unfortunate. I actually have quite a few repositories and I have never needed to diff CRLF files before. I wish I had known about this option a long time ago. Thank you for enlighten me. – Barefoot

You're not working with CRLF files here, at least not in the example you posted. That's an old-style mac file (just uses \r for EOL). That's why the diff is being shown on one line. A file using dos EOL would show each line distinctly with a trailing ^M, which you could tell get to handle via git config core.whitespace cr-at-eol. – Maxama

@Maxama yeah I just realized that too.. some of the files uses only CR and others LF. There are no CRLF involved. So only the CR files are causing trouble. – Barefoot

If I may quote my original, unmodified answer: "Of course, this is said to convert crlf to lf, while you want to convert cr to lf. I hope this still works …" – Zlatoust

the git config core.whitespace cr-at-eol thing would work. and of course you can write your own commit-hook that transforms cr to lf. – Zlatoust

I'm trying this, but I keep getting warning: LF will be replaced by CRLF instead of warning: CRLF will be replaced by LF, and I'm in Linux. Any idea why? I want all to end with LF, not CRLF! – Radish

@trusktr, it happened the same to me. In linux, with accidental CRLF, use git config --global core.autocrlf input, do the steps in this answer(rm, add, commit), and you will get warning: CRLF will be replaced by LF. The file will have its original line endings in your working directory.. Remove the files (because they have the original, wrong CRLF) and checkout them again from the last "Fix CRLF" commit. – Unpaid

Once the git config setting has been updated, you can also refresh your local copy with rm .git/index && git reset command. – Tolerant

I was working on Windows and now I use Debian Jessie. For me, what the @Zlatoust mentioned helped me. – Dusky

This doesn't work for files with mixed line endings though. – Slav

Version control is not the place to solve this problem. Changing files on the fly adds even more complexity and even stranger corner cases. git even had to add a core.safecrlf and SAMBA never implemented anything like it. Use the newer cr-at-eol and don't go anywhere autocrlf. – Mccomb

If you have files that match your git ignore you need to use git add -f in xargs – Ansley

Answer 2

Developing on Windows, I ran into this problem when using git tfs. I solved it this way:

git config --global core.whitespace cr-at-eol

This basically tells Git that an end-of-line CR is not an error. As a result, those annoying ^M characters no longer appear at the end of lines in git diff, git show, etc.

It appears to leave other settings as-is; for instance, extra spaces at the end of a line still show as errors (highlighted in red) in the diff.

(Other answers have alluded to this, but the above is exactly how to set the setting. To set the setting for only one project, omit the --global.)

EDIT:

After many line-ending travails, I've had the best luck, when working on a .NET team, with these settings:

• NO core.eol setting
• NO core.whitespace setting
• NO core.autocrlf setting
• When running the Git installer for Windows, you'll get these three options:
  • Checkout Windows-style, commit Unix-style line endings <-- choose this one
  • Checkout as-is, commit Unix-style line endings
  • Checkout as-is, commit as-is

If you need to use the whitespace setting, you should probably enable it only on a per-project basis if you need to interact with TFS. Just omit the --global:

git config core.whitespace cr-at-eol

If you need to remove some core.* settings, the easiest way is to run this command:

git config --global -e

This opens your global .gitconfig file in a text editor, and you can easily delete the lines you want to remove. (Or you can put '#' in front of them to comment them out.)

Entanglement

For those who find this now, it's worth noting that Checkout Windows-style, commit Unix-style line endings auto-sets core.autocrlf to true – Leticialetisha

Note that the line git config --global core.whitespace cr-at-eol would turn off other settings that are default. There are three defaults: blank-at-eol, blank-at-eof and space-before-tab. So to enable cr-at-eol while keeping the others you would need to use git config --global core.whitespace blank-at-eol,blank-at-eof,space-before-tab,cr-at-eol. – Baillargeon

For my project (it was checkout out on Windows and I'm viewing it on Linux), cr-at-eol got rid of ^M at the end of lines in git diff all right, but GIT still showed those lines as different, although the line ending was the only difference. – Chiller

SourceInsight keep pushing the ^M character, and git still shows the difference at line endings. @Zitrax's command is the right answer to my case, git diff show nice and clean output. – Cutaway

@Zitrax's comment is false; the default value of core.whitespace is "". If you wanted to, say, enable cr-at-eol and disable blank-at-eol, you would need to set core.whitespace to "cr-at-eol -blank-at-eol". See also git help config. – Coacher

I think git needs a bit more complexity, a few more conflicting settings for end of line. I think git should be more concerned about my whitespaces. For example throw an unrelated fatal error and leave the repository in a corrupt state when encountering Mac line endings on a Windows (but not Linux) machine. I mean why would I use a VCS that would mind it's business and let me use whichever line endings I want? I see they're trying, but they should throw in half a dozen more line-ending behaviors, to solve a problem that doesn't exist. They're almost there! Keep it up. – Hair

super valuable. I spent a long time fooling with editor settings in VSCode and Sublime text trying to get them to stop fooling with line-endings. I finally used a hex-viewer to convince myself that the line-ending on the line I was editing perfectly matched the original version, which let me know the display of the ^M was coming from git diff behavior and didn't represent an actual change to the line ending. – Parks

If you need to remove some core.* settings, the easiest way is to run this command: git config --global -e. similarly, if you need to edit the current repo's settings, use git config -e – Seaden

Answer 3

Try git diff --ignore-space-at-eol, or git diff --ignore-space-change, or git diff --ignore-all-space.

Gonsalves

None of that really affects the character that identifies the newline. – Zlatoust

I also tried with "-w" but no luck, still treats it as a single line. Next project I must remember to never ever get any CR into the source code. – Barefoot

Just remember git config --global core.autocrlf true, or bug the git folks until they make it default :) – Zlatoust

This solved my problem without having to change my autocrlf settings. Thanks! – Fennel

grr.. none of the 3 worked for me. I am using git on Windows 7 on files with \r line separator character – Muley

these flags have no effect for me... still shows ^M as diffs – Barmy

@Magnus: these flags are about what Git considers a difference. Their use makes it that Git doesn't show whole file as changed if end-of-line convention changed. This is not an output filter - if there is change, it would be shown with ^M. You can try to either configure less, or change Git pager. HTH – Blockage

When I diff --stat two branches to see which files have changed, files where ^M is the only difference still appear on the list – Barmy

@Magnus: Do you see the difference between git diff -w --stat and git diff --stat? The former should not show any changes if the difference is only in ^M (i.e. CR LF vs LF). – Blockage

OK I see what's happening. -w --stat and --ignore-all-space --stat yield identical diff results, with an identical file count to a naked --stat. But I realized that all the files coincidentally ALSO happen to have one or two legitimate diffs mixed in amidst the hundreds of meaningless ^M differences, thus they still appear on the list! Makes sense really since presumably the Windows user had an actual change to check in when the commit was made. Thanks Jakub for helping me pinpoint this. – Barmy

after using git diff --ignore-space-at-eol , all the irritating line endings changes that wre previosly visible with git diff commands were gone. However, git-diff shows only the way the files differ. During commit i.e git status , one still sees the files with Ctrl-M as a changed file. Under unix, one can change the line ending using dos2unix command. find ./ -name . | xargs dos2unix – Razzledazzle

Worked great for me, but I was just trying to get diffs that included real code changes and not the added carriage returns. – Destructive

Can I use all three flags? – Matthieu

Answer 4

Also see:

core.whitespace = cr-at-eol

or equivalently,

[core]
    whitespace = cr-at-eol

where whitespace is preceded by a tab character.

Puryear

Yep, this made the git diff tool (also used in git show) stop bugging me about the ^Ms on the changed lines! :) – Guttle

for whatever reason this did not work for me. Tried it both with = and no = sign. git diff still shows ^M characters. – Muley

Two ways to do this: one, add the line above verbatim to your .gitconfig either in .git/config, or in ~/.gitconfig; two, git config --global core.whitespace cr-at-eol (where --global is optional if you just want it on the repo you're on) – Leticialetisha

This worked for me on Windows 7, although I just put it under [core] so I can replace the core. prefix with a TAB character. – Electronarcosis

This question was above how to hide ^M in git diff, not about how to not put in ^M in the first place. That means the accepted answer of changing core.autocrlf is not the best because it silently alters the files without user's confirmation. – Roguery

Answer 5

Why do you get these ^M in your git diff?

In my case I was working on a project which was developed in Windows and I used Linux. When I changed some code, I saw ^M at the end of the lines I added in git diff. I think the ^M were showing up because they were different line endings than the rest of the file. Because the rest of the file was developed in Windows it used CRLF line endings, and in Linux it uses LF line endings.

Apparently, the Windows developer didn't use the option "Checkout Windows-style, commit Unix-style line endings" during the installation of Git.

So what should we do about this?

You can have the Windows users reinstall git and use the "Checkout Windows-style, commit Unix-style line endings" option. This is what I would prefer, because I see Windows as an exception in its line ending characters and Windows fixes its own issue this way.

If you go for this option, you should however fix the current files (because they're still using the CRLF line endings). I did this by following these steps:

1. Remove all files from the repository, but not from your filesystem.

    git rm --cached -r .
   

2. Add a .gitattributes file that enforces certain files to use a LF as line endings. Put this in the file:

    * text=auto eol=lf
   

3. Add all the files again.

    git add .
   

   This will show messages like this:

    warning: CRLF will be replaced by LF in <filename>.
    The file will have its original line endings in your working directory.
   

4. You could remove the .gitattributes file unless you have stubborn Windows users that don't want to use the "Checkout Windows-style, commit Unix-style line endings" option.

5. Commit and push it all.

6. Remove and checkout the applicable files on all the systems where they're used. On the Windows systems, make sure they now use the "Checkout Windows-style, commit Unix-style line endings" option. You should also do this on the system where you executed these tasks because when you added the files git said:

    The file will have its original line endings in your working directory.
   

   You can do something like this to remove the files:

    git ls | grep ".ext$" | xargs rm -f
   

   And then this to get them back with the correct line endings:

    git ls | grep ".ext$" | xargs git checkout
   

   Replacing .ext with the file extensions you want to match.

Now your project only uses LF characters for the line endings, and the nasty CR characters won't ever come back :).

The other option is to enforce windows style line endings. You can also use the .gitattributes file for this.

More info: https://help.github.com/articles/dealing-with-line-endings/#platform-all

Hedi

To fix all line endings in a specific file, if using Sublime Text, you can go to View -> Line Endings and click on Unix. – Saturnian

What exactly does this ^M means? Is it a windows or linux newline? Or is it just a "different" newline compared to the other newlines in the file? – Hetman

Good one, I think it's just a "different" newline (different than most others) – Hedi

-1 as reinstalling git to accomplish git config --global core.autocrlf true is overkill, and the anti-Windows/anti-CR campaign seems tangential to the question. – Schizothymia

should'nt it be *.ext text eol=lf inplace of of crlf ? I think that was a typo, that noone noticed ! – Razzledazzle

You state that Windows use CR but actually it use CRLF. Files with CR only at e.o.l. are from mac os 9 and prior. en.wikipedia.org/wiki/Newline – Cheer

Answer 6

Is there an option like "treat ^M as newline when diffing" ?

There will be one with Git 2.16 (Q1 2018), as the "diff" family of commands learned to ignore differences in carriage return at the end of line.

See commit e9282f0 (26 Oct 2017) by Junio C Hamano (gitster).
Helped-by: Johannes Schindelin (dscho).
^{(Merged by Junio C Hamano -- gitster -- in commit 10f65c2, 27 Nov 2017)}

diff: --ignore-cr-at-eol

A new option --ignore-cr-at-eol tells the diff machinery to treat a carriage-return at the end of a (complete) line as if it does not exist.

Just like other "--ignore-*" options to ignore various kinds of whitespace differences, this will help reviewing the real changes you made without getting distracted by spurious CRLF<->LF conversion made by your editor program.

Desulphurize

@kaartic Thank you for editing the answer and referencing the right commit! – Desulphurize

Whilst it's generally good practice to set git config --global core.autocrlf true as in the accepted answer, this answers the OP's question more directly: 'Is there an option like "treat ^M as newline when diffing" ?' – Elly

As of Git 2.20 this doesn't hide ^M's – Labile

@user1944491 I didn't notice any regression, meaning it sill ignores eol when diffing with this option in Git 2.26. – Desulphurize

@Desulphurize Using this argument in the git diff command didn't work. Nor did setting my core.whitespace value on git version 2.20.1 (Apple Git-117) but adding Jason Pyeron's core.pager answer fixed it. YMMV obviously. – Labile

This does not invoke git config and benefit people using git diff --no-index. – Tachygraphy

Answer 7

In my case, what did it was this command:

git config  core.whitespace cr-at-eol

Source: https://public-inbox.org/git/[email protected]/T/

Isaacs

git config core.whitespace cr-at-eol this should be the accepted answer. None of these other fixes work on latest MacOS. Other fixes do seem to work on older Mac OSes. – Thievish

I chose to apply it only for my local environment with: git config --local core.whitespace cr-at-eol I am on macOS Ventura 13.4.1. This suggestion worked well for me. The suggestions given in other comments had no effect. – Oppose

Answer 8

TL;DR

Change the core.pager to "tr -d '\r' | less -REX", not the source code

This is why

Those pesky ^M shown are an artifact of the colorization and the pager. It is caused by less -R, a default git pager option. (git's default pager is less -REX)

The first thing to note is that git diff -b will not show changes in white space (e.g. the \r\n vs \n)

setup:

git clone https://github.com/CipherShed/CipherShed
cd CipherShed

A quick test to create a unix file and change the line endings will show no changes with git diff -b:

echo -e 'The quick brown fox\njumped over the lazy\ndogs.' > test.txt
git add test.txt
unix2dos.exe test.txt
git diff -b test.txt

We note that forcing a pipe to less does not show the ^M, but enabling color and less -R does:

git diff origin/v0.7.4.0 origin/v0.7.4.1 | less
git -c color.ui=always diff origin/v0.7.4.0 origin/v0.7.4.1 | less -R

The fix is shown by using a pipe to strip the \r (^M) from the output:

git diff origin/v0.7.4.0 origin/v0.7.4.1
git -c core.pager="tr -d '\r' | less -REX"  diff origin/v0.7.4.0 origin/v0.7.4.1

An unwise alternative is to use less -r, because it will pass through all control codes, not just the color codes.

If you want to just edit your git config file directly, this is the entry to update/add:

[core]
        pager = tr -d '\\r' | less -REX

Tumbleweed

I had this problem in a repo where some of the files had \r\n line endings and some had \n line endings (I don't know if that's relevant); diffs of the former showed the ^M in the modified lines (that is, the + lines). core.autocrlf was set to true. Running git config core.pager "tr -d '\r' | less -REX" got rid of the pesky ^Ms. Thanks! – Daggerboard

Thanks for this. This is the only answer if you must work with differing line endings in your repo(s) -- e.g. you use checkout as-is, commit as-is, purposefully. – Appear

git diff -b is what I was looking for, but I do appreciate the thorough explanation. – Succory

Yes! Of all the answers to this question, modifying the git "config" file's [core] section by adding pager = tr -d '\\r' | less -REX was the only answer that worked for me. Thank you! – Gotama

git diff -b did not work, but the config modification did. – Labile

I know we aren't supposed to say thank you, but THANK YOU. This restores some sanity to my git diff. May I never see another ^M again! – Langton

Answer 9

I struggled with this problem for a long time. By far the easiest solution is to not worry about the ^M characters and just use a visual diff tool that can handle them.

Instead of typing:

git diff <commitHash> <filename>

try:

git difftool <commitHash> <filename>

Compendious

Thanks! Also I just ran "git difftool" and it basically compared all changed files in a loop – Ike

like, what tool can handle it? – Rhpositive

Answer 10

If you just want a quick line that makes the git diff but does not show the different endings (thus the ^M) use the one in the first comments to the original question, it worked for me:

 git diff -b

Take into account that, in the long run, you should get your line endings configuration right, as all other answers suggest.

Charteris

Answer 11

As noted by VonC, this has already been included in git 2.16+. Unfortunately, the name of the option (--ignore-cr-at-eol) differs from the one used by GNU diff that I'm used to (--strip-trailing-cr).

When I was confronted with this problem, my solution was to invoke GNU diff instead of git's built-in diff, because my git is older than 2.16. I did that using this command line:

GIT_EXTERNAL_DIFF='diff -u --strip-trailing-cr "$2" "$5";true;#' git diff --ext-diff

That allows using --strip-trailing-cr and any other GNU diff options.

There's also this other way:

git difftool -y -x 'diff -u --strip-trailing-cr'

but it doesn't use the configured pager settings, which is why I prefer the former.

Baltimore

Answer 12

If the git patch is already generated in a windows machine and you are using it you can format the patch with dos2unix utility in Linux.

find -name "*.patch"| xargs dos2unix

This will solve the ^M at EOL and you will be able to git apply patch in your linux machine.

Apyretic

Answer 13

Combine the core.autocrlf=true setting with the --ignore-space-at-eol paramerer to ignore line ending changes:

git -c "core.autocrlf=true" diff --ignore-space-at-eol

Bruise