Trying to fix line-endings with git filter-branch, but having no luck

Q

9

287

I have been bitten by the Windows/Linux line-ending issue with git. It seems, via GitHub, MSysGit, and other sources, that the best solution is to have your local repos set to use linux-style line endings, but set core.autocrlf to true. Unfortunately, I didn't do this early enough, so now every time I pull changes the line endings are borked.

I thought I had found an answer here but I can't get it to work for me. My Linux command line knowledge is limited at best, so i am not even sure what the "xargs fromdos" line does in his script. I keep getting messages about no such file or directory existing, and when I manage to point it to an existing directory, it tells me I don't have permissions.

I've tried this with MSysGit on Windows and via the Mac OS X terminal.

Quickie answered 2/10, 2009 at 17:12 Comment(2)

I can't upvote this thread even nearly enough. +1 ++ for it providing the best answer on the matter. – Meal 19/7, 2012 at 4:45

Agree with Charles. However, in my case (using Mac OS X 10.8) > git config core.autocrlf false worked, not > git config core.autocrlf input – Katekatee 4/9, 2013 at 19:24

L

204

The git documentation for gitattributes now documents another approach for "fixing" or normalizing all the line endings in your project. Here's the gist of it:

$ echo "* text=auto" >.gitattributes
$ git add --renormalize .
$ git status        # Show files that will be normalized
$ git commit -m "Introduce end-of-line normalization"

If any files that should not be normalized show up in git status, unset their text attribute before running git add -u.

manual.pdf -text

Conversely, text files that git does not detect can have normalization enabled manually.

weirdchars.txt text

This leverages a new --renormalize flag added in git v2.16.0, released Jan 2018.
But it may fail if you have "un-staged deleted files", hence stage those first, like:

git ls-files -z --deleted | xargs -0 git add

For older versions of git, there are a few more steps:

$ echo "* text=auto" >>.gitattributes
$ rm .git/index     # Remove the index to force git to
$ git reset         # re-scan the working directory
$ git status        # Show files that will be normalized
$ git add -u
$ git add .gitattributes
$ git commit -m "Introduce end-of-line normalization"

Library answered 13/1, 2011 at 18:32 Comment(13)

Could you tell me what the purpose of the git reset is, please? – Pax 13/9, 2012 at 15:58

forces git to rebuild the index, during which it scans each file to make a guess about whether its binary. The rm deletes the old index, reset builds the new index. – Library 15/10, 2012 at 18:30

Whenever I've done this I've omitted the git reset bit because the results have always been the same whether I did it or not. The git status after rm .git/index shows me all the files that need to be normalised, whether I reset or not, which is why I asked. Perhaps it's not necessary with new versions? – Pax 15/10, 2012 at 20:15

Hm. It seems that by doing git add . I was effectively doing the same thing. – Pax 15/10, 2012 at 20:44

Thanks, this worked for me. A useful command after running git status is to run git diff --ignore-space-at-eol just to be sure that the only changes you are committing are the line endings. – Forbade 2/5, 2014 at 19:19

Great solution. I had to change my core.safecrlf=true setting temporarily to false before running git add -u, then change it back to true. I did this in the repos I was doing this in by running git config core.safecrlf false && git add -u && git config --unset core.safecrlf in place of the $ git add -u line that Russ has above. This left my global setting as core.safecrlf=true because my one-liner only sets the local (non-global) git config, then unsets it when done. – Sever 15/8, 2014 at 19:58

This approach looks to be better since it doesn't try to "fix" the binary files. – Arcuation 20/11, 2014 at 18:46

Note: The only "real" difference between this and the "old" solution is in the presence of .gitattributes (with the appropriate content). Without this, git reset will detect no modifications, and is thus useless. – Etheleneethelin 31/5, 2015 at 15:56

Important: with this new procedure you can even git reset --hard as third step. Contrary to what one would expect, a hard reset still indicates a modified state of files (git status) and a hard reset can ONLY be effective as intended when .gitattributes is deleted or parts of it are outcommented (git 1.9). – Etheleneethelin 31/5, 2015 at 16:16

I'm a newbe using Git (used SVN for years until now). Two questions: a) the reset or reset --hard will erase all my changes, Right? b) If I "a" is true, and I backup my changes, I will re-introduce the problem all over again upon restoring, Right? – Sherr 7/4, 2016 at 23:33

It depends what changes you are talking about. This procedure it best performed when your working directory is clean. In other words, before starting this, run git status. If it shows any pending changes, commit them first. This procedure will then result in a new commit on top of that which just normalizes your line endings. You won't lose any other changes. – Library 11/4, 2016 at 13:32

Migrated project from TFS to Git and major problems med line endings, suddenly change files which could not be undo'ed etc. Tried a bunch of solutions, but this actually solved it! – Lelahleland 23/12, 2016 at 9:33

The instructions on the gitattributes page have been updated to take advantage of the --renormalize flag added in git v2.16.0 which was released in January 2018. The --renormalize flag consolidates the process of re-processing line endings for each tracked file into a single command: git add --renormalize .. – Fullmouthed 8/2, 2018 at 14:44

H

407

The easiest way to fix this is to make one commit that fixes all the line endings. Assuming that you don't have any modified files, then you can do this as follows.

# From the root of your repository remove everything from the index
git rm --cached -r .

# Change the autocrlf setting of the repository (you may want 
#  to use true on windows):
git config core.autocrlf input

# Re-add all the deleted files to the index
# (You should get lots of messages like:
#   warning: CRLF will be replaced by LF in <file>.)
git diff --cached --name-only -z | xargs -0 git add

# Commit
git commit -m "Fixed crlf issue"

# If you're doing this on a Unix/Mac OSX clone then optionally remove
# the working tree and re-check everything out with the correct line endings.
git ls-files -z | xargs -0 rm
git checkout .

Hudnut answered 2/10, 2009 at 19:3 Comment(11)

I got the same recommendation from Github. Not in love with it, but rewriting history also makes me nervous. I may just go with this solution for now. – Quickie 4/10, 2009 at 19:32

I'm actually going to mark your answer the fix because it was the one I went with. It was the only one that actually did seem to get all files at once. Others seemed to only get a few files at a time and each pull and checkout exposed more files that needed to be changed. Thanks! – Quickie 4/10, 2009 at 20:11

P.S. I recommended your fix to the guys at github.com and they updated their help guide to use your solution (previously it had just recommended a fresh clone and a hard reset, which did not seem to get all files.) help.github.com/dealing-with-lineendings – Quickie 5/10, 2009 at 14:22

You may also want to check out config.safecrlf to ensure that you aren't changing crlfs in non-text files (such as binary). Check it out in the docs kernel.org/pub/software/scm/git/docs/git-config.html. – Scale 23/7, 2010 at 16:11

@vrish88: If you're in this situation, though, you're likely to be suffering from mixed lined endings and core.safecrlf may actually prevent you from doing what you need to do. It's probably easier to not use safecrlf. git doesn't often get binary file detection wrong and if it does you can manually mark it as binary with a .gitattribute and recover the correct version from the previous commit. – Hudnut 23/7, 2010 at 16:27

Seems you want autocrlf=input on OSX/Unix and autocrlf=true on Windows – Brandabrandais 16/3, 2011 at 19:54

The newer solution recommended in Russ Egan's answer below is simpler and does not involve scary things like deleting all your source code, so I would really recommend people use that, even though this old solution has 10 times as many votes! – Release 20/7, 2011 at 21:54

Just be careful when some files (like test data) intentionally have mixed line endings. – Boxer 3/1, 2013 at 23:56

Seconding Russ Egan's answer below. I had bad line endings from a TFS -> GIT migration that I had to fix. The approach above missed a large number of files while it appears that Russ's did not. Unfortunately I don't know enough about Git to know why that is. – Dagmar 3/6, 2014 at 14:17

Two notes: 1) you may want to do git commit ... --no-verify to skip pre-commit hooks, 2) I've noticed (on Windows) that generally I have to repeat the procedure twice to really normalize all files (yep it's pretty weird). See my answer – Jaborandi 4/12, 2015 at 13:48

This steps will help you to fix eol issue but notice that files history (git blame) will be corrupted. – Playwright 21/8, 2016 at 21:33

L

204

The git documentation for gitattributes now documents another approach for "fixing" or normalizing all the line endings in your project. Here's the gist of it:

$ echo "* text=auto" >.gitattributes
$ git add --renormalize .
$ git status        # Show files that will be normalized
$ git commit -m "Introduce end-of-line normalization"

If any files that should not be normalized show up in git status, unset their text attribute before running git add -u.

manual.pdf -text

Conversely, text files that git does not detect can have normalization enabled manually.

weirdchars.txt text

This leverages a new --renormalize flag added in git v2.16.0, released Jan 2018.
But it may fail if you have "un-staged deleted files", hence stage those first, like:

git ls-files -z --deleted | xargs -0 git add

For older versions of git, there are a few more steps:

$ echo "* text=auto" >>.gitattributes
$ rm .git/index     # Remove the index to force git to
$ git reset         # re-scan the working directory
$ git status        # Show files that will be normalized
$ git add -u
$ git add .gitattributes
$ git commit -m "Introduce end-of-line normalization"