How to normalize working tree line endings in Git?
Asked Answered
R

8

117

I have cloned a repository that had inconsistend line endings. I have added a .gitattributes that sets the text attribute for the files I want to normalize. Now when I commit changes I get the message:

warning: CRLF will be replaced by LF in FILE.
The file will have its original line endings in your working directory.

How can I make git normalize my working copy of the file for me? Preferably I would like git to normalize the entire working tree.

Radiocommunication answered 26/3, 2013 at 15:42 Comment(1)
For people coming to this question in 2019, note that the second answer (by @philippn) is now better than the currently accepted oneHaphazard
D
126

With Git client 2.16 and higher there is now a much simpler way to do this. Just use:

git add --renormalize .

Note: it's better to do this with a clean workspace. For details, see:

Dihedron answered 1/6, 2018 at 13:54 Comment(8)
I just tried this on Git Bash for Windows and it staged every file in the repo for commit.Compensation
I have a error message: error: unknown option 'renormalize'Portrait
@RyanNghiem What does git --version say? I would guess it's simply older than 2.16 and you need to upgrade. HTHDihedron
@cja "Clean workspace" as in: the command git status returns something like nothing to commit, working tree cleanDihedron
This just doesn't do anything at all for me. :-/Deportment
Doesnt work. DownvoteBullough
@Bullough works like a charm for those who read the first 4 words of the answer and not just the copy paste the command. And it helps to have an up up to date version of git.Brachycephalic
This command does absolute nothing for me.Sf
C
139

For those using v2.16 or better, you can simply use:

git add --renormalize .  # Update index with renormalized files
git status               # Show the files that will be normalized
git commit -m "Introduce end-of-line normalization"

These directions are straight out of the gitattributes. For older versions, the docs (prior to v2.12) provide a different answer:

rm .git/index     # Remove the index to force git to
git reset         # re-scan the working directory
git status        # Show files that will be normalized
git add -u
git add .gitattributes
git commit -m "Introduce end-of-line normalization"

Do this sequence after you have edited .gitattributes.

Update

It appears some users have had trouble with the above instructions. Updated docs for gitattributes (2.12 to 2.14) shows a new set of instructions (after editing the .gitattributes files):

git read-tree --empty   # Clean index, force re-scan of working directory
git add .
git status        # Show files that will be normalized
git commit -m "Introduce end-of-line normalization"

Thanks to @vossad01 for pointing this out.

Also, with either solution the files in your working copy still retain their old line endings. If you want to update them, make sure your working tree is clean and use:

git rm --cached -r .
git reset --hard

Now the line endings will be correct in your working tree.

Carlcarla answered 26/3, 2013 at 20:30 Comment(15)
This worked, thank you! I was looking through the gitattributes documentation but it was so much to read so I sort of glossed over it and couldn't find what I was looking for. I feel stupid now.Radiocommunication
No worries, it happens! It's not as easy to find as it probably should be. :-)Carlcarla
This works when adding .gitattributes for the first time or changing the setting, but it doesn't work if the working tree has a different EOL (at least with MsysGit it doesn't). For that, it appears that git rm --cached -r . and then git reset --hard worked (warning: destroys your working tree). From help.github.com/articles/dealing-with-line-endings.Clementeclementi
Has something changed with Git that has broken this? I have used this many times, but now am in a repository where it is not working (git 2.13.0, linux). Instead I had to do git add . after removing the index (being mindful of anything in the working directory that was previously untracked).Limitation
@Limitation Can you explain what failed in the process? I don't see a reason why the current answer shouldn't work. I can certainly update the answer with the new procedure too--I just want to understand the issue a little more.Carlcarla
@jszakmeister It does not normalize any files, see my previous comment about wondering if something changed with Git that broke it. I don't know (and am not motivated enough to research) what changed, but I have now bounced between enough different environments to be confident something has changed causing new directions to be needed.Limitation
@Limitation Interesting. I guess I'm going to have to play with it more and see. Thanks for the heads up.Carlcarla
@Limitation So I've tried both ways and the old way still works fine. In fact, with both ways I end up with exactly the same results. I don't know what you did, but the one requirement for git add -u to work is that the file must have been versioned already (it must be part of the repo). So maybe that where things fail? I tried this with 2.15.0, so it's possible there was some other bug affecting this for some short time too.Carlcarla
Only the second method worked for me. Came here as other guides all have the first method, which wasn't working, but git read-tree --empty method was the trickLynn
Be cautious with the git read-tree --empty; git add . variant. Any tracked files that would be ignored by .gitignore will be deleted when using it. The documentation now recommends git add --renormalize .Limitation
Is there a setting I need to change somewhere? I have ran these commands multiple times now and it says succesfull in bash, but when I open the file in notepad++ with lineendings showing, it still shows CRLF.Wring
@RogerFar what OS are you running on? Windows? What does git config --get core.autocrlf say?Carlcarla
It says false. But even when it "tries" to convert line endings it too quick, like 5 seconds for 1000 files. In the end I used notepad++ to do a global search and replace.Wring
@RogerFar Do you have a .gitattributes set up? I've never seen this be a real problem, even under Windows. It sounds like you've missed a step or there is something environmental going on. And 5 seconds for 1,000 is quite possible. The conversion is pretty darn easy, and Git is awfully efficient in what it does. Sorry it was an issue for you though.Carlcarla
I did setup .gitattributes yeah, but it's strange. It did commit "something" but a git blame didn't show it changed line endings at all, neither did notepad++.Wring
D
126

With Git client 2.16 and higher there is now a much simpler way to do this. Just use:

git add --renormalize .

Note: it's better to do this with a clean workspace. For details, see:

Dihedron answered 1/6, 2018 at 13:54 Comment(8)
I just tried this on Git Bash for Windows and it staged every file in the repo for commit.Compensation
I have a error message: error: unknown option 'renormalize'Portrait
@RyanNghiem What does git --version say? I would guess it's simply older than 2.16 and you need to upgrade. HTHDihedron
@cja "Clean workspace" as in: the command git status returns something like nothing to commit, working tree cleanDihedron
This just doesn't do anything at all for me. :-/Deportment
Doesnt work. DownvoteBullough
@Bullough works like a charm for those who read the first 4 words of the answer and not just the copy paste the command. And it helps to have an up up to date version of git.Brachycephalic
This command does absolute nothing for me.Sf
P
12

Alternative approach (differs only in command used)

Make sure you have no any pending changes in repository:

$ git status
$ git stash

Modify .gitattributes so CRLF interpretation will changed:

$ echo "*.txt  text" >>.gitattributes
$ git commit -m "Made .txt files a subject to CRLF normalization." -- .gitattributes

Remove data from index and refresh working directory:

$ git rm --cached -r .
$ git reset --hard

Review CRLF fixes that Git proposes:

$ git ls-files --eol
$ git status
$ git diff

Agree with Git decision:

$ git add -u
$ git commit -m "Normalized CRLF for .txt files"

Reload changes as if clean clone was done:

$ git rm --cached -r .
$ git reset --hard
Phono answered 7/8, 2017 at 12:11 Comment(2)
Note: the --eol option to git ls-files is new in Git 2.8 (not that new any more, but some people are still suffering with Git 1.7, even late in 2018!).Magnanimity
git ls-files --eol helps verify that your line endings in all files are as intended, thanks!Eidetic
P
4

The .gitattributes settings will only affect new commits. If this repository has no history published (no others depending on it), you might want to go through the whole history. In Unix/Linux, you can use dos2unix(1) to fix all files in combination with find(1), and using the history rewriting of filter-branch (see the discussion in the git book) you can even clean up the full history of the project.

Use with utmost care, on a fresh clone. Get in contact with anybody who might have a clone, and advise them what you want to do.

Pyrogenic answered 26/3, 2013 at 19:42 Comment(2)
I would generally advise against this solution, unless a perfect history is important. There are so many ways to shoot oneself in the foot by rewriting the history. Normalize the files in the repo in a single commit, and let .gitattributes do the work from there on out.Fidelia
@angularsen, if it is important to publish a clean, useable history... that is why I advise to work on a clean, new clone and (re)publish that once it is checked sane.Pyrogenic
E
3

I had to re-clone repo with this flag -c core.autocrlf=false, and it worked, no other config required.

Like this:

git clone -c core.autocrlf=false https://github.com/any-repo.git

Our project initially was made with LF on Mac, but on Windows it was automatically converted to CRLF. We use eslint and it underlined every line of code, until I re-cloned it.

Enterotomy answered 21/2, 2022 at 18:9 Comment(0)
S
2

The * text=auto option in .gitattributes leaves the Git repository in an 'illegal state' if it contains files with CRLF (Windows) line endings which are now marked as text (see https://marc.info/?l=git&m=154484903528621&w=2). The standard renormalize option does not work correctly with the LFS filters, so the instructions in the other answers or for example at https://help.github.com/en/articles/dealing-with-line-endings, do not work correctly. Instead these steps worked for us:

Situation:

  • On Windows
  • Git repository contained files with both CR and CRLF line endings
  • Added * text=auto to .gitattributes (so not depended on user having set core.crlf=auto on Windows)
  • Also changed the -crlf to -text for LFS tracked files, not sure that is needed.

    1. Create a new branch from the branch with the line ending problem (assuming no uncommitted changes there): git checkout -b feature/doing-stuff-fix-eol
    2. Remove the LFS filters from .gitattributes (replace all 'filter=lfs diff=lfs merge=lfs ' with nothing)
    3. Commit and push: git commit -a -m "Disable LFS filters for EOL fix"
    4. Move to non-git folder
    5. Uninstall LFS globally: git lfs uninstall
    6. Create a new repository clone: git clone -b feature/doing-stuff-fix-eol [remote repository url] fix-eol
    7. Normalize the line endings: git add --renormalize . (note the dot to renormalize all files)
    8. Check only the correct files normalized. It should not include files normally handled by LFS!
    9. Commit and push (save the hash): git commit -m "Fix line endings"
    10. Move to non-git folder
    11. Install LFS globally: git lfs install
    12. Go to original repository clone and pull
    13. Checkout your original branch: git checkout feature/doing-stuff
    14. Cherry pick the eol fix commit and push: git cherry-pick [hash]
    15. Delete the eol branch and push
    16. Delete the eol repository clone (or keep around if you need to fix more branches)
Samira answered 28/6, 2019 at 11:2 Comment(0)
P
1

I've had CRLF line endings in working tree and LF in repository, because project folder was just copied from a Windows machine to a Linux one with Git 1.8.3. This is how I've reset line endings to LF:

  • git status returns a list of all changed files
  • git diff ignores line endings and returns files which are "really" changed
  • comm -3 compares lists and returns files from git status list which are not in git diff list
  • git checkout resets these files to HEAD
comm -3 <(git status --porcelain --untracked-files=no | cut -c4- | sort) <(git diff --name-only | sort) | xargs git checkout HEAD --
Placoid answered 23/6, 2023 at 11:0 Comment(0)
Y
0

The merge.renormalize configuration setting may be useful.

Yorker answered 1/6, 2020 at 3:25 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.