Removing '.git' directory from git repo?

Question

I'm trying to migrate a git repo from Kiln to Github. I can add the new remote just fine, but when I try to push master to the new remote, I get the following error:

Counting objects: 8691, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3000/3000), done.
remote: error: object a9ee490ac00987835de30bdbc851da5e8d45d28b:contains '.git'
remote: fatal: Error in object
error: pack-objects died of signal 13
error: failed to push some refs to '[email protected]:Account/repo.git'

Commit a9ee490ac00987835de30bdbc851da5e8d45d28b has the following files in it:

.git/
CHANGELOG.md
JSONKit.h
JSONKit.m
README.md

Obviously someone in the past, using hg, checked in a full git repo in a subdirectory.

I'd like to just kill that directory entirely, but am having trouble removing that file from the git history.

The answer in pushing a git repo fails with error: contains '.git' doesn't help, because I've the the repo as a git repo, not a mercurial one.

I tried git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch .git', but I get the error:

Rewrite 7dbd0970d6c79215d11994b4a9b8091b2e954cfb (326/442)error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/HEAD'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/config'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/description'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/applypatch-msg.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/commit-msg.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/post-update.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/pre-applypatch.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/pre-commit.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/pre-rebase.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/prepare-commit-msg.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/hooks/update.sample'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/index'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/info/exclude'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/logs/HEAD'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/logs/refs/heads/master'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/objects/pack/pack-43fac03d375df5c1e380c5e522ba6bcb9b4e1ec1.idx'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/objects/pack/pack-43fac03d375df5c1e380c5e522ba6bcb9b4e1ec1.pack'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/packed-refs'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/refs/heads/master'
error: Invalid path 'Account/Libraries/ShareKit/Submodules/JSONKit/.git/refs/remotes/origin/HEAD'

and history doesn't seem to be changed. I'm at the limit of my git knowledge. Can anyone help?

Combustible

Is the repo open-source? are you able to share a copy of it? – Pettigrew

Ah, no need to share a copy of the repo in question, I've managed to construct a Git repository with the same kind of bad tree, which I can use for testing purposes. – Pettigrew

possible duplicate of pushing a git repo fails with error: contains '.git' – Pourboire

Thanks @dana! I was facing the exact same problem right now (the .git folder having been added by mistake previously, while the repo was still in hg) – Pyrostat

Answer 1

Use the BFG Repo-Cleaner, a simpler, faster alternative to git-filter-branch specifically designed for removing files from Git history.

Carefully follow the steps here: https://rtyley.github.io/bfg-repo-cleaner/#usage - but the core bit is just this: download the BFG jar (requires Java 8 or above) and run this command:

$ java -jar bfg.jar --delete-folders .git --delete-files .git --no-blob-protection my-repo.git

Your entire repository history will be scanned, and any file or folder named .git will be removed. I've tested this against a specially-constructed test repo containing a .git folder, and it worked just fine.

Thanks to Michel Jouvin for reminding me that files named '.git' are also illegal in Git (and the error message reported by Git doesn't make it immediately obvious that they are files, rather than folders), I've updated this answer to reflect that.

Full disclosure: I'm the author of the BFG Repo-Cleaner.

Pettigrew

I had the same problem and this solution did the trick for me. – Trustee

Thanks!! I had the exact same problem too and this saved my life. If I could I'd mark this as the right answer. – Thetic

I tried the git filter-branch approach and it did not work, but using the bfg repo-cleaner worked like a charm. thanks! – Leff

Thanks! I agree with @JanSegre, this should definitely be marked as the correct answer. Worked like a charm for me. – Pyrostat

This one did the trick for me too. Super easy and fast approach. Although I did not use the --no-blob-protection as it was a repo i just imported from Perforce. – Immaterial

Also worked the first time for me - thanks for the great tool Roberto! – Ballistic

This worked for me. Big thank you. BFG tool did the job. – Zealot

Answer 2

As of git 2.2.22 the way to do this is with the git-filter-repo tool.

Easy and fast:

git filter-repo --force --invert-paths --path-glob '*/.git' --path '.git'

Anderton

Worked like a charm. Thanks! Only required 4.5 mins for an 18-GB repo for me. – Kuth

Answer 3

Have you tried with the following command ?

git filter-branch --tree-filter 'rm -Rf .git' HEAD

git filter-branch documentation indicates that this would be slower than --index-filter, but it could do the job.

Incidental

Unfortunately, I don't think this approach will work- running against a test repo that I constructed, this is the output I get: $ git filter-branch --tree-filter 'rm -Rf foo/.git' HEAD ..... Rewrite 6652877677cf01a7b668704134d5e8bb62bfe317 (6/6)error: Invalid path 'foo/.git/folder/zero' error: Invalid path 'foo/.git/hero' error: Invalid path 'foo/.git/zero' Ignoring path foo/.git/folder/zero Ignoring path foo/.git/hero Ignoring path foo/.git/zero WARNING: Ref 'refs/heads/master' is unchanged – Pettigrew

...so it looks like even using 'git filter-branch --tree-filter' has the problem that @Combustible encountered in the question. I would think that using The BFG is probably a better idea at this point. – Pettigrew

Answer 4

Another way to handle the case where a Mercurial (Hg) repo has a .git folder embedded in it is to first "fix" the Hg repo before exporting to Git. @laurens-holst has a great way of doing this.

If the .git folder was in the root of your project, then your create a text file, my-filemap that looks like:

exclude ".git"

Enable the Convert extension in your .hgrc file:

[extensions]
convert =

Then filter out the .git folder using the Hg convert extension:

hg convert --filemap my-filemap originalrepo newrepo

Now you can export from newrepo and won't have the issue of the embedded .git folder.

You can also use git fsck to make sure the new Git repo is ok before pushing it.

Uball

Answer 5

As I understand it kiln will let you clone a repository as either git or Mercurial these days, so clone it as mercurial, use the mercurial filemap solution you already found push that Mercurial repo as a new repo to Kiln and then clone it down as git. Then you'll have a git repo you can push to github.

Wilhelmina