P

26

1983

I have a Git repository which contains a number of subdirectories. Now I have found that one of the subdirectories is unrelated to the other and should be detached to a separate repository.

How can I do this while keeping the history of the files within the subdirectory?

I guess I could make a clone and remove the unwanted parts of each clone, but I suppose this would give me the complete tree when checking out an older revision etc. This might be acceptable, but I would prefer to be able to pretend that the two repositories doesn't have a shared history.

Just to make it clear, I have the following structure:

XYZ/
    .git/
    XY1/
    ABC/
    XY2/

But I would like this instead:

XYZ/
    .git/
    XY1/
    XY2/
ABC/
    .git/
    ABC/

Pilliwinks answered 11/12, 2008 at 13:57 Comment(4)

This is trivial now with git filter-branch see my answer below. – Insufficient 20/8, 2014 at 14:12

@Insufficient is right. This is no longer difficult to do but it is difficult to find the right answer on Google because all the old answers dominate the results. – Abampere 14/10, 2014 at 5:39

Use of git filter-branch is discouraged. See warning in docs. – Inhere 5/11, 2021 at 16:59

Please use an alternative history filtering tool such as git filter-repo. – Stare 13/9, 2022 at 4:6

D

1273

Update: This process is so common, that the git team made it much simpler with a new tool, git subtree. See here: Detach (move) subdirectory into separate Git repository

You want to clone your repository and then use git filter-branch to mark everything but the subdirectory you want in your new repo to be garbage-collected.

To clone your local repository:
```
git clone /XYZ /ABC
```
(Note: the repository will be cloned using hard-links, but that is not a problem since the hard-linked files will not be modified in themselves - new ones will be created.)
Now, let us preserve the interesting branches which we want to rewrite as well, and then remove the origin to avoid pushing there and to make sure that old commits will not be referenced by the origin:
```
cd /ABC
for i in branch1 br2 br3; do git branch -t $i origin/$i; done
git remote rm origin
```
or for all remote branches:
```
cd /ABC
for i in $(git branch -r | sed "s/.*origin\///"); do git branch -t $i origin/$i; done
git remote rm origin
```
Now you might want to also remove tags which have no relation with the subproject; you can also do that later, but you might need to prune your repo again. I did not do so and got a WARNING: Ref 'refs/tags/v0.1' is unchanged for all tags (since they were all unrelated to the subproject); additionally, after removing such tags more space will be reclaimed. Apparently git filter-branch should be able to rewrite other tags, but I could not verify this. If you want to remove all tags, use git tag -l | xargs git tag -d.
Then use filter-branch and reset to exclude the other files, so they can be pruned. Let's also add --tag-name-filter cat --prune-empty to remove empty commits and to rewrite tags (note that this will have to strip their signature):
```
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC -- --all
```
or alternatively, to only rewrite the HEAD branch and ignore tags and other branches:
```
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC HEAD
```
Then delete the backup reflogs so the space can be truly reclaimed (although now the operation is destructive)
```
git reset --hard
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git gc --aggressive --prune=now
```
and now you have a local git repository of the ABC sub-directory with all its history preserved.

Note: For most uses, git filter-branch should indeed have the added parameter -- --all. Yes that's really --space-- all. This needs to be the last parameters for the command. As Matli discovered, this keeps the project branches and tags included in the new repo.

Edit: various suggestions from comments below were incorporated to make sure, for instance, that the repository is actually shrunk (which was not always the case before).

Disqualify answered 11/12, 2008 at 13:57 Comment(28)

Good point! That would apply to most people using git filter-branch this way. In my case, I was segregating a library which had its own series of release numbers in tags, so I didn't want the project tags in my new repo. I'll edit that into the answer. – Disqualify 15/12, 2008 at 18:51

As of today, filter-branch is not supported on Windows. It looks like it is coming soon though. Check the msysgit discussion group (at google groups) for details. – Tirza 1/4, 2009 at 15:2

Why do you need --no-hardlinks? Removing one hardlink won't affect the other file. Git objects are immutable too. Only if you'd change owner/file permissions you need --no-hardlinks. – Trashy 1/2, 2010 at 9:58

git-filter-branch set_ident() calls LANG=C LC_ALL=C sed ... for maximum compatibility with non UTF-8 aware sed. Therefore, git-filter-branch dies on commits with UTF-8 characters in the author: or committer: fields (see git show --pretty=raw <commit SHA1>). A workaround, use a UTF-8 aware sed (e.g. GNU sed) and edit git-filter-branch: LANG=en_US.UTF-8 sed -ne ... – Bencher 20/8, 2010 at 15:4

Also the not-yet-included 'git filter-branch --split-submodule' can be used to preserve consistent history. vi-server.org/vi/bin/git-extract-submodule.patch – Glassware 27/12, 2010 at 1:48

I find that if the /XYZ repo is large, the /ABC repo won't be as small as desirable. To further shrink things: git clone /ABC /ABC.small ; cd /ABC.small ; git repack -A -d ; git gc --prune=DATE where DATE is date +%Y-%m-%d. For some reason if I don't git clone and work on the clone, nothing shrinks. :-/ – Drubbing 15/7, 2011 at 20:33

And if you want to rewrite your tags to not reference the old structure, add --tag-name-filter cat – Armed 6/10, 2011 at 11:58

Like Paul, I did not want project tags in my new repo, so I did not use -- --all. I also ran git remote rm origin, and git tag -l | xargs git tag -d before the git filter-branch command. This shrunk my .git directory from 60M to ~300K. Note that I needed to run both of these commands to in order to get the size reduction. – Trypsin 17/11, 2011 at 21:18

The git man page recommends, instead of rm -rf .git/refs/original/, git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d; I guess the latter is more robust if refs are not stored in the right place. Moreover, I believe that 'git remote rm origin' is also needed to shrink the repo, otherwise the refs from origin will keep objects referenced. @jonp, I think that was the problem for you. Finally, to also rewrite other branches, one must set them up manually with git branch after cloninng, -- --all and remove HEAD (which stops rewriting of other branches). – Flambeau 20/2, 2012 at 0:7

For many readers (including me), the answer by itself was quite imprecise. Therefore, I incorporated most of the suggestions from comments after testing them. In particular, I made sure that after following this procedure, the repository is actually shrunk (which was not the case for me and various other people). I hope that is OK. – Flambeau 20/2, 2012 at 0:44

To remove references in Windows you can use: for /f %a IN ('git for-each-ref --format="%(refname)" refs/original/') DO git update-ref -d %a – Philps 6/4, 2012 at 19:20

I found git-subtree to be effective as well, and much simpler. Found via this related question/answer: https://mcmap.net/q/12522/-how-to-extract-a-git-subdirectory-and-make-a-submodule-out-of-it – Teerell 18/4, 2012 at 15:2

On Windows, I first tried the steps from the powershell prompt, but step 5b failed. Then I re-tried all steps from the git bash, and everything worked perfectly. Thanks! – Peplos 7/7, 2012 at 9:23

From the 5 or so similar questions, this is the most valuable answer for me together with this link: gbayer.com/development/… – Warms 24/9, 2012 at 15:43

@Gragg has suggested to edit and remove the --aggressive option to gc "as it makes the size of the repo explode: see also link" – Nahshon 29/11, 2012 at 10:40

One important note is that if you have multiple branches being preserved in Step #2, then you'll have to run Step #4 and Step #5 through each one. Or at least thats what happened for me. – Rakel 5/2, 2013 at 0:9

Does this not create ABC/ instead of ABC/ABC/? – Carmichael 21/5, 2013 at 8:39

So, I've probably used this answer 10 times now and I keep wanting to upvote it again. Tons of stuff on stackoverflow has been useful, but this answer is the most useful so far. I've even put it in evernote for super quick referencing. Thanks! – Lapboard 13/6, 2013 at 19:7

How to use this if you want to take 2 directories? – Groveman 18/6, 2013 at 13:59

N.B. this leaves the new repo structured as /ABC/ABC, instead of just /ABC (though someone edited the question after the fact to say that was what was wanted) – Freiburg 11/12, 2013 at 19:28

@Disqualify - it should be noted that this approach fails if the ABC folder ever had a different name - in this case, the history stops at the point when it was renamed. – Peptize 24/4, 2014 at 16:17

That didn't reclaim disk space for me, the .git folder must have a bunch of unused history still in there as the pruned folder is 10s of KB yet the .git is 1.5 GB! Any ideas? – Gastrocnemius 30/4, 2014 at 19:13

After I used filter-branch, all of the files in that directory were removed. – Turbid 26/5, 2014 at 12:49

Really useful this is. However, if /ABC used to be called /XY3 until 3 commits before HEAD, will the resulting ABC repository not only have these 3 commits? At least this is what I have found. – Spoken 21/11, 2014 at 4:52

These answers are a bit confusing... Are the five steps in this answer now N/A? Is the "The Easy Way" answer below the right answer? When I first saw this answer, I figured the 'Detach subdirectory into separate Git repository' link was to more in depth documentation and was summarized by the five steps that followed. Can this be clarified? Perhaps add a header above the five steps, like "The Old Way (pre v1.7.11). – Hatchery 22/1, 2015 at 0:34

Github wiki with the same steps help.github.com/articles/… – Iceberg 28/9, 2016 at 0:7

Also, instead of cloning a repo in the first place, you can directly add a submodule which adds the new repo to git and creates the new repo into your current repo as well ! – Concede 30/4, 2018 at 7:0

github.com/newren/git-filter-repo is the suggested way by git itself to easily do this. – Mylesmylitta 28/10, 2020 at 15:25

T

1603

The Easy Way™

It turns out that this is such a common and useful practice that the overlords of Git made it really easy (added in version 1.7.11 - May 2012). Also, there's a real-world example in the walkthrough below.

Prepare the old repo
```
 cd <big-repo>
 git subtree split -P <name-of-folder> -b <name-of-new-branch>
```
Note: <name-of-folder> must NOT contain leading or trailing characters. For instance, the folder named subproject MUST be passed as subproject, NOT ./subproject/

Note: <name-of-new-branch> is a branch you will be creating in the existing/old repo, NOT the new one [that comes later].

Note for Windows users: When your folder depth is > 1, <name-of-folder> must have *nix style folder separator (/). For instance, the folder named path1\path2\subproject MUST be passed as path1/path2/subproject

Create the new repo

 mkdir ~/<new-repo> && cd ~/<new-repo>
 git init
 git pull </path/to/big-repo> <name-of-new-branch>

Link the new repo to GitHub or wherever

 git remote add origin <[email protected]:user/new-repo.git>
 git push -u origin master

Cleanup inside <big-repo>, if desired
```
 git rm -rf <name-of-folder>
```
Note: This leaves all the historical references in the repository. See the Appendix below if you're actually concerned about having committed a password or you need to decreasing the file size of your .git folder.

Walkthrough

These are the same steps as above, but following my exact steps for my repository instead of using <meta-named-things>.

Here's a project I have for implementing JavaScript browser modules in node:

tree ~/node-browser-compat

node-browser-compat
├── ArrayBuffer
├── Audio
├── Blob
├── FormData
├── atob
├── btoa
├── location
└── navigator

I want to split out a single folder, btoa, into a separate Git repository

cd ~/node-browser-compat/
git subtree split -P btoa -b btoa-only

I now have a new branch, btoa-only, that only has commits for btoa and I want to create a new repository.

mkdir ~/btoa/ && cd ~/btoa/
git init
git pull ~/node-browser-compat btoa-only

Next, I create a new repo on GitHub or Bitbucket, or whatever and add it as the origin

git remote add origin [email protected]:node-browser-compat/btoa.git
git push -u origin master

Happy day!

Note: If you created a repo with a README.md, .gitignore and LICENSE, you will need to pull first:

git pull origin master
git push origin master

Lastly, I'll want to remove the folder from the bigger repo

git rm -rf btoa

Clearing your history

By default, removing files from Git doesn't actually remove them; it just commits that they aren't there any more. If you want to actually remove the historical references (i.e. you committed a password), you need to do this:

git filter-branch --prune-empty --tree-filter 'rm -rf <name-of-folder>' HEAD

After that, you can check that your file or folder no longer shows up in the Git history at all:

git log -- <name-of-folder> # should show nothing

However, you can't "push" deletes to GitHub and the like. If you try, you'll get an error and you'll have to git pull before you can git push - and then you're back to having everything in your history.

So if you want to delete history from the "origin" - meaning to delete it from GitHub, Bitbucket, etc - you'll need to delete the repo and re-push a pruned copy of the repo. But wait - there's more! - if you're really concerned about getting rid of a password or something like that you'll need to prune the backup (see below).

Making `.git` smaller

The aforementioned delete history command still leaves behind a bunch of backup files - because Git is all too kind in helping you to not ruin your repo by accident. It will eventually delete orphaned files over the days and months, but it leaves them there for a while in case you realize that you accidentally deleted something you didn't want to.

So if you really want to empty the trash to reduce the clone size of a repo immediately you have to do all of this really weird stuff:

rm -rf .git/refs/original/ && \
git reflog expire --all && \
git gc --aggressive --prune=now

git reflog expire --all --expire-unreachable=0
git repack -A -d
git prune

That said, I'd recommend not performing these steps unless you know that you need to - just in case you did prune the wrong subdirectory, y'know? The backup files shouldn't get cloned when you push the repo, they'll just be in your local copy.

Credit

Toronto answered 25/7, 2013 at 17:10 Comment(41)

...in particular, not on Ubuntu, even not when using ppa:git-core/ppa :-( – Cardona 15/8, 2013 at 15:5

@rui.araujo: I ended up with creating a small shell script in my ~/bin directory that sources git-subtree.sh: gist.github.com/6338779. Of course this has to be chmod +x-d, too, – Cardona 26/8, 2013 at 7:19

If you have pushed a password to a public repository, you should change the password, not try to remove it from the public repo and hope nobody saw it. – Cathodoluminescence 18/9, 2013 at 3:43

You can override the push error when pushing to GitHub by doing git push -f to force the overwriting. – Palmitin 17/10, 2013 at 21:17

@rui.araujo your comment is very useful and makes the answer complete. It would be good if it was added to the actual answer. – Sallie 7/11, 2013 at 12:58

This doesn't work for me. When I call git pull </path/to/big-repo> <name-of-new-branch>, I get an error: Couldn't find remote ref <name-of-new-branch>. – Dripstone 16/12, 2013 at 1:43

This really helped me with a lot of repositories, specially using svn2git and then splitting the repository into multiple Git repos... Look at github.com/marcellodesales/svnedge-* for examples. – Nutrient 16/12, 2013 at 8:50

according to kernel.org/pub/software/scm/git/docs/git-filter-branch.html for git filter-branch, using git rm instead of rm is faster. – Ignoramus 20/1, 2014 at 15:46

on ubuntu 13.10 you can find it in the specified location: /usr/share/doc/git/contrib/subtree/git-subtree.sh (without having to use the ppa). – Underbrush 19/3, 2014 at 15:34

When cleaning up, shouldn't the branch created with git subtree be deleted as well? – Polyphone 26/3, 2014 at 8:59

What about if we have a subdirectory which was previously renamed? I tried this on a <name-of-folder> which at some point in time was renamed from <another-name-of-folder>. When I look at the history in <new-repo>, the initial commit is the one which renamed the folder - all of the history before that is erased from <new-repo> – Peptize 24/4, 2014 at 16:22

re "latest git on OS X": there is no need to mess with brew or any other extra package manager because the XCode command line tools include Git. Available from Apple free of charge, installs/updates without any hassle. – Cystitis 3/10, 2014 at 12:40

this seems to make a new repo with the contents of ABC/, but the new repo doesn't contain the folder ABC/ itself, as the question asked. How would you do this? – Cephalization 8/10, 2014 at 12:14

In the example it looks like it's just foo names to me. xyz, xyz1, xyz2, etc. Anyway mkdir ABC/; git mv ./* ABC/. I'm sure there's some way to give a prefix - maybe git read-tree --prefix=spoon-knife/ -u spoon-knife/master. See help.github.com/articles/about-git-subtree-merges. – Toronto 10/10, 2014 at 6:43

To pull the split subtree into a folder: git subtree add --prefix <new/folder/path> </path/to/big-repo> <name-of-new-branch> – Getraer 12/11, 2014 at 6:18

@Getraer I tried this on an empty repo and it didn't work. But I added my first commit and after that your solution worked for me. – Ambivalence 8/2, 2015 at 14:40

Is there a way to apply this method in order to split multiple sub folders into a new git repository? – Frizzy 15/6, 2015 at 8:55

What I finally did to split multiple sub folders into a new project was in 1. before the subtree split, move the sub folders I wanted into a unique sub-folder, rewriting history to keep it when pulling (step 2.): git filter-branch --tree-filter "mkdir <name-of-folder>; mv sub1 sub2 <name-of-folder>/" HEAD – Frizzy 15/6, 2015 at 12:15

@AnthonyO. Could you elaborate on your method ? Or even write an answer ? I am stuck here and would upvote it for sure. – Uprise 15/6, 2015 at 16:29

I finally used the solution given in stackoverflow.com/questions/19954485/… it works well for me so far. – Uprise 15/6, 2015 at 17:33

Hey @AnthonyO. I am trying something similar to what you wrote to move multiple sub-directories, but its failing for me. The error I get is rename sub1 to folder/sub1: No such file or directory and same for the second directory. Am I missing something ? – Cominform 5/8, 2015 at 17:14

@Cominform & PierreHenry : I've written what I did in a separated answer. Here it is: https://mcmap.net/q/12521/-detach-move-subdirectory-into-separate-git-repository – Frizzy 6/8, 2015 at 15:28

Hey @AnthonyO. I also asked a similar question here: Splitting many subdirectories into a new, separate Git repository and got it to work. But thanks, I will try out your answer too. – Cominform 6/8, 2015 at 15:45

It's probably a good idea to pass --prune-empty to filter-branch to remove the empty commits that happen due to removing the directory – Complaisant 12/8, 2015 at 11:22

As a corollary to this very useful answer - should anyone else use the subtree approach, be advised to manually split-out the .gitattributes file! I fell victim to this and got a CRLF/LF warning after adding a single ' ' to a file. – Gild 20/8, 2015 at 14:2

As far as I understand with this method (git subtree) you will loose both branches and tags. Using git filter-branch-method as described in https://mcmap.net/q/12521/-detach-move-subdirectory-into-separate-git-repository it is possible to keep/rewrite branches and tags. – Gragg 14/10, 2015 at 11:26

This solution doesn't preserve history. – Aricaarick 11/11, 2015 at 2:6

If your objective is to link the new repository as a submodule of the original project, then you need this answer (Usage §) of "How to extract a git subdirectory and make a submodule out of it?". – Bloomsbury 14/11, 2015 at 14:7

If at some point in the past you ran git subtree, you may get a fatal: bad object error when running the command again. Avoid this by passing in the --ignore-joins option. – Louvre 22/1, 2016 at 13:26

This method turns out to be essentially useless -- if you follow it you lose ALL branches other than master, and all the history that may be in those branches. – Easement 6/4, 2016 at 23:40

See rogerdpack's answer if you ever did any directory renames/moves in your commits which you wish to preserve. The filter-branch and subtree methods will not keep truncate commits at directory renames. – Peptize 30/10, 2017 at 17:17

The special directory path requirements highlight the big problem of git being internally inconsistent sometimes, which makes it that much more obtuse. – Homozygote 25/4, 2018 at 16:31

The popd and pushd command make this rather implicit and harder to grok what it intends to do ... – Lorant 4/6, 2018 at 16:39

The git subtree split command is much slower than the filter-branch approach. – Churchwoman 3/12, 2019 at 5:51

On CentOS 7, do sudo yum install git-subtree to get the subtree command. – Psychomotor 20/12, 2019 at 16:53

what about if your initial repository has many branches? this solution just preserves the master branch – Insignificance 27/1, 2020 at 13:58

Using the git subtree approach describe here, is there a way to preserve existing branches (and tags)? Since the whole relevant history is kept, that should be about restoring a pointer to a given commit? – Sleeper 5/3, 2020 at 13:52

How would you keep the btoa folder in the new repository? – Dancette 3/8, 2021 at 15:41

For those coming here in 2022: A very good (probably better) alternative to the answer provided here, based on git filter-repo, can be found further down in stackoverflow.com/questions/359424/… – Beale 11/5, 2022 at 17:51

Could I be getting an issue because original subfolder was in a private repo and the new one is public? – Isagoge 14/1, 2023 at 13:19

@AnatolyBugakov No. git is completely unaware any settings on GitHub. The CLI program was created without any relation to the SaaS product. – Toronto 17/1, 2023 at 5:37

D

1273

Update: This process is so common, that the git team made it much simpler with a new tool, git subtree. See here: Detach (move) subdirectory into separate Git repository

You want to clone your repository and then use git filter-branch to mark everything but the subdirectory you want in your new repo to be garbage-collected.

To clone your local repository:
```
git clone /XYZ /ABC
```
(Note: the repository will be cloned using hard-links, but that is not a problem since the hard-linked files will not be modified in themselves - new ones will be created.)
Now, let us preserve the interesting branches which we want to rewrite as well, and then remove the origin to avoid pushing there and to make sure that old commits will not be referenced by the origin:
```
cd /ABC
for i in branch1 br2 br3; do git branch -t $i origin/$i; done
git remote rm origin
```
or for all remote branches:
```
cd /ABC
for i in $(git branch -r | sed "s/.*origin\///"); do git branch -t $i origin/$i; done
git remote rm origin
```
Now you might want to also remove tags which have no relation with the subproject; you can also do that later, but you might need to prune your repo again. I did not do so and got a WARNING: Ref 'refs/tags/v0.1' is unchanged for all tags (since they were all unrelated to the subproject); additionally, after removing such tags more space will be reclaimed. Apparently git filter-branch should be able to rewrite other tags, but I could not verify this. If you want to remove all tags, use git tag -l | xargs git tag -d.
Then use filter-branch and reset to exclude the other files, so they can be pruned. Let's also add --tag-name-filter cat --prune-empty to remove empty commits and to rewrite tags (note that this will have to strip their signature):
```
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC -- --all
```
or alternatively, to only rewrite the HEAD branch and ignore tags and other branches:
```
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter ABC HEAD
```
Then delete the backup reflogs so the space can be truly reclaimed (although now the operation is destructive)
```
git reset --hard
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git gc --aggressive --prune=now
```
and now you have a local git repository of the ABC sub-directory with all its history preserved.