Git merge submodule into parent tree cleanly and preserving commit history

Question

I have a repository with two submodules that I want to convert into a single project. Many answers involve scripts, and some seem to be overcomplicated.

[submodule "site"]
    path = wp-content/themes/site
    url = https://[email protected]/ajf-/site.git
    fetchRecurseSubmodules = true
    ignore = all
[submodule "wpsite"]
    path = wp-content/themes/wpsite
    url = https://[email protected]/ajf-/wpsite.git
    fetchRecurseSubmodules = true
    ignore = all

Is there an officially supported / documented way to merge these submodules into the parent repository?

Nonsense

Possible duplicate of un-submodule a git submodule – Philipines

Good article; medium.com/walkme-engineering/… – Bubb

Answer 1

The best approach is to do subtree merging.

First, remove the submodules and related configuration from your superproject; Edit your .gitmodules file to remove the submodules affected, or delete the file entirely if you intend to merge all submodules. Delete the submodule directories as well.

Next, add the submodule repositories as proper remotes to your superproject:

git remote add site https://[email protected]/ajf-/site.git
git remote add wpsite https://[email protected]/ajf-/wpsite.git

Then, fetch the remotes:

git fetch --all

Now, check out the branches that you want to graft to your main project from each sub-project:

git checkout -b site-branch site/some_branch
git checkout -b wpsite-branch wpsite/some_other_branch

After that, return to the master branch, or to the branch where you want to create the combined superproject:

git checkout master

If you want to create a (possibly temporary) extra branch for the operation, do this instead:

git checkout -b new-superproject master

You're now ready to merge the module branches as subtrees with your main project (in the master branch in this example):

git merge --strategy=ours --no-commit --allow-unrelated-histories site-branch
git read-tree --prefix=site/ -u site-branch
git commit -m "Merge 'site' from submodule"

Repeat this step for all branches created from the submodules. Check the final result with gitk --all or another history visualizer.

If you do not want to keep the history of the submodules, then only do the read-tree step and omit the merge/commit steps.

Since you want to convert into a single project, you're not going to update the subprojects independently, so I'm not going to describe how that works.

You can read up on this in the chapter on advanced merging from Pro Git. Subtree merging is explained at the end of the chapter, but omits the step that creates a proper merge commit, so it will not preserve history of the submodule.

Outlandish

I had a problem with this method, see #23447957 – Nonsense

@AlainJacometForte: The question you link to is 410 Gone. (404ing). Did you turn out to be mistaken? – Concertize

@Concertize #1760087 maybe? – Rubi

pretty elegant solution! – Beller

I think you should probably add git checkout master prior to the read-tree step – Thigh

Note that this answer (at least for me) does not preserve the history of the submodules. – Flameout

@StephenHandley I have edited the post to reflect your suggestion. – Outlandish

I updated the answer to include a merge commit which will preserve history from the child repositories. – Outlandish

Answer 2

A bit late to the party, but for those still looking for help, checkout these:

1. https://gitirc.eu/howto/using-merge-subtree.html
2. https://help.github.com/en/github/using-git/about-git-subtree-merges

Below is a near verbatim copy of the first post:

1. git remote add -f Bproject /path/to/B
2. git merge --strategy=ours --no-commit --allow-unrelated-histories Bproject/master
3. git read-tree --prefix=dir-B/ -u Bproject/master
4. git commit -m "Merge B project as our subdirectory"

5. git pull -s subtree Bproject master

Explanation:

1. name the other project "Bproject", and fetch.
2. prepare for the later step to record the result as a merge.
3. read "master" branch of Bproject to the subdirectory "dir-B".
4. record the merge result.

5. pull in subsequent update from Bproject using "subtree" merge

As an alternative before step 4, you might want to update .gitmodules file, or just remove it:

3.1 git rm --cached .gitmodules

With this the history from the submodule is well preserved.

Derive

This is a very good answer. I should update my answer to reflect your suggestion of adding the merge step to preserve history. – Outlandish

This creates a subtree, instead of adding the history to the actual repository as a branch which is being merged in. – Getty

Answer 3

Although the merge solution described in other answers works, it creates a single big merge commit which makes it difficult to track the submodule history.

Here is my alternative solution, which uses rebase instead of merge when merging the submodule. This solution is not as safe and straightforward as the merge solution, but in my opinion creates nicer history. Also, this solution may fail if you have file name conflicts between the parent and submodule repository (see the step 5 for details):

1. Name the submodule Bproject and fetch it to the parent repository:

   git remote add -f Bproject /path/to/B
   

2. Switch to the submodule:

   git checkout Bproject
   

3. (Optional) Filter the branch in order to preserve only files you want:

   git filter-branch --subdirectory-filter requested/subdirectory
   

   The command above filters requested/subdirectory subdirectory from the submodule and makes a new repository root from it. This step helped me to avoid conflicts of well-known files like .gitignore and .gitmodules by simply removing them from the submodule history.

4. (Optional) Move the submodule files to the future location in the parent repository. If the Bproject files are supposed to be located in the src/Bproject subdirectory, create that directory and move the files in.

   mkdir -p src/Bproject
   git mv *.* src/Bproject
   git commit -m "Move Bproject files to the src/Bproject subdirectory."
   

   After this step the repository looks like a "file system layer" which can be put over the master branch of the parent project.

5. Finally, rebase the Bproject branch over the master branch of the parent project. This is the most critical step because it will work only if there are no file name conflicts between the master and Bproject branch.

   git rebase --interactive master
   

   E.g. if you have a README.md file in the root of both repositories, this will certainly result in a rebase conflict. In this case, you have to either use optional step 3 to get rid of such file, resolve the conflict manually or just don't use the rebasing strategy and get back to the merge strategy.

6. If the rebasing went well, you can use git log to check the resulting Git history. You will see your master commits followed by the Bproject commits, and the commit dates will no longer be chronological. If you moved the Bproject files, don't forget to use --follow command-line switch to follow the file renaming.

7. If you are satisfied with the result, merge it to the master branch. You will get a trivial fast-forward merge without the merge commit:

   git checkout master
   git merge Bproject
   

Koblick

Answer 4

In the end I did something simpler.

1. I entered the submodule repository and moved everything to a subdirectory.

   mkdir subdirectory
   git mv -k * .* subdirectory/
   git commit -am "Prepare to merge repositories"
   

2. I then went back to the parent repository, removed the submodule, committed and merged the child repository into it.

   git remote add submodule ../submodule
   git merge --allow-unrelated-histories submodule/master 
   

And that's it. The history is perfect and everything is in a single repository.

Credit to https://medium.com/walkme-engineering/how-to-merge-a-git-submodule-into-its-main-repository-d83a215a319c

Fervor