Consider the following scenario:
I have developed a small experimental project A in its own Git repo. It has now matured, and I'd like A to be part of larger project B, which has its own big repository. I'd now like to add A as a subdirectory of B.
How do I merge A into B, without losing history on any side?
A single branch of another repository can be easily placed under a subdirectory retaining its history. For example:
git subtree add --prefix=rails git://github.com/rails/rails.git master
This will appear as a single commit where all files of Rails master branch are added into "rails" directory. However the commit's title contains a reference to the old history tree:
Add 'rails/' from commit
<rev>
Where <rev> is a SHA-1 commit hash. You can still see the history, blame some changes.
git log <rev>
git blame <rev> -- README.md
Note that you can't see the directory prefix from here since this is an actual old branch left intact. You should treat this like a usual file move commit: you will need an extra jump when reaching it.
# finishes with all files added at once commit
git log rails/README.md
# then continue from original tree
git log <rev> -- README.md
There are more complex solutions like doing this manually or rewriting the history as described in other answers.
The git-subtree command is a part of official git-contrib, some packet managers install it by default (OS X Homebrew). But you might have to install it by yourself in addition to git.
git co v1.7.11.3 with ... v1.8.3). –
Tragedian git log rails/somefile will not display that file's commits history except the merge commit. As @Oliva suggested, check Greg Hewgill's answer. And you might need to use git filter-branch on the repo you want to include. –
Vaughan chmod +x /usr/share/doc/git/contrib/subtree/git-subtree.sh then ln -s /usr/share/doc/git/contrib/subtree/git-subtree.sh /usr/lib/git-core/git-subtree as root –
Lunneta git subtree may not do what you think! See here for a more complete solution. –
Vietnamese If you want to merge project-a into project-b:
cd path/to/project-b
git remote add project-a /path/to/project-a
git fetch project-a --tags
git merge --allow-unrelated-histories project-a/master # or any branch you want to merge
git remote remove project-a
Taken from: git merge different repositories?
This method worked pretty well for me, it's shorter and in my opinion a lot cleaner.
In case you want to put project-a into a subdirectory, you can use git-filter-repo (filter-branch is discouraged). Run the following commands before the commands above:
cd path/to/project-a
git filter-repo --to-subdirectory-filter project-a
An example of merging 2 big repositories, putting one of them into a subdirectory: https://gist.github.com/x-yuri/9890ab1079cf4357d6f269d073fd9731
Note:
The --allow-unrelated-histories parameter only exists since git >= 2.9. See Git - git merge Documentation / --allow-unrelated-histories
Update:
Added --tags as suggested by @jstadler in order to keep tags.
fatal: This operation must be run in a work tree, i want to merge two bare git repository. –
Thunderpeal git mv source-dir/ dest/new-source-dir –
Dremadremann git rebase -i [last commit of project-b] to make the logs look better. –
Trapp git merge step fails here with fatal: refusing to merge unrelated histories; --allow-unrelated-histories fixes that as explained in the docs. –
Ankylosis error: unknown option 'allow-unrelated-histories' –
Exclaim --allow-unrelated-histories worked –
Exclaim git fetch /path/to/project-a master; git merge --allow-unrelated-histories FETCH_HEAD. –
Heinrick git merge -s ours --allow-unrelated-histories project-a/master and it works perfectly! –
Brachiopod Project A: to all of project-a's commit messages, so that the context of the messages would be clear from within project-b. –
Pallbearer git fetch project-a --tags –
Trahern git rebase project-a/master to have a linear history. –
Gregggreggory project-a's tags? Because project-a and project-b had similar tags and now I got this ! [rejected] 0.1.0 -> 0.1.0 (would clobber existing tag) when running git fetch project-a --tags –
Aleciaaleck --tags to git fetch since that is the default. I can trace it back to at least year 2007 (git-1.5.0). @Shinebayar git filter-repo --tag-rename :project-a- See this for an example. –
Incase git filter-repo is not available with the standard Apple development tools on Mojave 10.14.6. The git version on this platform is 2.21.1 –
Agley git remote remove project-a may not be important for everyone. Someone may need to stay in sync with project-a. –
Pip get-filter-repo => sudo apt-get install git-filter-repo –
Hugibert Here are two possible solutions:
Either copy repository A into a separate directory in larger project B, or (perhaps better) clone repository A into a subdirectory in project B. Then use git submodule to make this repository a submodule of a repository B.
This is a good solution for loosely-coupled repositories, where development in repository A continues, and the major portion of development is a separate stand-alone development in A. See also SubmoduleSupport and GitSubmoduleTutorial pages on Git Wiki.
You can merge repository A into a subdirectory of a project B using the subtree merge strategy. This is described in Subtree Merging and You by Markus Prinz.
git remote add -f Bproject /path/to/B
git merge -s ours --allow-unrelated-histories --no-commit Bproject/master
git read-tree --prefix=dir-B/ -u Bproject/master
git commit -m "Merge B project as our subdirectory"
git pull -s subtree Bproject master
(Option --allow-unrelated-histories is needed for Git >= 2.9.0.)
Or you can use git subtree tool (repository on GitHub) by apenwarr (Avery Pennarun), announced for example in his blog post A new alternative to Git submodules: git subtree.
I think in your case (A is to be part of larger project B) the correct solution would be to use subtree merge.
git log dir-B/somefile won't show anything except the one merge. See Greg Hewgill's answer references this important issue. –
Oliva git subtree split tool is now in mainline git as of 1.7.11 –
Royall git stree – supposedly better than subtree: medium.com/@porteneuve/mastering-git-subtrees-943d29a798ec. –
Amersham A single branch of another repository can be easily placed under a subdirectory retaining its history. For example:
git subtree add --prefix=rails git://github.com/rails/rails.git master
This will appear as a single commit where all files of Rails master branch are added into "rails" directory. However the commit's title contains a reference to the old history tree:
Add 'rails/' from commit
<rev>
Where <rev> is a SHA-1 commit hash. You can still see the history, blame some changes.
git log <rev>
git blame <rev> -- README.md
Note that you can't see the directory prefix from here since this is an actual old branch left intact. You should treat this like a usual file move commit: you will need an extra jump when reaching it.
# finishes with all files added at once commit
git log rails/README.md
# then continue from original tree
git log <rev> -- README.md
There are more complex solutions like doing this manually or rewriting the history as described in other answers.
The git-subtree command is a part of official git-contrib, some packet managers install it by default (OS X Homebrew). But you might have to install it by yourself in addition to git.
git co v1.7.11.3 with ... v1.8.3). –
Tragedian git log rails/somefile will not display that file's commits history except the merge commit. As @Oliva suggested, check Greg Hewgill's answer. And you might need to use git filter-branch on the repo you want to include. –
Vaughan chmod +x /usr/share/doc/git/contrib/subtree/git-subtree.sh then ln -s /usr/share/doc/git/contrib/subtree/git-subtree.sh /usr/lib/git-core/git-subtree as root –
Lunneta git subtree may not do what you think! See here for a more complete solution. –
Vietnamese The submodule approach is good if you want to maintain the project separately. However, if you really want to merge both projects into the same repository, then you have a bit more work to do.
The first thing would be to use git filter-branch to rewrite the names of everything in the second repository to be in the subdirectory where you would like them to end up. So instead of foo.c, bar.html, you would have projb/foo.c and projb/bar.html.
Then, you should be able to do something like the following:
git remote add projb [wherever]
git pull projb
The git pull will do a git fetch followed by a git merge. There should be no conflicts, if the repository you're pulling to does not yet have a projb/ directory.
Further searching indicates that something similar was done to merge gitk into git. Junio C Hamano writes about it here: http://www.mail-archive.com/[email protected]/msg03395.html
git pull step may complain about the inability to find a common ancestor. Some other details may need to be ironed out too. –
Overmodest git filter-branch to achieve this. In the man page it says about the opposite way around: making subdir/ become the root, but not the other way around. –
Oliva git filter-branch example to move the root into a subdirectory: git filter-branch -f --index-filter 'git ls-files -s | awk -v prefix="SUBDIRECTORY/" "{ sub(/\t/, \"\t\" prefix); print}" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD (replace SUBDIRECTORY with actual directory name) –
Sunder git-subtree is nice, but it is probably not the one you want.
For example, if projectA is the directory created in B, after git subtree,
git log projectA
lists only one commit: the merge. The commits from the merged project are for different paths, so they don't show up.
Greg Hewgill's answer comes closest, although it doesn't actually say how to rewrite the paths.
The solution is surprisingly simple.
(1) In A,
PREFIX=projectA #adjust this
git filter-branch --index-filter '
git ls-files -s |
sed "s,\t,&'"$PREFIX"'/," |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
' HEAD
Note: This rewrites history; you may want to first make a backup of A.
Note Bene: You have to modify the substitute script inside the sed command in the case that you use non-ascii characters (or white characters) in file names or path. In that case the file location inside a record produced by "ls-files -s" begins with quotation mark.
(2) Then in B, run
git pull path/to/A
Voila! You have a projectA directory in B. If you run git log projectA, you will see all commits from A.
In my case, I wanted two subdirectories, projectA and projectB. In that case, I did step (1) to B as well.
"$GIT_INDEX_FILE" must be quoted (twice), otherwise your method will fail if e.g. the path contains spaces. –
Sakmar Ctrl-V <tab> –
Rebec git pull step. First, "fatal: 'projectB' does not appear to be a git repository fatal: Could not read from remote repository." This was solved by using the absolute path, i.e. /path/to/A. Next, I got: fatal: refusing to merge unrelated histories. This seems to happen since git v2.9, and can be solved by specifying git pull --allow-unrelated-histories /path/to/A. –
Julie git pull --allow-unrelated-histories /path/to/A instead. –
Archiphoneme git log doesn't show it when looking at the history of just that sub-directory. Subtree doesn't rewrite the commits so the trees of the merged commits are still at the "root". At the end of the day, its a trade-off -- maintain the history (and commit hashes) "as-is" with subtree, but recognize that older commits maintain their old trees, or rewrite the trees of older commits, so that the history appears as if the repo has always existed with the current structure. –
Abutilon If both repositories have same kind of files (like two Rails repositories for different projects), you can fetch data of the secondary repository to your current repository:
git fetch git://repository.url/repo.git master:branch_name
and then merge it to current repository:
git merge --allow-unrelated-histories branch_name
If your Git version is smaller than 2.9, remove --allow-unrelated-histories.
After this, conflicts may occur. You can resolve them for example with git mergetool. kdiff3 can be used solely with keyboard, so 5 conflict file takes when reading the code just few minutes.
Remember to finish the merge:
git commit
git pull --allow-unrelated-histories? –
Naara I kept losing history when using merge, so I ended up using rebase since in my case the two repositories are different enough not to end up merging at every commit:
git clone git@gitorious/projA.git projA
git clone git@gitorious/projB.git projB
cd projB
git remote add projA ../projA/
git fetch projA
git rebase projA/master HEAD
=> resolve conflicts, then continue, as many times as needed...
git rebase --continue
Doing this leads to one project having all commits from projA followed by commits from projB
In my case, I had a my-plugin repository and a main-project repository, and I wanted to pretend that my-plugin had always been developed in the plugins subdirectory of main-project.
Basically, I rewrote the history of the my-plugin repository so that it appeared all development took place in the plugins/my-plugin subdirectory. Then, I added the development history of my-plugin into the main-project history, and merged the two trees together. Since there was no plugins/my-plugin directory already present in the main-project repository, this was a trivial no-conflicts merge. The resulting repository contained all history from both original projects, and had two roots.
$ cp -R my-plugin my-plugin-dirty
$ cd my-plugin-dirty
$ git filter-branch -f --tree-filter "zsh -c 'setopt extended_glob && setopt glob_dots && mkdir -p plugins/my-plugin && (mv ^(.git|plugins) plugins/my-plugin || true)'" -- --all
$ cd ../main-project
$ git checkout master
$ git remote add --fetch my-plugin ../my-plugin-dirty
$ git merge my-plugin/master --allow-unrelated-histories
$ cd ..
$ rm -rf my-plugin-dirty
First, create a copy of the my-plugin repository, because we're going to be rewriting the history of this repository.
Now, navigate to the root of the my-plugin repository, check out your main branch (probably master), and run the following command. Of course, you should substitute for my-plugin and plugins whatever your actual names are.
$ git filter-branch -f --tree-filter "zsh -c 'setopt extended_glob && setopt glob_dots && mkdir -p plugins/my-plugin && (mv ^(.git|plugins) plugins/my-plugin || true)'" -- --all
Now for an explanation. git filter-branch --tree-filter (...) HEAD runs the (...) command on every commit that is reachable from HEAD. Note that this operates directly on the data stored for each commit, so we don't have to worry about notions of "working directory", "index", "staging", and so on.
If you run a filter-branch command that fails, it will leave behind some files in the .git directory and the next time you try filter-branch it will complain about this, unless you supply the -f option to filter-branch.
As for the actual command, I didn't have much luck getting bash to do what I wanted, so instead I use zsh -c to make zsh execute a command. First I set the extended_glob option, which is what enables the ^(...) syntax in the mv command, as well as the glob_dots option, which allows me to select dotfiles (such as .gitignore) with a glob (^(...)).
Next, I use the mkdir -p command to create both plugins and plugins/my-plugin at the same time.
Finally, I use the zsh "negative glob" feature ^(.git|plugins) to match all files in the root directory of the repository except for .git and the newly created my-plugin folder. (Excluding .git might not be necessary here, but trying to move a directory into itself is an error.)
In my repository, the initial commit did not include any files, so the mv command returned an error on the initial commit (since nothing was available to move). Therefore, I added a || true so that git filter-branch would not abort.
The --all option tells filter-branch to rewrite the history for all branches in the repository, and the extra -- is necessary to tell git to interpret it as a part of the option list for branches to rewrite, instead of as an option to filter-branch itself.
Now, navigate to your main-project repository and check out whatever branch you want to merge into. Add your local copy of the my-plugin repository (with its history modified) as a remote of main-project with:
$ git remote add --fetch my-plugin $PATH_TO_MY_PLUGIN_REPOSITORY
You will now have two unrelated trees in your commit history, which you can visualize nicely using:
$ git log --color --graph --decorate --all
To merge them, use:
$ git merge my-plugin/master --allow-unrelated-histories
Note that in pre-2.9.0 Git, the --allow-unrelated-histories option does not exist. If you are using one of these versions, just omit the option: the error message that --allow-unrelated-histories prevents was also added in 2.9.0.
You should not have any merge conflicts. If you do, it probably means that either the filter-branch command did not work correctly or there was already a plugins/my-plugin directory in main-project.
Make sure to enter an explanatory commit message for any future contributors wondering what hackery was going on to make a repository with two roots.
You can visualize the new commit graph, which should have two root commits, using the above git log command. Note that only the master branch will be merged. This means that if you have important work on other my-plugin branches that you want to merge into the main-project tree, you should refrain from deleting the my-plugin remote until you have done these merges. If you don't, then the commits from those branches will still be in the main-project repository, but some will be unreachable and susceptible to eventual garbage collection. (Also, you will have to refer to them by SHA, because deleting a remote removes its remote-tracking branches.)
Optionally, after you have merged everything you want to keep from my-plugin, you can remove the my-plugin remote using:
$ git remote remove my-plugin
You can now safely delete the copy of the my-plugin repository whose history you changed. In my case, I also added a deprecation notice to the real my-plugin repository after the merge was complete and pushed.
Tested on Mac OS X El Capitan with git --version 2.9.0 and zsh --version 5.2. Your mileage may vary.
References:
--allow-unrelated-histories are coming from? –
Parry man git-merge. By default, git merge command refuses to merge histories that do not share a common ancestor. This option can be used to override this safety when merging histories of two projects that started their lives independently. As that is a very rare occasion, no configuration variable to enable this by default exists and will not be added. –
Korten git version 2.7.2.windows.1? –
Parry Ref 'refs/tags/v1.0' was rewritten WARNING: You said to rewrite tagged commits, but not the corresponding tag. WARNING: Perhaps use '--tag-name-filter cat' to rewrite the tag. in one of my repos. When I checkout the tag, all the contents are in main directory. –
Elsworth --tag-name-filter cat to the filter-branch command rewrites the tags correctly. So the third command should read $ git filter-branch -f --tag-name-filter cat --tree-filter "zsh -c 'setopt extended_glob && setopt glob_dots && mkdir -p plugins/my-plugin && (mv ^(.git|plugins) plugins/my-plugin || true)'" -- --all –
Elsworth I have gathered a lot of information here on Stack OverFlow, etc., and have manage to put a script together which solves the problem for me.
The caveat is that it only takes into account the 'develop' branch of each repository and merges it into a separate directory in a completely new repository.
Tags and other branches are ignored - this might not be what you want.
The script even handles feature branches and tags - renaming them in the new project so you know where they came from.
#!/bin/bash
#
################################################################################
## Script to merge multiple git repositories into a new repository
## - The new repository will contain a folder for every merged repository
## - The script adds remotes for every project and then merges in every branch
## and tag. These are renamed to have the origin project name as a prefix
##
## Usage: mergeGitRepositories.sh <new_project> <my_repo_urls.lst>
## - where <new_project> is the name of the new project to create
## - and <my_repo_urls.lst> is a file contaning the URLs to the respositories
## which are to be merged on separate lines.
##
## Author: Robert von Burg
## [email protected]
##
## Version: 0.3.2
## Created: 2018-02-05
##
################################################################################
#
# disallow using undefined variables
shopt -s -o nounset
# Script variables
declare SCRIPT_NAME="${0##*/}"
declare SCRIPT_DIR="$(cd ${0%/*} ; pwd)"
declare ROOT_DIR="$PWD"
IFS=$'\n'
# Detect proper usage
if [ "$#" -ne "2" ] ; then
echo -e "ERROR: Usage: $0 <new_project> <my_repo_urls.lst>"
exit 1
fi
## Script variables
PROJECT_NAME="${1}"
PROJECT_PATH="${ROOT_DIR}/${PROJECT_NAME}"
TIMESTAMP="$(date +%s)"
LOG_FILE="${ROOT_DIR}/${PROJECT_NAME}_merge.${TIMESTAMP}.log"
REPO_FILE="${2}"
REPO_URL_FILE="${ROOT_DIR}/${REPO_FILE}"
# Script functions
function failed() {
echo -e "ERROR: Merging of projects failed:"
echo -e "ERROR: Merging of projects failed:" >>${LOG_FILE} 2>&1
echo -e "$1"
exit 1
}
function commit_merge() {
current_branch="$(git symbolic-ref HEAD 2>/dev/null)"
if [[ ! -f ".git/MERGE_HEAD" ]] ; then
echo -e "INFO: No commit required."
echo -e "INFO: No commit required." >>${LOG_FILE} 2>&1
else
echo -e "INFO: Committing ${sub_project}..."
echo -e "INFO: Committing ${sub_project}..." >>${LOG_FILE} 2>&1
if ! git commit -m "[Project] Merged branch '$1' of ${sub_project}" >>${LOG_FILE} 2>&1 ; then
failed "Failed to commit merge of branch '$1' of ${sub_project} into ${current_branch}"
fi
fi
}
# Make sure the REPO_URL_FILE exists
if [ ! -e "${REPO_URL_FILE}" ] ; then
echo -e "ERROR: Repo file ${REPO_URL_FILE} does not exist!"
exit 1
fi
# Make sure the required directories don't exist
if [ -e "${PROJECT_PATH}" ] ; then
echo -e "ERROR: Project ${PROJECT_NAME} already exists!"
exit 1
fi
# create the new project
echo -e "INFO: Logging to ${LOG_FILE}"
echo -e "INFO: Creating new git repository ${PROJECT_NAME}..."
echo -e "INFO: Creating new git repository ${PROJECT_NAME}..." >>${LOG_FILE} 2>&1
echo -e "===================================================="
echo -e "====================================================" >>${LOG_FILE} 2>&1
cd ${ROOT_DIR}
mkdir ${PROJECT_NAME}
cd ${PROJECT_NAME}
git init
echo "Initial Commit" > initial_commit
# Since this is a new repository we need to have at least one commit
# thus were we create temporary file, but we delete it again.
# Deleting it guarantees we don't have conflicts later when merging
git add initial_commit
git commit --quiet -m "[Project] Initial Master Repo Commit"
git rm --quiet initial_commit
git commit --quiet -m "[Project] Initial Master Repo Commit"
echo
# Merge all projects into the branches of this project
echo -e "INFO: Merging projects into new repository..."
echo -e "INFO: Merging projects into new repository..." >>${LOG_FILE} 2>&1
echo -e "===================================================="
echo -e "====================================================" >>${LOG_FILE} 2>&1
for url in $(cat ${REPO_URL_FILE}) ; do
if [[ "${url:0:1}" == '#' ]] ; then
continue
fi
# extract the name of this project
export sub_project=${url##*/}
sub_project=${sub_project%*.git}
echo -e "INFO: Project ${sub_project}"
echo -e "INFO: Project ${sub_project}" >>${LOG_FILE} 2>&1
echo -e "----------------------------------------------------"
echo -e "----------------------------------------------------" >>${LOG_FILE} 2>&1
# Fetch the project
echo -e "INFO: Fetching ${sub_project}..."
echo -e "INFO: Fetching ${sub_project}..." >>${LOG_FILE} 2>&1
git remote add "${sub_project}" "${url}"
if ! git fetch --tags --quiet ${sub_project} >>${LOG_FILE} 2>&1 ; then
failed "Failed to fetch project ${sub_project}"
fi
# add remote branches
echo -e "INFO: Creating local branches for ${sub_project}..."
echo -e "INFO: Creating local branches for ${sub_project}..." >>${LOG_FILE} 2>&1
while read branch ; do
branch_ref=$(echo $branch | tr " " "\t" | cut -f 1)
branch_name=$(echo $branch | tr " " "\t" | cut -f 2 | cut -d / -f 3-)
echo -e "INFO: Creating branch ${branch_name}..."
echo -e "INFO: Creating branch ${branch_name}..." >>${LOG_FILE} 2>&1
# create and checkout new merge branch off of master
if ! git checkout -b "${sub_project}/${branch_name}" master >>${LOG_FILE} 2>&1 ; then failed "Failed preparing ${branch_name}" ; fi
if ! git reset --hard ; then failed "Failed preparing ${branch_name}" >>${LOG_FILE} 2>&1 ; fi
if ! git clean -d --force ; then failed "Failed preparing ${branch_name}" >>${LOG_FILE} 2>&1 ; fi
# Merge the project
echo -e "INFO: Merging ${sub_project}..."
echo -e "INFO: Merging ${sub_project}..." >>${LOG_FILE} 2>&1
if ! git merge --allow-unrelated-histories --no-commit "remotes/${sub_project}/${branch_name}" >>${LOG_FILE} 2>&1 ; then
failed "Failed to merge branch 'remotes/${sub_project}/${branch_name}' from ${sub_project}"
fi
# And now see if we need to commit (maybe there was a merge)
commit_merge "${sub_project}/${branch_name}"
# relocate projects files into own directory
if [ "$(ls)" == "${sub_project}" ] ; then
echo -e "WARN: Not moving files in branch ${branch_name} of ${sub_project} as already only one root level."
echo -e "WARN: Not moving files in branch ${branch_name} of ${sub_project} as already only one root level." >>${LOG_FILE} 2>&1
else
echo -e "INFO: Moving files in branch ${branch_name} of ${sub_project} so we have a single directory..."
echo -e "INFO: Moving files in branch ${branch_name} of ${sub_project} so we have a single directory..." >>${LOG_FILE} 2>&1
mkdir ${sub_project}
for f in $(ls -a) ; do
if [[ "$f" == "${sub_project}" ]] ||
[[ "$f" == "." ]] ||
[[ "$f" == ".." ]] ; then
continue
fi
git mv -k "$f" "${sub_project}/"
done
# commit the moving
if ! git commit --quiet -m "[Project] Move ${sub_project} files into sub directory" ; then
failed "Failed to commit moving of ${sub_project} files into sub directory"
fi
fi
echo
done < <(git ls-remote --heads ${sub_project})
# checkout master of sub probject
if ! git checkout "${sub_project}/master" >>${LOG_FILE} 2>&1 ; then
failed "sub_project ${sub_project} is missing master branch!"
fi
# copy remote tags
echo -e "INFO: Copying tags for ${sub_project}..."
echo -e "INFO: Copying tags for ${sub_project}..." >>${LOG_FILE} 2>&1
while read tag ; do
tag_ref=$(echo $tag | tr " " "\t" | cut -f 1)
tag_name_unfixed=$(echo $tag | tr " " "\t" | cut -f 2 | cut -d / -f 3)
# hack for broken tag names where they are like 1.2.0^{} instead of just 1.2.0
tag_name="${tag_name_unfixed%%^*}"
tag_new_name="${sub_project}/${tag_name}"
echo -e "INFO: Copying tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}..."
echo -e "INFO: Copying tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}..." >>${LOG_FILE} 2>&1
if ! git tag "${tag_new_name}" "${tag_ref}" >>${LOG_FILE} 2>&1 ; then
echo -e "WARN: Could not copy tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}"
echo -e "WARN: Could not copy tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}" >>${LOG_FILE} 2>&1
fi
done < <(git ls-remote --tags --refs ${sub_project})
# Remove the remote to the old project
echo -e "INFO: Removing remote ${sub_project}..."
echo -e "INFO: Removing remote ${sub_project}..." >>${LOG_FILE} 2>&1
git remote rm ${sub_project}
echo
done
# Now merge all project master branches into new master
git checkout --quiet master
echo -e "INFO: Merging projects master branches into new repository..."
echo -e "INFO: Merging projects master branches into new repository..." >>${LOG_FILE} 2>&1
echo -e "===================================================="
echo -e "====================================================" >>${LOG_FILE} 2>&1
for url in $(cat ${REPO_URL_FILE}) ; do
if [[ ${url:0:1} == '#' ]] ; then
continue
fi
# extract the name of this project
export sub_project=${url##*/}
sub_project=${sub_project%*.git}
echo -e "INFO: Merging ${sub_project}..."
echo -e "INFO: Merging ${sub_project}..." >>${LOG_FILE} 2>&1
if ! git merge --allow-unrelated-histories --no-commit "${sub_project}/master" >>${LOG_FILE} 2>&1 ; then
failed "Failed to merge branch ${sub_project}/master into master"
fi
# And now see if we need to commit (maybe there was a merge)
commit_merge "${sub_project}/master"
echo
done
# Done
cd ${ROOT_DIR}
echo -e "INFO: Done."
echo -e "INFO: Done." >>${LOG_FILE} 2>&1
echo
exit 0
You can also get it from http://paste.ubuntu.com/11732805
First create a file with the URL to each repository, e.g.:
[email protected]:eitchnet/ch.eitchnet.parent.git
[email protected]:eitchnet/ch.eitchnet.utils.git
[email protected]:eitchnet/ch.eitchnet.privilege.git
Then call the script giving a name of the project and the path to the script:
./mergeGitRepositories.sh eitchnet_test eitchnet.lst
The script itself has a lot of comments which should explain what it does.
I've been trying to do the same thing for days, I am using git 2.7.2. Subtree does not preserve the history.
You can use this method if you will not be using the old project again.
I would suggest that you branch B first and work in the branch.
Here are the steps without branching:
cd B
# You are going to merge A into B, so first move all of B's files into a sub dir
mkdir B
# Move all files to B, till there is nothing in the dir but .git and B
git mv <files> B
git add .
git commit -m "Moving content of project B in preparation for merge from A"
# Now merge A into B
git remote add -f A <A repo url>
git merge A/<branch>
mkdir A
# move all the files into subdir A, excluding .git
git mv <files> A
git commit -m "Moved A into subdir"
# Move B's files back to root
git mv B/* ./
rm -rf B
git commit -m "Reset B to original state"
git push
If you now log any of the files in subdir A you will get the full history
git log --follow A/<file>
This was the post that help me do this:
If you want to put the files from a branch in repo B in a subtree of repo A and also preserve the history, keep reading. (In the example below, I am assuming that we want repo B's master branch merged into repo A's master branch.)
In repo A, first do the following to make repo B available:
git remote add B ../B # Add repo B as a new remote.
git fetch B
Now we create a brand new branch (with only one commit) in repo A that we call new_b_root. The resulting commit will have the files that were committed in the first commit of repo B's master branch but put in a subdirectory called path/to/b-files/.
git checkout --orphan new_b_root master
git rm -rf . # Remove all files.
git cherry-pick -n `git rev-list --max-parents=0 B/master`
mkdir -p path/to/b-files
git mv README path/to/b-files/
git commit --date="$(git log --format='%ai' $(git rev-list --max-parents=0 B/master))"
Explanation: The --orphan option to the checkout command checks out the files from A's master branch but doesn't create any commit. We could have selected any commit because next we clear out all the files anyway. Then, without committing yet (-n), we cherry-pick the first commit from B's master branch. (The cherry-pick preserves the original commit message which a straight checkout doesn't seem to do.) Then we create the subtree where we want to put all files from repo B. We then have to move all files that were introduced in the cherry-pick to the subtree. In the example above, there's only a README file to move. Then we commit our B-repo root commit, and, at the same time, we also preserve the timestamp of the original commit.
Now, we'll create a new B/master branch on top of the newly created new_b_root. We call the new branch b:
git checkout -b b B/master
git rebase -s recursive -Xsubtree=path/to/b-files/ new_b_root
Now, we merge our b branch into A/master:
git checkout master
git merge --allow-unrelated-histories --no-commit b
git commit -m 'Merge repo B into repo A.'
Finally, you can remove the B remote and temporary branches:
git remote remove B
git branch -D new_b_root b
The final graph will have a structure like this:
I know it's long after the fact, but I wasn't happy with the other answers I found here, so I wrote this:
me=$(basename $0)
TMP=$(mktemp -d /tmp/$me.XXXXXXXX)
echo
echo "building new repo in $TMP"
echo
sleep 1
set -e
cd $TMP
mkdir new-repo
cd new-repo
git init
cd ..
x=0
while [ -n "$1" ]; do
repo="$1"; shift
git clone "$repo"
dirname=$(basename $repo | sed -e 's/\s/-/g')
if [[ $dirname =~ ^git:.*\.git$ ]]; then
dirname=$(echo $dirname | sed s/.git$//)
fi
cd $dirname
git remote rm origin
git filter-branch --tree-filter \
"(mkdir -p $dirname; find . -maxdepth 1 ! -name . ! -name .git ! -name $dirname -exec mv {} $dirname/ \;)"
cd ..
cd new-repo
git pull --no-commit ../$dirname
[ $x -gt 0 ] && git commit -m "merge made by $me"
cd ..
x=$(( x + 1 ))
done
if [[ $dirname =~ ^.*\.git$ ]]; then –
Escobar If you're trying to simply glue two repositories together, submodules and subtree merges are the wrong tool to use because they don't preserve all of the file history (as people have noted on other answers). See this answer here for the simple and correct way to do this.
I had a similar challenge, but in my case, we had developed one version of the codebase in repo A, then cloned that into a new repo, repo B, for the new version of the product. After fixing some bugs in repo A, we needed to FI the changes into repo B. Ended up doing the following:
Worked a treat :)
Merging 2 repos
git clone ssh://<project-repo> project1
cd project1
git remote add -f project2 project2
git merge --allow-unrelated-histories project2/master
git remote rm project2
delete the ref to avoid errors
git update-ref -d refs/remotes/project2/master
Similar to @Smar but uses file system paths, set in PRIMARY and SECONDARY:
PRIMARY=~/Code/project1
SECONDARY=~/Code/project2
cd $PRIMARY
git remote add test $SECONDARY && git fetch test
git merge test/master
Then you manually merge.
(adapted from post by Anar Manafov)
When you want to merge three or more projects in a single commit, do the steps as described in the other answers (remote add -f, merge). Then, (soft) reset the index to old head (where no merge happened). Add all files (git add -A) and commit them (message "Merging projects A, B, C, and D into one project). This is now the commit-id of master.
Now, create .git/info/grafts with following content:
<commit-id of master> <list of commit ids of all parents>
Run git filter-branch -- head^..head head^2..head head^3..head. If you have more than three branches, just add as much head^n..head as you have branches. To update tags, append --tag-name-filter cat. Do not always add that, because this might cause a rewrite of some commits. For details see man page of filter-branch, search for "grafts".
Now, your last commit has the right parents associated.
To merge a A within B:
1) In the project A
git fast-export --all --date-order > /tmp/ProjectAExport
2) In the project B
git checkout -b projectA
git fast-import --force < /tmp/ProjectAExport
In this branch do all operations you need to do and commit them.
C) Then back to the master and a classical merge between the two branches:
git checkout master
git merge projectA
https://github.com/hraban/tomono as another mention of a script-based solution.
I am not the author but used it and it does the job.
One positive aspect is that you get all the branches and all the history into the final repo. For my repos (no duplicate folders in repos - actually, they came out of tfs2git migration) there were no conflicts and everything ran automated.
It is mainly used (see name) to create monorepos.
For Windows users: git bash can execute the .sh file. It comes with the standard git installation.
MONOREPO_NAME=A tomono --continue <<<"[email protected]:user/B.git B" –
Ruel There is a Copybara tool used by Google for more complex use cases - https://github.com/google/copybara
I merge projects slightly manually, which allows me to avoid needing to deal with merge conflicts.
first, copy in the files from the other project however you want them.
cp -R myotherproject newdirectory
git add newdirectory
next pull in the history
git fetch path_or_url_to_other_repo
tell git to merge in the history of last fetched thing
echo 'FETCH_HEAD' > .git/MERGE_HEAD
now commit however you normally would commit
git commit
I had to solve it as follows today: Project A was in bitbucket and Project B was in code commit .. both are the same projects but had to merge changes from A to B. (The trick is to create the same name branch in Project A, same as in Project B)
This function will clone remote repo into local repo dir, after merging all commits will be saved, git log will be show the original commits and proper paths:
function git-add-repo
{
repo="$1"
dir="$(echo "$2" | sed 's/\/$//')"
path="$(pwd)"
tmp="$(mktemp -d)"
remote="$(echo "$tmp" | sed 's/\///g'| sed 's/\./_/g')"
git clone "$repo" "$tmp"
cd "$tmp"
git filter-branch --index-filter '
git ls-files -s |
sed "s,\t,&'"$dir"'/," |
GIT_INDEX_FILE="$GIT_INDEX_FILE.new" git update-index --index-info &&
mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"
' HEAD
cd "$path"
git remote add -f "$remote" "file://$tmp/.git"
git pull "$remote/master"
git merge --allow-unrelated-histories -m "Merge repo $repo into master" --edit "$remote/master"
git remote remove "$remote"
rm -rf "$tmp"
}
How to use:
cd current/package
git-add-repo https://github.com/example/example dir/to/save
If make a little changes you can even move files/dirs of merged repo into different paths, for example:
repo="https://github.com/example/example"
path="$(pwd)"
tmp="$(mktemp -d)"
remote="$(echo "$tmp" | sed 's/\///g' | sed 's/\./_/g')"
git clone "$repo" "$tmp"
cd "$tmp"
GIT_ADD_STORED=""
function git-mv-store
{
from="$(echo "$1" | sed 's/\./\\./')"
to="$(echo "$2" | sed 's/\./\\./')"
GIT_ADD_STORED+='s,\t'"$from"',\t'"$to"',;'
}
# NOTICE! This paths used for example! Use yours instead!
git-mv-store 'public/index.php' 'public/admin.php'
git-mv-store 'public/data' 'public/x/_data'
git-mv-store 'public/.htaccess' '.htaccess'
git-mv-store 'core/config' 'config/config'
git-mv-store 'core/defines.php' 'defines/defines.php'
git-mv-store 'README.md' 'doc/README.md'
git-mv-store '.gitignore' 'unneeded/.gitignore'
git filter-branch --index-filter '
git ls-files -s |
sed "'"$GIT_ADD_STORED"'" |
GIT_INDEX_FILE="$GIT_INDEX_FILE.new" git update-index --index-info &&
mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"
' HEAD
GIT_ADD_STORED=""
cd "$path"
git remote add -f "$remote" "file://$tmp/.git"
git pull "$remote/master"
git merge --allow-unrelated-histories -m "Merge repo $repo into master" --edit "$remote/master"
git remote remove "$remote"
rm -rf "$tmp"
Notices
Paths replaces via sed, so make sure it moved in proper paths after merging.
The --allow-unrelated-histories parameter only exists since git >= 2.9.
Given command is the best possible solution I suggest.
git subtree add --prefix=MY_PROJECT git://github.com/project/my_project.git master
In addition to all the answers using remote add -> fetch -> merge strategy: if you want to preserve tags from the other repository but don't want to spill them all into a common namespace (and possibly get collisions) you might want to change the fetch command a little bit:
git fetch --no-tags other_repo
git fetch --no-tags other_repo 'refs/tags/*:refs/tags/other_repo/*'
First command fetches all the branches as usual, but omits tags attached to commits, the second one also omits the usual tag fetching mechanism (git help fetch for more), and fetches all the tags mapping them from X to other_repo/X using git's refspec functionality.
References (branches, tags) are just files in git, and you can use directories for namespacing. The two commands above will preserve tags from the first repository as-is, and the ones from the other one will be prefixed with other_repo/
After the operation it's best to remove the other remote, so you don't accidentally fetch the tags the normal way and make a mess.
I wanted to move a small project to a subdirectory of a larger one. Since my small project did not have many commits, I used git format-patch --output-directory /path/to/patch-dir. Then on the larger project, I used git am --directory=dir/in/project /path/to/patch-dir/*.
This feels way less scary and way more cleaner than a filter-branch. Granted, it may not be applicable to all cases.
Note: this solution may push uneeded LFS binaries (that are not used by the merge's ancestor commits), so you might want to improve this solution.
To complete the good answer by Andresch Serj in the case the project-a uses Git LFS in order to be able to push I had to first do:
cd /path/to/project-a
git lfs fetch --all # some old LFS objects might only be on the server of project-a and not in your clone of project-a
# possible alternative to the following two commands (untested): do a git lfs fetch from path/to/project-b
git remote add project-b-origin url-to-project-b-server
git lfs push --all project-b-origin
In my case this fixed the errors I got when attempting to git push the merge result in project-a clone directory:
remote: GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".
and
Unable to find source for object <hash> (try running `git lfs fetch --all`)
Git LFS upload failed:
(missing) <path> (<hash>)
© 2022 - 2024 — McMap. All rights reserved.