Git mirroring a repo with all submodules to another repo

Question

I mirrored https://github.com/boostorg/boost.git to my own repository with command:

git clone --recursive https://github.com/boostorg/boost.git
cd boost
git push --mirror 'URLOfMyOwnRepo'

in order to mirror all the submodules. However, it didn't work out the way I expected.

Basically what I want to do is a N-to-1 mirroring, so I can review the source on my own repo in different branches/tags.

I tried:

git clone --mirror --recursive

and

git submodule init git submodule sync git submodule update --init --recursive

even though I got these submodules on my local machine, I still can't mirror them to my repository.

Reunion

Did you manage to resolve the issue ? If so how ? – Devalue

@KarthikJ Sorry, I didn't resolve this. – Reunion

I've spend my days on it. But still now haven't found any solution! :( – Ocarina

I have this question too, still struggling with it. – Aphesis

Answer 1

This scenario applies to the Boost Libraries:

Don't quote me on this, but i think this is historic. Back in the days there was SVN and i vaguely remember that the libraries were linked via symlink. Today the hierarchical layout is simply that all the libraries reside under https://github.com/boostorg/ and are implemented as submodules within https://github.com/boostorg/boost/. You can see this by taking a look at .gitmodules:

url = ../smart_ptr.git
url = ../accumulators.git

etc. This maps to

boost -> https://github.com/boostorg/boost/
    [submodule "smart_ptr"] in libs/smart_ptr via https://github.com/boostorg/boost/../smart_ptr.git
boost -> https://github.com/boostorg/boost/
    [submodule "accumulators"] in libs/accumulators via https://github.com/boostorg/boost/../accumulators.git

The only thing you have to do to replicate this is setting up your local mirror repository in the same way. And of cause, there are stray cats like "numeric_conversion", "disjoint_sets" and "ublas", that map into libs/numeric or other directories. And at all, this may change in the future.

I took neonTTY's approach and modified it for my use case, fixed the "broken references". I use this for local gentoo ebuild shenanigans.

Changes:

• Don't pollute the repository with temporary files.
• Using .git as the standard repository terminator, https://git-scm.com/docs/gitrepository-layout
• no cloning in another private git server setup needed

Code:

#!/bin/bash
# Replicate a local copy of the boost git repositories.
# The current pwd is the root for the hierarchy of boost repositories
# from https://github.com/boostorg/ and https://github.com/boostorg/boost.git
# is the root of your local boost mirror.
# Jedzia (p) 2020, EvePanix 
# based on neonTTY's script via 
# https://mcmap.net/q/1828920/-git-mirroring-a-repo-with-all-submodules-to-another-repo on Apr 25 '19

git clone https://github.com/boostorg/boost.git boost.git

cd boost.git
git submodule status | awk '{print $2}' > ../boost_git_submodules_list.txt
cd ..
while read line; do
    module=`cut -d'/' -f2 <<< $line`

    if [ $module == *"libs/numeric"* ] || [ $module == "numeric" ]; then
      echo "skip $module.git"
    else
      echo "git clone https://github.com/boostorg/$module.git"
      git clone https://github.com/boostorg/$module.git $module.git
    fi
done < boost_git_submodules_list.txt

git clone https://github.com/boostorg/numeric_conversion.git numeric_conversion.git
git clone https://github.com/boostorg/interval.git interval.git
git clone https://github.com/boostorg/odeint.git odeint.git
git clone https://github.com/boostorg/ublas.git ublas.git

git clone https://github.com/boostorg/disjoint_sets.git disjoint_sets.git

After that you can simply run git clone --recursive /path/where/you/started/the/script/boost.git and the submodules are correctly referenced.

P.S.: Btw. Why this happened not to work for the OP (@zdunker) is that you don't have write access to the root of boost.git. This would make no sense, because you want to have a deep copy of a submoduled repository at your own. Therefore you need the submodules also as local copy. Unfortunately this also gives you the burden of keeping track of all changes, yours and the ones from upstream. Welcome to the life of a repository maintainer:) Use good documentation and be exact, especially when you make more than simple changes and in different locations. This can quickly become hairy and confusing.

Burrow

Answer 2

I just went through the same experience and got stuck after doing a recursive clone of boost from github.

In the end I created the following bash script to jump into each submodule folder and manually push a mirror to my private gitlab repo. Then the main repo seemed to find each of the submodules without any other configuration at their new url when I tried git cloning from my private gitlab.

#!/bin/bash
# This file is meant to help a user mirror the submodule-rich github repo 'boost' on our private gitlab
# start in boost root directory of boost cloned with:
# git clone --recursive https://github.com/boostorg/boost.git

myhost=example.com

git submodule status | awk '{print $2}' > boost_git_submodules_list.txt
while read line; do
    cd $line
    module=`cut -d'/' -f2 <<< $line`
    echo "git push --mirror git@$myhost:boostorg/$module.git"
    git push --mirror git@$myhost:boostorg/$module.git
    cd -
done < boost_git_submodules_list.txt

It takes a while and there are some broken references when they push, but I hope this helps!

Craver