How do I clone a local repo with submodules?

Question

Say I recursively clone a repo.

$ git clone --recursive ssh://server/project/client
Cloning into 'client'...
remote: Counting objects: 191, done
remote: Finding sources: 100% (191/191)
remote: Total 191 (delta 53), reused 159 (delta 53)
Receiving objects: 100% (191/191), 27.59 KiB | 0 bytes/s, done.
Resolving deltas: 100% (53/53), done.
Checking connectivity... done.
Submodule 'gui' (ssh://server/project/client/gui.git) registered for path 'gui'
Cloning into 'gui'...
remote: Counting objects: 3213, done
remote: Finding sources: 100% (3213/3213)
remote: Total 3213 (delta 1272), reused 3107 (delta 1272)
Receiving objects: 100% (3213/3213), 47.88 MiB | 12.05 MiB/s, done.
Resolving deltas: 100% (1272/1272), done.
Checking connectivity... done.
Submodule path 'gui': checked out '7315db8d7a8b36929f7874dc5477359839ec51ce'

Now I want to create a local clone of that local repo (perhaps after making and committing changes locally).

$ git clone --recursive client/ client_copy
Cloning into 'client_copy'...
done.
Submodule 'gui' (/home/deployer/client/gui.git) registered for path 'gui'
fatal: repository '/home/deployer/client/gui.git' does not exist
Clone of '/home/deployer/client/gui.git' into submodule path 'gui' failed

My .gitmodules file looks like this:

[submodule "gui"]
        path = gui
        url = ../client/gui.git

Why does this fail, and how can I address the issue?

Galloping

You should probably clone a bare repository, or create one first, push the contents of the normal repository there, and then clone the bare repository. – Percentage

@J.J.Hakala is that a hack, or is that the idiomatic way of handling local repositories in git? it seems needlessly complicated. – Galloping

Answer 1

The issue is with your .gitmodules file. The submodule url in your project is defined as a relative path from the superproject repository, but when the submodules are cloned, they are placed using the path location.

In other words, git is trying to pull the submodules from the url location, but on your local machine, they're actually at the path location.

To fix this, clone just the local superproject repository (git clone /path/to/superproject), then go into the new clone's .gitsubmodules and change the urls to be ./<whatever-the-path-is>. For example, your gui submodule would become:

[submodule "gui"]
        path = gui
        url = ./gui

Change each submodule in .gitmodules to be like this, then run:

git submodule sync
git submodule update --init --recursive

and that should do it!

Quintessence

Answer 2

If we want to clone the submodules from the locally cloned submodules, I come to the version below. This allows for only one level of submodules.

git clone $src $dst
modules=$(git -C $src config --file .gitmodules --name-only --get-regexp url)
for module in $modules; do
  module_path=$(git -C $src config --file .gitmodules ${module%.url}.path)
  git -C $dst config ${module} $src/$module_path
done
git -C $dst submodule update --init

Acquiescence

Answer 3

I needed a more comprehensive solution than this, due to the strange way gitlab clones things (SRC is toplevel source repo folder and DST is the requested toplevel destination folder):

git clone $SRC $DST
MODULES=$(git -C $SRC config --file .gitmodules --name-only --get-regexp url)
for MODULE in ${MODULES}; do
    MODULE_PATH=$(git -C $SRC config ${MODULE});
    git -C $DST config ${MODULE} ${MODULE_PATH};
done
git -C $DST submodule update --init --recursive;

It would be great if there were a built-in way to do this though...

Mayst

Nice. It clones the submodules from their origin though. I used this as a basis for a script to clone the submodules from the locally cloned submodules. – Acquiescence

In my particular case I was working in a gitlab build environment, so I want to use the actual url (the local folders are empty). I have another version of this that I use that works off the local folders, in that case I use this: MODULE_PATH=$SRC/$(git -C $SRC config --file=.gitmodules ${MODULE}) where I grab "--get-regexp path" instead of url. – Mayst

Answer 4

Note: with Git 2.12 or less, this git submodule update --init --recursive can fail because of some unusual path of the submodules.
This is fixed in Git 2.13 (Q2 2017)

See commit cf9e55f (07 Apr 2017) by Brandon Williams (mbrandonw).
^{(Merged by Junio C Hamano -- gitster -- in commit 5bceab4, 24 Apr 2017)}

submodule: prevent backslash expansion in submodule names

When attempting to add a submodule with backslashes in its name 'git submodule' fails in a funny way. We can see that some of the backslashes are expanded resulting in a bogus path:

git -C main submodule add ../sub\\with\\backslash
fatal: repository '/tmp/test/sub\witackslash' does not exist
fatal: clone of '/tmp/test/sub\witackslash' into submodule path

To solve this, convert calls to 'read' to 'read -r' in git-submodule.sh in order to prevent backslash expansion in submodule names.

Aret