cloning a repo with nested submodules does not work

Question

I have multiple separate git repos which have no submodules in them. The task is to assemble a hierarchical tree of those repos and use it to share between users. This is trivial with 'subtree' or 'subrepo' schemes, but it seems not to work with 'submodules'. The reason to try submodules is slow git performance on the nfs systems. In my case checkout takes more than 2 hours

I am trying to create a shared repo which contains submodules. So far the very first attempt to clone fails. Here is the test case:

 mkdir m1 ; cd m1 ; git init ; date > a.txt ; git add --all ; git commit -m added ; cd -
 mkdir m2 ; cd m2 ; git init ; date > b.txt ; git add --all ; git commit -m added ; cd -
 mkdir m3 ; cd m3 ; git init ; date > c.txt ; git add --all ; git commit -m added ; cd -
 mkdir msub; cd msub; git init; date > d.txt; git add --all; git commit -m added;
 git submodule add `realpath ../m1` m1
 cd m1
 git submodule add `realpath ../../m2` m2
 git submodule add `realpath ../../m3` m3
 git commit -m 'added submodules'
 cd ..
 git commit -m 'added a submodule'
 cd ..
 git clone --recursive msub msub1

As a result it creates msub1 with a single top-level submodule (m1).

In other cases I was getting a fatal error similar to this after the clone of the first submodule.

fatal: git upload-pack: not our ref 89434ad65c1e697bfa311cd0260dfe1997985e65
fatal: remote error: upload-pack: not our ref 89434ad65c1e697bfa311cd0260dfe1997985e65
Fetched in submodule path 'soc', but it did not contain 89434ad65c1e697bfa311cd0260dfe1997985e65. Direct fetching of that commit failed.

I tried adding submodules to 'm1' directly and it seemed to improve the situation, but I cannot do it with the real repos.

So, the desired scheme does not seem to work. Is there a way to fix it?

Casar

We ditched submodules a long time ago due to the pain of managing them. It sounds to me like you could just provide the same functionality by shipping a script containing a series of git clone commands, perhaps individually selectable. – Leontina

@Leontina thanks, it tried it and rejected the idea as too shaky. it has to rely on hierarchical .gitignores and needs a lot of scripting. With big customer base it is not going to work well. So, i hoped submodules would make it more manageable. – Casar

Answer 1

git submodule add `realpath ../m1` m1
cd m1
git submodule add `realpath ../../m2` m2
git submodule add `realpath ../../m3` m3

Here you modified your locally cloned copy of m1 but you didn't push the change back to the original m1.

git commit -m 'added submodules'
cd ..
git commit -m 'added a submodule'

You've forgotten to add changes in the submodule.

cd ..
git clone --recursive msub msub1

When git clones msub into msub1 it tries to clone m1 from its original directory, not from msub/m1. Simply because in the top-level .gitmodules there is the path to the original m1. And the original m1 doesn't have submodules.

To fix the entire workflow you need:

• git add changed m1 before committing it;
• cd m1 && git push origin master (well, push to a non-bare repo wouldn't work so cd to the original and pull instead).

So the entire fixed script is:

#! /bin/sh
set -e

mkdir m1 ; cd m1 ; git init ; date > a.txt ; git add --all ; git commit -m added ; cd -
mkdir m2 ; cd m2 ; git init ; date > b.txt ; git add --all ; git commit -m added ; cd -
mkdir m3 ; cd m3 ; git init ; date > c.txt ; git add --all ; git commit -m added ; cd -
mkdir msub; cd msub; git init; date > d.txt; git add --all; git commit -m added;
git submodule add `realpath ../m1` m1
cd m1
git submodule add `realpath ../../m2` m2
git submodule add `realpath ../../m3` m3
git commit -m 'added submodules'
cd ../../m1
git pull ../msub/m1 master
cd ../msub
git add m1
git commit -m 'added a submodule'
cd ..
git clone --recursive msub msub1

Corves

Thanks. I got the picture. It looks like submodule flow violates git philosophy. It does not clone the assembled repo, rather it clones original repos of individual submodules and rebuilds the tree on the way. Any commits made in submodules of the original tree are not reflected in its clone unless pushed in the original submodule repos. Bummer! – Casar

.gitmodules contains the original URL, not an URL into local clone. Such a local clone usually doesn't have a public URL at all, so pushing changes to the original URL is the only way to publish changes. If you want to clone the entire repository with all vendored subrepositories use git subtree. – Corves

Answer 2

The "not our ref" response usually means that your server is configured to restrict direct fetching of objects by ID, and there isn't a suitable reference that allows fetching that object.

Git provides three options which control whether you can fetch an arbitrary object ID: one which allows fetching any arbitrary object that Git has access to, one which allows fetching any object reachable from a reference, and one which additionally allows fetching objects reachable from hidden references. Most server providers choose to disable one or more of these, which often means that you can only request an object ID if a non-hidden reference (i.e., branch or tag) points to it.

The "not our ref" message means that you're trying to fetch an object by object ID, which is used for submodules, but the server doesn't allow it for the aforementioned reason. If you're using Bitbucket Server's ref caching, this can also mean that the server has cached stale data; in such a case, you should disable ref caching if you need things to work.

There are a couple things you can do. If you need the ability to check out an arbitrary revision, you can create a branch that points to it. Or, if your submodule doesn't need a particular revision, but only the latest of a branch, you can set the submodule.<name>.branch option (see man gitmodules) and then you'll always check out the latest branch. If you're using a self-hosted server, you can set uploadpack.allowAnySHA1InWant to true. Finally, you can manually fetch the submodule (possibly with git submodule foreach), which will usually have the revision you want.

Lampedusa

this all happened for local clones, no configurable server involved. But i will try the config. Thanks. – Casar