fatal: remote error: upload-pack: not our ref

Question

We are seeing an odd error only on our CI. We have a git repository (let's call it "OUTER") which contains one submodule ("INNER"). If we clone "OUTER" normally on our development machines and update submodules, we see no issues. On CI, after the odd way Bitrise "clones" the repo, calling git fetch throws an error when attempting to fetch the submodule.

Here is an example output. Notice the first few lines simulate how Bitrise "clones" the repo.

➜ test git "init"
git "remote" "add" "origin" "[email protected]:COMPANY/OUTER.git"
git "fetch" "--jobs=10" "--no-tags" "origin" "refs/heads/master"
git "checkout" "master"
git "merge" "origin/master"
git "submodule" "update" "--init" "--recursive" "--jobs=10"

hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint:   git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint:   git branch -m <name>
Initialized empty Git repository in /Users/lnatan/Desktop/test/.git/
remote: Enumerating objects: 48542, done.
remote: Counting objects: 100% (4094/4094), done.
remote: Compressing objects: 100% (989/989), done.
remote: Total 48542 (delta 3383), reused 3623 (delta 3082), pack-reused 44448
Receiving objects: 100% (48542/48542), 178.80 MiB | 5.71 MiB/s, done.
Resolving deltas: 100% (35468/35468), done.
From github.com:COMPANY/OUTER
 * branch              master     -> FETCH_HEAD
 * [new branch]        master     -> origin/master
Branch 'master' set up to track remote branch 'master' from 'origin'.
Already on 'master'
Already up to date.
Submodule 'Submodules/INNER' ([email protected]:COMPANY/INNER.git) registered for path 'Submodules/INNER'
Cloning into '/Users/lnatan/Desktop/test/Submodules/INNER'...
Submodule path 'Submodules/INNER': checked out '174a9f60014e3cead89242178c534aeaff769dc7'
➜  test git:(master) 
➜  test git:(master)                                                                                                                                                                                             
➜  test git:(master) 
➜  test git:(master) git fetch
remote: Enumerating objects: 9001, done.
remote: Counting objects: 100% (4953/4953), done.
remote: Compressing objects: 100% (2833/2833), done.
remote: Total 4461 (delta 3443), reused 2361 (delta 1527), pack-reused 0
Receiving objects: 100% (4461/4461), 6.16 MiB | 3.41 MiB/s, done.
Resolving deltas: 100% (3443/3443), completed with 399 local objects.
From github.com:COMPANY/OUTER
 = [up to date]        master                                              -> origin/master
 * [new branch]        ...
 * [new tag]           ...
Fetching submodule Submodules/INNER
fatal: remote error: upload-pack: not our ref 57df880c914a482c26a1db9b388a34ee55e22730
Errors during submodule fetch:
    Submodules/INNER

Using GIT_TRACE=2 and --verbose produces the following output:

➜  test git:(master) GIT_TRACE=2 git fetch --tags --verbose
02:04:41.644506 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/bin/git
02:04:41.644795 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/bin
02:04:41.645153 git.c:455               trace: built-in: git fetch --tags --verbose
02:04:41.645516 run-command.c:667       trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''COMPANY/OUTER.git'\'''
remote: Enumerating objects: 9300, done.
remote: Counting objects: 100% (5243/5243), done.
remote: Compressing objects: 100% (3091/3091), done.
02:04:48.411851 run-command.c:667       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 30246 on Leo-Natans-Inv-MBP.local' --pack_header=2,4750
02:04:48.417901 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git
02:04:48.418587 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core
02:04:48.419008 git.c:455               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 30246 on Leo-Natans-Inv-MBP.local' --pack_header=2,4750
remote: Total 4750 (delta 3673), reused 2442 (delta 1552), pack-reused 0
Receiving objects: 100% (4750/4750), 6.22 MiB | 3.51 MiB/s, done.
Resolving deltas: 100% (3673/3673), completed with 400 local objects.
02:04:50.742376 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
02:04:50.747795 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git
02:04:50.748441 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core
02:04:50.748944 git.c:455               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
From github.com:COMPANY/OUTER
 = [up to date]        master                                              -> origin/master
 * [new branch]        ...
 * [new tag]           ...
02:04:51.133204 run-command.c:1628      run_processes_parallel: preparing to run up to 1 tasks
02:04:51.133359 run-command.c:667       trace: run_command: cd /Users/lnatan/Desktop/test/.git/modules/Submodules/INNER; unset GIT_PREFIX; GIT_DIR=. git fetch --no-prune --no-prune-tags --tags -v --recurse-submodules-default on-demand --submodule-prefix Submodules/INNER/
Fetching submodule Submodules/INNER
02:04:51.135930 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git
02:04:51.136231 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core
02:04:51.136416 git.c:455               trace: built-in: git fetch --no-prune --no-prune-tags --tags -v --recurse-submodules-default on-demand --submodule-prefix Submodules/INNER/
02:04:51.136872 run-command.c:667       trace: run_command: unset GIT_DIR GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''COMPANY/INNER.git'\'''
02:04:55.589678 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
02:04:55.600024 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
02:04:55.603178 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git
02:04:55.603586 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core
02:04:55.603916 git.c:455               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
From github.com:COMPANY/INNER
 = [up to date]      ...
02:04:55.938232 run-command.c:1628      run_processes_parallel: preparing to run up to 1 tasks
02:04:55.938375 run-command.c:1660      run_processes_parallel: done
02:04:55.938496 run-command.c:667       trace: run_command: git maintenance run --auto --no-quiet
02:04:55.946682 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git
02:04:55.947423 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core
02:04:55.948094 git.c:455               trace: built-in: git maintenance run --auto --no-quiet
02:04:55.951412 run-command.c:667       trace: run_command: cd /Users/lnatan/Desktop/test/.git/modules/Submodules/INNER; unset GIT_PREFIX; GIT_DIR=. git fetch --no-prune --no-prune-tags --tags -v --recurse-submodules-default on-demand --submodule-prefix Submodules/INNER/ origin 57df880c914a482c26a1db9b388a34ee55e22730
02:04:55.956634 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git
02:04:55.957121 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core
02:04:55.957479 git.c:455               trace: built-in: git fetch --no-prune --no-prune-tags --tags -v --recurse-submodules-default on-demand --submodule-prefix Submodules/INNER/ origin 57df880c914a482c26a1db9b388a34ee55e22730
02:04:55.958394 run-command.c:667       trace: run_command: unset GIT_DIR GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''COMPANY/INNER.git'\'''
fatal: remote error: upload-pack: not our ref 57df880c914a482c26a1db9b388a34ee55e22730
02:05:00.643367 run-command.c:1660      run_processes_parallel: done
Errors during submodule fetch:
    Submodules/INNER
02:05:00.644990 run-command.c:667       trace: run_command: git maintenance run --auto --no-quiet
02:05:00.652674 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git
02:05:00.653655 exec-cmd.c:238          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core
02:05:00.654379 git.c:455               trace: built-in: git maintenance run --auto --no-quiet

I can't figure out where 57df880c914a482c26a1db9b388a34ee55e22730 comes from. I cannot find any reference to it anywhere, but I might not know where to look. Any assistance?

Thanks

Allisan

git fetch --no-recurse-submodules is a workaround – Perry

Answer 1

57df880c914a482c26a1db9b388a34ee55e22730 comes from a commit in the superproject (the repository that refers to the submodule). Your git fetch operation is running in "recursive" mode, where it is trying to use the instructions in the superproject to run git fetch in each submodule. Unfortunately the instructions in the superproject are either completely wrong, or unintelligible to the Git software on the remote (the place from which the submodule was initially cloned).

As you noted in your own comment, git fetch --no-recurse-submodules avoids trying to run a recursive fetch of that one specific commit. You could also configure recursion off by default, with git config fetch.recurseSubmodules false. Setting it to on-demand may or may not work here.

Denumerable

How can I find which commit references that commit? I have recreated the submodule from scratch, and the behavior is the same. I've read elsewhere, that it could be a rogue branch which somehow links to this commit, but I went over all branches that were created after the submodule was introduced, and was not able to find any that references this commit. Thanks – Perry

It's kind of a pain to find, but if you know the submodule path, use git rev-parse: for instance in the Git repository for Git, git rev-parse HEAD:sha1colllisiondetection prints the number of interest. If not, use git ls-tree -r on each commit: git ls-tree -r HEAD | grep 855827c583bc30645ba427885caa40c5b81764d2 shows 160000 commit 855827c583bc30645ba427885caa40c5b81764d2 sha1collisiondetection. – Denumerable

In shell script this can be written as a small loop: git rev-list --all | while read hash; do <commands>; done where <commands> is either the desired rev-parse (plus something to show you what $hash is first / conditionally on match), or the desired git ls-tree -r $hash (plus same). – Denumerable

cat .gitmodules in the repo's root folder – Helotism

@milahu: you need to look at the .gitmodules file in each commit, since it may vary from commit to commit, but you also need to consider the fact that Git copies .gitmodules data out to .git/config and then stops using .gitmodules. So this is actually very tricky sometimes. However, that's all for the URL and other settings: the .gitmodules file never gives the hash ID. – Denumerable