git shallow clone (clone --depth) misses remote branches

Question

After cloning a remote repository it does not show any remote branch by -a option. What could be the problem? How to debug it? In this snippet two of the remote branches are not shown:

$ git clone --depth 1 git://git.savannah.gnu.org/pythonwebkit.git
$ cd pythonwebkit
$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
$ git --version
git version 1.8.3.1

Tried the same command on another machine, it works well:

$ git clone --depth 1 git://git.savannah.gnu.org/pythonwebkit.git
Receiving objects: 100% (186886/186886), 818.91 MiB | 3.44 MiB/s, done.
$ cd pythonwebkit/
$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/debian
  remotes/origin/master
  remotes/origin/python_codegen
$ git --version
git version 1.7.1

Tried also cloning another repo, it works well. Though I can try it on this machine again, but it would be better to know what's wrong.

Any suggestions or hints will be more than welcome.

Edit: Answer summary: Since git version 1.8.3.2 the "--depth" and "--no-single-branch" need to be used together to get the same behavior as before. This is deemed a bug fix.

Filamentous

master is your local branch. remotes/origin/master is the corresponding remote branch. What exactly is the question? – Harappa

Did you perhaps forget the verbosity? Try git branch -avv – Unlettered

To michas etc: we usually do not refer master as a branch, sorry for the confusion. added "two remote branches are not shown". To jthill: thanks for reminding, you are correct. – Filamentous

Thanks for introducing git clone --depth=1 --no-single-branch, this is what I need in most cases. – Faiyum

Answer 1

The behavior is correct, after the last revision the master-branch is (since this is the primary remote's HEAD) the only remote-branch in the repository:

florianb$ git branch -a
        * master
          remotes/origin/HEAD -> origin/master
          remotes/origin/master

The full clone offers new (all) branches:

florianb$ git branch -a
        * master
          remotes/origin/HEAD -> origin/master
          remotes/origin/debian
          remotes/origin/master
          remotes/origin/python_codegen

---

Shallow clones

Due to the shallow-description in the technical documentation, a "git-clone --depth 20 repo [...] result[s in] commit chains with a length of at most 20." A shallow clone therefore should contain the requested depth of commits, from the tip of a branch.

As - in addition - the documentation of git clone for the --single-branch-option describes:

"Clone only the history leading to the tip of a single branch, either specified by the --branch option or the primary branch remote's HEAD points at. When creating a shallow clone with the --depth option, this is the default, unless --no-single-branch is given to fetch the histories near the tips of all branches."

Therefore a shallow clone (with the depth-option) only fetches only one single branch (at your requested depth).

---

Unfortunately both options (--depth and --single-branch) have been faulty in the past and the use of shallow clones implicits unresolved problems (as you can read in the link I posted above), which is caused by the given history-rewrite. This leads in overall to somewhat complicated behavior in special cases.

Francyne

florianb: what is your git version? thanks for trying it out. I did the --depth 1 on 1.7.1 just now it shows all the remote branches. updated the question with this. +1 for verifying the problem. – Filamentous

@minghua: I'm using 1.8.4 - i'll do a little investigation if there was a patch on that issue. – Francyne

I'm confused. Is it a bug or not? My 1.8.3.1 behave the same as your 1.8.4, but different from 1.7. Which one is correct behavior? The release note means 1.8.3.1 should be different from 1.8.3.2 and later versions... IMHO shallow history should not cut current branches. Since I can check out shallow and still see all remote branches with 1.7.1, they are still there. The branches can be seen on the git web interface too. I don't think 1.8.4 is right... or I'm missing something? – Filamentous

@minghua: i edited to reflect new findings about "shallowed clones". – Francyne

It's almost perfect except only one thing: what does it mean by saying "the repo-owner decided to cut the other branches off"? I think those branches are still there. – Filamentous

@Filamentous - you're absolutely right, i modified the answer. – Francyne

--no-single-branch also clones all tags. We can avoid that by creating a new repo, using the same config to fetch all remotes, i.e. fetch = +refs/heads/*:refs/remotes/origin/*, and running git fetch --depth 1 (without --tags). We can also add specific tags to be fetched, using config like fetch = +refs/tags/v2.0.0:refs/tags/v2.0.0. – Jeffcott

I'm not happy with this answer as it doesn't touch upon the OPs implicit question: how to get all of the branches while not fetching the whole repo. – Conservator

@Conservator - i am sorry, but i couldn't read any implicit question about fetching. I hope the OP flagged this answer as answer because it answered her question. – Francyne

Answer 2

After doing a shallow clone, to be able to checkout other branches from remote do:

1. Run (thanks @jthill) doc about set-branches:

   git remote set-branches origin '*'
   

2. After that, do a git fetch -v --depth=1

3. Finally git checkout the-branch-i-ve-been-looking-for

---

Step 1 can also be done manually by editing .git/config.

For instance, change the following line from:

fetch = +refs/heads/master:refs/remotes/origin/master

to (replace master with *):

fetch = +refs/heads/*:refs/remotes/origin/*

Gavan

You can also use git remote set-branches origin '*' for all branches, replace the * with a branchname for one. – Unlettered

What does -vvv mean ingit fetch -vvv? I coundn't find any information about it in git-fetch doc – Innsbruck

@Innsbruck it is for the verbosity or debug logging level of git. It is not from fetch method. – Gavan

This defeats the purpose of shallow cloning. – Schweitzer

@Schweitzer this is useful if you have so many branches and the size is to big for the 'internet connection' before and now could afford to get all those branches. :) – Gavan

It worked fine for me, this should be the accepted answer! – Venenose

git remote set-branches origin '*' has a minor issue, '*' single quotes in the command surrounding asterisk causes an issue. The command changes the git config file to fetch = +refs/heads/'*':refs/remotes/origin/'*', which causes an issue and does not fetch remaining branches in repository. – Forcemeat

@KrutiParekh Hi. In my git config file, the command with single quotes generated +refs/heads/*:refs/remotes/origin/* and it works fine. I'm using "git version 2.31.1.windows.1", what version are you using? – Myer

You can also use git remote show origin to see the remote branches that have been tracked. – Myer

For me step 2. was git fetch -v --depth=1 to keep the amount of data I downloaded bearable. – Bywoods

This will make you download everything (which defeats the purpose of shallow cloning). See @alejandrodnm answer below. – Bergius

@KrutiParekh using double quotes instead worked for me: git remote set-branches origin "*" – Polky

I'm on a slower network. So, I first downloaded the repo with --depth 1. Then in step 2, instead of --depth=1 I used git fetch --unshallow that downloaded everything for every branch. – Avail

I played with this approach a little bit. I have a relatively large repository (a lot of branches with not so many content inside each). After a regular git clone the size of .git folder is 2.82GB. If I do just clone --depth=1 it is 18MB. After the steps described in this answer the size becomes 121MB. So I would not say "it is pointless". I am going to use this approach to see how it works in practice. – Slapjack

Answer 3

From reading the responses and the comment from @jthill, the thing that worked best for me was to use the set-branches option on the git remote command:

$ git clone --depth 1 https://github.com/dogescript/dogescript.git
$ git remote set-branches origin 'remote_branch_name'
$ git fetch --depth 1 origin remote_branch_name
$ git checkout remote_branch_name

This changes the list of branches tracked by the named remote so that we can fetch and checkout just the required branch.

Crispi

It might be better to use git remote set-branches --add origin 'remote_branch_name' so that the new branch is in addition to existing ones, rather than replacing them in the remote's list of branches (or branch patterns) to fetch in the .git/config file. – Tilefish

OMG, the single quote ' is important in git remote set-branches --add origin 'remote_branch_name' – Marmoset

@Marmoset I couldn't get this working until I left the single quotes out – Intitule

@Intitule You are probably on Windows. The "$" sign in the answer implies Bash (on Unix or Cygwin/MSYS). – Interclavicle

I don't see anything about the single quotes being necessary in the docs and it seems to work fine without one on macOS at least. – Mesoglea