Is there any way to continue Git clone from the point where it failed?

Question

I was cloning the open embedded project yesterday. Because of connection problems, the cloning failed. I started the command again and cloning started from the beginning.

Is there any way to start my clone from the point where it failed?

Tawannatawdry

Answer 1

Unfortunately this cannot be done. See

• Continue interrupted git clone

  No. git clone cannot be restarted. You'll need to rm -rf common, and then restart then clone from the beginning.

• Continue git clone after interruption

  Unfortunately, we did not have enough GSoC slots for the project to allow restartable clones.

  There were discussions about how to implement this on the list, though.

  Unfortunately, those of us who know how the native protocol works can't come to an agreement on how it might be restartable. If you really read the archives on this topic, you'll see that Nico and I disagree about how to do this. IIRC Nico's position is, it isn't really possible to implement a restart.

You have to restart the clone.

Hilmahilt

I just faced this. The clone was from bitbucket repo and it broke at ~50% then when I restarted the command, it seemed to use a cache and breezed past till 40% and then it started normal download. A loss of about 10% which is not bad. There seems to be checkpointing? So it seems to do with a specific (bitbucket) implementation? Or, Am I missing something? – Lamppost

"Unfortunately, those of us who know how the native protocol works can't come to an agreement..." - can someone say me, how has spoken this? It looks like this Nico is just a self-absorbed person, which can't admit he is wrong, while others suffers because of it. A failed clone should be like cloning a whole repository and then removing/corrupting a part of it locally. If you can fix such a repo, you should be able to restart a clone, maybe after some adjustments in the protocol, but you should. – Hefty

Answer 2

With poor connectivity between the git server and your local computer, a git clone may continue for many hours and then fail. Restarting the clone just restarts the process, which is likely to fail again.

As a workaround, use a hosted server that has good connectivity to the git repository and ssh access from your local. Clone to the server, then rsync to your local over ssh, and resume the rsync as needed.

On your cloud server:

`git clone -n git://<repo>.git`

On your local computer (if fails, repeat to resume):

`rsync -a -P -e ssh <user>@<cloud-server>:<path-to-cloned-repository> <local-target-path>`

Lichee

this is great, that way, you can even delete the .git folder before you rsync, if it is not needed – Babbler

now I wish I had a clould server, I cant complete download of unreal engine 5 ... it is maddening... – Woolsey

Answer 3

Note: for project including submodules, git 2.9.1 (July 2016) can help.

See commit bb9d91b (09 Jun 2016), and commit 665b35e (10 Jun 2016) by Stefan Beller (stefanbeller).
^{(Merged by Junio C Hamano -- gitster -- in commit bb2d8a8, 11 Jul 2016)}

submodule--helper: initial clone learns retry logic

Each submodule that is attempted to be cloned, will be retried once in case of failure after all other submodules were cloned.
This helps to mitigate ephemeral server failures and increases chances of a reliable

If the retry fails, the error message will be:

Failed to clone '%s' a second time, aborting

clone of a repo with hundreds of submodules immensely.

Al

Answer 4

Here's one trick that I came up with.

In my case the git clone would fail because of the packfiles were too big and the server would close the connection (timeout) when git was downloading it.

First, I made a bare clone, without any of the blobs:

git clone --filter=blob:none --bare https://the.git/repo.git

This should be very fast even if the repository is huge.

Then, I tricked git into downloading all the blobs:

cd repo.git/
git rev-list --objects --all

The important part here is that if rev-list fails, you can safely retry it.

This took a very long time, and I did have to retry it a few times. But at least I was able to download the whole (almost 4Gb) repository.

Now I can move on to cleaning up the repository's history.

Azedarach