Re: git fails on large repo clone on intermittent, or intermittently-high-latency, connections

From: Jakub Narebski <jnareb@gmail.com>
Date: Jan 5, 2011, 15:26 UTC
Message-ID: <m3tyhnbcf7.fsf@localhost.localdomain>
In-Reply-To: <AANLkTinkhmHpAQjraviKjqFAczThR5GtT_qdyf6Sb3nm@mail.gmail.com>

Zenaan Harkness <zen@freedbms.net> writes:

Show 25 quoted lines

> Hi, I am trying to clone opentaps.git. The following is my third try,
> and I am giving up now. As you can see I pressed <Return> every now
> and then, and for the last long while, absolutely nothing downloading
> - the connection has clearly died once again. My last run was
> yesterday and I left it run overnight.
> 
> I cannot download (with a git clone/ initial repo), more than roughly
> 100MiB. I am on a satellite connection. I have also experienced this
> with wireless connection about 18 months ago.
> 
> I have in the middle there (a few months ago) spent a couple months at
> a friend's place, and never had the same problem - nice ADSL2+
> connection, ~1.5MiB/s connections. As you can see with my satellite
> (rural) and also with my older wireless (also rural) connections, I do
> not get more than about 64KiB/s, but it's usually slower, and I've
> always had satellite latency issues in the order of 450ms, and
> sometimes the odd dropout.
> 
> As opposed to git, wget on the other hand not only retries and
> continues from where it left off when it retries (HTTP protocol)
> (default retry 10 times I think), but I can completely INTerrupt wget,
> and start it again from an entirely different computer if I want
> (using wget's --continue option), and it will (with HTTP) happily
> continue right where it left off, and eventually my download
> completes.

[...]

> What can I do to work around my flaky link?

Ask project in question to provide bundle of repository for seeding initial clone (see git-bundle manpage); this is an ordinary file, and can be downloaded via HTTP or even P2P.

> 
> How hard would it be to add a wget-like mode to git, for the initial
> repo download?

Very hard; tthough "resumable clone" was often requested (25% responders in "Git User's Survey 2010", see [1]), and there was even some discussion about possible implementation, it was not implemented yet, even as proof of concept.

The trouble is that packfile is *generated for a client*, and bit-for-bit representation of said pack can vary (e.g. if multithreaded packing is enabled; usually a good idea).

[1]: https://git.wiki.kernel.org/index.php/GitSurvey2010#17._Which_of_the_following_features_would_you_like_to_see_implemented_in_git.3F

-- 
Jakub Narebski
Poland
ShadeHawk on #git