Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Feb 10, 2016, 22:17 UTC
Message-ID: <20160210221758.GC10155@google.com>
In-Reply-To: <20160210214945.GA5853@sigill.intra.peff.net>

Jeff King wrote:

> On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote:

Show 13 quoted lines

>> Several of us at $DAY_JOB talked about this more today and thought a
>> variation makes more sense:
>>
>> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack
>> like they do today.
>>
>> 2. Servers that support resumable clone include a "resumable"
>> capability in the advertisement.
>
> Because the magic happens in the git protocol, that would mean this does
> not have to be limited to git-over-http. It could be "resumable=<url>"
> to point the client anywhere (the same server over a different protocol,
> another server, etc).

Thanks for bringing this up. A worry with putting the URL in the capabilities line is that it makes it easy to run into the 1000-byte limit. It's been a while since v1.8.3-rc0~148^2~6 (pkt-line: provide a LARGE_PACKET_MAX static buffer, 2013-02-20) but we still can't rely on clients having that applied.

(I also haven't checked whether current versions of git are able to handle longer capability strings with that patch applied.)

Another nice thing about using a 302 is that you can set cookies during the redirect, which might make authenticated access easier. (That said, authenticated access through e.g. signed URLs can work fine without that.)

[...]

Show 5 quoted lines

> Clients do not have to _just_ fetch a packfile. They could get a bundle
> file that contains the roots along with the packfile. I know that one of
> your goals is not duplicating the storage of the packfile on the server,
> but it would not be hard for the server to store the packfile and the
> bundle header separately, and concatenate them on the fly.

Doesn't that prevent using a git-unaware file transfer service to serve the files?

It also means the client can't use the downloaded file as-is --- they need to separate the root list from the packfile (that's not a big deal; just some added complication to switch files at the appropriate moment during download if you want to avoid temporarily using twice the space).

That said, both these problems are avoided by serving the 'split bundle' you described as-is instead of concatenating.

[...]

> And you'll notice, too, that all of the bundle-http magic kicks in
> during step 2 because the client sees they're grabbing a bundle. Which
> means that the <url> in step 1 doesn't _have_ to be a bundle. It can be
> "go fetch from kernel.org, then come back to me".

I think that use case brings in complications that make it not necessarily worth it. In this example, if kernel.org is serving pack files, why shouldn't I point directly at the advertised pack CDN URL instead of adding an extra hop that puts added load on kernel.org servers?

Allowing an arbitrary "fetch from here first" capability is very flexible. I guess my fear comes from not knowing what the flexibility buys beyond aesthetics. (My motivation comes from the example of alternates: it is pretty and very flexible and ended up as a support and maintenance headache instead of being widely useful. I think what you are proposing is more harmless but I'd still want to have an example of what it's used for before going in that direction.)

Thanks, Jonathan