Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

From: Shawn Pearce <spearce@spearce.org>
Date: Feb 14, 2016, 02:14 UTC
Message-ID: <CAJo=hJv-GWZOsv31iekW+AdNazLGQ=XYD=UXMO+RuB15baTsow@mail.gmail.com>
In-Reply-To: <20160210214945.GA5853@sigill.intra.peff.net>

On Wed, Feb 10, 2016 at 1:49 PM, Jeff King <peff@peff.net> wrote:

Show 30 quoted lines

> On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote:
>
>> On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce <spearce@spearce.org> wrote:
>> >
>> > ... Thoughts?
>>
>> Several of us at $DAY_JOB talked about this more today and thought a
>> variation makes more sense:
>>
>> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack
>> like they do today.
>>
>> 2. Servers that support resumable clone include a "resumable"
>> capability in the advertisement.
>
> Because the magic happens in the git protocol, that would mean this does
> not have to be limited to git-over-http. It could be "resumable=<url>"
> to point the client anywhere (the same server over a different protocol,
> another server, etc).
>
>> 3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.
>>
>> 4. The server may return a 302 Redirect to its current "mostly whole"
>> pack file. This can be more flexible than "refs/heads/*", it just
>> needs to be a mostly complete pack file that contains a complete graph
>> from any arbitrary roots.
>
> And with "resumable=<url>", the client does not have to hit the server
> to do a redirect; it can go straight to the final URL, saving a
> round-trip.

It occurred to me today that to make the above ("resumable=<url>") as efficient as possible, we should allow HTTP clients to include &resumable=1 in the GET /info/refs URL as a hint to the server that if it serves a resumable=<url> it can shrink the ref advertisement to 1 line containing the capabilities.

Clients are going to follow the returned <url> to get a bundle header, which contains the references. And then incremental fetch from the server after downloading the pack. So returning references with the resumable URL during a clone is an unnecessary waste of bandwidth.

We could also consider allowing resumable=<url> to be relative to the repository, so that on HTTP schemes a server could just reply resumable=pack-HASH.info or something short and not worry about overflowing the capabilities line.