Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

From: Jeff King <peff@peff.net>
Date: Feb 14, 2016, 17:05 UTC
Message-ID: <20160214170525.GB10219@sigill.intra.peff.net>
In-Reply-To: <CAJo=hJv-GWZOsv31iekW+AdNazLGQ=XYD=UXMO+RuB15baTsow@mail.gmail.com>

On Sat, Feb 13, 2016 at 06:14:31PM -0800, Shawn Pearce wrote:

Show 9 quoted lines

> > And with "resumable=<url>", the client does not have to hit the server
> > to do a redirect; it can go straight to the final URL, saving a
> > round-trip.
> 
> It occurred to me today that to make the above ("resumable=<url>") as
> efficient as possible, we should allow HTTP clients to include
> &resumable=1 in the GET /info/refs URL as a hint to the server that if
> it serves a resumable=<url> it can shrink the ref advertisement to 1
> line containing the capabilities.

I'm slightly wary of this. The client advertising "resumable=1" does not mean "I will definitely do a resumable clone". It means "I support the resumable keyword, and if you share a resumable URL with me, I _might_ use it, depending on things that are none of your business" (e.g., if it does not like the server URL's protocol).

It is recoverable by having the client re-contact the server without the resumable flag, so it could still be a net win if the client will select the resumable URL a majority of the time.

I'm also not happy about having an HTTP-only feature in the protocol. I liked Stefan's proposal for the "v2" protocol that would let the two sides exchange capabilities before the ref advertisement. Then the client, having seen the server's resumable URL, knows whether or not to proceed with the advertisement.

> Clients are going to follow the returned <url> to get a bundle header,
> which contains the references. And then incremental fetch from the
> server after downloading the pack. So returning references with the
> resumable URL during a clone is an unnecessary waste of bandwidth.

If the bundle is up to date, the client can skip the follow-up incremental fetch, as it knows that it has everything needed for the original ref advertisement it got. Whether that's a net win depends on how up-to-date the bundles are.

If "C" is the cost to contact the server at all and "A" is the cost of the advertisement, then a "hit" with this scheme means the overhead is C+A (we contact the server only once). A "miss" means we have do the followup fetch anyway, and we pay 2C+2A (paying the advertisement cost both times). Whereas with your scheme, we pay 2C+A always; two contacts, but only the second has an advertisement.

So it depends on the relative cost of C and A, and how often we expect it to kick in.

In practice, I suspect it's mostly dominated by the cost of the actual clone objects anyway, but maybe that is different for Gerrit. I hear refs/changes/ can get pretty big. :)

But if that is the case, then "C" is almost certainly negligible compared to "A".

> We could also consider allowing resumable=<url> to be relative to the
> repository, so that on HTTP schemes a server could just reply
> resumable=pack-HASH.info or something short and not worry about
> overflowing the capabilities line.

I think that's orthogonal, but yeah, it might be a nice feature for admins setting up the server side config.

-Peff