Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

From: Junio C Hamano <gitster@pobox.com>
Date: Feb 11, 2016, 21:32 UTC
Message-ID: <xmqq4mdfvspl.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <20160210214945.GA5853@sigill.intra.peff.net>

Jeff King <peff@peff.net> writes:

Show 5 quoted lines

> ... One
> alternative would be to amend the bundle format so that rather than a
> single file, you get a bundle header whose end says "...and my matching
> packfile is 1234-abcd". And then the client knows that they can fetch
> that separately from the same source.

I would imagine that we would introduce bundle v3 format for this.

It may want to say "my matching packfiles are these" to accomodate a set of packs split at max-pack-size, but I am perfectly fine to say you must create a single pack when you use a bundle with separate header to keep things simpler.

Show 6 quoted lines

> It's an extra HTTP request, but it makes the code for client _and_
> server way simpler. So the whole thing is basically then:
>
>   0. During gc, server generates pack-1234abcd.pack. It writes matching
>      tips into pack-1234abcd.info, which is essentially a bundle file
>      whose final line says "pack-1234abcd.pack".

OK.

Show 8 quoted lines

>   1. Client contacts server via any git protocol. Server says
>      "resumable=<url>". Let's says that <url> is
>      https://example.com/repo/clones/1234abcd.bundle.
>
>   2. Client goes to <url>. They see that they are fetching a bundle,
>      and know not to do the usual smart-http or dumb-http protocols.
>      They can fetch the bundle header resumably (though it's tiny, so it
>      doesn't really matter).

Might be in megabytes range, though, with many refs. It still is tiny, though ;-).

Show 5 quoted lines

>   3. After finishing the bundle header, they see they need to grab the
>      packfile. Based on the bundle header's URL and the filename
>      contained within it, they know to get
>      https://example.com/repo/clones/pack-1234abcd.pack". This is
>      resumable, too.

OK.

>   4. Client clones from bundled pack as normal; no root-finding magic
>      required.

I like this part the most.

Show 6 quoted lines

>   5. Client runs incremental fetch against original repo from step 1.
>
> And you'll notice, too, that all of the bundle-http magic kicks in
> during step 2 because the client sees they're grabbing a bundle. Which
> means that the <url> in step 1 doesn't _have_ to be a bundle. It can be
> "go fetch from kernel.org, then come back to me".

Or it could be a packfile (and the client discovers roots), as you mentioned in a separate message. I personally do not think it buys us much, as long as we do a bundle represented as a header and a separate pack.