RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

Sorry, no code, only words today. Previously people have proposed a few different options for resumable clone, like "clone.bundle" (currently used by Android as a hack) or "skip start of regenerated pack". Here is another option.

We could implement resumable clone by making a bit of a hybrid of the smart and dumb HTTP protocol.

Overview -------- During `git gc` break out the refs/heads/* namespace into its own packfile, and pack the remaining references into an "other" pack file. This is already done by JGit's DfsGarbageCollector and normal filesystem based GC, we have many years of experience with this approach working well.

The GC process saves the roots used for the "head" pack to a new pack-$HASH.info file, which is written as a sibling to the existing .pack, .idx, and .bitmap files produced by GC. The .info file contains metadata about the pack in a text format such as:

  pack-name objects/pack/pack-$HASH.pack
  pack-trailer $FOOTER_HASH
  pack-size $LENGTH_IN_BYTES

  ff4ea6004fb48146330d663d64a71e7774f059f9 refs/heads/master
  ced85554d3d871f66dc4442631320a96a5d702f5 refs/heads/next

Ideally $HASH = $FOOTER_HASH here so that any repack with the same object set but different compression settings yields a different pack-name in the info file.

During `git clone` HTTP clients first fetch:

  https://host/repo/info/clone

This returns the most recent pack-*.info file available in the repository, assuming the client has read permission on all branches contained in the info file. If there is no info file present or the client has read restrictions that requires eliding content, the URL 404s. On 404 the client tries a normal smart HTTP ref advertisement (/info/refs?service=git-upload-pack).

With the info file in hand the client writes this to a temporary file under $GIT_DIR and then uses dumb HTTP byte range requests to download the pack identified by pack-name:

  https://host/repo/$pack-name
aka
  https://host/repo/objects/pack/pack-$HASH.pack

Smart HTTP servers that have info/clone support will need to return this pack as-is to clients, if the client still has read permission over all branches in the associated pack-*.info file.

Clients store the file data locally and check the last 20 bytes matches $FOOTER_HASH from the info file before running git-index-pack to verify the contents and build the local .idx file. If the footer hash does not match or any resume attempt for the pack-name URL fails, the client deletes everything and starts over from the beginning.

Post Clone Fetch ---------------- After the pack file is downloaded and indexed the client creates local tracking references matching the info file it obtained at the start. Once the references are created the client issues a standard `git fetch` with the origin server to become current.

Since the client is working off the server’s last GC pack, the client will be very close to the server, and the final fetch will be very small, however this depends on the frequency of the server’s GCs. With a volume based GC trigger such as `git gc` in a post-receive hook, clients will be very close to the server after this clone.

Redirects --------- Clients should follow HTTP redirects, allowing servers to position both the info/clone file and the pack-name on CDNs. Clients should not remember the redirected locations and instead always use the base repository URL during resume. This allows servers to embed short-lived authentication tokens into the redirect URL for processing at the edge CDN.

Parallel Download ----------------- Servers may optionally include a header indicating they are willing to perform parallel download of a file iff the client is well connected:

  parallel-download 5 8388608

in the header of the info/clone file indicates the server will allow clients to send up to 5 parallel byte-range requests on 8 MiB alignments.

Clients should start 1 HTTP download, track incoming bandwidth, then add a parallel download for a later segment of the file. If bandwidth increases with the additional stream, clients may try adding a 3rd stream, etc. until the maximum of 5 streams is reached.

Racing with GC -------------- Servers may choose to keep a pack file around several hours after GC to permit clients to continue to resume and download a repository they started cloning before the GC began.

By sending the most recent pack-*.info file to new clients, servers can choose to keep older packs on disk for a time window (e.g. 4 hours) that older sessions can continue to retrieve before the file is unlinked from the filesystem and its disk is reclaimed.

This is an optional service that is subject to disk constraints on the server. Unlike clone.bundle, the server does not need 2x disk capacity for all repositories; the additional disk required depends on GC rate and size of repositories.

End user experience ------------------- From the end-user side experience, I think we want the following to happen:

1. When initiating a `git clone`, the user does not have to do anything special.

2. A `git clone` eventually calls into the transport layer, and
git-remote-curl will probe for the info/clone URL; if the resource
fails to load, everything goes through the traditional codepath.

3. When git-remote-curl detects the support of dumb clone, it does the
"retry until successfully download the pack data fully" dance
internally, tentatively updates the remote tracking refs, and then
pretends as if it was asked to do an incremental fetch. If this
succeeds without any die(), everybody is happy.

4. If the above step 3. has to die() for some reason (including
impatience hitting ^C), leave the $GIT_DIR, downloaded .info file and
partially downloaded .pack file. Tell the user that the cloning can be
resumed and how.

5. `git clone --resume` would take the leftover $GIT_DIR and friends
from step 4. `git clone --resume`, when it is given a $GIT_DIR that
lacks .info, will refuse to start. When `--resume` is given, there is
no other option acceptable (perhaps other than naming which $GIT_DIR
the partial result lives).

... Thoughts?

On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce <spearce@spearce.org> wrote:

Several of us at $DAY_JOB talked about this more today and thought a variation makes more sense:

1. Clients attempting clone ask for /info/refs?service=git-upload-pack
like they do today.

2. Servers that support resumable clone include a "resumable"
capability in the advertisement.

3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.

4. The server may return a 302 Redirect to its current "mostly whole"
pack file. This can be more flexible than "refs/heads/*", it just
needs to be a mostly complete pack file that contains a complete graph
from any arbitrary roots.

5. Clients fetch the file using standard HTTP GET, possibly with
byte-ranges to resume.

6. Once stored and indexed with .idx, clients run `git fsck
--lost-found` to discover the roots of the pack it downloaded. These
are saved as temporary references.

7. Client runs incremental fetch, and then deletes the temporary
references from 6.

An advantage to this process is its much more flexible for the server. There is no additional pack-*.info file required. GC can organize packs anyway it wants, etc.

To make step 4 really resume well, clients may need to save the first Location header it gets back from /info/refs?service=git-resumable-clone and use that on resume. Servers are likely to embed the pack SHA-1 in the Location header, and the client wants to use this on subsequent GET attempts to abort early if the server has deleted the pack the client is trying to obtain.

On Wed, Feb 10, 2016 at 12:11 PM, Shawn Pearce <spearce@spearce.org> wrote:

like "resumable-token=hash" similar to a push cert advertisement?

Or just in the non-http case, they would terminate after the ls-remote (including capability advertisement) was done and connect again to a different service such as git-upload-stale-pack with the resumable token to identify the pack.

In the non-http case the git-upload-stale-pack would be rsync with the resume token to determine the file name of the pack, such that we have resumeability.

jrn:

    index-pack --check-self-contained-and-connected

Stefan Beller <sbeller@google.com> writes:

I somehow doubt it. Both index-pack and lost-found need to trace "object A depends on object B", but the similarity ends there. index-pack traces the delta dependency, which does not have any relation with the reachability, which is what lost-found needs to compute.

I am not quite sure if that is an advantage, though. The second message proposes that the lost-found computation to be done by the client using *.pack, but any client, given the same *.pack, will compute the same result, so if the result is computed on the server side just once when the *.pack is prepared and downloaded to the client, it would give us a better overall resource utilization. And in essence, that was what the *.info file in the first message was.

The original proposal explicitly specified that the initial bulk transfer pack must cover all heads and no tags. For the purpose of offloading the clone traffic and making it resumable, there is no need for such a restriction, and I do like the fact that the second one loosened the requirement.

Junio C Hamano wrote:

You already responded about this in another side-thread, so the following is mostly redundant. :) I think this message can still be useful to future readers since it more explicitly goes through the trade-offs.

The relevant similarity is that both index-pack and lost-found need to inflate all commit, tag, and tree objects.

This is the same idea that led to the introduction of "git index-pack --strict".

[...]

Advantages of not providing the list of roots:
 1. only need one round-trip to serve the packfile as-is
 2. less data sent over the wire (not important unless the list of roots
    is long)
 3. can be enabled on the server for existing repositories without an
    extra step of generating .info files

Advantage of providing the list of roots:
- speedup because the client does not have to compute the list of roots

For a client that is already iterating over all objects and inspecting FLAG_LINK, the advantage (3) seems compelling enough to prefer the protocol that doesn't sent a list of roots.

Except when people pass --depth, "git clone" sets 'check_self_contained_and_connected = 1'. That means clients that already iterate over all objects and inspect FLAG_LINK are the usual case.

Thanks for your thoughtfulness, Jonathan

On Wed, Feb 10, 2016 at 01:22:07PM -0800, Jonathan Nieder wrote:

I'm not sure how compelling (3) is, since we are relying on the server to make certain packing choices. I guess a stock "git repack -ad" would do in a pinch; it should at least contain all needed objects, but it's going to potentially have extra cruft objects (from reflogs, for example).

I outlined some alternatives to Shawn's proposal elsewhere in the thread. I think it's a useful feature for this redirect to not just be "go fetch this packfile", but "go clone from here and come back to me". That opens up a lot of flexibility.

It does make "go fetch this packfile without roots" a little harder, but I think it's still do-able. Right now when git hits an http URL, we pass the smart-http "?service=" magic, and we look at the response to figure out whether we got:

  1. A smart-http server.

  2. A dumb-http server.

  N. Something else, in which case we die.

The alternative I outlined elsewhere (and the patches I posted long ago) basically adds:

  3. If it's a bundle, fetch the bundle and then clone from that.

But we could also do:

  4. If it's a packfile, fetch the packfile and then do the
     find-the-roots magic.

Somewhat related, but I've wondered if we could do something similar even for non-clone cases. That is, `index-pack` could tell us the set of referenced but missing objects, and we could verify that each of those is reachable (we _could_ just have it verify that we have the object at all; traditionally we only guaranteed that reachable objects were kept, but these days we keep anything reachable from another object we are keeping, so if you have X, you should always have X^, etc).

Anyway, that's quite a tangent from this topic.

-Peff

Stefan Beller wrote:

It could just be the string 'resumable'.

But I wonder if it would be possible to save a round-trip by getting the 302 response in the initial request. If the client requests

	/info/refs?service=git-upload-pack&want_resumable=true

then allow the server to make a 302 in response to its current mostly whole pack. Current clients would never send such a request because the current protocol requires that for smart clients

	The request MUST contain exactly one query parameter,
	`service=$servicename`, where `$servicename` MUST be the service
	name the client wishes to contact to complete the operation.
	The request MUST NOT contain additional query parameters.

Current http-backend ignores extra query parameters. I haven't checked other smart http server implementations, though.

HTTP supports range requests and existing CDNs speak HTTP, so I suspect it would work better if the git-resumable-clone service printed an HTTP URL from which to grab the packfile.

I think the details are something that could be figured out after trying out the idea with http first, though.

[...]

How do I tunnel rsync over git protocol?

So I think in the non-http case the git-resumable-clone service would have to print a URL to be served using a possibly different protocol (e.g., a signed https URL for getting the file from a service like S3, or an rsync URL for getting the file using the same ssh creds that were used for the initial request).

[...]

--strict + --check-self-contained-and-connected check that the pack is self-contained. In the process they mark each object that is reachable from another object in the pack with FLAG_LINK.

The objects not marked with FLAG_LINK are the roots.

[...]

Yes.

I really like this design. I'm tempted to implement it (since it lacks a bunch of the downsides of clone.bundle).

Thanks, Jonathan

Jonathan Nieder <jrnieder@gmail.com> writes:

Ahh, OK, if we already have the code that does it, I have no objection. At some point before finalizing the pack on the disk we would want to run fsck on all objects anyway, so having to pass "--strict" to find objects that lack FLAG_LINK does not sound too big a downside.

Thanks.

Jonathan Nieder <jrnieder@gmail.com> writes:

Just to see people are not stepping on each others toe, implementing slightly different components in parallel based on the same theme to solve the same problem, it may be beneficial to have a list of a bit more detailed breakdown of the necessary parts.

I think possible small first steps include:

 * A new "--split-header" option to "git bundle" that allows the
   command to write out two files;

 * An update to the "bundle" transport so that it understand the new
   split bundle format (i.e. when you have base.bndl that refers to
   pack-deadbeef.pack, you can still say "git clone base.bndl" and
   "git ls-remote base.bndl");

 * A new "--bundle-header" option to "git index-pack", which makes
   it to write out the bundle header file that references the
   resulting packfile in addition to the pack .idx file (this should
   also be able to "fix" thin pack, and also by keeping track of the
   actual "foreign" objects not just number of them, compute and
   record the list of prerequisite objects in the resulting bundle
   header file).

Am I on the right track?  Assuming I am, anything I missed?

On Wed, Feb 10, 2016 at 7:43 PM, Junio C Hamano <gitster@pobox.com> wrote:

--bundle-header option for pack-objects so that repack/gc can create the matching .info file?

I waffled on the URL in the capabilities line, but I think Peff convinced me earlier in the thread that its reasonably safe to do. So this split bundle with header URL in the capabilities sounds like a good approach, its just a matter of implementation. :)

On Thu, Feb 11, 2016 at 4:01 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:

A bit disappointed we still do not address resumable fetch. But I'm happy that it makes people excited and one of them even tempted to implement. This thread sounds like we'll see resumable clone before 2017 :)

Duy Nguyen <pclouds@gmail.com> writes:

As you already said, the initial "clone" and subsequent incremental "fetch" are orthogonal issues.

Because the proposed update to "clone" has much larger payoff than the proposed change to "fetch", i.e.

 * The amount of data that gets transferred is much larger, hence
   the chance of network timing out in a poor network environment is
   much higher, need for resuming much larger.

 * Not only the approach makes "clone" resumable and helping
   clients, it helps the server offload bulk transfer out to CDN.

and it has much smaller damage to the existing code, i.e.

 * We do not have to pessimize the packing process, only to discard
   the bulk of bytes that were generated, like the proposed approach
   for "fetch".

 * The area new code is needed is well isolated and the switch to
   new protocol happens very early in the exchange without sharing
   code to existing codepath; these properties make it less risky to
   introduce regression.

it is simply a good taste to give "clone" higher priority between the two.

So I do not think there is much reason to feel disappointed.

On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote:

Because the magic happens in the git protocol, that would mean this does not have to be limited to git-over-http. It could be "resumable=<url>" to point the client anywhere (the same server over a different protocol, another server, etc).

And with "resumable=<url>", the client does not have to hit the server to do a redirect; it can go straight to the final URL, saving a round-trip.

Clients do not have to _just_ fetch a packfile. They could get a bundle file that contains the roots along with the packfile. I know that one of your goals is not duplicating the storage of the packfile on the server, but it would not be hard for the server to store the packfile and the bundle header separately, and concatenate them on the fly.

Right now the clients can't clone from bundles directly via HTTP. I wrote patches for that ages ago, but got stuck on this very issue (basically that I had to spool the bundle and then clone from it, which temporarily doubled the client's disk space requirements). One alternative would be to amend the bundle format so that rather than a single file, you get a bundle header whose end says "...and my matching packfile is 1234-abcd". And then the client knows that they can fetch that separately from the same source.

It's an extra HTTP request, but it makes the code for client _and_ server way simpler. So the whole thing is basically then:

  0. During gc, server generates pack-1234abcd.pack. It writes matching
     tips into pack-1234abcd.info, which is essentially a bundle file
     whose final line says "pack-1234abcd.pack".

  1. Client contacts server via any git protocol. Server says
     "resumable=<url>". Let's says that <url> is
     https://example.com/repo/clones/1234abcd.bundle.

  2. Client goes to <url>. They see that they are fetching a bundle,
     and know not to do the usual smart-http or dumb-http protocols.
     They can fetch the bundle header resumably (though it's tiny, so it
     doesn't really matter).

  3. After finishing the bundle header, they see they need to grab the
     packfile. Based on the bundle header's URL and the filename
     contained within it, they know to get
     https://example.com/repo/clones/pack-1234abcd.pack". This is
     resumable, too.

  4. Client clones from bundled pack as normal; no root-finding magic
     required.

  5. Client runs incremental fetch against original repo from step 1.

And you'll notice, too, that all of the bundle-http magic kicks in during step 2 because the client sees they're grabbing a bundle. Which means that the <url> in step 1 doesn't _have_ to be a bundle. It can be "go fetch from kernel.org, then come back to me".

Yes, it's much better than your original email, at least for GitHub servers. We're not very flexible with GC tricks, because we need bitmaps to work, and because we get a lot of benefit from sharing the object storage for forks of a single repository.

You could possibly do away with this trick if the server hands out a unique URL in its "resumable" header. Though I imagine it might be convenient for server admins to always point to a generic url, and put the logic in the HTTP layer.

OTOH, if you do the "split bundle" thing I mentioned above, then this happens for free. The client caches the bundle header it grabs in my step 2, and then that contains the unique pack name to fetch in step 3.

-Peff

Jeff King wrote:

Thanks for bringing this up. A worry with putting the URL in the capabilities line is that it makes it easy to run into the 1000-byte limit. It's been a while since v1.8.3-rc0~148^2~6 (pkt-line: provide a LARGE_PACKET_MAX static buffer, 2013-02-20) but we still can't rely on clients having that applied.

(I also haven't checked whether current versions of git are able to handle longer capability strings with that patch applied.)

Another nice thing about using a 302 is that you can set cookies during the redirect, which might make authenticated access easier. (That said, authenticated access through e.g. signed URLs can work fine without that.)

[...]

Doesn't that prevent using a git-unaware file transfer service to serve the files?

It also means the client can't use the downloaded file as-is --- they need to separate the root list from the packfile (that's not a big deal; just some added complication to switch files at the appropriate moment during download if you want to avoid temporarily using twice the space).

That said, both these problems are avoided by serving the 'split bundle' you described as-is instead of concatenating.

[...]

I think that use case brings in complications that make it not necessarily worth it. In this example, if kernel.org is serving pack files, why shouldn't I point directly at the advertised pack CDN URL instead of adding an extra hop that puts added load on kernel.org servers?

Allowing an arbitrary "fetch from here first" capability is very flexible. I guess my fear comes from not knowing what the flexibility buys beyond aesthetics. (My motivation comes from the example of alternates: it is pretty and very flexible and ended up as a support and maintenance headache instead of being widely useful. I think what you are proposing is more harmless but I'd still want to have an example of what it's used for before going in that direction.)

Thanks, Jonathan

On Wed, Feb 10, 2016 at 02:17:58PM -0800, Jonathan Nieder wrote:

I hadn't considered that, but I'm not sure how much of a problem it is in practice. The first line is 40-hex sha1, space, HEAD, NUL, then capabilities. The current capabilities string from github.com on a sample repo (which has a rather large agent string, and a normal-sized symref pointer for HEAD) is 188 bytes. So that's still over 750 bytes available for a URL. The space isn't infinite, and we may want to add more capabilities later. But I'd think that devoting even 256 bytes to a url would be reasonable, and leave us with fair room to grow.

Yeah, I can see there are advantages to assuming all the world is HTTP, but it just doesn't seem that practical to me.

Sort of. They would just need to serve the combined file. But _if_ you have a git-unaware service (rsync?) _and_ its hitting the same storage (so you could in theory not store the packfile twice), _and_ it cannot be taught to do any kind of concatenation, then yes, it would be a problem.

I do think I favor the "split bundle" anyway, though, just for simplicity on both ends.

Sure, that would be more efficient if kernel.org is providing such a CDN URL. But they aren't now, because it doesn't exist yet, and this feature can be used by you without having to coordinate their use of the feature. And you can replace kernel.org in my example with any other server that happens to be preferable to fetching from you, for whatever reason.

I guess one man's trash is another's treasure. Alternates are in wide use at GitHub, and are pretty much what make our data model feasible. I'm pretty sure that git.or.cz uses them for similar purposes, too.

Unless there is a big immediate downside, I'd rather err in the opposite direction: keep orthogonal concerns separate (e.g., redirecting to X versus protocol details of X).

-Peff

Jeff King <peff@peff.net> writes:

Yeah, I was wondering about that. Just storing them as separate two-file pair on the server side and serving it as if it is a single file to the client does not sound like a rocket science, and a reasonable webserver should be able to serve even a byte-range request out of such a thing.

Of course, updating the clients to understand such a two-file pair as a valid bundle is a good thing to do, and I like that step 0. you outlined below.

Jeff King <peff@peff.net> writes:

I would imagine that we would introduce bundle v3 format for this.

It may want to say "my matching packfiles are these" to accomodate a set of packs split at max-pack-size, but I am perfectly fine to say you must create a single pack when you use a bundle with separate header to keep things simpler.

OK.

Might be in megabytes range, though, with many refs. It still is tiny, though ;-).

OK.

I like this part the most.

Or it could be a packfile (and the client discovers roots), as you mentioned in a separate message. I personally do not think it buys us much, as long as we do a bundle represented as a header and a separate pack.

On Thu, Feb 11, 2016 at 01:32:22PM -0800, Junio C Hamano wrote:

Yeah, I think so. And in fact, the "here are my packfiles..." bit should probably be in the v3 header.

Interesting. My initial thought is that one could replace "git bundle create foo.bundle --all && split foo.bundle" with this (for storing or transferring a bundle somewhere that cannot handle the whole thing in one go). It has the advantage that you do not need to recreate the full bundle to extract the data.

But I think the negatives of splitting across packs would outweigh that. You cannot have cross-pack deltas, so your total size would be much larger (in general, I have yet to see a case where max-pack-size is beneficial, beyond the obvious "your filesystem cannot store files larger than N bytes").

So I don't think it would be helpful for normal bundle use.

It _could_ be helpful in the context we're talking about here, though. If I create the split-bundle so that people can resumable-clone from me, they can only clone up to that bundle's creation point (and incrementally fetch the rest). But with a single pack, I can't update the split-bundle without doing a full repack. With multiple packs, I could regenerate the split-bundle header and just mention the new pack.

It wouldn't be as _efficient_ as a full repack of course, but it may be a good, cheap interim solution between repacks.

Yes, but it's the same amount we already spew for a ref advertisement, which isn't resumable, either. :)

I think I'd probably make this a straight fetch in the first iteration, and we can worry about making it resumable later on if people actually care.

Yeah, I think I agree, and if it were just me, I'd implement the bundle part and call it done. But the important thing to me is that we haven't eliminated the possibility of doing the pure-pack thing on top, if we choose to.

-Peff

On Wed, Feb 10, 2016 at 3:49 PM, Jeff King <peff@peff.net> wrote:

I'd like to call this out as a possible security issue before it gets implemented. Allowing the server to instruct the client what protocol to use is a security risk. This sounds like a fine feature, just do it carefully.

I reported a similar issue was discussed off list which eventually became CVE-2015-7545. Basically, git-submodule allowed a repository to specify any protocol via .gitmodules, causing the client to fetch an arbitrary URL using a protocol of the attacker's choice. Sadly, the existence of git-remote-ext allows easily executing arbitrary shell commands if the server can tell the client to use it. Furthermore, it's possible the client has some insecure or sensitive custom git remote helpers installed.

To address this GIT_ALLOW_PROTOCOL was introduced, and git-submodule now uses it as of 33cfccb. This environment variable specifies a default whitelist of protocols. Whoever implements this should probably make use of GIT_ALLOW_PROTOCOL to limit resumable clones to the same default whitelist that git-submodule now uses.

On Fri, Feb 12, 2016 at 07:40:43PM -0600, Blake Burkhart wrote:

Thanks for mentioning this. I agree it's a potential issue, and we should use the same solution as submodules, as you pointed out:

-Peff

On Wed, Feb 10, 2016 at 1:49 PM, Jeff King <peff@peff.net> wrote:

It occurred to me today that to make the above ("resumable=<url>") as efficient as possible, we should allow HTTP clients to include &resumable=1 in the GET /info/refs URL as a hint to the server that if it serves a resumable=<url> it can shrink the ref advertisement to 1 line containing the capabilities.

Clients are going to follow the returned <url> to get a bundle header, which contains the references. And then incremental fetch from the server after downloading the pack. So returning references with the resumable URL during a clone is an unnecessary waste of bandwidth.

We could also consider allowing resumable=<url> to be relative to the repository, so that on HTTP schemes a server could just reply resumable=pack-HASH.info or something short and not worry about overflowing the capabilities line.

On Sat, Feb 13, 2016 at 06:14:31PM -0800, Shawn Pearce wrote:

I'm slightly wary of this. The client advertising "resumable=1" does not mean "I will definitely do a resumable clone". It means "I support the resumable keyword, and if you share a resumable URL with me, I _might_ use it, depending on things that are none of your business" (e.g., if it does not like the server URL's protocol).

It is recoverable by having the client re-contact the server without the resumable flag, so it could still be a net win if the client will select the resumable URL a majority of the time.

I'm also not happy about having an HTTP-only feature in the protocol. I liked Stefan's proposal for the "v2" protocol that would let the two sides exchange capabilities before the ref advertisement. Then the client, having seen the server's resumable URL, knows whether or not to proceed with the advertisement.

If the bundle is up to date, the client can skip the follow-up incremental fetch, as it knows that it has everything needed for the original ref advertisement it got. Whether that's a net win depends on how up-to-date the bundles are.

If "C" is the cost to contact the server at all and "A" is the cost of the advertisement, then a "hit" with this scheme means the overhead is C+A (we contact the server only once). A "miss" means we have do the followup fetch anyway, and we pay 2C+2A (paying the advertisement cost both times). Whereas with your scheme, we pay 2C+A always; two contacts, but only the second has an advertisement.

So it depends on the relative cost of C and A, and how often we expect it to kick in.

In practice, I suspect it's mostly dominated by the cost of the actual clone objects anyway, but maybe that is different for Gerrit. I hear refs/changes/ can get pretty big. :)

But if that is the case, then "C" is almost certainly negligible compared to "A".

I think that's orthogonal, but yeah, it might be a nice feature for admins setting up the server side config.

-Peff

On Sun, Feb 14, 2016 at 9:05 AM, Jeff King <peff@peff.net> wrote:

That was my thinking; if the client starts out with &resumable=1 and then doesn't like the URL the server offers in the advertisement it can drop the query parameter and relist the references for a standard dynamic clone.

Ok, that's fair. Lets forget the "&resumable=1" for now so that the various protocols are more identical. Later if "v2" makes enough progress we can rely on its capability exchange before advertisement to get this potential bandwidth savings.

Well, that comes down to "was the repository repacked since last push". If yes the pack contains everything and its pack-*.info / split bundle header thing contains the same contents as packed-refs.

I think in practice clients aren't going to bother with the implementation detail of saving the initial ref advertisement aside, grabbing the bundle header, and comparing them to see if it needs a subsequent fetch. They are just going to discard the ref advertisement, retrieve the pack, anchor the objects using the bundle header, and then reconnect.

Remember this isn't just about holding the refs in memory. To be resumable the client process may have to be restarted, so that initial ref advertisement has to be written out to disk to persist.

I hear refs/pulls/ can also get big. :)

But yes, some Gerrit advertisements are not small.

On Sun, Feb 14, 2016 at 9:05 AM, Jeff King <peff@peff.net> wrote:

I completely stalled that side project, so if anyone wants to pick it up, feel free.

The server side was complete (and reviewed and people liked it). The only missing part was the client side to be completed. (There were some patches back and forth and it worked mostly, the design seemed to be accepted and well). The code is at https://github.com/stefanbeller/git/commits/protocol2-10

Thanks, Stefan