Is anyone working on a next-gen Git protocol?

On Wed, Oct 3, 2012 at 9:13 PM, Junio C Hamano <gitster@pobox.com> wrote:

Has anyone started working on a next-gen Git protocol as a result of this discussion? If not I thought I'd give it a shot if/when I have time.

The current protocol is basically (S = Server, C = Client)

 S: Spew out first ref
 S: Advertisement of capabilities
 S: Dump of all our refs
 C/S: Declare wanted refs, negotiate with server
 S: Send pack to client, if needed

And I thought I'd basically turn it into:

 C: Connect to server, declare what protocol we understand
 C: Advertisement of capabilities
 S: Advertisement of capabilities
 C/S: Negotiate what we want
 C/S: Same as v1, without the advertisement of capabilities, and maybe
don't dump refs at all

Basically future-proofing it by having the client say what it supports to begin with along with what it can handle (like in HTTP).

Then in the negotiation phase the client & server would go back & forth about what they want & how they want it. I'd planned to implement something like:

    C: want_refs refs/heads/*
    S: OK to that
    C: want_refs refs/tags/*
    S: OK to that

Or:

    C: want_refs refs/heads/master
    S: OK to that
    C: want_refs refs/tags/v*
    S: OK to that

As a proof of concept (and also something that'll solve the issue I had), but by adding an initial negotiation phase the protocol should be open to any future extensions without making assumptions about the client wanting to know about all of the server's refs, unlike the current protocol.

On Sun, Oct 07, 2012 at 09:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:

Unfortunately, client signaling the version is nasty to do in ways that wouldn't cause current servers to hang up or do other undesirable things.

git://: Git-daemon will hang up[1] if it receives command it won't understand (and one can't add arguments either).

ssh://: Commands are NAKed in non-standard ways (e.g. Gitolite vs. shell) and one can't add arguments.

file://: That's easy.

CONNECT: The helper needs to be told that v2 is supported (helper doing
the rest).

Maybe with git://, one could hack the stuff in similar way as virtual hosting was added. But that won't work with SSH (nor one can use environment with SSH).

:-/

[1] And there is no guarantee that the server end of git:// is git-daemon. There's at least one git:// server implemetation that responds to unknown commands by ERR packet followed by hangup.

-Ilari

On Sun, Oct 07, 2012 at 09:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:

I haven't, and don't really plan on it soon (I have a few smaller things I'm working on, then I'd like to look into the EWAH bitmap stuff from Shawn next).

In the "C" portion there, there is also "client acknowledges a subset of capabilities shown by server" while it is declaring wanted refs.

The capability negotiation right now is that the server offers and the client accepts. Are you swapping that so that the client offers and the server accepts? Or are you thinking that they would be sent simultaneously here? That could drop one round-trip (it's probably not that important for git-over-tcp, but smart-http cares a lot about round trips). But it also introduces a complexity with future additions (one side may not know how to present its capabilities until understanding what the other side can do).

Refs we want, or capabilities we want?

I feel like this "maybe..." bit needs more fleshed out before designing the first part. I like the idea of future-proofing first and then adding new features second, but what does the "don't advertise all refs" protocol look like? Presumably the client is going to say "I'm interested in refs/heads/* and refs/tags/*" or something. Does that come with the capabilities? Or is it a new protocol phase?

I think we need to know what the second half of the two-step process will look like to be sure the first half will accommodate it (and the answer may be as simple as saying "they're not sending capabilities, they're sending arbitrary key/value items, with the knowledge that the other side may not understand particular keys, and we have to be prepared to handle both cases).

That seems simple. But how will it work over smart-http? Are we adding a round-trip to do want_refs negotiation?

-Peff

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

I and Shawn helped privately somebody from Gerrit circle, where the initial ref advertisement is a huge problem (primarily because they add tons of refs to one commit that eventually goes to their integration branch), to coming up with a problem description and proposal document to kick-start a discussion some time ago, but not much has happened since. Unless I hear from them soonish, I'll send a cleaned-up version of the draft before I leave for my vacation.

The gist of it is that the current protocol cannot be upgraded in place because "who speaks first" is not something you can update with capability, so we would need upload-pack-v2 that lets the fetching side speak first.

"What is spoken in the first message" is a separate issue, and one of the things it can address is to allow the ends to reduce the amount of ref advertisement that ends up not getting used in the end, but once we allow the fetcher to speak first, we have much wider possibilities.

On Sun, Oct 7, 2012 at 3:31 PM, Junio C Hamano <gitster@pobox.com> wrote:

Which I forgot and never happened. Here is a link (not cleaned-up)

http://tinyurl.com/WhoSpeaksFirstInGit

On 10/07/2012 09:57 PM, Ævar Arnfjörð Bjarmason wrote:

You'll want that to be a single "wants" message to avoid incurring insane amounts of roundtrip latency with lots of refs. github and other hosted services are quite popular, but with my 120ms ping rtt I'd be spending half a minute just telling the other side what I want when I fetch from a repo with 250 refs.

It's a flagday and a half to change the protocol though, so I expect it'll have to wait for 2.0, unless the current client-side part of it is dumb and ignores existing refs when requesting its "wants", in which case the server can just stop advertising existing refs and most of the speedup is already done.

Andreas Ericsson <ae@op5.se> writes:

Peff's recent patch when applied on the server side would help alleviate the load to produce these refs, but it obviously would not cut the network cost. In order to change this, we need to swap "who speaks first".

Once we go into "want/have" phase, I do not think there is a need for fundamental change in the protocol (by this, I am not counting a change to send "have"s sparsely and possibly backtracking to bisect history, etc. as "fundamental").

On Oct 8, 2012, at 6:27 PM, Junio C Hamano wrote:

I've recently discovered that the current protocol can be amazingly inefficient when it comes to transferring binary objects. Assuming two repositories that are in sync. After a 'git checkout --orphan && git commit', a subsequent transfers sends all the blobs attached to the new commit, although the other side already has all the blobs.

This behavior is especially annoying when (mis)using git to store binary files. I was thinking for a while that it might be a reasonable idea to store binary files in a submodule and frequently cut the history in order to save space. The history would have little value anyway, since diff and merge don't make much sense with binary files.

Eventually, I abandoned the idea due to the current behavior of the protocol. I had expected that git would be smarter and behave more like rsync, for example, by skipping big blobs as soon as it recognizes that they are already available at both sides.

Maybe the new protocol could include an optimization for the described case. I don't know whether this would be a fundamental change.

    Steffen

Steffen Prohaska <prohaska@zib.de> writes:

I do not think it has anything to do with binary, but what you deserve from using orphan, where you declared that the history does not have anything to do with the original.

If both of your repositories had the two paralle lines of these histories as branches, the transfer would have went well with or without binary objects.

From: "Junio C Hamano" <gitster@pobox.com>

Steffen, An alternative could be a shallow clone for just those branches with the binary objects, so that the git objects are still identical. Or use a replace/graft to trim the line of development. It's still a fudge, but something you could look at.

On Thu, Oct 11, 2012 at 3:46 AM, Junio C Hamano <gitster@pobox.com> wrote:

On the same inefficient subject, git does not try to share common objects for non-commit refs, for example tags pointing to trees. I have such a peculiar repo and if a new tag shares 90% the tree with existing tags, git-fetch to sends the whole tree of the new tag over the wire. It does not seem easy to fix though and is probably rare enough that does not justify proper support. As a work around, I generate commits that link all these tags/trees together in a predetermined order. Not nice but works ok.

On Wed, Oct 10, 2012 at 6:44 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:

Aside from saving a huge amount of CPU during the "Counting objects" phase, the compressed bitmap work we presented in JGit solves this by working off the complete reachability graph, and not just some subset related to a cut made across the commit graph. Unfortunately we took a shortcut and didn't create bitmaps for non-commits, but this is a trivial modification to the algorithm and the storage.