Re: Request for detailed documentation of git pack protocol

From: Scott Chacon <schacon@gmail.com>
Date: May 14, 2009, 13:55 UTC
Message-ID: <d411cc4a0905140655y244f21aem44f1e246dd74d80c@mail.gmail.com>
In-Reply-To: <20090512233450.GY30527@spearce.org>

Hey,

On Tue, May 12, 2009 at 4:34 PM, Shawn O. Pearce <spearce@spearce.org> wrote:

Show 23 quoted lines

> Jakub Narebski <jnareb@gmail.com> wrote:
>> We have now proliferation of different (re)implementations of git:
>> JGit in Java, Dulwich in Python, Grit in Ruby; and there are other
>> planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit
>> in Objective-C (for iPhone IIRC).  At some time they would reach
>> the point (or reached it already) of implementing git-daemon...
>> but currently the documentation of git protocol is lacking.
>
> Well, lets see...
>
> JGit - me and Robin, here on git ML.  JGit is the closest
> reimplementation effort to the canonical C implementation.
> JGit runs in production servers for many folks, e.g. quite
> a few Google engineers use the JGit server every day.  Its
> our main git daemon.
>
> Grit - GitHub folks.  They know where to find us.  And their
> business is Git.  If Grit isn't right, they'll make it right,
> or possibly suffer a loss of customers.  I'm fairly certain
> that GitHub runs Grit in production.
>
> ObjectGit - Scott Chacon, again, a GitHub folk.  Though he has
> expressed interest in moving to JGit or libgit2 where/when possible.

Actually, all of this work has moved to CocoaGit, which is much farther along than ObjectiveGit ever was. Although I would love to use libgit2 when it gets that far, this was for Mac/iPhone native client work which JGit would not be helpful for.

>
> Dulwich - off in its own world and not even trying to match basic
> protocol rules by just watching what happens when you telnet to a
> git port.  No clue how that's going to fair.

Oddly enough, I'm in Dulwich land too. I've been working on a Mercurial plugin that will provide a two way lossless bridge for Hg to be able to push and pull to/from a Git server. I've fixed some of the issues I've found with the client side work and both pushes and pulls will work now. (I did turn off 'thin-pack' capability announcement, since you're correct that it simply was not properly implemented).

If we're going to round out the list, I've also worked on an ActionScript partial implementation, but it never got to the packfile level, and some of the Erlang guys are interested in writing at least a partial Erlang implementation too, which I may get involved in at some point.

It seems like if anyone would do what you're asking, it's probably me. In the next few weeks, I do what I can to fix up the remainder of the Dulwich code as part of my hg-git work. I'm also working with Shawn on the Apress book, where I was going to try to document much of this information, perhaps I could try writing an RFC as an appendix or something - then that will force him to spend time correcting everything I got wrong :) At least that might be a good starting point - I'm unfamiliar with the actual RFC process, so I'll research that a bit today. I don't mind writing it, I think it would be really really useful to have, I just am unfamiliar with the process.

Thanks, Scott

Show 53 quoted lines

>
> git# - We'll see.  Mono GSoC Git projects have a really bad
> reputation of ignoring the existing git knowledge and hoping
> they can invent the wheel on their own.
>
>> This can lead, as you can read from recent post on git mailing, to
>> implementing details wrong (like Dulwich not using full SHA-1 where
>> it should, leading to ordinary git clients to failing to fetch from it),
>> or fail at best practices of implementation (like JGit last issue with
>> deadlocking for multi_ack extension).
>
> Dulwich is just busted.
>
> No existing developers knew that the fetch-pack/upload-pack protocol
> has this required implicit buffering consideration until JGit
> deadlocked over it.  But even then I'm still not 100% sure this
> is true, or if it is just an artifact of the JGit upload-pack side
> implementation being partially wrong.
>
>> The current documentation of git protocol is very sparse; the docs
>> in Documentation/technical/pack-protocol.txt offer only a sketch of
>> exchange.  You can find more, including pkt-line format, a way sideband
>> is multiplexed, and how capabilities are negotiated between server and
>> client in design document for "smart" HTTP server, for example in
>>   Subject: Re: More on git over HTTP POST
>>   Message-ID: <20080803025602.GB27465@spearce.org>
>>   URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196
>
> Seriously?  Don't link to that.  Its a horrible version of the smart
> HTTP RFC, and worse, it doesn't describe what you say it describes.
>
>> It would be really nice, I think, to have RFC for git pack protocol.
>> And it would help avoid incompatibilities between different clients
>> and servers.  If the document would contain expected behaviour of
>> client and server and Best Current Practices it would help avoid
>> pitfals when implementing git-daemon in other implementation.
>
> Yea, it would be nice.  But find me someone who knows the protocol
> and who has the time to document the #!@* thing.  Maybe I'll try
> to work on this myself, but I'm strapped for time, especially over
> the next two-to-three months.
>
> And lets not even start to mention Dulwich not completing a thin
> pack before storing it on disk.  Those sorts of on disk things
> matter to other more popular Git implementations (c git, jgit).
>
> --
> Shawn.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>