Re: Request for detailed documentation of git pack protocol

From: Shawn O. Pearce <spearce@spearce.org>
Date: May 12, 2009, 23:34 UTC
Message-ID: <20090512233450.GY30527@spearce.org>
In-Reply-To: <200905122329.15379.jnareb@gmail.com>

Jakub Narebski <jnareb@gmail.com> wrote:

Show 6 quoted lines

> We have now proliferation of different (re)implementations of git:
> JGit in Java, Dulwich in Python, Grit in Ruby; and there are other
> planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit
> in Objective-C (for iPhone IIRC).  At some time they would reach
> the point (or reached it already) of implementing git-daemon...
> but currently the documentation of git protocol is lacking.

Well, lets see...

JGit - me and Robin, here on git ML. JGit is the closest reimplementation effort to the canonical C implementation. JGit runs in production servers for many folks, e.g. quite a few Google engineers use the JGit server every day. Its our main git daemon.

Grit - GitHub folks. They know where to find us. And their business is Git. If Grit isn't right, they'll make it right, or possibly suffer a loss of customers. I'm fairly certain that GitHub runs Grit in production.

ObjectGit - Scott Chacon, again, a GitHub folk. Though he has expressed interest in moving to JGit or libgit2 where/when possible.

Dulwich - off in its own world and not even trying to match basic protocol rules by just watching what happens when you telnet to a git port. No clue how that's going to fair.

git# - We'll see. Mono GSoC Git projects have a really bad reputation of ignoring the existing git knowledge and hoping they can invent the wheel on their own.

Show 5 quoted lines

> This can lead, as you can read from recent post on git mailing, to
> implementing details wrong (like Dulwich not using full SHA-1 where
> it should, leading to ordinary git clients to failing to fetch from it),
> or fail at best practices of implementation (like JGit last issue with
> deadlocking for multi_ack extension).

Dulwich is just busted.

No existing developers knew that the fetch-pack/upload-pack protocol has this required implicit buffering consideration until JGit deadlocked over it. But even then I'm still not 100% sure this is true, or if it is just an artifact of the JGit upload-pack side implementation being partially wrong.

Show 8 quoted lines

> The current documentation of git protocol is very sparse; the docs
> in Documentation/technical/pack-protocol.txt offer only a sketch of
> exchange.  You can find more, including pkt-line format, a way sideband
> is multiplexed, and how capabilities are negotiated between server and
> client in design document for "smart" HTTP server, for example in
>   Subject: Re: More on git over HTTP POST
>   Message-ID: <20080803025602.GB27465@spearce.org>
>   URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196

Seriously?  Don't link to that.  Its a horrible version of the smart
HTTP RFC, and worse, it doesn't describe what you say it describes.

Show 5 quoted lines

> It would be really nice, I think, to have RFC for git pack protocol.
> And it would help avoid incompatibilities between different clients
> and servers.  If the document would contain expected behaviour of
> client and server and Best Current Practices it would help avoid
> pitfals when implementing git-daemon in other implementation.

Yea, it would be nice. But find me someone who knows the protocol and who has the time to document the #!@* thing. Maybe I'll try to work on this myself, but I'm strapped for time, especially over the next two-to-three months.

And lets not even start to mention Dulwich not completing a thin
pack before storing it on disk.  Those sorts of on disk things
matter to other more popular Git implementations (c git, jgit).

-- 
Shawn.