Re: Request for detailed documentation of git pack protocol

From: Shawn O. Pearce <spearce@spearce.org>
Date: May 14, 2009, 14:57 UTC
Message-ID: <20090514145724.GE30527@spearce.org>
In-Reply-To: <200905141024.17525.jnareb@gmail.com>

Jakub Narebski <jnareb@gmail.com> wrote:

Show 15 quoted lines

> On Wed, 13 May 2009 Shawn O. Pearce wrote:
> > Jakub Narebski <jnareb@gmail.com> wrote:
> > > We have now proliferation of different (re)implementations of git:
> > > JGit in Java, Dulwich in Python, Grit in Ruby; and there are other
> > > planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit
> > > in Objective-C (for iPhone IIRC).  At some time they would reach
> > > the point (or reached it already) of implementing git-daemon...
> > > but currently the documentation of git protocol is lacking.
> > 
> > Well, lets see...
> 
> So you are saying that even if detailed pack protocol specification
> isn't written down (Documentation/technical/pack-protocol.txt is more
> of a sketch than reference documentation), the knowledge is there,
> and it is not that hard to get (just ask on git mailing list), isn't it?

Yup. We've tried to keep JGit right here on this list just to keep the knowledge concentrated here, so git@vger is the place anyone can ask questions, and get good answers.

Show 9 quoted lines

> > No existing developers knew that the fetch-pack/upload-pack protocol
> > has this required implicit buffering consideration until JGit
> > deadlocked over it.  But even then I'm still not 100% sure this
> > is true, or if it is just an artifact of the JGit upload-pack side
> > implementation being partially wrong.
> 
> Well... I guess that section on Best Current Practices to avoid 
> deadlocking would not be there to avoid this issue in JGit, but
> would be added for the future later.

So I'm actually right (and Junio confirmed it off list), the fetch-pack/upload-pack protocol with multi_ack enabled requires a buffer on the client side of at least 2952 bytes which can be drained to the server after the client enters its receive phase.

In practical implementations like git:// TCP and SSH, there is enough inherit buffering in the TX side of the client that this isn't an issue.

In loopback mode for local file URIs, it may become an issue. C Git is just getting lucky by the pipe size I think. Though I thought I read somewhere yesterday pipe FIFOs in Linux were being allocated at 512 bytes, not one system page. Of course other systems could allocate whatever size they want too, and may allocate something below the 2952 minimum, and we'd most likely see a deadlock on them.

Show 17 quoted lines

> > > The current documentation of git protocol is very sparse; the docs
> > > in Documentation/technical/pack-protocol.txt offer only a sketch of
> > > exchange.  You can find more, including pkt-line format, a way sideband
> > > is multiplexed, and how capabilities are negotiated between server and
> > > client in design document for "smart" HTTP server, for example in
> > >   Subject: Re: More on git over HTTP POST
> > >   Message-ID: <20080803025602.GB27465@spearce.org>
> > >   URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196
> > 
> > Seriously?  Don't link to that.  Its a horrible version of the smart
> > HTTP RFC, and worse, it doesn't describe what you say it describes.
> 
> Ooops, I am sorry. This was my bookmark into this thread (which is very
> interesting, and contain host of otherwise unknown to me information
> about pack protocol), but the post in this thread was quite arbitrary
> (a random post where I decided that this thread is interesting enough
> to bookmark, and long enough to not want to save all interesting posts).

GMane is dead right now, otherwise I'd try to find the link you
were more likely talking about.  I think you were right, there may
have been a much better post in that particular thread.

Show 6 quoted lines

> > And lets not even start to mention Dulwich not completing a thin
> > pack before storing it on disk.  Those sorts of on disk things
> > matter to other more popular Git implementations (c git, jgit).
> 
> Ugh! Errr... aren't thin packs send only if other side has the
> capability for it?

Yes.

> What is then Dulwich doing announcing such 
> capability when not supporting it correctly...

Because the implementation is just busted.

-- 
Shawn.