Re: Git Scaling: What factors most affect Git performance for a large repo?

From: David Turner <dturner@twopensource.com>
Date: Feb 21, 2015, 00:41 UTC
Message-ID: <1424479319.31528.18.camel@leckie>
In-Reply-To: <3770789.lx2BnZ41U2@mfick1-lnx>

On Fri, 2015-02-20 at 13:37 -0700, Martin Fick wrote:

Show 18 quoted lines

> On Friday, February 20, 2015 01:29:12 PM David Turner wrote:
> >...
> > For a more general solution, perhaps a log of ref updates
> > could be used. Every time a ref is updated on the server,
> > that ref would be written into an append-only log.  Every
> > time a client pulls, their pull data includes an index
> > into that log.  Then on push, the client could say, "I
> > have refs as-of $index", and the server could read the
> > log (or do something more-optimized) and send only refs
> > updated since that index.
> 
> Interesting idea, I like it.
> 
> How would you make this reliable?  It relies on updates 
> being reliably recorded which would mean that you would have 
> to ensure that any tool which touches the repo follows this 
> convention.  That is unfortunately a tough thing to enforce 
> for most people.

I think it only truly relies on the server reliably updating its state on ref updates. Which of course the server will do because why would you let arbitrary processes write to your central git repo? (That is, most people use git in a roughly-centralized way, and if you turn on this config option, you promise to only do ref updates that write to the log).

If the client fails to update its state (on a fetch), it will send larger-than-necessary packs but not otherwise fail. And this situation is sometimes be detectable on the client side -- if mtime(.git/refs/remotes/$remote ) > mtime (.git/server-ref-log-index/$remote), then we know our server-ref-log-index is out-of-date.