Re: [PATCH 0/5] Suggested for PU: revision caching system to significantly speed up packing/walking

From: Nicolas Pitre <nico@cam.org>
Date: Aug 7, 2009, 15:00 UTC
Message-ID: <alpine.LFD.2.00.0908071029580.16073@xanadu.home>
In-Reply-To: <alpine.DEB.1.00.0908070808340.8306@pacific.mpi-cbg.de>

On Fri, 7 Aug 2009, Johannes Schindelin wrote:

Show 11 quoted lines

> On Fri, 7 Aug 2009, Sam Vilain wrote:
> 
> > Especially as in this case as Nick mentions, the domain is subtly 
> > different (ie pack vs dag). Unfortunately you just can't try to pretend 
> > that they will always be the same; you can't force a full repack on 
> > every ref change!
> 
> No, but you do not need that, either.  In the setting that is most likely 
> the most thankful one, i.e. a git:// server, you _want_ to keep the 
> repository "as packed as possible", otherwise the rev cache improvements 
> will be lost in the bad packing performance anyway.

Yes and no.

Currently, the number #1 latency in any initial git clone is the famous "counting objects" phase, even if the repo is perfectly packed. And that's all this rev cache can and will improve. The packing does play its performance role of course, but for a totally different reason. Hence the repository needs no be perfectly packed for a rev cache to speed up its own part of the game.

Show 11 quoted lines

> > > Still, there is some redundancy between the pack index and your cache, 
> > > as you have to write out the whole list of SHA-1s all over again.  I 
> > > guess it is time to look at the code instead of asking stupid 
> > > questions.
> > >   
> > 
> > "Disk is cheap" :-)
> 
> Disk I/O ain't.
> 
> (Size of the I/O caches, yaddayadda, I'm sure you get my point).

I don't know about the size of the rev cache on disk yet (I asked Nick about that) nor do I really know how this cache is implemented. But I know damn well about git packs and associated index and I for sure don't want to see a revision cache coupled with it.

And for a clone the disk IO will certainly be a magnitude larger than for the cache (or so I hope). Maybe the IO for the rev cache might be a significant overhead for operations performed on a client (aka developer) repository, in which case it would be a good idea to have a config variable to control the cache size, or even to turn it off entirely. We do it for delta depth and many other things already.

Nicolas