Re: pack operation is thrashing my server

From: Shawn O. Pearce <spearce@spearce.org>
Date: Aug 13, 2008, 17:19 UTC
Message-ID: <20080813171921.GF3782@spearce.org>
In-Reply-To: <alpine.LFD.1.10.0808131228270.4352@xanadu.home>

Nicolas Pitre <nico@cam.org> wrote:

Show 5 quoted lines

> On Wed, 13 Aug 2008, Shawn O. Pearce wrote:
> > Doing the object
> > enumeration is pointless as a security measure.
> 
> It is good for network bandwidth efficiency as I mentioned.

The network bandwidth efficiency is the most valid argument for the enumeration.

Show 5 quoted lines

> > I'm too busy to write a pack concat implementation proposal
> 
> A much better solution would consist of finding just _why_ object 
> enumeration is so slow.  This is indeed my biggest grip with git 
> performance at the moment.

...

Show 9 quoted lines

> |nico@xanadu:gcc> time git rev-list --objects --all > /dev/null
> |
> |real    1m51.591s
> |user    1m50.757s
> |sys     0m0.810s
> 
> That's for 1267993 objects, or about 11400 objects/sec.
> 
> Clearly something is not scaling here.

Yikes. Last time I was looking at this sort of thing I think we spent around 60% of our time dealing with inflating, patching and parsing commit and tree objects. pack v4's formatting spawned out of that particular point, but we never really finished that. Its been years so I can't trust my memory enough to say pack v4 is the solution to this, without redoing the profiling. But I think that is what one would find.

Though the decreasing objects/sec rate with increased total number of objects suggets the object hash isn't scaling.

-- 
Shawn.