Re: pack operation is thrashing my server

From: Nicolas Pitre <nico@cam.org>
Date: Aug 14, 2008, 18:38 UTC
Message-ID: <alpine.LFD.1.10.0808141416540.4352@xanadu.home>
In-Reply-To: <alpine.LFD.1.10.0808141014410.3324@nehalem.linux-foundation.org>

On Thu, 14 Aug 2008, Linus Torvalds wrote:

Show 17 quoted lines

> 
> 
> On Wed, 13 Aug 2008, Nicolas Pitre wrote:
> > 
> > A much better solution would consist of finding just _why_ object 
> > enumeration is so slow.  This is indeed my biggest grip with git 
> > performance at the moment.
> > 
> > |nico@xanadu:linux-2.6> time git rev-list --objects --all > /dev/null
> > |
> > |real    0m21.742s
> > |user    0m21.379s
> > |sys     0m0.360s
> > 
> > That's way too long for 1030198 objects (roughly 48k objects/sec).
> 
> Why do you think that's horribly slow?

Call it gut feeling.  Or 60% CPU wasted in zlib.

Show 5 quoted lines

> Doing a rev-list of all objects is a fairly rare operation, but even if 
> you want to clone/repack all of your archives the whole time, please 
> realize that listing objects is _not_ a simple operation. It opens up and 
> parses every single tree in the whole history. That's a _lot_ of data to 
> unpack.

I disagree. Well, right _now_ it is not a simple operation. But if you remember, I'm one of the co-investigator of the pack v4 format which goal is to make history and tree walking much much cheaper, while making their packed representation denser too. Even with early prototypes of the format with the overhead of converting objects back into the current format on the fly in unpack_entry() the object enumeration was _faster_ than current git.

So this might just be what was needed to bring back some incentive behind the pack v4 effort.

Nicolas