Re: pack operation is thrashing my server

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Aug 15, 2008, 16:15 UTC
Message-ID: <alpine.LFD.1.10.0808150907190.3324@nehalem.linux-foundation.org>
In-Reply-To: <20080814213033.GE13814@one.firstfloor.org>

On Thu, 14 Aug 2008, Andi Kleen wrote:

> 
> I would have thought that zlib has a sequential access pattern that the
> CPU prefetchers have a easy time with hiding latency.

No, the lookup tables for the patterns are quite non-sequential. It does do a lot of indirect accesses, ie it loads data from the input stream and then looks things up through that.

But it's quite possible that we should use different compression factors for different object types. Right now we have different (configurable) compression levels for loose objects and packs, but it might be interesting to see what happens for just "packed tree objects".

The trees really end up having rather different access patterns in pack-files. They also tend to be rather less compressible than other blobs, since the SHA1's in there are just random binary data. They also delta very well - obviously regular blobs do that _too_, but regular blobs are seldom as performance-critical in git (ie once you actually unpack a blob, there are other things going on like actually generating a diff - but trees get unpacked over and over for "internal git reasons")

		Linus