Re: cleaner/better zlib sources?

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mar 16, 2007, 16:51 UTC
Message-ID: <Pine.LNX.4.64.0703160946060.3816@woody.linux-foundation.org>
In-Reply-To: <45FAC75B.3030902@garzik.org>

On Fri, 16 Mar 2007, Jeff Garzik wrote:

Show 5 quoted lines

>
> Although it sounds like zlib could indeed be optimized to reduce its startup
> and shutdown overhead, I wonder if switching compression algorithms to a pure
> Huffman or even RLE compression (with associated lower startup/shutdown costs)
> would perform better in the face of all those small objects.

Well, the thing is, I personally much prefer to have just a single compression algorithm and object layout. Most of the performance-critical objects from a decompression standpoint during commit traversal are all small (especially if you do pathname limiting), but when you do something like a "git add ." most objects are actually random blob objects and you need to have a compression algorithm that works in the general case too.

Of course, pack-v4 may (likely will) end up using different strategies for different objects (delta's in particular), but the "one single object compression type" was a big deal for initial implementation.

It's may not be fundamental to git operation (so we can fairly easily change it and make it more complex without any higher-level stuff even noticing), but it was definitely fundamental to "get something stable and working" up and running quickly..

> And another random thought, though it may be useless in this thread:  I bet
> using a pre-built (compiled into git) static zlib dictionary for git commit
> and tree objects might improve things a bit.

That's kind of pack-v4 area. It will happen, but I'd actually like to see if we can just avoid stupid performance problems with zlib, independently of trying to make more tuned formats.

		Linus