Re: [PATCH] write-tree performance problems

From: Linus Torvalds <torvalds@osdl.org>
Date: Apr 20, 2005, 15:46 UTC
Message-ID: <Pine.LNX.4.58.0504200840240.6467@ppc970.osdl.org>
In-Reply-To: <Pine.LNX.4.61.0504201128550.2630@cag.csail.mit.edu>

On Wed, 20 Apr 2005, C. Scott Ananian wrote:

> 
> Hmm.  Are our index files too large, or is there some other factor?

They _are_ pretty large, but they have to be,

For the kernel, the index file is about 1.6MB. That's

 - 17,000+ files and filenames
 - stat information for all of them
 - the sha1 for them all

ie for the kernel it averages to 93.5 bytes per file. Which is actually pretty dense (just the sha1 and stat information is about half of it, and those are required).

> I was considering using a chunked representation for *all* files (not just 
> blobs), which would avoid the original 'trees must reference other trees 
> or they become too large' issue -- and maybe the performance issue you're 
> referring to, as well?

No. The most common index file operation is reading, and that's the one that has to be _fast_. And it is - it's a single "mmap" and some parsing.

In fact, writing it is pretty fast too, exactly because the index file is totally linear and isn't compressed or anything fancy like that. It's a _lot_ faster than the "tree objects", exactly because it doesn't need to be as careful.

The main cost of the index file is probably the fact that I add a sha1 signature of the file into itself to verify that it's ok. The advantage is that the signature means that the file is ok, and the parsing of it can be much more relaxed. You win some, you lose some.

		Linus