Re: Index/hash order

From: Linus Torvalds <torvalds@osdl.org>
Date: Apr 13, 2005, 22:11 UTC
Message-ID: <Pine.LNX.4.58.0504131503180.4501@ppc970.osdl.org>
In-Reply-To: <87aco2gxu2.fsf@deneb.enyo.de>

On Wed, 13 Apr 2005, Florian Weimer wrote:

Show 6 quoted lines

> 
> I've run a few tests, just to get a few numbers of the overhead
> involved.  I used the last ~8,000 changesets from the BKCVS kernel
> repository.  With cold cache, a checkout from cold cache takes about
> 250 seconds on my laptop.  I don't have git numbers, but a mere copy
> of the kernel tree needs 40 seconds.

I will bet you that a git checkout is _faster_ than a kernel source tree copy. The time will be dominated by the IO costs (in particular the read costs), and the IO costs are lower thanks to compression. So I think that the cold-cache case will beat your 40 seconds by a clear margin. It generally compresses to half the size, so 20 seconds is not impossible (although seek costs would tend to stay constant, so I'd expect it to be somewhere in between the two).

Show 5 quoted lines

> For the hot-cache case, the difference is 140 seconds vs. 2.5 seconds
> (or 6 seconds with decompression).
> 
> Uh-oh.  I wouldn't have imaged the difference would be *that*
> dramatic.  The file system layer is *fast*.

Did I mention that I designed git for speed?

Yes. The whole damn design is really about performance, distribution, and built-in integrity checking.

> On the storage front, we have 220 MB for the skip deltas vs. 106 MB
> for pure deltas-to-previous vs. 1.1 GB for uncompressed files
> (directories are always delta-compressed, so to speak[1]).

That's actually pretty encouraging. Your 1.1GB number implies to me that a compressed file setup should be about half that, which in turn says that the cost of full-file is not at all outrageous. Sure, it's 2-3 times larger than your skip deltas, but considering that the performance is about fifty times faster (and I can do distributed stuff without any locking synchronization and you can't), that's a tradeoff I'm more than happy with.

Or maybe I misunderstood what you were comparing?

Of course, the numbers will all depend on how the history looks etc, so this is all pretty much just guidelines.

			Linus