Re: Index/hash order

From: Florian Weimer <fw@deneb.enyo.de>
Date: Apr 13, 2005, 22:48 UTC
Message-ID: <87br8ithsv.fsf@deneb.enyo.de>
In-Reply-To: <Pine.LNX.4.58.0504131503180.4501@ppc970.osdl.org>

* Linus Torvalds:

Show 7 quoted lines

> I will bet you that a git checkout is _faster_ than a kernel source tree
> copy. The time will be dominated by the IO costs (in particular the read
> costs), and the IO costs are lower thanks to compression. So I think that
> the cold-cache case will beat your 40 seconds by a clear margin. It
> generally compresses to half the size, so 20 seconds is not impossible
> (although seek costs would tend to stay constant, so I'd expect it to be
> somewhere in between the two).

It's indeed slightly faster (34 seconds). The hot-cache case is about 6 seconds. Still okay.

However, I should redo these tests with a real git. The numbers could be quite different because seek overhead is a bit hard to predict. Which version should I try?

> That's actually pretty encouraging. Your 1.1GB number implies to me that a
> compressed file setup should be about half that, which in turn says that
> the cost of full-file is not at all outrageous.

I usually try to avoid the typical O(f(n)) fallacy because constant factors do matter in practice. But the way you put it -- maybe delta compression isn't worth the complexity after all. At least I'm beginning to have doubts.

Especially since the same Subversion repository, stored by the Berkeley DB backend, requires a whopping 1.3 GB of disk space.

> Or maybe I misunderstood what you were comparing?

My estimates only cover file data, not metadata. Based on the Subversion dumps, it might be possible to get some rough estimates for the cost of storing directory information. What is the average size of a directory blob? Is it true that for each tree revision, you need to store a new directory blob for each directory which indirectly contains a modified file?

Does your 50% estimate include wasted space due to the file system block size?