Re: Partitioned packs

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Apr 4, 2007, 02:14 UTC
Message-ID: <Pine.LNX.4.64.0704031858470.6730@woody.linux-foundation.org>
In-Reply-To: <db69205d0704031836u3b3dfc2pb9825dd649aca58@mail.gmail.com>

On Tue, 3 Apr 2007, Chris Lee wrote:

Show 7 quoted lines

> 
> These experiments include trying to see if there's a noticeable
> performance improvement by splitting out objects of different types
> into different packs. So far, it definitely seems to make a
> difference, though not the one I was initially expecting. For all of
> these tests, I did 'sysctl -w vm.drop_caches=3' before running, to
> effectively simulate a cold-cache run.

Ok, the wordwrap makes it a bit hard to read, but it looks like the single-pack always wins. Sometimes by a huge amount.

The reason is simple: not only are single packs well sorted anyway (so if you only look at commits, it will only look at the head of the pack anyway), but a single pack is much faster to look things up in: you can do a single binary lookup.

If you have multiple packs, you *may* be able to do a single binary lookup, but quite often you'll do one *failing* binary lookup, and then go on to the next pack - in other words, you'll do a linear search over a set of binary lookups.

So trying to partition things doesn't help (because the objects are already well sorted), and it does hurt.

That said, for most operations it's probably in the noise. Something bad happened for your "git-blame" thing for the "commits" and "everything else" case. Perhaps just unlucky ordering of packs.

		Linus