Re: git-index-pack really does suck..

From: Chris Lee <clee@kde.org>
Date: Apr 3, 2007, 22:49 UTC
Message-ID: <db69205d0704031549g7273da53g817f885705735db2@mail.gmail.com>
In-Reply-To: <Pine.LNX.4.64.0704031357470.6730@woody.linux-foundation.org>

On 4/3/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Tue, 3 Apr 2007, Junio C Hamano wrote:
> > So maybe we should retitle this thread from "git-index-pack
> > really does suck.." to "I used git-index-pack in a stupid way"?

Well, I never claimed to be a genius. :)

Show 10 quoted lines

> See my separate timing numbers, although I bet that Chris can give even
> better ones..
>
> Chris, try applying my patch, and then inside the KDE repo you have, do
>
>         git index-pack --paranoid --stdin --fix-thin new.pack < ~/git/.git/objects/pack/pack-*.pack
>
> (ie index the objects of the *git* repository, not the KDE one). That
> should approximate doing a fair-sized "git pull" - getting new objects. Do
> it with and without --paranoid, and time it.

% time git-index-pack --paranoid --stdin --fix-thin paranoid.pack < /usr/local/src/git/.git/objects/pack/*pack pack bf8ba7897da9c84d1981ecdc92c0b1979506a4b9 git-index-pack --paranoid --stdin --fix-thin paranoid.pack < 5.28s user 0.24s system 98% cpu 5.592 total

% time git-index-pack --stdin --fix-thin trusting.pack < /usr/local/src/git/.git/objects/pack/*pack pack bf8ba7897da9c84d1981ecdc92c0b1979506a4b9 git-index-pack --stdin --fix-thin trusting.pack < 5.07s user 0.12s system 99% cpu 5.202 total

So, in my case, at least... not really much of a difference, which is puzzling.

> I bet that what I see as a 7% slowdown will be much bigger for you, just
> because the negative lookups will be all that much more expensive when you
> have tons of objects.

I applied exactly the patch you sent, and it applied perfectly cleanly
- no failures.

I also mailed out the DVD with the repo on it to hpa today, so hopefully by tomorrow he'll get it. (He's not even two cities over, and I suspect I could have just driven it to his place, but that might have been a little awkward since I've never met him.)

Anyway, so, hopefully once he gets it he can put it up somewhere that you guys can grab it. For reference, the KDE repo is pretty big, but a "real" conversion of the repo would be bigger; the one that I've been playing with only has the KDE svn trunk, and only the first 409k revisions - there are, as of right now, over 650k revisions in KDE's svn repo. So, realistically speaking, a fully-converted KDE git repo would probably take up at least 6GB, packed, if not more. Subproject support would probably be *really* helpful to mitigate that.

-clee