Re: git gc --aggressive led to about 40 times slower "git log --raw"

From: David Kastrup <dak@gnu.org>
Date: Feb 18, 2014, 08:55 UTC
Message-ID: <87r470ssuc.fsf@fencepost.gnu.org>
In-Reply-To: <CAEjYwfU==yYtQBDzZzEPdvbqz1N=gZtbMr5ccRaC_U7NfViQLA@mail.gmail.com>

Christian Jaeger <chrjae@gmail.com> writes:

Show 12 quoted lines

> I've got a repository where "git log --raw > _somefile" took a few
> seconds in the past, but after an attempt at merging some commits that
> were collected in a clone of the same repo that was created about a
> year ago, I noticed that this command was now taking 3 minutes 7
> seconds. "git gc", "git fsck", "git clone file:///the/repo/.git" also
> now each took between ~4-10 minutes, also "git log --raw somefile" got
> equally unusably slow. With the help of the people on the IRC, I
> tracked it down to my recent use of "git gc --aggressive" in this
> repo. Running "git repack -a -d -f" solved it, now it's again taking
> 4-5 seconds. After running "git gc --aggressive" again for
> confirmation, "git log --raw > _somefile" was again slowed down,
> although now 'only' to 1 minute 34 seconds;

[...]

> I've now learned to avoid "git gc --aggressive". Perhaps there are
> some other conclusions to be drawn, I don't know.

I've seen the same with my ongoing work on git-blame with the current Emacs Git mirror. Aggressive packing reduces the repository size to about a quarter, but it blows up the system time (mainly I/O) significantly, quite reducing the total benefits of my algorithmic improvements there.

There is also some quite visible additional time spent in zlib, so a wild guess would be that zlib is not really suited to the massive amount of directory entries of a Git object store. Since the system time still dominates, this guess would only make sense if Git over zlib kept rereading the directory section of whatever compressed file we are talking about. But that's really a rather handwavy wild guess without anything better than a hunch to back it up. I don't even know what kind of compression and/or packs are used: I've only ever messed myself with the delta coding of the normal "unpacked" operation (there are a few older commits from me on that).

-- 
David Kastrup