Re: Poor performance of git describe in big repos

From: Alex Bennée <kernel-hacker@bennee.com>
Date: May 31, 2013, 09:57 UTC
Message-ID: <CAJ-05NN8cARpPTnsCfHt3kY6gTnhZ=Vq55EzqxWBV_3ju-oczQ@mail.gmail.com>
In-Reply-To: <87y5av4jvj.fsf@linux-k42r.v.cablecom.net>

On 31 May 2013 09:46, Thomas Rast <trast@inf.ethz.ch> wrote:

Show 20 quoted lines

> Alex Bennée <kernel-hacker@bennee.com> writes:
>
>> I think you are right. I was brave (well I assumed the tags would come
>> back from the upstream repo) and ran:
>>
>> git for-each-ref | grep "refs/tags" | grep "commit" | cut -d '/' -f 3
>> | xargs git tag -d
>
> So that deleted all unannotated tags pointing at commits, and then it
> was fast.  Curious.
>
>> However I have some big commits it seems:
>>
>> 09:37 ajb@sloy/x86_64 [work.git] >(git for-each-ref | grep ' commit' |
>> cut -d\  -f1 | xargs -n1 git cat-file commit) | wc -c
>> 1147231984
>
> How many unique entries are there in that list, i.e., what does
>
>   git for-each-ref | grep ' commit' | cut -d\  -f1 | sort -u | wc -l

09:49 ajb@sloy/x86_64 [work.git] >git for-each-ref | grep ' commit' | cut -d\ -f1 | sort -u | wc -l 1508

Show 5 quoted lines

> say?  Perhaps you can also find the biggest commit, e.g. like so:
>
>   git for-each-ref | grep ' commit' | cut -d\  -f1 |
>   while read sha; do git cat-file commit $sha | wc -c; done |
>   sort -n

Yeah there is a range from a few hundred bytes to a large number of 3M commits. I guess I need to identify which commits they are and remove the tags or convert them to annotated reference tags.

Show 5 quoted lines

> However, if that turns out to be the culprit, it's not fixable
> currently[1].  Having commits with insanely long messages is just, well,
> insane.
>
>

> [1]  unless we do a major rework of the loading infrastructure, so that
> we can teach it to load only the beginning of a commit as long as we are
> only interested in parents and such

I'll do a bit of scripting to dig into the nature of these uber-commits and try and work out how they cam about. I suspect they are simply start of branch states in our broken and disparate history.

I'll get back to you once I've dug a little deeper.

>
> --
> Thomas Rast
> trast@{inf,student}.ethz.ch

-- 
Alex, homepage: http://www.bennee.com/~alex/