Re: Poor performance of git describe in big repos

From: Ramkumar Ramachandra <artagnon@gmail.com>
Date: May 30, 2013, 15:17 UTC
Message-ID: <CALkWK0=6xg8X7ZLXoaTk_ZgRnhgXWsftsTzr6JAWFBJvmVOgFw@mail.gmail.com>
In-Reply-To: <CAJ-05NMt6h=JFLLCP+LASKMcToENhF=BSsk1dPML0024hJTwTw@mail.gmail.com>

Alex Bennée wrote:

Show 9 quoted lines

> 15:50 ajb@sloy/x86_64 [work.git] >time git log --pretty=oneline | wc -l
> 24648
>
> real    0m0.434s
> user    0m0.388s
> sys     0m0.112s
>
> Although it doesn't take too long to walk the whole mainline history
> (obviously ignoring all the other branches).

Damn, non-starter.  linux.git has 361k+ commits in mainline history.

Nit: use git rev-list --count HEAD next time.

Show 9 quoted lines

> 15:52 ajb@sloy/x86_64 [work.git] >git count-objects -v -H
> count: 581
> size: 5.09 MiB
> in-pack: 399307
> packs: 1
> size-pack: 1.49 GiB
> prune-packable: 0
> garbage: 0
> size-garbage: 0 bytes

linux.git has 2.9m+ in-pack. The pack-size is much lower at about 800+ MiB, but I don't think 1.49 GiB is a problem in itself. Looking forward to your big-files report to see why it's so big.

Show 6 quoted lines

> It is a pick repo. The gc --aggressive nearly took out my machine keeping
> around 4gb resident for most of the half hour and using nearly 8gb of VM.
>
> Of course most of the history is not needed for day to day stuff. Maybe
> if I split the pack files up it wouldn't be quite such a strain to work
> through them?

Really out of my depth here, sorry. Let's see what Duy (or the others) have to say.

Show 13 quoted lines

>> 2. You have have huge (binary) files checked into your repository.  Do
>> you?  If so, why isn't the code in streaming.c kicking in?
>
> We do have some binary blobs in the repository (mainly DSP and FPGA images)
> although not a huge number:
>
> 15:58 ajb@sloy/x86_64 [work.git] >time git log --pretty=oneline -- xxx
> xxx/xxxxxx/*.out ./xxx/xxx/*.out ./xxx/xxxxxxx/*.out | wc -l
> 234
>
> real    0m0.590s
> user    0m0.552s
> sys     0m0.040s

log is streaming, and is not a good measure: it doesn't even walk the entire commit graph. How big are these files?

> How can I tell if streaming is kicking in or now?

I use callgrind (and kcachegrind to visualize). Can you post callgrind output? It will be helpful in figuring out where exactly git is spending time.