Re: Poor performance of git describe in big repos
- From
- Alex Bennée <kernel-hacker@bennee.com>
- Date
- May 30, 2013, 15:01 UTC
- Message-ID
- <CAJ-05NMt6h=JFLLCP+LASKMcToENhF=BSsk1dPML0024hJTwTw@mail.gmail.com>
- In-Reply-To
- <CALkWK0=bgM+fYcVEwjHHF8k2Q8wMmjdbM5bxXdPH6s9StDH_Ng@mail.gmail.com>
On 30 May 2013 15:32, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
Show 20 quoted lines
> Alex Bennée wrote: >> And through my "special" repo: >> >> 41.58% git libcrypto.so.1.0.0 [.] sha1_block_data_order_ssse3 >> 33.62% git libz.so.1.2.3.4 [.] inflate_fast >> 10.39% git libz.so.1.2.3.4 [.] adler32 >> 2.03% git [kernel.kallsyms] [k] clear_page_c >> >> I'm not sure why libcrypto features so highly in the results > > While Duy churns on the delta chain, let me try to make a (rather > crude) observation: > > What does it mean for libcrypto to be so high in your perf report? > sha1_block_data_order is ultimately by object.c:parse_object. While > it indicates that deltas are taking a long time to apply (or are > somehow not optimally organized for IO), I think it indicates either: > > 1. Your history is very deep and there are an unusually high number of > deltas for each blob. What are the total number of commits?
Well the history does en-compose about 10 years of product development and has a high number of files in the repo (including about 3 copies of the kernel - sans upstream history).
15:50 ajb@sloy/x86_64 [work.git] >time git log --pretty=oneline | wc -l 24648
real 0m0.434s user 0m0.388s sys 0m0.112s
Although it doesn't take too long to walk the whole mainline history (obviously ignoring all the other branches).
15:52 ajb@sloy/x86_64 [work.git] >git count-objects -v -H count: 581 size: 5.09 MiB in-pack: 399307 packs: 1 size-pack: 1.49 GiB prune-packable: 0 garbage: 0 size-garbage: 0 bytes
It is a pick repo. The gc --aggressive nearly took out my machine keeping around 4gb resident for most of the half hour and using nearly 8gb of VM.
Of course most of the history is not needed for day to day stuff. Maybe if I split the pack files up it wouldn't be quite such a strain to work through them?
> 2. You have have huge (binary) files checked into your repository. Do > you? If so, why isn't the code in streaming.c kicking in?
We do have some binary blobs in the repository (mainly DSP and FPGA images) although not a huge number:
15:58 ajb@sloy/x86_64 [work.git] >time git log --pretty=oneline -- xxx xxx/xxxxxx/*.out ./xxx/xxx/*.out ./xxx/xxxxxxx/*.out | wc -l 234
real 0m0.590s user 0m0.552s sys 0m0.040s
How can I tell if streaming is kicking in or now?
-- Alex, homepage: http://www.bennee.com/~alex/