Re: Poor performance of git describe in big repos

From: Thomas Rast <trast@inf.ethz.ch>
Date: May 30, 2013, 16:21 UTC
Message-ID: <87ip20bfq4.fsf@linux-k42r.v.cablecom.net>
In-Reply-To: <CAJ-05NOjVhb+3Cab7uQE8K3VE0Q2GhqR3FE=WzJZvSn8Djt6tw@mail.gmail.com>

Alex Bennée <kernel-hacker@bennee.com> writes:

Show 13 quoted lines

> On 30 May 2013 16:33, Thomas Rast <trast@inf.ethz.ch> wrote:
>> Alex Bennée <kernel-hacker@bennee.com> writes:
>>
>>>  41.58%   git  libcrypto.so.1.0.0  [.] sha1_block_data_order_ssse3
>>>  33.62%   git  libz.so.1.2.3.4     [.] inflate_fast
>>>  10.39%   git  libz.so.1.2.3.4     [.] adler32
>>>   2.03%   git  [kernel.kallsyms]   [k] clear_page_c
>>
>> Do you have any large blobs in the repo that are referenced directly by
>> a tag?
>
> Most probably. I've certainly done a bunch of releases (which are tagged) were
> the last thing that was updated was an FPGA image.

[...]

Show 10 quoted lines

>> git-describe should probably be fixed to avoid loading blobs, though I'm
>> not sure off hand if we have any infrastructure to infer the type of a
>> loose object without inflating it.  (This could probably be added by
>> inflating only the first block.)  We do have this for packed objects, so
>> at least for packed repos there's a speedup to be had.
>
> Will it be loading the blob for every commit it traverses or just ones that hit
> a tag? Why does it need to load the blob at all? Surely the commit
> tree state doesn't
> need to be walked down?

No, my theory is that you tagged *the blobs*.  Git supports this.

git-describe needs to look at the commit (if any) obtained by peeling each tag (i.e. dereferencing tags until it reaches a non-tag). So to do that, it resolves the tag's referent and loads it. Usually this will be a commit, in which case it is marked as reached by the tag.

As my example shows, it also resolves tags' referents if they refer to non-commits, in particular, it will decompress large blobs that are (directly) referenced by a tag.

Note that while annotated tags provide the type information themselves, e.g.

  $ git cat-file tag junio-gpg-pub
  object fe113d3f96636710600c6b02d5fd421fa7e87dd6
  type blob
  tag junio-gpg-pub
  [...]

unannotated tags are simply refs, so it is not enough to just look at the tag objects' referent type.

I had a brief look around sha1_file.c, in particular sha1_object_info, and it turns out we lack the "deflate only early part" logic as I suspected. So that'll have to be fixed first. After that I *think* it should automatically carry over into the tag readers.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch