Re: Git in Outreachy?

From: ZheNing Hu <adlternative@gmail.com>
Date: Sep 5, 2021, 08:58 UTC
Message-ID: <CAOLTT8S1Tfu6YWcoHhZcydQYd_yBBCavdqyV_TzoOrEW6zHXGQ@mail.gmail.com>
In-Reply-To: <YTNrehKnfPo3E5RI@coredump.intra.peff.net>

Jeff King <peff@peff.net> 于2021年9月4日周六 下午8:50写道：

Show 22 quoted lines

>
> On Sat, Sep 04, 2021 at 03:40:41PM +0800, ZheNing Hu wrote:
>
> > This may be a place to promote my patches: See [1][2][3].
> > It can provide some extra atoms for git cat-file --batch | --batch-check,
> > like %(tree), %(author), %(tagger) etc. Although some performance
> > optimizations have been made, It still has small performance gap.
> >
> > If the community still expects git cat-file --batch to reuse the logic
> > of ref-filter,
> > I expect it to get the attention of reviewers.
> >
> > The solutions I can think of to further optimize performance are:
> > 1. Delay the evaluation of some ref-filter intermediate data.
> > 2. Let ref-filter code reentrant and can be called in multi-threaded  to take
> > advantage of multi-core.
>
> I don't think trying to thread it will help much. For expensive formats,
> where we have to actually open and parse objects, in theory we could do
> that in parallel. But most of our time there is spent in zlib getting
> the object data, and that all needs to be done under a big lock.
>

This big lock is "obj_read_lock()", right? If there are indeed the limitations of these locks, I am afraid that the parallel scheme is not good.

> For little formats (e.g., just printing "%(refname)"), we need to
> serialize the output anyway. So our unit of work is so tiny, I suspect
> that the threading overhead would be a net negative.
>

Make sence.

> I was coincidentally looking at ref-filter last week, and it seemed to
> me that a lot of the slowness is because of the over-use of malloc

Agree. malloc() and data-copy is the reason for the poor performance of ref-filter.

Show 6 quoted lines

> (e.g., we allocate a substring for every atom_value, and then form them
> into a separate buffer). If we could parse the original format into a
> form that could be traversed without having to do further allocations,
> just writing directly to a strbuf (or even a file handle), I think that
> would be a big improvement.
>

This patch has been tried to eliminate some malloc and data-copy: https://lore.kernel.org/git/3760ff032bb1dec3812881fd408f8d78ec125477.1629184489.git.gitgitgadget@gmail.com/ It is indeed possible to obtain some optimizations.

Show 9 quoted lines

> I just posted the results of some of my experiments to the list:
>
>   https://lore.kernel.org/git/YTNpQ7Od1U%2F5i0R7@coredump.intra.peff.net/
>
> I don't think that gives any kind of useful base to build on, but it
> shows what's possible by skipping past various segments of the
> ref-filter code.
>
> -Peff

Thanks. -- ZheNing Hu