Re: Git in Outreachy?

From: Jeff King <peff@peff.net>
Date: Sep 4, 2021, 12:50 UTC
Message-ID: <YTNrehKnfPo3E5RI@coredump.intra.peff.net>
In-Reply-To: <CAOLTT8QufEU5Q64JfQyEOs4FYCsrNX2jgj8PdmYziVtKnRyu4w@mail.gmail.com>

On Sat, Sep 04, 2021 at 03:40:41PM +0800, ZheNing Hu wrote:

Show 13 quoted lines

> This may be a place to promote my patches: See [1][2][3].
> It can provide some extra atoms for git cat-file --batch | --batch-check,
> like %(tree), %(author), %(tagger) etc. Although some performance
> optimizations have been made, It still has small performance gap.
> 
> If the community still expects git cat-file --batch to reuse the logic
> of ref-filter,
> I expect it to get the attention of reviewers.
> 
> The solutions I can think of to further optimize performance are:
> 1. Delay the evaluation of some ref-filter intermediate data.
> 2. Let ref-filter code reentrant and can be called in multi-threaded  to take
> advantage of multi-core.

I don't think trying to thread it will help much. For expensive formats, where we have to actually open and parse objects, in theory we could do that in parallel. But most of our time there is spent in zlib getting the object data, and that all needs to be done under a big lock.

For little formats (e.g., just printing "%(refname)"), we need to serialize the output anyway. So our unit of work is so tiny, I suspect that the threading overhead would be a net negative.

I was coincidentally looking at ref-filter last week, and it seemed to me that a lot of the slowness is because of the over-use of malloc (e.g., we allocate a substring for every atom_value, and then form them into a separate buffer). If we could parse the original format into a form that could be traversed without having to do further allocations, just writing directly to a strbuf (or even a file handle), I think that would be a big improvement.

I just posted the results of some of my experiments to the list:

  https://lore.kernel.org/git/YTNpQ7Od1U%2F5i0R7@coredump.intra.peff.net/

I don't think that gives any kind of useful base to build on, but it shows what's possible by skipping past various segments of the ref-filter code.

-Peff