From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Sat, 24 Dec 2011 01:39:11 GMT
Subject: Re: [PATCH v2 3/3] grep: disable threading in all but worktree case
Message-ID: <CACBZZX67WhcdhXdqOm8gZHW7C3YMbV2KzeytwjHwsnF=8-M_+w@mail.gmail.com>
In-Reply-To: <87mxaihpiq.fsf@thomas.inf.ethz.ch>

2011/12/23 Thomas Rast <trast@student.ethz.ch>:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> On Fri, Dec 2, 2011 at 14:07, Thomas Rast <trast@student.ethz.ch> wrote:
>>
>>> I conjecture that this is caused by contention on
>>> read_sha1_mutex. [...] So disable threading entirely when not
>>> scanning the worktree
>>
>> Why does git-grep even need to keep a mutex to call read_sha1_file()?
>> It's inherently a read-only operation isn't it? If the lock is needed
>> because data is being shared between threads in sha1_file.c shouldn't
>> we tackle that instead of completely disabling threading?
>
> The problem is that all sorts of data is shared.  See
>
>  http://thread.gmane.org/gmane.comp.version-control.git/186618
>
> But I need to go through it again, there are some races and double locks
> in the posted version.

I mentioned this on IRC, but I thought I'd bring it up here too.

Is the expensive part of git-grep all the setup work, or the actual
traversal and searching? I'm guessing it's the latter.

In that case an easy way to do git-grep in parallel would be to simply
spawn multiple sub-processes, e.g. if we had 1000 files and 4 cores:

 1. Split the 1000 into 4 parts 250 each.
 2. Spawn 4 processes as: git grep <pattern> -- <250 files>
 3. Aggregate all of the results in the parent process