Re: RFC: Enable delayed responses to Git clean/smudge filter requests

From: Jakub Narębski <jnareb@gmail.com>
Date: Nov 16, 2016, 22:41 UTC
Message-ID: <5eb682e8-13cb-67f2-a8a9-ec1fa1d139c6@gmail.com>
In-Reply-To: <17709AFF-3C2D-4EC0-97DC-BD750F514D0B@gmail.com>

W dniu 16.11.2016 o 10:53, Lars Schneider pisze:

Show 7 quoted lines

> On 15 Nov 2016, at 19:03, Junio C Hamano <gitster@pobox.com> wrote:
>> Lars Schneider <larsxschneider@gmail.com> writes:
>>
>>>> The filter itself would need to be aware of parallelism
>>>> if it lives for multiple objects, right?
>>>
>>> Correct. This way Git doesn't need to deal with threading...

[...]

Show 7 quoted lines

>> * You'd need to rein in the maximum parallelism somehow, as you do
>>   not want to see hundreds of competing filter processes starting
>>   only to tell the main loop over an index with hundreds of entries
>>   that they are delayed checkouts.
> 
> I intend to implement this feature only for the new long running filter
> process protocol. OK with you?

If I remember and understand it correctly, current version of long running process protocol processes files sequentially, one by one: git sends file to filter wholly, and receives response wholly.

In the single-file filter case, git calls filter process as async task, in a separate thread, so that one thread feeds the filter, and main thread (I think?) reads from it, to avoid deadlocks.

Couldn't something like this be done for long running filter process, via protocol extension? Namely, Git would send in an async task content to be filtered, and filter process would stream the response back, in any order. If it would be not enough, we could borrow idea of channels, and be sending few files back concurrently in parallel, as separate channels... though that would probably require quite a bit of change in caller.

Best,

-- 
Jakub Narębski