Re: Looking for help to understand external filter driver code

From: Lars Schneider <larsxschneider@gmail.com>
Date: Jul 19, 2016, 22:01 UTC
Message-ID: <BDD74384-AA02-4B60-A246-504B4C6DDF02@gmail.com>
In-Reply-To: <xmqqr3ape2ju.fsf@gitster.mtv.corp.google.com>

On 19 Jul 2016, at 23:33, Junio C Hamano <gitster@pobox.com> wrote:

Show 7 quoted lines

> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>> Git writes --> 4 byte filename length
>> Git writes --> filename string
> 
> Why limit to 32GB?  Perhaps NUL termination is more appropriate
> here?

OK, I will use NUL termination for the filename. You're also right about the limit - I will use 8 byte to encode the content length.

Show 10 quoted lines

>> Git writes --> 4 byte content length
>> Git writes --> content string
>> Git reads <-- 4 byte filtered content length
>> Git reads <-- filtered content
> 
> Do you really need to force the sender to know the length in
> advance?  Together with the sequential nature of the above exchange,
> i.e. the filter is forbidden from producing even a single byte of
> its output before reading everything Git feeds it, you are making it
> impossible to use filters that perform streaming conversion.

That is correct. However, for my particular use case streaming conversion wouldn't be useful anyways: https://github.com/github/git-lfs/pull/1382

> Of course, with the "sequential" thing, you do not have to worry
> about deadlocking hence no need for poll/select, but I am not sure
> that is a good thing.

Thanks for the confirmation. I consider to exchange a "filter protocol version" right after the filter process has started. That way someone could add a more evolved "filter driver protocol" later on that supports streaming and the external filter could pick whatever protocol is most appropriate (and supported). Could that be an acceptable compromise to get a serious review of the "sequential" thing?

Thanks, Lars