Re: git.git as of tonight

From: Stefan Beller <sbeller@google.com>
Date: Nov 3, 2015, 18:18 UTC
Message-ID: <CAGZ79kbAj-tZS3X1EwFw_gGh1=YU8M0OmEX91W8f++fqs6TiBA@mail.gmail.com>
In-Reply-To: <xmqq37wndndp.fsf@gitster.mtv.corp.google.com>

On Tue, Nov 3, 2015 at 9:05 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 5 quoted lines

> Johannes Sixt <j6t@kdbg.org> writes:
>
>> My findings so far are negative. The only short-term and mid-term
>> solution I see so far is to opt-out from the framework during
>> build-time.

So I started reading up on that[1]. As far as I understand, we don't need to mark a file descriptor to be non blocking, but rather we could use ReadFileEx[2] with a flag set for "overlapped" operation.

So that said, we can make set_nonblocking a noop and provide another implementation for strbuf_read_once depending on NO_PTHREADS being set. Maybe not even strbuf_read_once, but rather the underlying xread_nonblock ?

[1] http://tinyclouds.org/iocp-links.html [2] https://msdn.microsoft.com/en-us/library/aa365468(v=VS.85).aspx

Show 10 quoted lines

>
> Now, from where I sit, it seems that the way forward would be
>
>  1. Make this an optional feature so that platforms can compile it
>     out, if it is not already done.  My preference, even if we go
>     that route, would be to see if we can find a way to preserve the
>     overall code structure (e.g. instead of spawning multiple
>     workers, which is why the code needs NONBLOCK to avoid getting
>     stuck on reading from one while others are working, perhaps we
>     can spawn only one and not do a nonblock read?).

Yeah that would be my understanding as well. If we don't come up with a good solution for parallelism in Windows now, we'd need to make it at least working in the jobs=1 case as well as it worked before.

Show 9 quoted lines

>
>  2. After that is done, the feature could graduate to 'master'.  As
>     this is a bigger framework change than others, however, we do
>     not necessarily want to rush it.  On the other hand, because
>     this only affects submodules, which means it has fewer users and
>     testers that would give us feedback while it is on 'next', we
>     may want to push it to 'master' sooner to give it a larger
>     exposure.  I dunno, and I do not want to decide this myself the
>     week before I'll go offline for a few weeks (i.e. today).

Yeah I guess cooking this well done has its benefits.

Show 9 quoted lines

>
>  3. Then we would enlist help from folks who are more familiar with
>     Windows platform (like you) to see how the "run parallel workers
>     and collect from them" can be (re)done with a nice level of
>     abstraction.  I am hoping that we can continue the tradition of
>     the evolution of run-command.c API (I am specifically impressed
>     by what you did for "async" that allows the callers not to worry
>     about threads and processes) aroundt this area.  That is
>     obviously a mid- to longer term goal.

I just wonder if we can skip step 1) and 2) by having the discussion now how to change the framework to work well without posix file descriptors here.

>
> Thanks for working together well, you two.