Re: [RFC 1/4 v2] Implement a basic remote helper for svn in C.

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Aug 12, 2012, 16:12 UTC
Message-ID: <20120812161258.GA3829@mannheim-rule.local>
In-Reply-To: <1636924.tANzCnKezB@flobuntu>

Hi again,

Florian Achleitner wrote:

> back to the pipe-topic.

Ok, thanks.

[...]

>> The way it's supposed to work is that in a bidi-import, the remote
>> helper reads in the entire list of refs to be imported and only once
>> the newline indicating that that list is over arrives starts writing
>> its fast-import stream.

[...]

> This would require all existing remote helpers that use 'import' to be ported 
> to the new concept, right? Probably there is no other..

You mean all existing remote helpers that use 'bidi-import', right? There are none.

[...]

> I still don't believe that sharing the input pipe of the remote helper is 
> worth the hazzle.
> It still requires an additional pipe to be setup, the one from fast-import to 
> the remote-helper, sharing one FD at the remote helper.

If I understand correctly, you misunderstood how sharing the input pipe works. Have you tried it?

It does not involve setting up an additional pipe. Standard input for the remote helper is already a pipe. That pipe is what allows transport-helper.c to communicate with the remote helper. Letting fast-import share that pipe involves passing that file descriptor to git fast-import. No additional pipe() calls.

Do you mean that it would be too much work to implement? This explanation just doesn't make sense to me, given that the version using pipe() *already* *exists* and is *tested*.

I get the feeling I am missing something very basic. I would welcome input from others that shows what I am missing.

[...]

> Another alternative would be to use the existing --cat-pipe-fd argument. But 
> that requires to open the fifo before execing fast-import and makes us 
> dependent on the posix model of forking and inheriting file descriptors, while 
> opening a fifo in fast-import would not.

I'm getting kind of frustrated with this conversation going nowhere. Here is a compromise to explain why. Suppose:

- fast-import learns a --cat-blob-file parameter, which tells it to open a
  file to write responses to "cat-blob" and "ls" commands to instead of
  using an inherited file descriptor

- transport-helper.c sets up a named pipe and passes it using --cat-blob-file.

- transport-helper.c reads from that named pipe and copies everything it sees
  to the remote helper's standard input, until fast-import exits.

This would:

 - allow us to continue to use a very simple protocol for communicating
   with the remote helper, where commands and fast-import backflow both
   come on standard input

 - work fine on both Windows and Unix

Meanwhile it would:

 - be 100% functionally equivalent to the solution where fast-import
   writes directly to the remote helper's standard input.  Two programs
   can have the same pipe open for writing at the same time for a few
   seconds and that is *perfectly fine*.  On Unix and on Windows.

   On Windows the only complication with the pipe()-based  is that we haven't
   wired up the low-level logic to pass file descriptors other than
   stdin, stdout, stderr to child processes; and if I have understood
   earlier messages correctly, the operating system *does* have a
   concept of that and this is just a todo item in msys
   implementation.

 - be more complicated than the code that already exists for this
   stuff.

So while I presented this as a compromise, I don't see the point.

Is your goal portability, a dislike of the interface, some implementation detail I have missed, or something else? Could you explain the problem as concisely but clearly as possible (perhaps using an example) so that others like Sverre, Peff, or David can help think through it and to explain it in a way that dim people like me understand what's going on?

Puzzled, Jonathan