Re: [RFC 1/4 v2] Implement a basic remote helper for svn in C.

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Jul 28, 2012, 06:54 UTC
Message-ID: <20120728065439.GB4739@burratino>
In-Reply-To: <9398008.kKbpiTCtsb@flomedio>

Hi,

Some quick details.

Florian Achleitner wrote:

Show 6 quoted lines

>                                        Anyways, specifying cat-blob-fd is not 
> allowed via the 'option' command (see Documentation and 85c62395).
> It wouldn't make too much sense, because the file descriptor must be set up by 
> the parent.
>
> But for the fifo, it would, probably.

More precisely, requiring that the cat-blob fd go on the command line instead of in the stream matches a model where fast-import's three standard streams are abstract:

 - its input, using the INPUT FORMAT described in git-fast-import(1)
 - its standard output, which echoes "progress" commands
 - its backflow stream, where responses to "cat-blob" and "ls" commands go

The stream format is not necessarily pinned to a Unix model where input and output are based on filesystems and file descriptors. We can imagine a fast-import backend that runs on a remote server and the transport used for these streams is sockets; for fast-import frontends to be usable in such a context, the streams they produce must not rely on particular fd numbers, nor on pathnames (except for mark state saved to relative paths with --relative-marks) representing anything in particular.

This goes just as much for a fifo set up on the filesystem where the fast-import backend runs as for an inherited file descriptor. In the current model, such backend-specific details of setup go on the command line.

Show 5 quoted lines

>                                       The backward channel is only used by the 
> commands 'cat-blob' and 'ls' of fast-import. If a remote helper wants to use 
> them, it would could make fast-import open the pipe by sending an 'option' 
> command with the fifo filename, otherwise it defaults to stdout (like now) and 
> is rather useless.

I'm getting confused by terminology again. Let's see if I have the cast of characters straight:

 - the fast-import backend (e.g., "git fast-import" or "hg-fastimport")
 - the fast-import frontend (e.g., "git fast-export" or svn-fe)
 - git's generic foreign VCS support plumbing, also known as
   transport-helper.c
 - the remote helper (e.g., "git remote-svn" or "git remote-testgit")

Why would the fast-import backend ever need to open a pipe? If I want it to write backflow to a fifo, I can use

	mkfifo my-favorite-fifo
	git fast-import --cat-blob-fd=3 3>my-favorite-fifo

If I want it to write backflow to a pipe, I can use (using ksh syntax)

	cat |&
	git fast-import --cat-blob-fd=3 3>&p

> This would take the fifo setup out of transport-helper. The remote-helper would 
> have to create it, if it needs it.

We can imagine transport-helper.c learning the name of a fifo set up by the remote helper by sending it the "capabilities" command:

	git> capabilities
	helper> option
	helper> import
	helper> cat-blob-file my-favorite-fifo
	helper> refspec refs/heads/*:refs/helper/remotename/*
	helper>

transport-helper.c could then use that information to invoke fast-import appropriately:

	git fast-import --cat-blob-fd=3 3>my-favorite-fifo

But this seems like pushing complication onto the remote helper; since there is expected to be one remote helper per foreign VCS, implementing the appropriate logic correctly once and for all in transport-helper.c for all interested remote helpers to take advantage of seems to me like a better policy.

> Apropos stdout. That leads to another idea. You already suggested that it 
> would be easiest to only use FDs 0..2. Currently stdout and stderr of fast-
> import go to the shell. We could connect stdout to the remote-helper and don't 
> need the additional channel at all.

The complication that makes this strategy not so easy is "progress" commands in the fast-import input stream. (Incidentally, it would be nice to teach transport-helper.c to display specially formatted "progress" commands using a progress bar some day.)

Hoping that clarifies, Jonathan