patchRFC: proxy-command support for git://

I spend some of my time using a network that only allows outgoing TCP connections to certain ports, and the git-daemon port is not one of them. This patch below implements an analogue to ssh's ProxyCommand feature for git, as a less messy alternative to ssh port forwarding. One can use it to ssh to a bastion host and netcat to the destination:

  $ cat ~/bin/my-git-proxy-command
  #!/bin/sh
  exec ssh bastionhost nc "$1" "$2"

I've done a few pulls and a clone with it, and it seems to work.

Questions:

* Can git already do this and I just failed to notice?

* Where should git_use_proxy() look?  Some git configuration file?
  An environment variable?  Both?  Somewhere else?

It also needs to support non-default ports and probably other things I missed.

Paul Collins <paul@briny.ondioline.org> writes:

Wonderful.

Maybe I just failed to notice this too, but I do not think so.

My preference is put something in .git/config to describe which proxy command (maybe the same one with different argument) to use depending on where you are going. When you have internal hosts and external hosts you would want this to apply only to external hosts. Maybe you have two or more gateways and depending on which external host you are going you may want to use different proxied connection. On top of the config file, making it overridable from an environment variable would be sensible.

On Thu, 3 Nov 2005, Junio C Hamano wrote:

Actually, you could. TWO ways, in fact, afaik.

Just use the "ssh://host/pathname" format (or just "host:pathname") and the GIT_SSH environment variable.

You could also override the local command-name with

	git-send-pack --exec=my-local-send-program /machine/repo/path

where the "my-local-send-program" will parse /machine/repo/path thing. At least that works with git-send-pack, but it's possible it doesn't work with some other logic (ie "git push" might decide that it's unhappy that /machine/repo/path doesn't exist locally because it thinks it's a local path).

		Linus

Another way to do this would be using the ~/.ssh/config file. It would look something like this:

Host host
  ProxyCommand ...

Or, more generically...

Host *.*
  ProxyCommand ...

Host *.* assumes that if the machine name has a '.' in it then you are trying to get outside the firewall. This might not be a good assumption but it works well where I am.

Then use host:pathname or ssh://host/pathname or whatever.

The advantage of using this is that it works for anything that uses ssh to get outside the firewall. Not just git. So, setup is minimal.

Carl

On Thu, Nov 03, 2005 at 11:22:35AM -0800, Linus Torvalds wrote:

Carl Baldwin <cnb@fc.hp.com> writes:

Maybe I am slow today, but wouldn't this require the other end (i.e. remote repo) to let you ssh in?

I think the point of the original patch was to give proxied connection to git:// transport, not git over ssh.

Junio C Hamano <junkio@cox.net> writes:

Here is an updated patch that first looks for GIT_PROXY_COMMAND in the environment and then git.proxycommand in the repository's configuration file. I have left the calling convention the same -- argv[1] is the host and argv[2] is the port.

I've taken the hostname parsing verbatim from git_tcp_connect(), so it should now support an explicit port number and whatever that business with the square brackets is. (Should I move this to a helper function?)

Regarding internal vs. external hosts, the proxy command can simply run netcat locally to internal hosts, so perhaps that is sufficient.

Paul Collins <paul@briny.ondioline.org> writes:

I was hoping this to become a bit more generalized mechanism than that; for example using outgoing plug over HTTP Connect or telnet proxy using tn-gw-nav.

Junio C Hamano <junkio@cox.net> writes:

I realize the above does not really convey my real objection.

Your "ssh to the proxy/firewall host and run netcat to the destination" would not work for me to reach the internal hosts at all (while it would work for external ones), because my firewall does not know names of our internal hosts (the same for using tn-gw-nav to cross http or telnet proxy).

Junio C Hamano <junkio@cox.net> writes:

"Run a program and talk to it via stdin/stdout" is as general as it gets, isn't it? ssh+netcat is just what I happen to use.

It doesn't have to be unconditional.  For example, one could have:

    if on_blargco_network; then
        # internal
        case "$1" in
            *.blargco.com)
                exec nc "$1" "$2"
                ;;
            *)
                exec ssh bastion.blargco.com nc "$1" "$2"
                ;;
        esac
    else
        # external
        case "$1" in
            *.blargco.com)
                exec ssh bastion.blargco.com nc "$1" "$2"
                ;;
            *)
                exec ssh bastion nc "$1" "$2"
                ;;
        esac
    fi

But perhaps I do not really understand your objection.

Paul Collins <paul@briny.ondioline.org> writes:

No, I think you are getting it right.

I just wanted to avoid using the proxy script for some hosts, depending on where you are going. Obviously you can teach the proxy script to do passthru for some hosts like you did in your message. The only difference is where the configuration is specified. I wanted it to be in the git configuration file (i.e. using different proxy script or no script, depending on the host). Your example has that configuration wired in the single script that is always called regardless of the destination, and the script itself switches how it proxies, depending on where it is going, perhaps using its own configuration file or hardcoding.

Junio C Hamano <junkio@cox.net> writes:

I had some ideas along those lines, but I didn't like any of them.

 * Extend the proxy-command "protocol" with a third argument, an
   action.  For example if 'query $host $port' returns successfully,
   then git should run it with arguments 'connect $host $port',
   otherwise use git_tcp_connect().

 * Add a Proxy-Command field to the files in .git/remotes, e.g.:

     URL: git://git.kernel.org/pub/scm/git/git.git/
     Pull: master:origin
     Proxy-command: my-git-proxy-command

 * If the git config syntax is extended to allow dots in section or
   key names:

     [proxy]
     git.kernel.org = "ssh-to-bastion-proxy-command"
     git.blargco.com = "blargco-proxy-command"

   or perhaps

     [git.kernel.org]
     proxycommand = "ssh-to-bastion-proxy-command"
     [git.blargco.com]
     proxycommand = "blargco-proxy-command"

On Fri, 4 Nov 2005, Paul Collins wrote:

I would suggest

	[proxy]
		command = "ssh-to-bastion-proxy-command" for git.kernel.org

and then it's easy enough to just parse the value "proxy.command" with code like

	host = value;
	cmd = strstr(value, " for ");
	if (!cmd)
		return -1;
	*cmd = 0;
	cmd += 5;

which would do the right thing..

The thing is, it's not just ".". I could well imagine that you'd have

	[proxy]
		command="ssh" for "ssh://kernel.org/"
		command="proxy-command" for kernel.org
		command="myprotocol-command" for "my://"

which would actually allow you to literally add your own protocol names (it would see that the target starts with "my://", and decide that it shoul drun the "myprotocol-command" for the proxy).

I'd rather allow free-form strings for the values than for the key names. If we allow free-form key-names, then random text files suddenly often become valid (but strange) config files.

Right now the non-free-form key names are the strongest syntax checker of the whole protocol.

		Linus

Paul Collins <paul@briny.ondioline.org> writes:

Sorry for holding onto your patch without any action (other than looking at it, which from your end you cannot tell I was doing ;-). I think we will benefit from your patch in some form, and here is what I tried on top to update it to the multivalue configuration syntax Linus suggested during the list discussion. Requesting for comments the original author and from the general public.

I suspect this touches the same area as the user-path series Andreas Ericsson has been working on, so it might need some adjusting after that patch series goes in. I'll see that soon enough when I make the proposed updates branch tomorrow.

 -- >8 --

This builds on top of the git-proxy mechanism Paul Collins did, and updates its configuration mechanism.

 * GIT_PROXY_COMMAND environment variable is used as the
   catch-all fallback, as in the original.  This has not
   changed.

 * Renames proxy configuration variables to core.gitproxy; this
   has become a multi-value variable per list discussion, most
   notably from suggestion by Linus.

	[core]
	;# matches www.kernel.org as well
	gitproxy = netcatter for kernel.org
	gitproxy = netscatter for sample.xz
	gitproxy = none for mydomain.xz
	gitproxy = netcatter-default

   The values are command names, followed by an optional " for "
   and domainname; the first tail-match of the domainname
   determines which proxy command is used.  An entry without "
   for " matches any domain and can be used as the default.

   The command name "none" is special -- it tells the mechanism
   not to use any proxy command and use the native git://
   connection.

Signed-off-by: Junio C Hamano <junkio@cox.net>

---

 connect.c |   52 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 45 insertions(+), 7 deletions(-)

applies-to: d4a5ace05fd892cb455f330d379cdaa69dc6005c c1d7ac7b4680202d3549a72188f372367ac8837c