git+http:// proof-of-concept (not using CONNECT)

I've been toying around with an HTTP server for various things over the past few months and a few weeks ago I came up with the idea of tunneling arbitrary stream protocols over HTTP with "Transfer-Encoding: chunked" bodies on both ends. This allowed me to tunnel the git:// protocol among other things over something that may conform to RFC 2616.

To facilitate this, curl needed only one minor modification (now in their CVS repository) to make stdin with "-T-" non-blocking:

  http://cool.haxx.se/cvs.cgi/curl/src/main.c.diff?r1=1.520&r2=1.521

Since git clone already supports proxy commands via GIT_PROXY_COMMAND, I just created a one line shell script to use as GIT_PROXY_COMMAND. No modifications were needed to git itself on either end:

cat > /path/to/script <<\EOF #!/bin/sh exec curl --no-buffer -sSfT- http://$1:$2/ EOF

chmod +x /path/to/script GIT_PROXY_COMMAND=/path/to/script export GIT_PROXY_COMMAND

# Then, to test it on a small project on my server: git clone git://git.bogomips.org:8080/pcu

# Of course, a larger project like git should work, too: git clone git://git.bogomips.org:8080/mirrors/git

Hopefully it doesn't blow up or die on you, this is only lightly tested. Let me know if you manage to break it permanently.

The origin server I'm running for this is the latest release of Unicorn[2], which supports sending a chunked HTTP response as it is receiving a chunked request. Unicorn just dechunks the request and pipes it to git-daemon. When git-daemon writes to stdout, Unicorn just grabs the output and chunks it (via Rack[3] middleware) for the client.

This doesn't work in the face of most HTTP-aware proxies[1], so it probably doesn't help those who have trouble accessing git:// servers in the first place... However, this could potentially be useful in places where a proxy providing CONNECT is not available.

[1] - I run nginx on port 80 on bogomips.org and nginx (attempts to) fully buffer all requests before proxying it to the backend, so it definitely won't fly here. HTTP proxies are perfectly alright with taking chunked requests/responses and remove chunking on them (or vice versa).

[2] - http://unicorn.bogomips.org/ [3] - http://rack.rubyforge.org/

On Thu, Jul 2, 2009 at 12:54 PM, Eric Wong<normalperson@yhbt.net> wrote:

The current lore is that to work with http proxies two separate streams are needed, one for input and one for outputs.

This technique is described at http://xmpp.org/extensions/xep-0124.html#technique and other places.

Regards, Constantine

Constantine Plotnikov <constantine.plotnikov@gmail.com> wrote:

Actually, the problem with my approach here is that any HTTP proxies are free to rewrite chunked requests/responses to non-chunked ones (for compression, ssl, or even performance reasons). That and my approach requires both the client and server to be simutaneously sending and receiving responses in full-duplex channel which makes it impossible to work without chunking. IOW, there's no chance any HTTP proxy that dechunks or queues/coalesces chunked requests/responses can work with my approach.

The big upside is that only one HTTP request is needed for a full clone, making this the most efficient way of using git over HTTP so far :)

On Fri, 3 Jul 2009, Eric Wong wrote:

Many HTTP servers will not be able to support it either because HTTP is a half-duplex protocol (modulo request pipelining).

Tony.

Tony Finch <dot@dotat.at> wrote:

Heh, as far as I know Unicorn is the only one :)

Any news about this approach? I've heard some noise about a CGI implementation....

Douglas Campos <douglas@theros.info> wrote:

Hi Douglass,

I've been busy with other things. This approach is probably too limited to get around HTTP-aware proxies to be useful for non-git:// users anyways.

It also turns out the curl patch that made it into 7.19.6 was still unsuitable for bidirectional tunneling, looks like I'll have to port curl over to the curl_multi_* interface to get it working properly instead...