{"thread":{"id":"19999","subject":"git+http:// proof-of-concept (not using CONNECT)","startedAt":"2009-07-02T08:54:40Z","lastAt":"2009-09-05T08:23:13Z","messageCount":7,"participants":["Eric Wong","Constantine Plotnikov","Tony Finch","Douglas Campos"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"117349","messageId":"20090702085440.GC11119@dcvr.yhbt.net","threadId":"19999","inReplyTo":null,"subject":"git+http:// proof-of-concept (not using CONNECT)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-07-02T08:54:40Z","receivedAt":"2009-07-02T08:54:40Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"I've been toying around with an HTTP server for various things over the\npast few months and a few weeks ago I came up with the idea of tunneling\narbitrary stream protocols over HTTP with \"Transfer-Encoding: chunked\"\nbodies on both ends.  This allowed me to tunnel the git:// protocol\namong other things over something that may conform to RFC 2616.\n\nTo facilitate this, curl needed only one minor modification (now in\ntheir CVS repository) to make stdin with \"-T-\" non-blocking:\n\n  http://cool.haxx.se/cvs.cgi/curl/src/main.c.diff?r1=1.520&r2=1.521\n\nSince git clone already supports proxy commands via GIT_PROXY_COMMAND, I\njust created a one line shell script to use as GIT_PROXY_COMMAND.  No\nmodifications were needed to git itself on either end:\n\ncat > /path/to/script <<\\EOF\n#!/bin/sh\nexec curl --no-buffer -sSfT- http://$1:$2/\nEOF\n\nchmod +x /path/to/script\nGIT_PROXY_COMMAND=/path/to/script\nexport GIT_PROXY_COMMAND\n\n# Then, to test it on a small project on my server:\ngit clone git://git.bogomips.org:8080/pcu\n\n# Of course, a larger project like git should work, too:\ngit clone git://git.bogomips.org:8080/mirrors/git\n\nHopefully it doesn't blow up or die on you, this is only lightly\ntested.  Let me know if you manage to break it permanently.\n\n\nThe origin server I'm running for this is the latest release of\nUnicorn[2], which supports sending a chunked HTTP response as it is\nreceiving a chunked request.  Unicorn just dechunks the request and\npipes it to git-daemon.  When git-daemon writes to stdout, Unicorn just\ngrabs the output and chunks it (via Rack[3] middleware) for the client.\n\n\nThis doesn't work in the face of most HTTP-aware proxies[1], so it\nprobably doesn't help those who have trouble accessing git:// servers in\nthe first place...  However, this could potentially be useful in places\nwhere a proxy providing CONNECT is not available.\n\n\n[1] - I run nginx on port 80 on bogomips.org and nginx (attempts to)\nfully buffer all requests before proxying it to the backend, so it\ndefinitely won't fly here.  HTTP proxies are perfectly alright with\ntaking chunked requests/responses and remove chunking on them (or\nvice versa).\n\n[2] - http://unicorn.bogomips.org/\n[3] - http://rack.rubyforge.org/\n\n-- \nEric Wong\n"},{"id":"117351","messageId":"85647ef50907020252u41e36187jaacacad3d8a10f75@mail.gmail.com","threadId":"19999","inReplyTo":"20090702085440.GC11119@dcvr.yhbt.net","subject":"Re: git+http:// proof-of-concept (not using CONNECT)","fromName":"Constantine Plotnikov","fromEmail":"constantine.plotnikov@gmail.com","sentAt":"2009-07-02T09:52:22Z","receivedAt":"2009-07-02T09:52:22Z","isPatch":false,"sender":{"key":"constantine.plotnikov@gmail.com","avatar":null},"body":"On Thu, Jul 2, 2009 at 12:54 PM, Eric Wong<normalperson@yhbt.net> wrote:\n> This doesn't work in the face of most HTTP-aware proxies[1], so it\n> probably doesn't help those who have trouble accessing git:// servers in\n> the first place...  However, this could potentially be useful in places\n> where a proxy providing CONNECT is not available.\n>\nThe current lore is that to work with http proxies two separate\nstreams are needed,\none for input and one for outputs.\n\nThis technique is described at\nhttp://xmpp.org/extensions/xep-0124.html#technique\nand other places.\n\nRegards,\nConstantine\n"},{"id":"117426","messageId":"20090703202839.GB12072@dcvr.yhbt.net","threadId":"19999","inReplyTo":"85647ef50907020252u41e36187jaacacad3d8a10f75@mail.gmail.com","subject":"Re: git+http:// proof-of-concept (not using CONNECT)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-07-03T20:28:39Z","receivedAt":"2009-07-03T20:28:39Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Constantine Plotnikov <constantine.plotnikov@gmail.com> wrote:\n> On Thu, Jul 2, 2009 at 12:54 PM, Eric Wong<normalperson@yhbt.net> wrote:\n> > This doesn't work in the face of most HTTP-aware proxies[1], so it\n> > probably doesn't help those who have trouble accessing git:// servers in\n> > the first place...  However, this could potentially be useful in places\n> > where a proxy providing CONNECT is not available.\n> >\n> The current lore is that to work with http proxies two separate\n> streams are needed,\n> one for input and one for outputs.\n> \n> This technique is described at\n> http://xmpp.org/extensions/xep-0124.html#technique\n> and other places.\n\nActually, the problem with my approach here is that any HTTP proxies are\nfree to rewrite chunked requests/responses to non-chunked ones (for\ncompression, ssl, or even performance reasons).  That and my approach\nrequires both the client and server to be simutaneously sending and\nreceiving responses in full-duplex channel which makes it impossible to\nwork without chunking.  IOW, there's no chance any HTTP proxy that\ndechunks or queues/coalesces chunked requests/responses can work with\nmy approach.\n\nThe big upside is that only one HTTP request is needed for a full clone,\nmaking this the most efficient way of using git over HTTP so far :)\n\n-- \nEric Wong\n"},{"id":"117568","messageId":"alpine.LSU.2.00.0907071712580.30197@hermes-2.csi.cam.ac.uk","threadId":"19999","inReplyTo":"20090703202839.GB12072@dcvr.yhbt.net","subject":"Re: git+http:// proof-of-concept (not using CONNECT)","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-07-07T16:14:31Z","receivedAt":"2009-07-07T16:14:31Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Fri, 3 Jul 2009, Eric Wong wrote:\n>\n> That and my approach requires both the client and server to be\n> simutaneously sending and receiving responses in full-duplex channel\n> which makes it impossible to work without chunking.  IOW, there's no\n> chance any HTTP proxy that dechunks or queues/coalesces chunked\n> requests/responses can work with my approach.\n\nMany HTTP servers will not be able to support it either because HTTP is a\nhalf-duplex protocol (modulo request pipelining).\n\nTony.\n-- \nf.anthony.n.finch  <dot@dotat.at>  http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"117592","messageId":"20090707205003.GA31195@dcvr.yhbt.net","threadId":"19999","inReplyTo":"alpine.LSU.2.00.0907071712580.30197@hermes-2.csi.cam.ac.uk","subject":"Re: git+http:// proof-of-concept (not using CONNECT)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-07-07T20:50:03Z","receivedAt":"2009-07-07T20:50:03Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Tony Finch <dot@dotat.at> wrote:\n> On Fri, 3 Jul 2009, Eric Wong wrote:\n> >\n> > That and my approach requires both the client and server to be\n> > simutaneously sending and receiving responses in full-duplex channel\n> > which makes it impossible to work without chunking.  IOW, there's no\n> > chance any HTTP proxy that dechunks or queues/coalesces chunked\n> > requests/responses can work with my approach.\n> \n> Many HTTP servers will not be able to support it either because HTTP is a\n> half-duplex protocol (modulo request pipelining).\n\nHeh, as far as I know Unicorn is the only one :)\n\n-- \nEric Wong\n"},{"id":"121804","messageId":"ed88cb980908260734x78382052t30d4cdcf07451134@mail.gmail.com","threadId":"19999","inReplyTo":"20090707205003.GA31195@dcvr.yhbt.net","subject":"Re: git+http:// proof-of-concept (not using CONNECT)","fromName":"Douglas Campos","fromEmail":"douglas@theros.info","sentAt":"2009-08-26T14:34:05Z","receivedAt":"2009-08-26T14:34:05Z","isPatch":false,"sender":{"key":"douglas@theros.info","avatar":"https://gravatar.com/avatar/bca8e0e43e860dab00ec17dd3f1b60df6af911655fae70a0bbdeebeb889de421?d=mp&s=160"},"body":"Any news about this approach? I've heard some noise about a CGI\nimplementation....\n\n-- \nDouglas Campos\nqmx.me\n"},{"id":"122496","messageId":"20090905082313.GA26131@dcvr.yhbt.net","threadId":"19999","inReplyTo":"ed88cb980908260734x78382052t30d4cdcf07451134@mail.gmail.com","subject":"Re: git+http:// proof-of-concept (not using CONNECT)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-09-05T08:23:13Z","receivedAt":"2009-09-05T08:23:13Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Douglas Campos <douglas@theros.info> wrote:\n> Any news about this approach? I've heard some noise about a CGI\n> implementation....\n\nHi Douglass,\n\nI've been busy with other things.  This approach is probably\ntoo limited to get around HTTP-aware proxies to be useful for\nnon-git:// users anyways.\n\nIt also turns out the curl patch that made it into 7.19.6 was still\nunsuitable for bidirectional tunneling, looks like I'll have to port\ncurl over to the curl_multi_* interface to get it working properly\ninstead...\n\n-- \nEric Wong\n"}]}