git over webdav: what can I do for improving http-push ?

Hi everybody.

I've just subscribed to the list, therefore I think it should be rude not to introduce myself. My name is Grégoire Barbier (non french-speaking people should call me Greg and don't bother with non-ascii characters), I'm working mainly as a consultant (using Powerpoint and wearing a tie) but have some personal and professional interest in programming, especially about middlewares. BTW I apologize for my poor english.

I'm using Git for a rather short time but enough to fall in love with it. For a few days I'm trying to use it over webdav, that is over http/https with write (push) access. As for me, the main rationale to use http(s) rather than git or ssh is to get through corporate firewalls, otherwise I would probably not bother with webdav.

With 1.5.3.6 and 1.5.4-rc2, I encounter severe issues that make me think that http-push is not totally ready for production. That's why I would like to have a discussion with some of you that use and maintain it, to see what I can do to improve it or to help you improve it.

Here are some issues I encountered:
- http-push does not release locks when failing due to syntax error 
(e.g. if one types "git push" instead of "git push origin master")
- http-push freezes with no message with urls not terminated by a slash
- http-push does not create directory for the object (objects/xx/) and 
if the directory exists, it does not actually push objects without 
having USE_CURL_MULTI defined (which is not the compilation default)

I've starting to look at the source code and make some little improvements, but I feel that I should rather discuss with you to understand why there are two rather independant modes in http-push (USE_CURL_MULTI or not) and what is the real target (I don't want to work twice, neither to mess up the work of someone else that would be currently reorginzing this part of the code).

I attached some patches I did against 1.5.4-rc2, but I'm not sure they are doing it the good way, so I wouldn't be suprised if you were not okay to apply them as is.

On Sun, 30 Dec 2007, Grégoire Barbier wrote:

In general, we've been able to either get through firewalls with ssh or it's all in the same VPN. So it's kind of unloved at this point. People poke at it occasionally, but mostly in the context of other fixes, I think.

I think the issue is the CURL_MULTI library code is either not supported or is broken in versions of curl that many distros still ship, but we can do a lot better with it, so the duplicate implementation is plausibly worthwhile.

One thing I personally thought would be worthwhile would be to separate out the logic for sending stuff like the fetching logic is in walker.{c,h}, and include the necessary methods in struct walker. There were people interested in sftp (for the case where you can get ssh through firewalls, but you aren't allowed to install programs on the file server and git isn't installed system-wide).

One thing that's worth doing when looking at the code is using "git blame" to find out where the lines you're changing came from, and "git log <hash>" to find out what the person writing them was trying to do. This will also turn up the people who've been working in the area, who you might want to cc, since they'll be good reviewers.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow wrote:

If you have a http proxy that you can use, the you can use ssh via that with something like corkscrew. http://wiki.kartbuilding.net/index.php/Corkscrew_-_ssh_over_https

A simple shell script wrapper around ssh to detect when you are behind a firewall can inject the ProxyCommand into the command line arguments with -o

Graham.

On Mon, Dec 31, 2007 at 10:57:52 -0600, Graham Barr wrote:

This, obviously, requires, that ssh is running on port 443, because most HTTP proxies won't let you CONNECT anywhere else. I have also heared of a HTTP proxy, that will check whether the session inside CONNECT starts with SSL handshake and will break your connection if it does not.

Most of the time simply setting the parameter in .ssh/config works better -- because you are often behind a proxy for some sites only.

Jan Hudec a écrit :

Hello Jan.

I think we have similar experiences. I have personnaly be faced to proxies that not only scan for the SSL handshake but do man-in-the-middle "attack" to break the SSL into two parts, checking for HTTP inside it (and probably scanning for viruses and things like hat, I think).

I first replied privatly to Graham because I didn't think it was interesting for the whole list. It was a mistake, here is my answer:

In fact, I already use this hack where it is possible.

However some well advised companies does not allow CONNECT through their HTTP proxy without some limitations that make this tip unusable (for instance: allowing only port 443, allowing only sites of a white-list, forcing a man-in-the-middle that not only breaks the confidentiality but too forbids the use of other protocols such as SSH, even on port 443).

BTW such circumvention of the security facilities is often (at less where I live and with my clients) forbidden in some corporate rules, even when it is technically possible. Therefore I'm not allowed to do so and, furthermore, I cannot tell my clients to do so and write documents that tell it's the good way.

I think that real HTTP support is better than all workarounds we will be able to find to get through firewalls (when CONNECT is not available, some awful VPNs that send Etherne over HTTP may work ;-)). That's why I'm ok to work several hours on git code to enhance real HTTP(S) support.

Grégoire Barbier <gb@gbarbier.org> writes:

There was also an idea to create a CGI program, or enhance gitweb to use for pushing. I don't know if it would be better way to pursue to work around corporate firewalls, or not...

On Tue, Jan 01, 2008 at 10:12:28 -0800, Jakub Narebski wrote:

It is what bzr and mercurial do and I think it would be quite good way to go for cases like this. Eg. while our corporate firewall does allow anything through connect on 443 (so I can use ssh that way), it does *not* support web-dav in non-ssl mode. So I eg. can't even get from public subversion repositories at work.

I have also thought about optimizing download using CGI, but than I thought, that maybe there is a way to statically generate packs so, that if the client wants n revisions, the number of revisions it downloads is O(n) and the number of packs it gets them from (and thus number of round-trips) is O(log(n)). Assuming the client always wants everything up to the tip, of course. Now this is trivial with linear history (pack first half, than half of what's left, etc., gives logarithmic number of packs and you always download at most twice as much as you need), but it would be nice if somebody found a way (even one that satisfies the conditions on average only) to do this with non-linear history, it would be very nice improvement to the http download -- native git server optimizes amount of data transfered very well, but at the cost of quite heavy CPU load on the server.

Jan Hudec a écrit :

I subscribe to this point of view. I will look at the list archive to search for what has been said before about this.

Ok, I will have to look at bzr and mercurial...

Well... frankly I don't think I'm able of such things. Writing a walker over webdav or a simple cgi is a thing I can do (I think), but I'm not tought enough (or not ready to take the time needed) to have a look on the internals of packing revisions (whereas I can imagine it would means that "my" walker would be suitable only for small projects in terms of code amount and commit frequency).

I had a quick look on bzr and hg, and it seems that bzr use the easy way (walker, no optimizations) and hg a cgi (therefore, maybe optimizations). By quick look I mean that I sniff the HTTP queries on the network during a clone. I need to look harder...

BTW I never looked at the git:// protocol. Do you think that by tunneling the git protocol in a cgi (hg uses URLs of the form "/mycgi?cmd=mycommand&...", therefore I think "tunnel" is not a bad word...) the performance would be good? Maybe it's not that hard to write a performant HTTP/CGI protocol for Git if it's based upon existing code such as the git protocol.

On Thu, Jan 03, 2008 at 20:14:09 +0100, Grégoire Barbier wrote:

[...]

Bzr is quite far, design-wise, I fear. The mercurial might be a little more interesting to study, but being in python and internally somewhat file-oriented, I wouldn't think it is of much use.

You should start with upload, leaving the download direction to the dumb machinery git currently uses.

[...]

Well, it does not depend on the walker -- the walker is quite simple and already written anyway.

That's not quite true -- bzr has both dumb (walker over plain HTTP) and smart (CGI) methods. But their CGI is really just tunelling their custom protocol over HTTP and that protocol will not be anywhere near what we want for git because of vastly different design of the storage.

Yes, mercurial uses a CGI. But I am not sure how similar their approach is to anything that would make sense for git, so looking at the details might or might not be useful.

It would be pretty hard to tunnel it and it would loose all it's nice properties. The git protocol, for pull, basically works like this:

 - server sends a list of it's refs
 - client tells server which ones it wants
 - client starts listing revisions it has, newest to oldest
 - server tells client whenever it finds common ancestor with one of the
   heads desired
 - client restarts the listing from next ref
 - server starts sending the data when client runs out of refs to list

The main point about the protocol is, that the client is listing the refs, as fast as it can and server will stop it when it sees a revision it knows. Therefore there will only be one round-trip to discover each common ancestor.

However, you can't do this over HTTP, because response won't be started until the request is received. You could be sending a lot of smallish requests and quick, often empty, responses to them. However, that will waste a lot of bandwidth (because of the HTTP overhead) and loose much of the speed anyway. Also the HTTP protocol is stateless, but this is inherently stateful, so you would have to work that around somehow too. Therefore a different approach is preferable on HTTP.

Now to keep it stateless, I thought that:
 - client would first ask for list of refs
 - client would than ask for pack containing the first ref
 - server would respond with pack containing just that commit plus all
   objects that are not referenced by any of it's parents
 - if client does not have it's parent, it would ask for pack containing that
 - since it's second request, server would pack 2 revisions (with necessary
   objects) this time
 - if client still does not have all parents, it would again ask for a pack,
   receiving 4 revisions this time (next 8, 16, etc...)

This would guarantee, that when you want n revisions, you make at most log2(n) requests and get at most 2*n revisions (well, the requests are for each ref separately, so it's more like m*log2(n) where m is number of refs, but still). Additionally, it would be stateless, because the client would simply say 'I want commit abcdef01 and this is my 3rd request' and server would provide that commit and 7 it's parents (ie. 2^3 commits).

Now generating the packs takes it's CPU. The servers like git.kernel.org have quite high CPU load. But in this schema, all clients would most of the time get the same packs (unlike native git protocol, where the client always gets single pack with exactly what it needs). So the idea struck me, that they could simply be statically generated and fetched via the existing dumb protocol. It would get the efficiency and save a lot of CPU power, which is allows one to serve quite busy git repository from a limited (and therefore cheap) virtual machine or even (yes, I saw such idea on the list) serving any repository from NSLU2.

Now to create a packing policy to create such packs, you don't actually need to touch any C -- because git-repack is still in shell -- and you don't really need to touch any internals, because you only need to change how the pack-objects command will be called and leave all the dirty details to that command.

I would personally not re-split the already generated packs. Only find some algorithm for choosing when packs are deep enough in history so they should be merged together. It would also might not make sense to ever pack unpacked objects to more than one pack -- a dumb-HTTP-served might have a requirement of running this kind of repack after every push and clients will rarely have part of single push to the server.

For push it might, or might not be easy. But in the worst case you should be able to calculate a pack to upload locally (fetching from the server beforehead if necessary), upload that pack (or bundle) and update all the refs.

For pull it certainly won't be. You might be able to reimplement the common ref discovery using some kind of gradually increasing ref list and then generate a bundle for the server, but optimizing the dumb protocol seems more useful to me. As I said, generating the packs only requires devising a way of selecting which objects should go together and git pack-objects will take care of the dirty details of generating the packs and git update-server-info will take care of the dirty details of presenting the list of packs to client.

On Thu, 3 Jan 2008, Jan Hudec wrote:

Well, tunnelling the native git protocol is *exactly* what you'd want to do with some git CGI thing. So no, I don't think the actual stuff you tunnel would have any relationship, but the actual code to set up a tunnel and making it all look like some fake html sequence might be something that can be used as a base.

		Linus

Jan Hudec wrote:

Perhaps we could also simply tunnel git protocol over HTTP / HTTPS?
 

Perhaps we could use AJAX (XMLHttpRequest for communication, plain HTTP or IFRAMES for sending data) or something like this for git protocol tunneling?

Jakub Narebski a écrit :

well... I think I may manage to avoid javascript... ;-) more seriously, I was thinking of using http in an automated, un-human-browsable manner, not a full html user interface.

On Jan 4, 2008 10:15 AM, Jan Hudec <bulb@ucw.cz> wrote:

...

That is still a lot! How about, for each ref

 - Client sends a POST listing the ref and the latest related commit
it has that the server is likely to have (from origin/heads/<ref>).
Optionally, it can provide a blacklist of <treeish> (where every
object refered is known) and blob sha1s.
 - Server sends the new sha1 of the ref, and a thin pack that covers the changes
 - The client can disconnect to stop the transaction. For example --
if it sees the sha1 of a huge object that it already has. It can
re-request, with a blacklist.

A good number of objects will be sent unnecesarily - with no option to the client to say "I have this" - but by using the hint of letting the server know we have origin/heads/<ref> I suspect that it will be minimal.

Also:
 - It will probably be useful to list all the refs the client knows
from that server in the request.
 - If the ref has changed with a non-fast-forward, the server needs to
say so, and provide a listing of the commits. As soon as the client
spots a common commit, it can close the connection -- it now knows
what ref to tell the server about in a subsequent command.

This way, you ideally have 1 request per ref, 2 if it has been rebased/rewound. This can probably get reorganised to do several refs in one request.

cheers,

m

On Fri, Jan 04, 2008 at 12:54:58 +1300, Martin Langhoff wrote:

The whole point of that is that the packs can be statically precomputed and served with quite low CPU load, which is useful for serving from shared computers (like servers in school computer labs or cheapo web hosting) or slow servers like NSLU2. Also it makes HTTP caching actually useful, because the set of possible requests is quite limited.

Also, while I said it's for each ref, the packs should really be optimized for the common case of fetching all refs, which would really make it just log2(n) packs and 2*n revisions for each whole download.

It would be better to only unnecesarily send revlists. Since each HTTP packed will likely have something like 1kb overhead, sending few kb worth of revlist is still pretty efficient. So just send part of revlist, than more revlist and so on until you find exactly which revisions you need and than ask for them. That will save *both* bandwidth *and* server CPU. The only reason to waste bandwidth is to save CPU and you are not doing that.