{"thread":{"id":"627","subject":"Re: Mercurial 0.4e vs git network pull","startedAt":"2005-05-15T11:22:19Z","lastAt":"2005-05-16T22:22:11Z","messageCount":7,"participants":["Adam J. Richter","Petr Baudis","Matt Mackall","Jeff Garzik","Matthias Urlichs","Tristan Wibberley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"3353","messageId":"200505151122.j4FBMJa01073@adam.yggdrasil.com","threadId":"627","inReplyTo":null,"subject":"Re: Mercurial 0.4e vs git network pull","fromName":"Adam J. Richter","fromEmail":"adam@yggdrasil.com","sentAt":"2005-05-15T11:22:19Z","receivedAt":"2005-05-15T11:22:19Z","isPatch":false,"sender":{"key":"adam@yggdrasil.com","avatar":null},"body":"On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote:\n>Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter\n>where Matt Mackall told me that...\n>> Does this need an HTTP request (and round trip) per object? It appears\n>> to. That's 2200 requests/round trips for my 800 patch benchmark.\n\n>Yes it does. On the other side, it needs no server-side CGI. But I guess\n>it should be pretty easy to write some kind of server-side CGI streamer,\n>and it would then easily take just a single HTTP request (telling the\n>server the commit ID and receiving back all the objects).\n\n\tI don't understand what was wrong with Jeff Garzik's previous\nsuggestion of using http/1.1 pipelining to coalesce the round trips.\nIf you're worried about queuing too many http/1.1 requests, the client\ncould adopt a policy of not having more than a certain number of\nrequests outstanding or perhaps even making a new http connection\nafter a certain number of requests to avoid starving other clients\nwhen the number of clients doing one of these transfers exceeds the\nnumber of threads that the http server uses.\n\n\tBeing able to do without a server side CGI script might\nencourage deployment a bit more, both for security reasons and\neffort of deployment.\n\n\tIn any case, using httpd or ftp makes it easier to deploy\nservers in cases where it might be harder to modify firewall rules,\nso I am glad to see that, even if it is through a CGI script.\n\n __ ______________\nAdam J. Richter \\ /\nadam@yggdrasil.com | g g d r a s i l\n"},{"id":"3354","messageId":"20050515124042.GE13024@pasky.ji.cz","threadId":"627","inReplyTo":"200505151122.j4FBMJa01073@adam.yggdrasil.com","subject":"Re: Mercurial 0.4e vs git network pull","fromName":"Petr Baudis","fromEmail":"pasky@ucw.cz","sentAt":"2005-05-15T12:40:42Z","receivedAt":"2005-05-15T12:40:42Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sun, May 15, 2005 at 01:22:19PM CEST, I got a letter\nwhere \"Adam J. Richter\" told me that...\n> On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote:\n> >Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter\n> >where Matt Mackall told me that...\n> >> Does this need an HTTP request (and round trip) per object? It appears\n> >> to. That's 2200 requests/round trips for my 800 patch benchmark.\n> \n> >Yes it does. On the other side, it needs no server-side CGI. But I guess\n> >it should be pretty easy to write some kind of server-side CGI streamer,\n> >and it would then easily take just a single HTTP request (telling the\n> >server the commit ID and receiving back all the objects).\n> \n> \tI don't understand what was wrong with Jeff Garzik's previous\n> suggestion of using http/1.1 pipelining to coalesce the round trips.\n> If you're worried about queuing too many http/1.1 requests, the client\n> could adopt a policy of not having more than a certain number of\n> requests outstanding or perhaps even making a new http connection\n> after a certain number of requests to avoid starving other clients\n> when the number of clients doing one of these transfers exceeds the\n> number of threads that the http server uses.\n\nThe problem is that to fetch a revision tree, you have to\n\n\tsend request for commit A\n\treceive commit A\n\tlook at commit A for list of its parents\n\tsend request for the parents\n\treceive the parents\n\tlook inside for list of its parents\n\t...\n\n(and same for the trees).\n\n> \tBeing able to do without a server side CGI script might\n> encourage deployment a bit more, both for security reasons and\n> effort of deployment.\n\nYou could still use it without the server side CGI script as it is now,\njust without the speedups.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nC++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor\n"},{"id":"3370","messageId":"20050515173923.GK5914@waste.org","threadId":"627","inReplyTo":"200505151122.j4FBMJa01073@adam.yggdrasil.com","subject":"Re: Mercurial 0.4e vs git network pull","fromName":"Matt Mackall","fromEmail":"mpm@selenic.com","sentAt":"2005-05-15T17:39:23Z","receivedAt":"2005-05-15T17:39:23Z","isPatch":false,"sender":{"key":"mpm@selenic.com","avatar":null},"body":"On Sun, May 15, 2005 at 04:22:19AM -0700, Adam J. Richter wrote:\n> On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote:\n> >Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter\n> >where Matt Mackall told me that...\n> >> Does this need an HTTP request (and round trip) per object? It appears\n> >> to. That's 2200 requests/round trips for my 800 patch benchmark.\n> \n> >Yes it does. On the other side, it needs no server-side CGI. But I guess\n> >it should be pretty easy to write some kind of server-side CGI streamer,\n> >and it would then easily take just a single HTTP request (telling the\n> >server the commit ID and receiving back all the objects).\n> \n> \tI don't understand what was wrong with Jeff Garzik's previous\n> suggestion of using http/1.1 pipelining to coalesce the round trips.\n\nYou can't do pipelining if you can't look ahead far enough to fill the pipe.\n\n-- \nMathematics is the supreme nostalgia of our time.\n"},{"id":"3373","messageId":"428793A1.5070004@pobox.com","threadId":"627","inReplyTo":"20050515173923.GK5914@waste.org","subject":"Re: Mercurial 0.4e vs git network pull","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-05-15T18:23:29Z","receivedAt":"2005-05-15T18:23:29Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Matt Mackall wrote:\n> On Sun, May 15, 2005 at 04:22:19AM -0700, Adam J. Richter wrote:\n> \n>>On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote:\n>>\n>>>Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter\n>>>where Matt Mackall told me that...\n>>>\n>>>>Does this need an HTTP request (and round trip) per object? It appears\n>>>>to. That's 2200 requests/round trips for my 800 patch benchmark.\n>>\n>>>Yes it does. On the other side, it needs no server-side CGI. But I guess\n>>>it should be pretty easy to write some kind of server-side CGI streamer,\n>>>and it would then easily take just a single HTTP request (telling the\n>>>server the commit ID and receiving back all the objects).\n>>\n>>\tI don't understand what was wrong with Jeff Garzik's previous\n>>suggestion of using http/1.1 pipelining to coalesce the round trips.\n> \n> \n> You can't do pipelining if you can't look ahead far enough to fill the pipe.\n\nEven if you cannot fill a pipeline, HTTP/1.1 is sufficiently useful \nsimply by removing the per-request connection overhead.\n\n\tJeff\n\n\n"},{"id":"3404","messageId":"20050516011209.GM5914@waste.org","threadId":"627","inReplyTo":"428793A1.5070004@pobox.com","subject":"Re: Mercurial 0.4e vs git network pull","fromName":"Matt Mackall","fromEmail":"mpm@selenic.com","sentAt":"2005-05-16T01:12:09Z","receivedAt":"2005-05-16T01:12:09Z","isPatch":false,"sender":{"key":"mpm@selenic.com","avatar":null},"body":"On Sun, May 15, 2005 at 02:23:29PM -0400, Jeff Garzik wrote:\n> Matt Mackall wrote:\n> >On Sun, May 15, 2005 at 04:22:19AM -0700, Adam J. Richter wrote:\n> >\n> >>On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote:\n> >>\n> >>>Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter\n> >>>where Matt Mackall told me that...\n> >>>\n> >>>>Does this need an HTTP request (and round trip) per object? It appears\n> >>>>to. That's 2200 requests/round trips for my 800 patch benchmark.\n> >>\n> >>>Yes it does. On the other side, it needs no server-side CGI. But I guess\n> >>>it should be pretty easy to write some kind of server-side CGI streamer,\n> >>>and it would then easily take just a single HTTP request (telling the\n> >>>server the commit ID and receiving back all the objects).\n> >>\n> >>\tI don't understand what was wrong with Jeff Garzik's previous\n> >>suggestion of using http/1.1 pipelining to coalesce the round trips.\n> >\n> >\n> >You can't do pipelining if you can't look ahead far enough to fill the \n> >pipe.\n> \n> Even if you cannot fill a pipeline, HTTP/1.1 is sufficiently useful \n> simply by removing the per-request connection overhead.\n\nSure. It cuts round trips by a factor of 2. But that's just about all\nit does.\n\nMercurial already does:\n - approximately O(log(new changesets)) requests/data to find new changesets\n - one request to get an entire changegroup (set of all new\n changesets), which comes back all nicely pipelined and sorted by file\n - delta transfer\n\nIn \"dumb http\" mode, ie what's been there since about day three, it\ncan do:\n - one request (size proportional to total number of changesets) to\n find new changesets\n - approximately two requests per changed file to pull all deltas\n (vs request per file revision)\n - delta transfer\n\n-- \nMathematics is the supreme nostalgia of our time.\n"},{"id":"3411","messageId":"pan.2005.05.16.09.29.09.407174@smurf.noris.de","threadId":"627","inReplyTo":"200505151122.j4FBMJa01073@adam.yggdrasil.com","subject":"Re: Mercurial 0.4e vs git network pull","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-05-16T09:29:10Z","receivedAt":"2005-05-16T09:29:10Z","isPatch":false,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi, Adam J. Richter wrote:\n\n> \tBeing able to do without a server side CGI script might\n> encourage deployment a bit more, both for security reasons and\n> effort of deployment.\n\nA simple server-side CGI would be a \"send me all changeset SHA-1s,\nstarting at HEAD until you reach FOO\" operation (FOO being the SHA1 of\nthe previous head you've pulled before). This operation is simple enough\nthat it people should have no problem installing such a CGI.\n\nYou could then stream-pull the actual contents over HTTP/1.1 without\nfurther CGI interaction.\n\n-- \nMatthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de\n\n\n"},{"id":"3426","messageId":"1116282131.6141.8.camel__31554.9111379825$1116283578$gmane$org@localhost.localdomain","threadId":"627","inReplyTo":"20050515124042.GE13024@pasky.ji.cz","subject":"Re: Mercurial 0.4e vs git network pull","fromName":"Tristan Wibberley","fromEmail":"maihem@maihem.org","sentAt":"2005-05-16T22:22:11Z","receivedAt":"2005-05-16T22:22:11Z","isPatch":false,"sender":{"key":"maihem@maihem.org","avatar":null},"body":"On Sun, 2005-05-15 at 14:40 +0200, Petr Baudis wrote:\n> Dear diary, on Sun, May 15, 2005 at 01:22:19PM CEST, I got a letter\n> where \"Adam J. Richter\" told me that...\n> > \n> > \tI don't understand what was wrong with Jeff Garzik's previous\n> > suggestion of using http/1.1 pipelining to coalesce the round trips.\n> > If you're worried about queuing too many http/1.1 requests, the client\n> > could adopt a policy of not having more than a certain number of\n> > requests outstanding or perhaps even making a new http connection\n> > after a certain number of requests to avoid starving other clients\n> > when the number of clients doing one of these transfers exceeds the\n> > number of threads that the http server uses.\n> \n> The problem is that to fetch a revision tree, you have to\n> \n> \tsend request for commit A\n> \treceive commit A\n> \tlook at commit A for list of its parents\n> \tsend request for the parents\n> \treceive the parents\n> \tlook inside for list of its parents\n> \t...\n\nWhat about IMAP? You could ask for just the parents for several messages\n(via a message header), then start asking for message bodies (with the\njuicy stuff in). You could also ask for a list of the new commits then\nask for each of the bodies (several at a time). Not as good as a \"Just\ngive me all new data\", but an *awful* lot more efficient than HTTP. And\nvery flexible. You just need to map changesets to IMAP messages (if such\na mapping can actually make sense :)\n\nProlly a bit more work though.\n\n--\nTristan Wibberley\n\nThe opinions expressed in this message are my own opinions and not those\nof my employer.\n\n\n"}]}