# Re: Mercurial 0.4e vs git network pull 7 messages from 2005-05-15 to 2005-05-16. Participants: Adam J. Richter, Petr Baudis, Matt Mackall, Jeff Garzik, Matthias Urlichs, Tristan Wibberley. Thread: https://gitlist.dev/t/627 ## Adam J. Richter, 2005-05-15 11:22 Subject: Re: Mercurial 0.4e vs git network pull Message-ID: <200505151122.j4FBMJa01073@adam.yggdrasil.com> URL: https://gitlist.dev/e/200505151122.j4FBMJa01073%40adam.yggdrasil.com ``` On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote: >Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter >where Matt Mackall told me that... >> Does this need an HTTP request (and round trip) per object? It appears >> to. That's 2200 requests/round trips for my 800 patch benchmark. >Yes it does. On the other side, it needs no server-side CGI. But I guess >it should be pretty easy to write some kind of server-side CGI streamer, >and it would then easily take just a single HTTP request (telling the >server the commit ID and receiving back all the objects). I don't understand what was wrong with Jeff Garzik's previous suggestion of using http/1.1 pipelining to coalesce the round trips. If you're worried about queuing too many http/1.1 requests, the client could adopt a policy of not having more than a certain number of requests outstanding or perhaps even making a new http connection after a certain number of requests to avoid starving other clients when the number of clients doing one of these transfers exceeds the number of threads that the http server uses. Being able to do without a server side CGI script might encourage deployment a bit more, both for security reasons and effort of deployment. In any case, using httpd or ftp makes it easier to deploy servers in cases where it might be harder to modify firewall rules, so I am glad to see that, even if it is through a CGI script. __ ______________ Adam J. Richter \ / adam@yggdrasil.com | g g d r a s i l ``` ## Petr Baudis, 2005-05-15 12:40 Subject: Re: Mercurial 0.4e vs git network pull Message-ID: <20050515124042.GE13024@pasky.ji.cz> URL: https://gitlist.dev/e/20050515124042.GE13024%40pasky.ji.cz In-Reply-To: <200505151122.j4FBMJa01073@adam.yggdrasil.com> ``` Dear diary, on Sun, May 15, 2005 at 01:22:19PM CEST, I got a letter where "Adam J. Richter" told me that... > On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote: > >Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter > >where Matt Mackall told me that... > >> Does this need an HTTP request (and round trip) per object? It appears > >> to. That's 2200 requests/round trips for my 800 patch benchmark. > > >Yes it does. On the other side, it needs no server-side CGI. But I guess > >it should be pretty easy to write some kind of server-side CGI streamer, > >and it would then easily take just a single HTTP request (telling the > >server the commit ID and receiving back all the objects). > > I don't understand what was wrong with Jeff Garzik's previous > suggestion of using http/1.1 pipelining to coalesce the round trips. > If you're worried about queuing too many http/1.1 requests, the client > could adopt a policy of not having more than a certain number of > requests outstanding or perhaps even making a new http connection > after a certain number of requests to avoid starving other clients > when the number of clients doing one of these transfers exceeds the > number of threads that the http server uses. The problem is that to fetch a revision tree, you have to send request for commit A receive commit A look at commit A for list of its parents send request for the parents receive the parents look inside for list of its parents ... (and same for the trees). > Being able to do without a server side CGI script might > encourage deployment a bit more, both for security reasons and > effort of deployment. You could still use it without the server side CGI script as it is now, just without the speedups. -- Petr "Pasky" Baudis Stuff: http://pasky.or.cz/ C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor ``` ## Matt Mackall, 2005-05-15 17:39 Subject: Re: Mercurial 0.4e vs git network pull Message-ID: <20050515173923.GK5914@waste.org> URL: https://gitlist.dev/e/20050515173923.GK5914%40waste.org In-Reply-To: <200505151122.j4FBMJa01073@adam.yggdrasil.com> ``` On Sun, May 15, 2005 at 04:22:19AM -0700, Adam J. Richter wrote: > On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote: > >Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter > >where Matt Mackall told me that... > >> Does this need an HTTP request (and round trip) per object? It appears > >> to. That's 2200 requests/round trips for my 800 patch benchmark. > > >Yes it does. On the other side, it needs no server-side CGI. But I guess > >it should be pretty easy to write some kind of server-side CGI streamer, > >and it would then easily take just a single HTTP request (telling the > >server the commit ID and receiving back all the objects). > > I don't understand what was wrong with Jeff Garzik's previous > suggestion of using http/1.1 pipelining to coalesce the round trips. You can't do pipelining if you can't look ahead far enough to fill the pipe. -- Mathematics is the supreme nostalgia of our time. ``` ## Jeff Garzik, 2005-05-15 18:23 Subject: Re: Mercurial 0.4e vs git network pull Message-ID: <428793A1.5070004@pobox.com> URL: https://gitlist.dev/e/428793A1.5070004%40pobox.com In-Reply-To: <20050515173923.GK5914@waste.org> ``` Matt Mackall wrote: > On Sun, May 15, 2005 at 04:22:19AM -0700, Adam J. Richter wrote: > >>On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote: >> >>>Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter >>>where Matt Mackall told me that... >>> >>>>Does this need an HTTP request (and round trip) per object? It appears >>>>to. That's 2200 requests/round trips for my 800 patch benchmark. >> >>>Yes it does. On the other side, it needs no server-side CGI. But I guess >>>it should be pretty easy to write some kind of server-side CGI streamer, >>>and it would then easily take just a single HTTP request (telling the >>>server the commit ID and receiving back all the objects). >> >> I don't understand what was wrong with Jeff Garzik's previous >>suggestion of using http/1.1 pipelining to coalesce the round trips. > > > You can't do pipelining if you can't look ahead far enough to fill the pipe. Even if you cannot fill a pipeline, HTTP/1.1 is sufficiently useful simply by removing the per-request connection overhead. Jeff ``` ## Matt Mackall, 2005-05-16 01:12 Subject: Re: Mercurial 0.4e vs git network pull Message-ID: <20050516011209.GM5914@waste.org> URL: https://gitlist.dev/e/20050516011209.GM5914%40waste.org In-Reply-To: <428793A1.5070004@pobox.com> ``` On Sun, May 15, 2005 at 02:23:29PM -0400, Jeff Garzik wrote: > Matt Mackall wrote: > >On Sun, May 15, 2005 at 04:22:19AM -0700, Adam J. Richter wrote: > > > >>On Sun, 15 May 2005 10:54:05 +0200, Petr Baudis wrote: > >> > >>>Dear diary, on Thu, May 12, 2005 at 10:57:35PM CEST, I got a letter > >>>where Matt Mackall told me that... > >>> > >>>>Does this need an HTTP request (and round trip) per object? It appears > >>>>to. That's 2200 requests/round trips for my 800 patch benchmark. > >> > >>>Yes it does. On the other side, it needs no server-side CGI. But I guess > >>>it should be pretty easy to write some kind of server-side CGI streamer, > >>>and it would then easily take just a single HTTP request (telling the > >>>server the commit ID and receiving back all the objects). > >> > >> I don't understand what was wrong with Jeff Garzik's previous > >>suggestion of using http/1.1 pipelining to coalesce the round trips. > > > > > >You can't do pipelining if you can't look ahead far enough to fill the > >pipe. > > Even if you cannot fill a pipeline, HTTP/1.1 is sufficiently useful > simply by removing the per-request connection overhead. Sure. It cuts round trips by a factor of 2. But that's just about all it does. Mercurial already does: - approximately O(log(new changesets)) requests/data to find new changesets - one request to get an entire changegroup (set of all new changesets), which comes back all nicely pipelined and sorted by file - delta transfer In "dumb http" mode, ie what's been there since about day three, it can do: - one request (size proportional to total number of changesets) to find new changesets - approximately two requests per changed file to pull all deltas (vs request per file revision) - delta transfer -- Mathematics is the supreme nostalgia of our time. ``` ## Matthias Urlichs, 2005-05-16 09:29 Subject: Re: Mercurial 0.4e vs git network pull Message-ID: URL: https://gitlist.dev/e/pan.2005.05.16.09.29.09.407174%40smurf.noris.de In-Reply-To: <200505151122.j4FBMJa01073@adam.yggdrasil.com> ``` Hi, Adam J. Richter wrote: > Being able to do without a server side CGI script might > encourage deployment a bit more, both for security reasons and > effort of deployment. A simple server-side CGI would be a "send me all changeset SHA-1s, starting at HEAD until you reach FOO" operation (FOO being the SHA1 of the previous head you've pulled before). This operation is simple enough that it people should have no problem installing such a CGI. You could then stream-pull the actual contents over HTTP/1.1 without further CGI interaction. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de ``` ## Tristan Wibberley, 2005-05-16 22:22 Subject: Re: Mercurial 0.4e vs git network pull Message-ID: <1116282131.6141.8.camel__31554.9111379825$1116283578$gmane$org@localhost.localdomain> URL: https://gitlist.dev/e/1116282131.6141.8.camel__31554.9111379825%241116283578%24gmane%24org%40localhost.localdomain In-Reply-To: <20050515124042.GE13024@pasky.ji.cz> ``` On Sun, 2005-05-15 at 14:40 +0200, Petr Baudis wrote: > Dear diary, on Sun, May 15, 2005 at 01:22:19PM CEST, I got a letter > where "Adam J. Richter" told me that... > > > > I don't understand what was wrong with Jeff Garzik's previous > > suggestion of using http/1.1 pipelining to coalesce the round trips. > > If you're worried about queuing too many http/1.1 requests, the client > > could adopt a policy of not having more than a certain number of > > requests outstanding or perhaps even making a new http connection > > after a certain number of requests to avoid starving other clients > > when the number of clients doing one of these transfers exceeds the > > number of threads that the http server uses. > > The problem is that to fetch a revision tree, you have to > > send request for commit A > receive commit A > look at commit A for list of its parents > send request for the parents > receive the parents > look inside for list of its parents > ... What about IMAP? You could ask for just the parents for several messages (via a message header), then start asking for message bodies (with the juicy stuff in). You could also ask for a list of the new commits then ask for each of the bodies (several at a time). Not as good as a "Just give me all new data", but an *awful* lot more efficient than HTTP. And very flexible. You just need to map changesets to IMAP messages (if such a mapping can actually make sense :) Prolly a bit more work though. -- Tristan Wibberley The opinions expressed in this message are my own opinions and not those of my employer. ```