From: Daniel Barkalow Date: Fri, 07 Oct 2005 16:23:01 GMT Subject: Re: [PATCH] Add support for parallel HTTP transfers Message-ID: In-Reply-To: <20051007000041.GH15593@reactrix.com> On Thu, 6 Oct 2005, Nick Hengeveld wrote: > On Thu, Oct 06, 2005 at 04:07:07PM -0400, Daniel Barkalow wrote: > > > Somewhat weirdly, the version of curl on my desktop doesn't actually have > > an implementation of curl_multi_info_read, although it's in the header > > file and documentation. So you'll want a version check somewhere, I think, > > which should probably just disable parallel transfers. > > I was afraid that was going to happen... From the archived versions on the > CURL download site, it looks as though multi support was added in 7.9.8 - > which version do you have installed on your desktop? I'll have to check, but I think it's 7.9.8 or close to that; it seems like they added multi support without a critical function, so you might need to bump the check from what the history would suggest. > > It should be fine to download objects and a pack that contains them at the > > same time, although there's currently a check in fetch.c which should be > > removed, so that it will call fetch() for an object if the object appears > > between the prefetch() and the fetch(). > > Can you provide a patch, or point me toward the right place to make that > change? It's line 168 of fetch.c; the "!has_sha1_file(obj->sha1)" part should go away. > > I should be able to review this over the weekend. What sort of performance > > are you getting at this point (in terms of bandwidth utilization)? > > I've done limited testing by using the time command to track real/user/sys > taken to run 'git fetch http://kernel.org/pub/scm/git/git.git master', > and have seen performance improve by a factor of ~2-10: That looks good. I think it might be good to set the default connection limit higher; I don't think we can generate enough parallelism that we'd cause problems for a server with a single client, and, with a constant stream of clients, this will just shuffle around when the connections happen; to the extent that a single client does more simultaneous connections, it'll overlap less with other clients. > About that "-r" arg - seems like it should be something else as -r is > used elsewhere in git to enable recursion. "-c" was my first thought, > but that's used to fetch commit objects. The other things that affect the behaviour of the HTTP fetch in particular are done as environment variables, which seems like a good idea to me. Alternatively, you could use a long option. I don't expect there will be much variation in what someone uses. -Daniel *This .sig left intentionally blank*