Re: [PATCH] Add support for parallel HTTP transfers

From: Daniel Barkalow <barkalow@iabervon.org>
Date: Oct 7, 2005, 16:23 UTC
Message-ID: <Pine.LNX.4.63.0510071149550.23242@iabervon.org>
In-Reply-To: <20051007000041.GH15593@reactrix.com>

On Thu, 6 Oct 2005, Nick Hengeveld wrote:

Show 10 quoted lines

> On Thu, Oct 06, 2005 at 04:07:07PM -0400, Daniel Barkalow wrote:
> 
> > Somewhat weirdly, the version of curl on my desktop doesn't actually have 
> > an implementation of curl_multi_info_read, although it's in the header 
> > file and documentation. So you'll want a version check somewhere, I think, 
> > which should probably just disable parallel transfers.
> 
> I was afraid that was going to happen...  From the archived versions on the
> CURL download site, it looks as though multi support was added in 7.9.8 -
> which version do you have installed on your desktop?

I'll have to check, but I think it's 7.9.8 or close to that; it seems like they added multi support without a critical function, so you might need to bump the check from what the history would suggest.

Show 7 quoted lines

> > It should be fine to download objects and a pack that contains them at the 
> > same time, although there's currently a check in fetch.c which should be 
> > removed, so that it will call fetch() for an object if the object appears 
> > between the prefetch() and the fetch().
> 
> Can you provide a patch, or point me toward the right place to make that
> change?

It's line 168 of fetch.c; the "!has_sha1_file(obj->sha1)" part should go away.

Show 6 quoted lines

> > I should be able to review this over the weekend. What sort of performance 
> > are you getting at this point (in terms of bandwidth utilization)?
> 
> I've done limited testing by using the time command to track real/user/sys
> taken to run 'git fetch http://kernel.org/pub/scm/git/git.git master',
> and have seen performance improve by a factor of ~2-10:

That looks good. I think it might be good to set the default connection limit higher; I don't think we can generate enough parallelism that we'd cause problems for a server with a single client, and, with a constant stream of clients, this will just shuffle around when the connections happen; to the extent that a single client does more simultaneous connections, it'll overlap less with other clients.

> About that "-r" arg - seems like it should be something else as -r is
> used elsewhere in git to enable recursion.  "-c" was my first thought,
> but that's used to fetch commit objects.

The other things that affect the behaviour of the HTTP fetch in particular are done as environment variables, which seems like a good idea to me. Alternatively, you could use a long option. I don't expect there will be much variation in what someone uses.

	-Daniel
*This .sig left intentionally blank*