Re: [PATCH] Add support for parallel HTTP transfers

From: Daniel Barkalow <barkalow@iabervon.org>
Date: Oct 6, 2005, 20:07 UTC
Message-ID: <Pine.LNX.4.63.0510061550510.23242@iabervon.org>
In-Reply-To: <20051005214447.GF15593@reactrix.com>

On Wed, 5 Oct 2005, Nick Hengeveld wrote:

Show 5 quoted lines

> Add support for parallel HTTP transfers.  Prefetch populates a queue of
> objects to transfer and starts feeding requests to an active request
> queue for processing; fetch_object keeps the active queue moving
> while the specified object is being transferred.  The size of the active
> queue can be restricted using -r and defaults to 5 concurrent transfers.

Somewhat weirdly, the version of curl on my desktop doesn't actually have an implementation of curl_multi_info_read, although it's in the header file and documentation. So you'll want a version check somewhere, I think, which should probably just disable parallel transfers.

Show 7 quoted lines

> ---
> 
> I could use extra eyes on this patch - it seems to be stable although I have
> seen periodic cases during testing where it detects an empty server response
> (from kernel.org?)  Are there implications to downloading a (potentially large)
> pack while objects contained in that pack have been prefetched and are in the
> transfer and/or active queue?

It should be fine to download objects and a pack that contains them at the same time, although there's currently a check in fetch.c which should be removed, so that it will call fetch() for an object if the object appears between the prefetch() and the fetch().

I should be able to review this over the weekend. What sort of performance are you getting at this point (in terms of bandwidth utilization)?

	-Daniel
*This .sig left intentionally blank*