From: Daniel Barkalow Date: Thu, 06 Oct 2005 20:07:07 GMT Subject: Re: [PATCH] Add support for parallel HTTP transfers Message-ID: In-Reply-To: <20051005214447.GF15593@reactrix.com> On Wed, 5 Oct 2005, Nick Hengeveld wrote: > Add support for parallel HTTP transfers. Prefetch populates a queue of > objects to transfer and starts feeding requests to an active request > queue for processing; fetch_object keeps the active queue moving > while the specified object is being transferred. The size of the active > queue can be restricted using -r and defaults to 5 concurrent transfers. Somewhat weirdly, the version of curl on my desktop doesn't actually have an implementation of curl_multi_info_read, although it's in the header file and documentation. So you'll want a version check somewhere, I think, which should probably just disable parallel transfers. > --- > > I could use extra eyes on this patch - it seems to be stable although I have > seen periodic cases during testing where it detects an empty server response > (from kernel.org?) Are there implications to downloading a (potentially large) > pack while objects contained in that pack have been prefetched and are in the > transfer and/or active queue? It should be fine to download objects and a pack that contains them at the same time, although there's currently a check in fetch.c which should be removed, so that it will call fetch() for an object if the object appears between the prefetch() and the fetch(). I should be able to review this over the weekend. What sort of performance are you getting at this point (in terms of bandwidth utilization)? -Daniel *This .sig left intentionally blank*