Re: [PATCH] Add support for parallel HTTP transfers

From: Jon Loeliger <jdl@freescale.com>
Date: Oct 10, 2005, 16:48 UTC
Message-ID: <1128962922.9069.11.camel@cashmere.sps.mot.com>
In-Reply-To: <Pine.LNX.4.63.0510071808071.23242@iabervon.org>

On Fri, 2005-10-07 at 17:39, Daniel Barkalow wrote:

Show 15 quoted lines

> 
> I believe that the situation is the one you describe in your previous 
> message: we determine we need to fetch A and B; we ask for A; we ask for 
> B; we find A isn't available alone, but is available in a pack; we get the 
> pack; we find we now have B (in the pack); the request for B (which would 
> probably fail) is left dangling.
> 
> The only actual problem I can see is if this happens with a whole bunch of 
> objects at the beginning of a big download, and all but one of your 
> connections are left in this state while you download all of the loose 
> objects over the one connection that got the pack.
> 
> I don't know if this is a problem for the new http code, but it could be 
> an issue in general if a transport method allocates resources in 
> prefetch().

So, this sounds like a classic resource scheduling problem with various solutions encoded in many compiler schedulers.

Doesn't this sort of scheduling problem get solved by a two-stage request pipeline model? In particular, you only "semi-request A" and "semi-request B" into a queue. When all of the sub-parts (that are needed) of some pack P have been requested you issue the one request for the common pack P holding all the sub-parts. Then all the sub-parts are can be retired from the queue.

Or am I just now finally catching up? :-)

Thanks, jdl