Re: Errors from http-fetch

From: Daniel Barkalow <barkalow@iabervon.org>
Date: Oct 19, 2005, 22:10 UTC
Message-ID: <Pine.LNX.4.64.0510191755280.25300@iabervon.org>
In-Reply-To: <20051019212644.GB6160@reactrix.com>

On Wed, 19 Oct 2005, Nick Hengeveld wrote:

Show 10 quoted lines

> On Wed, Oct 19, 2005 at 03:46:10PM -0400, Daniel Barkalow wrote:
> 
> > I've now started getting errors from http-fetch; my guess for the output 
> > is that it has problems if an object is not available loose, and is only 
> > available in a pack file which is already being downloaded for a different 
> > object, possibly only if the first request to be made for the loose 
> > object is not the first one to fail.
> 
> Are you getting a curl return code of 52/CURLE_GOT_NOTHING?  Any chance
> you have access to the associated server logs?

I'm getting 404 for the object, which it reports. This was pulling from kernel.org.

> In the situation you describe, fetch() won't be called for the second
> object until fetch() for the first object is finished - at which point
> the pack will have been fully downloaded for the first object (or
> failed, causing fetch() for the first object to fail as well.)

My theory is that you have: prefetch(#2), prefetch(#1), (other stuff), #1 gives an error, start downloading the pack, fetch(#2), #2 gives an error, and there are no more packs to try.

That is, it was a later object that prompted getting the pack, because either the server decided to send that error message sooner, some packet got dropped and retransmitted, the connection got a lower-numbered file descriptor and they were ready at the same time, or something like that.

Another thing I noticed was that it was in the middle of downloading the pack when it suddenly exitted due to not being able to find the object; it would probably be worth having a call to finish up the active transfers after giving up on completing the whole thing, since we probably do actually want to finish downloading a big pack if we've started.

Consider the situation where you happen to catch the server in the middle of mirroring, and you're halfway through a big pack; restarting is nice, but it might as well let you finish downloading the pack, so you have less to do when you try again later.

	-Daniel
*This .sig left intentionally blank*