Errors from http-fetch

I've now started getting errors from http-fetch; my guess for the output is that it has problems if an object is not available loose, and is only available in a pack file which is already being downloaded for a different object, possibly only if the first request to be made for the loose object is not the first one to fail.

On the other hand, it looks like just trying again works fine (or, at least, makes progress), so there's no need to fall back to rsync or such.

But that also means that, in order to try to make a repeatable test, you need to copy your state before you run it and get the error; I'll probably try this evening to work it out.

	-Daniel
*This .sig left intentionally blank*

On Wed, Oct 19, 2005 at 03:46:10PM -0400, Daniel Barkalow wrote:

Are you getting a curl return code of 52/CURLE_GOT_NOTHING? Any chance you have access to the associated server logs?

In the situation you describe, fetch() won't be called for the second object until fetch() for the first object is finished - at which point the pack will have been fully downloaded for the first object (or failed, causing fetch() for the first object to fail as well.)

On Wed, 19 Oct 2005, Nick Hengeveld wrote:

I'm getting 404 for the object, which it reports. This was pulling from kernel.org.

My theory is that you have: prefetch(#2), prefetch(#1), (other stuff), #1 gives an error, start downloading the pack, fetch(#2), #2 gives an error, and there are no more packs to try.

That is, it was a later object that prompted getting the pack, because either the server decided to send that error message sooner, some packet got dropped and retransmitted, the connection got a lower-numbered file descriptor and they were ready at the same time, or something like that.

Another thing I noticed was that it was in the middle of downloading the pack when it suddenly exitted due to not being able to find the object; it would probably be worth having a call to finish up the active transfers after giving up on completing the whole thing, since we probably do actually want to finish downloading a big pack if we've started.

Consider the situation where you happen to catch the server in the middle of mirroring, and you're halfway through a big pack; restarting is nice, but it might as well let you finish downloading the pack, so you have less to do when you try again later.

	-Daniel
*This .sig left intentionally blank*

On Wed, Oct 19, 2005 at 06:10:52PM -0400, Daniel Barkalow wrote:

Something similar was reported yesterday, and when I investigated I found that one of the kernel.org servers was returning a 404 for the object in question and the other wasn't. Vger bounced my reply because it contained Content-type: headers...

I could see that happen if the pack appeared on the repo after #1 404ed and subsequently decided there were no packs, and before #2 404ed. Or, as was the case I saw yesterday, servers with the same DNS name aren't in sync. However, the pack download for #2 should never start in that case because fetch() for #1 would fail and exit.

I'm not sure how that could happen - once a pack download request starts, it has to finish before any fetch() calls can fail. However, there could certainly be other object requests in process when one fails, and it would be polite to let them finish.

On Wed, 19 Oct 2005, Nick Hengeveld wrote:

Hmm; it definitely cut off a pack download in progress. When I ran it again, it resumed that download, which contained the missing object. Last night, I didn't see any problems, though. I'd certainly believe that server disagreement could be a factor, though.

	-Daniel
*This .sig left intentionally blank*