{"thread":{"id":"2154","subject":"Errors from http-fetch","startedAt":"2005-10-19T19:46:10Z","lastAt":"2005-10-20T20:02:23Z","messageCount":5,"participants":["Daniel Barkalow","Nick Hengeveld"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"10291","messageId":"Pine.LNX.4.64.0510191536350.25300@iabervon.org","threadId":"2154","inReplyTo":null,"subject":"Errors from http-fetch","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-19T19:46:10Z","receivedAt":"2005-10-19T19:46:10Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"I've now started getting errors from http-fetch; my guess for the output \nis that it has problems if an object is not available loose, and is only \navailable in a pack file which is already being downloaded for a different \nobject, possibly only if the first request to be made for the loose \nobject is not the first one to fail.\n\nOn the other hand, it looks like just trying again works fine (or, at \nleast, makes progress), so there's no need to fall back to rsync or such.\n\nBut that also means that, in order to try to make a repeatable test, you \nneed to copy your state before you run it and get the error; I'll probably \ntry this evening to work it out.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"10304","messageId":"20051019212644.GB6160@reactrix.com","threadId":"2154","inReplyTo":"Pine.LNX.4.64.0510191536350.25300@iabervon.org","subject":"Re: Errors from http-fetch","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-10-19T21:26:44Z","receivedAt":"2005-10-19T21:26:44Z","isPatch":false,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"On Wed, Oct 19, 2005 at 03:46:10PM -0400, Daniel Barkalow wrote:\n\n> I've now started getting errors from http-fetch; my guess for the output \n> is that it has problems if an object is not available loose, and is only \n> available in a pack file which is already being downloaded for a different \n> object, possibly only if the first request to be made for the loose \n> object is not the first one to fail.\n\nAre you getting a curl return code of 52/CURLE_GOT_NOTHING?  Any chance\nyou have access to the associated server logs?\n\nIn the situation you describe, fetch() won't be called for the second\nobject until fetch() for the first object is finished - at which point\nthe pack will have been fully downloaded for the first object (or\nfailed, causing fetch() for the first object to fail as well.)\n\n-- \nFor a successful technology, reality must take precedence over public\nrelations, for nature cannot be fooled.\n"},{"id":"10310","messageId":"Pine.LNX.4.64.0510191755280.25300@iabervon.org","threadId":"2154","inReplyTo":"20051019212644.GB6160@reactrix.com","subject":"Re: Errors from http-fetch","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-19T22:10:52Z","receivedAt":"2005-10-19T22:10:52Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 19 Oct 2005, Nick Hengeveld wrote:\n\n> On Wed, Oct 19, 2005 at 03:46:10PM -0400, Daniel Barkalow wrote:\n> \n> > I've now started getting errors from http-fetch; my guess for the output \n> > is that it has problems if an object is not available loose, and is only \n> > available in a pack file which is already being downloaded for a different \n> > object, possibly only if the first request to be made for the loose \n> > object is not the first one to fail.\n> \n> Are you getting a curl return code of 52/CURLE_GOT_NOTHING?  Any chance\n> you have access to the associated server logs?\n\nI'm getting 404 for the object, which it reports. This was pulling from \nkernel.org.\n\n> In the situation you describe, fetch() won't be called for the second\n> object until fetch() for the first object is finished - at which point\n> the pack will have been fully downloaded for the first object (or\n> failed, causing fetch() for the first object to fail as well.)\n\nMy theory is that you have: prefetch(#2), prefetch(#1), (other stuff), #1 \ngives an error, start downloading the pack, fetch(#2), #2 gives an error, \nand there are no more packs to try.\n\nThat is, it was a later object that prompted getting the pack, because \neither the server decided to send that error message sooner, some packet \ngot dropped and retransmitted, the connection got a lower-numbered file \ndescriptor and they were ready at the same time, or something like that.\n\nAnother thing I noticed was that it was in the middle of downloading the \npack when it suddenly exitted due to not being able to find the object; it \nwould probably be worth having a call to finish up the active transfers \nafter giving up on completing the whole thing, since we probably do \nactually want to finish downloading a big pack if we've started.\n\nConsider the situation where you happen to catch the server in the middle \nof mirroring, and you're halfway through a big pack; restarting is nice, \nbut it might as well let you finish downloading the pack, so you have less \nto do when you try again later.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"10312","messageId":"20051019223708.GC6160@reactrix.com","threadId":"2154","inReplyTo":"Pine.LNX.4.64.0510191755280.25300@iabervon.org","subject":"Re: Errors from http-fetch","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-10-19T22:37:08Z","receivedAt":"2005-10-19T22:37:08Z","isPatch":false,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"On Wed, Oct 19, 2005 at 06:10:52PM -0400, Daniel Barkalow wrote:\n\n> I'm getting 404 for the object, which it reports. This was pulling from \n> kernel.org.\n\nSomething similar was reported yesterday, and when I investigated I\nfound that one of the kernel.org servers was returning a 404 for the\nobject in question and the other wasn't.  Vger bounced my reply because\nit contained Content-type: headers...\n\n> That is, it was a later object that prompted getting the pack, because \n> either the server decided to send that error message sooner, some packet \n> got dropped and retransmitted, the connection got a lower-numbered file \n> descriptor and they were ready at the same time, or something like that.\n\nI could see that happen if the pack appeared on the repo after #1 404ed\nand subsequently decided there were no packs, and before #2 404ed.  Or,\nas was the case I saw yesterday, servers with the same DNS name aren't\nin sync.  However, the pack download for #2 should never start in that\ncase because fetch() for #1 would fail and exit.\n\n> Another thing I noticed was that it was in the middle of downloading the \n> pack when it suddenly exitted due to not being able to find the object; it \n> would probably be worth having a call to finish up the active transfers \n> after giving up on completing the whole thing, since we probably do \n> actually want to finish downloading a big pack if we've started.\n\nI'm not sure how that could happen - once a pack download request\nstarts, it has to finish before any fetch() calls can fail.  However,\nthere could certainly be other object requests in process when one\nfails, and it would be polite to let them finish.\n\n-- \nFor a successful technology, reality must take precedence over public\nrelations, for nature cannot be fooled.\n"},{"id":"10368","messageId":"Pine.LNX.4.64.0510201558231.25300@iabervon.org","threadId":"2154","inReplyTo":"20051019223708.GC6160@reactrix.com","subject":"Re: Errors from http-fetch","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-20T20:02:23Z","receivedAt":"2005-10-20T20:02:23Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 19 Oct 2005, Nick Hengeveld wrote:\n\n> > Another thing I noticed was that it was in the middle of downloading the \n> > pack when it suddenly exitted due to not being able to find the object; it \n> > would probably be worth having a call to finish up the active transfers \n> > after giving up on completing the whole thing, since we probably do \n> > actually want to finish downloading a big pack if we've started.\n> \n> I'm not sure how that could happen - once a pack download request\n> starts, it has to finish before any fetch() calls can fail.  However,\n> there could certainly be other object requests in process when one\n> fails, and it would be polite to let them finish.\n\nHmm; it definitely cut off a pack download in progress. When I ran it \nagain, it resumed that download, which contained the missing object. Last \nnight, I didn't see any problems, though. I'd certainly believe that \nserver disagreement could be a factor, though.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"}]}