{"thread":{"id":"20899","subject":"Cannot clone redirecting stdout","startedAt":"2009-09-10T22:33:21Z","lastAt":"2009-09-11T22:47:32Z","messageCount":10,"participants":["Aloisio","Stefan Naewe","Jean-Luc Herren","Jeff King","Johan Sørensen","Daniel Barkalow"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"122851","messageId":"fc8ab2ad0909101533l135c8003m80091cb40ec93f16@mail.gmail.com","threadId":"20899","inReplyTo":null,"subject":"Cannot clone redirecting stdout","fromName":"Aloisio","fromEmail":"aloisiojr@gmail.com","sentAt":"2009-09-10T22:33:21Z","receivedAt":"2009-09-10T22:33:21Z","isPatch":false,"sender":{"key":"aloisiojr@gmail.com","avatar":null},"body":"Hi all,\n\nI faced a problem when trying to clone git://gitorious.org/qt/qt.git\n\nthis works:\ngit clone -n git://gitorious.org/qt/qt.git repo\n\nthis doesn't:\ngit clone -n git://gitorious.org/qt/qt.git repo >log\nfatal: The remote end hung up unexpectedly\nfatal: early EOF\nfatal: index-pack failed\n\nI reproduced the error in the following versions:\ngit version 1.6.0.4\ngit version 1.6.4.2\ngit version 1.5.4.7\n\nAny clues?\nThanks\n\nAloisio Almeida\n"},{"id":"122874","messageId":"4AA9F0A2.6050105@atlas-elektronik.com","threadId":"20899","inReplyTo":"fc8ab2ad0909101533l135c8003m80091cb40ec93f16@mail.gmail.com","subject":"Re: Cannot clone redirecting stdout","fromName":"Stefan Naewe","fromEmail":"stefan.naewe@atlas-elektronik.com","sentAt":"2009-09-11T06:39:30Z","receivedAt":"2009-09-11T06:39:30Z","isPatch":false,"sender":{"key":"stefan.naewe@gmail.com","avatar":"https://avatars.githubusercontent.com/u/4468?v=4"},"body":"On 9/11/2009 12:33 AM, Aloisio wrote:\n> Hi all,\n> \n> I faced a problem when trying to clone git://gitorious.org/qt/qt.git\n> \n> this works:\n> git clone -n git://gitorious.org/qt/qt.git repo\n> \n> this doesn't:\n> git clone -n git://gitorious.org/qt/qt.git repo >log\n> fatal: The remote end hung up unexpectedly\n> fatal: early EOF\n> fatal: index-pack failed\n> \n> I reproduced the error in the following versions:\n> git version 1.6.0.4\n> git version 1.6.4.2\n> git version 1.5.4.7\n> \n> Any clues?\n> Thanks\n\nI'd say:\n\nTake a look at /proc//fd\n\nfd 0 (stdout) is linked through a pipe to 'git index-pack'.\nRedirecting this link breaks everything.\n\nRegards,\nStefan\n"},{"id":"122876","messageId":"4AA9FBD3.5040404@atlas-elektronik.com","threadId":"20899","inReplyTo":"4AA9F0A2.6050105@atlas-elektronik.com","subject":"Re: Cannot clone redirecting stdout","fromName":"Stefan Naewe","fromEmail":"stefan.naewe@atlas-elektronik.com","sentAt":"2009-09-11T07:27:15Z","receivedAt":"2009-09-11T07:27:15Z","isPatch":false,"sender":{"key":"stefan.naewe@gmail.com","avatar":"https://avatars.githubusercontent.com/u/4468?v=4"},"body":"On 9/11/2009 8:39 AM, Stefan Naewe wrote:\n> On 9/11/2009 12:33 AM, Aloisio wrote:\n>> Hi all,\n>>\n>> I faced a problem when trying to clone git://gitorious.org/qt/qt.git\n>>\n>> this works:\n>> git clone -n git://gitorious.org/qt/qt.git repo\n>>\n>> this doesn't:\n>> git clone -n git://gitorious.org/qt/qt.git repo >log\n>> fatal: The remote end hung up unexpectedly\n>> fatal: early EOF\n>> fatal: index-pack failed\n>>\n>> I reproduced the error in the following versions:\n>> git version 1.6.0.4\n>> git version 1.6.4.2\n>> git version 1.5.4.7\n>>\n>> Any clues?\n>> Thanks\n> \n> I'd say:\n> \n> Take a look at /proc//fd\n> \n> fd 0 (stdout) is linked through a pipe to 'git index-pack'.\n> Redirecting this link breaks everything.\n\nstdout is of course fd 1 ;-)\n\nRegards,\nStefan\n-- \n/dev/random says: Never trust a computer you can't lift. - Stan Masor\n"},{"id":"122878","messageId":"4AAA0737.1030005@gmx.ch","threadId":"20899","inReplyTo":"fc8ab2ad0909101533l135c8003m80091cb40ec93f16@mail.gmail.com","subject":"Re: Cannot clone redirecting stdout","fromName":"Jean-Luc Herren","fromEmail":"jlh@gmx.ch","sentAt":"2009-09-11T08:15:51Z","receivedAt":"2009-09-11T08:15:51Z","isPatch":false,"sender":{"key":"jlh@gmx.ch","avatar":null},"body":"Hi list!\n\nAloisio wrote:\n> this doesn't:\n> git clone -n git://gitorious.org/qt/qt.git repo >log\n> fatal: The remote end hung up unexpectedly\n> fatal: early EOF\n> fatal: index-pack failed\n\nWhat are you trying to store in that file? The only line I\nobserve being printed to stdout while cloning is this one:\n\n Initialized empty Git repository in [...]/repo/.git/\n\nEverything else (progress status) is printed to stderr and I\npresume that any error messages would also go to stderr too, if\nany. So possibly you would be happy with this:\n\n git clone -n git://gitorious.org/qt/qt.git repo 2> log\n\nwhich works fine here with git-1.6.4.2. Though I agree that I\nwould expect redirecting stdout to work.\n\nCheers,\njlh\n"},{"id":"122880","messageId":"20090911102329.GA13044@sigill.intra.peff.net","threadId":"20899","inReplyTo":"fc8ab2ad0909101533l135c8003m80091cb40ec93f16@mail.gmail.com","subject":"Re: Cannot clone redirecting stdout","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-09-11T10:23:29Z","receivedAt":"2009-09-11T10:23:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Sep 10, 2009 at 07:33:21PM -0300, Aloisio wrote:\n\n> I faced a problem when trying to clone git://gitorious.org/qt/qt.git\n> \n> this works:\n> git clone -n git://gitorious.org/qt/qt.git repo\n> \n> this doesn't:\n> git clone -n git://gitorious.org/qt/qt.git repo >log\n> fatal: The remote end hung up unexpectedly\n> fatal: early EOF\n> fatal: index-pack failed\n\nI can reproduce the problem here. But after staring at the strace for a\nlong time, I don't think the problem is on the client side. The remote\nend _does_ hang up unexpectedly.\n\nLooking at what we send, the only difference between the redirected and\nunredirected case I could find is that we send the \"no-progress\" flag to\nthe server, which then hangs up on us instead of sending us the pack.\nWhich makes no sense.\n\nConfused,\n-Peff\n"},{"id":"122888","messageId":"20090911135110.GA30860@coredump.intra.peff.net","threadId":"20899","inReplyTo":"20090911102329.GA13044@sigill.intra.peff.net","subject":"Re: Cannot clone redirecting stdout","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-09-11T13:51:11Z","receivedAt":"2009-09-11T13:51:11Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 11, 2009 at 06:23:29AM -0400, Jeff King wrote:\n\n> > I faced a problem when trying to clone git://gitorious.org/qt/qt.git\n> > \n> > this works:\n> > git clone -n git://gitorious.org/qt/qt.git repo\n> > \n> > this doesn't:\n> > git clone -n git://gitorious.org/qt/qt.git repo >log\n> > fatal: The remote end hung up unexpectedly\n> > fatal: early EOF\n> > fatal: index-pack failed\n> \n> I can reproduce the problem here. But after staring at the strace for a\n> long time, I don't think the problem is on the client side. The remote\n> end _does_ hang up unexpectedly.\n> \n> Looking at what we send, the only difference between the redirected and\n> unredirected case I could find is that we send the \"no-progress\" flag to\n> the server, which then hangs up on us instead of sending us the pack.\n> Which makes no sense.\n\nI did a little more testing, and I can't reproduce the problem against a\nlocal git-daemon. I tried using several versions for the server, going\nall the way back to v1.5.0, which pre-dates no-progress, and all of them\nworked just fine.\n\nSo I am inclined to think there is something non-standard or broken at\ngitorious.org. I'm cc'ing support@gitorious to see if they have any\ncomment.\n\n-Peff\n"},{"id":"122896","messageId":"9e0f31700909110846h54959ae6u466ceda40799ba37@mail.gmail.com","threadId":"20899","inReplyTo":"20090911135110.GA30860@coredump.intra.peff.net","subject":"Re: Cannot clone redirecting stdout","fromName":"Johan Sørensen","fromEmail":"johan@johansorensen.com","sentAt":"2009-09-11T15:46:23Z","receivedAt":"2009-09-11T15:46:23Z","isPatch":false,"sender":{"key":"johan@johansorensen.com","avatar":"https://gravatar.com/avatar/dad7869db9d9711098ad213108d1b09967181bb51b667d5a5088c7634ce46616?d=mp&s=160"},"body":"On Fri, Sep 11, 2009 at 3:51 PM, Jeff King wrote:\n> On Fri, Sep 11, 2009 at 06:23:29AM -0400, Jeff King wrote:\n>\n>> > I faced a problem when trying to clone git://gitorious.org/qt/qt.git\n>> >\n>> > this works:\n>> > git clone -n git://gitorious.org/qt/qt.git repo\n>> >\n>> > this doesn't:\n>> > git clone -n git://gitorious.org/qt/qt.git repo >log\n>> > fatal: The remote end hung up unexpectedly\n>> > fatal: early EOF\n>> > fatal: index-pack failed\n>>\n>> I can reproduce the problem here. But after staring at the strace for a\n>> long time, I don't think the problem is on the client side. The remote\n>> end _does_ hang up unexpectedly.\n>>\n>> Looking at what we send, the only difference between the redirected and\n>> unredirected case I could find is that we send the \"no-progress\" flag to\n>> the server, which then hangs up on us instead of sending us the pack.\n>> Which makes no sense.\n>\n> I did a little more testing, and I can't reproduce the problem against a\n> local git-daemon. I tried using several versions for the server, going\n> all the way back to v1.5.0, which pre-dates no-progress, and all of them\n> worked just fine.\n>\n> So I am inclined to think there is something non-standard or broken at\n> gitorious.org. I'm cc'ing support@gitorious to see if they have any\n> comment.\n\nSome quick tests seem to indicate it's related to the fact that our\nwonderful little fork+exec git-daemon[1] (which is different from the\none distributed with git) exec's to \"git-upload-pack --strict\n--timeout=30 /path/to/repo\". Now, why exactly that'll trigger when the\nno-progress flag is given I'm not sure of. The daemon itself execs as\nsoon as it figures out what repo the client requested, so apart from\nthe timeout the only thing it reacts to is the header (the\n\"${headersize}git-upload-pack project/repo.git\\0host=gitorious.org\\0\"\npart).\n\nWe also do redirect stderr to /dev/null for reasons I cannot remember\n(so probably not good ones), that may be related as well. Well run\nsome more tests...\n\n[1]: http://gitorious.org/gitorious/mainline/blobs/master/script/git-daemon\n\n>\n> -Peff\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"122898","messageId":"20090911160510.GA10848@coredump.intra.peff.net","threadId":"20899","inReplyTo":"9e0f31700909110846h54959ae6u466ceda40799ba37@mail.gmail.com","subject":"Re: Cannot clone redirecting stdout","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-09-11T16:05:10Z","receivedAt":"2009-09-11T16:05:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 11, 2009 at 05:46:23PM +0200, Johan Sørensen wrote:\n\n> Some quick tests seem to indicate it's related to the fact that our\n> wonderful little fork+exec git-daemon[1] (which is different from the\n> one distributed with git) exec's to \"git-upload-pack --strict\n> --timeout=30 /path/to/repo\". Now, why exactly that'll trigger when the\n> no-progress flag is given I'm not sure of. The daemon itself execs as\n> soon as it figures out what repo the client requested, so apart from\n> the timeout the only thing it reacts to is the header (the\n> \"${headersize}git-upload-pack project/repo.git\\0host=gitorious.org\\0\"\n> part).\n> \n> We also do redirect stderr to /dev/null for reasons I cannot remember\n> (so probably not good ones), that may be related as well. Well run\n> some more tests...\n\nAh. I have a theory. If I do a clone of git://gitorious.org/qt/qt.git,\nthe counting/compressing stages take a long time (I timed it at 1m40\nbefore it actually sends any data). And looking at upload-pack.c, we\nleave the 30-second alarm set while creating the pack. Meaning we die 30\nseconds into creating the pack.\n\nWhen progress is being displayed, however, the progress timer actually\nuses SIGALRM, as well. So we are constantly resetting the timer and it\nnever goes off.\n\nAnd we should be able to test this theory. How long does it take for the\nfailure case to fail:\n\n $ time git clone -n git://gitorious.org/qt/qt.git repo >log\n fatal: The remote end hung up unexpectedly\n fatal: early EOF\n fatal: index-pack failed\n\n real 0m31.106s\n user 0m0.000s\n sys 0m0.012s\n\nHmm. Suspicious. :)\n\nSo that implies to me a few things:\n\n 1. You guys should really pack your repos, as you are wasting over a\n minute of CPU time every time somebody clones this repo.\n\n 2. Upload-pack has what I consider a bug. The --timeout should be\n suspended while we are actually crunching numbers to create the\n pack. We probably don't want it when sending the pack, either, as\n people with slow connections (or big repos) will get timed out\n during the send. Probably we just want to apply it to times when we\n are waiting to get the list of refs from the client.\n\n 3. Upload-pack and the progress code are both using the global alarm\n timer and signal, and that is papering over the bug in (2) when\n progress is enabled. I'm not sure of the simplest way of having\n those interact. But maybe we can just ignore it, because we\n probably don't want to using the --timeout alarm during the packing\n phase, anyway.\n\n-Peff\n"},{"id":"122899","messageId":"20090911162013.GA10939@coredump.intra.peff.net","threadId":"20899","inReplyTo":"20090911160510.GA10848@coredump.intra.peff.net","subject":"Re: Cannot clone redirecting stdout","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-09-11T16:20:13Z","receivedAt":"2009-09-11T16:20:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 11, 2009 at 12:05:10PM -0400, Jeff King wrote:\n\n> Ah. I have a theory. If I do a clone of git://gitorious.org/qt/qt.git,\n> the counting/compressing stages take a long time (I timed it at 1m40\n> before it actually sends any data). And looking at upload-pack.c, we\n> leave the 30-second alarm set while creating the pack. Meaning we die 30\n> seconds into creating the pack.\n> \n> When progress is being displayed, however, the progress timer actually\n> uses SIGALRM, as well. So we are constantly resetting the timer and it\n> never goes off.\n\nHmm. Actually, this is not quite right. It looks like we call out to\npack-objects as an external program, so there is no conflict with the\nsignal. And we do proxy the output of pack-objects, which will keep our\ntimer resetting every time we see a chunk of output. But pack-objects\nproduces no output during the deltification phase, unless progress is\nturned on. So we still hit our timeout in upload-pack during that\nphase.\n\nSo our options are:\n\n 1. Turn off the timer during deltification, which could mean that it\n would potentially go forever. But it's not controlled by the user.\n I think the 'timeout' feature is really about the client just\n opening the connection and sitting.\n\n 2. Keep progress on during deltification, but just throw it away\n instead of relaying it if no-progress is in effect.\n\n 3. Accept that hitting the timeout during deltification _should_ cause\n it to die. In that case, then the case with progress is wrong, and\n we should stop resetting the timer just because we got some\n progress output from pack-objects. But this may be redefining the\n intent of --timeout. I don't know what the original intent was, or\n what users of the feature are expecting.\n\n-Peff\n"},{"id":"122923","messageId":"alpine.LNX.2.00.0909111820450.28290@iabervon.org","threadId":"20899","inReplyTo":"20090911162013.GA10939@coredump.intra.peff.net","subject":"Re: Cannot clone redirecting stdout","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-09-11T22:47:32Z","receivedAt":"2009-09-11T22:47:32Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Fri, 11 Sep 2009, Jeff King wrote:\n\n> On Fri, Sep 11, 2009 at 12:05:10PM -0400, Jeff King wrote:\n> \n> > Ah. I have a theory. If I do a clone of git://gitorious.org/qt/qt.git,\n> > the counting/compressing stages take a long time (I timed it at 1m40\n> > before it actually sends any data). And looking at upload-pack.c, we\n> > leave the 30-second alarm set while creating the pack. Meaning we die 30\n> > seconds into creating the pack.\n> > \n> > When progress is being displayed, however, the progress timer actually\n> > uses SIGALRM, as well. So we are constantly resetting the timer and it\n> > never goes off.\n> \n> Hmm. Actually, this is not quite right. It looks like we call out to\n> pack-objects as an external program, so there is no conflict with the\n> signal. And we do proxy the output of pack-objects, which will keep our\n> timer resetting every time we see a chunk of output. But pack-objects\n> produces no output during the deltification phase, unless progress is\n> turned on. So we still hit our timeout in upload-pack during that\n> phase.\n> \n> So our options are:\n> \n> 1. Turn off the timer during deltification, which could mean that it\n> would potentially go forever. But it's not controlled by the user.\n> I think the 'timeout' feature is really about the client just\n> opening the connection and sitting.\n> \n> 2. Keep progress on during deltification, but just throw it away\n> instead of relaying it if no-progress is in effect.\n> \n> 3. Accept that hitting the timeout during deltification _should_ cause\n> it to die. In that case, then the case with progress is wrong, and\n> we should stop resetting the timer just because we got some\n> progress output from pack-objects. But this may be redefining the\n> intent of --timeout. I don't know what the original intent was, or\n> what users of the feature are expecting.\n\nYou don't remember October 2005? HPA introduced it in 960decc, which has a \npretty good explanation: we doesn't want to get DoS'd if clients just send \nSYNs. So it's supposed to time out only if we spend that long waiting \nfor a protocol item from the client.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"}]}