{"thread":{"id":"32227","subject":"git fetch pack freezes","startedAt":"2012-11-28T14:12:49Z","lastAt":"2012-11-28T19:22:53Z","messageCount":4,"participants":["Ivan Kanis","Shawn Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"204158","messageId":"87624pizoe.fsf@googlemail.com","threadId":"32227","inReplyTo":null,"subject":"git fetch pack freezes","fromName":"Ivan Kanis","fromEmail":"ivan.kanis@googlemail.com","sentAt":"2012-11-28T14:12:49Z","receivedAt":"2012-11-28T14:12:49Z","isPatch":false,"sender":{"key":"ivan.kanis@googlemail.com","avatar":null},"body":"Hello,\n\nWe are using git with the smart HTTP protocol. We are seeing the client\nfreeze with Jenkins poll. It stops the continuous build which is a bit\nof a pain. It happens with the latest release 1.8.0 on the client side.\nThe server is using 1.7.0.4.\n\nOn the server we are seeing the following error message:\n\nvar/log/apache2/error.log.1:[Mon Nov 19 15:17:12 2012] [error] [client 192.168.50.15] fatal: git upload-pack: not our ref aa92ca2076c2421433f89ee0b0ae89cd36d27481 multi_ack_detailed side-band-64k thin-pack no-progress ofs-delta\n\nThe error is located in upload-pack.c:\n\n * Hmph. We may later want to allow \"want\" line that\n * asks for something like \"master~10\" (symbolic)...\n * would it make sense? I don't know.\n */\n o = lookup_object(sha1_buf);\n if (!o || !(o->flags & OUR_REF))\n die(\"git upload-pack: not our ref %s\", line+5);\n\nIt calls die with I believe is doing the right thing(TM).\n\nOn the client side I compiled git with debugging symbols. I then\nattached the debugger to the process that gets hosed. I get the\nfollowing backtrace:\n\n#0 0x00007fff85389af0 in read ()\n#1 0x000000010f900913 in xread (fd=0, buf=0x7fff6f376e2c, len=4) at wrapper.c:142\n#2 0x000000010f900a11 in read_in_full (fd=0, buf=0x7fff6f376e2c, count=4) at wrapper.c:171\n#3 0x000000010f8a14e4 in safe_read (fd=0, buffer=0x7fff6f376e2c, size=4, return_line_fail=0) at pkt-line.c:140\n#4 0x000000010f8a110e in packet_read_internal (fd=0, buffer=0x10f981eb0 \"\", size=1000, return_line_fail=0) at pkt-line.c:183\n#5 0x000000010f8a1246 in packet_read_line (fd=0, buffer=0x10f981eb0 \"\", size=1000) at pkt-line.c:211\n#6 0x000000010f7b5196 in get_ack (fd=0, result_sha1=0x7fff6f377450 \"\") at builtin/fetch-pack.c:227\n#7 0x000000010f7b4074 in find_common (fd=0x7fff6f377668, result_sha1=0x7fff6f377450 \"\", refs=0x0) at builtin/fetch-pack.c:416\n#8 0x000000010f7b32a5 in do_fetch_pack (fd=0x7fff6f377668, orig_ref=0x10fb007a0, sought=0x7fff6f377670, pack_lockfile=0x7fff6f377660)\n at builtin/fetch-pack.c:832\n#9 0x000000010f7b2cbd in fetch_pack (my_args=0x10f973fe8, fd=0x7fff6f377668, conn=0x0, ref=0x10fb007a0, \n dest=0x7fff6f377a78 \"https://foo.visionobjects.com/git/a-project-name/\", sought=0x7fff6f377670, pack_lockfile=0x7fff6f377660)\n at builtin/fetch-pack.c:1069\n#10 0x000000010f7b2a25 in cmd_fetch_pack (argc=7, argv=0x7fff6f377890, prefix=0x0) at builtin/fetch-pack.c:1012\n#11 0x000000010f77aa5b in run_builtin (p=0x10f971990, argc=7, argv=0x7fff6f377890) at git.c:306\n#12 0x000000010f779ac7 in handle_internal_command (argc=7, argv=0x7fff6f377890) at git.c:467\n#13 0x000000010f77a299 in run_argv (argcp=0x7fff6f377858, argv=0x7fff6f377850) at git.c:513\n#14 0x000000010f7798fd in main (argc=7, argv=0x7fff6f377890) at git.c:588\n\nThe file descriptor being 0 in #6 seems a bit suspicious. I am not\nfamiliar with the git protocol. Could I get some guidance on how to\ntroubleshoot some more so that I come up with a patch?\n-- \nIvan Kanis, Release Manager, Vision Objects,\nTel +33 2 28 01 49 50, Fax +33 2 40 25 89 20\nhttp://www.visionobjects.com\n\nThe essence of science: ask an impertinent question, and you are on\nthe way to a pertinent answer.\n -- Jacob Bronowski\n"},{"id":"204164","messageId":"CAJo=hJvGNUtT+z_E7rSaOr77=7dSo1N+7QiEoHNdEQDV3n7JWw@mail.gmail.com","threadId":"32227","inReplyTo":"87624pizoe.fsf@googlemail.com","subject":"Re: git fetch pack freezes","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2012-11-28T16:41:30Z","receivedAt":"2012-11-28T16:41:30Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis wrote:\n> We are using git with the smart HTTP protocol. We are seeing the client\n> freeze with Jenkins poll. It stops the continuous build which is a bit\n> of a pain. It happens with the latest release 1.8.0 on the client side.\n> The server is using 1.7.0.4.\n>\n> On the server we are seeing the following error message:\n>\n> var/log/apache2/error.log.1:[Mon Nov 19 15:17:12 2012] [error] [client 192.168.50.15] fatal: git upload-pack: not our ref aa92ca2076c2421433f89ee0b0ae89cd36d27481 multi_ack_detailed side-band-64k thin-pack no-progress ofs-delta\n\nUpgrade your server. What happened here is a race condition that is\nonly possible in the smart HTTP protocol. The client makes an initial\nrequest to ask where the branches are. Then it makes a second (and\npossibly more) subsequent request to negotiate a common ancestor and\nactually obtain the necessary objects. If a branch is updated between\nthat initial request and any subsequent request, the server fails with\nthis error message.\n\nMore recent servers handle this more gracefully by looking to see if\nthe object the client wants is reachable within some reasonable\ndistance from a current branch. If so, the request is still allowed to\ncontinue, rather than die() with a failure.\n\n> On the client side I compiled git with debugging symbols. I then\n> attached the debugger to the process that gets hosed. I get the\n> following backtrace:\n>\n> #5 0x000000010f8a1246 in packet_read_line (fd=0, buffer=0x10f981eb0 \"\", size=1000) at pkt-line.c:211\n> #6 0x000000010f7b5196 in get_ack (fd=0, result_sha1=0x7fff6f377450 \"\") at builtin/fetch-pack.c:227\n> #7 0x000000010f7b4074 in find_common (fd=0x7fff6f377668, result_sha1=0x7fff6f377450 \"\", refs=0x0) at builtin/fetch-pack.c:416\n>\n> The file descriptor being 0 in #6 seems a bit suspicious. I am not\n> familiar with the git protocol. Could I get some guidance on how to\n> troubleshoot some more so that I come up with a patch?\n\nThere is actually a parent process in front of this one called\ngit-remote-http (or git-remote-https depending on protocol). That\nparent is doing the HTTP transport and is sending whatever the server\nprovides into this child's stdin. I'm not surprised there is yet\nanother deadlock condition in here, its a rather heinous process setup\nand way to handle the protocol.\n\nfd in frame #7 is an address of an int array. fd in frame #6 is an\nactual fd, the one to read from, which in this case should be 0 to\nread from stdin, because its trying to get data from the HTTP server\nby way of reading it from the pipe the parent supplied to it on stdin.\n\nSo the stack frames are correct. Its just a problem that the parent\ndidn't identify the server crashing and closing its side of the pipe\non stdin to force it to EOF to prevent the child from getting hung\nhere in a read.\n"},{"id":"204169","messageId":"87obihhc6q.fsf@visionobjects.com","threadId":"32227","inReplyTo":"CAJo=hJvGNUtT+z_E7rSaOr77=7dSo1N+7QiEoHNdEQDV3n7JWw@mail.gmail.com","subject":"Re: git fetch pack freezes","fromName":"Ivan Kanis","fromEmail":"ivan.kanis@googlemail.com","sentAt":"2012-11-28T17:25:33Z","receivedAt":"2012-11-28T17:25:33Z","isPatch":false,"sender":{"key":"ivan.kanis@googlemail.com","avatar":null},"body":"Hi Shawn,\n\nThanks for the quick reply.\n\nShawn Pearce wrote:\n\n> On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis wrote:\n>>\n>> On the server we are seeing the following error message:\n>\n> Upgrade your server.\n\nOK we'll look into it. I have a question: will a 1.8 server still serve\n1.7 git client?\n\n> So the stack frames are correct. Its just a problem that the parent\n> didn't identify the server crashing and closing its side of the pipe\n> on stdin to force it to EOF to prevent the child from getting hung\n> here in a read.\n\nIt sounds like a bug on the client, doesn't it?\n-- \nIvan Kanis, Release Manager, Vision Objects,\nTel +33 2 28 01 49 50, Fax +33 2 40 25 89 20\nhttp://www.visionobjects.com\n\nLet a fool hold his tongue and he will pass for a sage.\n -- Publilius Syrus\n"},{"id":"204188","messageId":"CAJo=hJs0dEY1WX4t+R1cz__5TK3Od+wtiRjnHJQxs-m1SU69Lw@mail.gmail.com","threadId":"32227","inReplyTo":"87obihhc6q.fsf@visionobjects.com","subject":"Re: git fetch pack freezes","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2012-11-28T19:22:53Z","receivedAt":"2012-11-28T19:22:53Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Nov 28, 2012 at 9:25 AM, Ivan Kanis wrote:\n> Shawn Pearce wrote:\n>\n>> On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis wrote:\n>>>\n>>> On the server we are seeing the following error message:\n>>\n>> Upgrade your server.\n>\n> OK we'll look into it. I have a question: will a 1.8 server still serve\n> 1.7 git client?\n\nYes.\n\n>> So the stack frames are correct. Its just a problem that the parent\n>> didn't identify the server crashing and closing its side of the pipe\n>> on stdin to force it to EOF to prevent the child from getting hung\n>> here in a read.\n>\n> It sounds like a bug on the client, doesn't it?\n\nYes.\n"}]}