{"thread":{"id":"44992","subject":"Deadlock between git-remote-http and git fetch-pack","startedAt":"2017-01-27T22:38:14Z","lastAt":"2017-01-28T00:12:02Z","messageCount":3,"participants":["tsuna","Jonathan Tan","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"310450","messageId":"CAFKYj4cMSK5nQ1nS66c4Opz8y7x+xQH+OdW8PTi7LmCiGBP1ZA@mail.gmail.com","threadId":"44992","inReplyTo":null,"subject":"Deadlock between git-remote-http and git fetch-pack","fromName":"tsuna","fromEmail":"tsunanet@gmail.com","sentAt":"2017-01-27T22:31:15Z","receivedAt":"2017-01-27T22:38:14Z","isPatch":false,"sender":{"key":"tsunanet@gmail.com","avatar":"https://avatars.githubusercontent.com/u/128281?v=4"},"body":"Hi there,\nWhile investigating a hung job in our CI system today, I think I found\na deadlock in git-remote-http\n\nGit version: 2.9.3\nLinux (amd64) kernel 4.9.0\n\nExcerpt from the process list:\n\njenkins 27316 0.0 0.0 18508 6024 ? S 19:30 0:00 |\n \\_ git -C ../../../arista fetch --unshallow\njenkins 27317 0.0 0.0 169608 10916 ? S 19:30 0:00 |\n \\_ git-remote-http origin http://gerrit/arista\njenkins 27319 0.0 0.0 24160 8260 ? S 19:30 0:00 |\n \\_ git fetch-pack --stateless-rpc --stdin\n--lock-pack --include-tag --thin --no-progress --depth=2147483647\nhttp://gerrit/arista/\n\nHere PID 27319 (git fetch-pack) is stuck reading on stdin, while its\nparent, PID 27317 (git-remote-http) is stuck reading on its child’s\nstdout. Nothing has moved for like 2h, it’s deadlocked.\n\n> strace -fp 27319\nstrace: Process 27319 attached\nread(0,\n\nHere FD 0 is a pipe:\n\n~ @8a33a534e2f7> lsof -np 27319 | grep 0r\ngit 27319 jenkins 0r FIFO 0,10 0t0 354519158 pipe\n\nThe writing end of which is owned by the parent process:\n\n~ @8a33a534e2f7> lsof -n 2>/dev/null | fgrep 354519158\ngit-remot 27317 jenkins 4w FIFO 0,10 0t0\n354519158 pipe\ngit 27319 jenkins 0r FIFO 0,10 0t0\n354519158 pipe\n\nAnd the parent process (git-remote-http) is stuck reading from another FD:\n\n> strace -fp 27317\nstrace: Process 27317 attached\nread(5,\n\nAnd here FD 5 is another pipe:\n\n~ @8a33a534e2f7> lsof -np 27317 | grep 5r\ngit-remot 27317 jenkins 5r FIFO 0,10 0t0 354519159 pipe\n\nWhich is the child’s stdout:\n\n> lsof -n 2>/dev/null | fgrep 354519159\ngit-remot 27317 jenkins 5r FIFO 0,10 0t0\n354519159 pipe\ngit 27319 jenkins 1w FIFO 0,10 0t0\n354519159 pipe\n\nHence the deadlock.\n\nStack trace in git-remote-http:\n\n(gdb) bt\n#0 0x00007f04f1e1363d in read () from target:/lib64/libpthread.so.0\n#1 0x0000562417472d73 in xread ()\n#2 0x0000562417472f2b in read_in_full ()\n#3 0x0000562417438a6e in get_packet_data ()\n#4 0x0000562417439129 in packet_read ()\n#5 0x00005624174245e0 in rpc_service ()\n#6 0x0000562417424f10 in fetch_git ()\n#7 0x00005624174233fd in main ()\n\nStack trace in git fetch-pack:\n\n(gdb) bt\n#0 0x00007fb3ab478620 in __read_nocancel () from target:/lib64/libpthread.so.0\n#1 0x000055f688827283 in xread ()\n#2 0x000055f68882743b in read_in_full ()\n#3 0x000055f6887ce35e in get_packet_data ()\n#4 0x000055f6887cea19 in packet_read ()\n#5 0x000055f6887ceb90 in packet_read_line ()\n#6 0x000055f68879dd05 in get_ack ()\n#7 0x000055f68879f6b4 in fetch_pack ()\n#8 0x000055f688710619 in cmd_fetch_pack ()\n#9 0x000055f6886dff7b in handle_builtin ()\n#10 0x000055f6886df026 in main ()\n\nI looked at the diff between v2.9.3 and HEAD on fetch-pack.c and\nremote-curl.c and didn’t see anything noteworthy in that area of the\ncode, so I presume the bug is still there in master.\n\n-- \nBenoit \"tsuna\" Sigoure\n"},{"id":"310451","messageId":"460fae08-d5af-0e18-e2ed-faca2cfc8826@google.com","threadId":"44992","inReplyTo":"CAFKYj4cMSK5nQ1nS66c4Opz8y7x+xQH+OdW8PTi7LmCiGBP1ZA@mail.gmail.com","subject":"Re: Deadlock between git-remote-http and git fetch-pack","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-01-27T23:19:28Z","receivedAt":"2017-01-27T23:26:55Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 01/27/2017 02:31 PM, tsuna wrote:\n> Hi there,\n> While investigating a hung job in our CI system today, I think I found\n> a deadlock in git-remote-http\n>\n> Git version: 2.9.3\n> Linux (amd64) kernel 4.9.0\n>\n> Excerpt from the process list:\n>\n> jenkins 27316 0.0 0.0 18508 6024 ? S 19:30 0:00 |\n> \\_ git -C ../../../arista fetch --unshallow\n> jenkins 27317 0.0 0.0 169608 10916 ? S 19:30 0:00 |\n> \\_ git-remote-http origin http://gerrit/arista\n> jenkins 27319 0.0 0.0 24160 8260 ? S 19:30 0:00 |\n> \\_ git fetch-pack --stateless-rpc --stdin\n> --lock-pack --include-tag --thin --no-progress --depth=2147483647\n> http://gerrit/arista/\n>\n> Here PID 27319 (git fetch-pack) is stuck reading on stdin, while its\n> parent, PID 27317 (git-remote-http) is stuck reading on its child’s\n> stdout. Nothing has moved for like 2h, it’s deadlocked.\n>\n>> strace -fp 27319\n> strace: Process 27319 attached\n> read(0,\n>\n> Here FD 0 is a pipe:\n>\n> ~ @8a33a534e2f7> lsof -np 27319 | grep 0r\n> git 27319 jenkins 0r FIFO 0,10 0t0 354519158 pipe\n>\n> The writing end of which is owned by the parent process:\n>\n> ~ @8a33a534e2f7> lsof -n 2>/dev/null | fgrep 354519158\n> git-remot 27317 jenkins 4w FIFO 0,10 0t0\n> 354519158 pipe\n> git 27319 jenkins 0r FIFO 0,10 0t0\n> 354519158 pipe\n>\n> And the parent process (git-remote-http) is stuck reading from another FD:\n>\n>> strace -fp 27317\n> strace: Process 27317 attached\n> read(5,\n>\n> And here FD 5 is another pipe:\n>\n> ~ @8a33a534e2f7> lsof -np 27317 | grep 5r\n> git-remot 27317 jenkins 5r FIFO 0,10 0t0 354519159 pipe\n>\n> Which is the child’s stdout:\n>\n>> lsof -n 2>/dev/null | fgrep 354519159\n> git-remot 27317 jenkins 5r FIFO 0,10 0t0\n> 354519159 pipe\n> git 27319 jenkins 1w FIFO 0,10 0t0\n> 354519159 pipe\n>\n> Hence the deadlock.\n>\n> Stack trace in git-remote-http:\n>\n> (gdb) bt\n> #0 0x00007f04f1e1363d in read () from target:/lib64/libpthread.so.0\n> #1 0x0000562417472d73 in xread ()\n> #2 0x0000562417472f2b in read_in_full ()\n> #3 0x0000562417438a6e in get_packet_data ()\n> #4 0x0000562417439129 in packet_read ()\n> #5 0x00005624174245e0 in rpc_service ()\n> #6 0x0000562417424f10 in fetch_git ()\n> #7 0x00005624174233fd in main ()\n>\n> Stack trace in git fetch-pack:\n>\n> (gdb) bt\n> #0 0x00007fb3ab478620 in __read_nocancel () from target:/lib64/libpthread.so.0\n> #1 0x000055f688827283 in xread ()\n> #2 0x000055f68882743b in read_in_full ()\n> #3 0x000055f6887ce35e in get_packet_data ()\n> #4 0x000055f6887cea19 in packet_read ()\n> #5 0x000055f6887ceb90 in packet_read_line ()\n> #6 0x000055f68879dd05 in get_ack ()\n> #7 0x000055f68879f6b4 in fetch_pack ()\n> #8 0x000055f688710619 in cmd_fetch_pack ()\n> #9 0x000055f6886dff7b in handle_builtin ()\n> #10 0x000055f6886df026 in main ()\n>\n> I looked at the diff between v2.9.3 and HEAD on fetch-pack.c and\n> remote-curl.c and didn’t see anything noteworthy in that area of the\n> code, so I presume the bug is still there in master.\n>\n\nI haven't looked into this in detail, but this might be related to \nsomething I discovered while writing my patch set. I noticed that \nupload-pack (the process on the \"other side\" of fetch-pack) can die \nwithout first writing any notification, causing fetch-pack to block \nforever on a read. A fix would probably look like that patch [1].\n\n[1] \n\n"},{"id":"310453","messageId":"xmqqefzoxeki.fsf@gitster.mtv.corp.google.com","threadId":"44992","inReplyTo":"CAFKYj4cMSK5nQ1nS66c4Opz8y7x+xQH+OdW8PTi7LmCiGBP1ZA@mail.gmail.com","subject":"Re: Deadlock between git-remote-http and git fetch-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-01-27T23:34:21Z","receivedAt":"2017-01-28T00:12:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"tsuna writes:\n\n> While investigating a hung job in our CI system today, I think I found\n> a deadlock in git-remote-http\n> ...\n> Here PID 27319 (git fetch-pack) is stuck reading on stdin, while its\n> parent, PID 27317 (git-remote-http) is stuck reading on its child’s\n> stdout. Nothing has moved for like 2h, it’s deadlocked.\n\nHmph, would this be related to 296b847c0d (\"remote-curl: don't hang\nwhen a server dies before any output\", 2016-11-18) I wonder...\n"}]}