{"thread":{"id":"38665","subject":"Deadlock during remote update","startedAt":"2015-02-27T11:03:18Z","lastAt":"2015-02-27T13:21:13Z","messageCount":3,"participants":["Heald, Mike","Duy Nguyen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"256739","messageId":"311995D1237EC0498D6D8FAFA73D6F458F1B67@G9W0762.americas.hpqcorp.net","threadId":"38665","inReplyTo":null,"subject":"Deadlock during remote update","fromName":"Heald, Mike","fromEmail":"mike.heald@hp.com","sentAt":"2015-02-27T11:03:18Z","receivedAt":"2015-02-27T11:03:18Z","isPatch":false,"sender":{"key":"mike.heald@hp.com","avatar":null},"body":"Hi,\n\nWe have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday:\n\n31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune\n    wait4'ing for 31630\n31630 git fetch --prune --multiple --all\n    wait4'ing for 31637\n31637 git fetch --append --prune upstream\n    waiting for read from pipe:[170381707] (fd 25)\n31638 git-remote-https upstream https://git.openstack.org/openstack/nova\n    has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]\n31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master\n    has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638\n\n31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.\n\nThis is git version 1.7.9.5, OS is ubuntu precise.\n\nAny ideas, or any way I can get more useful info?\n\nThanks,\nMike\n"},{"id":"256740","messageId":"CACsJy8AG9ZdhZj_pAb3sZPntBt+GPLj06dXDAV=w+5HyL6D+0Q@mail.gmail.com","threadId":"38665","inReplyTo":"311995D1237EC0498D6D8FAFA73D6F458F1B67@G9W0762.americas.hpqcorp.net","subject":"Re: Deadlock during remote update","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-27T11:27:28Z","receivedAt":"2015-02-27T11:27:28Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Feb 27, 2015 at 6:03 PM, Heald, Mike <mike.heald@hp.com> wrote:\n> Hi,\n>\n> We have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday:\n>\n> 31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune\n>     wait4'ing for 31630\n> 31630 git fetch --prune --multiple --all\n>     wait4'ing for 31637\n> 31637 git fetch --append --prune upstream\n>     waiting for read from pipe:[170381707] (fd 25)\n> 31638 git-remote-https upstream https://git.openstack.org/openstack/nova\n>     has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]\n> 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master\n>     has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638\n>\n> 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.\n>\n> This is git version 1.7.9.5, OS is ubuntu precise.\n\nThere's this commit 2a45520 (remote-curl: always parse incoming refs -\n2013-02-20) that fixes a deadlock between remote-http(s) and\nfetch-pack. I'm not sure if it's the same problem. Jeff  knows more\nabout this. The fix is available since v1.8.3\n-- \nDuy\n"},{"id":"256742","messageId":"20150227132113.GB13005@peff.net","threadId":"38665","inReplyTo":"CACsJy8AG9ZdhZj_pAb3sZPntBt+GPLj06dXDAV=w+5HyL6D+0Q@mail.gmail.com","subject":"Re: Deadlock during remote update","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-02-27T13:21:13Z","receivedAt":"2015-02-27T13:21:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 27, 2015 at 06:27:28PM +0700, Duy Nguyen wrote:\n\n> > 31638 git-remote-https upstream https://git.openstack.org/openstack/nova\n> >     has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]\n> > 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master\n> >     has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638\n> >\n> > 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.\n> >\n> > This is git version 1.7.9.5, OS is ubuntu precise.\n> \n> There's this commit 2a45520 (remote-curl: always parse incoming refs -\n> 2013-02-20) that fixes a deadlock between remote-http(s) and\n> fetch-pack. I'm not sure if it's the same problem. Jeff  knows more\n> about this. The fix is available since v1.8.3\n\nYou are more optimistic about my memory than I am. :)\n\nGiven the symptoms of the deadlock above, I agree it is a likely\nculprit. There was a spree of deadlock fixes around the same time, but I\nthink that is the only one that was found on the client side of a fetch.\n\n-Peff\n"}]}