# Deadlock during remote update 3 messages from 2015-02-27 to 2015-02-27. Participants: Heald, Mike, Duy Nguyen, Jeff King. Thread: https://gitlist.dev/t/38665 ## Heald, Mike, 2015-02-27 11:03 Subject: Deadlock during remote update Message-ID: <311995D1237EC0498D6D8FAFA73D6F458F1B67@G9W0762.americas.hpqcorp.net> URL: https://gitlist.dev/e/311995D1237EC0498D6D8FAFA73D6F458F1B67%40G9W0762.americas.hpqcorp.net ``` Hi, We have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday: 31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune wait4'ing for 31630 31630 git fetch --prune --multiple --all wait4'ing for 31637 31637 git fetch --append --prune upstream waiting for read from pipe:[170381707] (fd 25) 31638 git-remote-https upstream https://git.openstack.org/openstack/nova has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472] 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/ efs/heads/master has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to. This is git version 1.7.9.5, OS is ubuntu precise. Any ideas, or any way I can get more useful info? Thanks, Mike ``` ## Duy Nguyen, 2015-02-27 11:27 Subject: Re: Deadlock during remote update Message-ID: URL: https://gitlist.dev/e/CACsJy8AG9ZdhZj_pAb3sZPntBt%2BGPLj06dXDAV%3Dw%2B5HyL6D%2B0Q%40mail.gmail.com In-Reply-To: <311995D1237EC0498D6D8FAFA73D6F458F1B67@G9W0762.americas.hpqcorp.net> ``` On Fri, Feb 27, 2015 at 6:03 PM, Heald, Mike wrote: > Hi, > > We have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday: > > 31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune > wait4'ing for 31630 > 31630 git fetch --prune --multiple --all > wait4'ing for 31637 > 31637 git fetch --append --prune upstream > waiting for read from pipe:[170381707] (fd 25) > 31638 git-remote-https upstream https://git.openstack.org/openstack/nova > has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472] > 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/ efs/heads/master > has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638 > > 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to. > > This is git version 1.7.9.5, OS is ubuntu precise. There's this commit 2a45520 (remote-curl: always parse incoming refs - 2013-02-20) that fixes a deadlock between remote-http(s) and fetch-pack. I'm not sure if it's the same problem. Jeff knows more about this. The fix is available since v1.8.3 -- Duy ``` ## Jeff King, 2015-02-27 13:21 Subject: Re: Deadlock during remote update Message-ID: <20150227132113.GB13005@peff.net> URL: https://gitlist.dev/e/20150227132113.GB13005%40peff.net In-Reply-To: ``` On Fri, Feb 27, 2015 at 06:27:28PM +0700, Duy Nguyen wrote: > > 31638 git-remote-https upstream https://git.openstack.org/openstack/nova > > has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472] > > 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/ efs/heads/master > > has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638 > > > > 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to. > > > > This is git version 1.7.9.5, OS is ubuntu precise. > > There's this commit 2a45520 (remote-curl: always parse incoming refs - > 2013-02-20) that fixes a deadlock between remote-http(s) and > fetch-pack. I'm not sure if it's the same problem. Jeff knows more > about this. The fix is available since v1.8.3 You are more optimistic about my memory than I am. :) Given the symptoms of the deadlock above, I agree it is a likely culprit. There was a spree of deadlock fixes around the same time, but I think that is the only one that was found on the client side of a fetch. -Peff ```