threads / discuss / 38665

Deadlock during remote update

Subject: Deadlock during remote update

## tl;dr

3 messages between Feb 27, 2015 and Feb 27, 2015.

replies: 2people: 3as markdown or json

Heald, Mike· Feb 27, 2015, 11:03 UTC · lore
Hi,
We have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday:
31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune
    wait4'ing for 31630
31630 git fetch --prune --multiple --all
    wait4'ing for 31637
31637 git fetch --append --prune upstream
    waiting for read from pipe:[170381707] (fd 25)
31638 git-remote-https upstream https://git.openstack.org/openstack/nova
    has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]
31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master
    has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638
31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.
This is git version 1.7.9.5, OS is ubuntu precise.
Any ideas, or any way I can get more useful info?

Thanks, Mike

Duy Nguyen· Feb 27, 2015, 11:27 UTC · re: Heald, Mike · lore

Re: Deadlock during remote update

On Fri, Feb 27, 2015 at 6:03 PM, Heald, Mike <mike.heald@hp.com> wrote:
Show 18 quoted lines
> Hi,
>
> We have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday:
>
> 31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune
>     wait4'ing for 31630
> 31630 git fetch --prune --multiple --all
>     wait4'ing for 31637
> 31637 git fetch --append --prune upstream
>     waiting for read from pipe:[170381707] (fd 25)
> 31638 git-remote-https upstream https://git.openstack.org/openstack/nova
>     has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]
> 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master
>     has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638
>
> 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.
>
> This is git version 1.7.9.5, OS is ubuntu precise.

There's this commit 2a45520 (remote-curl: always parse incoming refs - 2013-02-20) that fixes a deadlock between remote-http(s) and fetch-pack. I'm not sure if it's the same problem. Jeff knows more about this. The fix is available since v1.8.3

-- 
Duy
Jeff King· Feb 27, 2015, 13:21 UTC · re: Duy Nguyen · lore

Re: Deadlock during remote update

On Fri, Feb 27, 2015 at 06:27:28PM +0700, Duy Nguyen wrote:
Show 13 quoted lines
> > 31638 git-remote-https upstream https://git.openstack.org/openstack/nova
> >     has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]
> > 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master
> >     has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638
> >
> > 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.
> >
> > This is git version 1.7.9.5, OS is ubuntu precise.
> 
> There's this commit 2a45520 (remote-curl: always parse incoming refs -
> 2013-02-20) that fixes a deadlock between remote-http(s) and
> fetch-pack. I'm not sure if it's the same problem. Jeff  knows more
> about this. The fix is available since v1.8.3
You are more optimistic about my memory than I am. :)

Given the symptoms of the deadlock above, I agree it is a likely culprit. There was a spree of deadlock fixes around the same time, but I think that is the only one that was found on the client side of a fetch.

-Peff

← back to recent threads