threads / discuss / 33908

git hangs on pthread_join

Subject: git hangs on pthread_join

## tl;dr

4 messages between May 23, 2013 and May 29, 2013.

replies: 3people: 3as markdown or json

Ian Kumlien· May 23, 2013, 13:01 UTC · lore

Hi,

I'm running a rather special configuration, basically i have a gerrit server pushing git data over openvpn connections (company regulations n' stuff)...

git 1.8.2.1 is started by xinetd
...
        port            = 9418
        socket_type     = stream
        wait            = no
        user            = gerrit2
        server          = /usr/bin/git
        server_args     =  daemon --inetd --syslog --export-all
--enable=receive-pack --init-timeout=3 --timeout=180 --base-path=<path>
...
        nice            = 10
        per_source      = UNLIMITED
        instances       = UNLIMITED
        flags           = KEEPALIVE NODELAY
---

Keepalive and nodelay has been added post fact, the same goes for the timeouts.

I have found "git receive-pack"s that has been running for days/weeks without terminating....

Attaching gdb and doing a trace results in: #0 0x0000003261207b35 in pthread_join () from /lib64/libpthread.so.0 #1 0x00000000004ce58b in finish_async () #2 0x000000000045744b in cmd_receive_pack () #3 0x0000000000404851 in handle_internal_command () #4 0x0000000000404c9d in main () (sorry don't have any debug data for the binary packages apparenlty (rpms was built from the official source))

(RHEL 5 machine with glibc 2.5-65.el5_7.1)

Anyone that has any clues about what could be going wrong?

Martin Fick· May 23, 2013, 19:45 UTC · re: Ian Kumlien · lore

Re: git hangs on pthread_join

On Thursday, May 23, 2013 07:01:43 am you wrote:

> 
> I'm running a rather special configuration, basically i
> have a gerrit server pushing

...

> I have found "git receive-pack"s that has been running
> for days/weeks without terminating....
>

...

> Anyone that has any clues about what could be going
> wrong? --

Have you narrowed down whether this is a git client problem, or a server problem (gerrit in your case). Is this a repeatable issue. Try the same operation against a clone of the repo using just git. Check on the server side for .noz files in you repo (a jgit thing),

-Martin

Jeff King· May 28, 2013, 17:51 UTC · re: Ian Kumlien · lore

Re: git hangs on pthread_join

On Thu, May 23, 2013 at 03:01:43PM +0200, Ian Kumlien wrote:

Show 11 quoted lines

> git 1.8.2.1 is started by xinetd
> [...]
> I have found "git receive-pack"s that has been running for days/weeks
> without terminating....
> 
> Attaching gdb and doing a trace results in:
> #0  0x0000003261207b35 in pthread_join () from /lib64/libpthread.so.0
> #1  0x00000000004ce58b in finish_async ()
> #2  0x000000000045744b in cmd_receive_pack ()
> #3  0x0000000000404851 in handle_internal_command ()
> #4  0x0000000000404c9d in main ()

I recently fixed a deadlock that could happen in receive-pack when clients hung up before sending a valid pack header. The fix is commit 49ecfa1, and it's in git v1.8.2.2.

The stack trace for the deadlock fixed by 49ecfa1 would have unpack_with_sideband between #1 and #2 above, but it is entirely possible that it is simply inlined in your build of git, depending on the -O level of your build (it is a static function that is only called from one place). So it seems likely that it is the culprit.

-Peff

Ian Kumlien· May 29, 2013, 08:29 UTC · re: Jeff King · lore

Re: git hangs on pthread_join

On Tue, May 28, 2013 at 01:51:09PM -0400, Jeff King wrote:

Show 17 quoted lines

> On Thu, May 23, 2013 at 03:01:43PM +0200, Ian Kumlien wrote:
> 
> > git 1.8.2.1 is started by xinetd
> > [...]
> > I have found "git receive-pack"s that has been running for days/weeks
> > without terminating....
> > 
> > Attaching gdb and doing a trace results in:
> > #0  0x0000003261207b35 in pthread_join () from /lib64/libpthread.so.0
> > #1  0x00000000004ce58b in finish_async ()
> > #2  0x000000000045744b in cmd_receive_pack ()
> > #3  0x0000000000404851 in handle_internal_command ()
> > #4  0x0000000000404c9d in main ()
> 
> I recently fixed a deadlock that could happen in receive-pack when
> clients hung up before sending a valid pack header. The fix is commit
> 49ecfa1, and it's in git v1.8.2.2.

With dodgy connections this could easily happen =)

Really nice catch!

Show 5 quoted lines

> The stack trace for the deadlock fixed by 49ecfa1 would have
> unpack_with_sideband between #1 and #2 above, but it is entirely
> possible that it is simply inlined in your build of git, depending on
> the -O level of your build (it is a static function that is only called
> from one place). So it seems likely that it is the culprit.

Yeah, since it's a RHEL 5 machine i don't even get a debug rpm package =P

I will upgrade all machines and keep monitoring, thanks!

> -Peff