# git hangs on pthread_join 4 messages from 2013-05-23 to 2013-05-29. Participants: Ian Kumlien, Martin Fick, Jeff King. Thread: https://gitlist.dev/t/33908 ## Ian Kumlien, 2013-05-23 13:01 Subject: git hangs on pthread_join Message-ID: <20130523130143.GA6376@pomac.netswarm.net> URL: https://gitlist.dev/e/20130523130143.GA6376%40pomac.netswarm.net ``` Hi, I'm running a rather special configuration, basically i have a gerrit server pushing git data over openvpn connections (company regulations n' stuff)... git 1.8.2.1 is started by xinetd ... port = 9418 socket_type = stream wait = no user = gerrit2 server = /usr/bin/git server_args = daemon --inetd --syslog --export-all --enable=receive-pack --init-timeout=3 --timeout=180 --base-path= ... nice = 10 per_source = UNLIMITED instances = UNLIMITED flags = KEEPALIVE NODELAY --- Keepalive and nodelay has been added post fact, the same goes for the timeouts. I have found "git receive-pack"s that has been running for days/weeks without terminating.... Attaching gdb and doing a trace results in: #0 0x0000003261207b35 in pthread_join () from /lib64/libpthread.so.0 #1 0x00000000004ce58b in finish_async () #2 0x000000000045744b in cmd_receive_pack () #3 0x0000000000404851 in handle_internal_command () #4 0x0000000000404c9d in main () (sorry don't have any debug data for the binary packages apparenlty (rpms was built from the official source)) (RHEL 5 machine with glibc 2.5-65.el5_7.1) Anyone that has any clues about what could be going wrong? ``` ## Martin Fick, 2013-05-23 19:45 Subject: Re: git hangs on pthread_join Message-ID: <201305231345.38190.mfick@codeaurora.org> URL: https://gitlist.dev/e/201305231345.38190.mfick%40codeaurora.org In-Reply-To: <20130523130143.GA6376@pomac.netswarm.net> ``` On Thursday, May 23, 2013 07:01:43 am you wrote: > > I'm running a rather special configuration, basically i > have a gerrit server pushing ... > I have found "git receive-pack"s that has been running > for days/weeks without terminating.... > ... > Anyone that has any clues about what could be going > wrong? -- Have you narrowed down whether this is a git client problem, or a server problem (gerrit in your case). Is this a repeatable issue. Try the same operation against a clone of the repo using just git. Check on the server side for .noz files in you repo (a jgit thing), -Martin ``` ## Jeff King, 2013-05-28 17:51 Subject: Re: git hangs on pthread_join Message-ID: <20130528175108.GA21210@sigill.intra.peff.net> URL: https://gitlist.dev/e/20130528175108.GA21210%40sigill.intra.peff.net In-Reply-To: <20130523130143.GA6376@pomac.netswarm.net> ``` On Thu, May 23, 2013 at 03:01:43PM +0200, Ian Kumlien wrote: > git 1.8.2.1 is started by xinetd > [...] > I have found "git receive-pack"s that has been running for days/weeks > without terminating.... > > Attaching gdb and doing a trace results in: > #0 0x0000003261207b35 in pthread_join () from /lib64/libpthread.so.0 > #1 0x00000000004ce58b in finish_async () > #2 0x000000000045744b in cmd_receive_pack () > #3 0x0000000000404851 in handle_internal_command () > #4 0x0000000000404c9d in main () I recently fixed a deadlock that could happen in receive-pack when clients hung up before sending a valid pack header. The fix is commit 49ecfa1, and it's in git v1.8.2.2. The stack trace for the deadlock fixed by 49ecfa1 would have unpack_with_sideband between #1 and #2 above, but it is entirely possible that it is simply inlined in your build of git, depending on the -O level of your build (it is a static function that is only called from one place). So it seems likely that it is the culprit. -Peff ``` ## Ian Kumlien, 2013-05-29 08:29 Subject: Re: git hangs on pthread_join Message-ID: <20130529082903.GC22308@pomac.netswarm.net> URL: https://gitlist.dev/e/20130529082903.GC22308%40pomac.netswarm.net In-Reply-To: <20130528175108.GA21210@sigill.intra.peff.net> ``` On Tue, May 28, 2013 at 01:51:09PM -0400, Jeff King wrote: > On Thu, May 23, 2013 at 03:01:43PM +0200, Ian Kumlien wrote: > > > git 1.8.2.1 is started by xinetd > > [...] > > I have found "git receive-pack"s that has been running for days/weeks > > without terminating.... > > > > Attaching gdb and doing a trace results in: > > #0 0x0000003261207b35 in pthread_join () from /lib64/libpthread.so.0 > > #1 0x00000000004ce58b in finish_async () > > #2 0x000000000045744b in cmd_receive_pack () > > #3 0x0000000000404851 in handle_internal_command () > > #4 0x0000000000404c9d in main () > > I recently fixed a deadlock that could happen in receive-pack when > clients hung up before sending a valid pack header. The fix is commit > 49ecfa1, and it's in git v1.8.2.2. With dodgy connections this could easily happen =) Really nice catch! > The stack trace for the deadlock fixed by 49ecfa1 would have > unpack_with_sideband between #1 and #2 above, but it is entirely > possible that it is simply inlined in your build of git, depending on > the -O level of your build (it is a static function that is only called > from one place). So it seems likely that it is the culprit. Yeah, since it's a RHEL 5 machine i don't even get a debug rpm package =P I will upgrade all machines and keep monitoring, thanks! > -Peff ```