Puzzling Git backtrace

Bryan Turner· Mar 28, 2019, 00:49 UTC · lore

I'm trying to assist a Bitbucket Server customer who is seeing some "git-upload-pack" processes "hang" on their server.

While investigating, we had them connect gdb to their processes (which are 2.10.0 built from source using a simple unzip-and-run-make approach) and get the backtraces for them. The output that they're seeing makes no sense to me, though, so I'm throwing this out to the list just to see if anyone has any idea how the processes could end up like this.

One "family" of "hung" processes looks like this in ps:

32432 7811 atlbitb+ Mar22 /bin/git upload-pack /path/to/repository 32433 32432 atlbitb+ Mar22 git-upload-pack /path/to/repository

Looks normal enough. The backtrace for process 32432 also looks like I'd expect:

#0 0x00007fa39842fc9c in __libc_waitpid (pid=pid@entry=32433, stat_loc=stat_loc@entry=0x7fffb42a4e4c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31 #1 0x00000000005098f2 in wait_or_whine (pid=32433, argv0=0xd6e0c0 "git-upload-pack", in_signal=in_signal@entry=0) at run-command.c:229 #2 0x000000000050a949 in finish_command (cmd=cmd@entry=0x7fffb42a4ea0) at run-command.c:537 #3 0x000000000050a9c1 in run_command (cmd=cmd@entry=0x7fffb42a4ea0) at run-command.c:558 #4 0x000000000050aa5e in run_command_v_opt_cd_env (argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40, dir=dir@entry=0x0, env=env@entry=0x0) at run-command.c:578 #5 0x000000000050aa79 in run_command_v_opt (argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40) at run-command.c:563 #6 0x0000000000405ad0 in execv_dashed_external (argv=0x7fffb42a5140) at git.c:569 #7 run_argv (argv=0x7fffb42a4f10, argcp=0x7fffb42a4f1c) at git.c:596 #8 cmd_main (argc=2, argc@entry=3, argv=0x7fffb42a5140, argv@entry=0x7fffb42a5138) at git.c:665 #9 0x0000000000404d9d in main (argc=3, argv=0x7fffb42a5138) at common-main.c:40

It's in "wait_or_whine" for "git-upload-pack".

When they attached to 32433 and printed its backtrace, though, things go a little sideways:

(gdb) attach 32433 Attaching to program: /usr/bin/git, process 32433 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007f79d1aca240 in ?? () (gdb) bt #0 0x00007f79d1aca240 in ?? () #1 0x000000000045179e in mktree_line (allow_missing=4, nul_term_line=0, len=<optimized out>, buf=<optimized out>) at builtin/mktree.c:103 #2 cmd_mktree (ac=<optimized out>, av=<optimized out>, prefix=<optimized out>) at builtin/mktree.c:173 #3 0x0000000000000000 in ?? ()

So ps shows "git-upload-pack", and the parent process shows it's waiting for "git-upload-pack", but gdb shows it's attaching to "git", not "git-upload-pack", and the stack trace on the child shows it's running "git mktree". I've searched through the source code and I don't see any path through upload-pack.c that could result in it essentially exec'ing "git mktree" over the top of itself, or even spawning a "git mktree" of its own.

Has anyone seen anything like this before? Any thoughts on how what should be a "git-upload-pack" could possibly end up being a "git mktree" instead?

Best regards, Bryan

Jeff King· Mar 28, 2019, 01:49 UTC · re: Bryan Turner · lore

Re: Puzzling Git backtrace

On Wed, Mar 27, 2019 at 05:49:27PM -0700, Bryan Turner wrote:

Show 9 quoted lines

> I'm trying to assist a Bitbucket Server customer who is seeing some
> "git-upload-pack" processes "hang" on their server.
> 
> While investigating, we had them connect gdb to their processes (which
> are 2.10.0 built from source using a simple unzip-and-run-make
> approach) and get the backtraces for them. The output that they're
> seeing makes no sense to me, though, so I'm throwing this out to the
> list just to see if anyone has any idea how the processes could end up
> like this.

upload-pack didn't become a builtin until v2.18, so...

Show 5 quoted lines

> When they attached to 32433 and printed its backtrace, though, things
> go a little sideways:
> 
> (gdb) attach 32433
> Attaching to program: /usr/bin/git, process 32433

The debugger needs to be using git-upload-pack as its executable, not "git".

Show 8 quoted lines

> (gdb) bt
> #0  0x00007f79d1aca240 in ?? ()
> #1  0x000000000045179e in mktree_line (allow_missing=4,
> nul_term_line=0, len=<optimized out>, buf=<optimized out>) at
> builtin/mktree.c:103
> #2  cmd_mktree (ac=<optimized out>, av=<optimized out>,
> prefix=<optimized out>) at builtin/mktree.c:173
> #3  0x0000000000000000 in ?? ()

And all of this is just trash, because it's matching the core with the wrong binary.

I'm sure there's a way to switch binaries within gdb, but I don't know it offhand. I'd just start a new gdb like:

  gdb git-upload-pack 32433

-Peff

Bryan Turner· Mar 28, 2019, 02:59 UTC · re: Jeff King · lore

Re: Puzzling Git backtrace

On Wed, Mar 27, 2019 at 6:50 PM Jeff King <peff@peff.net> wrote:

Show 23 quoted lines

>
> On Wed, Mar 27, 2019 at 05:49:27PM -0700, Bryan Turner wrote:
>
> > I'm trying to assist a Bitbucket Server customer who is seeing some
> > "git-upload-pack" processes "hang" on their server.
> >
> > While investigating, we had them connect gdb to their processes (which
> > are 2.10.0 built from source using a simple unzip-and-run-make
> > approach) and get the backtraces for them. The output that they're
> > seeing makes no sense to me, though, so I'm throwing this out to the
> > list just to see if anyone has any idea how the processes could end up
> > like this.
>
> upload-pack didn't become a builtin until v2.18, so...
>
> > When they attached to 32433 and printed its backtrace, though, things
> > go a little sideways:
> >
> > (gdb) attach 32433
> > Attaching to program: /usr/bin/git, process 32433
>
> The debugger needs to be using git-upload-pack as its executable, not
> "git".

Derp. Of course. Sorry, clearly it's been too long since I used C in anger!

Thanks for taking the time to point out what should have been obvious, and for doing it so kindly!

Bryan