{"thread":{"id":"50831","subject":"Puzzling Git backtrace","startedAt":"2019-03-28T00:49:41Z","lastAt":"2019-03-28T03:00:06Z","messageCount":3,"participants":["Bryan Turner","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"372597","messageId":"CAGyf7-F8cmzOuhi6zeJJ13iwGh_ie-uTiJYUzBwssvH+kCY+yA@mail.gmail.com","threadId":"50831","inReplyTo":null,"subject":"Puzzling Git backtrace","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2019-03-28T00:49:27Z","receivedAt":"2019-03-28T00:49:41Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"I'm trying to assist a Bitbucket Server customer who is seeing some\n\"git-upload-pack\" processes \"hang\" on their server.\n\nWhile investigating, we had them connect gdb to their processes (which\nare 2.10.0 built from source using a simple unzip-and-run-make\napproach) and get the backtraces for them. The output that they're\nseeing makes no sense to me, though, so I'm throwing this out to the\nlist just to see if anyone has any idea how the processes could end up\nlike this.\n\nOne \"family\" of \"hung\" processes looks like this in ps:\n\n32432 7811 atlbitb+ Mar22 /bin/git upload-pack /path/to/repository\n32433 32432 atlbitb+ Mar22 git-upload-pack /path/to/repository\n\nLooks normal enough. The backtrace for process 32432 also looks like I'd expect:\n\n#0 0x00007fa39842fc9c in __libc_waitpid (pid=pid@entry=32433,\nstat_loc=stat_loc@entry=0x7fffb42a4e4c, options=options@entry=0) at\n../sysdeps/unix/sysv/linux/waitpid.c:31\n#1 0x00000000005098f2 in wait_or_whine (pid=32433, argv0=0xd6e0c0\n\"git-upload-pack\", in_signal=in_signal@entry=0) at run-command.c:229\n#2 0x000000000050a949 in finish_command\n(cmd=cmd@entry=0x7fffb42a4ea0) at run-command.c:537\n#3 0x000000000050a9c1 in run_command (cmd=cmd@entry=0x7fffb42a4ea0)\nat run-command.c:558\n#4 0x000000000050aa5e in run_command_v_opt_cd_env\n(argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40, dir=dir@entry=0x0,\nenv=env@entry=0x0) at run-command.c:578\n#5 0x000000000050aa79 in run_command_v_opt\n(argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40) at\nrun-command.c:563\n#6 0x0000000000405ad0 in execv_dashed_external (argv=0x7fffb42a5140)\nat git.c:569\n#7 run_argv (argv=0x7fffb42a4f10, argcp=0x7fffb42a4f1c) at git.c:596\n#8 cmd_main (argc=2, argc@entry=3, argv=0x7fffb42a5140,\nargv@entry=0x7fffb42a5138) at git.c:665\n#9 0x0000000000404d9d in main (argc=3, argv=0x7fffb42a5138) at common-main.c:40\n\nIt's in \"wait_or_whine\" for \"git-upload-pack\".\n\nWhen they attached to 32433 and printed its backtrace, though, things\ngo a little sideways:\n\n(gdb) attach 32433\nAttaching to program: /usr/bin/git, process 32433\nReading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols\nfrom /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.\ndone.\nLoaded symbols for /lib64/ld-linux-x86-64.so.2\n0x00007f79d1aca240 in ?? ()\n(gdb) bt\n#0 0x00007f79d1aca240 in ?? ()\n#1 0x000000000045179e in mktree_line (allow_missing=4,\nnul_term_line=0, len=, buf=) at\nbuiltin/mktree.c:103\n#2 cmd_mktree (ac=, av=,\nprefix=) at builtin/mktree.c:173\n#3 0x0000000000000000 in ?? ()\n\nSo ps shows \"git-upload-pack\", and the parent process shows it's\nwaiting for \"git-upload-pack\", but gdb shows it's attaching to \"git\",\nnot \"git-upload-pack\", and the stack trace on the child shows it's\nrunning \"git mktree\". I've searched through the source code and I\ndon't see any path through upload-pack.c that could result in it\nessentially exec'ing \"git mktree\" over the top of itself, or even\nspawning a \"git mktree\" of its own.\n\nHas anyone seen anything like this before? Any thoughts on how what\nshould be a \"git-upload-pack\" could possibly end up being a \"git\nmktree\" instead?\n\nBest regards,\nBryan\n"},{"id":"372599","messageId":"20190328014957.GA7887@sigill.intra.peff.net","threadId":"50831","inReplyTo":"CAGyf7-F8cmzOuhi6zeJJ13iwGh_ie-uTiJYUzBwssvH+kCY+yA@mail.gmail.com","subject":"Re: Puzzling Git backtrace","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-03-28T01:49:58Z","receivedAt":"2019-03-28T01:50:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 27, 2019 at 05:49:27PM -0700, Bryan Turner wrote:\n\n> I'm trying to assist a Bitbucket Server customer who is seeing some\n> \"git-upload-pack\" processes \"hang\" on their server.\n> \n> While investigating, we had them connect gdb to their processes (which\n> are 2.10.0 built from source using a simple unzip-and-run-make\n> approach) and get the backtraces for them. The output that they're\n> seeing makes no sense to me, though, so I'm throwing this out to the\n> list just to see if anyone has any idea how the processes could end up\n> like this.\n\nupload-pack didn't become a builtin until v2.18, so...\n\n> When they attached to 32433 and printed its backtrace, though, things\n> go a little sideways:\n> \n> (gdb) attach 32433\n> Attaching to program: /usr/bin/git, process 32433\n\nThe debugger needs to be using git-upload-pack as its executable, not\n\"git\".\n\n> (gdb) bt\n> #0 0x00007f79d1aca240 in ?? ()\n> #1 0x000000000045179e in mktree_line (allow_missing=4,\n> nul_term_line=0, len=, buf=) at\n> builtin/mktree.c:103\n> #2 cmd_mktree (ac=, av=,\n> prefix=) at builtin/mktree.c:173\n> #3 0x0000000000000000 in ?? ()\n\nAnd all of this is just trash, because it's matching the core with the\nwrong binary.\n\nI'm sure there's a way to switch binaries within gdb, but I don't know\nit offhand. I'd just start a new gdb like:\n\n gdb git-upload-pack 32433\n\n-Peff\n"},{"id":"372603","messageId":"CAGyf7-EeeuaHKwn776i4YfFEX_rckKoszvn+K5mtD-aAafT=7w@mail.gmail.com","threadId":"50831","inReplyTo":"20190328014957.GA7887@sigill.intra.peff.net","subject":"Re: Puzzling Git backtrace","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2019-03-28T02:59:52Z","receivedAt":"2019-03-28T03:00:06Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Wed, Mar 27, 2019 at 6:50 PM Jeff King wrote:\n>\n> On Wed, Mar 27, 2019 at 05:49:27PM -0700, Bryan Turner wrote:\n>\n> > I'm trying to assist a Bitbucket Server customer who is seeing some\n> > \"git-upload-pack\" processes \"hang\" on their server.\n> >\n> > While investigating, we had them connect gdb to their processes (which\n> > are 2.10.0 built from source using a simple unzip-and-run-make\n> > approach) and get the backtraces for them. The output that they're\n> > seeing makes no sense to me, though, so I'm throwing this out to the\n> > list just to see if anyone has any idea how the processes could end up\n> > like this.\n>\n> upload-pack didn't become a builtin until v2.18, so...\n>\n> > When they attached to 32433 and printed its backtrace, though, things\n> > go a little sideways:\n> >\n> > (gdb) attach 32433\n> > Attaching to program: /usr/bin/git, process 32433\n>\n> The debugger needs to be using git-upload-pack as its executable, not\n> \"git\".\n\nDerp. Of course. Sorry, clearly it's been too long since I used C in anger!\n\nThanks for taking the time to point out what should have been obvious,\nand for doing it so kindly!\n\nBryan\n"}]}