{"thread":{"id":"51170","subject":"'git stash list' => Segmentation fault","startedAt":"2019-05-24T17:27:15Z","lastAt":"2019-05-29T20:10:56Z","messageCount":10,"participants":["Brendan Boerner","Johannes Schindelin","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"376195","messageId":"CAKkfZL2p8yFr3ecsQ63HzeZ+u-Jukf7YcYHk_8iBaKcA4WbEfg@mail.gmail.com","threadId":"51170","inReplyTo":null,"subject":"'git stash list' => Segmentation fault","fromName":"Brendan Boerner","fromEmail":"bboerner.biz@gmail.com","sentAt":"2019-05-24T17:27:02Z","receivedAt":"2019-05-24T17:27:15Z","isPatch":false,"sender":{"key":"bboerner.biz@gmail.com","avatar":"https://gravatar.com/avatar/075d7885f747885c271c5bcef7ed27d9f6368b582588db63acc8844b6ae3ec69?d=mp&s=160"},"body":"Ubuntu 14.04\n\nI was using some prior v2.x using linuxbrew, everything was fine, took\nan update, this started happening.\n\nI verified also happens using the 2.21.0 from the Ubuntu 14.04 PPA\n(this reproduction was using that build, not linuxbrew build).\n\nbboerner@myhost:~/funproject/Src/bash$ git --version\ngit version 2.21.0\n\nbboerner@myhost:~/funproject/Src/bash$ git stash list\nSegmentation fault (core dumped)\nSegmentation fault (core dumped)\nSegmentation fault (core dumped)\nstash@{0}: On devel: NYFL optimization\nstash@{1}: On GL285: gl285\n\nRegards,\nBrendan\n"},{"id":"376243","messageId":"nycvar.QRO.7.76.6.1905272046250.47@tvgsbejvaqbjf.bet","threadId":"51170","inReplyTo":"CAKkfZL2p8yFr3ecsQ63HzeZ+u-Jukf7YcYHk_8iBaKcA4WbEfg@mail.gmail.com","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-05-27T18:47:56Z","receivedAt":"2019-05-27T18:47:59Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brendan,\n\nOn Fri, 24 May 2019, Brendan Boerner wrote:\n\n> Ubuntu 14.04\n>\n> I was using some prior v2.x using linuxbrew, everything was fine, took\n> an update, this started happening.\n>\n> I verified also happens using the 2.21.0 from the Ubuntu 14.04 PPA\n> (this reproduction was using that build, not linuxbrew build).\n>\n> bboerner@myhost:~/funproject/Src/bash$ git --version\n> git version 2.21.0\n>\n> bboerner@myhost:~/funproject/Src/bash$ git stash list\n> Segmentation fault (core dumped)\n> Segmentation fault (core dumped)\n> Segmentation fault (core dumped)\n> stash@{0}: On devel: NYFL optimization\n> stash@{1}: On GL285: gl285\n\nUnfortunately, this leaves *very* little in the way of actionable\nevidence. Would it be possible for you to share a tar'red up copy of the\n.git/ directory? Or do you find that this happens for all of your stashes?\nWhat is the output when you run this with `GIT_TRACE=1`?\n\nCiao,\nJohannes\n"},{"id":"376275","messageId":"20190528063451.GG7946@sigill.intra.peff.net","threadId":"51170","inReplyTo":"nycvar.QRO.7.76.6.1905272046250.47@tvgsbejvaqbjf.bet","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-28T06:34:51Z","receivedAt":"2019-05-28T06:34:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 27, 2019 at 08:47:56PM +0200, Johannes Schindelin wrote:\n\n> > bboerner@myhost:~/funproject/Src/bash$ git stash list\n> > Segmentation fault (core dumped)\n> > Segmentation fault (core dumped)\n> > Segmentation fault (core dumped)\n> > stash@{0}: On devel: NYFL optimization\n> > stash@{1}: On GL285: gl285\n> \n> Unfortunately, this leaves *very* little in the way of actionable\n> evidence. Would it be possible for you to share a tar'red up copy of the\n> .git/ directory? Or do you find that this happens for all of your stashes?\n> What is the output when you run this with `GIT_TRACE=1`?\n\nAlternatively, if you cannot share the repo, a backtrace from gdb might\nhelp. There should be a core file either in the working directory, or in\nthe \".git/\" directory. Try \"gdb git /path/to/core\" and then \"bt\" to get\nthe backtrace.\n\n-Peff\n"},{"id":"376345","messageId":"CAKkfZL30QuBM6vT69OSS_keNuGi1U-bJ+jDiyDfhZmnr9L9xvA@mail.gmail.com","threadId":"51170","inReplyTo":"20190528063451.GG7946@sigill.intra.peff.net","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Brendan Boerner","fromEmail":"bboerner.biz@gmail.com","sentAt":"2019-05-28T19:56:13Z","receivedAt":"2019-05-28T19:56:27Z","isPatch":false,"sender":{"key":"bboerner.biz@gmail.com","avatar":"https://gravatar.com/avatar/075d7885f747885c271c5bcef7ed27d9f6368b582588db63acc8844b6ae3ec69?d=mp&s=160"},"body":"Yes, providing the repo isn't an option.\n\nThanks for the suggestions, here's what I can provide. Unfortunately\nnot a lot in the stack trace.\n\nDo you need the whole repo or would bits of it suffice e.g. index?\n\nbboerner@myhost:~/funproject/Src$ GIT_TRACE=1 git stash list\n14:52:22.633388 git.c:670               trace: exec: git-stash list\n14:52:22.633448 run-command.c:643       trace: run_command: git-stash list\nSegmentation fault (core dumped)\nSegmentation fault (core dumped)\nSegmentation fault (core dumped)\n14:52:24.904494 git.c:419               trace: built-in: git rev-parse --git-dir\n14:52:24.944879 git.c:419               trace: built-in: git rev-parse\n--git-path objects\n14:52:24.949126 git.c:419               trace: built-in: git rev-parse\n--show-prefix\n14:52:24.951414 git.c:419               trace: built-in: git rev-parse\n--show-toplevel\n14:52:24.953447 git.c:419               trace: built-in: git rev-parse\n--git-path index\n14:52:24.955670 git.c:419               trace: built-in: git config\n--get-colorbool color.interactive\n14:52:24.957435 git.c:419               trace: built-in: git config\n--get-color color.interactive.help 'red bold'\n14:52:24.958753 git.c:419               trace: built-in: git config\n--get-color  reset\n14:52:24.960461 git.c:419               trace: built-in: git rev-parse\n--verify --quiet refs/stash\n14:52:25.040053 git.c:419               trace: built-in: git log\n'--format=%gd: %gs' -g --first-parent -m refs/stash --\n14:52:26.436274 run-command.c:643       trace: run_command: unset\nGIT_PAGER_IN_USE; LESS=FRX LV=-c less\nstash@{0}: On devel: NYFL optimization\nstash@{1}: On GL285: gl285\n\n\n(gdb) bt\n#0  0x00007fd321805fdf in ?? ()\n#1  0x0000000000000000 in ?? ()\n\nRegards,\nBrendan\n\nOn Tue, May 28, 2019 at 1:34 AM Jeff King <peff@peff.net> wrote:\n>\n> On Mon, May 27, 2019 at 08:47:56PM +0200, Johannes Schindelin wrote:\n>\n> > > bboerner@myhost:~/funproject/Src/bash$ git stash list\n> > > Segmentation fault (core dumped)\n> > > Segmentation fault (core dumped)\n> > > Segmentation fault (core dumped)\n> > > stash@{0}: On devel: NYFL optimization\n> > > stash@{1}: On GL285: gl285\n> >\n> > Unfortunately, this leaves *very* little in the way of actionable\n> > evidence. Would it be possible for you to share a tar'red up copy of the\n> > .git/ directory? Or do you find that this happens for all of your stashes?\n> > What is the output when you run this with `GIT_TRACE=1`?\n>\n> Alternatively, if you cannot share the repo, a backtrace from gdb might\n> help. There should be a core file either in the working directory, or in\n> the \".git/\" directory. Try \"gdb git /path/to/core\" and then \"bt\" to get\n> the backtrace.\n>\n> -Peff\n"},{"id":"376352","messageId":"20190528204051.GA24650@sigill.intra.peff.net","threadId":"51170","inReplyTo":"CAKkfZL30QuBM6vT69OSS_keNuGi1U-bJ+jDiyDfhZmnr9L9xvA@mail.gmail.com","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-28T20:40:52Z","receivedAt":"2019-05-28T20:40:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 28, 2019 at 02:56:13PM -0500, Brendan Boerner wrote:\n\n> Unfortunately not a lot in the stack trace.\n> [...]\n> (gdb) bt\n> #0  0x00007fd321805fdf in ?? ()\n> #1  0x0000000000000000 in ?? ()\n\nI'll say. :)\n\nI'd suspect either:\n\n  1. Your git isn't built with debugging symbols. Is it possible to\n     build from source? The default CFLAGS if you just run \"make\" should\n     be enough, then you can run it straight out of the build directory\n     as \"$YOUR_BUILD_DIR/bin-wrappers/git\".\n\n  2. It's not the main git binary that's segfaulting. Most commands are\n     built-ins of that binary these days, but a few are not. We should\n     be able to figure out which command is having a problem from the\n     trace output, but...\n\n> 14:52:22.633448 run-command.c:643       trace: run_command: git-stash list\n> Segmentation fault (core dumped)\n> Segmentation fault (core dumped)\n> Segmentation fault (core dumped)\n\nI'd have expected to see more sub-commands here. And it's weird that\nthere are _three_ segfaults. Which version of Git is this? In the\nupcoming release stash is written in C, but from the trace output it\nlooks like this is the old shell-script version (which is what I was\nassuming, but I just want to double check).\n\nIs it possible that your shell is segfaulting? Does \"gdb /bin/sh core\"\nreveal anything more useful (a long shot, I know; it probably doesn't\nhave symbols either)?\n\n> Do you need the whole repo or would bits of it suffice e.g. index?\n\nProbably the whole thing. \"stash list\" wouldn't generally look at\nthe index at all. It should just be doing a \"git log\" on the reflog\nentries found in .git/logs/refs/stash. But...\n\n> 14:52:25.040053 git.c:419               trace: built-in: git log\n> '--format=%gd: %gs' -g --first-parent -m refs/stash --\n> 14:52:26.436274 run-command.c:643       trace: run_command: unset\n> GIT_PAGER_IN_USE; LESS=FRX LV=-c less\n> stash@{0}: On devel: NYFL optimization\n> stash@{1}: On GL285: gl285\n\nWe can see that the \"log\" command actually works! The segfaults seem not\nto derail the command from actually performing its task. So what\nprocesses are actually segfaulting?\n\nMaybe \"strace -o trace.out -f git stash list\" would be instructive. You\nshould be able to find the segfaulting process in that trace, and then\ncan grep for that PID to see what it was doing (especially what command\nwas exec'd, and possibly what it was doing right before the segfault).\n\nI can also help picking through the strace output if you're comfortable\nsending it off-list (it's likely to be much too large to send on-list\nanyway).\n\n-Peff\n"},{"id":"376353","messageId":"CAKkfZL3ZNgNFzxrYbjbhCF5BR12vQPy2sNuYdxmQgmuG+mo9Gw@mail.gmail.com","threadId":"51170","inReplyTo":"20190528204051.GA24650@sigill.intra.peff.net","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Brendan Boerner","fromEmail":"bboerner.biz@gmail.com","sentAt":"2019-05-28T20:53:51Z","receivedAt":"2019-05-28T20:54:04Z","isPatch":false,"sender":{"key":"bboerner.biz@gmail.com","avatar":"https://gravatar.com/avatar/075d7885f747885c271c5bcef7ed27d9f6368b582588db63acc8844b6ae3ec69?d=mp&s=160"},"body":"Hey Jeff,\n\nLot of good tips for debugging for me to work on - I'll work on that.\n\nBoth to avoid spamming the list and if sending large files e.g. strace\nis it ok if I continue correspondence with you directly? (And\nJohannes?)\n\nThanks!\nBrendan\n\nOn Tue, May 28, 2019 at 3:40 PM Jeff King <peff@peff.net> wrote:\n>\n> On Tue, May 28, 2019 at 02:56:13PM -0500, Brendan Boerner wrote:\n>\n> > Unfortunately not a lot in the stack trace.\n> > [...]\n> > (gdb) bt\n> > #0  0x00007fd321805fdf in ?? ()\n> > #1  0x0000000000000000 in ?? ()\n>\n> I'll say. :)\n>\n> I'd suspect either:\n>\n>   1. Your git isn't built with debugging symbols. Is it possible to\n>      build from source? The default CFLAGS if you just run \"make\" should\n>      be enough, then you can run it straight out of the build directory\n>      as \"$YOUR_BUILD_DIR/bin-wrappers/git\".\n>\n>   2. It's not the main git binary that's segfaulting. Most commands are\n>      built-ins of that binary these days, but a few are not. We should\n>      be able to figure out which command is having a problem from the\n>      trace output, but...\n>\n> > 14:52:22.633448 run-command.c:643       trace: run_command: git-stash list\n> > Segmentation fault (core dumped)\n> > Segmentation fault (core dumped)\n> > Segmentation fault (core dumped)\n>\n> I'd have expected to see more sub-commands here. And it's weird that\n> there are _three_ segfaults. Which version of Git is this? In the\n> upcoming release stash is written in C, but from the trace output it\n> looks like this is the old shell-script version (which is what I was\n> assuming, but I just want to double check).\n>\n> Is it possible that your shell is segfaulting? Does \"gdb /bin/sh core\"\n> reveal anything more useful (a long shot, I know; it probably doesn't\n> have symbols either)?\n>\n> > Do you need the whole repo or would bits of it suffice e.g. index?\n>\n> Probably the whole thing. \"stash list\" wouldn't generally look at\n> the index at all. It should just be doing a \"git log\" on the reflog\n> entries found in .git/logs/refs/stash. But...\n>\n> > 14:52:25.040053 git.c:419               trace: built-in: git log\n> > '--format=%gd: %gs' -g --first-parent -m refs/stash --\n> > 14:52:26.436274 run-command.c:643       trace: run_command: unset\n> > GIT_PAGER_IN_USE; LESS=FRX LV=-c less\n> > stash@{0}: On devel: NYFL optimization\n> > stash@{1}: On GL285: gl285\n>\n> We can see that the \"log\" command actually works! The segfaults seem not\n> to derail the command from actually performing its task. So what\n> processes are actually segfaulting?\n>\n> Maybe \"strace -o trace.out -f git stash list\" would be instructive. You\n> should be able to find the segfaulting process in that trace, and then\n> can grep for that PID to see what it was doing (especially what command\n> was exec'd, and possibly what it was doing right before the segfault).\n>\n> I can also help picking through the strace output if you're comfortable\n> sending it off-list (it's likely to be much too large to send on-list\n> anyway).\n>\n> -Peff\n"},{"id":"376355","messageId":"20190528205608.GC24650@sigill.intra.peff.net","threadId":"51170","inReplyTo":"CAKkfZL3ZNgNFzxrYbjbhCF5BR12vQPy2sNuYdxmQgmuG+mo9Gw@mail.gmail.com","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-28T20:56:08Z","receivedAt":"2019-05-28T20:56:11Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 28, 2019 at 03:53:51PM -0500, Brendan Boerner wrote:\n\n> Lot of good tips for debugging for me to work on - I'll work on that.\n> \n> Both to avoid spamming the list and if sending large files e.g. strace\n> is it ok if I continue correspondence with you directly? (And\n> Johannes?)\n\nIn general, please keep substantive responses on the list, so everybody\ncan follow along. Adding to the list volume is fine. :)\n\nBut certainly if there are sensitive bits, feel free to send them\ndirectly off-list.\n\n-Peff\n"},{"id":"376360","messageId":"CAKkfZL1BmOjzJ1QrQ43gJx89si3-M_1KMz=EWXN=pK6HSjCEkQ@mail.gmail.com","threadId":"51170","inReplyTo":"20190528205608.GC24650@sigill.intra.peff.net","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Brendan Boerner","fromEmail":"bboerner.biz@gmail.com","sentAt":"2019-05-28T21:29:13Z","receivedAt":"2019-05-28T21:29:25Z","isPatch":false,"sender":{"key":"bboerner.biz@gmail.com","avatar":"https://gravatar.com/avatar/075d7885f747885c271c5bcef7ed27d9f6368b582588db63acc8844b6ae3ec69?d=mp&s=160"},"body":"Thanks Jeff.\n\nTo answer your previous question I'm using git v2.21.0. I'm using the\nUbuntu 14.04 PPA as well as Linuxbew (also v2.21.0).\n\nYour help helped me narrow this down to linuxbrew. Take it out of my\npath no segfault.\n\nUninstall linuxbrew git, linuxbrew still in PATH, use PPA git, segfault.\n\nNarrowed down further to be the \"gettext\" package - remove it from\nlinuxbrew and PPA git stash list => no segfault.\n\nI'll ping the Linuxbrew folks.\n\nThanks for you help!\nBrendan\n\nOn Tue, May 28, 2019 at 3:56 PM Jeff King <peff@peff.net> wrote:\n>\n> On Tue, May 28, 2019 at 03:53:51PM -0500, Brendan Boerner wrote:\n>\n> > Lot of good tips for debugging for me to work on - I'll work on that.\n> >\n> > Both to avoid spamming the list and if sending large files e.g. strace\n> > is it ok if I continue correspondence with you directly? (And\n> > Johannes?)\n>\n> In general, please keep substantive responses on the list, so everybody\n> can follow along. Adding to the list volume is fine. :)\n>\n> But certainly if there are sensitive bits, feel free to send them\n> directly off-list.\n>\n> -Peff\n"},{"id":"376363","messageId":"20190528213912.GH24650@sigill.intra.peff.net","threadId":"51170","inReplyTo":"CAKkfZL1BmOjzJ1QrQ43gJx89si3-M_1KMz=EWXN=pK6HSjCEkQ@mail.gmail.com","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-05-28T21:39:13Z","receivedAt":"2019-05-28T21:39:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 28, 2019 at 04:29:13PM -0500, Brendan Boerner wrote:\n\n> To answer your previous question I'm using git v2.21.0. I'm using the\n> Ubuntu 14.04 PPA as well as Linuxbew (also v2.21.0).\n> \n> Your help helped me narrow this down to linuxbrew. Take it out of my\n> path no segfault.\n> \n> Uninstall linuxbrew git, linuxbrew still in PATH, use PPA git, segfault.\n> \n> Narrowed down further to be the \"gettext\" package - remove it from\n> linuxbrew and PPA git stash list => no segfault.\n> \n> I'll ping the Linuxbrew folks.\n\nAh, that makes sense. You're indeed running the shell version, and it's\ninvoking \"gettext\" for messages, which is probably what's segfaulting.\nAnd that explains the multiple segfaults and the fact that they didn't\nactually cause the overall operation to fail.\n\nSo probably nothing here for Git to fix. Thanks for following up with\nthe resolution!\n\n-Peff\n"},{"id":"376420","messageId":"CAKkfZL385B99fJ10V-yXVK3KpHNNA-fpM31zwvsBrfr02pD3jw@mail.gmail.com","threadId":"51170","inReplyTo":"20190528213912.GH24650@sigill.intra.peff.net","subject":"Re: 'git stash list' => Segmentation fault","fromName":"Brendan Boerner","fromEmail":"bboerner.biz@gmail.com","sentAt":"2019-05-29T20:10:43Z","receivedAt":"2019-05-29T20:10:56Z","isPatch":false,"sender":{"key":"bboerner.biz@gmail.com","avatar":"https://gravatar.com/avatar/075d7885f747885c271c5bcef7ed27d9f6368b582588db63acc8844b6ae3ec69?d=mp&s=160"},"body":"Hey Jeff,\n\nYes, if I install gettext build from source envsubst and gettext no\nsegfaults. tset from ncurses segfaults as well unless I install from\nsource.\n\nSo weird breakage which manifested as a red herring. :-(\n\nThanks for your help helping me sort it out.\n\nRegards,\nBrendan\n\n\n\nOn Tue, May 28, 2019 at 4:39 PM Jeff King <peff@peff.net> wrote:\n>\n> On Tue, May 28, 2019 at 04:29:13PM -0500, Brendan Boerner wrote:\n>\n> > To answer your previous question I'm using git v2.21.0. I'm using the\n> > Ubuntu 14.04 PPA as well as Linuxbew (also v2.21.0).\n> >\n> > Your help helped me narrow this down to linuxbrew. Take it out of my\n> > path no segfault.\n> >\n> > Uninstall linuxbrew git, linuxbrew still in PATH, use PPA git, segfault.\n> >\n> > Narrowed down further to be the \"gettext\" package - remove it from\n> > linuxbrew and PPA git stash list => no segfault.\n> >\n> > I'll ping the Linuxbrew folks.\n>\n> Ah, that makes sense. You're indeed running the shell version, and it's\n> invoking \"gettext\" for messages, which is probably what's segfaulting.\n> And that explains the multiple segfaults and the fact that they didn't\n> actually cause the overall operation to fail.\n>\n> So probably nothing here for Git to fix. Thanks for following up with\n> the resolution!\n>\n> -Peff\n"}]}