{"thread":{"id":"47197","subject":"Recovering from gc errors","startedAt":"2017-11-13T21:21:47Z","lastAt":"2017-11-15T00:19:19Z","messageCount":5,"participants":["Marc Branchaud","Jeff King","Eric Sunshine"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"332450","messageId":"4f548c23-7bb5-a672-21bb-6c1dd6de6139@xiplink.com","threadId":"47197","inReplyTo":null,"subject":"Recovering from gc errors","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2017-11-13T21:13:19Z","receivedAt":"2017-11-13T21:21:47Z","isPatch":false,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"(I'm using git 2.15.0.)\n\nSo today \"git gc\" started complaining:\n\nerror: Could not read 2bc277bcb7e9cc6ef2ea677dd1c3dcd1f9af0c2b\nfatal: Failed to traverse parents of commit \n9c355a7726e31b3033b8e714cf7edb4f0a41d8d4\nerror: failed to run repack\n\nI suspect I'm a victim of the worktree+submodule bugs -- as a longtime \nuser of contrib/workdir/git-new-workdir, I've been playing with the \n\"worktree\" command since it was first introduced.  The \"git gc\" error \noccurs when it's run in my main repo; I have not tried it in any of my \nworktrees/workdirs.\n\nVarious incantations of \"git show ... 9c355a7726e31\" only fail with the \nsame error, so I can't determine much about the problematic commit. \nLuckily I'm not particularly concerned with losing objects, as I push \nany important progress to named refs in backup repos.\n\nBut I would like to clean this up in my local repo so that gc stops \nfailing.  I tried simply removing this and other loose commits that trip \nup gc (i.e. the objects/9c/355a7726e31b3033b8e714cf7edb4f0a41d8d4 file \n-- there are 49 such files, all of which are several months old), but \nnow gc complains of a bad tree object:\n\nerror: Could not read c1a99c3520f0b456b8025c50302a4cc9b0b2d777\nfatal: bad tree object c1a99c3520f0b456b8025c50302a4cc9b0b2d777\nerror: failed to run repack\n\nThis object is not lying around loose.  \"git fsck\" lists several \ndangling blob/commit/tree objects, but none of them are c1a99c3520f0b4.\n\nSo I'm not sure what to do next.\n\nAny suggestions?\n\nThanks,\n\n\t\tM.\n"},{"id":"332512","messageId":"20171114055306.3tfi726wzmkcfluk@sigill.intra.peff.net","threadId":"47197","inReplyTo":"4f548c23-7bb5-a672-21bb-6c1dd6de6139@xiplink.com","subject":"Re: Recovering from gc errors","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-11-14T05:53:06Z","receivedAt":"2017-11-14T05:53:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 13, 2017 at 04:13:19PM -0500, Marc Branchaud wrote:\n\n> Various incantations of \"git show ... 9c355a7726e31\" only fail with the same\n> error, so I can't determine much about the problematic commit. Luckily I'm\n> not particularly concerned with losing objects, as I push any important\n> progress to named refs in backup repos.\n\nDoing \"git show\" will require looking at the parent commit to produce\nthe diff. Probably \"git show -s\" would work. But in general for poking\nat corruption, something bare-bones like \"git cat-file commit 9c355a77\"\nis going to be your best bet.\n\n> But I would like to clean this up in my local repo so that gc stops failing.\n> I tried simply removing this and other loose commits that trip up gc (i.e.\n> the objects/9c/355a7726e31b3033b8e714cf7edb4f0a41d8d4 file -- there are 49\n> such files, all of which are several months old), but now gc complains of a\n> bad tree object:\n\nYou can't generally fix corruption issues by deleting objects[1]. The\n\"source\" that makes Git want to have these objects is the refs and\nreflogs. So your best bet is to find which of those point to the\nproblematic objects and delete them.\n\nI'd start by seeing if the breakage is reachable from any refs:\n\n  git rev-list --objects --all >/dev/null\n\nIf that command succeeds, then all your refs are intact and the problem\nis in the reflogs. You can try to figure out which, but I'd probably\njust blow them all away:\n\n  rm -rf .git/logs\n\nIf the rev-list fails, then one or more branch is corrupted.\nUnfortunately the usual efficient tools for asking \"which branch\ncontains this object\" are likely to be broken by the corruption. But you\ncan brute-force it, like:\n\n  git for-each-ref --format='%(refname)' |\n  while read ref; do\n    git rev-list --objects \"$ref\" >/dev/null 2>&1 ||\n    echo \"$ref is broken\"\n  done\n\nHopefully that turns up only branches with little value, and you can\ndelete them:\n\n  git update-ref -d $broken_ref\n\n-Peff\n\n[1] A note on my \"you can't fix corruption by deleting objects\".\n\n    Since abcb86553d (pack-objects: match prune logic for discarding\n    objects, 2014-10-15) , git-gc also traverses the history graph of\n    unreachable but \"recent\" objects. This is to keep whole chunks of\n    the history graph intact during the gc grace period (which is 2\n    weeks by default). So object themselves _can_ be a source of\n    traversal for git-gc.\n\n    We do that traversal with the ignore_missing_links flag, so\n    breakages in the unreachable objects _shouldn't_ cause what you're\n    seeing. IIRC we did turn up a bug or two with ignore_missing_links.\n    The only one I could find was a3ba6bf10a (revision.c: ignore broken\n    tags with ignore_missing_links, 2017-05-20), which I think wouldn't\n    generate the output you're seeing.\n"},{"id":"332546","messageId":"3af0f8cc-09f3-bcf2-04c8-f076e0ddcea2@xiplink.com","threadId":"47197","inReplyTo":"20171114055306.3tfi726wzmkcfluk@sigill.intra.peff.net","subject":"Re: Recovering from gc errors","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2017-11-14T15:39:31Z","receivedAt":"2017-11-14T15:39:47Z","isPatch":false,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"(It turned out that this problem is related to worktrees.  CCing some \nworktree folks.)\n\nOn 2017-11-14 12:53 AM, Jeff King wrote:\n> On Mon, Nov 13, 2017 at 04:13:19PM -0500, Marc Branchaud wrote:\n> \n>> Various incantations of \"git show ... 9c355a7726e31\" only fail with the same\n>> error, so I can't determine much about the problematic commit. Luckily I'm\n>> not particularly concerned with losing objects, as I push any important\n>> progress to named refs in backup repos.\n> \n> Doing \"git show\" will require looking at the parent commit to produce\n> the diff. Probably \"git show -s\" would work. But in general for poking\n> at corruption, something bare-bones like \"git cat-file commit 9c355a77\"\n> is going to be your best bet.\n\nThanks, I'd forgotten about cat-file (show's -s did not work).\n\nOnly one or two of the bad commits could possibly belong in a submodule, \nso I don't think I'm seeing a worktree+submodule problem.\n\nThere are some definite \"rebase -i\" commits (e.g. \"fixup!\"), and a lot \nof what were probably cherry-picks.  I know I did these operations in a \nworktree (see below).\n\n>> But I would like to clean this up in my local repo so that gc stops failing.\n>> I tried simply removing this and other loose commits that trip up gc (i.e.\n>> the objects/9c/355a7726e31b3033b8e714cf7edb4f0a41d8d4 file -- there are 49\n>> such files, all of which are several months old), but now gc complains of a\n>> bad tree object:\n> \n> You can't generally fix corruption issues by deleting objects[1]. The\n> \"source\" that makes Git want to have these objects is the refs and\n> reflogs. So your best bet is to find which of those point to the\n> problematic objects and delete them.\n> \n> I'd start by seeing if the breakage is reachable from any refs:\n> \n>    git rev-list --objects --all >/dev/null\n\nThat command does succeed.\n\n> If that command succeeds, then all your refs are intact and the problem\n> is in the reflogs. You can try to figure out which, but I'd probably\n> just blow them all away:\n> \n>    rm -rf .git/logs\n\nUnfortunately, removing the logs directory does not fix \"git gc\".  So I \nrestored it.\n\nHowever I did find all of the bad SHAs in the HEAD logs of four of my \nworktrees.\n\nAll of those worktrees have directories in .git/worktrees/, but \"git \nworktree list\" does not show two of them.  \"git worktree prune -v\" \ndisplays and does nothing.  (I do not want to play with --expire, \nbecause I'd rather keep my other worktrees.)\n\nI removed all of those worktrees' directories from .git/worktrees/, and \nnow \"git gc\" succeeds.  I've also removed those worktrees' working \ndirectories, as I don't really need them anymore.\n\nThanks for your help!\n\nI'm willing to chalk this up to bugs in the early worktree code, unless \none of the CC'd worktree developers thinks otherwise.\n\nAn explicit \"git worktree delete\" command would be nice for manually \ncleaning things up.  It's easy to just delete the directory, but having \na \"delete\" command gives the user assurance that they're not missing \nsomething.\n\n\t\tM.\n\n> If the rev-list fails, then one or more branch is corrupted.\n> Unfortunately the usual efficient tools for asking \"which branch\n> contains this object\" are likely to be broken by the corruption. But you\n> can brute-force it, like:\n> \n>    git for-each-ref --format='%(refname)' |\n>    while read ref; do\n>      git rev-list --objects \"$ref\" >/dev/null 2>&1 ||\n>      echo \"$ref is broken\"\n>    done\n> \n> Hopefully that turns up only branches with little value, and you can\n> delete them:\n> \n>    git update-ref -d $broken_ref\n> \n> -Peff\n> \n> [1] A note on my \"you can't fix corruption by deleting objects\".\n> \n>      Since abcb86553d (pack-objects: match prune logic for discarding\n>      objects, 2014-10-15) , git-gc also traverses the history graph of\n>      unreachable but \"recent\" objects. This is to keep whole chunks of\n>      the history graph intact during the gc grace period (which is 2\n>      weeks by default). So object themselves _can_ be a source of\n>      traversal for git-gc.\n> \n>      We do that traversal with the ignore_missing_links flag, so\n>      breakages in the unreachable objects _shouldn't_ cause what you're\n>      seeing. IIRC we did turn up a bug or two with ignore_missing_links.\n>      The only one I could find was a3ba6bf10a (revision.c: ignore broken\n>      tags with ignore_missing_links, 2017-05-20), which I think wouldn't\n>      generate the output you're seeing.\n> \n"},{"id":"332553","messageId":"CAPig+cS8fg7UEX7eO4LLrVuuuBCxOVbhqmc5xFGqv3mT6s0=ng@mail.gmail.com","threadId":"47197","inReplyTo":"3af0f8cc-09f3-bcf2-04c8-f076e0ddcea2@xiplink.com","subject":"Re: Recovering from gc errors","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-14T16:48:07Z","receivedAt":"2017-11-14T16:48:27Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Nov 14, 2017 at 10:39 AM, Marc Branchaud <marcnarc@xiplink.com> wrote:\n> I'm willing to chalk this up to bugs in the early worktree code, unless one\n> of the CC'd worktree developers thinks otherwise.\n>\n> An explicit \"git worktree delete\" command would be nice for manually\n> cleaning things up.  It's easy to just delete the directory, but having a\n> \"delete\" command gives the user assurance that they're not missing\n> something.\n\nDuy does have a series in 'pu' which adds this functionality, but I\nguess it's stalled for the moment. From \"What's Cooking\":\n\n    * nd/worktree-move (2017-04-20) 6 commits\n    - worktree remove: new command\n    - worktree move: refuse to move worktrees with submodules\n    - worktree move: accept destination as directory\n    - worktree move: new command\n    - worktree.c: add update_worktree_location()\n    - worktree.c: add validate_worktree()\n\n    \"git worktree\" learned move and remove subcommands.\n\n    Expecting a reroll.\n    cf. <20170420101024.7593-1-pclouds@gmail.com>\n    cf. <20170421145916.mknekgqzhxffu7di@sigill.intra.peff.net>\n    cf. <d0e81b1e-5869-299e-f462-4d43dc997bd1@ramsayjones.plus.com>\n"},{"id":"332579","messageId":"20171115001907.vno7fo4t4bglh35n@sigill.intra.peff.net","threadId":"47197","inReplyTo":"3af0f8cc-09f3-bcf2-04c8-f076e0ddcea2@xiplink.com","subject":"Re: Recovering from gc errors","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-11-15T00:19:08Z","receivedAt":"2017-11-15T00:19:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 14, 2017 at 10:39:31AM -0500, Marc Branchaud wrote:\n\n> >    rm -rf .git/logs\n> \n> Unfortunately, removing the logs directory does not fix \"git gc\".  So I\n> restored it.\n> \n> However I did find all of the bad SHAs in the HEAD logs of four of my\n> worktrees.\n\nAh, right. There are more logs than .git/logs these days. That makes\nsense.\n\n> I'm willing to chalk this up to bugs in the early worktree code, unless one\n> of the CC'd worktree developers thinks otherwise.\n\nPerhaps fixed by acd9544a8f (revision.c: --reflog add HEAD reflog from\nall worktrees, 2017-08-23). I think before then that prune/repack would\nconsider those objects unreachable.\n\n-Peff\n"}]}