{"thread":{"id":"49074","subject":"Help with \"fatal: unable to read ....\" error during GC?","startedAt":"2018-08-08T15:22:46Z","lastAt":"2018-08-12T09:29:59Z","messageCount":13,"participants":["Paul Smith","Jeff King","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"354826","messageId":"1b2f649f0ece2ff46801c7bbd971c736e257af83.camel@mad-scientist.net","threadId":"49074","inReplyTo":null,"subject":"Help with \"fatal: unable to read ....\" error during GC?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2018-08-08T14:30:11Z","receivedAt":"2018-08-08T15:22:46Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"I recently upgraded from Git 2.9.2 to 2.18.0 (note, I have no\nparticular reason to believe this is related just passing info). I'm\nrunning on Linux (64bit Ubuntu 18.04.1 but I've compiled Git myself\nfrom source, I'm not using the distro version).\n\nI have a local repository I've been using for about two years (the\n.git/description file, which I don't use, has a TLM of July 31, 2016),\nwith lots of worktrees being created/pruned/etc. during that time.\n\nNote I'm doing all these operations in the 'main' repository, not in\nany of the worktrees.\n\nYesterday, when I tried to fetch from my upstream I got a notification\nabout GC needed. Then GC failed with these errors (HEAD is set to\nmaster which is the same as origin/master):\n\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n warning: reflog of 'HEAD' references pruned commits\n fatal: unable to read c104b8fb3631b5c54695206b2f73310c023c9963\n error: failed to run repack\n\nI ran a git fsck --full which showed me a lot of dangling commits and\nblobs, but no errors, no broken link messages, etc.\n\nI ran git reflog expire --all --stale-fix but no change.\n\nI can't find that SHA anywhere: I looked in .git/objects, etc. I also\ncan't find any problems with my repo; obviously I haven't checked\neverything but I can show the git log back to the initial commit, all\nmy stashes look fine, all my worktrees seem to be OK (git status etc.\nwork fine in all of them).\n\nBut whenever I pull etc. Git wants to run gc and I get this set of\nerrors again. FWIW other repos created from the same remote don't show\nany issues so it appears to be just this local copy of the repo.\n\nI've seen many SO and blog posts about issues like this but all were\nconcentrating on recovering things and I don't even know if I've lost\nanything... and anyway the operations they suggest don't work for me\nbecause nothing can access that SHA; I just get \"bad object\".\n\nAny ideas on what to look at next?\n\nI would hate to have to throw this setup away since it has 23 stashes\nand 25 worktrees in various states that would be annoying to have to\nrecreate... \n"},{"id":"354858","messageId":"20180808160612.GC1607@sigill.intra.peff.net","threadId":"49074","inReplyTo":"1b2f649f0ece2ff46801c7bbd971c736e257af83.camel@mad-scientist.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-08T16:06:12Z","receivedAt":"2018-08-08T16:06:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 08, 2018 at 10:30:11AM -0400, Paul Smith wrote:\n\n> I recently upgraded from Git 2.9.2 to 2.18.0 (note, I have no\n> particular reason to believe this is related just passing info). I'm\n> running on Linux (64bit Ubuntu 18.04.1 but I've compiled Git myself\n> from source, I'm not using the distro version).\n> \n> I have a local repository I've been using for about two years (the\n> .git/description file, which I don't use, has a TLM of July 31, 2016),\n> with lots of worktrees being created/pruned/etc. during that time.\n> \n> Note I'm doing all these operations in the 'main' repository, not in\n> any of the worktrees.\n\nHrm, there was a pretty serious corruption bug in early versions of the\nworktree code (IIRC, pruning would not consider detached HEADs from\nother worktrees, and could drop that object).\n\n> Yesterday, when I tried to fetch from my upstream I got a notification\n> about GC needed. Then GC failed with these errors (HEAD is set to\n> master which is the same as origin/master):\n> \n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> fatal: unable to read c104b8fb3631b5c54695206b2f73310c023c9963\n> error: failed to run repack\n\nSo that definitely looks like the corruption I'd expect from the\nworktree bug, but...\n\n> I ran a git fsck --full which showed me a lot of dangling commits and\n> blobs, but no errors, no broken link messages, etc.\n\nI'd have expected fsck to find it, too. However, looking at the code,\nI'm not convinced that fsck is actually considering detached worktree\nheads properly, either. Try:\n\n git rev-list --all --reflog --objects >/dev/null\n\nwhich I know checks worktrees correctly. I'd expect that to fail.\n\nIf it does, then we need to narrow down which worktree is corrupt.\nPerhaps something like:\n\n git worktree list |\n while read worktree head junk; do\n\tgit rev-list --objects $head >/dev/null ||\n\techo \"$worktree seems corrupt\"\n done\n\n> I can't find that SHA anywhere: I looked in .git/objects, etc. I also\n> can't find any problems with my repo; obviously I haven't checked\n> everything but I can show the git log back to the initial commit, all\n> my stashes look fine, all my worktrees seem to be OK (git status etc.\n> work fine in all of them).\n\n\"git status\" might succeed if the corruption is further back in the\nhistory.\n\n> I would hate to have to throw this setup away since it has 23 stashes\n> and 25 worktrees in various states that would be annoying to have to\n> recreate... \n\nDefinitely don't throw it away. I suspect you have a single corrupt\nworktree, and everything else is fine.\n\n-Peff\n"},{"id":"354874","messageId":"b247434b62ccd30f32adbebb83fa6ea12b51b6ff.camel@mad-scientist.net","threadId":"49074","inReplyTo":"20180808160612.GC1607@sigill.intra.peff.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2018-08-08T17:35:30Z","receivedAt":"2018-08-08T17:56:04Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Wed, 2018-08-08 at 12:06 -0400, Jeff King wrote:\n> I'd have expected fsck to find it, too. However, looking at the code,\n> I'm not convinced that fsck is actually considering detached worktree\n> heads properly, either. Try:\n> \n> git rev-list --all --reflog --objects >/dev/null\n> \n> which I know checks worktrees correctly. I'd expect that to fail.\n> \n> If it does, then we need to narrow down which worktree is corrupt.\n> Perhaps something like:\n> \n> git worktree list |\n> while read worktree head junk; do\n> git rev-list --objects $head >/dev/null ||\n> echo \"$worktree seems corrupt\"\n> done\n\nThanks for the note! Unhappily for me none of these operations seem to\nfind any actionable problems...\n\n$ git rev-list --all --reflog --objects >/dev/null\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\n$ echo $?\n0\n\n$ git worktree list | while read wt head junk; do \\\n git rev-list --objects $head >/dev/null || echo \"$wt seems corrupt\"; \\\n done\n$\n\nJust to be sure I updated the loop above to echo $wt and $head and they\nwere correct. I also re-ran git gc after the above and still got the\noriginal error output so it didn't magically fix itself :).\n"},{"id":"354877","messageId":"20180808182436.GA19096@sigill.intra.peff.net","threadId":"49074","inReplyTo":"b247434b62ccd30f32adbebb83fa6ea12b51b6ff.camel@mad-scientist.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-08T18:24:36Z","receivedAt":"2018-08-08T18:24:39Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 08, 2018 at 01:35:30PM -0400, Paul Smith wrote:\n\n> Thanks for the note! Unhappily for me none of these operations seem to\n> find any actionable problems...\n> [...]\n\nDrat.\n\nOne other option is that it _could_ be related to the \"old unreachable\nobjects that are reachable from recent unreachable objects should be\nkept\" code. That's supposed to quietly ignore broken links in\nunreachable objects, but there could be a bug.\n\nLet's narrow it down first and make sure we're dying where I expect. Can\nyou try:\n\n GIT_TRACE=1 git gc\n\nand confirm the program running when the fatal error is produced?\n\nFrom what you've shown it's going to be git-repack, but what I'm not\nclear on is whether it is repack itself that is complaining, or the\npack-objects process it spawns. I'd guess the latter.\n\nIf so, can you try running it under gdb and getting a stack trace?\nSomething like:\n\n gdb git\n [and then inside gdb...]\n set args pack-objects --all --reflog --indexed-objects foo Let's narrow it down first and make sure we're dying where I expect.\n> Can\n> you try:\n> \n> GIT_TRACE=1 git gc\n> \n> and confirm the program running when the fatal error is produced?\n> \n> From what you've shown it's going to be git-repack, but what I'm not\n> clear on is whether it is repack itself that is complaining, or the\n> pack-objects process it spawns. I'd guess the latter.\n\nYou are correct:\n\n15:27:24.264161 git.c:415 trace: built-in: git pack-\nobjects --keep-true-parents --honor-pack-keep --non-empty --all --\nreflog --indexed-objects --unpack-unreachable=2.weeks.ago --local --\ndelta-base-offset .git/objects/pack/.tmp-17617-pack\n\n> If so, can you try running it under gdb and getting a stack trace?\n\nI would... but I discovered all my Git binaries are stripped to the max\nand no symbols available.\n\nI'll do a quick rebuild with some debug info and get back to you.\n\nThanks for the pointers!\n"},{"id":"354966","messageId":"249be5d3dada9a4b1b5282896a9a11e12c1ffd2a.camel@mad-scientist.net","threadId":"49074","inReplyTo":"20180808182436.GA19096@sigill.intra.peff.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2018-08-09T02:45:49Z","receivedAt":"2018-08-09T03:32:09Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Wed, 2018-08-08 at 14:24 -0400, Jeff King wrote:\n> If so, can you try running it under gdb and getting a stack trace?\n> Something like:\n> \n> gdb git\n> [and then inside gdb...]\n> set args pack-objects --all --reflog --indexed-objects foo break die\n> run\n> bt\n> \n> That might give us a clue where the broken object reference is coming\n\nHere we go. I can rebuild with -Og or -O0 if more detailed debugging\nis needed; most everything appears to be optimized out:\n\n ...\nCompressing objects: 100% (107777/107777), done.\nWriting objects: 54% (274416/508176) \nThread 1 \"git\" hit Breakpoint 1, die (err=err@entry=0x5a373a \"unable to read %s\") at usage.c:119\n119 {\n(gdb) bt\n#0 die (err=err@entry=0x5a373a \"unable to read %s\") at usage.c:119\n#1 0x00000000004563f3 in get_delta (entry=) at builtin/pack-objects.c:143\n#2 write_no_reuse_object () at builtin/pack-objects.c:308\n#3 0x0000000000456592 in write_reuse_object (usable_delta=, limit=, entry=, f=) at builtin/pack-objects.c:516\n#4 write_object (write_offset=, entry=0x7fffc9a8d940, f=0x198fb70) at builtin/pack-objects.c:518\n#5 write_one () at builtin/pack-objects.c:576\n#6 0x00000000004592f0 in write_pack_file () at builtin/pack-objects.c:849\n#7 cmd_pack_objects (argc=, argv=, prefix=) at builtin/pack-objects.c:3354\n#8 0x0000000000404f06 in run_builtin (argv=, argc=, p=) at git.c:417\n#9 handle_builtin (argc=, argv=) at git.c:632\n#10 0x0000000000405f21 in run_argv (argv=0x7fffffffe210, argcp=0x7fffffffe21c) at git.c:761\n#11 cmd_main (argc=, argc@entry=6, argv=, argv@entry=0x7fffffffe448) at git.c:761\n#12 0x0000000000404b15 in main (argc=6, argv=0x7fffffffe448) at common-main.c:45\n"},{"id":"355010","messageId":"20180809170609.GE1439@sigill.intra.peff.net","threadId":"49074","inReplyTo":"249be5d3dada9a4b1b5282896a9a11e12c1ffd2a.camel@mad-scientist.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-09T17:06:10Z","receivedAt":"2018-08-09T17:06:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 08, 2018 at 10:45:49PM -0400, Paul Smith wrote:\n\n> On Wed, 2018-08-08 at 14:24 -0400, Jeff King wrote:\n> > If so, can you try running it under gdb and getting a stack trace?\n> > Something like:\n> > \n> > gdb git\n> > [and then inside gdb...]\n> > set args pack-objects --all --reflog --indexed-objects foo > break die\n> > run\n> > bt\n> > \n> > That might give us a clue where the broken object reference is coming\n> \n> Here we go. I can rebuild with -Og or -O0 if more detailed debugging\n> is needed; most everything appears to be optimized out:\n\nNo, I think this is enough to give a general sense of the problem\nlocation.\n\n> Compressing objects: 100% (107777/107777), done.\n> Writing objects: 54% (274416/508176) \n> Thread 1 \"git\" hit Breakpoint 1, die (err=err@entry=0x5a373a \"unable to read %s\") at usage.c:119\n> 119 {\n> (gdb) bt\n> #0 die (err=err@entry=0x5a373a \"unable to read %s\") at usage.c:119\n> #1 0x00000000004563f3 in get_delta (entry=) at builtin/pack-objects.c:143\n> #2 write_no_reuse_object () at builtin/pack-objects.c:308\n> #3 0x0000000000456592 in write_reuse_object (usable_delta=, limit=, entry=, f=) at builtin/pack-objects.c:516\n> #4 write_object (write_offset=, entry=0x7fffc9a8d940, f=0x198fb70) at builtin/pack-objects.c:518\n> #5 write_one () at builtin/pack-objects.c:576\n> #6 0x00000000004592f0 in write_pack_file () at builtin/pack-objects.c:849\n> #7 cmd_pack_objects (argc=, argv=, prefix=) at builtin/pack-objects.c:3354\n> #8 0x0000000000404f06 in run_builtin (argv=, argc=, p=) at git.c:417\n> #9 handle_builtin (argc=, argv=) at git.c:632\n> #10 0x0000000000405f21 in run_argv (argv=0x7fffffffe210, argcp=0x7fffffffe21c) at git.c:761\n> #11 cmd_main (argc=, argc@entry=6, argv=, argv@entry=0x7fffffffe448) at git.c:761\n> #12 0x0000000000404b15 in main (argc=6, argv=0x7fffffffe448) at common-main.c:45\n\nSo that's quite unexpected. I assumed we'd have hit this problem while\ndeciding _which_ objects to write. But we get all the way to the point\nof writing out the result before we notice it's missing.\n\nI don't think I've run such a case before, but I wonder if \"pack-objects\n--all\" is too lax about adding missing blobs during its object traversal\n(especially during the \"unreachable but recent\" part of the traversal\nthat I mentioned, which should silently omit missing objects). I played\naround with recreating this situation, though, and I don't think it's\npossible to cause the results you're seeing. We come up with a list of\nrecent objects, but we only use it as a look-up index for discarding\ntoo-old objects. So:\n\n - it wouldn't ever cause us to choose to write an object into a pack,\n which is what you're seeing\n\n - we'd never consider a missing object; it's a pure lookup table, and\n the actual list of objects we consider is found by walking the set\n of packs\n\nSo that's probably a dead end.\n\nWhat I really wonder is where we found out about that object name in the\nfirst place. Can you instrument your Git build like this:\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 71056d8294..5ff6de5ddf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1112,6 +1112,13 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tstruct packed_git *found_pack = NULL;\n \toff_t found_offset = 0;\n \tuint32_t index_pos;\n+\tstatic const struct object_id funny_oid = {\n+\t\t\"\\xc1\\x04\\xb8\\xfb\\x36\\x31\\xb5\\xc5\\x46\\x95\"\n+\t\t\"\\x20\\x6b\\x2f\\x73\\x31\\x0c\\x02\\x3c\\x99\\x63\"\n+\t};\n+\n+\tif (!oidcmp(oid, &funny_oid))\n+\t\twarning(\"found funny oid\");\n \n \tdisplay_progress(progress_state, ++nr_seen);\n \n\nand similarly get a backtrace when we hit that warning()? (Or if you're\na gdb expert, you could probably use a conditional breakpoint, but I\nfind just modifying the source easier).\n\n-Peff\n"},{"id":"355245","messageId":"be46349efde84f158b80e96f2fbbcf4304a71208.camel@mad-scientist.net","threadId":"49074","inReplyTo":"20180808182436.GA19096@sigill.intra.peff.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2018-08-11T12:13:17Z","receivedAt":"2018-08-11T12:13:26Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Wed, 2018-08-08 at 14:24 -0400, Jeff King wrote:\n> If so, can you try running it under gdb and getting a stack trace?\n> Something like:\n> \n> gdb git\n> [and then inside gdb...]\n> set args pack-objects --all --reflog --indexed-objects foo break die\n> run\n> bt\n> \n> That might give us a clue where the broken object reference is coming\n> from.\n\nOh no. I messed up :(.\n\nI rebuilt Git 2.18.0 without optimization to try to get more debug\ninformation. Unfortunately I didn't think to create a backup of my\nproblematic .git directory.\n\nWhen I ran the above command under the debugger using the non-optimized \nversion of Git... it worked! That fixed the problem so that now when I\nrun \"git gc\" using the original optimized version I no longer see the\nissue there either.\n\nSo... clearly something is wrong but because I was dumb and didn't make\na backup I can no longer reproduce the problem :(. On the other hand,\nmy repository is no longer throwing errors so that's good.\n\nI do still have these warnings and no amount of git gc/git fsck/etc.\nhas reduced them in any way:\n\n$ git gc\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nEnumerating objects: 506556, done.\nCounting objects: 100% (506556/506556), done.\nDelta compression using up to 8 threads.\nCompressing objects: 100% (101199/101199), done.\nWriting objects: 100% (506556/506556), done.\nTotal 506556 (delta 358957), reused 506556 (delta 358957)\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nwarning: reflog of 'HEAD' references pruned commits\nChecking connectivity: 506556, done.\n\nI've run git gc --prune=all then git fsck reports only these dangling\ncommits:\n\ndangling commit cef0678a5e0765506e3fac41286696fd37a9b1e9\ndangling commit 1729195f021a1b95ea8ca10b9c32e76bf2257e67\ndangling commit 08385b9731291607a8c6d4bf10272002d8f31e1f\ndangling commit c4ddfb2139eeb5a3c132dbfc84cc6e27fdeb46d1\ndangling commit 1df8ebcc1cd5f59dd224ce1f3ba39f24370cf4e7\n\n(this is down from probably 50 or so \"dangling ...\" commits, blobs, and\ntrees before).\n"},{"id":"355248","messageId":"20180811142341.GA17605@sigill.intra.peff.net","threadId":"49074","inReplyTo":"be46349efde84f158b80e96f2fbbcf4304a71208.camel@mad-scientist.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-11T14:23:41Z","receivedAt":"2018-08-11T14:23:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 11, 2018 at 08:13:17AM -0400, Paul Smith wrote:\n\n> I rebuilt Git 2.18.0 without optimization to try to get more debug\n> information. Unfortunately I didn't think to create a backup of my\n> problematic .git directory.\n> \n> When I ran the above command under the debugger using the non-optimized \n> version of Git... it worked! That fixed the problem so that now when I\n> run \"git gc\" using the original optimized version I no longer see the\n> issue there either.\n> \n> So... clearly something is wrong but because I was dumb and didn't make\n> a backup I can no longer reproduce the problem :(. On the other hand,\n> my repository is no longer throwing errors so that's good.\n\nHeh. Well, I would like to have known what the problem was. But if it\nnever happens again, we can go on with our lives. And if it does, then\nwe'll have another reproduction. :)\n\nThe fact that disabling optimizations changed things is worrisome. It\nmakes me wonder if this is somehow related to the struct-packing changes\nin pack-objects. I don't know of any problems there, but in some\nmodifications to them post-v2.18, we had to deal with some race\nconditions.\n\nOne alternative theory is that it wasn't the optimizations at all, but\nrather the clock moving forward. The repack process cares about the\ncurrent time with respect to the mtime of the unreachable objects on\ndisk. It's possible that between yesterday and today, some objects\ncrossed the line to \"too old to keep\" (though from the earlier digging,\nI'm not sure this is related to unreachable objects at all).\n\nThanks for your patience in digging into this, and please let us know if\nyou run into similar problems again.\n\n> I do still have these warnings and no amount of git gc/git fsck/etc.\n> has reduced them in any way:\n> \n> $ git gc\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n> warning: reflog of 'HEAD' references pruned commits\n\nI think these would go away via \"reflog expire\" (I'd have thought \"git\ngc\" would do so, though). I wonder if this is yet another tool that\nneeds to be taught about worktree heads.\n\n> I've run git gc --prune=all then git fsck reports only these dangling\n> commits:\n> \n> dangling commit cef0678a5e0765506e3fac41286696fd37a9b1e9\n> dangling commit 1729195f021a1b95ea8ca10b9c32e76bf2257e67\n> dangling commit 08385b9731291607a8c6d4bf10272002d8f31e1f\n> dangling commit c4ddfb2139eeb5a3c132dbfc84cc6e27fdeb46d1\n> dangling commit 1df8ebcc1cd5f59dd224ce1f3ba39f24370cf4e7\n> \n> (this is down from probably 50 or so \"dangling ...\" commits, blobs, and\n> trees before).\n\nI'd also expect \"--prune=all\" to drop all dangling heads. But I think\nthis is the worktree thing, again. The code in fsck starts it\nconnectivity check with this:\n\n if (head_points_at && !is_null_oid(&head_oid))\n fsck_handle_ref(\"HEAD\", &head_oid, 0, NULL);\n for_each_rawref(fsck_handle_ref, NULL);\n if (include_reflogs)\n for_each_reflog(fsck_handle_reflog, NULL);\n\nbut looking at the similar code in revision.c that has been upgraded to\nhandle worktrees (e.g., add_reflogs_to_pending()), I think that is not\ngoing to look at worktree HEADs nor reflogs.\n\nI'd hoped to give you a one-liner to try out, but I think it will\nrequire some refactoring.\n\n-Peff\n"},{"id":"355249","messageId":"20180811142527.GB17605@sigill.intra.peff.net","threadId":"49074","inReplyTo":"20180811142341.GA17605@sigill.intra.peff.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-11T14:25:28Z","receivedAt":"2018-08-11T14:27:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 11, 2018 at 10:23:41AM -0400, Jeff King wrote:\n\n> > I do still have these warnings and no amount of git gc/git fsck/etc.\n> > has reduced them in any way:\n> > \n> > $ git gc\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> \n> I think these would go away via \"reflog expire\" (I'd have thought \"git\n> gc\" would do so, though). I wonder if this is yet another tool that\n> needs to be taught about worktree heads.\n> \n> > I've run git gc --prune=all then git fsck reports only these dangling\n> > commits:\n> > \n> > dangling commit cef0678a5e0765506e3fac41286696fd37a9b1e9\n> > dangling commit 1729195f021a1b95ea8ca10b9c32e76bf2257e67\n> > dangling commit 08385b9731291607a8c6d4bf10272002d8f31e1f\n> > dangling commit c4ddfb2139eeb5a3c132dbfc84cc6e27fdeb46d1\n> > dangling commit 1df8ebcc1cd5f59dd224ce1f3ba39f24370cf4e7\n> > \n> > (this is down from probably 50 or so \"dangling ...\" commits, blobs, and\n> > trees before).\n> \n> I'd also expect \"--prune=all\" to drop all dangling heads. But I think\n> this is the worktree thing, again. The code in fsck starts it\n> connectivity check with this:\n> \n> if (head_points_at && !is_null_oid(&head_oid))\n> fsck_handle_ref(\"HEAD\", &head_oid, 0, NULL);\n> for_each_rawref(fsck_handle_ref, NULL);\n> if (include_reflogs)\n> for_each_reflog(fsck_handle_reflog, NULL);\n> \n> but looking at the similar code in revision.c that has been upgraded to\n> handle worktrees (e.g., add_reflogs_to_pending()), I think that is not\n> going to look at worktree HEADs nor reflogs.\n> \n> I'd hoped to give you a one-liner to try out, but I think it will\n> require some refactoring.\n\nResponding myself and adding Duy to the cc to increase visibility among\nworktree experts. :)\n\n-Peff\n"},{"id":"355250","messageId":"CACsJy8DE+V3GK0f3fDAtjseLYU_5Ct3V0RHxs3N=aNmA0hO3cw@mail.gmail.com","threadId":"49074","inReplyTo":"20180811142527.GB17605@sigill.intra.peff.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-08-11T14:38:00Z","receivedAt":"2018-08-11T14:38:28Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Aug 11, 2018 at 4:25 PM Jeff King wrote:\n> Responding myself and adding Duy to the cc to increase visibility among\n> worktree experts. :)\n\nI do silently watch this thread (and yes I still have to fix that fsck\nthing, hit a roadblock with ref names but I should really restart it\nsoon). Now you have found one more thing for me to do. Why Jeff why?\nj/k\n-- \nDuy\n"},{"id":"355253","messageId":"20180811163954.GA27393@sigill.intra.peff.net","threadId":"49074","inReplyTo":"CACsJy8DE+V3GK0f3fDAtjseLYU_5Ct3V0RHxs3N=aNmA0hO3cw@mail.gmail.com","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-08-11T16:39:55Z","receivedAt":"2018-08-11T16:39:58Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 11, 2018 at 04:38:00PM +0200, Duy Nguyen wrote:\n\n> On Sat, Aug 11, 2018 at 4:25 PM Jeff King wrote:\n> > Responding myself and adding Duy to the cc to increase visibility among\n> > worktree experts. :)\n> \n> I do silently watch this thread (and yes I still have to fix that fsck\n> thing, hit a roadblock with ref names but I should really restart it\n> soon). Now you have found one more thing for me to do. Why Jeff why?\n> j/k\n\n:) I was actually thinking about doing it myself, but was worried that\nthe refactoring might complicate things. And it sounds from the fact\nthat you looked into it and hit a roadblock that it is more complicated\nthan I thought.\n\nSo I'll leave it for now, but I'm happy to review or discuss ideas.\n\n-Peff\n"},{"id":"355298","messageId":"CACsJy8DyDpgGJ4FS=KrJpchpHpUFsfUt1zaQeoWYthAiBtSqpA@mail.gmail.com","threadId":"49074","inReplyTo":"20180811142341.GA17605@sigill.intra.peff.net","subject":"Re: Help with \"fatal: unable to read ....\" error during GC?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-08-12T09:29:31Z","receivedAt":"2018-08-12T09:29:59Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Aug 11, 2018 at 4:25 PM Jeff King wrote:\n> > I do still have these warnings and no amount of git gc/git fsck/etc.\n> > has reduced them in any way:\n> >\n> > $ git gc\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n> > warning: reflog of 'HEAD' references pruned commits\n>\n> I think these would go away via \"reflog expire\" (I'd have thought \"git\n> gc\" would do so, though). I wonder if this is yet another tool that\n> needs to be taught about worktree heads.\n\nYou would need \"reflog expire --expire-unreachable=now\" because the\ndefault 30 days are probably too long for this case. And yes \"reflog\nexpire --all\" needs to be aware of other heads (and other per-worktree\nrefs in general). I'm pretty sure right now it only cares about the\ncurrent worktree's head.\n-- \nDuy\n"}]}