{"thread":{"id":"45085","subject":"[PATCH 0/2] Demonstrate a critical worktree/gc bug","startedAt":"2017-02-08T12:18:24Z","lastAt":"2017-02-08T12:35:45Z","messageCount":4,"participants":["Johannes Schindelin","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"311053","messageId":"cover.1486556262.git.johannes.schindelin@gmx.de","threadId":"45085","inReplyTo":null,"subject":"[PATCH 0/2] Demonstrate a critical worktree/gc bug","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-08T12:17:52Z","receivedAt":"2017-02-08T12:18:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"... and a half-working workaround for the auto-gc case.\n\nThis patch series really is just another attempt to prod Duy into fixing\nthis instead of dabbling with shiny new toys ;-)\n\nChanges since \"v0\":\n\n- split out the test case, both to make it easier for Duy to integrate\n  it into the patch series that fixes the bug, as well as to provide a\n  little more, very gentle pressure to demonstrate the severity of this\n  problem.\n\n\nJohannes Schindelin (2):\n  worktree: demonstrate data lost to auto-gc\n  worktree: demonstrate a half-working workaround for auto-gc\n\n t/t6500-gc.sh | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 63 insertions(+)\n\n\nbase-commit: 6e3a7b3398559305c7a239a42e447c21a8f39ff8\nPublished-As: https://github.com/dscho/git/releases/tag/gc-worktree-v1\nFetch-It-Via: git fetch https://github.com/dscho/git gc-worktree-v1\n\n-- \n2.11.1.windows.1\n\n"},{"id":"311055","messageId":"674c16ee94c7609d82d14cac498ba62e5df65ab1.1486556262.git.johannes.schindelin@gmx.de","threadId":"45085","inReplyTo":"cover.1486556262.git.johannes.schindelin@gmx.de","subject":"[PATCH 1/2] worktree: demonstrate data lost to auto-gc","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-08T12:17:58Z","receivedAt":"2017-02-08T12:18:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"When gc --auto is called in the presence of worktrees, it fails to take\n*all* reflogs into account when trying to retain reachable objects, and as\na consequence data integrity goes pretty much to hell.\n\nThis patch provides a test case in the hopes that this bug gets fixed,\nafter an initial attempt has stalled for eight months already.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n t/t6500-gc.sh | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 1762dfa6a3..e24a4fb611 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -67,5 +67,44 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre\n \ttest_line_count = 2 new # There is one new pack and its .idx\n '\n \n+trigger_auto_gc () {\n+\t# This is unfortunately very, very ugly\n+\tgdir=\"$(git rev-parse --git-common-dir)\" &&\n+\tmkdir -p \"$gdir\"/objects/17 &&\n+\ttouch \"$gdir\"/objects/17/17171717171717171717171717171717171717 &&\n+\ttouch \"$gdir\"/objects/17/17171717171717171717171717171717171718 &&\n+\tgit -c gc.auto=1 -c gc.pruneexpire=now -c gc.autodetach=0 gc --auto\n+}\n+\n+test_expect_failure 'gc respects refs/reflogs in all worktrees' '\n+\ttest_commit something &&\n+\tgit worktree add worktree &&\n+\t(\n+\t\tcd worktree &&\n+\t\tgit checkout --detach &&\n+\t\t# avoid implicit tagging of test_commit\n+\t\techo 1 >something.t &&\n+\t\ttest_tick &&\n+\t\tgit commit -m worktree-reflog something.t &&\n+\t\tgit rev-parse --verify HEAD >../commit-reflog &&\n+\t\techo 2 >something.t &&\n+\t\ttest_tick &&\n+\t\tgit commit -m worktree-ref something.t &&\n+\t\tgit rev-parse --verify HEAD >../commit-ref\n+\t) &&\n+\ttrigger_auto_gc &&\n+\tgit rev-parse --verify $(cat commit-ref)^{commit} &&\n+\tgit rev-parse --verify $(cat commit-reflog)^{commit} &&\n+\n+\t# Now, add a reflog in the top-level dir and verify that `git gc` in\n+\t# the worktree does not purge that\n+\tgit checkout --detach &&\n+\techo 3 >something.t &&\n+\ttest_tick &&\n+\tgit commit -m commondir-reflog something.t &&\n+\tgit rev-parse --verify HEAD >commondir-reflog &&\n+\t(cd worktree && trigger_auto_gc) &&\n+\tgit rev-parse --verify $(cat commondir-reflog)^{commit}\n+'\n \n test_done\n-- \n2.11.1.windows.1\n\n\n"},{"id":"311056","messageId":"da25bfcebccdeb5ccdbf1f98027859f894fe5c25.1486556262.git.johannes.schindelin@gmx.de","threadId":"45085","inReplyTo":"cover.1486556262.git.johannes.schindelin@gmx.de","subject":"[PATCH 2/2] worktree: demonstrate a half-working workaround for auto-gc","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-08T12:18:04Z","receivedAt":"2017-02-08T12:18:33Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"While worktrees are marked \"experimental\", there is really no alternative\nto using them when trying to work on multiple patch series in parallel.\n\nSadly, there is still a bug that causes irretrievable data loss caused by\nworktree's incomplete handling of reflogs (and of the multiple indices of\nthe various worktrees), as this developer can testify a dozen times over.\n\nIn the --auto case, we can install a hook that runs before-hand,\naccumulates all the worktree-specific refs and reflogs and installs them\ninto a very special reflog in the common refs namespace. The only\npurpose of this stunt is to let gc pick up on those refs and reflogs, of\ncourse, and *not* ignore them.\n\nUnfortunately, this still does not address the \"git gc\" case, but\nhopefully a real fix will be here some time soon.\n\nAlso, if there are simply too many loose objects for Git's liking, it will\nkick off auto-gc (including the hook, which takes a couple of seconds to\nrun) that subsequently only says that it won't do anything, so the next\nGit operation will kick off another auto-gc (including the multi-second\nhook run).\n\nNeedless to say: this patch is not actually intended for inclusion, but\ninstead tries to demonstrate the pain and the dear need for a real bug fix\n(and not another 8 months of buggy Git).\n\nSo here is hoping to a timely fix. Cheers.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n t/t6500-gc.sh | 26 +++++++++++++++++++++++++-\n 1 file changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex e24a4fb611..2f1e52825d 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -67,6 +67,30 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre\n \ttest_line_count = 2 new # There is one new pack and its .idx\n '\n \n+test_expect_success 'install pre-auto-gc hook for worktrees' '\n+\tmkdir -p .git/hooks &&\n+\twrite_script .git/hooks/pre-auto-gc <<-\\EOF\n+\techo \"Preserving refs/reflogs of worktrees\" >&2 &&\n+\tdir=\"$(git rev-parse --git-common-dir)\" &&\n+\trefsdir=\"$dir/logs/refs\" &&\n+\trm -f \"$refsdir\"/preserve &&\n+\tident=\"$(GIT_COMMITTER_DATE= git var GIT_COMMITTER_IDENT)\" &&\n+\t(\n+\t\tfind \"$dir\"/logs \"$dir\"/worktrees/*/logs \\\n+\t\t\t-type f -exec cat {} \\; |\n+\t\tcut -d\" \" -f1\n+\t\tfind \"$dir\"/HEAD \"$dir\"/worktrees/*/HEAD \"$dir\"/refs \\\n+\t\t\t\"$dir\"/worktrees/*/refs -type f -exec cat {} \\; |\n+\t\tgrep -v \"^ref:\"\n+\t) 2>/dev/null |\n+\tsort |\n+\tuniq |\n+\tsed \"s/.*/& & $ident\tdummy/\" >\"$dir\"/preserve &&\n+\tmkdir -p \"$refsdir\" &&\n+\tmv \"$dir\"/preserve \"$refsdir\"/\n+\tEOF\n+'\n+\n trigger_auto_gc () {\n \t# This is unfortunately very, very ugly\n \tgdir=\"$(git rev-parse --git-common-dir)\" &&\n@@ -76,7 +100,7 @@ trigger_auto_gc () {\n \tgit -c gc.auto=1 -c gc.pruneexpire=now -c gc.autodetach=0 gc --auto\n }\n \n-test_expect_failure 'gc respects refs/reflogs in all worktrees' '\n+test_expect_success 'gc respects refs/reflogs in all worktrees' '\n \ttest_commit something &&\n \tgit worktree add worktree &&\n \t(\n-- \n2.11.1.windows.1\n"},{"id":"311058","messageId":"CACsJy8AmjkPbwRAykQM+aQosVwSamGwKpHox7FTPr6H7MgZQ7g@mail.gmail.com","threadId":"45085","inReplyTo":"cover.1486556262.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 0/2] Demonstrate a critical worktree/gc bug","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2017-02-08T12:28:13Z","receivedAt":"2017-02-08T12:35:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 8, 2017 at 7:17 PM, Johannes Schindelin\n<johannes.schindelin@gmx.de> wrote:\n> ... and a half-working workaround for the auto-gc case.\n>\n> This patch series really is just another attempt to prod Duy into fixing\n> this instead of dabbling with shiny new toys ;-)\n\nFYI work is ongoing [1] [2]. If you want it even faster, go do it yourself.\n\n[1] https://github.com/pclouds/git/commits/prune-in-worktrees-2\n[2] https://public-inbox.org/git/20170208113144.8201-1-pclouds@gmail.com/\n-- \nDuy\n"}]}