{"thread":{"id":"61289","subject":"[PATCH] midx: disable replace objects","startedAt":"2024-04-07T13:11:52Z","lastAt":"2024-04-18T16:06:41Z","messageCount":9,"participants":["blanet via GitGitGadget","Taylor Blau","鑫邢","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"492460","messageId":"pull.1711.git.1712495507815.gitgitgadget@gmail.com","threadId":"61289","inReplyTo":null,"subject":"[PATCH] midx: disable replace objects","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-07T13:11:47Z","receivedAt":"2024-04-07T13:11:52Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"From: Xing Xin <xingxin.xx@bytedance.com>\n\nWe observed a series of clone failures arose in a specific set of\nrepositories after we fully enabled the MIDX bitmap feature within our\nCodebase service. These failures were accompanied with error messages\nsuch as:\n\n  fatal: did not receive expected object ...\n  fatal: fetch-pack: invalid index-pack output\n\nTemporarily disabling the MIDX feature eliminated the reported issues.\nAfter some investigation we found that all repositories experiencing\nfailures contain replace references, which seem to be improperly\nacknowledged by the MIDX bitmap generation logic. During cloning or\nfetching, git-pack-objects, which may make use of MIDX bitmap to find\nobjects to pack, would give wrong objects even if we explicitly\nspecified not to enable replace refs by GIT_NO_REPLACE_OBJECTS=1.\nIndeed, this issue appears to have persisted since the introduction of\nMIDX.\n\nThis patch updates the MIDX logic to disable replace objects during\noperations, mirroring the handling seen in single pack index scenarios,\ni.e. git-index-pack and git-pack-objects. The added test uses\ngit-rev-list to give a more intuitive check.\n\nSigned-off-by: Xing Xin <xingxin.xx@bytedance.com>\n---\n    midx: disable replace objects\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1711%2Fblanet%2Fxx%2Fmidx-ignore-replace-objects-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1711/blanet/xx/midx-ignore-replace-objects-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1711\n\n builtin/multi-pack-index.c    |  3 +++\n t/t5326-multi-pack-bitmaps.sh | 24 ++++++++++++++++++++++++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex a72aebecaa2..8360932d2e7 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"trace2.h\"\n #include \"object-store-ll.h\"\n+#include \"replace-object.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\" \\\n@@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t};\n \tstruct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts);\n \n+\tdisable_replace_refs();\n+\n \tgit_config(git_default_config, NULL);\n \n \tif (the_repository &&\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 70d1b58709a..5e4cdef6a8b 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -434,6 +434,30 @@ test_expect_success 'tagged commits are selected for bitmapping' '\n \t)\n '\n \n+test_expect_success 'do not follow replace objects for MIDX bitmap' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\tA=$(git rev-parse HEAD) &&\n+\t\ttest_commit B &&\n+\t\tB=$(git rev-parse HEAD) &&\n+\t\tgit checkout --orphan=orphan $A &&\n+\t\ttest_commit orphan &&\n+\t\tC=$(git rev-parse HEAD) &&\n+\t\tgit rev-list --objects --no-object-names $B |sort >expected &&\n+\n+\t\tgit replace $A $C &&\n+\t\tgit repack -ad &&\n+\t\tgit multi-pack-index write --bitmap &&\n+\t\tgit rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n corrupt_file () {\n \tchmod a+w \"$1\" &&\n \tprintf \"bogus\" | dd of=\"$1\" bs=1 seek=\"12\" conv=notrunc\n\nbase-commit: 3c2a3fdc388747b9eaf4a4a4f2035c1c9ddb26d0\n-- \ngitgitgadget\n"},{"id":"492464","messageId":"ZhKqvA1NQwrVfnfE@nand.local","threadId":"61289","inReplyTo":"pull.1711.git.1712495507815.gitgitgadget@gmail.com","subject":"Re: [PATCH] midx: disable replace objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-07T14:16:28Z","receivedAt":"2024-04-07T14:16:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Apr 07, 2024 at 01:11:47PM +0000, blanet via GitGitGadget wrote:\n> From: Xing Xin <xingxin.xx@bytedance.com>\n>\n> We observed a series of clone failures arose in a specific set of\n> repositories after we fully enabled the MIDX bitmap feature within our\n> Codebase service. These failures were accompanied with error messages\n> such as:\n>\n>   fatal: did not receive expected object ...\n>   fatal: fetch-pack: invalid index-pack output\n>\n> Temporarily disabling the MIDX feature eliminated the reported issues.\n> After some investigation we found that all repositories experiencing\n> failures contain replace references, which seem to be improperly\n> acknowledged by the MIDX bitmap generation logic.\n\nI was suspicious that this might be related to the MIDX or MIDX bitmap,\nbut noticed something curious upon digging in. Applying the following on\ntop of your patch:\n\n--- 8< ---\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 5e4cdef6a8..8543f8d097 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -451,9 +451,7 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' '\n \t\tgit rev-list --objects --no-object-names $B |sort >expected &&\n\n \t\tgit replace $A $C &&\n-\t\tgit repack -ad &&\n-\t\tgit multi-pack-index write --bitmap &&\n-\t\tgit rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&\n+\t\tgit rev-list --objects --no-object-names $B |sort >actual &&\n \t\ttest_cmp expected actual\n \t)\n '\n--- >8 ---\n\n, I can still produce the failure that you are seeing here. So I suspect\nthat while it's entirely possible that there is a bug in the MIDX/bitmap\ncode, that this test is not exercising it.\n\nI think the first step to demonstrate a bug in the MIDX/bitmap machinery\nwould be to provide a reproducer that fails only when using a MIDX\nand/or bitmap.\n\n> @@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n>  \t};\n>  \tstruct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts);\n>\n> +\tdisable_replace_refs();\n> +\n\nSupposing for a moment that this issue is in the MIDX, we know that\nregardless of what replace refs might be in place, the MIDX should only\nbe storing the objects that are in the packs being indexed, not the\nobjects which are their replacements.\n\nAre we storing objects in the MIDX that are replacements? Looking\nat midx.c::fill_pack_entry(), I think the answer is \"no\", since we're\nlooking up packed objects by calling nth_packed_object_id(), which is\njust a table read into the .idx, all of which is beneath the level of\nreplace refs.\n\n> @@ -434,6 +434,30 @@ test_expect_success 'tagged commits are selected for bitmapping' '\n>  \t)\n>  '\n>\n> +test_expect_success 'do not follow replace objects for MIDX bitmap' '\n> +\trm -fr repo &&\n> +\tgit init repo &&\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit A &&\n> +\t\tA=$(git rev-parse HEAD) &&\n\nIt's possible that much of this will be moot if the current test gets\nrewritten, but here are a couple of suggestions for writing tests in\nGit's suite:\n\n- test_commit will create a tag for you, so there is no need to store\n  \"$A\", \"$B\", and \"$C\".\n\n> +\t\ttest_commit B &&\n> +\t\tB=$(git rev-parse HEAD) &&\n> +\t\tgit checkout --orphan=orphan $A &&\n> +\t\ttest_commit orphan &&\n> +\t\tC=$(git rev-parse HEAD) &&\n> +\t\tgit rev-list --objects --no-object-names $B |sort >expected &&\n\n- We do not allow Git invocations on the left-hand side of a pipe, since\n  doing so will squelch its exit code. Instead, try:\n\n    git rev-list --objects --no-object-names B >expect.raw &&\n    sort expect.raw >expect &&\n\nThanks,\nTaylor\n"},{"id":"492472","messageId":"ZhLfqU9VNUW+2mmV@nand.local","threadId":"61289","inReplyTo":"ZhKqvA1NQwrVfnfE@nand.local","subject":"Re: [PATCH] midx: disable replace objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-07T18:02:17Z","receivedAt":"2024-04-07T18:02:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Apr 07, 2024 at 10:16:28AM -0400, Taylor Blau wrote:\n> , I can still produce the failure that you are seeing here. So I suspect\n> that while it's entirely possible that there is a bug in the MIDX/bitmap\n> code, that this test is not exercising it.\n>\n> I think the first step to demonstrate a bug in the MIDX/bitmap machinery\n> would be to provide a reproducer that fails only when using a MIDX\n> and/or bitmap.\n\nI had a some more time to look into this, and I think that your original\nfix is correct.\n\nThe issue is, as you suggest, due to the following (from your original\npatch):\n\n> After some investigation we found that all repositories experiencing\n> failures contain replace references, which seem to be improperly\n> acknowledged by the MIDX bitmap generation logic.\n\nIndeed, the pack-bitmap-write machinery does not itself call\ndisable_replace_refs(). So when it generates a reachability bitmap, it\nis doing so with the replace refs in mind. You can see that this is\nindeed the cause of the problem by looking at the output of an\ninstrumented version of Git that indicates what bits are being set\nduring the bitmap generation phase.\n\nWith replace refs (incorrectly) enabled, we get:\n\n    [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]\n\nand doing the same after calling disable_replace_refs(), we instead get:\n\n    [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]\n\nSingle pack bitmaps are unaffected by this issue because we generate\nthem from within pack-objects, which does call disable_replace_refs().\n\nIt is tempting to instead do something like:\n\n--- 8< ---\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c6c8f94cc5..cbc543caad 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -17,6 +17,7 @@\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n+#include \"replace-object.h\"\n\n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -223,6 +224,8 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \tmemset(bb, 0, sizeof(*bb));\n \tinit_bb_data(&bb->data);\n\n+\tparsed_object_pool_clear(the_repository->parsed_objects);\n+\n \treset_revision_walk();\n \trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n \trevs.topo_order = 1;\n--- >8 ---\n\nBut by then it is too late, because the replace refs have already been\ntaken into account for parsed objects.\n\nAn alternative is to clear the parsed_object_pool before (or after)\ncalling disable_replace_refs(), but I think that approach that feels\nsub-optimal for a couple of reasons:\n\n  - We're wasting time re-parsing objects that we've already seen\n\n  - We're banking on the fact that the MIDX generation does not lookup\n    objects with the OBJECT_INFO_LOOKUP_REPLACE flag set, which would\n    cause the MIDX to be broken in the same way.\n\nSo I think that disabling replace refs at the outset within the\nmulti-pack-index builtin is the right way to go. In addition to the test\nfixes I suggested earlier, I would instead demonstrate the bug by\nshowing a clone (which fails with MIDXs, but doesn't without MIDXs) like\nso:\n\n--- 8< ---\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 5e4cdef6a8..1fb3b0f9d7 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -442,19 +442,16 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' '\n \t\tcd repo &&\n\n \t\ttest_commit A &&\n-\t\tA=$(git rev-parse HEAD) &&\n \t\ttest_commit B &&\n-\t\tB=$(git rev-parse HEAD) &&\n-\t\tgit checkout --orphan=orphan $A &&\n+\t\tgit checkout --orphan=orphan A &&\n \t\ttest_commit orphan &&\n-\t\tC=$(git rev-parse HEAD) &&\n-\t\tgit rev-list --objects --no-object-names $B |sort >expected &&\n\n-\t\tgit replace $A $C &&\n-\t\tgit repack -ad &&\n-\t\tgit multi-pack-index write --bitmap &&\n-\t\tgit rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&\n-\t\ttest_cmp expected actual\n+\t\tgit replace A HEAD &&\n+\t\tgit repack -ad --write-midx --write-bitmap-index &&\n+\n+\t\t# generating reachability bitmaps with replace refs\n+\t\t# enabled will result in broken clones\n+\t\tgit clone --no-local --bare . clone.git\n \t)\n '\n--- >8 ---\n\nWith the change in your patch to call disable_replace_refs() in\nbuiltin/multi-pack-index.c, this test passes as expected. With that\nchange compiled out, we instead get:\n\n[...]\n+ git clone --no-local --bare . clone.git\nCloning into bare repository 'clone.git'...\nremote: Enumerating objects: 8, done.\nremote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)\nReceiving objects: 100% (8/8), done.\nfatal: did not receive expected object da5497437fd67ca928333aab79c4b4b55036ea66\nfatal: fetch-pack: invalid index-pack output\nerror: last command exited with $?=128\nnot ok 352 - do not follow replace objects for MIDX bitmap\n\nas expected.\n\nThanks,\nTaylor\n\n"},{"id":"492473","messageId":"ZhLgLhv4LRcYrm7X@nand.local","threadId":"61289","inReplyTo":"ZhLfqU9VNUW+2mmV@nand.local","subject":"Re: [PATCH] midx: disable replace objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-07T18:04:30Z","receivedAt":"2024-04-07T18:04:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Apr 07, 2024 at 02:02:17PM -0400, Taylor Blau wrote:\n> It is tempting to instead do something like:\n>\n> --- 8< ---\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index c6c8f94cc5..cbc543caad 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -17,6 +17,7 @@\n>  #include \"trace2.h\"\n>  #include \"tree.h\"\n>  #include \"tree-walk.h\"\n> +#include \"replace-object.h\"\n>\n>  struct bitmapped_commit {\n>  \tstruct commit *commit;\n> @@ -223,6 +224,8 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n>  \tmemset(bb, 0, sizeof(*bb));\n>  \tinit_bb_data(&bb->data);\n>\n> +\tparsed_object_pool_clear(the_repository->parsed_objects);\n> +\n>  \treset_revision_walk();\n>  \trepo_init_revisions(writer->to_pack->repo, &revs, NULL);\n>  \trevs.topo_order = 1;\n> --- >8 ---\n\nOops. I meant to attach a diff here that calls disable_replace_refs(),\nnot parsed_object_pool_clear().\n\n> But by then it is too late, because the replace refs have already been\n> taken into account for parsed objects.\n\nThanks,\nTaylor\n"},{"id":"492484","messageId":"pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com","threadId":"61289","inReplyTo":"pull.1711.git.1712495507815.gitgitgadget@gmail.com","subject":"[PATCH v2] midx: disable replace objects","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-08T05:26:57Z","receivedAt":"2024-04-08T05:27:01Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"From: Xing Xin <xingxin.xx@bytedance.com>\n\nWe observed a series of clone failures arose in a specific set of\nrepositories after we fully enabled the MIDX bitmap feature within our\nCodebase service. These failures were accompanied with error messages\nsuch as:\n\n    Cloning into bare repository 'clone.git'...\n    remote: Enumerating objects: 8, done.\n    remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)\n    Receiving objects: 100% (8/8), done.\n    fatal: did not receive expected object ...\n    fatal: fetch-pack: invalid index-pack output\n\nTemporarily disabling the MIDX feature eliminated the reported issues.\nAfter some investigation we found that all repositories experiencing\nfailures contain replace references, which seem to be improperly\nacknowledged by the MIDX bitmap generation logic.\n\nA more thorough explanation about the root cause from Taylor Blau says:\n\nIndeed, the pack-bitmap-write machinery does not itself call\ndisable_replace_refs(). So when it generates a reachability bitmap, it\nis doing so with the replace refs in mind. You can see that this is\nindeed the cause of the problem by looking at the output of an\ninstrumented version of Git that indicates what bits are being set\nduring the bitmap generation phase.\n\nWith replace refs (incorrectly) enabled, we get:\n\n    [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]\n\nand doing the same after calling disable_replace_refs(), we instead get:\n\n    [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]\n\nSingle pack bitmaps are unaffected by this issue because we generate\nthem from within pack-objects, which does call disable_replace_refs().\n\nThis patch updates the MIDX logic to disable replace objects within the\nmulti-pack-index builtin, and a test showing a clone (which would fail\nwith MIDX bitmap) is added to demonstrate the bug.\n\nHelped-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Xing Xin <xingxin.xx@bytedance.com>\n---\n    midx: disable replace objects\n    \n    cc: Taylor Blau me@ttaylorr.com\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1711%2Fblanet%2Fxx%2Fmidx-ignore-replace-objects-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1711/blanet/xx/midx-ignore-replace-objects-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1711\n\nRange-diff vs v1:\n\n 1:  b1c838965ab ! 1:  1be25b55c5a midx: disable replace objects\n     @@ Commit message\n          Codebase service. These failures were accompanied with error messages\n          such as:\n      \n     -      fatal: did not receive expected object ...\n     -      fatal: fetch-pack: invalid index-pack output\n     +        Cloning into bare repository 'clone.git'...\n     +        remote: Enumerating objects: 8, done.\n     +        remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)\n     +        Receiving objects: 100% (8/8), done.\n     +        fatal: did not receive expected object ...\n     +        fatal: fetch-pack: invalid index-pack output\n      \n          Temporarily disabling the MIDX feature eliminated the reported issues.\n          After some investigation we found that all repositories experiencing\n          failures contain replace references, which seem to be improperly\n     -    acknowledged by the MIDX bitmap generation logic. During cloning or\n     -    fetching, git-pack-objects, which may make use of MIDX bitmap to find\n     -    objects to pack, would give wrong objects even if we explicitly\n     -    specified not to enable replace refs by GIT_NO_REPLACE_OBJECTS=1.\n     -    Indeed, this issue appears to have persisted since the introduction of\n     -    MIDX.\n     +    acknowledged by the MIDX bitmap generation logic.\n      \n     -    This patch updates the MIDX logic to disable replace objects during\n     -    operations, mirroring the handling seen in single pack index scenarios,\n     -    i.e. git-index-pack and git-pack-objects. The added test uses\n     -    git-rev-list to give a more intuitive check.\n     +    A more thorough explanation about the root cause from Taylor Blau says:\n      \n     +    Indeed, the pack-bitmap-write machinery does not itself call\n     +    disable_replace_refs(). So when it generates a reachability bitmap, it\n     +    is doing so with the replace refs in mind. You can see that this is\n     +    indeed the cause of the problem by looking at the output of an\n     +    instrumented version of Git that indicates what bits are being set\n     +    during the bitmap generation phase.\n     +\n     +    With replace refs (incorrectly) enabled, we get:\n     +\n     +        [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]\n     +\n     +    and doing the same after calling disable_replace_refs(), we instead get:\n     +\n     +        [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]\n     +\n     +    Single pack bitmaps are unaffected by this issue because we generate\n     +    them from within pack-objects, which does call disable_replace_refs().\n     +\n     +    This patch updates the MIDX logic to disable replace objects within the\n     +    multi-pack-index builtin, and a test showing a clone (which would fail\n     +    with MIDX bitmap) is added to demonstrate the bug.\n     +\n     +    Helped-by: Taylor Blau <me@ttaylorr.com>\n          Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>\n      \n       ## builtin/multi-pack-index.c ##\n     @@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'tagged commits are selected\n      +\t\tcd repo &&\n      +\n      +\t\ttest_commit A &&\n     -+\t\tA=$(git rev-parse HEAD) &&\n      +\t\ttest_commit B &&\n     -+\t\tB=$(git rev-parse HEAD) &&\n     -+\t\tgit checkout --orphan=orphan $A &&\n     ++\t\tgit checkout --orphan=orphan A &&\n      +\t\ttest_commit orphan &&\n     -+\t\tC=$(git rev-parse HEAD) &&\n     -+\t\tgit rev-list --objects --no-object-names $B |sort >expected &&\n      +\n     -+\t\tgit replace $A $C &&\n     -+\t\tgit repack -ad &&\n     -+\t\tgit multi-pack-index write --bitmap &&\n     -+\t\tgit rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&\n     -+\t\ttest_cmp expected actual\n     ++\t\tgit replace A HEAD &&\n     ++\t\tgit repack -ad --write-midx --write-bitmap-index &&\n     ++\n     ++\t\t# generating reachability bitmaps with replace refs\n     ++\t\t# enabled will result in broken clones\n     ++\t\tgit clone --no-local --bare . clone.git\n      +\t)\n      +'\n      +\n\n\n builtin/multi-pack-index.c    |  3 +++\n t/t5326-multi-pack-bitmaps.sh | 21 +++++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex a72aebecaa2..8360932d2e7 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"trace2.h\"\n #include \"object-store-ll.h\"\n+#include \"replace-object.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\" \\\n@@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t};\n \tstruct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts);\n \n+\tdisable_replace_refs();\n+\n \tgit_config(git_default_config, NULL);\n \n \tif (the_repository &&\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 70d1b58709a..1fb3b0f9d7a 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -434,6 +434,27 @@ test_expect_success 'tagged commits are selected for bitmapping' '\n \t)\n '\n \n+test_expect_success 'do not follow replace objects for MIDX bitmap' '\n+\trm -fr repo &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\tgit checkout --orphan=orphan A &&\n+\t\ttest_commit orphan &&\n+\n+\t\tgit replace A HEAD &&\n+\t\tgit repack -ad --write-midx --write-bitmap-index &&\n+\n+\t\t# generating reachability bitmaps with replace refs\n+\t\t# enabled will result in broken clones\n+\t\tgit clone --no-local --bare . clone.git\n+\t)\n+'\n+\n corrupt_file () {\n \tchmod a+w \"$1\" &&\n \tprintf \"bogus\" | dd of=\"$1\" bs=1 seek=\"12\" conv=notrunc\n\nbase-commit: 3c2a3fdc388747b9eaf4a4a4f2035c1c9ddb26d0\n-- \ngitgitgadget\n"},{"id":"492486","messageId":"CAER=4vi71KWKJUJF4Qg8eMax8D6xr4+pxTuE1_Pw6-yv40EOiw@mail.gmail.com","threadId":"61289","inReplyTo":"ZhLfqU9VNUW+2mmV@nand.local","subject":"Re: [External] Re: [PATCH] midx: disable replace objects","fromName":"鑫邢","fromEmail":"xingxin.xx@bytedance.com","sentAt":"2024-04-08T05:45:03Z","receivedAt":"2024-04-08T05:45:16Z","isPatch":true,"sender":{"key":"xingxin.xx@bytedance.com","avatar":null},"body":"> I had a some more time to look into this, and I think that your original\n> fix is correct.\n>\n> The issue is, as you suggest, due to the following (from your original\n> patch):\n>\n> > After some investigation we found that all repositories experiencing\n> > failures contain replace references, which seem to be improperly\n> > acknowledged by the MIDX bitmap generation logic.\n>\n> Indeed, the pack-bitmap-write machinery does not itself call\n> disable_replace_refs(). So when it generates a reachability bitmap, it\n> is doing so with the replace refs in mind. You can see that this is\n> indeed the cause of the problem by looking at the output of an\n> instrumented version of Git that indicates what bits are being set\n> during the bitmap generation phase.\n>\n> With replace refs (incorrectly) enabled, we get:\n>\n>     [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]\n>\n> and doing the same after calling disable_replace_refs(), we instead get:\n>\n>     [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]\n>\n> Single pack bitmaps are unaffected by this issue because we generate\n> them from within pack-objects, which does call disable_replace_refs().\n\nThank you for the comprehensive investigation. I have quoted them in the\ncommit message to provide a clearer explanation of the patch.\n\n> In addition to the test fixes I suggested earlier, I would instead demonstrate\n> the bug by showing a clone (which fails with MIDXs, but doesn't without\n> MIDXs) like so:\n>\n> --- 8< ---\n> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\n> index 5e4cdef6a8..1fb3b0f9d7 100755\n> --- a/t/t5326-multi-pack-bitmaps.sh\n> +++ b/t/t5326-multi-pack-bitmaps.sh\n> @@ -442,19 +442,16 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' '\n>                 cd repo &&\n>\n>                 test_commit A &&\n> -               A=$(git rev-parse HEAD) &&\n>                 test_commit B &&\n> -               B=$(git rev-parse HEAD) &&\n> -               git checkout --orphan=orphan $A &&\n> +               git checkout --orphan=orphan A &&\n>                 test_commit orphan &&\n> -               C=$(git rev-parse HEAD) &&\n> -               git rev-list --objects --no-object-names $B |sort >expected &&\n>\n> -               git replace $A $C &&\n> -               git repack -ad &&\n> -               git multi-pack-index write --bitmap &&\n> -               git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&\n> -               test_cmp expected actual\n> +               git replace A HEAD &&\n> +               git repack -ad --write-midx --write-bitmap-index &&\n> +\n> +               # generating reachability bitmaps with replace refs\n> +               # enabled will result in broken clones\n> +               git clone --no-local --bare . clone.git\n>         )\n>  '\n> --- >8 ---\n>\n> With the change in your patch to call disable_replace_refs() in\n> builtin/multi-pack-index.c, this test passes as expected. With that\n> change compiled out, we instead get:\n>\n> [...]\n> + git clone --no-local --bare . clone.git\n> Cloning into bare repository 'clone.git'...\n> remote: Enumerating objects: 8, done.\n> remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)\n> Receiving objects: 100% (8/8), done.\n> fatal: did not receive expected object da5497437fd67ca928333aab79c4b4b55036ea66\n> fatal: fetch-pack: invalid index-pack output\n> error: last command exited with $?=128\n> not ok 352 - do not follow replace objects for MIDX bitmap\n>\n> as expected.\n>\n\nApplied! The test looks much clearer now, thanks!\n\nXing Xin\n"},{"id":"493109","messageId":"xmqqplun6aws.fsf@gitster.g","threadId":"61289","inReplyTo":"pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] midx: disable replace objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-17T19:34:27Z","receivedAt":"2024-04-17T19:34:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"blanet via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Xing Xin <xingxin.xx@bytedance.com>\n> ...\n> Helped-by: Taylor Blau <me@ttaylorr.com>\n> Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>\n> ---\n\nI think this took the review in\n\n  https://lore.kernel.org/git/ZhLfqU9VNUW+2mmV@nand.local/\n\ninto account and is in good shape?\n\nThanks, both.\n\n"},{"id":"493138","messageId":"ZiEee8ySuy6kCvBZ@nand.local","threadId":"61289","inReplyTo":"xmqqplun6aws.fsf@gitster.g","subject":"Re: [PATCH v2] midx: disable replace objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-18T13:22:03Z","receivedAt":"2024-04-18T13:22:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Apr 17, 2024 at 12:34:27PM -0700, Junio C Hamano wrote:\n> \"blanet via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n> > From: Xing Xin <xingxin.xx@bytedance.com>\n> > ...\n> > Helped-by: Taylor Blau <me@ttaylorr.com>\n> > Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>\n> > ---\n>\n> I think this took the review in\n>\n>   https://lore.kernel.org/git/ZhLfqU9VNUW+2mmV@nand.local/\n>\n> into account and is in good shape?\n\nYes, sorry for not explicitly ack-ing, this version looks good to me.\n\nThanks,\nTaylor\n"},{"id":"493145","messageId":"xmqq5xwe4pvf.fsf@gitster.g","threadId":"61289","inReplyTo":"ZiEee8ySuy6kCvBZ@nand.local","subject":"Re: [PATCH v2] midx: disable replace objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-18T16:06:28Z","receivedAt":"2024-04-18T16:06:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> On Wed, Apr 17, 2024 at 12:34:27PM -0700, Junio C Hamano wrote:\n>> \"blanet via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>>\n>> > From: Xing Xin <xingxin.xx@bytedance.com>\n>> > ...\n>> > Helped-by: Taylor Blau <me@ttaylorr.com>\n>> > Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>\n>> > ---\n>>\n>> I think this took the review in\n>>\n>>   https://lore.kernel.org/git/ZhLfqU9VNUW+2mmV@nand.local/\n>>\n>> into account and is in good shape?\n>\n> Yes, sorry for not explicitly ack-ing, this version looks good to me.\n\nThanks.\n"}]}