{"thread":{"id":"61479","subject":"[PATCH] bundle-uri: refresh packed_git if unbundle succeed","startedAt":"2024-05-15T03:01:13Z","lastAt":"2024-06-19T06:31:16Z","messageCount":66,"participants":["blanet via GitGitGadget","Patrick Steinhardt","Karthik Nayak","Junio C Hamano","Xing Xin","Xing Xin via GitGitGadget"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"494752","messageId":"pull.1730.git.1715742069966.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":null,"subject":"[PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-15T03:01:09Z","receivedAt":"2024-05-15T03:01:13Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not being discovered, resulting in rather slow\nclones. This was particularly problematic when employing the heuristic\n`creationTokens`.\n\nAnd this is easy to reproduce. Suppose we have a repository with a\nsingle branch `main` pointing to commit `A`, firstly we create a base\nbundle with\n\n git bundle create base.bundle main\n\nThen let's add a new commit `B` on top of `A`, so that an incremental\nbundle for `main` can be created with\n\n git bundle create incr.bundle A..main\n\nNow we can generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above would give the expected\n`refs/bundles/main` pointing at `B` in new repository, in other words we\nalready had everything locally from the bundles, but git would still\ndownload everything from server as if we got nothing.\n\nSo why the `refs/bundles/main` is not discovered? After some digging I\nfound that:\n\n1. when unbundling a downloaded bundle, a `verify_bundle` is called to\n check its prerequisites if any. The verify procedure would find oids\n so `packed_git` is initialized.\n\n2. after unbundled all bundles, we would enter `do_fetch_pack_v2`,\n during which `mark_complete_and_common_ref` and `mark_tips` would\n find oids with `OBJECT_INFO_QUICK` flag set, so no new packs would be\n enlisted if `packed_git` has already initialized in 1.\n\nBack to the example above, when unbunding `incr.bundle`, `base.pack` is\nenlisted to `packed_git` bacause of the prerequisites to verify. Then we\ncan not find `B` for negotiation at a latter time bacause `B` exists in\n`incr.pack` which is not enlisted in `packed_git`.\n\nThis commit fixes this by adding a `reprepare_packed_git` call for every\nsuccessfully unbundled bundle, which ensures to enlist all generated\npacks from bundle uri. And a set of negotiation related tests are added.\n\nSigned-off-by: Xing Xin \n---\n bundle-uri: refresh packed_git if unbundle succeed\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\n bundle-uri.c | 3 +\n t/t5558-clone-bundle-uri.sh | 129 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 132 insertions(+)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex ca32050a78f..2b9d36cfd8e 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -7,6 +7,7 @@\n #include \"refs.h\"\n #include \"run-command.h\"\n #include \"hashmap.h\"\n+#include \"packfile.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n #include \"remote.h\"\n@@ -376,6 +377,8 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\t\t VERIFY_BUNDLE_QUIET)))\n \t\treturn 1;\n \n+\treprepare_packed_git(r);\n+\n \t/*\n \t * Convert all refs/heads/ from the bundle into refs/bundles/\n \t * in the local repository.\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..a5b04d6f187 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -20,7 +20,10 @@ test_expect_success 'fail to clone from non-bundle file' '\n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n \tgit -C clone-from checkout -b topic &&\n+\n \ttest_commit -C clone-from A &&\n+\tgit -C clone-from bundle create A.bundle topic &&\n+\n \ttest_commit -C clone-from B &&\n \tgit -C clone-from bundle create B.bundle topic\n '\n@@ -259,6 +262,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+#########################################################################\n+# Clone negotiation related tests begin here\n+\n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n\nbase-commit: 83f1add914c6b4682de1e944ec0d1ac043d53d78\n-- \ngitgitgadget\n"},{"id":"494923","messageId":"ZkbkgT46mdNEhcf0@tanuki","threadId":"61479","inReplyTo":"pull.1730.git.1715742069966.gitgitgadget@gmail.com","subject":"Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-17T05:00:49Z","receivedAt":"2024-05-17T05:00:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, May 15, 2024 at 03:01:09AM +0000, blanet via GitGitGadget wrote:\n> From: Xing Xin \n\nLong time no see :)\n\n> When using the bundle-uri mechanism with a bundle list containing\n> multiple interrelated bundles, we encountered a bug where tips from\n> downloaded bundles were not being discovered, resulting in rather slow\n> clones. This was particularly problematic when employing the heuristic\n> `creationTokens`.\n> \n> And this is easy to reproduce. Suppose we have a repository with a\n> single branch `main` pointing to commit `A`, firstly we create a base\n> bundle with\n> \n> git bundle create base.bundle main\n> \n> Then let's add a new commit `B` on top of `A`, so that an incremental\n> bundle for `main` can be created with\n> \n> git bundle create incr.bundle A..main\n> \n> Now we can generate a bundle list with the following content:\n> \n> [bundle]\n> version = 1\n> mode = all\n> heuristic = creationToken\n> \n> [bundle \"base\"]\n> uri = base.bundle\n> creationToken = 1\n> \n> [bundle \"incr\"]\n> uri = incr.bundle\n> creationToken = 2\n> \n> A fresh clone with the bundle list above would give the expected\n> `refs/bundles/main` pointing at `B` in new repository, in other words we\n> already had everything locally from the bundles, but git would still\n> download everything from server as if we got nothing.\n> \n> So why the `refs/bundles/main` is not discovered? After some digging I\n> found that:\n> \n> 1. when unbundling a downloaded bundle, a `verify_bundle` is called to\n> check its prerequisites if any. The verify procedure would find oids\n> so `packed_git` is initialized.\n> \n> 2. after unbundled all bundles, we would enter `do_fetch_pack_v2`,\n> during which `mark_complete_and_common_ref` and `mark_tips` would\n> find oids with `OBJECT_INFO_QUICK` flag set, so no new packs would be\n> enlisted if `packed_git` has already initialized in 1.\n\nAnd I assume we do not want it to not use `OBJECT_INFO_QUICK`?\n\n> Back to the example above, when unbunding `incr.bundle`, `base.pack` is\n> enlisted to `packed_git` bacause of the prerequisites to verify. Then we\n> can not find `B` for negotiation at a latter time bacause `B` exists in\n> `incr.pack` which is not enlisted in `packed_git`.\n\nOkay, the explanation feels sensible.\n\n> This commit fixes this by adding a `reprepare_packed_git` call for every\n> successfully unbundled bundle, which ensures to enlist all generated\n> packs from bundle uri. And a set of negotiation related tests are added.\n\nThis makes me wonder though. Do we really need to call\n`reprepare_packed_git()` once for every bundle, or can't we instead call\nit once at the end once we have fetched all bundles? Reading on.\n\n> Signed-off-by: Xing Xin \n> ---\n> bundle-uri: refresh packed_git if unbundle succeed\n> \n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1730\n> \n> bundle-uri.c | 3 +\n> t/t5558-clone-bundle-uri.sh | 129 ++++++++++++++++++++++++++++++++++++\n> 2 files changed, 132 insertions(+)\n> \n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index ca32050a78f..2b9d36cfd8e 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -7,6 +7,7 @@\n> #include \"refs.h\"\n> #include \"run-command.h\"\n> #include \"hashmap.h\"\n> +#include \"packfile.h\"\n> #include \"pkt-line.h\"\n> #include \"config.h\"\n> #include \"remote.h\"\n> @@ -376,6 +377,8 @@ static int unbundle_from_file(struct repository *r, const char *file)\n> \t\t\t VERIFY_BUNDLE_QUIET)))\n> \t\treturn 1;\n> \n> +\treprepare_packed_git(r);\n> +\n\nSo what's hidden here is that `unbundle_from_file()` will also try to\naccess the bundle's refs right away. Surprisingly, we do so by calling\n`refs_update_ref()` with `REF_SKIP_OID_VERIFICATION`, which has the\neffect that we basically accept arbitrary object IDs here even if we do\nnot know those. That's why we didn't have to `reprepare_packed_git()`\nbefore this change.\n\nNow there are two conflicting thoughts here:\n\n - Either we can now drop `REF_SKIP_OID_VERIFICATION` as the object IDs\n should now be accessible.\n\n - Or we can avoid calling `reprepare_packed_git()` inside the loop and\n instead call it once after we have fetched all bundles.\n\nThe second one feels a bit like premature optimization to me. But the\nfirst item does feel like it could help us to catch broken bundles\nbecause we wouldn't end up creating refs for objects that neither we nor\nthe bundle have.\n\nPatrick\n"},{"id":"494935","messageId":"CAOLa=ZT=x6bmnvJO2KtbJvVOvt=4LH1RgU1NMJYpTgW2XXfGVg@mail.gmail.com","threadId":"61479","inReplyTo":"pull.1730.git.1715742069966.gitgitgadget@gmail.com","subject":"Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-17T07:36:53Z","receivedAt":"2024-05-17T07:36:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"blanet via GitGitGadget\" writes:\n\n> From: Xing Xin >\n> When using the bundle-uri mechanism with a bundle list containing\n> multiple interrelated bundles, we encountered a bug where tips from\n> downloaded bundles were not being discovered, resulting in rather slow\n> clones. This was particularly problematic when employing the heuristic\n> `creationTokens`.\n>\n> And this is easy to reproduce. Suppose we have a repository with a\n> single branch `main` pointing to commit `A`, firstly we create a base\n> bundle with\n>\n> git bundle create base.bundle main\n>\n> Then let's add a new commit `B` on top of `A`, so that an incremental\n> bundle for `main` can be created with\n>\n> git bundle create incr.bundle A..main\n>\n> Now we can generate a bundle list with the following content:\n>\n> [bundle]\n> version = 1\n> mode = all\n> heuristic = creationToken\n>\n> [bundle \"base\"]\n> uri = base.bundle\n> creationToken = 1\n>\n> [bundle \"incr\"]\n> uri = incr.bundle\n> creationToken = 2\n>\n> A fresh clone with the bundle list above would give the expected\n> `refs/bundles/main` pointing at `B` in new repository, in other words we\n> already had everything locally from the bundles, but git would still\n> download everything from server as if we got nothing.\n>\n> So why the `refs/bundles/main` is not discovered? After some digging I\n> found that:\n>\n> 1. when unbundling a downloaded bundle, a `verify_bundle` is called to\n\ns/a//\n\n> check its prerequisites if any. The verify procedure would find oids\n> so `packed_git` is initialized.\n>\n\nSo the flow is:\n1. `fetch_bundle_list` fetches all the bundles advertised via\n`download_bundle_list` to local files.\n2. It then calls `unbundle_all_bundles` to unbundle all the bundles.\n3. Each bundle is then unbundled using `unbundle_from_file`.\n4. Here, we first read the bundle header to get all the prerequisites\nfor the bundle, this is done in `read_bundle_header`.\n5. Then we call `unbundle`, which calls `verify_bundle` to ensure that\nthe repository does indeed contain the prerequisites mentioned in the\nbundle. Then it creates the index from the bundle file.\n\nSo because the objects are being checked, the `prepare_packed_git`\nfunction is eventually called, which means that the\n`raw_object_store->packed_git` data gets filled in and\n`packed_git_initialized` is set.\n\nThis means consecutive calls to `prepare_packed_git` doesn't\nre-initiate `raw_object_store->packed_git` since\n`packed_git_initialized` already is set.\n\nSo your explanation makes sense, as a _nit_ I would perhaps add the part\nabout why consecutive calls to `prepare_packed_git` are ineffective.\n\n> 2. after unbundled all bundles, we would enter `do_fetch_pack_v2`,\n\ns/unbundled/unbundling\n\n> during which `mark_complete_and_common_ref` and `mark_tips` would\n> find oids with `OBJECT_INFO_QUICK` flag set, so no new packs would be\n> enlisted if `packed_git` has already initialized in 1.\n> Back to the example above, when unbunding `incr.bundle`, `base.pack` is\n> enlisted to `packed_git` bacause of the prerequisites to verify. Then we\n> can not find `B` for negotiation at a latter time bacause `B` exists in\n> `incr.pack` which is not enlisted in `packed_git`.\n>\n> This commit fixes this by adding a `reprepare_packed_git` call for every\n> successfully unbundled bundle, which ensures to enlist all generated\n> packs from bundle uri. And a set of negotiation related tests are added.\n>\n\nThe solution makes sense.\n\n> Signed-off-by: Xing Xin \n> ---\n> bundle-uri: refresh packed_git if unbundle succeed\n>\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1730\n>\n> bundle-uri.c | 3 +\n> t/t5558-clone-bundle-uri.sh | 129 ++++++++++++++++++++++++++++++++++++\n> 2 files changed, 132 insertions(+)\n>\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index ca32050a78f..2b9d36cfd8e 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -7,6 +7,7 @@\n> #include \"refs.h\"\n> #include \"run-command.h\"\n> #include \"hashmap.h\"\n> +#include \"packfile.h\"\n> #include \"pkt-line.h\"\n> #include \"config.h\"\n> #include \"remote.h\"\n> @@ -376,6 +377,8 @@ static int unbundle_from_file(struct repository *r, const char *file)\n> \t\t\t VERIFY_BUNDLE_QUIET)))\n> \t\treturn 1;\n>\n> +\treprepare_packed_git(r);\n> +\n>\n\nWould it make sense to move this to `bundle.c:unbundle()`, since that is\nalso where the idx is created?\n\n[snip]\n"},{"id":"494979","messageId":"xmqqikzcl8js.fsf@gitster.g","threadId":"61479","inReplyTo":"ZkbkgT46mdNEhcf0@tanuki","subject":"Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-17T16:14:47Z","receivedAt":"2024-05-17T16:14:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Now there are two conflicting thoughts here:\n>\n> - Either we can now drop `REF_SKIP_OID_VERIFICATION` as the object IDs\n> should now be accessible.\n>\n> - Or we can avoid calling `reprepare_packed_git()` inside the loop and\n> instead call it once after we have fetched all bundles.\n>\n> The second one feels a bit like premature optimization to me. But the\n> first item does feel like it could help us to catch broken bundles\n> because we wouldn't end up creating refs for objects that neither we nor\n> the bundle have.\n\nI like the way your thoughts are structured around here.\n\nI do agree that the latter is a wrong approach---we shouldn't be\ntrusting what came from elsewhere over the network without first\nchecking. We should probably be running the \"index-pack --fix-thin\"\nthe unbundling process runs with also the \"--fsck-objects\" option if\nwe are not doing so already, and even then, we should make sure that\nthe object we are making our ref point at have everything behind it.\n\n"},{"id":"495071","messageId":"6d2a424e.a65f.18f95606177.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZkbkgT46mdNEhcf0@tanuki","subject":"Re:Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-20T09:41:12Z","receivedAt":"2024-05-20T09:41:32Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-17 13:00:49, \"Patrick Steinhardt\" wrote:\n>On Wed, May 15, 2024 at 03:01:09AM +0000, blanet via GitGitGadget wrote:\n>> From: Xing Xin \n>\n>Long time no see :)\n\nGlad to see you again here :)\n\n>> \n>> So why the `refs/bundles/main` is not discovered? After some digging I\n>> found that:\n>> \n>> 1. when unbundling a downloaded bundle, a `verify_bundle` is called to\n>> check its prerequisites if any. The verify procedure would find oids\n>> so `packed_git` is initialized.\n>> \n>> 2. after unbundled all bundles, we would enter `do_fetch_pack_v2`,\n>> during which `mark_complete_and_common_ref` and `mark_tips` would\n>> find oids with `OBJECT_INFO_QUICK` flag set, so no new packs would be\n>> enlisted if `packed_git` has already initialized in 1.\n>\n>And I assume we do not want it to not use `OBJECT_INFO_QUICK`?\n\nI think so. For clones or fetches without using bundle-uri, we can hardly\nencounter the case that new packs are added during the negotiation process.\nSo using `OBJECT_INFO_QUICK` can get some performance gain.\n\n>\n>> Back to the example above, when unbunding `incr.bundle`, `base.pack` is\n>> enlisted to `packed_git` bacause of the prerequisites to verify. Then we\n>> can not find `B` for negotiation at a latter time bacause `B` exists in\n>> `incr.pack` which is not enlisted in `packed_git`.\n>\n>Okay, the explanation feels sensible.\n>\n>> This commit fixes this by adding a `reprepare_packed_git` call for every\n>> successfully unbundled bundle, which ensures to enlist all generated\n>> packs from bundle uri. And a set of negotiation related tests are added.\n>\n>This makes me wonder though. Do we really need to call\n>`reprepare_packed_git()` once for every bundle, or can't we instead call\n>it once at the end once we have fetched all bundles? Reading on.\n>\n>> Signed-off-by: Xing Xin \n>> ---\n>> bundle-uri: refresh packed_git if unbundle succeed\n>> \n>> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v1\n>> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v1\n>> Pull-Request: https://github.com/gitgitgadget/git/pull/1730\n>> \n>> bundle-uri.c | 3 +\n>> t/t5558-clone-bundle-uri.sh | 129 ++++++++++++++++++++++++++++++++++++\n>> 2 files changed, 132 insertions(+)\n>> \n>> diff --git a/bundle-uri.c b/bundle-uri.c\n>> index ca32050a78f..2b9d36cfd8e 100644\n>> --- a/bundle-uri.c\n>> +++ b/bundle-uri.c\n>> @@ -7,6 +7,7 @@\n>> #include \"refs.h\"\n>> #include \"run-command.h\"\n>> #include \"hashmap.h\"\n>> +#include \"packfile.h\"\n>> #include \"pkt-line.h\"\n>> #include \"config.h\"\n>> #include \"remote.h\"\n>> @@ -376,6 +377,8 @@ static int unbundle_from_file(struct repository *r, const char *file)\n>> \t\t\t VERIFY_BUNDLE_QUIET)))\n>> \t\treturn 1;\n>> \n>> +\treprepare_packed_git(r);\n>> +\n>\n>So what's hidden here is that `unbundle_from_file()` will also try to\n>access the bundle's refs right away. Surprisingly, we do so by calling\n>`refs_update_ref()` with `REF_SKIP_OID_VERIFICATION`, which has the\n>effect that we basically accept arbitrary object IDs here even if we do\n>not know those. That's why we didn't have to `reprepare_packed_git()`\n>before this change.\n\nYou are right! I tried dropping this `REF_SKIP_OID_VERIFICATION` flag and\nthe negotiation works as expected.\n\nAfter some further digging I find that without `REF_SKIP_OID_VERIFICATION`,\nboth `write_ref_to_lockfile` for files backend and `reftable_be_transaction_prepare`\nfor reftable backend would call `parse_object` to check the oid. `parse_object`\ncan help refresh `packed_git` via `reprepare_packed_git`.\n\n>\n>Now there are two conflicting thoughts here:\n>\n> - Either we can now drop `REF_SKIP_OID_VERIFICATION` as the object IDs\n> should now be accessible.\n>\n> - Or we can avoid calling `reprepare_packed_git()` inside the loop and\n> instead call it once after we have fetched all bundles.\n>\n>The second one feels a bit like premature optimization to me. But the\n>first item does feel like it could help us to catch broken bundles\n>because we wouldn't end up creating refs for objects that neither we nor\n>the bundle have.\n\nI favor the first approach because a validation on the object IDs we are\nwriting is a safe guard . And the flag itself was designed to be used in\ntesting scenarios.\n\n/*\n * Blindly write an object_id. This is useful for testing data corruption\n * scenarios.\n */\n#define REF_SKIP_OID_VERIFICATION (1 << 10)\n\n\n"},{"id":"495075","messageId":"59d5f488.ad90.18f9583cbc2.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"CAOLa=ZT=x6bmnvJO2KtbJvVOvt=4LH1RgU1NMJYpTgW2XXfGVg@mail.gmail.com","subject":"Re:Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-20T10:19:53Z","receivedAt":"2024-05-20T10:20:06Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-17 15:36:53, \"Karthik Nayak\" wrote:\n>\"blanet via GitGitGadget\" writes:\n>\n>> From: Xing Xin >\n>> When using the bundle-uri mechanism with a bundle list containing\n>> multiple interrelated bundles, we encountered a bug where tips from\n>> downloaded bundles were not being discovered, resulting in rather slow\n>> clones. This was particularly problematic when employing the heuristic\n>> `creationTokens`.\n>>\n>> And this is easy to reproduce. Suppose we have a repository with a\n>> single branch `main` pointing to commit `A`, firstly we create a base\n>> bundle with\n>>\n>> git bundle create base.bundle main\n>>\n>> Then let's add a new commit `B` on top of `A`, so that an incremental\n>> bundle for `main` can be created with\n>>\n>> git bundle create incr.bundle A..main\n>>\n>> Now we can generate a bundle list with the following content:\n>>\n>> [bundle]\n>> version = 1\n>> mode = all\n>> heuristic = creationToken\n>>\n>> [bundle \"base\"]\n>> uri = base.bundle\n>> creationToken = 1\n>>\n>> [bundle \"incr\"]\n>> uri = incr.bundle\n>> creationToken = 2\n>>\n>> A fresh clone with the bundle list above would give the expected\n>> `refs/bundles/main` pointing at `B` in new repository, in other words we\n>> already had everything locally from the bundles, but git would still\n>> download everything from server as if we got nothing.\n>>\n>> So why the `refs/bundles/main` is not discovered? After some digging I\n>> found that:\n>>\n>> 1. when unbundling a downloaded bundle, a `verify_bundle` is called to\n>\n>s/a//\n\nThanks!\n\n>\n>> check its prerequisites if any. The verify procedure would find oids\n>> so `packed_git` is initialized.\n>>\n>\n>So the flow is:\n>1. `fetch_bundle_list` fetches all the bundles advertised via\n>`download_bundle_list` to local files.\n>2. It then calls `unbundle_all_bundles` to unbundle all the bundles.\n>3. Each bundle is then unbundled using `unbundle_from_file`.\n>4. Here, we first read the bundle header to get all the prerequisites\n>for the bundle, this is done in `read_bundle_header`.\n>5. Then we call `unbundle`, which calls `verify_bundle` to ensure that\n>the repository does indeed contain the prerequisites mentioned in the\n>bundle. Then it creates the index from the bundle file.\n>\n>So because the objects are being checked, the `prepare_packed_git`\n>function is eventually called, which means that the\n>`raw_object_store->packed_git` data gets filled in and\n>`packed_git_initialized` is set.\n>\n>This means consecutive calls to `prepare_packed_git` doesn't\n>re-initiate `raw_object_store->packed_git` since\n>`packed_git_initialized` already is set.\n>\n>So your explanation makes sense, as a _nit_ I would perhaps add the part\n>about why consecutive calls to `prepare_packed_git` are ineffective.\n\nThanks my friend, you have expressed this issue more clearly. I will\npost a new description based on your explanation with the creationToken\ncase covered.\n\n>\n>> 2. after unbundled all bundles, we would enter `do_fetch_pack_v2`,\n>\n>s/unbundled/unbundling\n\nCopy that.\n\n>\n>> +#include \"packfile.h\"\n>> #include \"pkt-line.h\"\n>> #include \"config.h\"\n>> #include \"remote.h\"\n>> @@ -376,6 +377,8 @@ static int unbundle_from_file(struct repository *r, const char *file)\n>> \t\t\t VERIFY_BUNDLE_QUIET)))\n>> \t\treturn 1;\n>>\n>> +\treprepare_packed_git(r);\n>> +\n>>\n>\n>Would it make sense to move this to `bundle.c:unbundle()`, since that is\n>also where the idx is created?\n>\n\nI wonder if we need a mental model that we should `reprepare_packed_git`\nthat when a new pack and its corresponding idx is generated? Currently\nwhether to call `reprepare_packed_git` is determined by the caller.\n\nBut within the scope of this bug, I tend to remove the\n`REF_SKIP_OID_VERIFICATION` flag when writing refs as Patrick suggested.\n\nXing Xin\n\n"},{"id":"495078","messageId":"5e61daf6.ba64.18f95d49895.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"xmqqikzcl8js.fsf@gitster.g","subject":"Re:Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-20T11:48:08Z","receivedAt":"2024-05-20T11:48:43Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-18 00:14:47, \"Junio C Hamano\" wrote:\n>Patrick Steinhardt writes:\n>\n>> Now there are two conflicting thoughts here:\n>>\n>> - Either we can now drop `REF_SKIP_OID_VERIFICATION` as the object IDs\n>> should now be accessible.\n>>\n>> - Or we can avoid calling `reprepare_packed_git()` inside the loop and\n>> instead call it once after we have fetched all bundles.\n>>\n>> The second one feels a bit like premature optimization to me. But the\n>> first item does feel like it could help us to catch broken bundles\n>> because we wouldn't end up creating refs for objects that neither we nor\n>> the bundle have.\n>\n>I like the way your thoughts are structured around here.\n>\n>I do agree that the latter is a wrong approach---we shouldn't be\n>trusting what came from elsewhere over the network without first\n>checking. We should probably be running the \"index-pack --fix-thin\"\n>the unbundling process runs with also the \"--fsck-objects\" option if\n>we are not doing so already, and even then, we should make sure that\n>the object we are making our ref point at have everything behind it.\n\nCurrently `unbundle` and all its callers are not adding \"--fsck-objects\".\nThere is a param `extra_index_pack_args` for `unbundle` but it is\nmainly used for progress related options.\n\nPersonally I think data from bundles and data received via network\nshould be treated equally. For \"fetch-pack\" we now have some configs\nsuch as \"fetch.fsckobjects\" and \"transfer.fsckobjects\" to decide the\nbehavior, these configs are invisible when we are fetching bundles.\n\nSo for bundles I think we have some different ways to go:\n\n - Acknowledge the configs mentioned above and behave like\n \"fetch-pack\".\n\n - Add \"--fsck-objects\" as a default in `unbundle`.\n\n - In `unbundle_from_file`, pass in \"--fsck-objects\" as\n `extra_index_pack_args` for `unbundle` so this only affect the\n bundle-uri related process.\n\nWhat do you think?\n\nXing Xin\n"},{"id":"495079","messageId":"pull.1730.v2.git.1716208605926.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.git.1715742069966.gitgitgadget@gmail.com","subject":"[PATCH v2] bundle-uri: verify oid before writing refs","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-20T12:36:45Z","receivedAt":"2024-05-20T12:36:50Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not discovered, thus resulting in rather slow\nclones. This was particularly problematic when employing the heuristic\n`creationTokens`.\n\nAnd this is easy to reproduce. Suppose we have a repository with a\nsingle branch `main` pointing to commit `A`, firstly we create a base\nbundle with\n\n git bundle create base.bundle main\n\nThen let's add a new commit `B` on top of `A`, so that an incremental\nbundle for `main` can be created with\n\n git bundle create incr.bundle A..main\n\nNow we can generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above would give the expected\n`refs/bundles/main` pointing at `B` in new repository, in other words we\nalready had everything locally from the bundles, but git would still\ndownload everything from server as if we got nothing.\n\nSo why the `refs/bundles/main` is not discovered? After some digging I\nfound that:\n\n1. Bundles in bundle list are downloaded to local files via\n `download_bundle_list` or via `fetch_bundles_by_token` for the\n creationToken heuristic case.\n2. Then it tries to unbundle each bundle via `unbundle_from_file`, which\n is called by `unbundle_all_bundles` or called within\n `fetch_bundles_by_token` for the creationToken heuristic case.\n3. Here, we first read the bundle header to get all the prerequisites\n for the bundle, this is done in `read_bundle_header`.\n4. Then we call `unbundle`, which calls `verify_bundle` to ensure that\n the repository does indeed contain the prerequisites mentioned in the\n bundle.\n5. The `verify_bundle` will call `parse_object`, within which the\n `prepare_packed_git` or `reprepare_packed_git` is eventually called,\n which means that the `raw_object_store->packed_git` data gets filled\n in and ``packed_git_initialized` is set. This also means consecutive\n calls to `prepare_packed_git` doesn't re-initiate\n `raw_object_store->packed_git` since `packed_git_initialized` already\n is set.\n6. If `unbundle` succeeds, it writes some refs via `refs_update_ref`\n with `REF_SKIP_OID_VERIFICATION` set. So the bundle refs which can\n target arbitrary objects are written to the repository.\n7. Finally in `do_fetch_pack_v2`, `mark_complete_and_common_ref` and\n `mark_tips` are called with `OBJECT_INFO_QUICK` set to find local\n tips. Here it won't call `reprepare_packed_git` anymore so it would\n fail to parse oids that only reside in the last bundle.\n\nBack to the example above, when unbunding `incr.bundle`, `base.pack` is\nenlisted to `packed_git` bacause of the prerequisites to verify. While\nwe can not find `B` for negotiation at a latter time because `B` exists\nin `incr.pack` which is not enlisted in `packed_git`.\n\nThis commit fixes this bug by dropping the `REF_SKIP_OID_VERIFICATION`\nflag when writing bundle refs, so we can:\n\n1. Ensure that the bundle refs we are writing are pointing to valid\n objects.\n2. Ensure all the tips from bundle refs can be correctly parsed.\n\nAnd a set of negotiation related tests for bundle-uri are added.\n\nSigned-off-by: Xing Xin \n---\n bundle-uri: refresh packed_git if unbundle succeed\n \n cc: Patrick Steinhardt ps@pks.im cc: Karthik Nayak karthik.188@gmail.com\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\nRange-diff vs v1:\n\n 1: d4841eea556 ! 1: 8bdeacf1360 bundle-uri: refresh packed_git if unbundle succeed\n @@ Metadata\n Author: Xing Xin \n \n ## Commit message ##\n - bundle-uri: refresh packed_git if unbundle succeed\n + bundle-uri: verify oid before writing refs\n \n When using the bundle-uri mechanism with a bundle list containing\n multiple interrelated bundles, we encountered a bug where tips from\n - downloaded bundles were not being discovered, resulting in rather slow\n + downloaded bundles were not discovered, thus resulting in rather slow\n clones. This was particularly problematic when employing the heuristic\n `creationTokens`.\n \n @@ Commit message\n So why the `refs/bundles/main` is not discovered? After some digging I\n found that:\n \n - 1. when unbundling a downloaded bundle, a `verify_bundle` is called to\n - check its prerequisites if any. The verify procedure would find oids\n - so `packed_git` is initialized.\n -\n - 2. after unbundled all bundles, we would enter `do_fetch_pack_v2`,\n - during which `mark_complete_and_common_ref` and `mark_tips` would\n - find oids with `OBJECT_INFO_QUICK` flag set, so no new packs would be\n - enlisted if `packed_git` has already initialized in 1.\n + 1. Bundles in bundle list are downloaded to local files via\n + `download_bundle_list` or via `fetch_bundles_by_token` for the\n + creationToken heuristic case.\n + 2. Then it tries to unbundle each bundle via `unbundle_from_file`, which\n + is called by `unbundle_all_bundles` or called within\n + `fetch_bundles_by_token` for the creationToken heuristic case.\n + 3. Here, we first read the bundle header to get all the prerequisites\n + for the bundle, this is done in `read_bundle_header`.\n + 4. Then we call `unbundle`, which calls `verify_bundle` to ensure that\n + the repository does indeed contain the prerequisites mentioned in the\n + bundle.\n + 5. The `verify_bundle` will call `parse_object`, within which the\n + `prepare_packed_git` or `reprepare_packed_git` is eventually called,\n + which means that the `raw_object_store->packed_git` data gets filled\n + in and ``packed_git_initialized` is set. This also means consecutive\n + calls to `prepare_packed_git` doesn't re-initiate\n + `raw_object_store->packed_git` since `packed_git_initialized` already\n + is set.\n + 6. If `unbundle` succeeds, it writes some refs via `refs_update_ref`\n + with `REF_SKIP_OID_VERIFICATION` set. So the bundle refs which can\n + target arbitrary objects are written to the repository.\n + 7. Finally in `do_fetch_pack_v2`, `mark_complete_and_common_ref` and\n + `mark_tips` are called with `OBJECT_INFO_QUICK` set to find local\n + tips. Here it won't call `reprepare_packed_git` anymore so it would\n + fail to parse oids that only reside in the last bundle.\n \n Back to the example above, when unbunding `incr.bundle`, `base.pack` is\n - enlisted to `packed_git` bacause of the prerequisites to verify. Then we\n - can not find `B` for negotiation at a latter time bacause `B` exists in\n - `incr.pack` which is not enlisted in `packed_git`.\n + enlisted to `packed_git` bacause of the prerequisites to verify. While\n + we can not find `B` for negotiation at a latter time because `B` exists\n + in `incr.pack` which is not enlisted in `packed_git`.\n +\n + This commit fixes this bug by dropping the `REF_SKIP_OID_VERIFICATION`\n + flag when writing bundle refs, so we can:\n \n - This commit fixes this by adding a `reprepare_packed_git` call for every\n - successfully unbundled bundle, which ensures to enlist all generated\n - packs from bundle uri. And a set of negotiation related tests are added.\n + 1. Ensure that the bundle refs we are writing are pointing to valid\n + objects.\n + 2. Ensure all the tips from bundle refs can be correctly parsed.\n +\n + And a set of negotiation related tests for bundle-uri are added.\n \n Signed-off-by: Xing Xin \n \n ## bundle-uri.c ##\n -@@\n - #include \"refs.h\"\n - #include \"run-command.h\"\n - #include \"hashmap.h\"\n -+#include \"packfile.h\"\n - #include \"pkt-line.h\"\n - #include \"config.h\"\n - #include \"remote.h\"\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *file)\n - \t\t\t VERIFY_BUNDLE_QUIET)))\n - \t\treturn 1;\n + \t\trefs_update_ref(get_main_ref_store(the_repository),\n + \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n + \t\t\t\thas_old ? &old_oid : NULL,\n +-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n +-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n ++\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n + \t}\n \n -+\treprepare_packed_git(r);\n -+\n - \t/*\n - \t * Convert all refs/heads/ from the bundle into refs/bundles/\n - \t * in the local repository.\n + \tbundle_header_release(&header);\n \n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'fail to clone from non-bundle file' '\n\n\n bundle-uri.c | 3 +-\n t/t5558-clone-bundle-uri.sh | 129 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 130 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91b3319a5c1..65666a11d9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\trefs_update_ref(get_main_ref_store(the_repository),\n \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n \t\t\t\thas_old ? &old_oid : NULL,\n-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n+\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n \t}\n \n \tbundle_header_release(&header);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..a5b04d6f187 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -20,7 +20,10 @@ test_expect_success 'fail to clone from non-bundle file' '\n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n \tgit -C clone-from checkout -b topic &&\n+\n \ttest_commit -C clone-from A &&\n+\tgit -C clone-from bundle create A.bundle topic &&\n+\n \ttest_commit -C clone-from B &&\n \tgit -C clone-from bundle create B.bundle topic\n '\n@@ -259,6 +262,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+#########################################################################\n+# Clone negotiation related tests begin here\n+\n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n\nbase-commit: d8ab1d464d07baa30e5a180eb33b3f9aa5c93adf\n-- \ngitgitgadget\n"},{"id":"495092","messageId":"xmqqh6esjta1.fsf@gitster.g","threadId":"61479","inReplyTo":"5e61daf6.ba64.18f95d49895.Coremail.bupt_xingxin@163.com","subject":"Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-20T17:19:02Z","receivedAt":"2024-05-20T17:19:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Xing Xin\" writes:\n\n> Personally I think data from bundles and data received via network\n> should be treated equally.\n\nYup, that is not personal ;-) but is universally accepted as a good\ndiscipline. In the case of bundle-uri, the bundle came over the\nnetwork so it is even more true that they should be treated the\nsame.\n\n> For \"fetch-pack\" we now have some configs\n> such as \"fetch.fsckobjects\" and \"transfer.fsckobjects\" to decide the\n> behavior, these configs are invisible when we are fetching bundles.\n\nWhen fetching over network, transport.c:fetch_refs_via_pack() calls\nfetch_pack.c:fetch_pack(), which eventually calls get_pack() and the\nconfiguration variables are honored there. It appears that the\ntransport layer is unaware of the .fsckobjects configuration knobs.\n\nWhen fetching from a bundle, transport.c:fetch_refs_from_bundle()\ncalls bundle.c:unbundle(). This function has three callers, i.e.\n\"git bundle unbundle\", normal fetching from a bundle, and more\nrecently added bundle-uri codepaths. \n\nI think one reasonable approach to take is to add an extra parameter\nthat takes one of three values: (never, use-config, always), and\nconditionally add \"--fsck-objects\" to the command line of the\nindex-pack. Teach \"git bundle unbundle\" the \"--fsck-objects\" option\nso that it can pass 'never' or 'always' from the command line, and\npass 'use-config' from the code paths for normal fetching from a\nbudnle and bundle-uri.\n\nTo implement use-config, you'd probably need to refactor a small\npart of fetch-pack.c:get_pack()\n\n\tif (fetch_fsck_objects >= 0\n\t ? fetch_fsck_objects\n\t : transfer_fsck_objects >= 0\n\t ? transfer_fsck_objects\n\t : 0)\n\t\tfsck_objects = 1;\n\ninto a public function (to support a caller like unbundle() that\ncomes from sideways, the new function may also need to call\nfetch_pack_setup() to prime them).\n\nA patch series may take a structure like so:\n\n * define enum { UNBUNDLE_FSCK_NEVER, UNBUNDLE_FSCK_ALWAYS } in\n bundle.h, have bundle.c:unbundle() accept a new parameter of that\n type, and conditionally add \"--fsck-objects\" to its call to\n \"index-pack\". \"git bundle unbundle\" can pass 'never' to its\n invocation to unbundle() as an easy way to test it. For the\n other two callers, we can start by passing 'always'.\n\n * (optional) teach \"git bundle unbundle\" a new \"--fsck-objects\"\n option to allow passing 'always' to its call to unbundle(). With\n that, add tests to feed it a bundle with questionable objects in\n it and make sure that unbundling notices.\n\n * refactor fetch-pack.c:get_pack() to make the fetch-then-transfer\n configuration logic available to external callers.\n\n * Add UNBUNDLE_FSCK_USE_CONFIG to the enum, enhance unbundle() to\n react to the value by calling the helper function you introduced\n in the previous step.\n\nThanks.\n"},{"id":"495171","messageId":"CAOLa=ZR2QU-76ovpMF4xRvAzY447ce+oAYCaiiMTcFGL7Q52-A@mail.gmail.com","threadId":"61479","inReplyTo":"pull.1730.v2.git.1716208605926.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] bundle-uri: verify oid before writing refs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-21T15:41:40Z","receivedAt":"2024-05-21T15:41:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"blanet via GitGitGadget\" writes:\n\n> From: Xing Xin \n>\n> When using the bundle-uri mechanism with a bundle list containing\n> multiple interrelated bundles, we encountered a bug where tips from\n> downloaded bundles were not discovered, thus resulting in rather slow\n> clones. This was particularly problematic when employing the heuristic\n> `creationTokens`.\n>\n> And this is easy to reproduce. Suppose we have a repository with a\n> single branch `main` pointing to commit `A`, firstly we create a base\n> bundle with\n>\n> git bundle create base.bundle main\n>\n> Then let's add a new commit `B` on top of `A`, so that an incremental\n> bundle for `main` can be created with\n>\n> git bundle create incr.bundle A..main\n>\n> Now we can generate a bundle list with the following content:\n>\n> [bundle]\n> version = 1\n> mode = all\n> heuristic = creationToken\n>\n> [bundle \"base\"]\n> uri = base.bundle\n> creationToken = 1\n>\n> [bundle \"incr\"]\n> uri = incr.bundle\n> creationToken = 2\n>\n> A fresh clone with the bundle list above would give the expected\n> `refs/bundles/main` pointing at `B` in new repository, in other words we\n> already had everything locally from the bundles, but git would still\n> download everything from server as if we got nothing.\n>\n> So why the `refs/bundles/main` is not discovered? After some digging I\n> found that:\n>\n> 1. Bundles in bundle list are downloaded to local files via\n> `download_bundle_list` or via `fetch_bundles_by_token` for the\n> creationToken heuristic case.\n> 2. Then it tries to unbundle each bundle via `unbundle_from_file`, which\n> is called by `unbundle_all_bundles` or called within\n> `fetch_bundles_by_token` for the creationToken heuristic case.\n> 3. Here, we first read the bundle header to get all the prerequisites\n> for the bundle, this is done in `read_bundle_header`.\n> 4. Then we call `unbundle`, which calls `verify_bundle` to ensure that\n> the repository does indeed contain the prerequisites mentioned in the\n> bundle.\n> 5. The `verify_bundle` will call `parse_object`, within which the\n> `prepare_packed_git` or `reprepare_packed_git` is eventually called,\n> which means that the `raw_object_store->packed_git` data gets filled\n> in and ``packed_git_initialized` is set. This also means consecutive\n> calls to `prepare_packed_git` doesn't re-initiate\n> `raw_object_store->packed_git` since `packed_git_initialized` already\n> is set.\n> 6. If `unbundle` succeeds, it writes some refs via `refs_update_ref`\n> with `REF_SKIP_OID_VERIFICATION` set. So the bundle refs which can\n> target arbitrary objects are written to the repository.\n> 7. Finally in `do_fetch_pack_v2`, `mark_complete_and_common_ref` and\n> `mark_tips` are called with `OBJECT_INFO_QUICK` set to find local\n> tips. Here it won't call `reprepare_packed_git` anymore so it would\n> fail to parse oids that only reside in the last bundle.\n>\n> Back to the example above, when unbunding `incr.bundle`, `base.pack` is\n> enlisted to `packed_git` bacause of the prerequisites to verify. While\n\ns/bacause/because\n\n[snip]\n\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 91b3319a5c1..65666a11d9c 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n> \t\trefs_update_ref(get_main_ref_store(the_repository),\n> \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n> \t\t\t\thas_old ? &old_oid : NULL,\n> -\t\t\t\tREF_SKIP_OID_VERIFICATION,\n> -\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n> +\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n> \t}\n>\n> \tbundle_header_release(&header);\n> diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\n> index 1ca5f745e73..a5b04d6f187 100755\n> --- a/t/t5558-clone-bundle-uri.sh\n> +++ b/t/t5558-clone-bundle-uri.sh\n> @@ -20,7 +20,10 @@ test_expect_success 'fail to clone from non-bundle file' '\n> test_expect_success 'create bundle' '\n> \tgit init clone-from &&\n> \tgit -C clone-from checkout -b topic &&\n> +\n> \ttest_commit -C clone-from A &&\n> +\tgit -C clone-from bundle create A.bundle topic &&\n> +\n> \ttest_commit -C clone-from B &&\n> \tgit -C clone-from bundle create B.bundle topic\n> '\n> @@ -259,6 +262,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n> \t! grep \"refs/bundles/\" refs\n> '\n>\n> +#########################################################################\n> +# Clone negotiation related tests begin here\n> +\n> +test_expect_success 'negotiation: bundle with part of wanted commits' '\n> +\ttest_when_finished rm -rf trace*.txt &&\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n> +\t\tclone-from nego-bundle-part &&\n> +\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/topic\n> +\tEOF\n> +\ttest_cmp expect actual &&\n> +\t# Ensure that refs/bundles/topic are sent as \"have\".\n> +\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n> +'\n> +\n> +test_expect_success 'negotiation: bundle with all wanted commits' '\n> +\ttest_when_finished rm -rf trace*.txt &&\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n> +\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n> +\t\tclone-from nego-bundle-all &&\n> +\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/topic\n> +\tEOF\n> +\ttest_cmp expect actual &&\n> +\t# We already have all needed commits so no \"want\" needed.\n> +\t! grep \"clone> want \" trace-packet.txt\n> +'\n> +\n> +test_expect_success 'negotiation: bundle list (no heuristic)' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\tcat >bundle-list <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n> +\tEOF\n> +\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n> +\t\tclone-from nego-bundle-list-no-heuristic &&\n> +\n> +\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/base\n> +\trefs/bundles/left\n> +\tEOF\n> +\ttest_cmp expect actual &&\n> +\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n> +'\n> +\n> +test_expect_success 'negotiation: bundle list (creationToken)' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\tcat >bundle-list <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n> +\t\tcreationToken = 1\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n> +\t\tcreationToken = 2\n> +\tEOF\n> +\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n> +\t\tclone-from nego-bundle-list-heuristic &&\n> +\n> +\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/base\n> +\trefs/bundles/left\n> +\tEOF\n> +\ttest_cmp expect actual &&\n> +\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n> +'\n> +\n> +test_expect_success 'negotiation: bundle list with all wanted commits' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\tcat >bundle-list <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\t\theuristic = creationToken\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n> +\t\tcreationToken = 1\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n> +\t\tcreationToken = 2\n> +\tEOF\n> +\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --single-branch --branch=left --no-tags \\\n> +\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n> +\t\tclone-from nego-bundle-list-all &&\n> +\n> +\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/base\n> +\trefs/bundles/left\n> +\tEOF\n> +\ttest_cmp expect actual &&\n> +\t# We already have all needed commits so no \"want\" needed.\n> +\t! grep \"clone> want \" trace-packet.txt\n> +'\n> +\n> #########################################################################\n> # HTTP tests begin here\n>\n>\n> base-commit: d8ab1d464d07baa30e5a180eb33b3f9aa5c93adf\n> --\n> gitgitgadget\n\nThis update looks good to me, Thanks.\n"},{"id":"495693","messageId":"pull.1730.v3.git.1716824518.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v2.git.1716208605926.gitgitgadget@gmail.com","subject":"[PATCH v3 0/4] object checking related additions and fixes for bundles in fetches","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-27T15:41:53Z","receivedAt":"2024-05-27T15:42:01Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"While attempting to fix a reference negotiation bug in bundle-uri, we\ndiscovered that the fetch process are missing some helpful object validation\nlogic when processing bundles. The main issues are:\n\n * In the bundle-uri scenario, we did not validate object IDs before writing\n bundle references. This is the root cause of the original negotiation bug\n in bundle-uri, and can cause potential repository corruption.\n * The existing fetch.fsckObjects and transfer.fsckObjects are not detected\n when directly fetching bundles. In fact there is no object validation\n support for unbundle.\n\nThe first patch fixes the bundle-uri negotiation issue by dropping the\nREF_SKIP_OID_VERIFICATION flag when writing bundle references.\n\nPatch 2~4 extend bundle.c:unbundle with a unbundle_fsck_flags to control\nobject fscking in different scenarios, the implementation mainly follows\nwhat Junio suggested on the mailing list.\n\nXing Xin (4):\n bundle-uri: verify oid before writing refs\n unbundle: introduce unbundle_fsck_flags for fsckobjects handling\n fetch-pack: expose fsckObjects configuration logic\n unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH\n\n builtin/bundle.c | 2 +-\n bundle-uri.c | 5 +-\n bundle.c | 20 ++++-\n bundle.h | 9 +-\n fetch-pack.c | 18 ++--\n fetch-pack.h | 2 +\n t/t5558-clone-bundle-uri.sh | 163 +++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 23 +++++\n transport.c | 2 +-\n 9 files changed, 227 insertions(+), 17 deletions(-)\n\n\nbase-commit: b9cfe4845cb2562584837bc0101c0ab76490a239\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\nRange-diff vs v2:\n\n 1: 8bdeacf1360 = 1: 8f488a5eeaa bundle-uri: verify oid before writing refs\n -: ----------- > 2: 057c697970f unbundle: introduce unbundle_fsck_flags for fsckobjects handling\n -: ----------- > 3: 67401d4fbcb fetch-pack: expose fsckObjects configuration logic\n -: ----------- > 4: c19b8f633cb unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH\n\n-- \ngitgitgadget\n"},{"id":"495694","messageId":"8f488a5eeaaa0cdb525c34c2c165e6ee74ee7691.1716824518.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v3.git.1716824518.gitgitgadget@gmail.com","subject":"[PATCH v3 1/4] bundle-uri: verify oid before writing refs","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-27T15:41:54Z","receivedAt":"2024-05-27T15:42:01Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not discovered, thus resulting in rather slow\nclones. This was particularly problematic when employing the heuristic\n`creationTokens`.\n\nAnd this is easy to reproduce. Suppose we have a repository with a\nsingle branch `main` pointing to commit `A`, firstly we create a base\nbundle with\n\n git bundle create base.bundle main\n\nThen let's add a new commit `B` on top of `A`, so that an incremental\nbundle for `main` can be created with\n\n git bundle create incr.bundle A..main\n\nNow we can generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above would give the expected\n`refs/bundles/main` pointing at `B` in new repository, in other words we\nalready had everything locally from the bundles, but git would still\ndownload everything from server as if we got nothing.\n\nSo why the `refs/bundles/main` is not discovered? After some digging I\nfound that:\n\n1. Bundles in bundle list are downloaded to local files via\n `download_bundle_list` or via `fetch_bundles_by_token` for the\n creationToken heuristic case.\n2. Then it tries to unbundle each bundle via `unbundle_from_file`, which\n is called by `unbundle_all_bundles` or called within\n `fetch_bundles_by_token` for the creationToken heuristic case.\n3. Here, we first read the bundle header to get all the prerequisites\n for the bundle, this is done in `read_bundle_header`.\n4. Then we call `unbundle`, which calls `verify_bundle` to ensure that\n the repository does indeed contain the prerequisites mentioned in the\n bundle.\n5. The `verify_bundle` will call `parse_object`, within which the\n `prepare_packed_git` or `reprepare_packed_git` is eventually called,\n which means that the `raw_object_store->packed_git` data gets filled\n in and ``packed_git_initialized` is set. This also means consecutive\n calls to `prepare_packed_git` doesn't re-initiate\n `raw_object_store->packed_git` since `packed_git_initialized` already\n is set.\n6. If `unbundle` succeeds, it writes some refs via `refs_update_ref`\n with `REF_SKIP_OID_VERIFICATION` set. So the bundle refs which can\n target arbitrary objects are written to the repository.\n7. Finally in `do_fetch_pack_v2`, `mark_complete_and_common_ref` and\n `mark_tips` are called with `OBJECT_INFO_QUICK` set to find local\n tips. Here it won't call `reprepare_packed_git` anymore so it would\n fail to parse oids that only reside in the last bundle.\n\nBack to the example above, when unbunding `incr.bundle`, `base.pack` is\nenlisted to `packed_git` bacause of the prerequisites to verify. While\nwe can not find `B` for negotiation at a latter time because `B` exists\nin `incr.pack` which is not enlisted in `packed_git`.\n\nThis commit fixes this bug by dropping the `REF_SKIP_OID_VERIFICATION`\nflag when writing bundle refs, so we can:\n\n1. Ensure that the bundle refs we are writing are pointing to valid\n objects.\n2. Ensure all the tips from bundle refs can be correctly parsed.\n\nAnd a set of negotiation related tests for bundle-uri are added.\n\nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 +-\n t/t5558-clone-bundle-uri.sh | 129 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 130 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91b3319a5c1..65666a11d9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\trefs_update_ref(get_main_ref_store(the_repository),\n \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n \t\t\t\thas_old ? &old_oid : NULL,\n-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n+\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n \t}\n \n \tbundle_header_release(&header);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..a5b04d6f187 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -20,7 +20,10 @@ test_expect_success 'fail to clone from non-bundle file' '\n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n \tgit -C clone-from checkout -b topic &&\n+\n \ttest_commit -C clone-from A &&\n+\tgit -C clone-from bundle create A.bundle topic &&\n+\n \ttest_commit -C clone-from B &&\n \tgit -C clone-from bundle create B.bundle topic\n '\n@@ -259,6 +262,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+#########################################################################\n+# Clone negotiation related tests begin here\n+\n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n-- \ngitgitgadget\n\n"},{"id":"495695","messageId":"057c697970ff49301cd9dc6adef099f53d440c3c.1716824518.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v3.git.1716824518.gitgitgadget@gmail.com","subject":"[PATCH v3 2/4] unbundle: introduce unbundle_fsck_flags for fsckobjects handling","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-27T15:41:55Z","receivedAt":"2024-05-27T15:42:02Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThis commit adds a new enum `unbundle_fsck_flags` which is designed to\ncontrol the fsck behavior when unbundling. `unbundle` can use this newly\npassed in enum to further decide whether to enable `--fsck-objects` for\n\"git-index-pack\".\n\nCurrently only `UNBUNDLE_FSCK_NEVER` and `UNBUNDLE_FSCK_ALWAYS` are\nsupported as the very basic options. Another interesting option would be\nadded in later commits.\n\nSigned-off-by: Xing Xin \n---\n builtin/bundle.c | 2 +-\n bundle-uri.c | 2 +-\n bundle.c | 12 +++++++++++-\n bundle.h | 8 +++++++-\n transport.c | 2 +-\n 5 files changed, 21 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 3ad11dc5d05..6c10961c640 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -212,7 +212,7 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n \t\tstrvec_pushl(&extra_index_pack_args, \"-v\", \"--progress-title\",\n \t\t\t _(\"Unbundling objects\"), NULL);\n \tret = !!unbundle(the_repository, &header, bundle_fd,\n-\t\t\t &extra_index_pack_args, 0) ||\n+\t\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_NEVER) ||\n \t\tlist_bundle_refs(&header, argc, argv);\n \tbundle_header_release(&header);\n cleanup:\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 65666a11d9c..80f02aac6f1 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET, UNBUNDLE_FSCK_ALWAYS)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 95367c2d0a0..a922d592782 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -612,7 +612,8 @@ int create_bundle(struct repository *r, const char *path,\n \n int unbundle(struct repository *r, struct bundle_header *header,\n \t int bundle_fd, struct strvec *extra_index_pack_args,\n-\t enum verify_bundle_flags flags)\n+\t enum verify_bundle_flags flags,\n+\t enum unbundle_fsck_flags fsck_flags)\n {\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n \n@@ -625,6 +626,15 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n+\tswitch (fsck_flags) {\n+\tcase UNBUNDLE_FSCK_ALWAYS:\n+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\t\tbreak;\n+\tcase UNBUNDLE_FSCK_NEVER:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/bundle.h b/bundle.h\nindex 021adbdcbb3..cfa9daddda6 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -30,6 +30,11 @@ int create_bundle(struct repository *r, const char *path,\n \t\t int argc, const char **argv, struct strvec *pack_options,\n \t\t int version);\n \n+enum unbundle_fsck_flags {\n+\tUNBUNDLE_FSCK_NEVER = 0,\n+\tUNBUNDLE_FSCK_ALWAYS,\n+};\n+\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n@@ -53,7 +58,8 @@ int verify_bundle(struct repository *r, struct bundle_header *header,\n */\n int unbundle(struct repository *r, struct bundle_header *header,\n \t int bundle_fd, struct strvec *extra_index_pack_args,\n-\t enum verify_bundle_flags flags);\n+\t enum verify_bundle_flags flags,\n+\t enum unbundle_fsck_flags fsck_flags);\n int list_bundle_refs(struct bundle_header *header,\n \t\tint argc, const char **argv);\n \ndiff --git a/transport.c b/transport.c\nindex 0ad04b77fd2..6799988f10c 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, 0);\n+\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_ALWAYS);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"495696","messageId":"67401d4fbcb3f07d31589bb8ec10060dcb77545e.1716824518.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v3.git.1716824518.gitgitgadget@gmail.com","subject":"[PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-27T15:41:56Z","receivedAt":"2024-05-27T15:42:04Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nCurrently we can use \"transfer.fsckObjects\" or \"fetch.fsckObjects\" to\ncontrol whether to enable checks for broken objects during fetching. But\nthese configs are only acknowledged by `fetch-pack.c:get_pack` and do\nnot make sense when fetching from bundles or using bundle-uris.\n\nThis commit exposed the fetch-then-transfer configuration logic by\nadding a new function `fetch_pack_fsck_objects` in fetch-pack.h. In next\ncommit, this new function will be used by `unbundle` in fetching\nscenarios.\n\nSigned-off-by: Xing Xin \n---\n fetch-pack.c | 18 ++++++++++++------\n fetch-pack.h | 2 ++\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d2aef21add..81a64be6951 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\tstrvec_push(&cmd.args, alternate_shallow_file);\n \t}\n \n-\tif (fetch_fsck_objects >= 0\n-\t ? fetch_fsck_objects\n-\t : transfer_fsck_objects >= 0\n-\t ? transfer_fsck_objects\n-\t : 0)\n-\t\tfsck_objects = 1;\n+\tfsck_objects = fetch_pack_fsck_objects();\n \n \tif (do_keep || args->from_promisor || index_pack_args || fsck_objects) {\n \t\tif (pack_lockfiles || fsck_objects)\n@@ -2046,6 +2041,17 @@ static const struct object_id *iterate_ref_map(void *cb_data)\n \treturn &ref->old_oid;\n }\n \n+int fetch_pack_fsck_objects(void)\n+{\n+\tfetch_pack_setup();\n+\n+\treturn fetch_fsck_objects >= 0\n+\t ? fetch_fsck_objects\n+\t : transfer_fsck_objects >= 0\n+\t ? transfer_fsck_objects\n+\t : 0;\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[],\n \t\t const struct ref *ref,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 6775d265175..38956d9b748 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -101,4 +101,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+int fetch_pack_fsck_objects(void);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"495697","messageId":"c19b8f633cb9f851948b3a17b5425f3310d900d5.1716824518.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v3.git.1716824518.gitgitgadget@gmail.com","subject":"[PATCH v3 4/4] unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-27T15:41:57Z","receivedAt":"2024-05-27T15:42:05Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThis commit adds a new option `UNBUNDLE_FSCK_FOLLOW_FETCH` to\n`unbundle_fsck_flags`, this new flag is currently used in the _fetch_\nprocess by\n\n- `transport.c:fetch_refs_from_bundle` for fetching directly from a\n bundle.\n- `bundle-uri.c:unbundle_from_file` for unbundling bundles downloaded\n from bundle-uri.\n\nSo we now have a relatively consistent logic for checking objects during\nfetching. Add tests for the above two situations are added.\n\nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 2 +-\n bundle.c | 10 +++++++++-\n bundle.h | 1 +\n t/t5558-clone-bundle-uri.sh | 36 +++++++++++++++++++++++++++++++-----\n t/t5607-clone-bundle.sh | 23 +++++++++++++++++++++++\n transport.c | 2 +-\n 6 files changed, 66 insertions(+), 8 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 80f02aac6f1..0da3e5a61b9 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET, UNBUNDLE_FSCK_ALWAYS)))\n+\t\t\t VERIFY_BUNDLE_QUIET, UNBUNDLE_FSCK_FOLLOW_FETCH)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex a922d592782..c7344543aa4 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -17,6 +17,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"connected.h\"\n #include \"write-or-die.h\"\n+#include \"fetch-pack.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -616,6 +617,7 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \t enum unbundle_fsck_flags fsck_flags)\n {\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n+\tint fsck_objects = 0;\n \n \tif (verify_bundle(r, header, flags))\n \t\treturn -1;\n@@ -628,13 +630,19 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \n \tswitch (fsck_flags) {\n \tcase UNBUNDLE_FSCK_ALWAYS:\n-\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\t\tfsck_objects = 1;\n+\t\tbreak;\n+\tcase UNBUNDLE_FSCK_FOLLOW_FETCH:\n+\t\tfsck_objects = fetch_pack_fsck_objects();\n \t\tbreak;\n \tcase UNBUNDLE_FSCK_NEVER:\n \tdefault:\n \t\tbreak;\n \t}\n \n+\tif (fsck_objects)\n+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/bundle.h b/bundle.h\nindex cfa9daddda6..c46488422ce 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -33,6 +33,7 @@ int create_bundle(struct repository *r, const char *path,\n enum unbundle_fsck_flags {\n \tUNBUNDLE_FSCK_NEVER = 0,\n \tUNBUNDLE_FSCK_ALWAYS,\n+\tUNBUNDLE_FSCK_FOLLOW_FETCH,\n };\n \n enum verify_bundle_flags {\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex a5b04d6f187..3df4d44e78f 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -19,13 +19,30 @@ test_expect_success 'fail to clone from non-bundle file' '\n \n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n-\tgit -C clone-from checkout -b topic &&\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b topic &&\n+\n+\t\ttest_commit A &&\n+\t\tgit bundle create A.bundle topic &&\n+\n+\t\ttest_commit B &&\n+\t\tgit bundle create B.bundle topic &&\n+\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n \n-\ttest_commit -C clone-from A &&\n-\tgit -C clone-from bundle create A.bundle topic &&\n+\t\tcommit: this is a commit with bad emails\n \n-\ttest_commit -C clone-from B &&\n-\tgit -C clone-from bundle create B.bundle topic\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad.bundle bad &&\n+\t\tgit update-ref -d refs/heads/bad\n+\t)\n '\n \n test_expect_success 'clone with path bundle' '\n@@ -36,6 +53,15 @@ test_expect_success 'clone with path bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone with bad bundle' '\n+\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad.bundle\" \\\n+\t\tclone-from clone-bad 2>err &&\n+\t# Unbundle fails, but clone can still proceed.\n+\ttest_grep \"missingEmail\" err &&\n+\tgit -C clone-bad for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\ndiff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\nindex 0d1e92d9963..423b35ac237 100755\n--- a/t/t5607-clone-bundle.sh\n+++ b/t/t5607-clone-bundle.sh\n@@ -138,6 +138,29 @@ test_expect_success 'fetch SHA-1 from bundle' '\n \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n '\n \n+test_expect_success 'clone bundle with fetch.fsckObjects' '\n+\ttest_create_repo bundle-fsck &&\n+\t(\n+\t\tcd bundle-fsck &&\n+\t\ttest_commit first &&\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad.bundle bad\n+\t) &&\n+\ttest_must_fail git -c fetch.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fsck-clone 2>err &&\n+\ttest_grep \"missingEmail\" err\n+'\n+\n test_expect_success 'git bundle uses expected default format' '\n \tgit bundle create bundle HEAD^.. &&\n \tcat >expect <<-EOF &&\ndiff --git a/transport.c b/transport.c\nindex 6799988f10c..a140d4b03c0 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_ALWAYS);\n+\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_FOLLOW_FETCH);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n"},{"id":"495698","messageId":"ff380ec.c5ee.18fbacbe47b.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"xmqqh6esjta1.fsf@gitster.g","subject":"Re:Re: [PATCH] bundle-uri: refresh packed_git if unbundle succeed","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-27T16:04:35Z","receivedAt":"2024-05-27T16:05:01Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-21 01:19:02, \"Junio C Hamano\" wrote:\n>\"Xing Xin\" writes:\n>\n>> Personally I think data from bundles and data received via network\n>> should be treated equally.\n>\n>Yup, that is not personal ;-) but is universally accepted as a good\n>discipline. In the case of bundle-uri, the bundle came over the\n>network so it is even more true that they should be treated the\n>same.\n>\n>> For \"fetch-pack\" we now have some configs\n>> such as \"fetch.fsckobjects\" and \"transfer.fsckobjects\" to decide the\n>> behavior, these configs are invisible when we are fetching bundles.\n>\n>When fetching over network, transport.c:fetch_refs_via_pack() calls\n>fetch_pack.c:fetch_pack(), which eventually calls get_pack() and the\n>configuration variables are honored there. It appears that the\n>transport layer is unaware of the .fsckobjects configuration knobs.\n>\n>When fetching from a bundle, transport.c:fetch_refs_from_bundle()\n>calls bundle.c:unbundle(). This function has three callers, i.e.\n>\"git bundle unbundle\", normal fetching from a bundle, and more\n>recently added bundle-uri codepaths. \n>\n>I think one reasonable approach to take is to add an extra parameter\n>that takes one of three values: (never, use-config, always), and\n>conditionally add \"--fsck-objects\" to the command line of the\n>index-pack. Teach \"git bundle unbundle\" the \"--fsck-objects\" option\n>so that it can pass 'never' or 'always' from the command line, and\n>pass 'use-config' from the code paths for normal fetching from a\n>budnle and bundle-uri.\n>\n>To implement use-config, you'd probably need to refactor a small\n>part of fetch-pack.c:get_pack()\n>\n>\tif (fetch_fsck_objects >= 0\n>\t ? fetch_fsck_objects\n>\t : transfer_fsck_objects >= 0\n>\t ? transfer_fsck_objects\n>\t : 0)\n>\t\tfsck_objects = 1;\n>\n>into a public function (to support a caller like unbundle() that\n>comes from sideways, the new function may also need to call\n>fetch_pack_setup() to prime them).\n>\n>A patch series may take a structure like so:\n>\n> * define enum { UNBUNDLE_FSCK_NEVER, UNBUNDLE_FSCK_ALWAYS } in\n> bundle.h, have bundle.c:unbundle() accept a new parameter of that\n> type, and conditionally add \"--fsck-objects\" to its call to\n> \"index-pack\". \"git bundle unbundle\" can pass 'never' to its\n> invocation to unbundle() as an easy way to test it. For the\n> other two callers, we can start by passing 'always'.\n>\n> * (optional) teach \"git bundle unbundle\" a new \"--fsck-objects\"\n> option to allow passing 'always' to its call to unbundle(). With\n> that, add tests to feed it a bundle with questionable objects in\n> it and make sure that unbundling notices.\n\nI just submitted a new series mainly focusing on the unbundle handling during\nfetches. I would like to submit a new one for teaching \"git bundle unbundle\" a\n\"--fsck-objects\" option after this to make changes more targeted.\n\n> * refactor fetch-pack.c:get_pack() to make the fetch-then-transfer\n> configuration logic available to external callers.\n>\n> * Add UNBUNDLE_FSCK_USE_CONFIG to the enum, enhance unbundle() to\n\nI tend to use `UNBUNDLE_FSCK_FOLLOW_FETCH` because this option is only\nused in fetches, though the current implementation is indeed reading configs.\n\n> react to the value by calling the helper function you introduced\n> in the previous step.\n\nThe new patch series is constructed right as you suggested, thanks a lot for\nyour help. \n\nXing Xin\n\n"},{"id":"495742","messageId":"ZlXGI4lkaXcD7_8C@tanuki","threadId":"61479","inReplyTo":"8f488a5eeaaa0cdb525c34c2c165e6ee74ee7691.1716824518.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/4] bundle-uri: verify oid before writing refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-28T11:55:15Z","receivedAt":"2024-05-28T11:55:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 27, 2024 at 03:41:54PM +0000, Xing Xin via GitGitGadget wrote:\n> From: Xing Xin \n[snip]\n> 5. The `verify_bundle` will call `parse_object`, within which the\n> `prepare_packed_git` or `reprepare_packed_git` is eventually called,\n> which means that the `raw_object_store->packed_git` data gets filled\n> in and ``packed_git_initialized` is set. This also means consecutive\n\ns/``/`/\n\n[snip]\n> This commit fixes this bug by dropping the `REF_SKIP_OID_VERIFICATION`\n> flag when writing bundle refs, so we can:\n> \n> 1. Ensure that the bundle refs we are writing are pointing to valid\n> objects.\n> 2. Ensure all the tips from bundle refs can be correctly parsed.\n\nI think one angle that your explanation doesn't cover is why exactly\ndropping the flag fixes the observed issue.\n\n> And a set of negotiation related tests for bundle-uri are added.\n\ns/And/Add/\n\n[snip]\n> +#########################################################################\n> +# Clone negotiation related tests begin here\n> +\n> +test_expect_success 'negotiation: bundle with part of wanted commits' '\n> +\ttest_when_finished rm -rf trace*.txt &&\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n> +\t\tclone-from nego-bundle-part &&\n> +\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/topic\n> +\tEOF\n> +\ttest_cmp expect actual &&\n> +\t# Ensure that refs/bundles/topic are sent as \"have\".\n> +\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n> +'\n\nAs far as I can see there is no test that verifies the case where the\nbundle contains refs, but misses the objects to satisfy the refs. Can we\ncraft such a bundle and exercise this new failure mode?\n\nPatrick\n"},{"id":"495743","messageId":"ZlXIBrs5Q91r_-qU@tanuki","threadId":"61479","inReplyTo":"67401d4fbcb3f07d31589bb8ec10060dcb77545e.1716824518.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-28T12:03:18Z","receivedAt":"2024-05-28T12:03:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 27, 2024 at 03:41:56PM +0000, Xing Xin via GitGitGadget wrote:\n> From: Xing Xin \n> \n> Currently we can use \"transfer.fsckObjects\" or \"fetch.fsckObjects\" to\n> control whether to enable checks for broken objects during fetching. But\n> these configs are only acknowledged by `fetch-pack.c:get_pack` and do\n> not make sense when fetching from bundles or using bundle-uris.\n\nDo they not make sense, or are they not effective? I assume you mean the\nlatter, right?\n\n> This commit exposed the fetch-then-transfer configuration logic by\n\ns/exposed/exposes/\n\n> adding a new function `fetch_pack_fsck_objects` in fetch-pack.h. In next\n> commit, this new function will be used by `unbundle` in fetching\n> scenarios.\n> \n> Signed-off-by: Xing Xin \n> ---\n> fetch-pack.c | 18 ++++++++++++------\n> fetch-pack.h | 2 ++\n> 2 files changed, 14 insertions(+), 6 deletions(-)\n> \n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 7d2aef21add..81a64be6951 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args,\n> \t\tstrvec_push(&cmd.args, alternate_shallow_file);\n> \t}\n> \n> -\tif (fetch_fsck_objects >= 0\n> -\t ? fetch_fsck_objects\n> -\t : transfer_fsck_objects >= 0\n> -\t ? transfer_fsck_objects\n> -\t : 0)\n> -\t\tfsck_objects = 1;\n\nThis statement is really weird to read, but that is certainly not the\nfault of this patch, but...\n\n> +\tfsck_objects = fetch_pack_fsck_objects();\n> \n> \tif (do_keep || args->from_promisor || index_pack_args || fsck_objects) {\n> \t\tif (pack_lockfiles || fsck_objects)\n> @@ -2046,6 +2041,17 @@ static const struct object_id *iterate_ref_map(void *cb_data)\n> \treturn &ref->old_oid;\n> }\n> \n> +int fetch_pack_fsck_objects(void)\n> +{\n> +\tfetch_pack_setup();\n> +\n> +\treturn fetch_fsck_objects >= 0\n> +\t ? fetch_fsck_objects\n> +\t : transfer_fsck_objects >= 0\n> +\t ? transfer_fsck_objects\n> +\t : 0;\n> +}\n\n... can we maybe rewrite it to something more customary here? The\nfollowing is way easier to read, at least for me.\n\n\tint fetch_pack_fsck_objects(void)\n\t{\n\t\tfetch_pack_setup();\n\t\tif (fetch_fsck_objects >= 0 ||\n\t\t transfer_fsck_objects >= 0)\n\t\t\treturn 1;\n\t\treturn 0;\n\t}\n\n> struct ref *fetch_pack(struct fetch_pack_args *args,\n> \t\t int fd[],\n> \t\t const struct ref *ref,\n> diff --git a/fetch-pack.h b/fetch-pack.h\n> index 6775d265175..38956d9b748 100644\n> --- a/fetch-pack.h\n> +++ b/fetch-pack.h\n> @@ -101,4 +101,6 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n> */\n> int report_unmatched_refs(struct ref **sought, int nr_sought);\n> \n> +int fetch_pack_fsck_objects(void);\n\nLet's add a comment here saying what this function does.\n\nPatrick\n"},{"id":"495744","messageId":"ZlXIDXfc_1lGb51J@tanuki","threadId":"61479","inReplyTo":"057c697970ff49301cd9dc6adef099f53d440c3c.1716824518.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/4] unbundle: introduce unbundle_fsck_flags for fsckobjects handling","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-28T12:03:25Z","receivedAt":"2024-05-28T12:03:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 27, 2024 at 03:41:55PM +0000, Xing Xin via GitGitGadget wrote:\n[snip]\n> diff --git a/bundle.h b/bundle.h\n> index 021adbdcbb3..cfa9daddda6 100644\n> --- a/bundle.h\n> +++ b/bundle.h\n> @@ -30,6 +30,11 @@ int create_bundle(struct repository *r, const char *path,\n> \t\t int argc, const char **argv, struct strvec *pack_options,\n> \t\t int version);\n> \n> +enum unbundle_fsck_flags {\n> +\tUNBUNDLE_FSCK_NEVER = 0,\n> +\tUNBUNDLE_FSCK_ALWAYS,\n> +};\n> +\n> enum verify_bundle_flags {\n> \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n> \tVERIFY_BUNDLE_QUIET = (1 << 1),\n\nWouldn't this have been a natural fit for the new flag, e.g. via\nsomething like `VERIFY_BUNDLE_FSCK`?\n\nPatrick\n"},{"id":"495745","messageId":"ZlXInJZn7oDfyvWv@tanuki","threadId":"61479","inReplyTo":"c19b8f633cb9f851948b3a17b5425f3310d900d5.1716824518.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/4] unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-28T12:05:48Z","receivedAt":"2024-05-28T12:05:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 27, 2024 at 03:41:57PM +0000, Xing Xin via GitGitGadget wrote:\n> From: Xing Xin \n> diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\n> index 0d1e92d9963..423b35ac237 100755\n> --- a/t/t5607-clone-bundle.sh\n> +++ b/t/t5607-clone-bundle.sh\n> @@ -138,6 +138,29 @@ test_expect_success 'fetch SHA-1 from bundle' '\n> \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n> '\n> \n> +test_expect_success 'clone bundle with fetch.fsckObjects' '\n> +\ttest_create_repo bundle-fsck &&\n> +\t(\n> +\t\tcd bundle-fsck &&\n> +\t\ttest_commit first &&\n> +\t\tcat >data <<-EOF &&\n> +\t\ttree $(git rev-parse HEAD^{tree})\n> +\t\tparent $(git rev-parse HEAD)\n> +\t\tauthor A U Thor\n> +\t\tcommitter A U Thor\n> +\n> +\t\tcommit: this is a commit with bad emails\n> +\n> +\t\tEOF\n> +\t\tgit hash-object --literally -t commit -w --stdin commit &&\n> +\t\tgit branch bad $(cat commit) &&\n> +\t\tgit bundle create bad.bundle bad\n> +\t) &&\n> +\ttest_must_fail git -c fetch.fsckObjects=true \\\n> +\t\tclone bundle-fsck/bad.bundle bundle-fsck-clone 2>err &&\n> +\ttest_grep \"missingEmail\" err\n> +'\n\nDo we also want to have a test for `transfer.fsckObjects`?\n\nPatrick\n"},{"id":"495763","messageId":"xmqqikyx6ew9.fsf@gitster.g","threadId":"61479","inReplyTo":"ZlXIBrs5Q91r_-qU@tanuki","subject":"Re: [PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-28T17:10:46Z","receivedAt":"2024-05-28T17:10:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> +int fetch_pack_fsck_objects(void)\n>> +{\n>> +\tfetch_pack_setup();\n>> +\n>> +\treturn fetch_fsck_objects >= 0\n>> +\t ? fetch_fsck_objects\n>> +\t : transfer_fsck_objects >= 0\n>> +\t ? transfer_fsck_objects\n>> +\t : 0;\n>> +}\n>\n> ... can we maybe rewrite it to something more customary here? The\n> following is way easier to read, at least for me.\n>\n> \tint fetch_pack_fsck_objects(void)\n> \t{\n> \t\tfetch_pack_setup();\n> \t\tif (fetch_fsck_objects >= 0 ||\n> \t\t transfer_fsck_objects >= 0)\n> \t\t\treturn 1;\n> \t\treturn 0;\n> \t}\n\nBut do they mean the same thing? In a repository where\n\n [fetch] fsckobjects = no\n\nis set, no matter what transfer.fsckobjects says (or left unspecified),\nwe want to return \"no, we are not doing fsck\".\n"},{"id":"495765","messageId":"xmqqbk4p6e98.fsf@gitster.g","threadId":"61479","inReplyTo":"xmqqikyx6ew9.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-28T17:24:35Z","receivedAt":"2024-05-28T17:24:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Patrick Steinhardt writes:\n>\n>>> +int fetch_pack_fsck_objects(void)\n>>> +{\n>>> +\tfetch_pack_setup();\n>>> +\n>>> +\treturn fetch_fsck_objects >= 0\n>>> +\t ? fetch_fsck_objects\n>>> +\t : transfer_fsck_objects >= 0\n>>> +\t ? transfer_fsck_objects\n>>> +\t : 0;\n>>> +}\n>>\n>> ... can we maybe rewrite it to something more customary here? The\n>> following is way easier to read, at least for me.\n>>\n>> \tint fetch_pack_fsck_objects(void)\n>> \t{\n>> \t\tfetch_pack_setup();\n>> \t\tif (fetch_fsck_objects >= 0 ||\n>> \t\t transfer_fsck_objects >= 0)\n>> \t\t\treturn 1;\n>> \t\treturn 0;\n>> \t}\n>\n> But do they mean the same thing? In a repository where\n>\n> [fetch] fsckobjects = no\n>\n> is set, no matter what transfer.fsckobjects says (or left unspecified),\n> we want to return \"no, we are not doing fsck\".\n\nThe original before it was made into a helper function was written\nas a cascade of ?: operators, because it had to be a single\nexpression. As the body of a helper function, we now can sprinkle\nmultiple return statements in it. I think the way that is easiest\nto understand is\n\n\t/* the most specific, if specified */\n\tif (fetch_fsck_objects >= 0)\n\t\treturn fetch_fsck_objects;\n\t/* the less specific, catch-all for both directions */\n\tif (transfer_fsck_objects >= 0)\n \treturn transfer_fsck_objects;\n\t/* the fallback hardcoded default */\n\treturn 0;\n\nwithout the /* comments */.\n"},{"id":"495776","messageId":"ZlbCiVsIwmqEFESo@tanuki","threadId":"61479","inReplyTo":"xmqqikyx6ew9.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-29T05:52:09Z","receivedAt":"2024-05-29T05:52:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 28, 2024 at 10:10:46AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> +int fetch_pack_fsck_objects(void)\n> >> +{\n> >> +\tfetch_pack_setup();\n> >> +\n> >> +\treturn fetch_fsck_objects >= 0\n> >> +\t ? fetch_fsck_objects\n> >> +\t : transfer_fsck_objects >= 0\n> >> +\t ? transfer_fsck_objects\n> >> +\t : 0;\n> >> +}\n> >\n> > ... can we maybe rewrite it to something more customary here? The\n> > following is way easier to read, at least for me.\n> >\n> > \tint fetch_pack_fsck_objects(void)\n> > \t{\n> > \t\tfetch_pack_setup();\n> > \t\tif (fetch_fsck_objects >= 0 ||\n> > \t\t transfer_fsck_objects >= 0)\n> > \t\t\treturn 1;\n> > \t\treturn 0;\n> > \t}\n> \n> But do they mean the same thing? In a repository where\n> \n> [fetch] fsckobjects = no\n> \n> is set, no matter what transfer.fsckobjects says (or left unspecified),\n> we want to return \"no, we are not doing fsck\".\n\nOh, of course they don't. This here would be a faithful conversion:\n\n\tint fetch_pack_fsck_objects(void)\n\t{\n\t\tfetch_pack_setup();\n\t\tif (fetch_fsck_objects >= 0)\n\t\t\treturn fetch_fsck_objects;\n\t\tif (transfer_fsck_objects >= 0)\n\t\t\treturn transfer_fsck_objects;\n\t\treturn 0;\n\t}\n\nStill easier to read in my opinion.\n\nPatrick\n"},{"id":"495777","messageId":"ZlbCtUr9906gq300@tanuki","threadId":"61479","inReplyTo":"xmqqbk4p6e98.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-29T05:52:53Z","receivedAt":"2024-05-29T05:52:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 28, 2024 at 10:24:35AM -0700, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Patrick Steinhardt writes:\n> >\n> >>> +int fetch_pack_fsck_objects(void)\n> >>> +{\n> >>> +\tfetch_pack_setup();\n> >>> +\n> >>> +\treturn fetch_fsck_objects >= 0\n> >>> +\t ? fetch_fsck_objects\n> >>> +\t : transfer_fsck_objects >= 0\n> >>> +\t ? transfer_fsck_objects\n> >>> +\t : 0;\n> >>> +}\n> >>\n> >> ... can we maybe rewrite it to something more customary here? The\n> >> following is way easier to read, at least for me.\n> >>\n> >> \tint fetch_pack_fsck_objects(void)\n> >> \t{\n> >> \t\tfetch_pack_setup();\n> >> \t\tif (fetch_fsck_objects >= 0 ||\n> >> \t\t transfer_fsck_objects >= 0)\n> >> \t\t\treturn 1;\n> >> \t\treturn 0;\n> >> \t}\n> >\n> > But do they mean the same thing? In a repository where\n> >\n> > [fetch] fsckobjects = no\n> >\n> > is set, no matter what transfer.fsckobjects says (or left unspecified),\n> > we want to return \"no, we are not doing fsck\".\n> \n> The original before it was made into a helper function was written\n> as a cascade of ?: operators, because it had to be a single\n> expression. As the body of a helper function, we now can sprinkle\n> multiple return statements in it. I think the way that is easiest\n> to understand is\n> \n> \t/* the most specific, if specified */\n> \tif (fetch_fsck_objects >= 0)\n> \t\treturn fetch_fsck_objects;\n> \t/* the less specific, catch-all for both directions */\n> \tif (transfer_fsck_objects >= 0)\n> \treturn transfer_fsck_objects;\n> \t/* the fallback hardcoded default */\n> \treturn 0;\n> \n> without the /* comments */.\n\nAh, right, didn't see this mail. My revised version looks the same as\nyours, except for the added comments.\n\nPatrick\n"},{"id":"495853","messageId":"1d097be.b642.18fc58dfb70.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZlXIDXfc_1lGb51J@tanuki","subject":"Re:Re: [PATCH v3 2/4] unbundle: introduce unbundle_fsck_flags for fsckobjects handling","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-29T18:12:47Z","receivedAt":"2024-05-29T18:13:08Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-28 20:03:25, \"Patrick Steinhardt\" wrote:\n>On Mon, May 27, 2024 at 03:41:55PM +0000, Xing Xin via GitGitGadget wrote:\n>[snip]\n>> diff --git a/bundle.h b/bundle.h\n>> index 021adbdcbb3..cfa9daddda6 100644\n>> --- a/bundle.h\n>> +++ b/bundle.h\n>> @@ -30,6 +30,11 @@ int create_bundle(struct repository *r, const char *path,\n>> \t\t int argc, const char **argv, struct strvec *pack_options,\n>> \t\t int version);\n>> \n>> +enum unbundle_fsck_flags {\n>> +\tUNBUNDLE_FSCK_NEVER = 0,\n>> +\tUNBUNDLE_FSCK_ALWAYS,\n>> +};\n>> +\n>> enum verify_bundle_flags {\n>> \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n>> \tVERIFY_BUNDLE_QUIET = (1 << 1),\n>\n>Wouldn't this have been a natural fit for the new flag, e.g. via\n>something like `VERIFY_BUNDLE_FSCK`?\n\nIt makes sense to me. Currently, verify_bundle_flags controls the amount\nof information displayed when checking a bundle's prerequisites. The\nnewly added unbundle_fsck_flags is designed to check for broken objects\nduring the unbundle process, which is essentially a form of bundle\nverification. I believe we should extend some object verification\ncapabilities to the git bundle verify command as well, perhaps by adding\na --fsck-objects option.\n\nWith this in mind, I support adding new options to verify_bundle_flags.\nSince bundle.c:unbundle needs to combine multiple options, we must\ndefine new options using bitwise shifting:\n\n\tenum verify_bundle_flags {\n\t\tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n\t\tVERIFY_BUNDLE_QUIET = (1 << 1),\n\t\tVERIFY_BUNDLE_FSCK_OBJECTS_ALWAYS = (1 << 2),\n\t\tVERIFY_BUNDLE_FSCK_OBJECTS_FOLLOW_FETCH = (1 << 3),\n\t};\n\nHow about the naming? I'm not very good at naming :)\n\nXing Xin\n"},{"id":"495884","messageId":"ZlgC2TaXGjcNv2C0@tanuki","threadId":"61479","inReplyTo":"1d097be.b642.18fc58dfb70.Coremail.bupt_xingxin@163.com","subject":"Re: Re: [PATCH v3 2/4] unbundle: introduce unbundle_fsck_flags for fsckobjects handling","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-30T04:38:49Z","receivedAt":"2024-05-30T04:38:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, May 30, 2024 at 02:12:47AM +0800, Xing Xin wrote:\n> At 2024-05-28 20:03:25, \"Patrick Steinhardt\" wrote:\n> >On Mon, May 27, 2024 at 03:41:55PM +0000, Xing Xin via GitGitGadget wrote:\n> >[snip]\n> >> diff --git a/bundle.h b/bundle.h\n> >> index 021adbdcbb3..cfa9daddda6 100644\n> >> --- a/bundle.h\n> >> +++ b/bundle.h\n> >> @@ -30,6 +30,11 @@ int create_bundle(struct repository *r, const char *path,\n> >> \t\t int argc, const char **argv, struct strvec *pack_options,\n> >> \t\t int version);\n> >> \n> >> +enum unbundle_fsck_flags {\n> >> +\tUNBUNDLE_FSCK_NEVER = 0,\n> >> +\tUNBUNDLE_FSCK_ALWAYS,\n> >> +};\n> >> +\n> >> enum verify_bundle_flags {\n> >> \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n> >> \tVERIFY_BUNDLE_QUIET = (1 << 1),\n> >\n> >Wouldn't this have been a natural fit for the new flag, e.g. via\n> >something like `VERIFY_BUNDLE_FSCK`?\n> \n> It makes sense to me. Currently, verify_bundle_flags controls the amount\n> of information displayed when checking a bundle's prerequisites. The\n> newly added unbundle_fsck_flags is designed to check for broken objects\n> during the unbundle process, which is essentially a form of bundle\n> verification. I believe we should extend some object verification\n> capabilities to the git bundle verify command as well, perhaps by adding\n> a --fsck-objects option.\n> \n> With this in mind, I support adding new options to verify_bundle_flags.\n> Since bundle.c:unbundle needs to combine multiple options, we must\n> define new options using bitwise shifting:\n> \n> \tenum verify_bundle_flags {\n> \t\tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n> \t\tVERIFY_BUNDLE_QUIET = (1 << 1),\n> \t\tVERIFY_BUNDLE_FSCK_OBJECTS_ALWAYS = (1 << 2),\n> \t\tVERIFY_BUNDLE_FSCK_OBJECTS_FOLLOW_FETCH = (1 << 3),\n> \t};\n> \n> How about the naming? I'm not very good at naming :)\n\nI later noticed that you extend the `unbundle_fsck_flags` in a later\npatch. With that in mind I don't think it's all that important anymore\nto merge those into the `verify_bundle_flags` as you would otherwise\nallow for weirdness. What happens for example when both `ALWAYS` and\n`FOLLOW_FETCH` are set?\n\nSo feel free to ignore this advice. If you still think it's a good idea\nthen the above naming looks okay to me.\n\nPatrick\n"},{"id":"495905","messageId":"pull.1730.v4.git.1717057290.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v3.git.1716824518.gitgitgadget@gmail.com","subject":"[PATCH v4 0/4] object checking related additions and fixes for bundles in fetches","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-30T08:21:26Z","receivedAt":"2024-05-30T08:21:34Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"While attempting to fix a reference negotiation bug in bundle-uri, we\nidentified that the fetch process lacks some crucial object validation\nchecks when processing bundles. The primary issues are:\n\n 1. In the bundle-uri scenario, object IDs were not validated before writing\n bundle references. This was the root cause of the original negotiation\n bug in bundle-uri and could lead to potential repository corruption.\n 2. The existing \"fetch.fsckObjects\" and \"transfer.fsckObjects\"\n configurations were not applied when directly fetching bundles or\n fetching with bundle-uri enabled. In fact, there were no object\n validation supports for unbundle.\n\nThe first patch addresses the bundle-uri negotiation issue by removing the\nREF_SKIP_OID_VERIFICATION flag when writing bundle references.\n\nPatches 2 through 4 extend verify_bundle_flags for bundle.c:unbundle to add\nsupport for object validation (fsck) in different scenarios, mainly\nfollowing the suggestions from Junio on the mailing list.\n\nXing Xin (4):\n bundle-uri: verify oid before writing refs\n unbundle: extend verify_bundle_flags to support fsck-objects\n fetch-pack: expose fsckObjects configuration logic\n unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH\n\n bundle-uri.c | 5 +-\n bundle.c | 10 ++\n bundle.h | 2 +\n fetch-pack.c | 17 ++--\n fetch-pack.h | 5 +\n t/t5558-clone-bundle-uri.sh | 186 +++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++\n transport.c | 2 +-\n 8 files changed, 246 insertions(+), 14 deletions(-)\n\n\nbase-commit: b9cfe4845cb2562584837bc0101c0ab76490a239\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\nRange-diff vs v3:\n\n 1: 8f488a5eeaa ! 1: e958a3ab20c bundle-uri: verify oid before writing refs\n @@ Commit message\n When using the bundle-uri mechanism with a bundle list containing\n multiple interrelated bundles, we encountered a bug where tips from\n downloaded bundles were not discovered, thus resulting in rather slow\n - clones. This was particularly problematic when employing the heuristic\n - `creationTokens`.\n + clones. This was particularly problematic when employing the\n + \"creationTokens\" heuristic.\n \n - And this is easy to reproduce. Suppose we have a repository with a\n - single branch `main` pointing to commit `A`, firstly we create a base\n - bundle with\n + To reproduce this issue, consider a repository with a single branch\n + \"main\" pointing to commit \"A\". Firstly, create a base bundle with:\n \n git bundle create base.bundle main\n \n - Then let's add a new commit `B` on top of `A`, so that an incremental\n - bundle for `main` can be created with\n + Then, add a new commit \"B\" on top of \"A\", and create an incremental\n + bundle for \"main\":\n \n git bundle create incr.bundle A..main\n \n - Now we can generate a bundle list with the following content:\n + Now, generate a bundle list with the following content:\n \n [bundle]\n version = 1\n @@ Commit message\n uri = incr.bundle\n creationToken = 2\n \n - A fresh clone with the bundle list above would give the expected\n - `refs/bundles/main` pointing at `B` in new repository, in other words we\n - already had everything locally from the bundles, but git would still\n - download everything from server as if we got nothing.\n + A fresh clone with the bundle list above should result in a reference\n + \"refs/bundles/main\" pointing to \"B\" in the new repository. However, git\n + would still download everything from the server, as if it had fetched\n + nothing locally.\n \n - So why the `refs/bundles/main` is not discovered? After some digging I\n + So why the \"refs/bundles/main\" is not discovered? After some digging I\n found that:\n \n 1. Bundles in bundle list are downloaded to local files via\n - `download_bundle_list` or via `fetch_bundles_by_token` for the\n - creationToken heuristic case.\n - 2. Then it tries to unbundle each bundle via `unbundle_from_file`, which\n - is called by `unbundle_all_bundles` or called within\n - `fetch_bundles_by_token` for the creationToken heuristic case.\n - 3. Here, we first read the bundle header to get all the prerequisites\n - for the bundle, this is done in `read_bundle_header`.\n - 4. Then we call `unbundle`, which calls `verify_bundle` to ensure that\n - the repository does indeed contain the prerequisites mentioned in the\n + `bundle-uri.c:download_bundle_list` or via\n + `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n + heuristic.\n + 2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which\n + is called by `bundle-uri.c:unbundle_all_bundles` or called within\n + `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n + heuristic.\n + 3. To get all prerequisites of the bundle, the bundle header is read\n + inside `bundle-uri.c:unbundle_from_file` to by calling\n + `bundle.c:read_bundle_header`.\n + 4. Then it calls `bundle.c:unbundle`, which calls\n + `bundle.c:verify_bundle` to ensure the repository contains all the\n + prerequisites.\n + 5. `bundle.c:verify_bundle` calls `parse_object`, which eventually\n + invokes `packfile.c:prepare_packed_git` or\n + `packfile.c:reprepare_packed_git`, filling\n + `raw_object_store->packed_git` and setting `packed_git_initialized`.\n + 6. If `bundle.c:unbundle` succeeds, it writes refs via\n + `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here\n + bundle refs which can target arbitrary objects are written to the\n + repository.\n + 7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions\n + `fetch-pack.c:mark_complete_and_common_ref` and\n + `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to\n + find local tips for negotiation. The `OBJECT_INFO_QUICK` flag\n + prevents `packfile.c:reprepare_packed_git` from being called,\n + resulting in failures to parse OIDs that reside only in the latest\n bundle.\n - 5. The `verify_bundle` will call `parse_object`, within which the\n - `prepare_packed_git` or `reprepare_packed_git` is eventually called,\n - which means that the `raw_object_store->packed_git` data gets filled\n - in and ``packed_git_initialized` is set. This also means consecutive\n - calls to `prepare_packed_git` doesn't re-initiate\n - `raw_object_store->packed_git` since `packed_git_initialized` already\n - is set.\n - 6. If `unbundle` succeeds, it writes some refs via `refs_update_ref`\n - with `REF_SKIP_OID_VERIFICATION` set. So the bundle refs which can\n - target arbitrary objects are written to the repository.\n - 7. Finally in `do_fetch_pack_v2`, `mark_complete_and_common_ref` and\n - `mark_tips` are called with `OBJECT_INFO_QUICK` set to find local\n - tips. Here it won't call `reprepare_packed_git` anymore so it would\n - fail to parse oids that only reside in the last bundle.\n -\n - Back to the example above, when unbunding `incr.bundle`, `base.pack` is\n - enlisted to `packed_git` bacause of the prerequisites to verify. While\n - we can not find `B` for negotiation at a latter time because `B` exists\n - in `incr.pack` which is not enlisted in `packed_git`.\n -\n - This commit fixes this bug by dropping the `REF_SKIP_OID_VERIFICATION`\n - flag when writing bundle refs, so we can:\n -\n - 1. Ensure that the bundle refs we are writing are pointing to valid\n - objects.\n - 2. Ensure all the tips from bundle refs can be correctly parsed.\n -\n - And a set of negotiation related tests for bundle-uri are added.\n \n + In the example above, when unbunding \"incr.bundle\", \"base.pack\" is added\n + to `packed_git` due to prerequisites verification. However, \"B\" cannot\n + be found for negotiation because it exists in \"incr.pack\", which is not\n + included in `packed_git`.\n +\n + This commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag\n + when writing bundle refs. When `refs.c:refs_update_ref` is called to to\n + write the corresponding bundle refs, it triggers\n + `refs.c:ref_transaction_commit`. This, in turn, invokes\n + `refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of\n + the refs storage backend. For files backend, this function is\n + `files-backend.c:files_transaction_prepare`, and for reftable backend,\n + it is `reftable-backend.c:reftable_be_transaction_prepare`. Both\n + functions eventually call `object.c:parse_object`, which can invoke\n + `packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures\n + that bundle refs point to valid objects and that all tips from bundle\n + refs are correctly parsed during subsequent negotiations.\n +\n + A test has been added to demonstrate that bundles with incorrect\n + headers, where refs point to non-existent objects, do not result in any\n + bundle refs being created in the repository. Additionally, a set of\n + negotiation-related tests for fetching with bundle-uri has been\n + included.\n +\n + Reviewed-by: Karthik Nayak \n + Reviewed-by: Patrick Steinhardt \n Signed-off-by: Xing Xin \n \n ## bundle-uri.c ##\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n \n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'fail to clone from non-bundle file' '\n + \n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n - \tgit -C clone-from checkout -b topic &&\n +-\tgit -C clone-from checkout -b topic &&\n +-\ttest_commit -C clone-from A &&\n +-\ttest_commit -C clone-from B &&\n +-\tgit -C clone-from bundle create B.bundle topic\n ++\t(\n ++\t\tcd clone-from &&\n ++\t\tgit checkout -b topic &&\n ++\n ++\t\ttest_commit A &&\n ++\t\tgit bundle create A.bundle topic &&\n +\n - \ttest_commit -C clone-from A &&\n -+\tgit -C clone-from bundle create A.bundle topic &&\n ++\t\ttest_commit B &&\n ++\t\tgit bundle create B.bundle topic &&\n +\n - \ttest_commit -C clone-from B &&\n - \tgit -C clone-from bundle create B.bundle topic\n ++\t\t# Create a bundle with reference pointing to non-existent object.\n ++\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n ++\t)\n '\n + \n + test_expect_success 'clone with path bundle' '\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with path bundle' '\n + \ttest_cmp expect actual\n + '\n + \n ++test_expect_success 'clone with bundle that has bad header' '\n ++\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n ++\t\tclone-from clone-bad-header 2>err &&\n ++\t# Write bundle ref fails, but clone can still proceed.\n ++\tcommit_b=$(git -C clone-from rev-parse B) &&\n ++\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n ++\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n ++\t! grep \"refs/bundles/\" refs\n ++'\n ++\n + test_expect_success 'clone with path bundle and non-default hash' '\n + \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n + \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n 2: 057c697970f ! 2: beb70735811 unbundle: introduce unbundle_fsck_flags for fsckobjects handling\n @@ Metadata\n Author: Xing Xin \n \n ## Commit message ##\n - unbundle: introduce unbundle_fsck_flags for fsckobjects handling\n + unbundle: extend verify_bundle_flags to support fsck-objects\n \n - This commit adds a new enum `unbundle_fsck_flags` which is designed to\n - control the fsck behavior when unbundling. `unbundle` can use this newly\n - passed in enum to further decide whether to enable `--fsck-objects` for\n - \"git-index-pack\".\n + This commit extends `verify_bundle_flags` by adding a new option\n + `VERIFY_BUNDLE_FSCK_ALWAYS`, which enables checks for broken objects in\n + `bundle.c:unbundle`. This option is now used as the default for fetches\n + involving bundles, specifically by `transport.c:fetch_refs_from_bundle`\n + for direct bundle fetches and by `bundle-uri.c:unbundle_from_file` for\n + _bundle-uri_ enabled fetches.\n \n - Currently only `UNBUNDLE_FSCK_NEVER` and `UNBUNDLE_FSCK_ALWAYS` are\n - supported as the very basic options. Another interesting option would be\n - added in later commits.\n + Upcoming commits will introduce another option as a replacement that\n + fits better with fetch operations. `VERIFY_BUNDLE_FSCK_ALWAYS` will be\n + further used to add \"--fsck-objects\" support for \"git bundle unbundle\"\n + and \"git bundle verify\".\n \n + Reviewed-by: Patrick Steinhardt \n Signed-off-by: Xing Xin \n \n - ## builtin/bundle.c ##\n -@@ builtin/bundle.c: static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n - \t\tstrvec_pushl(&extra_index_pack_args, \"-v\", \"--progress-title\",\n - \t\t\t _(\"Unbundling objects\"), NULL);\n - \tret = !!unbundle(the_repository, &header, bundle_fd,\n --\t\t\t &extra_index_pack_args, 0) ||\n -+\t\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_NEVER) ||\n - \t\tlist_bundle_refs(&header, argc, argv);\n - \tbundle_header_release(&header);\n - cleanup:\n -\n ## bundle-uri.c ##\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n -\t\t\t VERIFY_BUNDLE_QUIET)))\n -+\t\t\t VERIFY_BUNDLE_QUIET, UNBUNDLE_FSCK_ALWAYS)))\n ++\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n \t\treturn 1;\n \n \t/*\n \n ## bundle.c ##\n -@@ bundle.c: int create_bundle(struct repository *r, const char *path,\n - \n - int unbundle(struct repository *r, struct bundle_header *header,\n - \t int bundle_fd, struct strvec *extra_index_pack_args,\n --\t enum verify_bundle_flags flags)\n -+\t enum verify_bundle_flags flags,\n -+\t enum unbundle_fsck_flags fsck_flags)\n - {\n - \tstruct child_process ip = CHILD_PROCESS_INIT;\n - \n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n -+\tswitch (fsck_flags) {\n -+\tcase UNBUNDLE_FSCK_ALWAYS:\n ++\tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n +\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n -+\t\tbreak;\n -+\tcase UNBUNDLE_FSCK_NEVER:\n -+\tdefault:\n -+\t\tbreak;\n -+\t}\n +\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \n ## bundle.h ##\n @@ bundle.h: int create_bundle(struct repository *r, const char *path,\n - \t\t int argc, const char **argv, struct strvec *pack_options,\n - \t\t int version);\n - \n -+enum unbundle_fsck_flags {\n -+\tUNBUNDLE_FSCK_NEVER = 0,\n -+\tUNBUNDLE_FSCK_ALWAYS,\n -+};\n -+\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n -@@ bundle.h: int verify_bundle(struct repository *r, struct bundle_header *header,\n - */\n - int unbundle(struct repository *r, struct bundle_header *header,\n - \t int bundle_fd, struct strvec *extra_index_pack_args,\n --\t enum verify_bundle_flags flags);\n -+\t enum verify_bundle_flags flags,\n -+\t enum unbundle_fsck_flags fsck_flags);\n - int list_bundle_refs(struct bundle_header *header,\n - \t\tint argc, const char **argv);\n ++\tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n + };\n \n + int verify_bundle(struct repository *r, struct bundle_header *header,\n \n ## transport.c ##\n @@ transport.c: static int fetch_refs_from_bundle(struct transport *transport,\n @@ transport.c: static int fetch_refs_from_bundle(struct transport *transport,\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n -\t\t &extra_index_pack_args, 0);\n -+\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_ALWAYS);\n ++\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n 3: 67401d4fbcb ! 3: 5ddc894c2c1 fetch-pack: expose fsckObjects configuration logic\n @@ Metadata\n ## Commit message ##\n fetch-pack: expose fsckObjects configuration logic\n \n - Currently we can use \"transfer.fsckObjects\" or \"fetch.fsckObjects\" to\n - control whether to enable checks for broken objects during fetching. But\n - these configs are only acknowledged by `fetch-pack.c:get_pack` and do\n - not make sense when fetching from bundles or using bundle-uris.\n + Currently, we can use \"transfer.fsckObjects\" and the more specific\n + \"fetch.fsckObjects\" to control checks for broken objects in received\n + packs during fetches. However, these configurations were only\n + acknowledged by `fetch-pack.c:get_pack` and did not take effect in\n + direct bundle fetches and fetches with _bundle-uri_ enabled.\n \n - This commit exposed the fetch-then-transfer configuration logic by\n - adding a new function `fetch_pack_fsck_objects` in fetch-pack.h. In next\n - commit, this new function will be used by `unbundle` in fetching\n - scenarios.\n + This commit exposes the fetch-then-transfer configuration logic by\n + adding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This\n + new function is used to replace the assignment for `fsck_objects` in\n + `fetch-pack.c:get_pack`. In the next commit, it will also be used by\n + `bundle.c:unbundle` to better fit fetching scenarios.\n \n + Helped-by: Junio C Hamano \n + Helped-by: Patrick Steinhardt \n Signed-off-by: Xing Xin \n \n ## fetch-pack.c ##\n @@ fetch-pack.c: static const struct object_id *iterate_ref_map(void *cb_data)\n +int fetch_pack_fsck_objects(void)\n +{\n +\tfetch_pack_setup();\n -+\n -+\treturn fetch_fsck_objects >= 0\n -+\t ? fetch_fsck_objects\n -+\t : transfer_fsck_objects >= 0\n -+\t ? transfer_fsck_objects\n -+\t : 0;\n ++\tif (fetch_fsck_objects >= 0)\n ++\t\treturn fetch_fsck_objects;\n ++\tif (transfer_fsck_objects >= 0)\n ++\t\treturn transfer_fsck_objects;\n ++\treturn 0;\n +}\n +\n struct ref *fetch_pack(struct fetch_pack_args *args,\n @@ fetch-pack.h: void negotiate_using_fetch(const struct oid_array *negotiation_tip\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n ++/*\n ++ * Return true if checks for broken objects in received pack are required.\n ++ */\n +int fetch_pack_fsck_objects(void);\n +\n #endif\n 4: c19b8f633cb ! 4: 68b9bca9f8b unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH\n @@ Metadata\n Author: Xing Xin \n \n ## Commit message ##\n - unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH\n + unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH\n \n - This commit adds a new option `UNBUNDLE_FSCK_FOLLOW_FETCH` to\n - `unbundle_fsck_flags`, this new flag is currently used in the _fetch_\n - process by\n + This commit introduces a new option `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` to\n + `verify_bundle_flags`. In `bundle.c:unbundle`, this new option controls\n + whether broken object checks should be enabled by invoking\n + `fetch-pack.c:fetch_pack_fsck_objects`. Note that the option\n + `VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over\n + `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`.\n \n - - `transport.c:fetch_refs_from_bundle` for fetching directly from a\n - bundle.\n - - `bundle-uri.c:unbundle_from_file` for unbundling bundles downloaded\n - from bundle-uri.\n + This flag is now used in the fetching process by:\n \n - So we now have a relatively consistent logic for checking objects during\n - fetching. Add tests for the above two situations are added.\n + - `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n + - `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n +\n + This addition ensures a consistent logic for object verification during\n + fetch operations. Tests have been added to confirm functionality in the\n + scenarios mentioned above.\n \n Signed-off-by: Xing Xin \n \n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n --\t\t\t VERIFY_BUNDLE_QUIET, UNBUNDLE_FSCK_ALWAYS)))\n -+\t\t\t VERIFY_BUNDLE_QUIET, UNBUNDLE_FSCK_FOLLOW_FETCH)))\n +-\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n ++\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n \t\treturn 1;\n \n \t/*\n @@ bundle.c\n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n - \t enum unbundle_fsck_flags fsck_flags)\n + \t enum verify_bundle_flags flags)\n {\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n +\tint fsck_objects = 0;\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \tif (verify_bundle(r, header, flags))\n \t\treturn -1;\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n + \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n - \tswitch (fsck_flags) {\n - \tcase UNBUNDLE_FSCK_ALWAYS:\n --\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n + \tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n +\t\tfsck_objects = 1;\n -+\t\tbreak;\n -+\tcase UNBUNDLE_FSCK_FOLLOW_FETCH:\n ++\telse if (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n +\t\tfsck_objects = fetch_pack_fsck_objects();\n - \t\tbreak;\n - \tcase UNBUNDLE_FSCK_NEVER:\n - \tdefault:\n - \t\tbreak;\n - \t}\n - \n -+\tif (fsck_objects)\n -+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n +\n ++\tif (fsck_objects)\n + \t\tstrvec_push(&ip.args, \"--fsck-objects\");\n + \n \tif (extra_index_pack_args) {\n - \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n - \t\tstrvec_clear(extra_index_pack_args);\n \n ## bundle.h ##\n -@@ bundle.h: int create_bundle(struct repository *r, const char *path,\n - enum unbundle_fsck_flags {\n - \tUNBUNDLE_FSCK_NEVER = 0,\n - \tUNBUNDLE_FSCK_ALWAYS,\n -+\tUNBUNDLE_FSCK_FOLLOW_FETCH,\n +@@ bundle.h: enum verify_bundle_flags {\n + \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n + \tVERIFY_BUNDLE_QUIET = (1 << 1),\n + \tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n ++\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 3),\n };\n \n - enum verify_bundle_flags {\n + int verify_bundle(struct repository *r, struct bundle_header *header,\n \n ## t/t5558-clone-bundle-uri.sh ##\n -@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'fail to clone from non-bundle file' '\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'create bundle' '\n + \t\tgit bundle create B.bundle topic &&\n \n - test_expect_success 'create bundle' '\n - \tgit init clone-from &&\n --\tgit -C clone-from checkout -b topic &&\n -+\t(\n -+\t\tcd clone-from &&\n -+\t\tgit checkout -b topic &&\n -+\n -+\t\ttest_commit A &&\n -+\t\tgit bundle create A.bundle topic &&\n -+\n -+\t\ttest_commit B &&\n -+\t\tgit bundle create B.bundle topic &&\n + \t\t# Create a bundle with reference pointing to non-existent object.\n +-\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n ++\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle &&\n +\n +\t\tcat >data <<-EOF &&\n +\t\ttree $(git rev-parse HEAD^{tree})\n +\t\tparent $(git rev-parse HEAD)\n +\t\tauthor A U Thor\n +\t\tcommitter A U Thor\n - \n --\ttest_commit -C clone-from A &&\n --\tgit -C clone-from bundle create A.bundle topic &&\n ++\n +\t\tcommit: this is a commit with bad emails\n - \n --\ttest_commit -C clone-from B &&\n --\tgit -C clone-from bundle create B.bundle topic\n ++\n +\t\tEOF\n +\t\tgit hash-object --literally -t commit -w --stdin commit &&\n +\t\tgit branch bad $(cat commit) &&\n -+\t\tgit bundle create bad.bundle bad &&\n ++\t\tgit bundle create bad-object.bundle bad &&\n +\t\tgit update-ref -d refs/heads/bad\n -+\t)\n + \t)\n '\n \n - test_expect_success 'clone with path bundle' '\n -@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with path bundle' '\n - \ttest_cmp expect actual\n +@@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with bundle that has bad header' '\n + \t! grep \"refs/bundles/\" refs\n '\n \n -+test_expect_success 'clone with bad bundle' '\n -+\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad.bundle\" \\\n -+\t\tclone-from clone-bad 2>err &&\n -+\t# Unbundle fails, but clone can still proceed.\n ++test_expect_success 'clone with bundle that has bad object' '\n ++\t# Unbundle succeeds if no fsckObjects confugured.\n ++\tgit clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n ++\t\tclone-from clone-bad-object-no-fsck &&\n ++\tgit -C clone-bad-object-no-fsck for-each-ref --format=\"%(refname)\" >refs &&\n ++\tgrep \"refs/bundles/\" refs >actual &&\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/bundles/bad\n ++\tEOF\n ++\ttest_cmp expect actual &&\n ++\n ++\t# Unbundle fails with fsckObjects set true, but clone can still proceed.\n ++\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n ++\t\tclone-from clone-bad-object-fsck 2>err &&\n +\ttest_grep \"missingEmail\" err &&\n -+\tgit -C clone-bad for-each-ref --format=\"%(refname)\" >refs &&\n ++\tgit -C clone-bad-object-fsck for-each-ref --format=\"%(refname)\" >refs &&\n +\t! grep \"refs/bundles/\" refs\n +'\n +\n @@ t/t5607-clone-bundle.sh: test_expect_success 'fetch SHA-1 from bundle' '\n \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n '\n \n -+test_expect_success 'clone bundle with fetch.fsckObjects' '\n ++test_expect_success 'clone bundle with different fsckObjects configurations' '\n +\ttest_create_repo bundle-fsck &&\n +\t(\n +\t\tcd bundle-fsck &&\n @@ t/t5607-clone-bundle.sh: test_expect_success 'fetch SHA-1 from bundle' '\n +\t\tgit branch bad $(cat commit) &&\n +\t\tgit bundle create bad.bundle bad\n +\t) &&\n ++\n ++\tgit clone bundle-fsck/bad.bundle bundle-no-fsck &&\n ++\n ++\tgit -c fetch.fsckObjects=false -c transfer.fsckObjects=true \\\n ++\t\tclone bundle-fsck/bad.bundle bundle-fetch-no-fsck &&\n ++\n +\ttest_must_fail git -c fetch.fsckObjects=true \\\n -+\t\tclone bundle-fsck/bad.bundle bundle-fsck-clone 2>err &&\n ++\t\tclone bundle-fsck/bad.bundle bundle-fetch-fsck 2>err &&\n ++\ttest_grep \"missingEmail\" err &&\n ++\n ++\ttest_must_fail git -c transfer.fsckObjects=true \\\n ++\t\tclone bundle-fsck/bad.bundle bundle-transfer-fsck 2>err &&\n +\ttest_grep \"missingEmail\" err\n +'\n +\n @@ transport.c: static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n --\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_ALWAYS);\n -+\t\t &extra_index_pack_args, 0, UNBUNDLE_FSCK_FOLLOW_FETCH);\n +-\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS);\n ++\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n\n-- \ngitgitgadget\n"},{"id":"495906","messageId":"e958a3ab20c0c06f52a00038f39605f14302032a.1717057290.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v4.git.1717057290.gitgitgadget@gmail.com","subject":"[PATCH v4 1/4] bundle-uri: verify oid before writing refs","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-30T08:21:27Z","receivedAt":"2024-05-30T08:21:35Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not discovered, thus resulting in rather slow\nclones. This was particularly problematic when employing the\n\"creationTokens\" heuristic.\n\nTo reproduce this issue, consider a repository with a single branch\n\"main\" pointing to commit \"A\". Firstly, create a base bundle with:\n\n git bundle create base.bundle main\n\nThen, add a new commit \"B\" on top of \"A\", and create an incremental\nbundle for \"main\":\n\n git bundle create incr.bundle A..main\n\nNow, generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above should result in a reference\n\"refs/bundles/main\" pointing to \"B\" in the new repository. However, git\nwould still download everything from the server, as if it had fetched\nnothing locally.\n\nSo why the \"refs/bundles/main\" is not discovered? After some digging I\nfound that:\n\n1. Bundles in bundle list are downloaded to local files via\n `bundle-uri.c:download_bundle_list` or via\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which\n is called by `bundle-uri.c:unbundle_all_bundles` or called within\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n3. To get all prerequisites of the bundle, the bundle header is read\n inside `bundle-uri.c:unbundle_from_file` to by calling\n `bundle.c:read_bundle_header`.\n4. Then it calls `bundle.c:unbundle`, which calls\n `bundle.c:verify_bundle` to ensure the repository contains all the\n prerequisites.\n5. `bundle.c:verify_bundle` calls `parse_object`, which eventually\n invokes `packfile.c:prepare_packed_git` or\n `packfile.c:reprepare_packed_git`, filling\n `raw_object_store->packed_git` and setting `packed_git_initialized`.\n6. If `bundle.c:unbundle` succeeds, it writes refs via\n `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here\n bundle refs which can target arbitrary objects are written to the\n repository.\n7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions\n `fetch-pack.c:mark_complete_and_common_ref` and\n `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to\n find local tips for negotiation. The `OBJECT_INFO_QUICK` flag\n prevents `packfile.c:reprepare_packed_git` from being called,\n resulting in failures to parse OIDs that reside only in the latest\n bundle.\n\nIn the example above, when unbunding \"incr.bundle\", \"base.pack\" is added\nto `packed_git` due to prerequisites verification. However, \"B\" cannot\nbe found for negotiation because it exists in \"incr.pack\", which is not\nincluded in `packed_git`.\n\nThis commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag\nwhen writing bundle refs. When `refs.c:refs_update_ref` is called to to\nwrite the corresponding bundle refs, it triggers\n`refs.c:ref_transaction_commit`. This, in turn, invokes\n`refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of\nthe refs storage backend. For files backend, this function is\n`files-backend.c:files_transaction_prepare`, and for reftable backend,\nit is `reftable-backend.c:reftable_be_transaction_prepare`. Both\nfunctions eventually call `object.c:parse_object`, which can invoke\n`packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures\nthat bundle refs point to valid objects and that all tips from bundle\nrefs are correctly parsed during subsequent negotiations.\n\nA test has been added to demonstrate that bundles with incorrect\nheaders, where refs point to non-existent objects, do not result in any\nbundle refs being created in the repository. Additionally, a set of\nnegotiation-related tests for fetching with bundle-uri has been\nincluded.\n\nReviewed-by: Karthik Nayak \nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 +-\n t/t5558-clone-bundle-uri.sh | 153 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 150 insertions(+), 6 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91b3319a5c1..65666a11d9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\trefs_update_ref(get_main_ref_store(the_repository),\n \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n \t\t\t\thas_old ? &old_oid : NULL,\n-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n+\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n \t}\n \n \tbundle_header_release(&header);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..8f4f802e4f1 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -19,10 +19,19 @@ test_expect_success 'fail to clone from non-bundle file' '\n \n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n-\tgit -C clone-from checkout -b topic &&\n-\ttest_commit -C clone-from A &&\n-\ttest_commit -C clone-from B &&\n-\tgit -C clone-from bundle create B.bundle topic\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b topic &&\n+\n+\t\ttest_commit A &&\n+\t\tgit bundle create A.bundle topic &&\n+\n+\t\ttest_commit B &&\n+\t\tgit bundle create B.bundle topic &&\n+\n+\t\t# Create a bundle with reference pointing to non-existent object.\n+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n+\t)\n '\n \n test_expect_success 'clone with path bundle' '\n@@ -33,6 +42,16 @@ test_expect_success 'clone with path bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone with bundle that has bad header' '\n+\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n+\t\tclone-from clone-bad-header 2>err &&\n+\t# Write bundle ref fails, but clone can still proceed.\n+\tcommit_b=$(git -C clone-from rev-parse B) &&\n+\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n+\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\n@@ -259,6 +278,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+#########################################################################\n+# Clone negotiation related tests begin here\n+\n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n-- \ngitgitgadget\n\n"},{"id":"495907","messageId":"beb7073581123c1a9096d466f1ccff7fa68f3a19.1717057290.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v4.git.1717057290.gitgitgadget@gmail.com","subject":"[PATCH v4 2/4] unbundle: extend verify_bundle_flags to support fsck-objects","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-30T08:21:28Z","receivedAt":"2024-05-30T08:21:35Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThis commit extends `verify_bundle_flags` by adding a new option\n`VERIFY_BUNDLE_FSCK_ALWAYS`, which enables checks for broken objects in\n`bundle.c:unbundle`. This option is now used as the default for fetches\ninvolving bundles, specifically by `transport.c:fetch_refs_from_bundle`\nfor direct bundle fetches and by `bundle-uri.c:unbundle_from_file` for\n_bundle-uri_ enabled fetches.\n\nUpcoming commits will introduce another option as a replacement that\nfits better with fetch operations. `VERIFY_BUNDLE_FSCK_ALWAYS` will be\nfurther used to add \"--fsck-objects\" support for \"git bundle unbundle\"\nand \"git bundle verify\".\n\nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 2 +-\n bundle.c | 3 +++\n bundle.h | 1 +\n transport.c | 2 +-\n 4 files changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 65666a11d9c..066ff788104 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 95367c2d0a0..26574e74bdd 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -625,6 +625,9 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n+\tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/bundle.h b/bundle.h\nindex 021adbdcbb3..cf23c8615d3 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -33,6 +33,7 @@ int create_bundle(struct repository *r, const char *path,\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n+\tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\ndiff --git a/transport.c b/transport.c\nindex 0ad04b77fd2..1b3d61ffcec 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, 0);\n+\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"495908","messageId":"5ddc894c2c16db109ef96a07d8a2b623b855d95d.1717057290.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v4.git.1717057290.gitgitgadget@gmail.com","subject":"[PATCH v4 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-30T08:21:29Z","receivedAt":"2024-05-30T08:21:37Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nCurrently, we can use \"transfer.fsckObjects\" and the more specific\n\"fetch.fsckObjects\" to control checks for broken objects in received\npacks during fetches. However, these configurations were only\nacknowledged by `fetch-pack.c:get_pack` and did not take effect in\ndirect bundle fetches and fetches with _bundle-uri_ enabled.\n\nThis commit exposes the fetch-then-transfer configuration logic by\nadding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This\nnew function is used to replace the assignment for `fsck_objects` in\n`fetch-pack.c:get_pack`. In the next commit, it will also be used by\n`bundle.c:unbundle` to better fit fetching scenarios.\n\nHelped-by: Junio C Hamano \nHelped-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n fetch-pack.c | 17 +++++++++++------\n fetch-pack.h | 5 +++++\n 2 files changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d2aef21add..3acff2baf09 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\tstrvec_push(&cmd.args, alternate_shallow_file);\n \t}\n \n-\tif (fetch_fsck_objects >= 0\n-\t ? fetch_fsck_objects\n-\t : transfer_fsck_objects >= 0\n-\t ? transfer_fsck_objects\n-\t : 0)\n-\t\tfsck_objects = 1;\n+\tfsck_objects = fetch_pack_fsck_objects();\n \n \tif (do_keep || args->from_promisor || index_pack_args || fsck_objects) {\n \t\tif (pack_lockfiles || fsck_objects)\n@@ -2046,6 +2041,16 @@ static const struct object_id *iterate_ref_map(void *cb_data)\n \treturn &ref->old_oid;\n }\n \n+int fetch_pack_fsck_objects(void)\n+{\n+\tfetch_pack_setup();\n+\tif (fetch_fsck_objects >= 0)\n+\t\treturn fetch_fsck_objects;\n+\tif (transfer_fsck_objects >= 0)\n+\t\treturn transfer_fsck_objects;\n+\treturn 0;\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[],\n \t\t const struct ref *ref,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 6775d265175..b5c579cdae2 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -101,4 +101,9 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+/*\n+ * Return true if checks for broken objects in received pack are required.\n+ */\n+int fetch_pack_fsck_objects(void);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"495909","messageId":"68b9bca9f8b19897997c2adc9a278ac5052e75cd.1717057290.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v4.git.1717057290.gitgitgadget@gmail.com","subject":"[PATCH v4 4/4] unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-05-30T08:21:30Z","receivedAt":"2024-05-30T08:21:37Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThis commit introduces a new option `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` to\n`verify_bundle_flags`. In `bundle.c:unbundle`, this new option controls\nwhether broken object checks should be enabled by invoking\n`fetch-pack.c:fetch_pack_fsck_objects`. Note that the option\n`VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over\n`VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`.\n\nThis flag is now used in the fetching process by:\n\n- `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n- `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n\nThis addition ensures a consistent logic for object verification during\nfetch operations. Tests have been added to confirm functionality in the\nscenarios mentioned above.\n\nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 2 +-\n bundle.c | 7 +++++++\n bundle.h | 1 +\n t/t5558-clone-bundle-uri.sh | 35 ++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++++++++++++++++++++++++++++\n transport.c | 2 +-\n 6 files changed, 77 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 066ff788104..e7ebac6ce57 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n+\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 26574e74bdd..53ac73834ea 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -17,6 +17,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"connected.h\"\n #include \"write-or-die.h\"\n+#include \"fetch-pack.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -615,6 +616,7 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \t enum verify_bundle_flags flags)\n {\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n+\tint fsck_objects = 0;\n \n \tif (verify_bundle(r, header, flags))\n \t\treturn -1;\n@@ -626,6 +628,11 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n \tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n+\t\tfsck_objects = 1;\n+\telse if (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n+\t\tfsck_objects = fetch_pack_fsck_objects();\n+\n+\tif (fsck_objects)\n \t\tstrvec_push(&ip.args, \"--fsck-objects\");\n \n \tif (extra_index_pack_args) {\ndiff --git a/bundle.h b/bundle.h\nindex cf23c8615d3..a39d8ea1a7e 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -34,6 +34,7 @@ enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n \tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n+\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 3),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 8f4f802e4f1..48be1b18802 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -30,7 +30,21 @@ test_expect_success 'create bundle' '\n \t\tgit bundle create B.bundle topic &&\n \n \t\t# Create a bundle with reference pointing to non-existent object.\n-\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle &&\n+\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad-object.bundle bad &&\n+\t\tgit update-ref -d refs/heads/bad\n \t)\n '\n \n@@ -52,6 +66,25 @@ test_expect_success 'clone with bundle that has bad header' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+test_expect_success 'clone with bundle that has bad object' '\n+\t# Unbundle succeeds if no fsckObjects confugured.\n+\tgit clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-no-fsck &&\n+\tgit -C clone-bad-object-no-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/bad\n+\tEOF\n+\ttest_cmp expect actual &&\n+\n+\t# Unbundle fails with fsckObjects set true, but clone can still proceed.\n+\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\tgit -C clone-bad-object-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\ndiff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\nindex 0d1e92d9963..5182efc0b45 100755\n--- a/t/t5607-clone-bundle.sh\n+++ b/t/t5607-clone-bundle.sh\n@@ -138,6 +138,39 @@ test_expect_success 'fetch SHA-1 from bundle' '\n \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n '\n \n+test_expect_success 'clone bundle with different fsckObjects configurations' '\n+\ttest_create_repo bundle-fsck &&\n+\t(\n+\t\tcd bundle-fsck &&\n+\t\ttest_commit first &&\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad.bundle bad\n+\t) &&\n+\n+\tgit clone bundle-fsck/bad.bundle bundle-no-fsck &&\n+\n+\tgit -c fetch.fsckObjects=false -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-no-fsck &&\n+\n+\ttest_must_fail git -c fetch.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\n+\ttest_must_fail git -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-transfer-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err\n+'\n+\n test_expect_success 'git bundle uses expected default format' '\n \tgit bundle create bundle HEAD^.. &&\n \tcat >expect <<-EOF &&\ndiff --git a/transport.c b/transport.c\nindex 1b3d61ffcec..6cd5683bb45 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS);\n+\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n"},{"id":"495911","messageId":"c990a2f.8915.18fc8a0cc48.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZlXGI4lkaXcD7_8C@tanuki","subject":"Re:Re: [PATCH v3 1/4] bundle-uri: verify oid before writing refs","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-30T08:32:12Z","receivedAt":"2024-05-30T08:32:33Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-28 19:55:15, \"Patrick Steinhardt\" wrote:\n[snip]\n>> which means that the `raw_object_store->packed_git` data gets filled\n>> in and ``packed_git_initialized` is set. This also means consecutive\n>\n>s/``/`/\n\nCopy that.\n\n[snip]\n>\n>> And a set of negotiation related tests for bundle-uri are added.\n>\n>s/And/Add/\n\nThe \"And\" is correct here, though I changed the commit message in the new\nversion. :)\n\n\n[snip]\n>As far as I can see there is no test that verifies the case where the\n>bundle contains refs, but misses the objects to satisfy the refs. Can we\n>craft such a bundle and exercise this new failure mode?\n\nA new test is added in [PATCH v4 1/4], which shows it correctly refuses to\ntarget a bundle reference to a non-existent object.\n\nXing Xin"},{"id":"495913","messageId":"4d855732.8dfe.18fc8ae32a0.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZlgC2TaXGjcNv2C0@tanuki","subject":"Re:Re: Re: [PATCH v3 2/4] unbundle: introduce unbundle_fsck_flags for fsckobjects handling","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-30T08:46:50Z","receivedAt":"2024-05-30T08:47:07Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-30 12:38:49, \"Patrick Steinhardt\" wrote:\n[snip]\n>> >\n>> >Wouldn't this have been a natural fit for the new flag, e.g. via\n>> >something like `VERIFY_BUNDLE_FSCK`?\n>> \n>> It makes sense to me. Currently, verify_bundle_flags controls the amount\n>> of information displayed when checking a bundle's prerequisites. The\n>> newly added unbundle_fsck_flags is designed to check for broken objects\n>> during the unbundle process, which is essentially a form of bundle\n>> verification. I believe we should extend some object verification\n>> capabilities to the git bundle verify command as well, perhaps by adding\n>> a --fsck-objects option.\n>> \n>> With this in mind, I support adding new options to verify_bundle_flags.\n>> Since bundle.c:unbundle needs to combine multiple options, we must\n>> define new options using bitwise shifting:\n>> \n>> \tenum verify_bundle_flags {\n>> \t\tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n>> \t\tVERIFY_BUNDLE_QUIET = (1 << 1),\n>> \t\tVERIFY_BUNDLE_FSCK_OBJECTS_ALWAYS = (1 << 2),\n>> \t\tVERIFY_BUNDLE_FSCK_OBJECTS_FOLLOW_FETCH = (1 << 3),\n>> \t};\n>> \n>> How about the naming? I'm not very good at naming :)\n>\n>I later noticed that you extend the `unbundle_fsck_flags` in a later\n>patch. With that in mind I don't think it's all that important anymore\n>to merge those into the `verify_bundle_flags` as you would otherwise\n>allow for weirdness. What happens for example when both `ALWAYS` and\n>`FOLLOW_FETCH` are set?\n>\n>So feel free to ignore this advice. If you still think it's a good idea\n>then the above naming looks okay to me.\n\nWith the idea of extending \"--fsck-objects\" support for \"git bundle verify\" and\n\"git bundle unbundle\", I prefer to grouping these options together. Especially\nin the \"git bundle verify\" scenario, adding a new parameter like\n`unbundle_fsck_flags` for `bundle.c:verify_bundle` is confusing.\n\nXing Xin"},{"id":"495914","messageId":"5cb91961.8ebd.18fc8b01748.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"xmqqbk4p6e98.fsf@gitster.g","subject":"Re:Re: [PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-30T08:48:54Z","receivedAt":"2024-05-30T08:49:20Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-29 01:24:35, \"Junio C Hamano\" wrote:\n[snip]\n>The original before it was made into a helper function was written\n>as a cascade of ?: operators, because it had to be a single\n>expression. As the body of a helper function, we now can sprinkle\n>multiple return statements in it. I think the way that is easiest\n>to understand is\n>\n>\t/* the most specific, if specified */\n>\tif (fetch_fsck_objects >= 0)\n>\t\treturn fetch_fsck_objects;\n>\t/* the less specific, catch-all for both directions */\n>\tif (transfer_fsck_objects >= 0)\n> \treturn transfer_fsck_objects;\n>\t/* the fallback hardcoded default */\n>\treturn 0;\n>\n>without the /* comments */.\n\nApplied, thanks!\n\nXing Xin\n"},{"id":"495915","messageId":"501f41c2.904b.18fc8b4c3e9.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZlXInJZn7oDfyvWv@tanuki","subject":"Re:Re: [PATCH v3 4/4] unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-05-30T08:54:00Z","receivedAt":"2024-05-30T08:54:15Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-05-28 20:05:48, \"Patrick Steinhardt\" wrote:\n[snip]\n>> \n>> +test_expect_success 'clone bundle with fetch.fsckObjects' '\n>> +\ttest_create_repo bundle-fsck &&\n>> +\t(\n>> +\t\tcd bundle-fsck &&\n>> +\t\ttest_commit first &&\n>> +\t\tcat >data <<-EOF &&\n>> +\t\ttree $(git rev-parse HEAD^{tree})\n>> +\t\tparent $(git rev-parse HEAD)\n>> +\t\tauthor A U Thor\n>> +\t\tcommitter A U Thor\n>> +\n>> +\t\tcommit: this is a commit with bad emails\n>> +\n>> +\t\tEOF\n>> +\t\tgit hash-object --literally -t commit -w --stdin commit &&\n>> +\t\tgit branch bad $(cat commit) &&\n>> +\t\tgit bundle create bad.bundle bad\n>> +\t) &&\n>> +\ttest_must_fail git -c fetch.fsckObjects=true \\\n>> +\t\tclone bundle-fsck/bad.bundle bundle-fsck-clone 2>err &&\n>> +\ttest_grep \"missingEmail\" err\n>> +'\n>\n>Do we also want to have a test for `transfer.fsckObjects`?\n\nSure, some more combinations of \"fetch.fsckObjects\" and \"transfer.fsckObjects\"\nare added in tests.\n\nXing Xin\n"},{"id":"496519","messageId":"ZmGmUcO8OYuPnH9H@tanuki","threadId":"61479","inReplyTo":"beb7073581123c1a9096d466f1ccff7fa68f3a19.1717057290.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/4] unbundle: extend verify_bundle_flags to support fsck-objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-06-06T12:06:41Z","receivedAt":"2024-06-06T12:06:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, May 30, 2024 at 08:21:28AM +0000, Xing Xin via GitGitGadget wrote:\n> From: Xing Xin \n\nTiny nit: the important change in this commit is not that you wire up\nthe flag, but rather that we start to execute git-fsck(1) now. I'd thus\npropose to adapt the commit title accordingly.\n\nPatrick\n"},{"id":"496520","messageId":"ZmGmV8-QNMN643ou@tanuki","threadId":"61479","inReplyTo":"68b9bca9f8b19897997c2adc9a278ac5052e75cd.1717057290.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 4/4] unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-06-06T12:06:47Z","receivedAt":"2024-06-06T12:06:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, May 30, 2024 at 08:21:30AM +0000, Xing Xin via GitGitGadget wrote:\n> From: Xing Xin \n\nSame here, the important part is not that we introduce the flag, but\nthat we start using it in `unbundle_from_file()`.\n\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 066ff788104..e7ebac6ce57 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n> \t * the prerequisite commits.\n> \t */\n> \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n> -\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n> +\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n> \t\treturn 1;\n> \n> \t/*\n\nOne thing that is a bit weird is that we first change `unbundle()` to\nuse `FSCK_ALWAYS` in a preceding patch, and then convert it to use\n`FSCK_FOLLOW_FETCH` in the same series. It could be restructured a bit\nto first introduce the flags, only, while not modifying any of the\ncallsites yet. Passing the respective flags would then be done in a\nseparate commit.\n\nPatrick\n"},{"id":"496827","messageId":"pull.1730.v5.git.1718088126.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v4.git.1717057290.gitgitgadget@gmail.com","subject":"[PATCH v5 0/4] object checking related additions and fixes for bundles in fetches","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T06:42:02Z","receivedAt":"2024-06-11T06:42:10Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"While attempting to fix a reference negotiation bug in bundle-uri, we\nidentified that the fetch process lacks some crucial object validation\nchecks when processing bundles. The primary issues are:\n\n 1. In the bundle-uri scenario, object IDs were not validated before writing\n bundle references. This was the root cause of the original negotiation\n bug in bundle-uri and could lead to potential repository corruption.\n 2. The existing \"fetch.fsckObjects\" and \"transfer.fsckObjects\"\n configurations were not applied when directly fetching bundles or\n fetching with bundle-uri enabled. In fact, there were no object\n validation supports for unbundle.\n\nThe first patch addresses the bundle-uri negotiation issue by removing the\nREF_SKIP_OID_VERIFICATION flag when writing bundle references.\n\nPatches 2 through 4 extend verify_bundle_flags for bundle.c:unbundle to add\nsupport for object validation (fsck) in different scenarios, mainly\nfollowing the suggestions from Junio on the mailing list.\n\nXing Xin (4):\n bundle-uri: verify oid before writing refs\n fetch-pack: expose fsckObjects configuration logic\n unbundle: extend options to support object verification\n unbundle: use VERIFY_BUNDLE_FSCK_FOLLOW_FETCH for fetches\n\n bundle-uri.c | 5 +-\n bundle.c | 10 ++\n bundle.h | 2 +\n fetch-pack.c | 17 ++--\n fetch-pack.h | 5 +\n t/t5558-clone-bundle-uri.sh | 186 +++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++\n transport.c | 2 +-\n 8 files changed, 246 insertions(+), 14 deletions(-)\n\n\nbase-commit: b9cfe4845cb2562584837bc0101c0ab76490a239\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\nRange-diff vs v4:\n\n 1: e958a3ab20c = 1: e958a3ab20c bundle-uri: verify oid before writing refs\n 3: 5ddc894c2c1 = 2: d21c236b8de fetch-pack: expose fsckObjects configuration logic\n 2: beb70735811 ! 3: 0a18d7839be unbundle: extend verify_bundle_flags to support fsck-objects\n @@ Metadata\n Author: Xing Xin \n \n ## Commit message ##\n - unbundle: extend verify_bundle_flags to support fsck-objects\n + unbundle: extend options to support object verification\n \n - This commit extends `verify_bundle_flags` by adding a new option\n - `VERIFY_BUNDLE_FSCK_ALWAYS`, which enables checks for broken objects in\n - `bundle.c:unbundle`. This option is now used as the default for fetches\n - involving bundles, specifically by `transport.c:fetch_refs_from_bundle`\n - for direct bundle fetches and by `bundle-uri.c:unbundle_from_file` for\n - _bundle-uri_ enabled fetches.\n + This commit extends object verification support in `bundle.c:unbundle`\n + by adding two new options to `verify_bundle_flags`:\n \n - Upcoming commits will introduce another option as a replacement that\n - fits better with fetch operations. `VERIFY_BUNDLE_FSCK_ALWAYS` will be\n - further used to add \"--fsck-objects\" support for \"git bundle unbundle\"\n - and \"git bundle verify\".\n + - `VERIFY_BUNDLE_FSCK_ALWAYS` explicitly enables checks for broken\n + objects. It will be used to add \"--fsck-objects\" support for \"git\n + bundle unbundle\" in a separate series.\n + - `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is designed to be used during fetch\n + operations, specifically for direct bundle fetches and _bundle-uri_\n + enabled fetches. When enabled, `bundle.c:unbundle` invokes\n + `fetch-pack.c:fetch_pack_fsck_objects` to determine whether to enable\n + checks for broken objects. Passing this flag during fetching will be\n + implemented in a subsequent commit.\n +\n + Note that the option `VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over\n + `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`.\n \n Reviewed-by: Patrick Steinhardt \n Signed-off-by: Xing Xin \n \n - ## bundle-uri.c ##\n -@@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *file)\n - \t * the prerequisite commits.\n - \t */\n - \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n --\t\t\t VERIFY_BUNDLE_QUIET)))\n -+\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n - \t\treturn 1;\n - \n - \t/*\n -\n ## bundle.c ##\n +@@\n + #include \"list-objects-filter-options.h\"\n + #include \"connected.h\"\n + #include \"write-or-die.h\"\n ++#include \"fetch-pack.h\"\n + \n + static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n + static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n +@@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n + \t enum verify_bundle_flags flags)\n + {\n + \tstruct child_process ip = CHILD_PROCESS_INIT;\n ++\tint fsck_objects = 0;\n + \n + \tif (verify_bundle(r, header, flags))\n + \t\treturn -1;\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n +\tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n ++\t\tfsck_objects = 1;\n ++\telse if (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n ++\t\tfsck_objects = fetch_pack_fsck_objects();\n ++\n ++\tif (fsck_objects)\n +\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n +\n \tif (extra_index_pack_args) {\n @@ bundle.h: int create_bundle(struct repository *r, const char *path,\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n +\tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n ++\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 3),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\n -\n - ## transport.c ##\n -@@ transport.c: static int fetch_refs_from_bundle(struct transport *transport,\n - \tif (!data->get_refs_from_bundle_called)\n - \t\tget_refs_from_bundle_inner(transport);\n - \tret = unbundle(the_repository, &data->header, data->fd,\n --\t\t &extra_index_pack_args, 0);\n -+\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS);\n - \ttransport->hash_algo = data->header.hash_algo;\n - \treturn ret;\n - }\n 4: 68b9bca9f8b ! 4: eb9f21f16b5 unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH\n @@ Metadata\n Author: Xing Xin \n \n ## Commit message ##\n - unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH\n + unbundle: use VERIFY_BUNDLE_FSCK_FOLLOW_FETCH for fetches\n \n - This commit introduces a new option `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` to\n - `verify_bundle_flags`. In `bundle.c:unbundle`, this new option controls\n - whether broken object checks should be enabled by invoking\n - `fetch-pack.c:fetch_pack_fsck_objects`. Note that the option\n - `VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over\n - `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`.\n -\n - This flag is now used in the fetching process by:\n + This commit passes `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` to `unbundle` in\n + the fetching process, including:\n \n - `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n - `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n @@ Commit message\n fetch operations. Tests have been added to confirm functionality in the\n scenarios mentioned above.\n \n + Reviewed-by: Patrick Steinhardt \n Signed-off-by: Xing Xin \n \n ## bundle-uri.c ##\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n --\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n +-\t\t\t VERIFY_BUNDLE_QUIET)))\n +\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n \t\treturn 1;\n \n \t/*\n \n - ## bundle.c ##\n -@@\n - #include \"list-objects-filter-options.h\"\n - #include \"connected.h\"\n - #include \"write-or-die.h\"\n -+#include \"fetch-pack.h\"\n - \n - static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n - static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n -@@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n - \t enum verify_bundle_flags flags)\n - {\n - \tstruct child_process ip = CHILD_PROCESS_INIT;\n -+\tint fsck_objects = 0;\n - \n - \tif (verify_bundle(r, header, flags))\n - \t\treturn -1;\n -@@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n - \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n - \n - \tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n -+\t\tfsck_objects = 1;\n -+\telse if (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n -+\t\tfsck_objects = fetch_pack_fsck_objects();\n -+\n -+\tif (fsck_objects)\n - \t\tstrvec_push(&ip.args, \"--fsck-objects\");\n - \n - \tif (extra_index_pack_args) {\n -\n - ## bundle.h ##\n -@@ bundle.h: enum verify_bundle_flags {\n - \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n - \tVERIFY_BUNDLE_QUIET = (1 << 1),\n - \tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n -+\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 3),\n - };\n - \n - int verify_bundle(struct repository *r, struct bundle_header *header,\n -\n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'create bundle' '\n \t\tgit bundle create B.bundle topic &&\n @@ transport.c: static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n --\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS);\n +-\t\t &extra_index_pack_args, 0);\n +\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n\n-- \ngitgitgadget\n"},{"id":"496828","messageId":"d21c236b8dec9fd8c81a4224c1f63c511b04d8b9.1718088127.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v5.git.1718088126.gitgitgadget@gmail.com","subject":"[PATCH v5 2/4] fetch-pack: expose fsckObjects configuration logic","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T06:42:04Z","receivedAt":"2024-06-11T06:42:12Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nCurrently, we can use \"transfer.fsckObjects\" and the more specific\n\"fetch.fsckObjects\" to control checks for broken objects in received\npacks during fetches. However, these configurations were only\nacknowledged by `fetch-pack.c:get_pack` and did not take effect in\ndirect bundle fetches and fetches with _bundle-uri_ enabled.\n\nThis commit exposes the fetch-then-transfer configuration logic by\nadding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This\nnew function is used to replace the assignment for `fsck_objects` in\n`fetch-pack.c:get_pack`. In the next commit, it will also be used by\n`bundle.c:unbundle` to better fit fetching scenarios.\n\nHelped-by: Junio C Hamano \nHelped-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n fetch-pack.c | 17 +++++++++++------\n fetch-pack.h | 5 +++++\n 2 files changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d2aef21add..3acff2baf09 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\tstrvec_push(&cmd.args, alternate_shallow_file);\n \t}\n \n-\tif (fetch_fsck_objects >= 0\n-\t ? fetch_fsck_objects\n-\t : transfer_fsck_objects >= 0\n-\t ? transfer_fsck_objects\n-\t : 0)\n-\t\tfsck_objects = 1;\n+\tfsck_objects = fetch_pack_fsck_objects();\n \n \tif (do_keep || args->from_promisor || index_pack_args || fsck_objects) {\n \t\tif (pack_lockfiles || fsck_objects)\n@@ -2046,6 +2041,16 @@ static const struct object_id *iterate_ref_map(void *cb_data)\n \treturn &ref->old_oid;\n }\n \n+int fetch_pack_fsck_objects(void)\n+{\n+\tfetch_pack_setup();\n+\tif (fetch_fsck_objects >= 0)\n+\t\treturn fetch_fsck_objects;\n+\tif (transfer_fsck_objects >= 0)\n+\t\treturn transfer_fsck_objects;\n+\treturn 0;\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[],\n \t\t const struct ref *ref,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 6775d265175..b5c579cdae2 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -101,4 +101,9 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+/*\n+ * Return true if checks for broken objects in received pack are required.\n+ */\n+int fetch_pack_fsck_objects(void);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"496829","messageId":"e958a3ab20c0c06f52a00038f39605f14302032a.1718088127.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v5.git.1718088126.gitgitgadget@gmail.com","subject":"[PATCH v5 1/4] bundle-uri: verify oid before writing refs","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T06:42:03Z","receivedAt":"2024-06-11T06:42:12Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not discovered, thus resulting in rather slow\nclones. This was particularly problematic when employing the\n\"creationTokens\" heuristic.\n\nTo reproduce this issue, consider a repository with a single branch\n\"main\" pointing to commit \"A\". Firstly, create a base bundle with:\n\n git bundle create base.bundle main\n\nThen, add a new commit \"B\" on top of \"A\", and create an incremental\nbundle for \"main\":\n\n git bundle create incr.bundle A..main\n\nNow, generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above should result in a reference\n\"refs/bundles/main\" pointing to \"B\" in the new repository. However, git\nwould still download everything from the server, as if it had fetched\nnothing locally.\n\nSo why the \"refs/bundles/main\" is not discovered? After some digging I\nfound that:\n\n1. Bundles in bundle list are downloaded to local files via\n `bundle-uri.c:download_bundle_list` or via\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which\n is called by `bundle-uri.c:unbundle_all_bundles` or called within\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n3. To get all prerequisites of the bundle, the bundle header is read\n inside `bundle-uri.c:unbundle_from_file` to by calling\n `bundle.c:read_bundle_header`.\n4. Then it calls `bundle.c:unbundle`, which calls\n `bundle.c:verify_bundle` to ensure the repository contains all the\n prerequisites.\n5. `bundle.c:verify_bundle` calls `parse_object`, which eventually\n invokes `packfile.c:prepare_packed_git` or\n `packfile.c:reprepare_packed_git`, filling\n `raw_object_store->packed_git` and setting `packed_git_initialized`.\n6. If `bundle.c:unbundle` succeeds, it writes refs via\n `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here\n bundle refs which can target arbitrary objects are written to the\n repository.\n7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions\n `fetch-pack.c:mark_complete_and_common_ref` and\n `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to\n find local tips for negotiation. The `OBJECT_INFO_QUICK` flag\n prevents `packfile.c:reprepare_packed_git` from being called,\n resulting in failures to parse OIDs that reside only in the latest\n bundle.\n\nIn the example above, when unbunding \"incr.bundle\", \"base.pack\" is added\nto `packed_git` due to prerequisites verification. However, \"B\" cannot\nbe found for negotiation because it exists in \"incr.pack\", which is not\nincluded in `packed_git`.\n\nThis commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag\nwhen writing bundle refs. When `refs.c:refs_update_ref` is called to to\nwrite the corresponding bundle refs, it triggers\n`refs.c:ref_transaction_commit`. This, in turn, invokes\n`refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of\nthe refs storage backend. For files backend, this function is\n`files-backend.c:files_transaction_prepare`, and for reftable backend,\nit is `reftable-backend.c:reftable_be_transaction_prepare`. Both\nfunctions eventually call `object.c:parse_object`, which can invoke\n`packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures\nthat bundle refs point to valid objects and that all tips from bundle\nrefs are correctly parsed during subsequent negotiations.\n\nA test has been added to demonstrate that bundles with incorrect\nheaders, where refs point to non-existent objects, do not result in any\nbundle refs being created in the repository. Additionally, a set of\nnegotiation-related tests for fetching with bundle-uri has been\nincluded.\n\nReviewed-by: Karthik Nayak \nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 +-\n t/t5558-clone-bundle-uri.sh | 153 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 150 insertions(+), 6 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91b3319a5c1..65666a11d9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\trefs_update_ref(get_main_ref_store(the_repository),\n \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n \t\t\t\thas_old ? &old_oid : NULL,\n-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n+\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n \t}\n \n \tbundle_header_release(&header);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..8f4f802e4f1 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -19,10 +19,19 @@ test_expect_success 'fail to clone from non-bundle file' '\n \n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n-\tgit -C clone-from checkout -b topic &&\n-\ttest_commit -C clone-from A &&\n-\ttest_commit -C clone-from B &&\n-\tgit -C clone-from bundle create B.bundle topic\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b topic &&\n+\n+\t\ttest_commit A &&\n+\t\tgit bundle create A.bundle topic &&\n+\n+\t\ttest_commit B &&\n+\t\tgit bundle create B.bundle topic &&\n+\n+\t\t# Create a bundle with reference pointing to non-existent object.\n+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n+\t)\n '\n \n test_expect_success 'clone with path bundle' '\n@@ -33,6 +42,16 @@ test_expect_success 'clone with path bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone with bundle that has bad header' '\n+\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n+\t\tclone-from clone-bad-header 2>err &&\n+\t# Write bundle ref fails, but clone can still proceed.\n+\tcommit_b=$(git -C clone-from rev-parse B) &&\n+\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n+\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\n@@ -259,6 +278,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+#########################################################################\n+# Clone negotiation related tests begin here\n+\n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n-- \ngitgitgadget\n\n"},{"id":"496830","messageId":"0a18d7839be67d6c0be137c7e15dff9663a161a8.1718088127.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v5.git.1718088126.gitgitgadget@gmail.com","subject":"[PATCH v5 3/4] unbundle: extend options to support object verification","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T06:42:05Z","receivedAt":"2024-06-11T06:42:13Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThis commit extends object verification support in `bundle.c:unbundle`\nby adding two new options to `verify_bundle_flags`:\n\n- `VERIFY_BUNDLE_FSCK_ALWAYS` explicitly enables checks for broken\n objects. It will be used to add \"--fsck-objects\" support for \"git\n bundle unbundle\" in a separate series.\n- `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is designed to be used during fetch\n operations, specifically for direct bundle fetches and _bundle-uri_\n enabled fetches. When enabled, `bundle.c:unbundle` invokes\n `fetch-pack.c:fetch_pack_fsck_objects` to determine whether to enable\n checks for broken objects. Passing this flag during fetching will be\n implemented in a subsequent commit.\n\nNote that the option `VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over\n`VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`.\n\nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle.c | 10 ++++++++++\n bundle.h | 2 ++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/bundle.c b/bundle.c\nindex 95367c2d0a0..53ac73834ea 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -17,6 +17,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"connected.h\"\n #include \"write-or-die.h\"\n+#include \"fetch-pack.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -615,6 +616,7 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \t enum verify_bundle_flags flags)\n {\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n+\tint fsck_objects = 0;\n \n \tif (verify_bundle(r, header, flags))\n \t\treturn -1;\n@@ -625,6 +627,14 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n+\tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n+\t\tfsck_objects = 1;\n+\telse if (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n+\t\tfsck_objects = fetch_pack_fsck_objects();\n+\n+\tif (fsck_objects)\n+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/bundle.h b/bundle.h\nindex 021adbdcbb3..a39d8ea1a7e 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -33,6 +33,8 @@ int create_bundle(struct repository *r, const char *path,\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n+\tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n+\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 3),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\n-- \ngitgitgadget\n\n"},{"id":"496831","messageId":"eb9f21f16b5d2b01e67510eaec160cecabbcc049.1718088127.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v5.git.1718088126.gitgitgadget@gmail.com","subject":"[PATCH v5 4/4] unbundle: use VERIFY_BUNDLE_FSCK_FOLLOW_FETCH for fetches","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T06:42:06Z","receivedAt":"2024-06-11T06:42:14Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThis commit passes `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` to `unbundle` in\nthe fetching process, including:\n\n- `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n- `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n\nThis addition ensures a consistent logic for object verification during\nfetch operations. Tests have been added to confirm functionality in the\nscenarios mentioned above.\n\nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 2 +-\n t/t5558-clone-bundle-uri.sh | 35 ++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++++++++++++++++++++++++++++\n transport.c | 2 +-\n 4 files changed, 69 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 65666a11d9c..e7ebac6ce57 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 8f4f802e4f1..48be1b18802 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -30,7 +30,21 @@ test_expect_success 'create bundle' '\n \t\tgit bundle create B.bundle topic &&\n \n \t\t# Create a bundle with reference pointing to non-existent object.\n-\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle &&\n+\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad-object.bundle bad &&\n+\t\tgit update-ref -d refs/heads/bad\n \t)\n '\n \n@@ -52,6 +66,25 @@ test_expect_success 'clone with bundle that has bad header' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+test_expect_success 'clone with bundle that has bad object' '\n+\t# Unbundle succeeds if no fsckObjects confugured.\n+\tgit clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-no-fsck &&\n+\tgit -C clone-bad-object-no-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/bad\n+\tEOF\n+\ttest_cmp expect actual &&\n+\n+\t# Unbundle fails with fsckObjects set true, but clone can still proceed.\n+\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\tgit -C clone-bad-object-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\ndiff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\nindex 0d1e92d9963..5182efc0b45 100755\n--- a/t/t5607-clone-bundle.sh\n+++ b/t/t5607-clone-bundle.sh\n@@ -138,6 +138,39 @@ test_expect_success 'fetch SHA-1 from bundle' '\n \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n '\n \n+test_expect_success 'clone bundle with different fsckObjects configurations' '\n+\ttest_create_repo bundle-fsck &&\n+\t(\n+\t\tcd bundle-fsck &&\n+\t\ttest_commit first &&\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad.bundle bad\n+\t) &&\n+\n+\tgit clone bundle-fsck/bad.bundle bundle-no-fsck &&\n+\n+\tgit -c fetch.fsckObjects=false -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-no-fsck &&\n+\n+\ttest_must_fail git -c fetch.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\n+\ttest_must_fail git -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-transfer-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err\n+'\n+\n test_expect_success 'git bundle uses expected default format' '\n \tgit bundle create bundle HEAD^.. &&\n \tcat >expect <<-EOF &&\ndiff --git a/transport.c b/transport.c\nindex 0ad04b77fd2..6cd5683bb45 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, 0);\n+\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n"},{"id":"496833","messageId":"1238d76b.6e25.190060c5b5d.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZmGmV8-QNMN643ou@tanuki","subject":"Re:Re: [PATCH v4 4/4] unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-06-11T06:46:31Z","receivedAt":"2024-06-11T06:46:50Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-06-06 20:06:47, \"Patrick Steinhardt\" wrote:\n>On Thu, May 30, 2024 at 08:21:30AM +0000, Xing Xin via GitGitGadget wrote:\n>> From: Xing Xin \n>\n>Same here, the important part is not that we introduce the flag, but\n>that we start using it in `unbundle_from_file()`.\n>\n>> diff --git a/bundle-uri.c b/bundle-uri.c\n>> index 066ff788104..e7ebac6ce57 100644\n>> --- a/bundle-uri.c\n>> +++ b/bundle-uri.c\n>> @@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n>> \t * the prerequisite commits.\n>> \t */\n>> \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n>> -\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS)))\n>> +\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n>> \t\treturn 1;\n>> \n>> \t/*\n>\n>One thing that is a bit weird is that we first change `unbundle()` to\n>use `FSCK_ALWAYS` in a preceding patch, and then convert it to use\n>`FSCK_FOLLOW_FETCH` in the same series. It could be restructured a bit\n>to first introduce the flags, only, while not modifying any of the\n>callsites yet. Passing the respective flags would then be done in a\n>separate commit.\n\nThis makes sense to me, thanks!\n\nXing Xin\n"},{"id":"496834","messageId":"5670c216.6e3d.190060c98fb.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZmGmUcO8OYuPnH9H@tanuki","subject":"Re:Re: [PATCH v4 2/4] unbundle: extend verify_bundle_flags to support fsck-objects","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-06-11T06:46:47Z","receivedAt":"2024-06-11T06:47:08Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-06-06 20:06:41, \"Patrick Steinhardt\" wrote:\n>On Thu, May 30, 2024 at 08:21:28AM +0000, Xing Xin via GitGitGadget wrote:\n>> From: Xing Xin \n>\n>Tiny nit: the important change in this commit is not that you wire up\n>the flag, but rather that we start to execute git-fsck(1) now. I'd thus\n\nTo be precise, it is adding a \"--fsck-objects\" flag to \"git-index-pack\".\n\n>propose to adapt the commit title accordingly.\n\nThis commit is mapped to [PATCH v5 3/4] due to some adjustments to the\ncommit order and implementation details. Hope the new title can better\ndescribe the new patch.\n\nXing Xin\n"},{"id":"496858","messageId":"ZmgUrQ6D1xtQghPy@tanuki","threadId":"61479","inReplyTo":"0a18d7839be67d6c0be137c7e15dff9663a161a8.1718088127.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 3/4] unbundle: extend options to support object verification","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-06-11T09:11:09Z","receivedAt":"2024-06-11T09:11:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jun 11, 2024 at 06:42:05AM +0000, Xing Xin via GitGitGadget wrote:\n> From: Xing Xin \n> \n> This commit extends object verification support in `bundle.c:unbundle`\n> by adding two new options to `verify_bundle_flags`:\n> \n> - `VERIFY_BUNDLE_FSCK_ALWAYS` explicitly enables checks for broken\n> objects. It will be used to add \"--fsck-objects\" support for \"git\n> bundle unbundle\" in a separate series.\n> - `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is designed to be used during fetch\n> operations, specifically for direct bundle fetches and _bundle-uri_\n> enabled fetches. When enabled, `bundle.c:unbundle` invokes\n> `fetch-pack.c:fetch_pack_fsck_objects` to determine whether to enable\n> checks for broken objects. Passing this flag during fetching will be\n> implemented in a subsequent commit.\n> \n> Note that the option `VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over\n> `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`.\n\nThanks, the new sequence of commits is much easier to follow. It also\nshows that there is no user of `VERIFY_BUNDLE_FSCK_ALWAYS` at the end of\nthis series. So maybe we should drop that flag?\n\nIf you do that, then I'd also propose to merge patches 2 and 3 into one\ngiven that both are quite trivial and related to each other.\n\nOther than that this series looks good to me.\n\nPatrick\n\n> Reviewed-by: Patrick Steinhardt \n> Signed-off-by: Xing Xin \n> ---\n> bundle.c | 10 ++++++++++\n> bundle.h | 2 ++\n> 2 files changed, 12 insertions(+)\n> \n> diff --git a/bundle.c b/bundle.c\n> index 95367c2d0a0..53ac73834ea 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -17,6 +17,7 @@\n> #include \"list-objects-filter-options.h\"\n> #include \"connected.h\"\n> #include \"write-or-die.h\"\n> +#include \"fetch-pack.h\"\n> \n> static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n> static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n> @@ -615,6 +616,7 @@ int unbundle(struct repository *r, struct bundle_header *header,\n> \t enum verify_bundle_flags flags)\n> {\n> \tstruct child_process ip = CHILD_PROCESS_INIT;\n> +\tint fsck_objects = 0;\n> \n> \tif (verify_bundle(r, header, flags))\n> \t\treturn -1;\n> @@ -625,6 +627,14 @@ int unbundle(struct repository *r, struct bundle_header *header,\n> \tif (header->filter.choice)\n> \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n> \n> +\tif (flags & VERIFY_BUNDLE_FSCK_ALWAYS)\n> +\t\tfsck_objects = 1;\n> +\telse if (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n> +\t\tfsck_objects = fetch_pack_fsck_objects();\n> +\n> +\tif (fsck_objects)\n> +\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n> +\n> \tif (extra_index_pack_args) {\n> \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n> \t\tstrvec_clear(extra_index_pack_args);\n> diff --git a/bundle.h b/bundle.h\n> index 021adbdcbb3..a39d8ea1a7e 100644\n> --- a/bundle.h\n> +++ b/bundle.h\n> @@ -33,6 +33,8 @@ int create_bundle(struct repository *r, const char *path,\n> enum verify_bundle_flags {\n> \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n> \tVERIFY_BUNDLE_QUIET = (1 << 1),\n> +\tVERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2),\n> +\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 3),\n> };\n> \n> int verify_bundle(struct repository *r, struct bundle_header *header,\n> -- \n> gitgitgadget\n> \n"},{"id":"496916","messageId":"pull.1730.v6.git.1718109943.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v5.git.1718088126.gitgitgadget@gmail.com","subject":"[PATCH v6 0/3] object checking related additions and fixes for bundles in fetches","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T12:45:40Z","receivedAt":"2024-06-11T12:45:47Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"While attempting to fix a reference negotiation bug in bundle-uri, we\nidentified that the fetch process lacks some crucial object validation\nchecks when processing bundles. The primary issues are:\n\n 1. In the bundle-uri scenario, object IDs were not validated before writing\n bundle references. This was the root cause of the original negotiation\n bug in bundle-uri and could lead to potential repository corruption.\n 2. The existing \"fetch.fsckObjects\" and \"transfer.fsckObjects\"\n configurations were not applied when directly fetching bundles or\n fetching with bundle-uri enabled. In fact, there were no object\n validation supports for unbundle.\n\nThe first patch addresses the bundle-uri negotiation issue by removing the\nREF_SKIP_OID_VERIFICATION flag when writing bundle references.\n\nPatches 2 through 3 extend verify_bundle_flags for bundle.c:unbundle to add\nsupport for object validation (fsck) in fetch scenarios, mainly following\nthe suggestions from Junio and Patrick on the mailing list.\n\nXing Xin (3):\n bundle-uri: verify oid before writing refs\n fetch-pack: expose fsckObjects configuration logic\n unbundle: support object verification for fetches\n\n bundle-uri.c | 5 +-\n bundle.c | 5 +\n bundle.h | 1 +\n fetch-pack.c | 17 ++--\n fetch-pack.h | 5 +\n t/t5558-clone-bundle-uri.sh | 186 +++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++\n transport.c | 2 +-\n 8 files changed, 240 insertions(+), 14 deletions(-)\n\n\nbase-commit: b9cfe4845cb2562584837bc0101c0ab76490a239\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\nRange-diff vs v5:\n\n 1: e958a3ab20c = 1: e958a3ab20c bundle-uri: verify oid before writing refs\n 2: d21c236b8de = 2: d21c236b8de fetch-pack: expose fsckObjects configuration logic\n 3: 0a18d7839be < -: ----------- unbundle: extend options to support object verification\n 4: eb9f21f16b5 ! 3: 53395e8c08a unbundle: use VERIFY_BUNDLE_FSCK_FOLLOW_FETCH for fetches\n @@ Metadata\n Author: Xing Xin \n \n ## Commit message ##\n - unbundle: use VERIFY_BUNDLE_FSCK_FOLLOW_FETCH for fetches\n + unbundle: support object verification for fetches\n \n - This commit passes `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` to `unbundle` in\n - the fetching process, including:\n + This commit extends object verification support for fetches in\n + `bundle.c:unbundle` by adding the `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`\n + option to `verify_bundle_flags`. When this option is enabled,\n + `bundle.c:unbundle` invokes `fetch-pack.c:fetch_pack_fsck_objects` to\n + determine whether to append the \"--fsck-objects\" flag to\n + \"git-index-pack\".\n +\n + `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is now passed to `unbundle` in the\n + fetching process, including:\n \n - `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n - `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n \n \t/*\n \n + ## bundle.c ##\n +@@\n + #include \"list-objects-filter-options.h\"\n + #include \"connected.h\"\n + #include \"write-or-die.h\"\n ++#include \"fetch-pack.h\"\n + \n + static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n + static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n +@@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n + \tif (header->filter.choice)\n + \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n + \n ++\tif (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n ++\t\tif (fetch_pack_fsck_objects())\n ++\t\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n ++\n + \tif (extra_index_pack_args) {\n + \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n + \t\tstrvec_clear(extra_index_pack_args);\n +\n + ## bundle.h ##\n +@@ bundle.h: int create_bundle(struct repository *r, const char *path,\n + enum verify_bundle_flags {\n + \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n + \tVERIFY_BUNDLE_QUIET = (1 << 1),\n ++\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 2),\n + };\n + \n + int verify_bundle(struct repository *r, struct bundle_header *header,\n +\n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'create bundle' '\n \t\tgit bundle create B.bundle topic &&\n\n-- \ngitgitgadget\n"},{"id":"496917","messageId":"e958a3ab20c0c06f52a00038f39605f14302032a.1718109943.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v6.git.1718109943.gitgitgadget@gmail.com","subject":"[PATCH v6 1/3] bundle-uri: verify oid before writing refs","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T12:45:41Z","receivedAt":"2024-06-11T12:45:47Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not discovered, thus resulting in rather slow\nclones. This was particularly problematic when employing the\n\"creationTokens\" heuristic.\n\nTo reproduce this issue, consider a repository with a single branch\n\"main\" pointing to commit \"A\". Firstly, create a base bundle with:\n\n git bundle create base.bundle main\n\nThen, add a new commit \"B\" on top of \"A\", and create an incremental\nbundle for \"main\":\n\n git bundle create incr.bundle A..main\n\nNow, generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above should result in a reference\n\"refs/bundles/main\" pointing to \"B\" in the new repository. However, git\nwould still download everything from the server, as if it had fetched\nnothing locally.\n\nSo why the \"refs/bundles/main\" is not discovered? After some digging I\nfound that:\n\n1. Bundles in bundle list are downloaded to local files via\n `bundle-uri.c:download_bundle_list` or via\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which\n is called by `bundle-uri.c:unbundle_all_bundles` or called within\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n3. To get all prerequisites of the bundle, the bundle header is read\n inside `bundle-uri.c:unbundle_from_file` to by calling\n `bundle.c:read_bundle_header`.\n4. Then it calls `bundle.c:unbundle`, which calls\n `bundle.c:verify_bundle` to ensure the repository contains all the\n prerequisites.\n5. `bundle.c:verify_bundle` calls `parse_object`, which eventually\n invokes `packfile.c:prepare_packed_git` or\n `packfile.c:reprepare_packed_git`, filling\n `raw_object_store->packed_git` and setting `packed_git_initialized`.\n6. If `bundle.c:unbundle` succeeds, it writes refs via\n `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here\n bundle refs which can target arbitrary objects are written to the\n repository.\n7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions\n `fetch-pack.c:mark_complete_and_common_ref` and\n `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to\n find local tips for negotiation. The `OBJECT_INFO_QUICK` flag\n prevents `packfile.c:reprepare_packed_git` from being called,\n resulting in failures to parse OIDs that reside only in the latest\n bundle.\n\nIn the example above, when unbunding \"incr.bundle\", \"base.pack\" is added\nto `packed_git` due to prerequisites verification. However, \"B\" cannot\nbe found for negotiation because it exists in \"incr.pack\", which is not\nincluded in `packed_git`.\n\nThis commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag\nwhen writing bundle refs. When `refs.c:refs_update_ref` is called to to\nwrite the corresponding bundle refs, it triggers\n`refs.c:ref_transaction_commit`. This, in turn, invokes\n`refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of\nthe refs storage backend. For files backend, this function is\n`files-backend.c:files_transaction_prepare`, and for reftable backend,\nit is `reftable-backend.c:reftable_be_transaction_prepare`. Both\nfunctions eventually call `object.c:parse_object`, which can invoke\n`packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures\nthat bundle refs point to valid objects and that all tips from bundle\nrefs are correctly parsed during subsequent negotiations.\n\nA test has been added to demonstrate that bundles with incorrect\nheaders, where refs point to non-existent objects, do not result in any\nbundle refs being created in the repository. Additionally, a set of\nnegotiation-related tests for fetching with bundle-uri has been\nincluded.\n\nReviewed-by: Karthik Nayak \nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 +-\n t/t5558-clone-bundle-uri.sh | 153 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 150 insertions(+), 6 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91b3319a5c1..65666a11d9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\trefs_update_ref(get_main_ref_store(the_repository),\n \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n \t\t\t\thas_old ? &old_oid : NULL,\n-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n+\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n \t}\n \n \tbundle_header_release(&header);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..8f4f802e4f1 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -19,10 +19,19 @@ test_expect_success 'fail to clone from non-bundle file' '\n \n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n-\tgit -C clone-from checkout -b topic &&\n-\ttest_commit -C clone-from A &&\n-\ttest_commit -C clone-from B &&\n-\tgit -C clone-from bundle create B.bundle topic\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b topic &&\n+\n+\t\ttest_commit A &&\n+\t\tgit bundle create A.bundle topic &&\n+\n+\t\ttest_commit B &&\n+\t\tgit bundle create B.bundle topic &&\n+\n+\t\t# Create a bundle with reference pointing to non-existent object.\n+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n+\t)\n '\n \n test_expect_success 'clone with path bundle' '\n@@ -33,6 +42,16 @@ test_expect_success 'clone with path bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone with bundle that has bad header' '\n+\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n+\t\tclone-from clone-bad-header 2>err &&\n+\t# Write bundle ref fails, but clone can still proceed.\n+\tcommit_b=$(git -C clone-from rev-parse B) &&\n+\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n+\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\n@@ -259,6 +278,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+#########################################################################\n+# Clone negotiation related tests begin here\n+\n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished rm -rf trace*.txt &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/topic\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished rm -f trace*.txt &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n-- \ngitgitgadget\n\n"},{"id":"496918","messageId":"d21c236b8dec9fd8c81a4224c1f63c511b04d8b9.1718109943.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v6.git.1718109943.gitgitgadget@gmail.com","subject":"[PATCH v6 2/3] fetch-pack: expose fsckObjects configuration logic","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T12:45:42Z","receivedAt":"2024-06-11T12:45:49Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nCurrently, we can use \"transfer.fsckObjects\" and the more specific\n\"fetch.fsckObjects\" to control checks for broken objects in received\npacks during fetches. However, these configurations were only\nacknowledged by `fetch-pack.c:get_pack` and did not take effect in\ndirect bundle fetches and fetches with _bundle-uri_ enabled.\n\nThis commit exposes the fetch-then-transfer configuration logic by\nadding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This\nnew function is used to replace the assignment for `fsck_objects` in\n`fetch-pack.c:get_pack`. In the next commit, it will also be used by\n`bundle.c:unbundle` to better fit fetching scenarios.\n\nHelped-by: Junio C Hamano \nHelped-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n fetch-pack.c | 17 +++++++++++------\n fetch-pack.h | 5 +++++\n 2 files changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d2aef21add..3acff2baf09 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\tstrvec_push(&cmd.args, alternate_shallow_file);\n \t}\n \n-\tif (fetch_fsck_objects >= 0\n-\t ? fetch_fsck_objects\n-\t : transfer_fsck_objects >= 0\n-\t ? transfer_fsck_objects\n-\t : 0)\n-\t\tfsck_objects = 1;\n+\tfsck_objects = fetch_pack_fsck_objects();\n \n \tif (do_keep || args->from_promisor || index_pack_args || fsck_objects) {\n \t\tif (pack_lockfiles || fsck_objects)\n@@ -2046,6 +2041,16 @@ static const struct object_id *iterate_ref_map(void *cb_data)\n \treturn &ref->old_oid;\n }\n \n+int fetch_pack_fsck_objects(void)\n+{\n+\tfetch_pack_setup();\n+\tif (fetch_fsck_objects >= 0)\n+\t\treturn fetch_fsck_objects;\n+\tif (transfer_fsck_objects >= 0)\n+\t\treturn transfer_fsck_objects;\n+\treturn 0;\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[],\n \t\t const struct ref *ref,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 6775d265175..b5c579cdae2 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -101,4 +101,9 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+/*\n+ * Return true if checks for broken objects in received pack are required.\n+ */\n+int fetch_pack_fsck_objects(void);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"496919","messageId":"53395e8c08a8487f3e53dca15766307854a24b3b.1718109943.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v6.git.1718109943.gitgitgadget@gmail.com","subject":"[PATCH v6 3/3] unbundle: support object verification for fetches","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-11T12:45:43Z","receivedAt":"2024-06-11T12:45:50Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThis commit extends object verification support for fetches in\n`bundle.c:unbundle` by adding the `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`\noption to `verify_bundle_flags`. When this option is enabled,\n`bundle.c:unbundle` invokes `fetch-pack.c:fetch_pack_fsck_objects` to\ndetermine whether to append the \"--fsck-objects\" flag to\n\"git-index-pack\".\n\n`VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is now passed to `unbundle` in the\nfetching process, including:\n\n- `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n- `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n\nThis addition ensures a consistent logic for object verification during\nfetch operations. Tests have been added to confirm functionality in the\nscenarios mentioned above.\n\nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 2 +-\n bundle.c | 5 +++++\n bundle.h | 1 +\n t/t5558-clone-bundle-uri.sh | 35 ++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++++++++++++++++++++++++++++\n transport.c | 2 +-\n 6 files changed, 75 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 65666a11d9c..e7ebac6ce57 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 95367c2d0a0..20bbfffbb43 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -17,6 +17,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"connected.h\"\n #include \"write-or-die.h\"\n+#include \"fetch-pack.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -625,6 +626,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n+\tif (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n+\t\tif (fetch_pack_fsck_objects())\n+\t\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/bundle.h b/bundle.h\nindex 021adbdcbb3..545df6e9d40 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -33,6 +33,7 @@ int create_bundle(struct repository *r, const char *path,\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n+\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 2),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 8f4f802e4f1..48be1b18802 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -30,7 +30,21 @@ test_expect_success 'create bundle' '\n \t\tgit bundle create B.bundle topic &&\n \n \t\t# Create a bundle with reference pointing to non-existent object.\n-\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle &&\n+\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad-object.bundle bad &&\n+\t\tgit update-ref -d refs/heads/bad\n \t)\n '\n \n@@ -52,6 +66,25 @@ test_expect_success 'clone with bundle that has bad header' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+test_expect_success 'clone with bundle that has bad object' '\n+\t# Unbundle succeeds if no fsckObjects confugured.\n+\tgit clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-no-fsck &&\n+\tgit -C clone-bad-object-no-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/bad\n+\tEOF\n+\ttest_cmp expect actual &&\n+\n+\t# Unbundle fails with fsckObjects set true, but clone can still proceed.\n+\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\tgit -C clone-bad-object-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\ndiff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\nindex 0d1e92d9963..5182efc0b45 100755\n--- a/t/t5607-clone-bundle.sh\n+++ b/t/t5607-clone-bundle.sh\n@@ -138,6 +138,39 @@ test_expect_success 'fetch SHA-1 from bundle' '\n \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n '\n \n+test_expect_success 'clone bundle with different fsckObjects configurations' '\n+\ttest_create_repo bundle-fsck &&\n+\t(\n+\t\tcd bundle-fsck &&\n+\t\ttest_commit first &&\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad.bundle bad\n+\t) &&\n+\n+\tgit clone bundle-fsck/bad.bundle bundle-no-fsck &&\n+\n+\tgit -c fetch.fsckObjects=false -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-no-fsck &&\n+\n+\ttest_must_fail git -c fetch.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\n+\ttest_must_fail git -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-transfer-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err\n+'\n+\n test_expect_success 'git bundle uses expected default format' '\n \tgit bundle create bundle HEAD^.. &&\n \tcat >expect <<-EOF &&\ndiff --git a/transport.c b/transport.c\nindex 0ad04b77fd2..6cd5683bb45 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, 0);\n+\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n"},{"id":"496920","messageId":"5b47b697.c049.19007573522.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"ZmgUrQ6D1xtQghPy@tanuki","subject":"Re:Re: [PATCH v5 3/4] unbundle: extend options to support object verification","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-06-11T12:47:54Z","receivedAt":"2024-06-11T12:48:09Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-06-11 17:11:09, \"Patrick Steinhardt\" wrote:\n>On Tue, Jun 11, 2024 at 06:42:05AM +0000, Xing Xin via GitGitGadget wrote:\n>> From: Xing Xin \n>> \n>> This commit extends object verification support in `bundle.c:unbundle`\n>> by adding two new options to `verify_bundle_flags`:\n>> \n>> - `VERIFY_BUNDLE_FSCK_ALWAYS` explicitly enables checks for broken\n>> objects. It will be used to add \"--fsck-objects\" support for \"git\n>> bundle unbundle\" in a separate series.\n>> - `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is designed to be used during fetch\n>> operations, specifically for direct bundle fetches and _bundle-uri_\n>> enabled fetches. When enabled, `bundle.c:unbundle` invokes\n>> `fetch-pack.c:fetch_pack_fsck_objects` to determine whether to enable\n>> checks for broken objects. Passing this flag during fetching will be\n>> implemented in a subsequent commit.\n>> \n>> Note that the option `VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over\n>> `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`.\n>\n>Thanks, the new sequence of commits is much easier to follow. It also\n>shows that there is no user of `VERIFY_BUNDLE_FSCK_ALWAYS` at the end of\n>this series. So maybe we should drop that flag?\n\nOK, let's focus on the fetching scenarios in this patch series.\n\n>If you do that, then I'd also propose to merge patches 2 and 3 into one\n>given that both are quite trivial and related to each other.\n\nI merged patches 3 and 4 instead to combine the new option definition and usage\nlogic, as the two are more closely related and more trivial. :)\n\nXing Xin\n\n>Other than that this series looks good to me.\n>\n>Patrick\n>\n\n"},{"id":"496922","messageId":"ZmhNwevTExnbvIdH@tanuki","threadId":"61479","inReplyTo":"pull.1730.v6.git.1718109943.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 0/3] object checking related additions and fixes for bundles in fetches","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-06-11T13:14:41Z","receivedAt":"2024-06-11T13:14:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jun 11, 2024 at 12:45:40PM +0000, blanet via GitGitGadget wrote:\n> While attempting to fix a reference negotiation bug in bundle-uri, we\n> identified that the fetch process lacks some crucial object validation\n> checks when processing bundles. The primary issues are:\n> \n> 1. In the bundle-uri scenario, object IDs were not validated before writing\n> bundle references. This was the root cause of the original negotiation\n> bug in bundle-uri and could lead to potential repository corruption.\n> 2. The existing \"fetch.fsckObjects\" and \"transfer.fsckObjects\"\n> configurations were not applied when directly fetching bundles or\n> fetching with bundle-uri enabled. In fact, there were no object\n> validation supports for unbundle.\n> \n> The first patch addresses the bundle-uri negotiation issue by removing the\n> REF_SKIP_OID_VERIFICATION flag when writing bundle references.\n> \n> Patches 2 through 3 extend verify_bundle_flags for bundle.c:unbundle to add\n> support for object validation (fsck) in fetch scenarios, mainly following\n> the suggestions from Junio and Patrick on the mailing list.\n\nThanks, this version looks good to me.\n\nPatrick\n"},{"id":"496956","messageId":"xmqqa5jruwkr.fsf@gitster.g","threadId":"61479","inReplyTo":"e958a3ab20c0c06f52a00038f39605f14302032a.1718109943.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 1/3] bundle-uri: verify oid before writing refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-11T19:08:04Z","receivedAt":"2024-06-11T19:08:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Xing Xin via GitGitGadget\" writes:\n\n> This commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag\n\n\"Fix the bug by ...\".\n\n> when writing bundle refs. When `refs.c:refs_update_ref` is called to to\n\n\"to to\"\n\n> write the corresponding bundle refs, it triggers\n> `refs.c:ref_transaction_commit`. This, in turn, invokes\n> `refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of\n> the refs storage backend. For files backend, this function is\n> `files-backend.c:files_transaction_prepare`, and for reftable backend,\n> it is `reftable-backend.c:reftable_be_transaction_prepare`. Both\n> functions eventually call `object.c:parse_object`, which can invoke\n> `packfile.c:reprepare_packed_git` to refresh `packed_git`.\n\nNice. That does sound like the right fix. The one who did\nsomething to _require_ us to reprepare causes the packed_git\nlist refreshed.\n\n> test_expect_success 'create bundle' '\n> \tgit init clone-from &&\n> -\tgit -C clone-from checkout -b topic &&\n> -\ttest_commit -C clone-from A &&\n> -\ttest_commit -C clone-from B &&\n> -\tgit -C clone-from bundle create B.bundle topic\n> +\t(\n> +\t\tcd clone-from &&\n> +\t\tgit checkout -b topic &&\n> +\n> +\t\ttest_commit A &&\n> +\t\tgit bundle create A.bundle topic &&\n> +\n> +\t\ttest_commit B &&\n> +\t\tgit bundle create B.bundle topic &&\n> +\n> +\t\t# Create a bundle with reference pointing to non-existent object.\n> +\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n\nI suspect that this would be terribly unportable. The early part of\na bundle file may be text and sed may be able to grok but are you\nsure everybody's implementation of sed would not barf (or even\nworse, corrupt) the pack stream data that follows?\n\nThe code used in t/lib-bundle.sh:convert_bundle_to_pack() has been\nin use since 8315588b (bundle: fix wrong check of read_header()'s\nreturn value & add tests, 2007-03-06), so munging the bundle with\na code similar to it may have a better portability story.\n\nAdd something like:\n\n corrupt_bundle_header () {\n sed -e '/^$/q' \"$@\"\n cat\n }\n\nto t/lib-bundle.sh, which can take an arbitrary sequence of command\nline parameters to drive \"sed\", and can be used like so:\n\n\tcorrupt_bundle_header \\\n\t\t-e \"s/^$(git rev-parse A) /$(git rev-parse B) /\" \\\n\t\tB.bndl\n\nperhaps?\n\n> @@ -33,6 +42,16 @@ test_expect_success 'clone with path bundle' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'clone with bundle that has bad header' '\n> +\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n> +\t\tclone-from clone-bad-header 2>err &&\n> +\t# Write bundle ref fails, but clone can still proceed.\n> +\tcommit_b=$(git -C clone-from rev-parse B) &&\n> +\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n> +\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n> +\t! grep \"refs/bundles/\" refs\n> +'\n> +\n\nSo this is the test the proposed log message discussed. The\ndescription gave a false impression that the \"broken header\" test\nthat has not much to do with the bug being fixed was the only added\ntest---we probably want to correct that impression.\n\n> @@ -259,6 +278,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n> \t! grep \"refs/bundles/\" refs\n> '\n> \n> +#########################################################################\n> +# Clone negotiation related tests begin here\n\nDrop this divider and comment. The HTTP tests you see below has a\nmuch better reason to be separated like that in order to warn test\nwriters (they shouldn't add their random new tests after that point,\nbecause everything after that one is skipped when HTTPD tests are\ndisabled---see the beginning of t/lib-httpd.sh which is included\nafter that divider line), but everything here you added is not\nspecial. Everybody should run these tests.\n\n> +test_expect_success 'negotiation: bundle with part of wanted commits' '\n> +\ttest_when_finished rm -rf trace*.txt &&\n\nDo not overly depend on glob not matching at this point when you\nestablish the when-finished handler (or glob matching the files you\nwant to catch and later test not adding anything you would want to\nclean). Quote \"rm -f trace*.txt\" and drop \"r\" if you do not absolutely\nneed it (and I would imagine you don't, given the .txt suffix).\n\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n> +\t\tclone-from nego-bundle-part &&\n> +\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/topic\n> +\tEOF\n\nHmph, if the expected pattern is only a few lines without any magic,\n\n\ttest_write_lines >expect refs/bundles/topic &&\n\nmay be easier to follow.\n\n> +\ttest_cmp expect actual &&\n> +\t# Ensure that refs/bundles/topic are sent as \"have\".\n> +\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n> +'\n\nUsing \"test_grep\" would make it easier to diagnose when test breaks.\nA failing \"grep\" will be silent (i.e., \"I didn't find anything you\ntold me to look for\"). A failing \"test_grep\" will tell you \"I was\ntold to find this, but didn't find any in that\".\n\n> +test_expect_success 'negotiation: bundle with all wanted commits' '\n> +\ttest_when_finished rm -rf trace*.txt &&\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n> +\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n> +\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n> +\t\tclone-from nego-bundle-all &&\n> +\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n> +\tgrep \"refs/bundles/\" refs >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\trefs/bundles/topic\n> +\tEOF\n> +\ttest_cmp expect actual &&\n> +\t# We already have all needed commits so no \"want\" needed.\n> +\t! grep \"clone> want \" trace-packet.txt\n> +'\n> +\n> +test_expect_success 'negotiation: bundle list (no heuristic)' '\n> +\ttest_when_finished rm -f trace*.txt &&\n> +\tcat >bundle-list <<-EOF &&\n> +\t[bundle]\n> +\t\tversion = 1\n> +\t\tmode = all\n> +\n> +\t[bundle \"bundle-1\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n> +\n> +\t[bundle \"bundle-2\"]\n> +\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n> +\tEOF\n\nOK. This is a good use of here-doc (as opposed to test_write_lines\nI sugested earlier for different purposes). I wondered if these\n$(pwd) and file://$(pwd) are safe (I always get confused by the need\nto sometimes use $PWD to help Windows), but I see them used in what\nDerrick wrote in this file, so they must be fine.\n\nBut there may be characters in the leading part of $(pwd) that we do\nnot control that needs quoting (like a double quote '\"'). The value\nof bundle.*.uri may need to be quoted a bit carefully. This is not\na new problem this patch introduces, so you do not have to rewrite\nthis part of the patch; I'll mark it with #leftoverbits here---the\nidea being somebody else who is too bored can come back, see if it\nis truly broken, and fix them after all dust settles.\n\nAbusing \"git config -f bundle-list\" might be safer, e.g.\n\n\t$ git config -f bundle.list bundle.bundle-1.uri \\\n\t\t'file:///home/abc\"def/t/trash dir/clone-from/b1.bndl'\n\t$ cat bundle.list\n [bundle \"bundle-1\"]\n uri = file:///home/abc\\\"def/t/trash dir/clone-from/b1.bndl\n\nas you do not know what other garbage character is in $(pwd) part.\n\n> +\t# We already have all needed commits so no \"want\" needed.\n> +\t! grep \"clone> want \" trace-packet.txt\n\nJust FYI, to negate test_grep, use\n\n\ttest_grep ! \"clone > want \" trace-packet.txt\n\nnot\t\n\n\t! test_grep \"clone > want \" trace-packet.txt ;# WRONG\n\n"},{"id":"496957","messageId":"xmqq5xufuw09.fsf@gitster.g","threadId":"61479","inReplyTo":"d21c236b8dec9fd8c81a4224c1f63c511b04d8b9.1718109943.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 2/3] fetch-pack: expose fsckObjects configuration logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-11T19:20:22Z","receivedAt":"2024-06-11T19:20:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Xing Xin via GitGitGadget\" writes:\n\n> -\tif (fetch_fsck_objects >= 0\n> -\t ? fetch_fsck_objects\n> -\t : transfer_fsck_objects >= 0\n> -\t ? transfer_fsck_objects\n> -\t : 0)\n> -\t\tfsck_objects = 1;\n> +\tfsck_objects = fetch_pack_fsck_objects();\n> ...\n> +int fetch_pack_fsck_objects(void)\n> +{\n> +\tfetch_pack_setup();\n\nOK, this one is designed to be as lightweight as possible when it\nhas already been called, so a potentially duplicated call would not\ncause too much worries here.\n\n> +\tif (fetch_fsck_objects >= 0)\n> +\t\treturn fetch_fsck_objects;\n> +\tif (transfer_fsck_objects >= 0)\n> +\t\treturn transfer_fsck_objects;\n> +\treturn 0;\n> +}\n\nMuch easier to follow. Nicely done.\n"},{"id":"496959","messageId":"xmqqsexjtfcg.fsf@gitster.g","threadId":"61479","inReplyTo":"53395e8c08a8487f3e53dca15766307854a24b3b.1718109943.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 3/3] unbundle: support object verification for fetches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-11T20:05:35Z","receivedAt":"2024-06-11T20:05:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Xing Xin via GitGitGadget\" writes:\n\n> From: Xing Xin \n>\n> This commit extends object verification support for fetches in\n> `bundle.c:unbundle` by adding the `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`\n> option to `verify_bundle_flags`. When this option is enabled,\n> `bundle.c:unbundle` invokes `fetch-pack.c:fetch_pack_fsck_objects` to\n> determine whether to append the \"--fsck-objects\" flag to\n> \"git-index-pack\".\n\nPlease start your proposed log message by stating what the perceived\nproblem without this patch in the current world is. Without it, you\ncannot easily answer the most basic question: \"why are we doing this?\"\n\nThe name VERIFY_BUNDLE_FSCK_FOLLOW_FETCH does not read very well.\nVERIFY_BUNDLE part is common across various flags, so what is\nspecific to the feature is \"FSCK_FOLLOW_FETCH\", and it is good that\nwe convey the fact that we do a bit more than the normal\nVERIFY_BUNDLE (which is, to read the prerequisite headers and make\nsure we have them in the sense that they are reachable from our\nrefs) with the word FSCK.\n\nBut is it necessary (or even a good idea) to limit its usability\nwith \"FOLLOW_FETCH\" (which does not look even grammatical)? Aren't\nwe closing the door to other folks who may want to do a more\nthorough fsck-like checks in other codepaths by saying \"if you are\nnot doing this immediately after you fetch, you are unwelcome to use\nthis flag\"?\n\n> `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is now passed to `unbundle` in the\n> fetching process, including:\n>\n> - `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n> - `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n>\n> This addition ensures a consistent logic for object verification during\n> fetch operations. Tests have been added to confirm functionality in the\n> scenarios mentioned above.\n>\n> Reviewed-by: Patrick Steinhardt \n> Signed-off-by: Xing Xin \n> ---\n> bundle-uri.c | 2 +-\n> bundle.c | 5 +++++\n> bundle.h | 1 +\n> t/t5558-clone-bundle-uri.sh | 35 ++++++++++++++++++++++++++++++++++-\n> t/t5607-clone-bundle.sh | 33 +++++++++++++++++++++++++++++++++\n> transport.c | 2 +-\n> 6 files changed, 75 insertions(+), 3 deletions(-)\n>\n> diff --git a/bundle-uri.c b/bundle-uri.c\n> index 65666a11d9c..e7ebac6ce57 100644\n> --- a/bundle-uri.c\n> +++ b/bundle-uri.c\n> @@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n> \t * the prerequisite commits.\n> \t */\n> \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n> -\t\t\t VERIFY_BUNDLE_QUIET)))\n> +\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n> \t\treturn 1;\n\nOK (modulo the flag name).\n\n> +\tif (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n> +\t\tif (fetch_pack_fsck_objects())\n> +\t\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n> +\n\nOK, that's quite straight-forward. We are running \"index-pack\n--fix-thin --stdin\" and feeding the bundle data to it. We just tell\nit to also work in the \"--fsck-objects\" mode.\n\n> diff --git a/transport.c b/transport.c\n> index 0ad04b77fd2..6cd5683bb45 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport,\n> \tif (!data->get_refs_from_bundle_called)\n> \t\tget_refs_from_bundle_inner(transport);\n> \tret = unbundle(the_repository, &data->header, data->fd,\n> -\t\t &extra_index_pack_args, 0);\n> +\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH);\n\nOK.\n\nI wonder if something like this is a potential follow-up topic\nsomebody may be interested in after the dust settles. That is\nexactly why the name of this bit matters.\n\n\n\n builtin/bundle.c | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git c/builtin/bundle.c w/builtin/bundle.c\nindex d5d41a8f67..eeb5963dcb 100644\n--- c/builtin/bundle.c\n+++ w/builtin/bundle.c\n@@ -194,10 +194,13 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n \tint bundle_fd = -1;\n \tint ret;\n \tint progress = isatty(2);\n+\tint fsck_objects = 0;\n \n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"progress\", &progress,\n \t\t\t N_(\"show progress meter\")),\n+\t\tOPT_BOOL(0, \"fsck-objects\", &fsck_objects,\n+\t\t\t N_(\"check the objects in the bundle\")),\n \t\tOPT_END()\n \t};\n \tchar *bundle_file;\n@@ -217,7 +220,8 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n \t\tstrvec_pushl(&extra_index_pack_args, \"-v\", \"--progress-title\",\n \t\t\t _(\"Unbundling objects\"), NULL);\n \tret = !!unbundle(the_repository, &header, bundle_fd,\n-\t\t\t &extra_index_pack_args, 0) ||\n+\t\t\t &extra_index_pack_args,\n+\t\t\t fsck_objects ? VERIFY_BUNDLE_FSCK_FOLLOW_FETCH : 0) ||\n \t\tlist_bundle_refs(&header, argc, argv);\n \tbundle_header_release(&header);\n cleanup:\n\n"},{"id":"497043","messageId":"5ee13e97.c24f.1900dba4822.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"xmqqsexjtfcg.fsf@gitster.g","subject":"Re:Re: [PATCH v6 3/3] unbundle: support object verification for fetches","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-06-12T18:33:50Z","receivedAt":"2024-06-12T18:34:17Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-06-12 04:05:35, \"Junio C Hamano\" wrote:\n>\"Xing Xin via GitGitGadget\" writes:\n>\n>> From: Xing Xin \n>>\n>> This commit extends object verification support for fetches in\n>> `bundle.c:unbundle` by adding the `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`\n>> option to `verify_bundle_flags`. When this option is enabled,\n>> `bundle.c:unbundle` invokes `fetch-pack.c:fetch_pack_fsck_objects` to\n>> determine whether to append the \"--fsck-objects\" flag to\n>> \"git-index-pack\".\n>\n>Please start your proposed log message by stating what the perceived\n>problem without this patch in the current world is. Without it, you\n>cannot easily answer the most basic question: \"why are we doing this?\"\n\nGot it.\n\n>The name VERIFY_BUNDLE_FSCK_FOLLOW_FETCH does not read very well.\n>VERIFY_BUNDLE part is common across various flags, so what is\n>specific to the feature is \"FSCK_FOLLOW_FETCH\", and it is good that\n>we convey the fact that we do a bit more than the normal\n>VERIFY_BUNDLE (which is, to read the prerequisite headers and make\n>sure we have them in the sense that they are reachable from our\n>refs) with the word FSCK.\n>\n>But is it necessary (or even a good idea) to limit its usability\n>with \"FOLLOW_FETCH\" (which does not look even grammatical)? Aren't\n>we closing the door to other folks who may want to do a more\n>thorough fsck-like checks in other codepaths by saying \"if you are\n>not doing this immediately after you fetch, you are unwelcome to use\n>this flag\"?\n\nI initially considered adding another option VERIFY_BUNDLE_FSCK_ALWAYS\nfor other scenarios, which would take a higher priority than\nVERIFY_BUNDLE_FSCK_FOLLOW_FETCH. However, that approach is also\nconfusing, as we would have two flags both controlling the fsck\nbehavior.\n\nHow about extending VERIFY_BUNDLE_FSCK and letting the callers decide\nwhether to add the flag for fscking?\n\nIn bundle.c, we can make a small change like:\n\n@@ -625,6 +626,9 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n\n+\tif (flags & VERIFY_BUNDLE_FSCK)\n+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\n\nFor example, in `bundle-uri.c:unbundle_from_file`, which is one of the\ncallers of unbundle, we can use `fetch_pack_fsck_objects` to decide\nwhether to add that option:\n\n@@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET | (fetch_pack_fsck_objects() ? VERIFY_BUNDLE_FSCK : 0))))\n \t\treturn 1;\n\nThis approach should streamline the code while maintaining flexibility.\nThe follow-up patch you mentioned below should just work then, it is not\nfor now because we are touching the unwanted `fetch_pack_fsck_objects`\nwithin `unbundle`.\n\nXing Xin\n\n>I wonder if something like this is a potential follow-up topic\n>somebody may be interested in after the dust settles. That is\n>exactly why the name of this bit matters.\n>\n>\n>\n> builtin/bundle.c | 6 +++++-\n> 1 file changed, 5 insertions(+), 1 deletion(-)\n>\n>diff --git c/builtin/bundle.c w/builtin/bundle.c\n>index d5d41a8f67..eeb5963dcb 100644\n>--- c/builtin/bundle.c\n>+++ w/builtin/bundle.c\n>@@ -194,10 +194,13 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n> \tint bundle_fd = -1;\n> \tint ret;\n> \tint progress = isatty(2);\n>+\tint fsck_objects = 0;\n> \n> \tstruct option options[] = {\n> \t\tOPT_BOOL(0, \"progress\", &progress,\n> \t\t\t N_(\"show progress meter\")),\n>+\t\tOPT_BOOL(0, \"fsck-objects\", &fsck_objects,\n>+\t\t\t N_(\"check the objects in the bundle\")),\n> \t\tOPT_END()\n> \t};\n> \tchar *bundle_file;\n>@@ -217,7 +220,8 @@ static int cmd_bundle_unbundle(int argc, const char **argv, const char *prefix)\n> \t\tstrvec_pushl(&extra_index_pack_args, \"-v\", \"--progress-title\",\n> \t\t\t _(\"Unbundling objects\"), NULL);\n> \tret = !!unbundle(the_repository, &header, bundle_fd,\n>-\t\t\t &extra_index_pack_args, 0) ||\n>+\t\t\t &extra_index_pack_args,\n>+\t\t\t fsck_objects ? VERIFY_BUNDLE_FSCK_FOLLOW_FETCH : 0) ||\n> \t\tlist_bundle_refs(&header, argc, argv);\n> \tbundle_header_release(&header);\n> cleanup:\n"},{"id":"497244","messageId":"77366503.bef9.1902679d458.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"xmqqa5jruwkr.fsf@gitster.g","subject":"Re:Re: [PATCH v6 1/3] bundle-uri: verify oid before writing refs","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-06-17T13:53:56Z","receivedAt":"2024-06-17T13:54:20Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-06-12 03:08:04, \"Junio C Hamano\" wrote:\n>\"Xing Xin via GitGitGadget\" writes:\n>\n>> This commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag\n>\n>\"Fix the bug by ...\".\n>\n>> when writing bundle refs. When `refs.c:refs_update_ref` is called to to\n>\n>\"to to\"\n\nAh, always typos. :-)\n\n[snip]\n>> test_expect_success 'create bundle' '\n>> \tgit init clone-from &&\n>> -\tgit -C clone-from checkout -b topic &&\n>> -\ttest_commit -C clone-from A &&\n>> -\ttest_commit -C clone-from B &&\n>> -\tgit -C clone-from bundle create B.bundle topic\n>> +\t(\n>> +\t\tcd clone-from &&\n>> +\t\tgit checkout -b topic &&\n>> +\n>> +\t\ttest_commit A &&\n>> +\t\tgit bundle create A.bundle topic &&\n>> +\n>> +\t\ttest_commit B &&\n>> +\t\tgit bundle create B.bundle topic &&\n>> +\n>> +\t\t# Create a bundle with reference pointing to non-existent object.\n>> +\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n>\n>I suspect that this would be terribly unportable. The early part of\n>a bundle file may be text and sed may be able to grok but are you\n>sure everybody's implementation of sed would not barf (or even\n>worse, corrupt) the pack stream data that follows?\n>\n>The code used in t/lib-bundle.sh:convert_bundle_to_pack() has been\n>in use since 8315588b (bundle: fix wrong check of read_header()'s\n>return value & add tests, 2007-03-06), so munging the bundle with\n>a code similar to it may have a better portability story.\n>\n>Add something like:\n>\n> corrupt_bundle_header () {\n> sed -e '/^$/q' \"$@\"\n> cat\n> }\n>\n>to t/lib-bundle.sh, which can take an arbitrary sequence of command\n>line parameters to drive \"sed\", and can be used like so:\n>\n>\tcorrupt_bundle_header \\\n>\t\t-e \"s/^$(git rev-parse A) /$(git rev-parse B) /\" \\\n>\t\tB.bndl\n>\n>perhaps?\n\nThanks, I never knew sed could be used this way! It's so concise!\n\nHowever, after applying these code, the added test \"t5558.5 clone with\nbundle that has bad header\" fails in some CI jobs showing that we can\nnot get expected error. For example CI \"linux-musl (alpine)\" gives:\n\n\t+ git clone '--bundle-uri=clone-from/bad-header.bundle' clone-from clone-bad-header\n\t+ git -C clone-from rev-parse B\n\t+ commit_b=d9df4505cb3522088b9e29d6051ac16f1564154a\n\t+ test_grep 'trying to write ref '\"'\"'refs/bundles/topic'\"'\"' with nonexistent object d9df4505cb3522088b9e29d6051ac16f1564154a' err\n\t+ eval 'last_arg=${2}'\n\t+ last_arg=err\n\t+ test -f err\n\t+ test 2 -lt 2\n\t+ test 'x!' '=' 'xtrying to write ref '\"'\"'refs/bundles/topic'\"'\"' with nonexistent object d9df4505cb3522088b9e29d6051ac16f1564154a'\n\t+ test 'x!' '=' 'xtrying to write ref '\"'\"'refs/bundles/topic'\"'\"' with nonexistent object d9df4505cb3522088b9e29d6051ac16f1564154a'\n\t+ grep 'trying to write ref '\"'\"'refs/bundles/topic'\"'\"' with nonexistent object d9df4505cb3522088b9e29d6051ac16f1564154a' err\n\t+ echo 'error: '\"'\"'grep trying to write ref '\"'\"'refs/bundles/topic'\"'\"' with nonexistent object d9df4505cb3522088b9e29d6051ac16f1564154a' 'err'\"'\"' didn'\"'\"'t find a match in:'\n\terror: 'grep trying to write ref 'refs/bundles/topic' with nonexistent object d9df4505cb3522088b9e29d6051ac16f1564154a err' didn't find a match in:\n\t+ test -s err\n\t+ cat err\n\tCloning into 'clone-bad-header'...\n\tfatal: pack signature mismatch\n\terror: index-pack died\n\tdone.\n\t+ return 1\n\terror: last command exited with $?=1\n\tnot ok 5 - clone with bundle that has bad header\n\n\t+# Create a bundle with reference pointing to non-existent object.\n\t+sed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n\t+\tbad-header.bundle &&\n\t+convert_bundle_to_pack \\\n\t+\t>bad-header.bundle\n\nMore details can be found at https://github.com/blanet/git/actions/runs/9541478191/job/26294731254.\n\nAfter some digging, I discovered that inside the Alpine container,\n`corrupt_bundle_header` is missing the leading \"PACK\\x00\" in the pack\nsection of the converted bundle. This is likely caused by the\nincompatibility of \"sed\" across different operating systems. I stopped\ninvestigating further due to my unfamiliarity with containers and \"sed\"\nitself. Instead, I found another approach that utilizes the suggested\nusage of \"sed\" and `lib-bundle.sh:convert_bundle_to_pack`.\n\n \t+# Create a bundle with reference pointing to non-existent object.\n\t+sed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n\t+\tbad-header.bundle &&\n\t+convert_bundle_to_pack \\\n\t+\t>bad-header.bundle\n\n>> @@ -33,6 +42,16 @@ test_expect_success 'clone with path bundle' '\n>> \ttest_cmp expect actual\n>> '\n>> \n>> +test_expect_success 'clone with bundle that has bad header' '\n>> +\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n>> +\t\tclone-from clone-bad-header 2>err &&\n>> +\t# Write bundle ref fails, but clone can still proceed.\n>> +\tcommit_b=$(git -C clone-from rev-parse B) &&\n>> +\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n>> +\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n>> +\t! grep \"refs/bundles/\" refs\n>> +'\n>> +\n>\n>So this is the test the proposed log message discussed. The\n>description gave a false impression that the \"broken header\" test\n>that has not much to do with the bug being fixed was the only added\n>test---we probably want to correct that impression.\n\nAdjusted the commit message.\n\n>> @@ -259,6 +278,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n>> \t! grep \"refs/bundles/\" refs\n>> '\n>> \n>> +#########################################################################\n>> +# Clone negotiation related tests begin here\n>\n>Drop this divider and comment. The HTTP tests you see below has a\n>much better reason to be separated like that in order to warn test\n>writers (they shouldn't add their random new tests after that point,\n>because everything after that one is skipped when HTTPD tests are\n>disabled---see the beginning of t/lib-httpd.sh which is included\n>after that divider line), but everything here you added is not\n>special. Everybody should run these tests.\n\nThanks for the explanation, dropped the divider in new series.\n\n>> +test_expect_success 'negotiation: bundle with part of wanted commits' '\n>> +\ttest_when_finished rm -rf trace*.txt &&\n>\n>Do not overly depend on glob not matching at this point when you\n>establish the when-finished handler (or glob matching the files you\n>want to catch and later test not adding anything you would want to\n>clean). Quote \"rm -f trace*.txt\" and drop \"r\" if you do not absolutely\n>need it (and I would imagine you don't, given the .txt suffix).\n\nYes, the \"-r\" is not safe and I just need to clean the \"*.txt\", not involving directories.\n\n>> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n>> +\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n>> +\t\tclone-from nego-bundle-part &&\n>> +\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n>> +\tgrep \"refs/bundles/\" refs >actual &&\n>> +\tcat >expect <<-\\EOF &&\n>> +\trefs/bundles/topic\n>> +\tEOF\n>\n>Hmph, if the expected pattern is only a few lines without any magic,\n>\n>\ttest_write_lines >expect refs/bundles/topic &&\n>\n>may be easier to follow.\n\nApplied!\n\n>> +\ttest_cmp expect actual &&\n>> +\t# Ensure that refs/bundles/topic are sent as \"have\".\n>> +\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n>> +'\n>\n>Using \"test_grep\" would make it easier to diagnose when test breaks.\n>A failing \"grep\" will be silent (i.e., \"I didn't find anything you\n>told me to look for\"). A failing \"test_grep\" will tell you \"I was\n>told to find this, but didn't find any in that\".\n\nThanks, it just accelerated the digging process of the test failure\nmentioned above. :)\n\n>> +test_expect_success 'negotiation: bundle with all wanted commits' '\n>> +\ttest_when_finished rm -rf trace*.txt &&\n>> +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n>> +\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n>> +\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n>> +\t\tclone-from nego-bundle-all &&\n>> +\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n>> +\tgrep \"refs/bundles/\" refs >actual &&\n>> +\tcat >expect <<-\\EOF &&\n>> +\trefs/bundles/topic\n>> +\tEOF\n>> +\ttest_cmp expect actual &&\n>> +\t# We already have all needed commits so no \"want\" needed.\n>> +\t! grep \"clone> want \" trace-packet.txt\n>> +'\n>> +\n>> +test_expect_success 'negotiation: bundle list (no heuristic)' '\n>> +\ttest_when_finished rm -f trace*.txt &&\n>> +\tcat >bundle-list <<-EOF &&\n>> +\t[bundle]\n>> +\t\tversion = 1\n>> +\t\tmode = all\n>> +\n>> +\t[bundle \"bundle-1\"]\n>> +\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n>> +\n>> +\t[bundle \"bundle-2\"]\n>> +\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n>> +\tEOF\n>\n>OK. This is a good use of here-doc (as opposed to test_write_lines\n>I sugested earlier for different purposes). I wondered if these\n>$(pwd) and file://$(pwd) are safe (I always get confused by the need\n>to sometimes use $PWD to help Windows), but I see them used in what\n>Derrick wrote in this file, so they must be fine.\n>\n>But there may be characters in the leading part of $(pwd) that we do\n>not control that needs quoting (like a double quote '\"'). The value\n>of bundle.*.uri may need to be quoted a bit carefully. This is not\n>a new problem this patch introduces, so you do not have to rewrite\n>this part of the patch; I'll mark it with #leftoverbits here---the\n>idea being somebody else who is too bored can come back, see if it\n>is truly broken, and fix them after all dust settles.\n>\n>Abusing \"git config -f bundle-list\" might be safer, e.g.\n>\n>\t$ git config -f bundle.list bundle.bundle-1.uri \\\n>\t\t'file:///home/abc\"def/t/trash dir/clone-from/b1.bndl'\n>\t$ cat bundle.list\n> [bundle \"bundle-1\"]\n> uri = file:///home/abc\\\"def/t/trash dir/clone-from/b1.bndl\n>\n>as you do not know what other garbage character is in $(pwd) part.\n>\n>> +\t# We already have all needed commits so no \"want\" needed.\n>> +\t! grep \"clone> want \" trace-packet.txt\n>\n>Just FYI, to negate test_grep, use\n>\n>\ttest_grep ! \"clone > want \" trace-packet.txt\n>\n>not\t\n>\n>\t! test_grep \"clone > want \" trace-packet.txt ;# WRONG\n\nThanks your through review!\n\nXing Xin\n\n"},{"id":"497245","messageId":"pull.1730.v7.git.1718632535.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v6.git.1718109943.gitgitgadget@gmail.com","subject":"[PATCH v7 0/3] object checking related additions and fixes for bundles in fetches","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-17T13:55:32Z","receivedAt":"2024-06-17T13:55:40Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"While attempting to fix a reference negotiation bug in bundle-uri, we\nidentified that the fetch process lacks some crucial object validation\nchecks when processing bundles. The primary issues are:\n\n 1. In the bundle-uri scenario, object IDs were not validated before writing\n bundle references. This was the root cause of the original negotiation\n bug in bundle-uri and could lead to potential repository corruption.\n 2. The existing \"fetch.fsckObjects\" and \"transfer.fsckObjects\"\n configurations were not applied when directly fetching bundles or\n fetching with bundle-uri enabled. In fact, there were no object\n validation supports for unbundle.\n\nThe first patch addresses the bundle-uri negotiation issue by removing the\nREF_SKIP_OID_VERIFICATION flag when writing bundle references.\n\nPatches 2 through 3 extend verify_bundle_flags for bundle.c:unbundle to add\nsupport for object validation (fsck) in fetch scenarios, mainly following\nthe suggestions from Junio and Patrick on the mailing list.\n\nXing Xin (3):\n bundle-uri: verify oid before writing refs\n fetch-pack: expose fsckObjects configuration logic\n unbundle: extend object verification for fetches\n\n bundle-uri.c | 6 +-\n bundle.c | 3 +\n bundle.h | 1 +\n fetch-pack.c | 17 ++--\n fetch-pack.h | 5 +\n t/t5558-clone-bundle-uri.sh | 181 +++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++\n transport.c | 3 +-\n 8 files changed, 235 insertions(+), 14 deletions(-)\n\n\nbase-commit: b9cfe4845cb2562584837bc0101c0ab76490a239\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v7\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v7\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\nRange-diff vs v6:\n\n 1: e958a3ab20c ! 1: fc9f44fda00 bundle-uri: verify oid before writing refs\n @@ Commit message\n be found for negotiation because it exists in \"incr.pack\", which is not\n included in `packed_git`.\n \n - This commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag\n - when writing bundle refs. When `refs.c:refs_update_ref` is called to to\n - write the corresponding bundle refs, it triggers\n - `refs.c:ref_transaction_commit`. This, in turn, invokes\n - `refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of\n - the refs storage backend. For files backend, this function is\n - `files-backend.c:files_transaction_prepare`, and for reftable backend,\n - it is `reftable-backend.c:reftable_be_transaction_prepare`. Both\n - functions eventually call `object.c:parse_object`, which can invoke\n + Fix the bug by removing `REF_SKIP_OID_VERIFICATION` flag when writing\n + bundle refs. When `refs.c:refs_update_ref` is called to write the\n + corresponding bundle refs, it triggers `refs.c:ref_transaction_commit`.\n + This, in turn, invokes `refs.c:ref_transaction_prepare`, which calls\n + `transaction_prepare` of the refs storage backend. For files backend, it\n + is `files-backend.c:files_transaction_prepare`, and for reftable\n + backend, it is `reftable-backend.c:reftable_be_transaction_prepare`.\n + Both functions eventually call `object.c:parse_object`, which can invoke\n `packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures\n that bundle refs point to valid objects and that all tips from bundle\n refs are correctly parsed during subsequent negotiations.\n \n - A test has been added to demonstrate that bundles with incorrect\n - headers, where refs point to non-existent objects, do not result in any\n - bundle refs being created in the repository. Additionally, a set of\n - negotiation-related tests for fetching with bundle-uri has been\n - included.\n + A set of negotiation-related tests for cloning with bundle-uri has been\n + included to demonstrate that downloaded bundles are utilized to\n + accelerate fetching.\n +\n + Additionally, another test has been added to show that bundles with\n + incorrect headers, where refs point to non-existent objects, do not\n + result in any bundle refs being created in the repository.\n \n Reviewed-by: Karthik Nayak \n Reviewed-by: Patrick Steinhardt \n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *fi\n \tbundle_header_release(&header);\n \n ## t/t5558-clone-bundle-uri.sh ##\n +@@\n + test_description='test fetching bundles with --bundle-uri'\n + \n + . ./test-lib.sh\n ++. \"$TEST_DIRECTORY\"/lib-bundle.sh\n + \n + test_expect_success 'fail to clone from non-existent file' '\n + \ttest_when_finished rm -rf test &&\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'fail to clone from non-bundle file' '\n \n test_expect_success 'create bundle' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'fail to clone from non-bundle\n +\t\tgit bundle create B.bundle topic &&\n +\n +\t\t# Create a bundle with reference pointing to non-existent object.\n -+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n ++\t\tsed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n ++\t\t\tbad-header.bundle &&\n ++\t\tconvert_bundle_to_pack \\\n ++\t\t\t>bad-header.bundle\n +\t)\n '\n \n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with path bundle' '\n '\n \n +test_expect_success 'clone with bundle that has bad header' '\n ++\t# Write bundle ref fails, but clone can still proceed.\n +\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n +\t\tclone-from clone-bad-header 2>err &&\n -+\t# Write bundle ref fails, but clone can still proceed.\n +\tcommit_b=$(git -C clone-from rev-parse B) &&\n +\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n +\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n \t! grep \"refs/bundles/\" refs\n '\n \n -+#########################################################################\n -+# Clone negotiation related tests begin here\n -+\n +test_expect_success 'negotiation: bundle with part of wanted commits' '\n -+\ttest_when_finished rm -rf trace*.txt &&\n ++\ttest_when_finished \"rm -f trace*.txt\" &&\n +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n +\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n +\t\tclone-from nego-bundle-part &&\n +\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n +\tgrep \"refs/bundles/\" refs >actual &&\n -+\tcat >expect <<-\\EOF &&\n -+\trefs/bundles/topic\n -+\tEOF\n ++\ttest_write_lines refs/bundles/topic >expect &&\n +\ttest_cmp expect actual &&\n +\t# Ensure that refs/bundles/topic are sent as \"have\".\n -+\tgrep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n ++\ttest_grep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle with all wanted commits' '\n -+\ttest_when_finished rm -rf trace*.txt &&\n ++\ttest_when_finished \"rm -f trace*.txt\" &&\n +\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n +\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n +\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n +\t\tclone-from nego-bundle-all &&\n +\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n +\tgrep \"refs/bundles/\" refs >actual &&\n -+\tcat >expect <<-\\EOF &&\n -+\trefs/bundles/topic\n -+\tEOF\n ++\ttest_write_lines refs/bundles/topic >expect &&\n +\ttest_cmp expect actual &&\n +\t# We already have all needed commits so no \"want\" needed.\n +\t! grep \"clone> want \" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle list (no heuristic)' '\n -+\ttest_when_finished rm -f trace*.txt &&\n ++\ttest_when_finished \"rm -f trace*.txt\" &&\n +\tcat >bundle-list <<-EOF &&\n +\t[bundle]\n +\t\tversion = 1\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n +\trefs/bundles/left\n +\tEOF\n +\ttest_cmp expect actual &&\n -+\tgrep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n ++\ttest_grep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle list (creationToken)' '\n -+\ttest_when_finished rm -f trace*.txt &&\n ++\ttest_when_finished \"rm -f trace*.txt\" &&\n +\tcat >bundle-list <<-EOF &&\n +\t[bundle]\n +\t\tversion = 1\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n +\trefs/bundles/left\n +\tEOF\n +\ttest_cmp expect actual &&\n -+\tgrep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n ++\ttest_grep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle list with all wanted commits' '\n -+\ttest_when_finished rm -f trace*.txt &&\n ++\ttest_when_finished \"rm -f trace*.txt\" &&\n +\tcat >bundle-list <<-EOF &&\n +\t[bundle]\n +\t\tversion = 1\n 2: d21c236b8de = 2: 3dc0d9dd22f fetch-pack: expose fsckObjects configuration logic\n 3: 53395e8c08a ! 3: 2f15099bbb9 unbundle: support object verification for fetches\n @@ Metadata\n Author: Xing Xin \n \n ## Commit message ##\n - unbundle: support object verification for fetches\n + unbundle: extend object verification for fetches\n \n - This commit extends object verification support for fetches in\n - `bundle.c:unbundle` by adding the `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`\n - option to `verify_bundle_flags`. When this option is enabled,\n - `bundle.c:unbundle` invokes `fetch-pack.c:fetch_pack_fsck_objects` to\n - determine whether to append the \"--fsck-objects\" flag to\n - \"git-index-pack\".\n + The existing fetch.fsckObjects and transfer.fsckObjects configurations\n + were not fully applied to bundle-involved fetches, including direct\n + bundle fetches and bundle-uri enabled fetches. Furthermore, there was no\n + object verification support for unbundle.\n \n - `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` is now passed to `unbundle` in the\n - fetching process, including:\n + This commit extends object verification support in `bundle.c:unbundle`\n + by adding the `VERIFY_BUNDLE_FSCK` option to `verify_bundle_flags`. When\n + this option is enabled, we append the `--fsck-objects` flag to\n + `git-index-pack`.\n +\n + The `VERIFY_BUNDLE_FSCK` option is now used by bundle-involved fetches,\n + where we use `fetch-pack.c:fetch_pack_fsck_objects` to determine whether\n + to enable this option for `bundle.c:unbundle`, specifically in:\n \n - `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n - `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n \n This addition ensures a consistent logic for object verification during\n - fetch operations. Tests have been added to confirm functionality in the\n - scenarios mentioned above.\n + fetches. Tests have been added to confirm functionality in the scenarios\n + mentioned above.\n \n Reviewed-by: Patrick Steinhardt \n Signed-off-by: Xing Xin \n \n ## bundle-uri.c ##\n +@@\n + #include \"hashmap.h\"\n + #include \"pkt-line.h\"\n + #include \"config.h\"\n ++#include \"fetch-pack.h\"\n + #include \"remote.h\"\n + \n + static struct {\n @@ bundle-uri.c: static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n -\t\t\t VERIFY_BUNDLE_QUIET)))\n -+\t\t\t VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)))\n ++\t\t\t VERIFY_BUNDLE_QUIET | (fetch_pack_fsck_objects() ? VERIFY_BUNDLE_FSCK : 0))))\n \t\treturn 1;\n \n \t/*\n \n ## bundle.c ##\n -@@\n - #include \"list-objects-filter-options.h\"\n - #include \"connected.h\"\n - #include \"write-or-die.h\"\n -+#include \"fetch-pack.h\"\n - \n - static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n - static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n -+\tif (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH)\n -+\t\tif (fetch_pack_fsck_objects())\n -+\t\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n ++\tif (flags & VERIFY_BUNDLE_FSCK)\n ++\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n +\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n @@ bundle.h: int create_bundle(struct repository *r, const char *path,\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n -+\tVERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 2),\n ++\tVERIFY_BUNDLE_FSCK = (1 << 2),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\n \n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'create bundle' '\n - \t\tgit bundle create B.bundle topic &&\n - \n - \t\t# Create a bundle with reference pointing to non-existent object.\n --\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle\n -+\t\tsed \"s/$(git rev-parse A)/$(git rev-parse B)/\" bad-header.bundle &&\n + \t\tsed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n + \t\t\tbad-header.bundle &&\n + \t\tconvert_bundle_to_pack \\\n +-\t\t\t>bad-header.bundle\n ++\t\t\t>bad-header.bundle &&\n +\n +\t\tcat >data <<-EOF &&\n +\t\ttree $(git rev-parse HEAD^{tree})\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with bundle that has bad\n +\t\tclone-from clone-bad-object-no-fsck &&\n +\tgit -C clone-bad-object-no-fsck for-each-ref --format=\"%(refname)\" >refs &&\n +\tgrep \"refs/bundles/\" refs >actual &&\n -+\tcat >expect <<-\\EOF &&\n -+\trefs/bundles/bad\n -+\tEOF\n ++\ttest_write_lines refs/bundles/bad >expect &&\n +\ttest_cmp expect actual &&\n +\n +\t# Unbundle fails with fsckObjects set true, but clone can still proceed.\n @@ transport.c: static int fetch_refs_from_bundle(struct transport *transport,\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n -\t\t &extra_index_pack_args, 0);\n -+\t\t &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH);\n ++\t\t &extra_index_pack_args,\n ++\t\t fetch_pack_fsck_objects() ? VERIFY_BUNDLE_FSCK : 0);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n\n-- \ngitgitgadget\n"},{"id":"497246","messageId":"fc9f44fda0032ab1e5ee0c9bcc2886ddb8e17722.1718632536.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v7.git.1718632535.gitgitgadget@gmail.com","subject":"[PATCH v7 1/3] bundle-uri: verify oid before writing refs","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-17T13:55:33Z","receivedAt":"2024-06-17T13:55:40Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not discovered, thus resulting in rather slow\nclones. This was particularly problematic when employing the\n\"creationTokens\" heuristic.\n\nTo reproduce this issue, consider a repository with a single branch\n\"main\" pointing to commit \"A\". Firstly, create a base bundle with:\n\n git bundle create base.bundle main\n\nThen, add a new commit \"B\" on top of \"A\", and create an incremental\nbundle for \"main\":\n\n git bundle create incr.bundle A..main\n\nNow, generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above should result in a reference\n\"refs/bundles/main\" pointing to \"B\" in the new repository. However, git\nwould still download everything from the server, as if it had fetched\nnothing locally.\n\nSo why the \"refs/bundles/main\" is not discovered? After some digging I\nfound that:\n\n1. Bundles in bundle list are downloaded to local files via\n `bundle-uri.c:download_bundle_list` or via\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which\n is called by `bundle-uri.c:unbundle_all_bundles` or called within\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n3. To get all prerequisites of the bundle, the bundle header is read\n inside `bundle-uri.c:unbundle_from_file` to by calling\n `bundle.c:read_bundle_header`.\n4. Then it calls `bundle.c:unbundle`, which calls\n `bundle.c:verify_bundle` to ensure the repository contains all the\n prerequisites.\n5. `bundle.c:verify_bundle` calls `parse_object`, which eventually\n invokes `packfile.c:prepare_packed_git` or\n `packfile.c:reprepare_packed_git`, filling\n `raw_object_store->packed_git` and setting `packed_git_initialized`.\n6. If `bundle.c:unbundle` succeeds, it writes refs via\n `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here\n bundle refs which can target arbitrary objects are written to the\n repository.\n7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions\n `fetch-pack.c:mark_complete_and_common_ref` and\n `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to\n find local tips for negotiation. The `OBJECT_INFO_QUICK` flag\n prevents `packfile.c:reprepare_packed_git` from being called,\n resulting in failures to parse OIDs that reside only in the latest\n bundle.\n\nIn the example above, when unbunding \"incr.bundle\", \"base.pack\" is added\nto `packed_git` due to prerequisites verification. However, \"B\" cannot\nbe found for negotiation because it exists in \"incr.pack\", which is not\nincluded in `packed_git`.\n\nFix the bug by removing `REF_SKIP_OID_VERIFICATION` flag when writing\nbundle refs. When `refs.c:refs_update_ref` is called to write the\ncorresponding bundle refs, it triggers `refs.c:ref_transaction_commit`.\nThis, in turn, invokes `refs.c:ref_transaction_prepare`, which calls\n`transaction_prepare` of the refs storage backend. For files backend, it\nis `files-backend.c:files_transaction_prepare`, and for reftable\nbackend, it is `reftable-backend.c:reftable_be_transaction_prepare`.\nBoth functions eventually call `object.c:parse_object`, which can invoke\n`packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures\nthat bundle refs point to valid objects and that all tips from bundle\nrefs are correctly parsed during subsequent negotiations.\n\nA set of negotiation-related tests for cloning with bundle-uri has been\nincluded to demonstrate that downloaded bundles are utilized to\naccelerate fetching.\n\nAdditionally, another test has been added to show that bundles with\nincorrect headers, where refs point to non-existent objects, do not\nresult in any bundle refs being created in the repository.\n\nReviewed-by: Karthik Nayak \nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 +-\n t/t5558-clone-bundle-uri.sh | 150 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 147 insertions(+), 6 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91b3319a5c1..65666a11d9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\trefs_update_ref(get_main_ref_store(the_repository),\n \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n \t\t\t\thas_old ? &old_oid : NULL,\n-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n+\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n \t}\n \n \tbundle_header_release(&header);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..2dcdd238a90 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -3,6 +3,7 @@\n test_description='test fetching bundles with --bundle-uri'\n \n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-bundle.sh\n \n test_expect_success 'fail to clone from non-existent file' '\n \ttest_when_finished rm -rf test &&\n@@ -19,10 +20,22 @@ test_expect_success 'fail to clone from non-bundle file' '\n \n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n-\tgit -C clone-from checkout -b topic &&\n-\ttest_commit -C clone-from A &&\n-\ttest_commit -C clone-from B &&\n-\tgit -C clone-from bundle create B.bundle topic\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b topic &&\n+\n+\t\ttest_commit A &&\n+\t\tgit bundle create A.bundle topic &&\n+\n+\t\ttest_commit B &&\n+\t\tgit bundle create B.bundle topic &&\n+\n+\t\t# Create a bundle with reference pointing to non-existent object.\n+\t\tsed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n+\t\t\tbad-header.bundle &&\n+\t\tconvert_bundle_to_pack \\\n+\t\t\t>bad-header.bundle\n+\t)\n '\n \n test_expect_success 'clone with path bundle' '\n@@ -33,6 +46,16 @@ test_expect_success 'clone with path bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone with bundle that has bad header' '\n+\t# Write bundle ref fails, but clone can still proceed.\n+\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n+\t\tclone-from clone-bad-header 2>err &&\n+\tcommit_b=$(git -C clone-from rev-parse B) &&\n+\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n+\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\n@@ -259,6 +282,125 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\ttest_write_lines refs/bundles/topic >expect &&\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\ttest_grep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\ttest_write_lines refs/bundles/topic >expect &&\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\ttest_grep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\ttest_grep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\t! grep \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n-- \ngitgitgadget\n\n"},{"id":"497247","messageId":"3dc0d9dd22f64f79f7cddc453c43701f73f0e133.1718632536.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v7.git.1718632535.gitgitgadget@gmail.com","subject":"[PATCH v7 2/3] fetch-pack: expose fsckObjects configuration logic","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-17T13:55:34Z","receivedAt":"2024-06-17T13:55:42Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nCurrently, we can use \"transfer.fsckObjects\" and the more specific\n\"fetch.fsckObjects\" to control checks for broken objects in received\npacks during fetches. However, these configurations were only\nacknowledged by `fetch-pack.c:get_pack` and did not take effect in\ndirect bundle fetches and fetches with _bundle-uri_ enabled.\n\nThis commit exposes the fetch-then-transfer configuration logic by\nadding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This\nnew function is used to replace the assignment for `fsck_objects` in\n`fetch-pack.c:get_pack`. In the next commit, it will also be used by\n`bundle.c:unbundle` to better fit fetching scenarios.\n\nHelped-by: Junio C Hamano \nHelped-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n fetch-pack.c | 17 +++++++++++------\n fetch-pack.h | 5 +++++\n 2 files changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d2aef21add..3acff2baf09 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\tstrvec_push(&cmd.args, alternate_shallow_file);\n \t}\n \n-\tif (fetch_fsck_objects >= 0\n-\t ? fetch_fsck_objects\n-\t : transfer_fsck_objects >= 0\n-\t ? transfer_fsck_objects\n-\t : 0)\n-\t\tfsck_objects = 1;\n+\tfsck_objects = fetch_pack_fsck_objects();\n \n \tif (do_keep || args->from_promisor || index_pack_args || fsck_objects) {\n \t\tif (pack_lockfiles || fsck_objects)\n@@ -2046,6 +2041,16 @@ static const struct object_id *iterate_ref_map(void *cb_data)\n \treturn &ref->old_oid;\n }\n \n+int fetch_pack_fsck_objects(void)\n+{\n+\tfetch_pack_setup();\n+\tif (fetch_fsck_objects >= 0)\n+\t\treturn fetch_fsck_objects;\n+\tif (transfer_fsck_objects >= 0)\n+\t\treturn transfer_fsck_objects;\n+\treturn 0;\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[],\n \t\t const struct ref *ref,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 6775d265175..b5c579cdae2 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -101,4 +101,9 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+/*\n+ * Return true if checks for broken objects in received pack are required.\n+ */\n+int fetch_pack_fsck_objects(void);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"497248","messageId":"2f15099bbb9fed4919e829274b7683ace9094d15.1718632536.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v7.git.1718632535.gitgitgadget@gmail.com","subject":"[PATCH v7 3/3] unbundle: extend object verification for fetches","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-17T13:55:35Z","receivedAt":"2024-06-17T13:55:43Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThe existing fetch.fsckObjects and transfer.fsckObjects configurations\nwere not fully applied to bundle-involved fetches, including direct\nbundle fetches and bundle-uri enabled fetches. Furthermore, there was no\nobject verification support for unbundle.\n\nThis commit extends object verification support in `bundle.c:unbundle`\nby adding the `VERIFY_BUNDLE_FSCK` option to `verify_bundle_flags`. When\nthis option is enabled, we append the `--fsck-objects` flag to\n`git-index-pack`.\n\nThe `VERIFY_BUNDLE_FSCK` option is now used by bundle-involved fetches,\nwhere we use `fetch-pack.c:fetch_pack_fsck_objects` to determine whether\nto enable this option for `bundle.c:unbundle`, specifically in:\n\n- `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n- `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n\nThis addition ensures a consistent logic for object verification during\nfetches. Tests have been added to confirm functionality in the scenarios\nmentioned above.\n\nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 ++-\n bundle.c | 3 +++\n bundle.h | 1 +\n t/t5558-clone-bundle-uri.sh | 33 ++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 33 +++++++++++++++++++++++++++++++++\n transport.c | 3 ++-\n 6 files changed, 73 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 65666a11d9c..ed9b49fdbc1 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -9,6 +9,7 @@\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"fetch-pack.h\"\n #include \"remote.h\"\n \n static struct {\n@@ -373,7 +374,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET | (fetch_pack_fsck_objects() ? VERIFY_BUNDLE_FSCK : 0))))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 95367c2d0a0..f124a2a5626 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -625,6 +625,9 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n+\tif (flags & VERIFY_BUNDLE_FSCK)\n+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/bundle.h b/bundle.h\nindex 021adbdcbb3..5ccc9a061a4 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -33,6 +33,7 @@ int create_bundle(struct repository *r, const char *path,\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n+\tVERIFY_BUNDLE_FSCK = (1 << 2),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 2dcdd238a90..38a25d08d0a 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -34,7 +34,21 @@ test_expect_success 'create bundle' '\n \t\tsed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n \t\t\tbad-header.bundle &&\n \t\tconvert_bundle_to_pack \\\n-\t\t\t>bad-header.bundle\n+\t\t\t>bad-header.bundle &&\n+\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad-object.bundle bad &&\n+\t\tgit update-ref -d refs/heads/bad\n \t)\n '\n \n@@ -56,6 +70,23 @@ test_expect_success 'clone with bundle that has bad header' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+test_expect_success 'clone with bundle that has bad object' '\n+\t# Unbundle succeeds if no fsckObjects confugured.\n+\tgit clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-no-fsck &&\n+\tgit -C clone-bad-object-no-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\ttest_write_lines refs/bundles/bad >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Unbundle fails with fsckObjects set true, but clone can still proceed.\n+\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\tgit -C clone-bad-object-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\t! grep \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\ndiff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\nindex 0d1e92d9963..5182efc0b45 100755\n--- a/t/t5607-clone-bundle.sh\n+++ b/t/t5607-clone-bundle.sh\n@@ -138,6 +138,39 @@ test_expect_success 'fetch SHA-1 from bundle' '\n \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n '\n \n+test_expect_success 'clone bundle with different fsckObjects configurations' '\n+\ttest_create_repo bundle-fsck &&\n+\t(\n+\t\tcd bundle-fsck &&\n+\t\ttest_commit first &&\n+\t\tcat >data <<-EOF &&\n+\t\ttree $(git rev-parse HEAD^{tree})\n+\t\tparent $(git rev-parse HEAD)\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n+\t\tgit branch bad $(cat commit) &&\n+\t\tgit bundle create bad.bundle bad\n+\t) &&\n+\n+\tgit clone bundle-fsck/bad.bundle bundle-no-fsck &&\n+\n+\tgit -c fetch.fsckObjects=false -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-no-fsck &&\n+\n+\ttest_must_fail git -c fetch.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-fetch-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\n+\ttest_must_fail git -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-transfer-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err\n+'\n+\n test_expect_success 'git bundle uses expected default format' '\n \tgit bundle create bundle HEAD^.. &&\n \tcat >expect <<-EOF &&\ndiff --git a/transport.c b/transport.c\nindex 0ad04b77fd2..a93c4171f7b 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,8 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, 0);\n+\t\t &extra_index_pack_args,\n+\t\t fetch_pack_fsck_objects() ? VERIFY_BUNDLE_FSCK : 0);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n"},{"id":"497295","messageId":"xmqqr0cuyww9.fsf@gitster.g","threadId":"61479","inReplyTo":"fc9f44fda0032ab1e5ee0c9bcc2886ddb8e17722.1718632536.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 1/3] bundle-uri: verify oid before writing refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-06-18T17:37:58Z","receivedAt":"2024-06-18T17:38:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Xing Xin via GitGitGadget\" writes:\n\n> @@ -33,6 +46,16 @@ test_expect_success 'clone with path bundle' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'clone with bundle that has bad header' '\n> +\t# Write bundle ref fails, but clone can still proceed.\n> +\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n> +\t\tclone-from clone-bad-header 2>err &&\n> +\tcommit_b=$(git -C clone-from rev-parse B) &&\n> +\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n> +\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n> +\t! grep \"refs/bundles/\" refs\n\nWhy not \"test_grep !\" here? There are other uses of bare grep in\nthe newly added lines, but I won't repeat them here; the same\ncomment applies to them.\n\n> +\ttest_write_lines refs/bundles/topic >expect &&\n> +\ttest_cmp expect actual &&\n> +\t# Ensure that refs/bundles/topic are sent as \"have\".\n> +\ttest_grep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n> +'\n\nCan this rev-parse fail (the failure would be hidden from the test)?\nIf so,\n\n\t...\n\ttest_cmp expect actual &&\n\t# Ensure that refs/bundles/topic is sent as \"have\"\n\ttip=$(git -C clone-from rev-parse A) &&\n\ttest_grep \"clone> have $tip\" trace-packet.txt\n\nwould catch such a failure. You are doing so in the previous test\nin the hunk starting at 33/46 above with commit_b variable already.\n\nThere are other uses of git command in $(command substitution) whose\nexit status are ignored in the newly added lines, but I won't\nrepeat them here; the same comment applies to them.\n\nThanks.\n"},{"id":"497326","messageId":"pull.1730.v8.git.1718770053.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v7.git.1718632535.gitgitgadget@gmail.com","subject":"[PATCH v8 0/3] object checking related additions and fixes for bundles in fetches","fromName":"blanet via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-19T04:07:30Z","receivedAt":"2024-06-19T04:07:37Z","isPatch":true,"sender":{"key":"name:blanet","avatar":null},"body":"While attempting to fix a reference negotiation bug in bundle-uri, we\nidentified that the fetch process lacks some crucial object validation\nchecks when processing bundles. The primary issues are:\n\n 1. In the bundle-uri scenario, object IDs were not validated before writing\n bundle references. This was the root cause of the original negotiation\n bug in bundle-uri and could lead to potential repository corruption.\n 2. The existing \"fetch.fsckObjects\" and \"transfer.fsckObjects\"\n configurations were not applied when directly fetching bundles or\n fetching with bundle-uri enabled. In fact, there were no object\n validation supports for unbundle.\n\nThe first patch addresses the bundle-uri negotiation issue by removing the\nREF_SKIP_OID_VERIFICATION flag when writing bundle references.\n\nPatches 2 through 3 extend verify_bundle_flags for bundle.c:unbundle to add\nsupport for object validation (fsck) in fetch scenarios, mainly following\nthe suggestions from Junio and Patrick on the mailing list.\n\nXing Xin (3):\n bundle-uri: verify oid before writing refs\n fetch-pack: expose fsckObjects configuration logic\n unbundle: extend object verification for fetches\n\n bundle-uri.c | 6 +-\n bundle.c | 3 +\n bundle.h | 1 +\n fetch-pack.c | 17 ++--\n fetch-pack.h | 5 +\n t/t5558-clone-bundle-uri.sh | 187 +++++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 35 +++++++\n transport.c | 3 +-\n 8 files changed, 243 insertions(+), 14 deletions(-)\n\n\nbase-commit: b9cfe4845cb2562584837bc0101c0ab76490a239\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1730%2Fblanet%2Fxx%2Fbundle-uri-bug-using-bundle-list-v8\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1730/blanet/xx/bundle-uri-bug-using-bundle-list-v8\nPull-Request: https://github.com/gitgitgadget/git/pull/1730\n\nRange-diff vs v7:\n\n 1: fc9f44fda00 ! 1: d8fbde2dcd4 bundle-uri: verify oid before writing refs\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'fail to clone from non-bundle\n +\t\tgit bundle create B.bundle topic &&\n +\n +\t\t# Create a bundle with reference pointing to non-existent object.\n -+\t\tsed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n ++\t\tcommit_a=$(git rev-parse A) &&\n ++\t\tcommit_b=$(git rev-parse B) &&\n ++\t\tsed -e \"/^$/q\" -e \"s/$commit_a /$commit_b /\" \\\n +\t\t\tbad-header.bundle &&\n +\t\tconvert_bundle_to_pack \\\n +\t\t\t>bad-header.bundle\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with path bundle' '\n +\tcommit_b=$(git -C clone-from rev-parse B) &&\n +\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n +\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n -+\t! grep \"refs/bundles/\" refs\n ++\ttest_grep ! \"refs/bundles/\" refs\n +'\n +\n test_expect_success 'clone with path bundle and non-default hash' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n +\ttest_write_lines refs/bundles/topic >expect &&\n +\ttest_cmp expect actual &&\n +\t# Ensure that refs/bundles/topic are sent as \"have\".\n -+\ttest_grep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n ++\ttip=$(git -C clone-from rev-parse A) &&\n ++\ttest_grep \"clone> have $tip\" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle with all wanted commits' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n +\ttest_write_lines refs/bundles/topic >expect &&\n +\ttest_cmp expect actual &&\n +\t# We already have all needed commits so no \"want\" needed.\n -+\t! grep \"clone> want \" trace-packet.txt\n ++\ttest_grep ! \"clone> want \" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle list (no heuristic)' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n +\trefs/bundles/left\n +\tEOF\n +\ttest_cmp expect actual &&\n -+\ttest_grep \"clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n ++\ttip=$(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left) &&\n ++\ttest_grep \"clone> have $tip\" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle list (creationToken)' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n +\trefs/bundles/left\n +\tEOF\n +\ttest_cmp expect actual &&\n -+\ttest_grep \"clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)\" trace-packet.txt\n ++\ttip=$(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left) &&\n ++\ttest_grep \"clone> have $tip\" trace-packet.txt\n +'\n +\n +test_expect_success 'negotiation: bundle list with all wanted commits' '\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone bundle list (file, any m\n +\tEOF\n +\ttest_cmp expect actual &&\n +\t# We already have all needed commits so no \"want\" needed.\n -+\t! grep \"clone> want \" trace-packet.txt\n ++\ttest_grep ! \"clone> want \" trace-packet.txt\n +'\n +\n #########################################################################\n 2: 3dc0d9dd22f ! 2: 518584c8698 fetch-pack: expose fsckObjects configuration logic\n @@ Commit message\n \"fetch.fsckObjects\" to control checks for broken objects in received\n packs during fetches. However, these configurations were only\n acknowledged by `fetch-pack.c:get_pack` and did not take effect in\n - direct bundle fetches and fetches with _bundle-uri_ enabled.\n + direct bundle fetches or fetches with _bundle-uri_ enabled.\n \n This commit exposes the fetch-then-transfer configuration logic by\n adding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This\n new function is used to replace the assignment for `fsck_objects` in\n - `fetch-pack.c:get_pack`. In the next commit, it will also be used by\n - `bundle.c:unbundle` to better fit fetching scenarios.\n + `fetch-pack.c:get_pack`. In the next commit, this function will also be\n + used to extend fsck support for bundle-involved fetches.\n \n Helped-by: Junio C Hamano \n Helped-by: Patrick Steinhardt \n 3: 2f15099bbb9 ! 3: 698dd6e49b7 unbundle: extend object verification for fetches\n @@ bundle.h: int create_bundle(struct repository *r, const char *path,\n \n ## t/t5558-clone-bundle-uri.sh ##\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'create bundle' '\n - \t\tsed -e \"/^$/q\" -e \"s/$(git rev-parse A) /$(git rev-parse B) /\" \\\n + \t\tsed -e \"/^$/q\" -e \"s/$commit_a /$commit_b /\" \\\n \t\t\tbad-header.bundle &&\n \t\tconvert_bundle_to_pack \\\n -\t\t\t>bad-header.bundle\n +\t\t\t>bad-header.bundle &&\n +\n ++\t\ttree_b=$(git rev-parse B^{tree}) &&\n +\t\tcat >data <<-EOF &&\n -+\t\ttree $(git rev-parse HEAD^{tree})\n -+\t\tparent $(git rev-parse HEAD)\n ++\t\ttree $tree_b\n ++\t\tparent $commit_b\n +\t\tauthor A U Thor\n +\t\tcommitter A U Thor\n +\n +\t\tcommit: this is a commit with bad emails\n +\n +\t\tEOF\n -+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n -+\t\tgit branch bad $(cat commit) &&\n ++\t\tbad_commit=$(git hash-object --literally -t commit -w --stdin refs &&\n @@ t/t5558-clone-bundle-uri.sh: test_expect_success 'clone with bundle that has bad\n +\t\tclone-from clone-bad-object-fsck 2>err &&\n +\ttest_grep \"missingEmail\" err &&\n +\tgit -C clone-bad-object-fsck for-each-ref --format=\"%(refname)\" >refs &&\n -+\t! grep \"refs/bundles/\" refs\n ++\ttest_grep ! \"refs/bundles/\" refs\n +'\n +\n test_expect_success 'clone with path bundle and non-default hash' '\n @@ t/t5607-clone-bundle.sh: test_expect_success 'fetch SHA-1 from bundle' '\n +\ttest_create_repo bundle-fsck &&\n +\t(\n +\t\tcd bundle-fsck &&\n -+\t\ttest_commit first &&\n ++\t\ttest_commit A &&\n ++\t\tcommit_a=$(git rev-parse A) &&\n ++\t\ttree_a=$(git rev-parse A^{tree}) &&\n +\t\tcat >data <<-EOF &&\n -+\t\ttree $(git rev-parse HEAD^{tree})\n -+\t\tparent $(git rev-parse HEAD)\n ++\t\ttree $tree_a\n ++\t\tparent $commit_a\n +\t\tauthor A U Thor\n +\t\tcommitter A U Thor\n +\n +\t\tcommit: this is a commit with bad emails\n +\n +\t\tEOF\n -+\t\tgit hash-object --literally -t commit -w --stdin commit &&\n -+\t\tgit branch bad $(cat commit) &&\n ++\t\tbad_commit=$(git hash-object --literally -t commit -w --stdin \n\nCurrently, we can use \"transfer.fsckObjects\" and the more specific\n\"fetch.fsckObjects\" to control checks for broken objects in received\npacks during fetches. However, these configurations were only\nacknowledged by `fetch-pack.c:get_pack` and did not take effect in\ndirect bundle fetches or fetches with _bundle-uri_ enabled.\n\nThis commit exposes the fetch-then-transfer configuration logic by\nadding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This\nnew function is used to replace the assignment for `fsck_objects` in\n`fetch-pack.c:get_pack`. In the next commit, this function will also be\nused to extend fsck support for bundle-involved fetches.\n\nHelped-by: Junio C Hamano \nHelped-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n fetch-pack.c | 17 +++++++++++------\n fetch-pack.h | 5 +++++\n 2 files changed, 16 insertions(+), 6 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d2aef21add..3acff2baf09 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\tstrvec_push(&cmd.args, alternate_shallow_file);\n \t}\n \n-\tif (fetch_fsck_objects >= 0\n-\t ? fetch_fsck_objects\n-\t : transfer_fsck_objects >= 0\n-\t ? transfer_fsck_objects\n-\t : 0)\n-\t\tfsck_objects = 1;\n+\tfsck_objects = fetch_pack_fsck_objects();\n \n \tif (do_keep || args->from_promisor || index_pack_args || fsck_objects) {\n \t\tif (pack_lockfiles || fsck_objects)\n@@ -2046,6 +2041,16 @@ static const struct object_id *iterate_ref_map(void *cb_data)\n \treturn &ref->old_oid;\n }\n \n+int fetch_pack_fsck_objects(void)\n+{\n+\tfetch_pack_setup();\n+\tif (fetch_fsck_objects >= 0)\n+\t\treturn fetch_fsck_objects;\n+\tif (transfer_fsck_objects >= 0)\n+\t\treturn transfer_fsck_objects;\n+\treturn 0;\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[],\n \t\t const struct ref *ref,\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 6775d265175..b5c579cdae2 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -101,4 +101,9 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips,\n */\n int report_unmatched_refs(struct ref **sought, int nr_sought);\n \n+/*\n+ * Return true if checks for broken objects in received pack are required.\n+ */\n+int fetch_pack_fsck_objects(void);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"497328","messageId":"d8fbde2dcd49d2a2a37e35f456830c51709cd743.1718770053.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v8.git.1718770053.gitgitgadget@gmail.com","subject":"[PATCH v8 1/3] bundle-uri: verify oid before writing refs","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-19T04:07:31Z","receivedAt":"2024-06-19T04:07:40Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nWhen using the bundle-uri mechanism with a bundle list containing\nmultiple interrelated bundles, we encountered a bug where tips from\ndownloaded bundles were not discovered, thus resulting in rather slow\nclones. This was particularly problematic when employing the\n\"creationTokens\" heuristic.\n\nTo reproduce this issue, consider a repository with a single branch\n\"main\" pointing to commit \"A\". Firstly, create a base bundle with:\n\n git bundle create base.bundle main\n\nThen, add a new commit \"B\" on top of \"A\", and create an incremental\nbundle for \"main\":\n\n git bundle create incr.bundle A..main\n\nNow, generate a bundle list with the following content:\n\n [bundle]\n version = 1\n mode = all\n heuristic = creationToken\n\n [bundle \"base\"]\n uri = base.bundle\n creationToken = 1\n\n [bundle \"incr\"]\n uri = incr.bundle\n creationToken = 2\n\nA fresh clone with the bundle list above should result in a reference\n\"refs/bundles/main\" pointing to \"B\" in the new repository. However, git\nwould still download everything from the server, as if it had fetched\nnothing locally.\n\nSo why the \"refs/bundles/main\" is not discovered? After some digging I\nfound that:\n\n1. Bundles in bundle list are downloaded to local files via\n `bundle-uri.c:download_bundle_list` or via\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which\n is called by `bundle-uri.c:unbundle_all_bundles` or called within\n `bundle-uri.c:fetch_bundles_by_token` for the \"creationToken\"\n heuristic.\n3. To get all prerequisites of the bundle, the bundle header is read\n inside `bundle-uri.c:unbundle_from_file` to by calling\n `bundle.c:read_bundle_header`.\n4. Then it calls `bundle.c:unbundle`, which calls\n `bundle.c:verify_bundle` to ensure the repository contains all the\n prerequisites.\n5. `bundle.c:verify_bundle` calls `parse_object`, which eventually\n invokes `packfile.c:prepare_packed_git` or\n `packfile.c:reprepare_packed_git`, filling\n `raw_object_store->packed_git` and setting `packed_git_initialized`.\n6. If `bundle.c:unbundle` succeeds, it writes refs via\n `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here\n bundle refs which can target arbitrary objects are written to the\n repository.\n7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions\n `fetch-pack.c:mark_complete_and_common_ref` and\n `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to\n find local tips for negotiation. The `OBJECT_INFO_QUICK` flag\n prevents `packfile.c:reprepare_packed_git` from being called,\n resulting in failures to parse OIDs that reside only in the latest\n bundle.\n\nIn the example above, when unbunding \"incr.bundle\", \"base.pack\" is added\nto `packed_git` due to prerequisites verification. However, \"B\" cannot\nbe found for negotiation because it exists in \"incr.pack\", which is not\nincluded in `packed_git`.\n\nFix the bug by removing `REF_SKIP_OID_VERIFICATION` flag when writing\nbundle refs. When `refs.c:refs_update_ref` is called to write the\ncorresponding bundle refs, it triggers `refs.c:ref_transaction_commit`.\nThis, in turn, invokes `refs.c:ref_transaction_prepare`, which calls\n`transaction_prepare` of the refs storage backend. For files backend, it\nis `files-backend.c:files_transaction_prepare`, and for reftable\nbackend, it is `reftable-backend.c:reftable_be_transaction_prepare`.\nBoth functions eventually call `object.c:parse_object`, which can invoke\n`packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures\nthat bundle refs point to valid objects and that all tips from bundle\nrefs are correctly parsed during subsequent negotiations.\n\nA set of negotiation-related tests for cloning with bundle-uri has been\nincluded to demonstrate that downloaded bundles are utilized to\naccelerate fetching.\n\nAdditionally, another test has been added to show that bundles with\nincorrect headers, where refs point to non-existent objects, do not\nresult in any bundle refs being created in the repository.\n\nReviewed-by: Karthik Nayak \nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 +-\n t/t5558-clone-bundle-uri.sh | 155 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 152 insertions(+), 6 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 91b3319a5c1..65666a11d9c 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t\trefs_update_ref(get_main_ref_store(the_repository),\n \t\t\t\t\"fetched bundle\", bundle_ref.buf, oid,\n \t\t\t\thas_old ? &old_oid : NULL,\n-\t\t\t\tREF_SKIP_OID_VERIFICATION,\n-\t\t\t\tUPDATE_REFS_MSG_ON_ERR);\n+\t\t\t\t0, UPDATE_REFS_MSG_ON_ERR);\n \t}\n \n \tbundle_header_release(&header);\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex 1ca5f745e73..a0895913fe9 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -3,6 +3,7 @@\n test_description='test fetching bundles with --bundle-uri'\n \n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-bundle.sh\n \n test_expect_success 'fail to clone from non-existent file' '\n \ttest_when_finished rm -rf test &&\n@@ -19,10 +20,24 @@ test_expect_success 'fail to clone from non-bundle file' '\n \n test_expect_success 'create bundle' '\n \tgit init clone-from &&\n-\tgit -C clone-from checkout -b topic &&\n-\ttest_commit -C clone-from A &&\n-\ttest_commit -C clone-from B &&\n-\tgit -C clone-from bundle create B.bundle topic\n+\t(\n+\t\tcd clone-from &&\n+\t\tgit checkout -b topic &&\n+\n+\t\ttest_commit A &&\n+\t\tgit bundle create A.bundle topic &&\n+\n+\t\ttest_commit B &&\n+\t\tgit bundle create B.bundle topic &&\n+\n+\t\t# Create a bundle with reference pointing to non-existent object.\n+\t\tcommit_a=$(git rev-parse A) &&\n+\t\tcommit_b=$(git rev-parse B) &&\n+\t\tsed -e \"/^$/q\" -e \"s/$commit_a /$commit_b /\" \\\n+\t\t\tbad-header.bundle &&\n+\t\tconvert_bundle_to_pack \\\n+\t\t\t>bad-header.bundle\n+\t)\n '\n \n test_expect_success 'clone with path bundle' '\n@@ -33,6 +48,16 @@ test_expect_success 'clone with path bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'clone with bundle that has bad header' '\n+\t# Write bundle ref fails, but clone can still proceed.\n+\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n+\t\tclone-from clone-bad-header 2>err &&\n+\tcommit_b=$(git -C clone-from rev-parse B) &&\n+\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n+\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n+\ttest_grep ! \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\n@@ -259,6 +284,128 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '\n \t! grep \"refs/bundles/\" refs\n '\n \n+test_expect_success 'negotiation: bundle with part of wanted commits' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"clone-from/A.bundle\" \\\n+\t\tclone-from nego-bundle-part &&\n+\tgit -C nego-bundle-part for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\ttest_write_lines refs/bundles/topic >expect &&\n+\ttest_cmp expect actual &&\n+\t# Ensure that refs/bundles/topic are sent as \"have\".\n+\ttip=$(git -C clone-from rev-parse A) &&\n+\ttest_grep \"clone> have $tip\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle with all wanted commits' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=topic --no-tags \\\n+\t\t--bundle-uri=\"clone-from/B.bundle\" \\\n+\t\tclone-from nego-bundle-all &&\n+\tgit -C nego-bundle-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\ttest_write_lines refs/bundles/topic >expect &&\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\ttest_grep ! \"clone> want \" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (no heuristic)' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-no-heuristic &&\n+\n+\tgit -C nego-bundle-list-no-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\ttip=$(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left) &&\n+\ttest_grep \"clone> have $tip\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list (creationToken)' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-heuristic &&\n+\n+\tgit -C nego-bundle-list-heuristic for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\ttip=$(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left) &&\n+\ttest_grep \"clone> have $tip\" trace-packet.txt\n+'\n+\n+test_expect_success 'negotiation: bundle list with all wanted commits' '\n+\ttest_when_finished \"rm -f trace*.txt\" &&\n+\tcat >bundle-list <<-EOF &&\n+\t[bundle]\n+\t\tversion = 1\n+\t\tmode = all\n+\t\theuristic = creationToken\n+\n+\t[bundle \"bundle-1\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-1.bundle\n+\t\tcreationToken = 1\n+\n+\t[bundle \"bundle-2\"]\n+\t\turi = file://$(pwd)/clone-from/bundle-2.bundle\n+\t\tcreationToken = 2\n+\tEOF\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace-packet.txt\" \\\n+\tgit clone --no-local --single-branch --branch=left --no-tags \\\n+\t\t--bundle-uri=\"file://$(pwd)/bundle-list\" \\\n+\t\tclone-from nego-bundle-list-all &&\n+\n+\tgit -C nego-bundle-list-all for-each-ref --format=\"%(refname)\" >refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\tcat >expect <<-\\EOF &&\n+\trefs/bundles/base\n+\trefs/bundles/left\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t# We already have all needed commits so no \"want\" needed.\n+\ttest_grep ! \"clone> want \" trace-packet.txt\n+'\n+\n #########################################################################\n # HTTP tests begin here\n \n-- \ngitgitgadget\n\n"},{"id":"497329","messageId":"698dd6e49b78d1f14c9eda1a1161fe3eff10028e.1718770053.git.gitgitgadget@gmail.com","threadId":"61479","inReplyTo":"pull.1730.v8.git.1718770053.gitgitgadget@gmail.com","subject":"[PATCH v8 3/3] unbundle: extend object verification for fetches","fromName":"Xing Xin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-06-19T04:07:33Z","receivedAt":"2024-06-19T04:07:41Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"From: Xing Xin \n\nThe existing fetch.fsckObjects and transfer.fsckObjects configurations\nwere not fully applied to bundle-involved fetches, including direct\nbundle fetches and bundle-uri enabled fetches. Furthermore, there was no\nobject verification support for unbundle.\n\nThis commit extends object verification support in `bundle.c:unbundle`\nby adding the `VERIFY_BUNDLE_FSCK` option to `verify_bundle_flags`. When\nthis option is enabled, we append the `--fsck-objects` flag to\n`git-index-pack`.\n\nThe `VERIFY_BUNDLE_FSCK` option is now used by bundle-involved fetches,\nwhere we use `fetch-pack.c:fetch_pack_fsck_objects` to determine whether\nto enable this option for `bundle.c:unbundle`, specifically in:\n\n- `transport.c:fetch_refs_from_bundle` for direct bundle fetches.\n- `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches.\n\nThis addition ensures a consistent logic for object verification during\nfetches. Tests have been added to confirm functionality in the scenarios\nmentioned above.\n\nReviewed-by: Patrick Steinhardt \nSigned-off-by: Xing Xin \n---\n bundle-uri.c | 3 ++-\n bundle.c | 3 +++\n bundle.h | 1 +\n t/t5558-clone-bundle-uri.sh | 34 +++++++++++++++++++++++++++++++++-\n t/t5607-clone-bundle.sh | 35 +++++++++++++++++++++++++++++++++++\n transport.c | 3 ++-\n 6 files changed, 76 insertions(+), 3 deletions(-)\n\ndiff --git a/bundle-uri.c b/bundle-uri.c\nindex 65666a11d9c..ed9b49fdbc1 100644\n--- a/bundle-uri.c\n+++ b/bundle-uri.c\n@@ -9,6 +9,7 @@\n #include \"hashmap.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"fetch-pack.h\"\n #include \"remote.h\"\n \n static struct {\n@@ -373,7 +374,7 @@ static int unbundle_from_file(struct repository *r, const char *file)\n \t * the prerequisite commits.\n \t */\n \tif ((result = unbundle(r, &header, bundle_fd, NULL,\n-\t\t\t VERIFY_BUNDLE_QUIET)))\n+\t\t\t VERIFY_BUNDLE_QUIET | (fetch_pack_fsck_objects() ? VERIFY_BUNDLE_FSCK : 0))))\n \t\treturn 1;\n \n \t/*\ndiff --git a/bundle.c b/bundle.c\nindex 95367c2d0a0..f124a2a5626 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -625,6 +625,9 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tif (header->filter.choice)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n+\tif (flags & VERIFY_BUNDLE_FSCK)\n+\t\tstrvec_push(&ip.args, \"--fsck-objects\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/bundle.h b/bundle.h\nindex 021adbdcbb3..5ccc9a061a4 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -33,6 +33,7 @@ int create_bundle(struct repository *r, const char *path,\n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n \tVERIFY_BUNDLE_QUIET = (1 << 1),\n+\tVERIFY_BUNDLE_FSCK = (1 << 2),\n };\n \n int verify_bundle(struct repository *r, struct bundle_header *header,\ndiff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh\nindex a0895913fe9..cd05321e176 100755\n--- a/t/t5558-clone-bundle-uri.sh\n+++ b/t/t5558-clone-bundle-uri.sh\n@@ -36,7 +36,22 @@ test_expect_success 'create bundle' '\n \t\tsed -e \"/^$/q\" -e \"s/$commit_a /$commit_b /\" \\\n \t\t\tbad-header.bundle &&\n \t\tconvert_bundle_to_pack \\\n-\t\t\t>bad-header.bundle\n+\t\t\t>bad-header.bundle &&\n+\n+\t\ttree_b=$(git rev-parse B^{tree}) &&\n+\t\tcat >data <<-EOF &&\n+\t\ttree $tree_b\n+\t\tparent $commit_b\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tbad_commit=$(git hash-object --literally -t commit -w --stdin refs &&\n+\tgrep \"refs/bundles/\" refs >actual &&\n+\ttest_write_lines refs/bundles/bad >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Unbundle fails with fsckObjects set true, but clone can still proceed.\n+\tgit -c fetch.fsckObjects=true clone --bundle-uri=\"clone-from/bad-object.bundle\" \\\n+\t\tclone-from clone-bad-object-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err &&\n+\tgit -C clone-bad-object-fsck for-each-ref --format=\"%(refname)\" >refs &&\n+\ttest_grep ! \"refs/bundles/\" refs\n+'\n+\n test_expect_success 'clone with path bundle and non-default hash' '\n \ttest_when_finished \"rm -rf clone-path-non-default-hash\" &&\n \tGIT_DEFAULT_HASH=sha256 git clone --bundle-uri=\"clone-from/B.bundle\" \\\ndiff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\nindex 0d1e92d9963..489c6570da5 100755\n--- a/t/t5607-clone-bundle.sh\n+++ b/t/t5607-clone-bundle.sh\n@@ -138,6 +138,41 @@ test_expect_success 'fetch SHA-1 from bundle' '\n \tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n '\n \n+test_expect_success 'clone bundle with different fsckObjects configurations' '\n+\ttest_create_repo bundle-fsck &&\n+\t(\n+\t\tcd bundle-fsck &&\n+\t\ttest_commit A &&\n+\t\tcommit_a=$(git rev-parse A) &&\n+\t\ttree_a=$(git rev-parse A^{tree}) &&\n+\t\tcat >data <<-EOF &&\n+\t\ttree $tree_a\n+\t\tparent $commit_a\n+\t\tauthor A U Thor\n+\t\tcommitter A U Thor\n+\n+\t\tcommit: this is a commit with bad emails\n+\n+\t\tEOF\n+\t\tbad_commit=$(git hash-object --literally -t commit -w --stdin err &&\n+\ttest_grep \"missingEmail\" err &&\n+\n+\ttest_must_fail git -c transfer.fsckObjects=true \\\n+\t\tclone bundle-fsck/bad.bundle bundle-transfer-fsck 2>err &&\n+\ttest_grep \"missingEmail\" err\n+'\n+\n test_expect_success 'git bundle uses expected default format' '\n \tgit bundle create bundle HEAD^.. &&\n \tcat >expect <<-EOF &&\ndiff --git a/transport.c b/transport.c\nindex 0ad04b77fd2..a93c4171f7b 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -184,7 +184,8 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \tif (!data->get_refs_from_bundle_called)\n \t\tget_refs_from_bundle_inner(transport);\n \tret = unbundle(the_repository, &data->header, data->fd,\n-\t\t &extra_index_pack_args, 0);\n+\t\t &extra_index_pack_args,\n+\t\t fetch_pack_fsck_objects() ? VERIFY_BUNDLE_FSCK : 0);\n \ttransport->hash_algo = data->header.hash_algo;\n \treturn ret;\n }\n-- \ngitgitgadget\n"},{"id":"497332","messageId":"ae431a7.611a.1902f30a243.Coremail.bupt_xingxin@163.com","threadId":"61479","inReplyTo":"xmqqr0cuyww9.fsf@gitster.g","subject":"Re:Re: [PATCH v7 1/3] bundle-uri: verify oid before writing refs","fromName":"Xing Xin","fromEmail":"bupt_xingxin@163.com","sentAt":"2024-06-19T06:30:34Z","receivedAt":"2024-06-19T06:31:16Z","isPatch":true,"sender":{"key":"bupt_xingxin@163.com","avatar":null},"body":"At 2024-06-19 01:37:58, \"Junio C Hamano\" wrote:\n>\"Xing Xin via GitGitGadget\" writes:\n>\n>> @@ -33,6 +46,16 @@ test_expect_success 'clone with path bundle' '\n>> \ttest_cmp expect actual\n>> '\n>> \n>> +test_expect_success 'clone with bundle that has bad header' '\n>> +\t# Write bundle ref fails, but clone can still proceed.\n>> +\tgit clone --bundle-uri=\"clone-from/bad-header.bundle\" \\\n>> +\t\tclone-from clone-bad-header 2>err &&\n>> +\tcommit_b=$(git -C clone-from rev-parse B) &&\n>> +\ttest_grep \"trying to write ref '\\''refs/bundles/topic'\\'' with nonexistent object $commit_b\" err &&\n>> +\tgit -C clone-bad-header for-each-ref --format=\"%(refname)\" >refs &&\n>> +\t! grep \"refs/bundles/\" refs\n>\n>Why not \"test_grep !\" here? There are other uses of bare grep in\n>the newly added lines, but I won't repeat them here; the same\n>comment applies to them.\n\nBoth `test_grep !` and `! grep` are widely used in tests. Sorry for not\nrealizing that the former is preferred.\n\n>> +\ttest_write_lines refs/bundles/topic >expect &&\n>> +\ttest_cmp expect actual &&\n>> +\t# Ensure that refs/bundles/topic are sent as \"have\".\n>> +\ttest_grep \"clone> have $(git -C clone-from rev-parse A)\" trace-packet.txt\n>> +'\n>\n>Can this rev-parse fail (the failure would be hidden from the test)?\n>If so,\n>\n>\t...\n>\ttest_cmp expect actual &&\n>\t# Ensure that refs/bundles/topic is sent as \"have\"\n>\ttip=$(git -C clone-from rev-parse A) &&\n>\ttest_grep \"clone> have $tip\" trace-packet.txt\n>\n>would catch such a failure. You are doing so in the previous test\n>in the hunk starting at 33/46 above with commit_b variable already.\n>\n>There are other uses of git command in $(command substitution) whose\n>exit status are ignored in the newly added lines, but I won't\n>repeat them here; the same comment applies to them.\n\nFixed in the new series. Thanks.\n\nXing Xin\n"}]}