{"thread":{"id":"54074","subject":"[PATCH] fetch-pack: in partial clone, pass --promisor","startedAt":"2020-08-20T17:51:32Z","lastAt":"2020-08-21T21:42:26Z","messageCount":4,"participants":["Jonathan Tan","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"404106","messageId":"20200820175116.3889786-1-jonathantanmy@google.com","threadId":"54074","inReplyTo":null,"subject":"[PATCH] fetch-pack: in partial clone, pass --promisor","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-08-20T17:51:16Z","receivedAt":"2020-08-20T17:51:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When fetching a pack from a promisor remote, the corresponding .promisor\nfile needs to be created. \"fetch-pack\" originally did this by passing\n\"--promisor\" to \"index-pack\", but in 5374a290aa (\"fetch-pack: write\nfetched refs to .promisor\", 2019-10-16), \"fetch-pack\" was taught to do\nthis itself instead, because it needed to store ref information in the\n.promisor file.\n\nThis causes a problem with superprojects when transfer.fsckobjects is\nset, because in the current implementation, it is \"index-pack\" that\ncalls fsck_finish() to check the objects; before 5374a290aa,\nfsck_finish() would see that .gitmodules is a promisor object and\ntolerate it being missing, but after, there is no .promisor file (at the\ntime of the invocation of fsck_finish() by \"index-pack\") to tell it that\n.gitmodules is a promisor object, so it returns an error.\n\nTherefore, teach \"fetch-pack\" to pass \"--promisor\" to index pack once\nagain. \"fetch-pack\" will subsequently overwrite this file with the ref\ninformation.\n\nAn alternative is to instead move object checking to \"fetch-pack\", and\nlet \"index-pack\" only index the files. However, since \"index-pack\" has\nto inflate objects in order to index them, it seems reasonable to also\nlet it check the objects (which also require inflated files).\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n fetch-pack.c             | 17 ++++++++++-------\n t/t5616-partial-clone.sh | 16 ++++++++++++++++\n 2 files changed, 26 insertions(+), 7 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7f20eca4f8..d467edc24e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -866,13 +866,16 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\t * have this responsibility.\n \t\t\t */\n \t\t\targs->check_self_contained_and_connected = 0;\n-\t\t/*\n-\t\t * If we're obtaining the filename of a lockfile, we'll use\n-\t\t * that filename to write a .promisor file with more\n-\t\t * information below. If not, we need index-pack to do it for\n-\t\t * us.\n-\t\t */\n-\t\tif (!(do_keep && pack_lockfiles) && args->from_promisor)\n+\n+\t\tif (args->from_promisor)\n+\t\t\t/*\n+\t\t\t * write_promisor_file() may be called afterwards but\n+\t\t\t * we still need index-pack to know that this is a\n+\t\t\t * promisor pack. For example, if transfer.fsckobjects\n+\t\t\t * is true, index-pack needs to know that .gitmodules\n+\t\t\t * is a promisor object (so that it won't complain if\n+\t\t\t * it is missing).\n+\t\t\t */\n \t\t\tstrvec_push(&cmd.args, \"--promisor\");\n \t}\n \telse {\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 8827c2ed18..5a01466db4 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -163,6 +163,22 @@ test_expect_success 'manual prefetch of missing objects' '\n \ttest_line_count = 0 observed.oids\n '\n \n+test_expect_success 'partial clone with transfer.fsckobjects=1 works with submodules' '\n+\ttest_create_repo submodule &&\n+\ttest_commit -C submodule mycommit &&\n+\n+\ttest_create_repo src_with_sub &&\n+\ttest_config -C src_with_sub uploadpack.allowfilter 1 &&\n+\ttest_config -C src_with_sub uploadpack.allowanysha1inwant 1 &&\n+\n+\tgit -C src_with_sub submodule add \"file://$(pwd)/submodule\" mysub &&\n+\tgit -C src_with_sub commit -m \"commit with submodule\" &&\n+\n+\tgit -c transfer.fsckobjects=1 \\\n+\t\tclone --filter=\"blob:none\" \"file://$(pwd)/src_with_sub\" dst &&\n+\ttest_when_finished rm -rf dst\n+'\n+\n test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack --fsck-objects' '\n \tgit init src &&\n \ttest_commit -C src x &&\n-- \n2.28.0.297.g1956fa8f8d-goog\n\n"},{"id":"404112","messageId":"xmqqimddaxhz.fsf@gitster.c.googlers.com","threadId":"54074","inReplyTo":"20200820175116.3889786-1-jonathantanmy@google.com","subject":"Re: [PATCH] fetch-pack: in partial clone, pass --promisor","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-20T20:11:20Z","receivedAt":"2020-08-20T20:11:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> When fetching a pack from a promisor remote, the corresponding .promisor\n> file needs to be created. \"fetch-pack\" originally did this by passing\n> \"--promisor\" to \"index-pack\", but in 5374a290aa (\"fetch-pack: write\n> fetched refs to .promisor\", 2019-10-16), \"fetch-pack\" was taught to do\n> this itself instead, because it needed to store ref information in the\n> .promisor file.\n\nSo is this patch a fix for a regression in v2.25?\n\n> This causes a problem ...\n> ...\n> An alternative is to instead move object checking to \"fetch-pack\", and\n> let \"index-pack\" only index the files. However, since \"index-pack\" has\n> to inflate objects in order to index them, it seems reasonable to also\n> let it check the objects (which also require inflated files).\n\nI can see why it might feel attractive to draw the line to divide\nthe labor between the two that way, but I 100% agree with your\nreasoning---index-pack needs to inspect the objects in order for it\nto be able to index, and in order for us to be able to trust the\nresulting index, it should be validating the objects it identified\nwhile it was indexing the pack stream.\n\n> +\n> +\t\tif (args->from_promisor)\n> +\t\t\t/*\n> +\t\t\t * write_promisor_file() may be called afterwards but\n> +\t\t\t * we still need index-pack to know that this is a\n> +\t\t\t * promisor pack. For example, if transfer.fsckobjects\n> +\t\t\t * is true, index-pack needs to know that .gitmodules\n> +\t\t\t * is a promisor object (so that it won't complain if\n> +\t\t\t * it is missing).\n> +\t\t\t */\n>  \t\t\tstrvec_push(&cmd.args, \"--promisor\");\n>  \t}\n>  \telse {\n> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> index 8827c2ed18..5a01466db4 100755\n> --- a/t/t5616-partial-clone.sh\n> +++ b/t/t5616-partial-clone.sh\n> @@ -163,6 +163,22 @@ test_expect_success 'manual prefetch of missing objects' '\n>  \ttest_line_count = 0 observed.oids\n>  '\n>  \n> +test_expect_success 'partial clone with transfer.fsckobjects=1 works with submodules' '\n> +\ttest_create_repo submodule &&\n> +\ttest_commit -C submodule mycommit &&\n> +\n> +\ttest_create_repo src_with_sub &&\n> +\ttest_config -C src_with_sub uploadpack.allowfilter 1 &&\n> +\ttest_config -C src_with_sub uploadpack.allowanysha1inwant 1 &&\n> +\n> +\tgit -C src_with_sub submodule add \"file://$(pwd)/submodule\" mysub &&\n> +\tgit -C src_with_sub commit -m \"commit with submodule\" &&\n> +\n> +\tgit -c transfer.fsckobjects=1 \\\n> +\t\tclone --filter=\"blob:none\" \"file://$(pwd)/src_with_sub\" dst &&\n> +\ttest_when_finished rm -rf dst\n> +'\n> +\n>  test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack --fsck-objects' '\n>  \tgit init src &&\n>  \ttest_commit -C src x &&\n"},{"id":"404214","messageId":"20200821210827.1203531-1-jonathantanmy@google.com","threadId":"54074","inReplyTo":"xmqqimddaxhz.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] fetch-pack: in partial clone, pass --promisor","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-08-21T21:08:27Z","receivedAt":"2020-08-21T21:08:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n> > When fetching a pack from a promisor remote, the corresponding .promisor\n> > file needs to be created. \"fetch-pack\" originally did this by passing\n> > \"--promisor\" to \"index-pack\", but in 5374a290aa (\"fetch-pack: write\n> > fetched refs to .promisor\", 2019-10-16), \"fetch-pack\" was taught to do\n> > this itself instead, because it needed to store ref information in the\n> > .promisor file.\n> \n> So is this patch a fix for a regression in v2.25?\n\nYes. (Just checked with \"git merge-base\" that 5374a290aa is in v2.25.0\nbut not v2.24.0.)\n"},{"id":"404232","messageId":"xmqqo8n37k1z.fsf@gitster.c.googlers.com","threadId":"54074","inReplyTo":"20200821210827.1203531-1-jonathantanmy@google.com","subject":"Re: [PATCH] fetch-pack: in partial clone, pass --promisor","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-21T21:42:16Z","receivedAt":"2020-08-21T21:42:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n>> Jonathan Tan <jonathantanmy@google.com> writes:\n>> \n>> > When fetching a pack from a promisor remote, the corresponding .promisor\n>> > file needs to be created. \"fetch-pack\" originally did this by passing\n>> > \"--promisor\" to \"index-pack\", but in 5374a290aa (\"fetch-pack: write\n>> > fetched refs to .promisor\", 2019-10-16), \"fetch-pack\" was taught to do\n>> > this itself instead, because it needed to store ref information in the\n>> > .promisor file.\n>> \n>> So is this patch a fix for a regression in v2.25?\n>\n> Yes. (Just checked with \"git merge-base\" that 5374a290aa is in v2.25.0\n> but not v2.24.0.)\n\nThanks.  \n\nI wonder how we missed the breakage back then, but better late than\nnever ;-)\n"}]}