{"thread":{"id":"55638","subject":"[PATCH] Packfile-uris support excluding commit objects","startedAt":"2021-05-07T02:11:51Z","lastAt":"2021-10-19T11:40:14Z","messageCount":72,"participants":["Teng Long","Ævar Arnfjörð Bjarmason","Junio C Hamano","Felipe Contreras","Bagas Sanjaya","ZheNing Hu"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"423835","messageId":"20210507021140.31372-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":null,"subject":"[PATCH] Packfile-uris support excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-05-07T02:11:40Z","receivedAt":"2021-05-07T02:11:51Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"On the server, more sophisticated means of excluding objects should be\nsupported, such as commit object. This commit introduces a new\nconfiguration `uploadpack.commitpackfileuri` for this.\n\nThis patch only pack the commit object, not including the that commit\nand all objects that it references. This work will be done in a further\npatch recently.\n\nSimilarly, there are related documents that will be included in\nsubsequent patches.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 8 ++---\n fetch-pack.c | 8 +++++\n t/t5702-protocol-v2.sh | 71 +++++++++++++++++++++++++++++++++---------\n upload-pack.c | 7 +++--\n 4 files changed, 73 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6d13cd3e1a..2f1817fe28 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2985,7 +2985,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n+ if (!strcmp(k, \"uploadpack.blobpackfileuri\") || !strcmp(k, \"uploadpack.commitpackfileuri\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n \t\tconst char *oid_end, *pack_end;\n \t\t/*\n@@ -2998,11 +2998,11 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t *oid_end != ' ' ||\n \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n \t\t *pack_end != ' ')\n-\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n-\t\t\t \"of the form ' ' (got '%s')\"), v);\n+ die(_(\"value of uploadpack.blobpackfileuri or upload.commitpackfileuri must be \"\n+ \"of the form ' ' (got '%s')\"), v);\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n \t\t\tdie(_(\"object already configured in another \"\n-\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n+\t\t\t \"uploadpack.blobpackfileuri or uploadpack.commitpackfileuri (got '%s')\"), v);\n \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n \t\tex->uri = xstrdup(pack_end + 1);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 2318ebe680..24a947835b 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -23,6 +23,7 @@\n #include \"fetch-negotiator.h\"\n #include \"fsck.h\"\n #include \"shallow.h\"\n+#include \"strmap.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -1677,6 +1678,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n+\tstruct strset uris;\n+\tstrset_init(&uris);\n \tfor (i = 0; i < packfile_uris.nr; i++) {\n \t\tint j;\n \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -1684,6 +1687,11 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\tconst char *uri = packfile_uris.items[i].string +\n \t\t\tthe_hash_algo->hexsz + 1;\n \n+ if (strset_contains(&uris, uri)) {\n+ continue;\n+ }\n+\n+ strset_add(&uris, uri);\n \t\tstrvec_push(&cmd.args, \"http-fetch\");\n \t\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n \t\t\t (int) the_hash_algo->hexsz,\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 2e1243ca40..d444177fb5 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -824,12 +824,22 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n '\n \n configure_exclusion () {\n-\tgit -C \"$1\" hash-object \"$2\" >objh &&\n-\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n-\tgit -C \"$1\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n-\tcat objh\n+ if test \"$1\" = \"blob\"\n+ then\n+ git -C \"$2\" hash-object \"$3\" >objh &&\n+ git -C \"$2\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n+ git -C \"$2\" config --add \\\n+ \t\t\"uploadpack.blobpackfileuri\" \\\n+ \t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+ cat objh\n+ else\n+ echo \"$3\" > objh &&\n+ git -C \"$2\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n+ git -C \"$2\" config --add \\\n+ \t\t\"uploadpack.commitpackfileuri\" \\\n+ \t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+ cat objh\n+ fi\n }\n \n test_expect_success 'part of packfile response provided as URI' '\n@@ -845,8 +855,8 @@ test_expect_success 'part of packfile response provided as URI' '\n \tgit -C \"$P\" add other-blob &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\tconfigure_exclusion \"$P\" other-blob >h2 &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" other-blob >h2 &&\n \n \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n \tgit -c protocol.version=2 \\\n@@ -881,7 +891,7 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest_line_count = 6 filelist\n '\n \n-test_expect_success 'packfile URIs with fetch instead of clone' '\n+test_expect_success 'blobs packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \trm -rf \"$P\" http_child log &&\n \n@@ -892,7 +902,7 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \tgit -C \"$P\" add my-blob &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tgit init http_child &&\n \n@@ -902,6 +912,37 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \t\tfetch \"$HTTPD_URL/smart/http_parent\"\n '\n \n+test_expect_success 'commits packfile URIs with fetch instead of clone' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\trm -rf \"$P\" http_child log &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\tgit -C \"$P\" commit -m x &&\n+\n+\n+\tmycommit=$(git -C \"$P\" rev-parse HEAD) &&\n+ echo other-blob >\"$P/other-blob\" &&\n+ git -C \"$P\" add other-blob &&\n+ git -C \"$P\" commit -m x &&\n+\tothercommit=$(git -C \"$P\" rev-parse HEAD) &&\n+\tconfigure_exclusion commit \"$P\" \"$mycommit\" >h &&\n+\tconfigure_exclusion commit \"$P\" \"$othercommit\" >h2 &&\n+\n+\tgit init http_child &&\n+\n+\tGIT_TRACE=1 GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch \"$HTTPD_URL/smart/http_parent\" &&\n+\tls http_child/.git/objects/pack/*.pack \\\n+ \t http_child/.git/objects/pack/*.idx >filelist &&\n+ \ttest_line_count = 6 filelist\n+'\n+\n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \trm -rf \"$P\" http_child log &&\n@@ -915,7 +956,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" add other-blob &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \t# Configure a URL for other-blob. Just reuse the hash of the object as\n \t# the hash of the packfile, since the hash does not matter for this\n \t# test as long as it is not the hash of the pack, and it is of the\n@@ -944,7 +985,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \tgit -C \"$P\" add my-blob &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -978,7 +1019,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \tgit -C \"$P\" add my-blob &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1000,7 +1041,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \tgit -C \"$P\" add .gitmodules &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1026,7 +1067,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n \tgit -C \"$P\" add .gitmodules &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5c1cd19612..34f8bb81a8 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1744,9 +1744,12 @@ int upload_pack_advertise(struct repository *r,\n \t\t allow_sideband_all_value))\n \t\t\tstrbuf_addstr(value, \" sideband-all\");\n \n-\t\tif (!repo_config_get_string(the_repository,\n+\t\tif ((!repo_config_get_string(the_repository,\n \t\t\t\t\t \"uploadpack.blobpackfileuri\",\n-\t\t\t\t\t &str) &&\n+\t\t\t\t\t &str) ||\n+ !repo_config_get_string(the_repository,\n+ \"uploadpack.commitpackfileuri\",\n+ &str)) &&\n \t\t str) {\n \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n \t\t\tfree(str);\n-- \n2.31.1.442.g7e39198978.dirty\n\n"},{"id":"424021","messageId":"87wns6u8pc.fsf@evledraar.gmail.com","threadId":"55638","inReplyTo":"20210507021140.31372-1-dyroneteng@gmail.com","subject":"Re: [PATCH] Packfile-uris support excluding commit objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-10T11:14:40Z","receivedAt":"2021-05-10T11:50:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, May 07 2021, Teng Long wrote:\n\nIt seems like this and your\nhttp://lore.kernel.org/git/20210506073354.27833-1-dyroneteng@gmail.com\nshould be part of one series, not split up.\n\n> On the server, more sophisticated means of excluding objects should be\n> supported, such as commit object. This commit introduces a new\n> configuration `uploadpack.commitpackfileuri` for this.\n\nPer my understanding in\nhttps://lore.kernel.org/git/87o8hk820f.fsf@evledraar.gmail.com/ this +\nJonathan's earlier bfc2a36ff2a (Doc: clarify contents of packfile sent\nas URI, 2021-01-20) still makes this whole thing more confusing that it\nneeds to be.\n\nI think we should just have a new uploadpack.excludeObject, and document\nthat uploadpack.blobpackfileuri is an (unfortunately named) synonym for\nit. I.e. the actual implementation doesn't care about the objec type it\njust excludes any object listed via an oidmap. No?\n\nAs for some comments on the implementation:\n\n> This patch only pack the commit object, not including the that commit\n> and all objects that it references. This work will be done in a further\n> patch recently.\n\nI realize you're probably not a native English speaker (neither am I),\nbut I honestly can't understand that \"This work will be done in a\nfurther patch recently.\". Do you mean something like:\n\n This change does not add support for recursively excluding things\n referenced by container objects such as \"commit\", \"tag\", and\n \"tree\". We'll still just dumbly exclude that specific object (this\n was originally meant for specific \"blobs\"). Smartly excluding things\n recursively might be implemented by a future change.\n\n> Similarly, there are related documents that will be included in\n> subsequent patches.\n\nPlease send the earlier doc cleanup + the spec change for this + any doc\nupdates as one series.\n\nNarrow comments on the patch:\n\n> Signed-off-by: Teng Long \n> ---\n> builtin/pack-objects.c | 8 ++---\n> fetch-pack.c | 8 +++++\n> t/t5702-protocol-v2.sh | 71 +++++++++++++++++++++++++++++++++---------\n> upload-pack.c | 7 +++--\n> 4 files changed, 73 insertions(+), 21 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 6d13cd3e1a..2f1817fe28 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -2985,7 +2985,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n> \t\treturn 0;\n> \t}\n> -\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n> + if (!strcmp(k, \"uploadpack.blobpackfileuri\") || !strcmp(k, \"uploadpack.commitpackfileuri\")) {\n\n\nNit: Split this across two lines.\n\n> \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n> \t\tconst char *oid_end, *pack_end;\n> \t\t/*\n> @@ -2998,11 +2998,11 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t *oid_end != ' ' ||\n> \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n> \t\t *pack_end != ' ')\n> -\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n> -\t\t\t \"of the form ' ' (got '%s')\"), v);\n> + die(_(\"value of uploadpack.blobpackfileuri or upload.commitpackfileuri must be \"\n> + \"of the form ' ' (got '%s')\"), v);\n\nIndending with spaces.\n\n> \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n> \t\t\tdie(_(\"object already configured in another \"\n> -\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n> +\t\t\t \"uploadpack.blobpackfileuri or uploadpack.commitpackfileuri (got '%s')\"), v);\n\nI think by having a uploadpack.excludeObject documented as the primary\ninterface to this we could just say \"object already listed by an earlier\nexclusion\" or something like that.\n\n> \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n> \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n> \t\tex->uri = xstrdup(pack_end + 1);\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 2318ebe680..24a947835b 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -23,6 +23,7 @@\n> #include \"fetch-negotiator.h\"\n> #include \"fsck.h\"\n> #include \"shallow.h\"\n> +#include \"strmap.h\"\n> \n> static int transfer_unpack_limit = -1;\n> static int fetch_unpack_limit = -1;\n> @@ -1677,6 +1678,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t}\n> \t}\n> \n> +\tstruct strset uris;\n> +\tstrset_init(&uris);\n> \tfor (i = 0; i < packfile_uris.nr; i++) {\n> \t\tint j;\n> \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n> @@ -1684,6 +1687,11 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\tconst char *uri = packfile_uris.items[i].string +\n> \t\t\tthe_hash_algo->hexsz + 1;\n> \n> + if (strset_contains(&uris, uri)) {\n> + continue;\n> + }\n\n\nMore indenting with spaces, also don't need the {} here.\n\n> +\n> + strset_add(&uris, uri);\n> \t\tstrvec_push(&cmd.args, \"http-fetch\");\n> \t\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n> \t\t\t (int) the_hash_algo->hexsz,\n> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\n> index 2e1243ca40..d444177fb5 100755\n> --- a/t/t5702-protocol-v2.sh\n> +++ b/t/t5702-protocol-v2.sh\n> @@ -824,12 +824,22 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n> '\n> \n> configure_exclusion () {\n> -\tgit -C \"$1\" hash-object \"$2\" >objh &&\n> -\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n> -\tgit -C \"$1\" config --add \\\n> -\t\t\"uploadpack.blobpackfileuri\" \\\n> -\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n> -\tcat objh\n> + if test \"$1\" = \"blob\"\n> + then\n\nDon't indent the \"then\", also spaces...\n\n> + git -C \"$2\" hash-object \"$3\" >objh &&\n> + git -C \"$2\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n> + git -C \"$2\" config --add \\\n> + \t\t\"uploadpack.blobpackfileuri\" \\\n> + \t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n> + cat objh\n> + else\n> + echo \"$3\" > objh &&\n\nUse \">objh\" not \"> objh\".\n\n> + git -C \"$2\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n> + git -C \"$2\" config --add \\\n> + \t\t\"uploadpack.commitpackfileuri\" \\\n> + \t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n> + cat objh\n\n\nThis whole if/else seems like it could be better split up by discovering\nthe variable first, using that as a variable, and then avoiding the\nduplication. But if we just used uploadpack.excludeObject...\n\n> + fi\n> }\n> \n> test_expect_success 'part of packfile response provided as URI' '\n> @@ -845,8 +855,8 @@ test_expect_success 'part of packfile response provided as URI' '\n> \tgit -C \"$P\" add other-blob &&\n> \tgit -C \"$P\" commit -m x &&\n> \n> -\tconfigure_exclusion \"$P\" my-blob >h &&\n> -\tconfigure_exclusion \"$P\" other-blob >h2 &&\n> +\tconfigure_exclusion blob \"$P\" my-blob >h &&\n> +\tconfigure_exclusion blob \"$P\" other-blob >h2 &&\n> \n> \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n> \tgit -c protocol.version=2 \\\n> @@ -881,7 +891,7 @@ test_expect_success 'part of packfile response provided as URI' '\n> \ttest_line_count = 6 filelist\n> '\n> \n> -test_expect_success 'packfile URIs with fetch instead of clone' '\n> +test_expect_success 'blobs packfile URIs with fetch instead of clone' '\n> \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> \trm -rf \"$P\" http_child log &&\n> \n> @@ -892,7 +902,7 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n> \tgit -C \"$P\" add my-blob &&\n> \tgit -C \"$P\" commit -m x &&\n> \n> -\tconfigure_exclusion \"$P\" my-blob >h &&\n> +\tconfigure_exclusion blob \"$P\" my-blob >h &&\n> \n> \tgit init http_child &&\n> \n> @@ -902,6 +912,37 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n> \t\tfetch \"$HTTPD_URL/smart/http_parent\"\n> '\n> \n> +test_expect_success 'commits packfile URIs with fetch instead of clone' '\n> +\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> +\trm -rf \"$P\" http_child log &&\n\nPut stuff like this in \"test_when_finished\"\n\n> +\n> +\tgit init \"$P\" &&\n> +\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n> +\n> +\techo my-blob >\"$P/my-blob\" &&\n> +\tgit -C \"$P\" add my-blob &&\n> +\tgit -C \"$P\" commit -m x &&\n\nYou can just use test_commit here, no?\n\n> +\n> +\n> +\tmycommit=$(git -C \"$P\" rev-parse HEAD) &&\n> + echo other-blob >\"$P/other-blob\" &&\n> + git -C \"$P\" add other-blob &&\n> + git -C \"$P\" commit -m x &&\n\nditto test_commit.\n\n> +\tothercommit=$(git -C \"$P\" rev-parse HEAD) &&\n> +\tconfigure_exclusion commit \"$P\" \"$mycommit\" >h &&\n> +\tconfigure_exclusion commit \"$P\" \"$othercommit\" >h2 &&\n\nPersonally I'd just skip this whole \"rev-parse HEAD\" etc. and just pass\nthe tag name(s) created by earlier test_commit, then have\nconfigure_exclusion ust always do a rev-parse...\n\n> +\n> +\tgit init http_child &&\n> +\n> +\tGIT_TRACE=1 GIT_TEST_SIDEBAND_ALL=1 \\\n> +\tgit -C http_child -c protocol.version=2 \\\n> +\t\t-c fetch.uriprotocols=http,https \\\n> +\t\tfetch \"$HTTPD_URL/smart/http_parent\" &&\n> +\tls http_child/.git/objects/pack/*.pack \\\n> + \t http_child/.git/objects/pack/*.idx >filelist &&\n> + \ttest_line_count = 6 filelist\n> +'\n> +\n> test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n> \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> \trm -rf \"$P\" http_child log &&\n> @@ -915,7 +956,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n> \tgit -C \"$P\" add other-blob &&\n> \tgit -C \"$P\" commit -m x &&\n> \n> -\tconfigure_exclusion \"$P\" my-blob >h &&\n> +\tconfigure_exclusion blob \"$P\" my-blob >h &&\n> \t# Configure a URL for other-blob. Just reuse the hash of the object as\n> \t# the hash of the packfile, since the hash does not matter for this\n> \t# test as long as it is not the hash of the pack, and it is of the\n> @@ -944,7 +985,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n> \tgit -C \"$P\" add my-blob &&\n> \tgit -C \"$P\" commit -m x &&\n> \n> -\tconfigure_exclusion \"$P\" my-blob >h &&\n> +\tconfigure_exclusion blob \"$P\" my-blob >h &&\n> \n> \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n> \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n> @@ -978,7 +1019,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n> \tgit -C \"$P\" add my-blob &&\n> \tgit -C \"$P\" commit -m x &&\n> \n> -\tconfigure_exclusion \"$P\" my-blob >h &&\n> +\tconfigure_exclusion blob \"$P\" my-blob >h &&\n> \n> \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n> \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n> @@ -1000,7 +1041,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n> \tgit -C \"$P\" add .gitmodules &&\n> \tgit -C \"$P\" commit -m x &&\n> \n> -\tconfigure_exclusion \"$P\" .gitmodules >h &&\n> +\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n> \n> \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n> \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n> @@ -1026,7 +1067,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n> \tgit -C \"$P\" add .gitmodules &&\n> \tgit -C \"$P\" commit -m x &&\n> \n> -\tconfigure_exclusion \"$P\" .gitmodules >h &&\n> +\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n> \n> \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n> \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 5c1cd19612..34f8bb81a8 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -1744,9 +1744,12 @@ int upload_pack_advertise(struct repository *r,\n> \t\t allow_sideband_all_value))\n> \t\t\tstrbuf_addstr(value, \" sideband-all\");\n> \n> -\t\tif (!repo_config_get_string(the_repository,\n> +\t\tif ((!repo_config_get_string(the_repository,\n> \t\t\t\t\t \"uploadpack.blobpackfileuri\",\n> -\t\t\t\t\t &str) &&\n> +\t\t\t\t\t &str) ||\n> + !repo_config_get_string(the_repository,\n> + \"uploadpack.commitpackfileuri\",\n> + &str)) &&\n> \t\t str) {\n> \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n> \t\t\tfree(str);\n\nNot a new issue, but I wonder if we shouldn't just export\nconfigset_find_element(). This is at least 2 stackframes down the chain\nof the \"does this key exist?\" we actually care about here.\n"},{"id":"424846","messageId":"cover.1621327467.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"20210507021140.31372-1-dyroneteng@gmail.com","subject":"[PATCH v2 0/3] packfile-uris: commit objects exclusion","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-05-18T08:49:50Z","receivedAt":"2021-05-18T08:50:25Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Changes since v2:\n\n* Rename the \"uploadpack.blobpackfileuri\" configuration to\n \"uploadpack.excludeobject\".\n* In addition to blobs, packfile-uris now supports the exclusion of\n commit objects (recursive and non-recursive).\n* Added a patch to modify the packfile-uris.txt file.\n* Added a patch for related tests in t5702.\n\nAbout renaming, I do not know whether it will bring some compatibility\nimpact, packfile-uris now is an experimental feature, how to deal\nwith this situation, hoping to get some advice.\n\nAlso, I did not consider implementing packfile-uri support for tree\nobjects, because in the design scenario of packfile-uris, it seems to\nbe of little use.\n\nTeng Long (3):\n packfile-uris: support for excluding commit object\n packfile-uris.txt: excluding commit object\n t5702: excluding commits with packfile-uris\n\n Documentation/technical/packfile-uri.txt | 20 ++--\n builtin/pack-objects.c | 53 ++++++---\n fetch-pack.c | 5 +\n t/t5702-protocol-v2.sh | 145 +++++++++++++++++------\n upload-pack.c | 5 +-\n 5 files changed, 166 insertions(+), 62 deletions(-)\n\nRange-diff against v1:\n1: 1f2fb5c85f < -: ---------- Packfile-uris support excluding commit objects\n-: ---------- > 1: 73e64147b1 packfile-uris: support for excluding commit object\n-: ---------- > 2: 4abab98a76 packfile-uris.txt: excluding commit object\n-: ---------- > 3: e824cc26a7 t5702: excluding commits with packfile-uris\n-- \n2.31.1.442.g7e39198978.dirty\n\n"},{"id":"424847","messageId":"73e64147b17cb382d34357c913616095b6169650.1621327467.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1621327467.git.dyroneteng@gmail.com","subject":"[PATCH v2 1/3] packfile-uris: support for excluding commit object","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-05-18T08:49:51Z","receivedAt":"2021-05-18T08:50:34Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"On the server, more sophisticated means of excluding objects should be\nsupported, such as commit object. This commit introduces a new\nconfiguration `uploadpack.excludeobject` for this.\n\nThe old configuration `uploadpack.blobpackfileuri` is only support to\nexclude blobs and the name has no abstract meaning, so the configruation\nname changes, to support more object types. Compatibility issues will\nnot be considered because packfile-uris now is an experimental feature.\n\nIn addition to the configuration name, the format of the configuration\nvalue has also been expanded. When excluding the commits (or trees in\nthe future) objects, the old format ` `\ncan not express the meaning of recursion. So, the format is expanded,\nthe new format ` ` should\ndeal with this scenario (When processing commit objects, whether they\nare absolutely recursively excluded).\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 53 ++++++++++++++++++++++++++++++------------\n fetch-pack.c | 5 ++++\n upload-pack.c | 5 ++--\n 3 files changed, 45 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6d13cd3e1a..e687061420 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -99,6 +99,8 @@ static struct bitmap *reuse_packfile_bitmap;\n static int use_bitmap_index_default = 1;\n static int use_bitmap_index = -1;\n static int allow_pack_reuse = 1;\n+static int in_commit_order;\n+static int exclude_until_next_commit;\n static enum {\n \tWRITE_BITMAP_FALSE = 0,\n \tWRITE_BITMAP_QUIET,\n@@ -132,6 +134,7 @@ struct configured_exclusion {\n \tstruct oidmap_entry e;\n \tchar *pack_hash_hex;\n \tchar *uri;\n+\tint recursively:1;\n };\n static struct oidmap configured_exclusions;\n \n@@ -1291,10 +1294,16 @@ static int want_object_in_pack_one(struct packed_git *p,\n * and its offset in these variables.\n */\n static int want_object_in_pack(const struct object_id *oid,\n+\t\t\t enum object_type type,\n \t\t\t int exclude,\n \t\t\t struct packed_git **found_pack,\n \t\t\t off_t *found_offset)\n {\n+\tif (exclude_until_next_commit && type != OBJ_COMMIT)\n+\t\treturn 0;\n+\tif (type == OBJ_COMMIT)\n+\t\texclude_until_next_commit = 0 ;\n+\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n@@ -1345,6 +1354,8 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\t\t\t\t&p) &&\n \t\t\t\t *p == ':') {\n \t\t\t\t\toidset_insert(&excluded_by_config, oid);\n+\t\t\t\t\tif(ex->recursively && type == OBJ_COMMIT)\n+\t\t\t\t\t\texclude_until_next_commit = 1;\n \t\t\t\t\treturn 0;\n \t\t\t\t}\n \t\t\t}\n@@ -1394,7 +1405,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tif (have_duplicate_entry(oid, exclude))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) {\n+\tif (!want_object_in_pack(oid, type, exclude, &found_pack, &found_offset)) {\n \t\t/* The pack is missing an object, so it will not have closure */\n \t\tif (write_bitmap_index) {\n \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n@@ -1420,7 +1431,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \tif (have_duplicate_entry(oid, 0))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, 0, &pack, &offset))\n+\tif (!want_object_in_pack(oid, type, 0, &pack, &offset))\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, pack, offset);\n@@ -2985,27 +2996,33 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n+\tif (!strcmp(k, \"uploadpack.excludeobject\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n-\t\tconst char *oid_end, *pack_end;\n+\t\tconst char *oid_end, *pack_end, *recursively_end;\n \t\t/*\n \t\t * Stores the pack hash. This is not a true object ID, but is\n \t\t * of the same form.\n \t\t */\n \t\tstruct object_id pack_hash;\n-\n+\t\tchar recursively[2];\n \t\tif (parse_oid_hex(v, &ex->e.oid, &oid_end) ||\n \t\t *oid_end != ' ' ||\n-\t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n+\t\t !strlcpy(recursively, oid_end + 1, sizeof(recursively)) ||\n+\t\t parse_oid_hex(oid_end + 3, &pack_hash, &pack_end) ||\n \t\t *pack_end != ' ')\n-\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n-\t\t\t \"of the form ' ' (got '%s')\"), v);\n+ die(_(\"value of uploadpack.excludeobject must be \"\n+ \"of the form ' ' (got '%s')\"), v);\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n-\t\t\tdie(_(\"object already configured in another \"\n-\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n-\t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n-\t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n+ die(_(\"object already configured by an earlier \"\n+ \"uploadpack.excludeobject (got '%s')\"), v);\n+\t\trecursively_end = oid_end + 2;\n+\t\tex->pack_hash_hex = xcalloc(1, pack_end - recursively_end);\n+\t\tmemcpy(ex->pack_hash_hex, recursively_end + 1, pack_end - recursively_end - 1);\n \t\tex->uri = xstrdup(pack_end + 1);\n+\t\tif (atoi(recursively)) {\n+\t\t\tex->recursively = 1;\n+\t\t\tin_commit_order = 1;\n+ }\n \t\toidmap_put(&configured_exclusions, ex);\n \t}\n \treturn git_default_config(k, v, cb);\n@@ -3023,7 +3040,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \tstruct rev_info *revs = _data;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \toff_t ofs;\n-\tenum object_type type;\n+\tstatic enum object_type type;\n \n \tdisplay_progress(progress_state, ++nr_seen);\n \n@@ -3031,7 +3048,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tofs = nth_packed_object_offset(p, pos);\n-\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n+\tif (!want_object_in_pack(oid, type, 0, &p, &ofs))\n \t\treturn 0;\n \n \toi.typep = &type;\n@@ -3831,7 +3848,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"respect islands during delta compression\")),\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n-\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject with this protocol\")),\n \t\tOPT_END(),\n \t};\n \n@@ -3903,6 +3920,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tfetch_if_missing = 0;\n \t\tstrvec_push(&rp, \"--exclude-promisor-objects\");\n \t}\n+\n+\tif (in_commit_order){\n+\t\tuse_internal_rev_list = 1;\n+\t\tstrvec_push(&rp, \"--in-commit-order\");\n+\t}\n+\n \tif (unpack_unreachable || keep_unreachable || pack_loose_unreachable)\n \t\tuse_internal_rev_list = 1;\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 2318ebe680..cdf8777907 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -23,6 +23,7 @@\n #include \"fetch-negotiator.h\"\n #include \"fsck.h\"\n #include \"shallow.h\"\n+#include \"strmap.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -1677,6 +1678,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n+\tstruct strset uris;\n+\tstrset_init(&uris);\n \tfor (i = 0; i < packfile_uris.nr; i++) {\n \t\tint j;\n \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -1684,6 +1687,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\tconst char *uri = packfile_uris.items[i].string +\n \t\t\tthe_hash_algo->hexsz + 1;\n \n+\t\tif (!strset_add(&uris, uri))\n+\t\t\tcontinue;\n \t\tstrvec_push(&cmd.args, \"http-fetch\");\n \t\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n \t\t\t (int) the_hash_algo->hexsz,\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5c1cd19612..4d994658d2 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1745,9 +1745,8 @@ int upload_pack_advertise(struct repository *r,\n \t\t\tstrbuf_addstr(value, \" sideband-all\");\n \n \t\tif (!repo_config_get_string(the_repository,\n-\t\t\t\t\t \"uploadpack.blobpackfileuri\",\n-\t\t\t\t\t &str) &&\n-\t\t str) {\n+\t\t\t\t\t \"uploadpack.excludeobject\",\n+\t\t\t\t\t &str) && str) {\n \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n \t\t\tfree(str);\n \t\t}\n-- \n2.31.1.442.g7e39198978.dirty\n\n"},{"id":"424848","messageId":"4abab98a76966a34a166fc758b2e9b7cca38c156.1621327467.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1621327467.git.dyroneteng@gmail.com","subject":"[PATCH v2 2/3] packfile-uris.txt: excluding commit object","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-05-18T08:49:52Z","receivedAt":"2021-05-18T08:50:38Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Modified the content related to the configuration of packfile-uris, that\nis, the modification of the configuration format and the support for\nexcluding commit objects, and cut some descriptions about future work.\n\nSigned-off-by: Teng Long \n---\n Documentation/technical/packfile-uri.txt | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt\nindex f7eabc6c76..6ed850930f 100644\n--- a/Documentation/technical/packfile-uri.txt\n+++ b/Documentation/technical/packfile-uri.txt\n@@ -35,13 +35,16 @@ include some sort of non-trivial implementation in the Minimum Viable Product,\n at least so that we can test the client.\n \n This is the implementation: a feature, marked experimental, that allows the\n-server to be configured by one or more `uploadpack.blobPackfileUri=\n-` entries. Whenever the list of objects to be sent is assembled, all such\n-blobs are excluded, replaced with URIs. As noted in \"Future work\" below, the\n-server can evolve in the future to support excluding other objects (or other\n-implementations of servers could be made that support excluding other objects)\n-without needing a protocol change, so clients should not expect that packfiles\n-downloaded in this way only contain single blobs.\n+server to be configured by one or more entries with the format:\n+\n+ uploadpack.excludeobject= \n+\n+Value is the key of entry, and the object type can be a blob\n+or commit. Value works for commit object, if \n+is configured as '1', then the commit and all the referenced objects by\n+the commit will be recursively excluded. Otherwise, only the commit itself\n+will be excluded. Whenever the list of objects to be sent is assembled, all\n+such objects are excluded, replaced with URIs.\n \n Client design\n -------------\n@@ -65,9 +68,6 @@ The protocol design allows some evolution of the server and client without any\n need for protocol changes, so only a small-scoped design is included here to\n form the MVP. For example, the following can be done:\n \n- * On the server, more sophisticated means of excluding objects (e.g. by\n- specifying a commit to represent that commit and all objects that it\n- references).\n * On the client, resumption of clone. If a clone is interrupted, information\n could be recorded in the repository's config and a \"clone-resume\" command\n can resume the clone in progress. (Resumption of subsequent fetches is more\n-- \n2.31.1.442.g7e39198978.dirty\n\n"},{"id":"424849","messageId":"e824cc26a7a85673800429fd320279f3b6198576.1621327467.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1621327467.git.dyroneteng@gmail.com","subject":"[PATCH v2 3/3] t5702: excluding commits with packfile-uris","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-05-18T08:49:53Z","receivedAt":"2021-05-18T08:50:41Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Modify the logic of configure_exclusion to support the processing of\ncommits objects and add test cases for excluding commit objects\n(recursive and non-recursive).\n\nReplace \"rm...\" in the original test with \"test_when_finished...\".\nReplace \"git commit...\" in the original test with \"test_commit...\".\n\nSigned-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 145 +++++++++++++++++++++++++++++++----------\n 1 file changed, 111 insertions(+), 34 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 2e1243ca40..2148bfcda1 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -753,7 +753,7 @@ test_expect_success 'ls-remote with v2 http sends only one POST' '\n '\n \n test_expect_success 'push with http:// and a config of v2 does not request v2' '\n-\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \t# Till v2 for push is designed, make sure that if a client has\n \t# protocol.version configured to use v2, that the client instead falls\n \t# back and uses v0.\n@@ -776,7 +776,7 @@ test_expect_success 'push with http:// and a config of v2 does not request v2' '\n '\n \n test_expect_success 'when server sends \"ready\", expect DELIM' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -796,7 +796,7 @@ test_expect_success 'when server sends \"ready\", expect DELIM' '\n '\n \n test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -824,17 +824,39 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n '\n \n configure_exclusion () {\n-\tgit -C \"$1\" hash-object \"$2\" >objh &&\n-\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n-\tgit -C \"$1\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n-\tcat objh\n+ objt=\"$1\"\n+ P=\"$2\"\n+ recursive=\"$4\"\n+\n+ if test \"$objt\" = \"blob\"\n+ then\n+ git -C \"$P\" hash-object \"$3\" >objh &&\n+ git -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n+ git -C \"$P\" config --add \\\n+ \"uploadpack.excludeobject\" \\\n+ \"$(cat objh) $recursive $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+ cat objh\n+ elif test \"$objt\" = \"commit\"\n+ then\n+ echo \"$3\" >objh\n+ if test \"$recursive\" = 0\n+ then\n+ git -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n+ else\n+ git -C \"$2\" pack-objects --revs \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n+ fi\n+ git -C \"$P\" config --add \\\n+ \"uploadpack.excludeobject\" \\\n+ \"$(cat objh) $recursive $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+ cat objh\n+ else\n+ echo \"unsupported object type in configure_exclusion (got $objt)\"\n+ fi\n }\n \n test_expect_success 'part of packfile response provided as URI' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -843,10 +865,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\tconfigure_exclusion \"$P\" other-blob >h2 &&\n+\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n+\tconfigure_exclusion blob \"$P\" other-blob 0 >h2 &&\n \n \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n \tgit -c protocol.version=2 \\\n@@ -881,18 +903,18 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest_line_count = 6 filelist\n '\n \n-test_expect_success 'packfile URIs with fetch instead of clone' '\n+test_expect_success 'blobs packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n \n \tgit init http_child &&\n \n@@ -902,9 +924,65 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \t\tfetch \"$HTTPD_URL/smart/http_parent\"\n '\n \n+test_expect_success 'commits(not recursively) packfile URIs with fetch instead of clone' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n+\n+\tmycommit=$(git -C \"$P\" rev-parse A) &&\n+\techo other-blob >\"$P/other-blob\" &&\n+\tgit -C \"$P\" add other-blob &&\n+\ttest_commit -C \"$P\" B &&\n+\tothercommit=$(git -C \"$P\" rev-parse B) &&\n+\n+\tconfigure_exclusion commit \"$P\" \"$mycommit\" 0 >h &&\n+\tconfigure_exclusion commit \"$P\" \"$othercommit\" 0 >h2 &&\n+\n+\tgit init http_child &&\n+\n+\tGIT_TRACE=1 GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch \"$HTTPD_URL/smart/http_parent\"\n+'\n+\n+test_expect_success 'commits(recursively) packfile URIs with fetch instead of clone' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n+\n+\tmycommit=$(git -C \"$P\" rev-parse A) &&\n+\techo other-blob >\"$P/other-blob\" &&\n+ git -C \"$P\" add other-blob &&\n+\ttest_commit -C \"$P\" B &&\n+\tothercommit=$(git -C \"$P\" rev-parse B) &&\n+\n+\tconfigure_exclusion commit \"$P\" \"$mycommit\" 1 >h2 &&\n+\tconfigure_exclusion commit \"$P\" \"$othercommit\" 1 >h2 &&\n+\n+\tgit init http_child &&\n+\n+\tGIT_TRACE=1 GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch \"$HTTPD_URL/smart/http_parent\"\n+'\n+\n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -913,9 +991,9 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n \t# Configure a URL for other-blob. Just reuse the hash of the object as\n \t# the hash of the packfile, since the hash does not matter for this\n \t# test as long as it is not the hash of the pack, and it is of the\n@@ -923,8 +1001,8 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" hash-object other-blob >objh &&\n \tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n \tgit -C \"$P\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+\t\t\"uploadpack.excludeobject\" \\\n+\t\t\"$(cat objh) 0 $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n \n \ttest_must_fail env GIT_TEST_SIDEBAND_ALL=1 \\\n \t\tgit -c protocol.version=2 \\\n@@ -935,16 +1013,15 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -959,7 +1036,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -976,9 +1053,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -989,7 +1066,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1000,7 +1077,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \tgit -C \"$P\" add .gitmodules &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules 0 >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1015,7 +1092,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child err &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child err\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1024,9 +1101,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n \techo \"path = include/foo\" >>\"$P/.gitmodules\" &&\n \techo \"url = git://example.com/git/lib.git\" >>\"$P/.gitmodules\" &&\n \tgit -C \"$P\" add .gitmodules &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules 0 >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n-- \n2.31.1.442.g7e39198978.dirty\n\n"},{"id":"424917","messageId":"xmqq5yzfz6sm.fsf@gitster.g","threadId":"55638","inReplyTo":"73e64147b17cb382d34357c913616095b6169650.1621327467.git.dyroneteng@gmail.com","subject":"Re: [PATCH v2 1/3] packfile-uris: support for excluding commit object","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-19T04:28:09Z","receivedAt":"2021-05-19T04:28:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Teng Long writes:\n\n> On the server, more sophisticated means of excluding objects should be\n> supported, such as commit object. This commit introduces a new\n> configuration `uploadpack.excludeobject` for this.\n\nThis \"should\" is not justfied at all, it seems? What is lacking in\nwhat we already have? What new things does it all us to do by\nadding a new configuration variable?\n\n> The old configuration `uploadpack.blobpackfileuri` is only support to\n> exclude blobs and the name has no abstract meaning, so the configruation\n> name changes, to support more object types. Compatibility issues will\n> not be considered because packfile-uris now is an experimental feature.\n\nI'll let Jonathan speak up, but even for an experimental feature,\nwhatever new and incompatible way to do things should have a clear\nadvantage compared to the old way. Sell the backward incomptibility\nalong that line---\"it is an experimental so I'll trash it\" is not,\nbut \"by doing this it gets this much better, and migrating existing\nusers won't be too taxing (it is just this simple thing)\" is an\nacceptable way to justify such a change.\n\nNote that I am not opposed to the proposed change (and I am not\nsupporting it, either). I do have a problem with the way the change\nis sold, though.\n\n> builtin/pack-objects.c | 53 ++++++++++++++++++++++++++++++------------\n> fetch-pack.c | 5 ++++\n> upload-pack.c | 5 ++--\n> 3 files changed, 45 insertions(+), 18 deletions(-)\n\nEven though the name of the configuration variable changed, and the\nsemantics of the value of it changed, there is no documentation\nchange, because...?\n\nBecause the original didn't even document the variable properly? It\nmay be another reason why changing it may not impact the existing\nusers too much.\n\n> @@ -132,6 +134,7 @@ struct configured_exclusion {\n> \tstruct oidmap_entry e;\n> \tchar *pack_hash_hex;\n> \tchar *uri;\n> +\tint recursively:1;\n> };\n> static struct oidmap configured_exclusions;\n> \n> @@ -1291,10 +1294,16 @@ static int want_object_in_pack_one(struct packed_git *p,\n> * and its offset in these variables.\n> */\n> static int want_object_in_pack(const struct object_id *oid,\n> +\t\t\t enum object_type type,\n> \t\t\t int exclude,\n> \t\t\t struct packed_git **found_pack,\n> \t\t\t off_t *found_offset)\n> {\n> +\tif (exclude_until_next_commit && type != OBJ_COMMIT)\n> +\t\treturn 0;\n> +\tif (type == OBJ_COMMIT)\n> +\t\texclude_until_next_commit = 0 ;\n\nLose SP before the semicolon.\n\nOur codebase does not allow statements before declarations. Move\nall of the above down to be below the block of decls at the\nbeginning of the function.\n\n> \tint want;\n> \tstruct list_head *pos;\n> \tstruct multi_pack_index *m;\n\n> @@ -1345,6 +1354,8 @@ static int want_object_in_pack(const struct object_id *oid,\n> \t\t\t\t\t\t&p) &&\n> \t\t\t\t *p == ':') {\n> \t\t\t\t\toidset_insert(&excluded_by_config, oid);\n> +\t\t\t\t\tif(ex->recursively && type == OBJ_COMMIT)\n> +\t\t\t\t\t\texclude_until_next_commit = 1;\n\nThis depends on a new file-scope global variable, which means two\nthings.\n\n * if two or more threads are deciding which object to pack and not\n to pack, this code will horribly break, as they are traversing\n totally different parts of the object DAG to find out which\n objects to pack, but one thread hitting a commit to be excluded\n and setting this flag will cause other thread skip unrelated\n blobs and trees that it discovers, doesn't it?\n\n * even if we assume there is no concurrency and reentrancy issues\n (e.g. by forcing single-threaded operation when this feature is\n in use), the code _assumes_ a concrete order in which this helper\n function gets called, namely, non-commit objects fed to this\n helper after the helper gets a single commit object *all* belong\n to that commit. With the current code that feeds objects as they\n are discovered during depth first traversal of the top-level tree\n starting at each commit, that assumption might hold, but it feels\n that the assumption is too much to be healty. For example, would\n it be possible for the bitmap code to cause this helper to be\n called in different order (i.e. it might find it more convenent\n to feed a tree, a blob or a tag that is unrelated to the commit\n that was last fed to the helper)? If so, the logic in this code\n will constrain the caller too much.\n\nI'll stop reading for now at this place; review of the remainder may\ncome at a later time, but not now.\n\nThanks.\n"},{"id":"425031","messageId":"xmqqlf8aui5y.fsf@gitster.g","threadId":"55638","inReplyTo":"73e64147b17cb382d34357c913616095b6169650.1621327467.git.dyroneteng@gmail.com","subject":"Re: [PATCH v2 1/3] packfile-uris: support for excluding commit object","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-20T04:46:01Z","receivedAt":"2021-05-20T04:46:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"(continuing from yesterday's review)\n\n> @@ -3023,7 +3040,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \tstruct rev_info *revs = _data;\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> \toff_t ofs;\n> -\tenum object_type type;\n> +\tstatic enum object_type type;\n> \n> \tdisplay_progress(progress_state, ++nr_seen);\n> \n> @@ -3031,7 +3048,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t\treturn 0;\n> \n> \tofs = nth_packed_object_offset(p, pos);\n> -\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n> +\tif (!want_object_in_pack(oid, type, 0, &p, &ofs))\n> \t\treturn 0;\n> \n> \toi.typep = &type;\n\nThis change is puzzling.\n\nThe first call to this helper will use BSS initialized type to call\nwant_object_in_pack(), and then after that call says \"yes, we want\nthat object\", packed_object_info() gets called to learn what type it\nis. And the second call would use the type of the object we\npreviously handled, because type is a function scope static. Or if\nwe are not that lucky and multiple threads are involved, the type we\npass is from a random and unrelated object some other thread has\ncalled this helper with.\n\n"},{"id":"431169","messageId":"cover.1627292424.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1621327467.git.dyroneteng@gmail.com","subject":"[PATCH v3 0/3] packfile-uris: commit objects exclusio","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-07-26T09:46:11Z","receivedAt":"2021-07-26T09:46:45Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Changes since v3:\n\n* Ensure the forward compatibility of the old\n configuration (uploadpack.blobpackfileuri).\n* Reimplementation of the commit object exclusion method (without rely\n on the \"--in-commit-order\" arg).\n* Extend `show_object` function.\n* Remove `exclude_until_next_commit` var in pack-objects.c (Concurrency\n issues).\n* Restore the definition of want_object_in_pack method (problems caused\n by the new \"type\" parameter)\n\nTeng Long (3):\n packfile-uris: support for excluding commit objects\n t5702: support for excluding commit objects\n packfile-uri.txt: support for excluding commit objects\n\n Documentation/technical/packfile-uri.txt | 20 +--\n builtin/describe.c | 4 +-\n builtin/pack-objects.c | 97 +++++++------\n builtin/rev-list.c | 2 +-\n fetch-pack.c | 6 +\n list-objects.c | 37 ++---\n list-objects.h | 2 +-\n object.c | 15 +-\n object.h | 4 +\n pack-bitmap.c | 8 +-\n reachable.c | 8 +-\n revision.c | 36 +++--\n revision.h | 4 +\n t/t5702-protocol-v2.sh | 166 ++++++++++++++++++-----\n upload-pack.c | 7 +\n 15 files changed, 291 insertions(+), 125 deletions(-)\n\nRange-diff against v2:\n-: ---------- > 1: 91dce385f6 packfile-uris: support for excluding commit objects\n-: ---------- > 2: 92def8c72b t5702: support for excluding commit objects\n-: ---------- > 3: 01ab2cbb34 packfile-uri.txt: support for excluding commit objects\n-- \n2.31.1.443.g55c63af4c9.dirty\n\n"},{"id":"431170","messageId":"91dce385f630f4741bc6bb1540ef3c65e95a113f.1627292425.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1627292424.git.dyroneteng@gmail.com","subject":"[PATCH v3 1/3] packfile-uris: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-07-26T09:46:12Z","receivedAt":"2021-07-26T09:47:02Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"On the server, more sophisticated means of excluding objects should be\nsupported, such as commit object. This commit introduces a new\nconfiguration `uploadpack.excludeobject` for this.\n\nThe reason for bringing a new configuration is for two considerations.\nFirst, the old configuration supports a single object type (blob), which\nlimits the use of this feature. Secondly, the name of the old\nconfiguration is not abstract enough, this make extension difficult. If\ndifferent object types use different configuration names, the\nconfiguration items will be bloated and difficult to maintain, so the\nnew configuration is more abstract in name and easy to extend.\n\nAlthough a new configuration has been introduced, the old one is\nstill available and compatible with the new configuration. The old\nconfiguration `uploadpack.blobpackfileuri` only supports excluding\nblobs. The new configuration `uploadpack.excludeobject` not only\nsupports excluding blob objects, but also supports excluding commit\nobjects, as well as recursively excluding tree objects and blob objects\nthey contain.\n\nSigned-off-by: Teng Long \n---\n builtin/describe.c | 4 +-\n builtin/pack-objects.c | 97 ++++++++++++++++++++++++------------------\n builtin/rev-list.c | 2 +-\n fetch-pack.c | 6 +++\n list-objects.c | 37 +++++++++-------\n list-objects.h | 2 +-\n object.c | 15 +++++--\n object.h | 4 ++\n pack-bitmap.c | 8 ++--\n reachable.c | 8 ++--\n revision.c | 36 +++++++++++-----\n revision.h | 4 ++\n upload-pack.c | 7 +++\n 13 files changed, 148 insertions(+), 82 deletions(-)\n\ndiff --git a/builtin/describe.c b/builtin/describe.c\nindex 40482d8e9f..045da79b5c 100644\n--- a/builtin/describe.c\n+++ b/builtin/describe.c\n@@ -485,9 +485,9 @@ static void process_commit(struct commit *commit, void *data)\n \tpcd->current_commit = commit->object.oid;\n }\n \n-static void process_object(struct object *obj, const char *path, void *data)\n+static void process_object(struct object *obj, const char *path, void *show_data, void *carry_data)\n {\n-\tstruct process_commit_data *pcd = data;\n+\tstruct process_commit_data *pcd = show_data;\n \n \tif (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) {\n \t\treset_revision_walk();\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6d13cd3e1a..154c98bcb6 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1188,6 +1188,24 @@ static int have_duplicate_entry(const struct object_id *oid,\n \treturn 1;\n }\n \n+static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n+{\n+\tint i;\n+\tconst char *p;\n+\n+\tif (ex) {\n+\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n+\t\t\tif (skip_prefix(ex->uri,\n+\t\t\t\t\turi_protocols.items[i].string,\n+\t\t\t\t\t&p) &&\n+\t\t\t *p == ':')\n+\t\t\t\treturn 1;\n+\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n static int want_found_object(const struct object_id *oid, int exclude,\n \t\t\t struct packed_git *p)\n {\n@@ -1293,7 +1311,8 @@ static int want_object_in_pack_one(struct packed_git *p,\n static int want_object_in_pack(const struct object_id *oid,\n \t\t\t int exclude,\n \t\t\t struct packed_git **found_pack,\n-\t\t\t off_t *found_offset)\n+\t\t\t off_t *found_offset,\n+\t\t\t struct object *referred_commit)\n {\n \tint want;\n \tstruct list_head *pos;\n@@ -1333,21 +1352,13 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n-\t\tstruct configured_exclusion *ex =\n-\t\t\toidmap_get(&configured_exclusions, oid);\n-\t\tint i;\n-\t\tconst char *p;\n-\n-\t\tif (ex) {\n-\t\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n-\t\t\t\tif (skip_prefix(ex->uri,\n-\t\t\t\t\t\turi_protocols.items[i].string,\n-\t\t\t\t\t\t&p) &&\n-\t\t\t\t *p == ':') {\n-\t\t\t\t\toidset_insert(&excluded_by_config, oid);\n-\t\t\t\t\treturn 0;\n-\t\t\t\t}\n-\t\t\t}\n+\t\tif (referred_commit) {\n+\t\t\tif (oidmap_get(&configured_exclusions, &referred_commit->oid) && match_packfile_uri_exclusions(referred_ex))\n+\t\t\t\treturn 0;\n+\t\t}\n+\t\tif (oidmap_get(&configured_exclusions, oid) && match_packfile_uri_exclusions(ex)) {\n+\t\t\toidset_insert(&excluded_by_config, oid);\n+\t\t\treturn 0;\n \t\t}\n \t}\n \n@@ -1384,7 +1395,8 @@ static const char no_closure_warning[] = N_(\n );\n \n static int add_object_entry(const struct object_id *oid, enum object_type type,\n-\t\t\t const char *name, int exclude)\n+\t\t\t const char *name, int exclude,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct packed_git *found_pack = NULL;\n \toff_t found_offset = 0;\n@@ -1394,7 +1406,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tif (have_duplicate_entry(oid, exclude))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) {\n+\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) {\n \t\t/* The pack is missing an object, so it will not have closure */\n \t\tif (write_bitmap_index) {\n \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n@@ -1420,7 +1432,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \tif (have_duplicate_entry(oid, 0))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, 0, &pack, &offset))\n+\tif (!want_object_in_pack(oid, 0, &pack, &offset, NULL))\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, pack, offset);\n@@ -1560,7 +1572,7 @@ static void add_pbase_object(struct tree_desc *tree,\n \t\tif (name[cmplen] != '/') {\n \t\t\tadd_object_entry(&entry.oid,\n \t\t\t\t\t object_type(entry.mode),\n-\t\t\t\t\t fullname, 1);\n+\t\t\t\t\t fullname, 1, NULL);\n \t\t\treturn;\n \t\t}\n \t\tif (S_ISDIR(entry.mode)) {\n@@ -1628,7 +1640,7 @@ static void add_preferred_base_object(const char *name)\n \tcmplen = name_cmp_len(name);\n \tfor (it = pbase_tree; it; it = it->next) {\n \t\tif (cmplen == 0) {\n-\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1);\n+\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL);\n \t\t}\n \t\telse {\n \t\t\tstruct tree_desc tree;\n@@ -2830,7 +2842,7 @@ static void add_tag_chain(const struct object_id *oid)\n \t\t\tdie(_(\"unable to pack objects reachable from tag %s\"),\n \t\t\t oid_to_hex(oid));\n \n-\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0);\n+\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL);\n \n \t\tif (tag->tagged->type != OBJ_TAG)\n \t\t\treturn;\n@@ -2985,7 +2997,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n+\tif (!strcmp(k, \"uploadpack.excludeobject\") || !strcmp(k, \"uploadpack.blobpackfileuri\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n \t\tconst char *oid_end, *pack_end;\n \t\t/*\n@@ -2998,11 +3010,11 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t *oid_end != ' ' ||\n \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n \t\t *pack_end != ' ')\n-\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n-\t\t\t \"of the form ' ' (got '%s')\"), v);\n+ die(_(\"value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be \"\n+ \"of the form ' ' (got '%s')\"), v);\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n-\t\t\tdie(_(\"object already configured in another \"\n-\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n+ die(_(\"object already configured by an earlier \"\n+ \"uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')\"), v);\n \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n \t\tex->uri = xstrdup(pack_end + 1);\n@@ -3031,7 +3043,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tofs = nth_packed_object_offset(p, pos);\n-\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n+\tif (!want_object_in_pack(oid, 0, &p, &ofs, NULL))\n \t\treturn 0;\n \n \toi.typep = &type;\n@@ -3059,7 +3071,7 @@ static void show_commit_pack_hint(struct commit *commit, void *_data)\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *_data)\n+\t\t\t\t void *show_data, void *carry_data)\n {\n \tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n \tif (!oe)\n@@ -3224,7 +3236,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(_(\"expected object ID, got garbage:\\n %s\"), line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL);\n \t}\n }\n \n@@ -3233,7 +3245,7 @@ static void read_object_list_from_stdin(void)\n \n static void show_commit(struct commit *commit, void *data)\n {\n-\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0);\n+\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n \tcommit->object.flags |= OBJECT_ADDED;\n \n \tif (write_bitmap_index)\n@@ -3243,10 +3255,11 @@ static void show_commit(struct commit *commit, void *data)\n \t\tpropagate_island_marks(commit);\n }\n \n-static void show_object(struct object *obj, const char *name, void *data)\n+static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n+\tstruct object *referred_commit = carry_data;\n \tadd_preferred_base_object(name);\n-\tadd_object_entry(&obj->oid, obj->type, name, 0);\n+\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_commit);\n \tobj->flags |= OBJECT_ADDED;\n \n \tif (use_delta_islands) {\n@@ -3265,7 +3278,7 @@ static void show_object(struct object *obj, const char *name, void *data)\n \t}\n }\n \n-static void show_object__ma_allow_any(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_any(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tassert(arg_missing_action == MA_ALLOW_ANY);\n \n@@ -3276,10 +3289,10 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void\n \tif (!has_object(the_repository, &obj->oid, 0))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, show_data, carry_data);\n }\n \n-static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tassert(arg_missing_action == MA_ALLOW_PROMISOR);\n \n@@ -3290,7 +3303,7 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,\n \tif (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, show_data, carry_data);\n }\n \n static int option_parse_missing_action(const struct option *opt,\n@@ -3397,7 +3410,7 @@ static void add_objects_in_unpacked_packs(void)\n \t\tQSORT(in_pack.array, in_pack.nr, ofscmp);\n \t\tfor (i = 0; i < in_pack.nr; i++) {\n \t\t\tstruct object *o = in_pack.array[i].object;\n-\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0);\n+\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0, NULL);\n \t\t}\n \t}\n \tfree(in_pack.array);\n@@ -3413,7 +3426,7 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t\treturn 0;\n \t}\n \n-\tadd_object_entry(oid, type, \"\", 0);\n+\tadd_object_entry(oid, type, \"\", 0, NULL);\n \treturn 0;\n }\n \n@@ -3538,7 +3551,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \n static void record_recent_object(struct object *obj,\n \t\t\t\t const char *name,\n-\t\t\t\t void *data)\n+\t\t\t\t void *show_data,\n+\t\t\t\t void *carry_data)\n {\n \toid_array_append(&recent_objects, &obj->oid);\n }\n@@ -3831,7 +3845,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"respect islands during delta compression\")),\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n-\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject or \"\n+\t\t\t\t \t\"uploadpack.blobpackfileuri with this protocol\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..1cad33d9e8 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -266,7 +266,7 @@ static int finish_object(struct object *obj, const char *name, void *cb_data)\n \treturn 0;\n }\n \n-static void show_object(struct object *obj, const char *name, void *cb_data)\n+static void show_object(struct object *obj, const char *name, void *cb_data, void *carry_data)\n {\n \tstruct rev_list_info *info = cb_data;\n \tstruct rev_info *revs = info->revs;\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 2318ebe680..39bb449586 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -23,6 +23,7 @@\n #include \"fetch-negotiator.h\"\n #include \"fsck.h\"\n #include \"shallow.h\"\n+#include \"strmap.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -1576,6 +1577,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct string_list packfile_uris = STRING_LIST_INIT_DUP;\n \tint i;\n \tstruct strvec index_pack_args = STRVEC_INIT;\n+\tstruct strset uris;\n \n \tnegotiator = &negotiator_alloc;\n \tfetch_negotiator_init(r, negotiator);\n@@ -1677,6 +1679,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n+\tstrset_init(&uris);\n \tfor (i = 0; i < packfile_uris.nr; i++) {\n \t\tint j;\n \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -1684,6 +1687,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\tconst char *uri = packfile_uris.items[i].string +\n \t\t\tthe_hash_algo->hexsz + 1;\n \n+\t\tif (!strset_add(&uris, uri))\n+\t\t\tcontinue;\n \t\tstrvec_push(&cmd.args, \"http-fetch\");\n \t\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n \t\t\t (int) the_hash_algo->hexsz,\n@@ -1727,6 +1732,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t\t\t\t get_object_directory(),\n \t\t\t\t\t\t packname));\n \t}\n+\tstrset_clear(&uris);\n \tstring_list_clear(&packfile_uris, 0);\n \tstrvec_clear(&index_pack_args);\n \ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..fa3156dc89 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -24,7 +24,8 @@ struct traversal_context {\n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n@@ -60,7 +61,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -97,11 +98,13 @@ static void process_gitlink(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name);\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit);\n \n static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t struct tree *tree,\n-\t\t\t\t struct strbuf *base)\n+\t\t\t\t struct strbuf *base,\n+\t\t\t\t struct object *referred_commit)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -129,7 +132,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t\tprocess_tree(ctx, t, base, entry.path, referred_commit);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n@@ -142,7 +145,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t\tprocess_blob(ctx, b, base, entry.path, referred_commit);\n \t\t}\n \t}\n }\n@@ -150,7 +153,8 @@ static void process_tree_contents(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n@@ -191,14 +195,14 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n-\t\tprocess_tree_contents(ctx, tree, base);\n+\t\tprocess_tree_contents(ctx, tree, base, referred_commit);\n \n \tr = list_objects_filter__filter_object(ctx->revs->repo,\n \t\t\t\t\t LOFS_END_TREE, obj,\n@@ -207,7 +211,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -314,9 +318,9 @@ void mark_edges_uninteresting(struct rev_info *revs,\n \t}\n }\n \n-static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n+static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit)\n {\n-\tadd_pending_object(revs, &tree->object, \"\");\n+\tadd_pending_object_with_referred_commit(revs, &tree->object, \"\", referred_commit);\n }\n \n static void traverse_trees_and_blobs(struct traversal_context *ctx,\n@@ -329,23 +333,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n+\t\tstruct object *referred_commit = pending->referred_commit;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n@@ -370,7 +375,7 @@ static void do_traverse(struct traversal_context *ctx)\n \t\telse if (get_commit_tree(commit)) {\n \t\t\tstruct tree *tree = get_commit_tree(commit);\n \t\t\ttree->object.flags |= NOT_USER_GIVEN;\n-\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t\tadd_pending_tree(ctx->revs, tree, &commit->object);\n \t\t} else if (commit->object.parsed) {\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e46..ab946d34db 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -6,7 +6,7 @@ struct object;\n struct rev_info;\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n-typedef void (*show_object_fn)(struct object *, const char *, void *);\n+typedef void (*show_object_fn)(struct object *, const char *, void *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\ndiff --git a/object.c b/object.c\nindex 14188453c5..6b1ce2fcde 100644\n--- a/object.c\n+++ b/object.c\n@@ -322,9 +322,10 @@ void object_list_free(struct object_list **list)\n */\n static char object_array_slopbuf[1];\n \n-void add_object_array_with_path(struct object *obj, const char *name,\n-\t\t\t\tstruct object_array *array,\n-\t\t\t\tunsigned mode, const char *path)\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name,\n+\t\t\t\t\t\t struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path,\n+\t\t\t\t\t\t struct object *referred_commit)\n {\n \tunsigned nr = array->nr;\n \tunsigned alloc = array->alloc;\n@@ -339,6 +340,7 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \t}\n \tentry = &objects[nr];\n \tentry->item = obj;\n+\tentry->referred_commit = referred_commit;\n \tif (!name)\n \t\tentry->name = NULL;\n \telse if (!*name)\n@@ -354,6 +356,13 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \tarray->nr = ++nr;\n }\n \n+void add_object_array_with_path(struct object *obj, const char *name,\n+\t\t\t\tstruct object_array *array,\n+\t\t\t\tunsigned mode, const char *path)\n+{\n+\tadd_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL);\n+}\n+\n void add_object_array(struct object *obj, const char *name, struct object_array *array)\n {\n \tadd_object_array_with_path(obj, name, array, S_IFINVALID, NULL);\ndiff --git a/object.h b/object.h\nindex 87a6da47c8..de9f15b97d 100644\n--- a/object.h\n+++ b/object.h\n@@ -43,6 +43,7 @@ struct object_array {\n \tunsigned int alloc;\n \tstruct object_array_entry {\n \t\tstruct object *item;\n+\t\tstruct object *referred_commit;\n \t\t/*\n \t\t * name or NULL. If non-NULL, the memory pointed to\n \t\t * is owned by this object *except* if it points at\n@@ -157,6 +158,9 @@ void object_list_free(struct object_list **list);\n /* Object array handling .. */\n void add_object_array(struct object *obj, const char *name, struct object_array *array);\n void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path);\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path,\n+\t\t\t\t\t\t struct object *referred_commit);\n \n /*\n * Returns NULL if the array is empty. Otherwise, returns the last object\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3ed15431cd..516eb235da 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -459,9 +459,9 @@ struct bitmap_show_data {\n \tstruct bitmap *base;\n };\n \n-static void show_object(struct object *object, const char *name, void *data_)\n+static void show_object(struct object *object, const char *name, void *show_data, void *carry_data)\n {\n-\tstruct bitmap_show_data *data = data_;\n+\tstruct bitmap_show_data *data = show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(data->bitmap_git, &object->oid);\n@@ -1268,9 +1268,9 @@ struct bitmap_test_data {\n };\n \n static void test_show_object(struct object *object, const char *name,\n-\t\t\t void *data)\n+\t\t\t void *show_data, void *carry_data)\n {\n-\tstruct bitmap_test_data *tdata = data;\n+\tstruct bitmap_test_data *tdata = show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..ebd817c446 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -47,14 +47,14 @@ static int add_one_ref(const char *path, const struct object_id *oid,\n * The traversal will have already marked us as SEEN, so we\n * only need to handle any progress reporting here.\n */\n-static void mark_object(struct object *obj, const char *name, void *data)\n+static void mark_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n-\tupdate_progress(data);\n+\tupdate_progress(show_data);\n }\n \n-static void mark_commit(struct commit *c, void *data)\n+static void mark_commit(struct commit *c, void *show_data)\n {\n-\tmark_object(&c->object, NULL, data);\n+\tmark_object(&c->object, NULL, show_data, NULL);\n }\n \n struct recent_data {\ndiff --git a/revision.c b/revision.c\nindex 4853c85d0b..da0ce0e3f2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit)\n \tcommit_stack_clear(&pending);\n }\n \n-static void add_pending_object_with_path(struct rev_info *revs,\n-\t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode,\n-\t\t\t\t\t const char *path)\n+static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t\t struct object *referred_commit)\n {\n \tstruct interpret_branch_name_options options = { 0 };\n \tif (!obj)\n@@ -326,20 +327,36 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n+\tadd_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit);\n+}\n+\n+static void add_pending_object_with_path(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t const char *path) {\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL);\n }\n \n static void add_pending_object_with_mode(struct rev_info *revs,\n \t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode)\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t struct object *referred_commit)\n {\n-\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n+\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit);\n+}\n+\n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct object *referred_commit)\n+{\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit);\n }\n \n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name)\n {\n-\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID);\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL);\n }\n \n void add_head_to_pending(struct rev_info *revs)\n@@ -2764,7 +2781,6 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\t\tcontinue;\n \t\t}\n \n-\n \t\tif (handle_revision_arg(arg, revs, flags, revarg_opt)) {\n \t\t\tint j;\n \t\t\tif (seen_dashdash || *arg == '^')\n@@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tif (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc))\n \t\t\tdiagnose_missing_default(revs->def);\n \t\tobject = get_reference(revs, revs->def, &oid, 0);\n-\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode);\n+\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL);\n \t}\n \n \t/* Did the user ask for any diff output? Run the diff! */\ndiff --git a/revision.h b/revision.h\nindex a24f72dcd1..8a632e3587 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -424,6 +424,10 @@ void show_object_with_name(FILE *, struct object *, const char *);\n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name);\n \n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct object *referred_commit);\n+\n void add_pending_oid(struct rev_info *revs,\n \t\t const char *name, const struct object_id *oid,\n \t\t unsigned int flags);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5c1cd19612..d26fb351a3 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r,\n \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n \t\t\tfree(str);\n \t\t}\n+\n+\t\tif (!repo_config_get_string(the_repository,\n+\t\t\t\t\t \"uploadpack.excludeobject\",\n+\t\t\t\t\t &str) && str) {\n+\t\t\tstrbuf_addstr(value, \" packfile-uris\");\n+\t\t\tfree(str);\n+\t\t}\n \t}\n \n \treturn 1;\n-- \n2.31.1.443.g55c63af4c9.dirty\n\n"},{"id":"431171","messageId":"92def8c72b8df57a41df3b53216e193b5d05a1d6.1627292425.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1627292424.git.dyroneteng@gmail.com","subject":"[PATCH v3 2/3] t5702: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-07-26T09:46:13Z","receivedAt":"2021-07-26T09:47:09Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 166 +++++++++++++++++++++++++++++++++--------\n 1 file changed, 133 insertions(+), 33 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 2e1243ca40..bcf21e1445 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -753,7 +753,7 @@ test_expect_success 'ls-remote with v2 http sends only one POST' '\n '\n \n test_expect_success 'push with http:// and a config of v2 does not request v2' '\n-\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \t# Till v2 for push is designed, make sure that if a client has\n \t# protocol.version configured to use v2, that the client instead falls\n \t# back and uses v0.\n@@ -776,7 +776,7 @@ test_expect_success 'push with http:// and a config of v2 does not request v2' '\n '\n \n test_expect_success 'when server sends \"ready\", expect DELIM' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -796,7 +796,7 @@ test_expect_success 'when server sends \"ready\", expect DELIM' '\n '\n \n test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -824,17 +824,44 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n '\n \n configure_exclusion () {\n-\tgit -C \"$1\" hash-object \"$2\" >objh &&\n-\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n-\tgit -C \"$1\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n-\tcat objh\n+ objt=\"$1\"\n+ P=\"$2\"\n+\tversion=\"$3\"\n+\n+ oldc=\"uploadpack.blobpackfileuri\"\n+ newc=\"uploadpack.excludeobject\"\n+\tconfigkey=\"\"\n+\tif test \"$version\" = \"0\"\n+ then\n+ \tconfigkey=\"$oldc\"\n+ else\n+ \tconfigkey=\"$newc\"\n+\tfi\n+\n+ if test \"$objt\" = \"blob\"\n+ then\n+ git -C \"$P\" hash-object \"$3\" >objh &&\n+ git -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n+ git -C \"$P\" config --add \\\n+ \"$configkey\" \\\n+ \"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+ cat objh\n+ elif test \"$objt\" = \"commit\" || test \"$objt\" = \"tag\"\n+ then\n+ echo \"$3\" >objh\n+\t\tgit -C \"$2\" pack-objects --revs \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n+ git -C \"$P\" config --add \\\n+ \"$configkey\" \\\n+ \"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+ cat objh\n+ else\n+ echo \"unsupported object type in configure_exclusion (got $objt)\"\n+ fi\n }\n \n test_expect_success 'part of packfile response provided as URI' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -843,10 +870,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\tconfigure_exclusion \"$P\" other-blob >h2 &&\n+\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n+\tconfigure_exclusion blob \"$P\" other-blob 0 >h2 &&\n \n \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n \tgit -c protocol.version=2 \\\n@@ -881,18 +908,40 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest_line_count = 6 filelist\n '\n \n-test_expect_success 'packfile URIs with fetch instead of clone' '\n+test_expect_success 'blobs packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n+\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n+\n+\tgit init http_child &&\n+\n+\tGIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch \"$HTTPD_URL/smart/http_parent\"\n+'\n+\n+test_expect_success 'blobs packfile URIs(Compatible with the old) with fetch instead of clone' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\t# with the old \"uploadpack.blobpackfileuri\" configure\n+\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n \n \tgit init http_child &&\n \n@@ -902,9 +951,60 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \t\tfetch \"$HTTPD_URL/smart/http_parent\"\n '\n \n+test_expect_success 'commits packfile URIs with fetch instead of clone' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n+\n+\tmycommit=$(git -C \"$P\" rev-parse A) &&\n+\techo other-blob >\"$P/other-blob\" &&\n+ git -C \"$P\" add other-blob &&\n+\ttest_commit -C \"$P\" B &&\n+\tothercommit=$(git -C \"$P\" rev-parse B) &&\n+\n+\tconfigure_exclusion commit \"$P\" \"$mycommit\" >h2 &&\n+\tconfigure_exclusion commit \"$P\" \"$othercommit\" >h2 &&\n+\n+\tgit init http_child &&\n+\n+\tGIT_TRACE=1 GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch \"$HTTPD_URL/smart/http_parent\"\n+'\n+\n+test_expect_success 'tags packfile URIs with fetch instead of clone' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+ \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n+\tgit -C \"$P\" tag -a -m \"annotated_tag\" tagA &&\n+\ttagObj=$(git -C \"$P\" rev-parse tagA) &&\n+\n+\tconfigure_exclusion tag \"$P\" \"$tagObj\" >h2 &&\n+\n+\tgit init http_child &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=1 GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch --tags \"$HTTPD_URL/smart/http_parent\"\n+'\n+\n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -913,9 +1013,9 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \t# Configure a URL for other-blob. Just reuse the hash of the object as\n \t# the hash of the packfile, since the hash does not matter for this\n \t# test as long as it is not the hash of the pack, and it is of the\n@@ -923,7 +1023,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" hash-object other-blob >objh &&\n \tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n \tgit -C \"$P\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n+\t\t\"uploadpack.excludeobject\" \\\n \t\t\"$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n \n \ttest_must_fail env GIT_TEST_SIDEBAND_ALL=1 \\\n@@ -933,18 +1033,18 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \ttest_i18ngrep \"pack downloaded from.*does not match expected hash\" err\n '\n \n+\n test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -959,7 +1059,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -976,9 +1076,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -989,7 +1089,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1000,7 +1100,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \tgit -C \"$P\" add .gitmodules &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1015,7 +1115,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child err &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child err\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1024,9 +1124,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n \techo \"path = include/foo\" >>\"$P/.gitmodules\" &&\n \techo \"url = git://example.com/git/lib.git\" >>\"$P/.gitmodules\" &&\n \tgit -C \"$P\" add .gitmodules &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n-- \n2.31.1.443.g55c63af4c9.dirty\n\n"},{"id":"431172","messageId":"01ab2cbb34e6eb5f9b421d34ac368ea4e266fcd1.1627292425.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1627292424.git.dyroneteng@gmail.com","subject":"[PATCH v3 3/3] packfile-uri.txt: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-07-26T09:46:14Z","receivedAt":"2021-07-26T09:47:10Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n Documentation/technical/packfile-uri.txt | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt\nindex f7eabc6c76..2532db0e99 100644\n--- a/Documentation/technical/packfile-uri.txt\n+++ b/Documentation/technical/packfile-uri.txt\n@@ -35,13 +35,16 @@ include some sort of non-trivial implementation in the Minimum Viable Product,\n at least so that we can test the client.\n \n This is the implementation: a feature, marked experimental, that allows the\n-server to be configured by one or more `uploadpack.blobPackfileUri=\n-` entries. Whenever the list of objects to be sent is assembled, all such\n-blobs are excluded, replaced with URIs. As noted in \"Future work\" below, the\n-server can evolve in the future to support excluding other objects (or other\n-implementations of servers could be made that support excluding other objects)\n-without needing a protocol change, so clients should not expect that packfiles\n-downloaded in this way only contain single blobs.\n+server to be configured by one or more entries with the format:\n+\n+ uploadpack.excludeobject= \n+\n+Value is the key of entry, and the object type can be a blob\n+or commit. Whenever the list of objects to be sent is assembled, all such\n+objects are excluded, replaced with URIs. At the same time, for the old\n+configuration `uploadpack.blobPackfileUri= ` is\n+still compatible for now, but this configuration only supports the\n+exclusion of blob objects.\n \n Client design\n -------------\n@@ -65,9 +68,6 @@ The protocol design allows some evolution of the server and client without any\n need for protocol changes, so only a small-scoped design is included here to\n form the MVP. For example, the following can be done:\n \n- * On the server, more sophisticated means of excluding objects (e.g. by\n- specifying a commit to represent that commit and all objects that it\n- references).\n * On the client, resumption of clone. If a clone is interrupted, information\n could be recorded in the repository's config and a \"clone-resume\" command\n can resume the clone in progress. (Resumption of subsequent fetches is more\n-- \n2.31.1.443.g55c63af4c9.dirty\n\n"},{"id":"431177","messageId":"87czr5s162.fsf@evledraar.gmail.com","threadId":"55638","inReplyTo":"cover.1627292424.git.dyroneteng@gmail.com","subject":"Re: [PATCH v3 0/3] packfile-uris: commit objects exclusio","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-26T12:34:49Z","receivedAt":"2021-07-26T12:35:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 26 2021, Teng Long wrote:\n\n> Range-diff against v2:\n> -: ---------- > 1: 91dce385f6 packfile-uris: support for excluding commit objects\n> -: ---------- > 2: 92def8c72b t5702: support for excluding commit objects\n> -: ---------- > 3: 01ab2cbb34 packfile-uri.txt: support for excluding commit objects\n\nIt looks like you provided the wrong base for the --range-diff (likely\nmaster?), so it's not a diff against v2, just whatever you used as a\nbase.\n"},{"id":"431193","messageId":"87a6m9ru85.fsf@evledraar.gmail.com","threadId":"55638","inReplyTo":"92def8c72b8df57a41df3b53216e193b5d05a1d6.1627292425.git.dyroneteng@gmail.com","subject":"Re: [PATCH v3 2/3] t5702: support for excluding commit objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-26T15:03:15Z","receivedAt":"2021-07-26T15:16:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 26 2021, Teng Long wrote:\n\n> Signed-off-by: Teng Long \n> ---\n> t/t5702-protocol-v2.sh | 166 +++++++++++++++++++++++++++++++++--------\n> 1 file changed, 133 insertions(+), 33 deletions(-)\n>\n> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\n> index 2e1243ca40..bcf21e1445 100755\n> --- a/t/t5702-protocol-v2.sh\n> +++ b/t/t5702-protocol-v2.sh\n> @@ -753,7 +753,7 @@ test_expect_success 'ls-remote with v2 http sends only one POST' '\n> '\n> \n> test_expect_success 'push with http:// and a config of v2 does not request v2' '\n> -\ttest_when_finished \"rm -f log\" &&\n> +\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n> \t# Till v2 for push is designed, make sure that if a client has\n> \t# protocol.version configured to use v2, that the client instead falls\n> \t# back and uses v0.\n> @@ -776,7 +776,7 @@ test_expect_success 'push with http:// and a config of v2 does not request v2' '\n> '\n> \n> test_expect_success 'when server sends \"ready\", expect DELIM' '\n> -\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child &&\n> +\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child\" &&\n> \n> \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n> @@ -796,7 +796,7 @@ test_expect_success 'when server sends \"ready\", expect DELIM' '\n> '\n> \n> test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n> -\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log &&\n> +\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n> \n> \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n> @@ -824,17 +824,44 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n> '\n\nThis looks like a good cleanup, but should be split into another cleanup\ncommit. It looks unrelated.\n\n> configure_exclusion () {\n> -\tgit -C \"$1\" hash-object \"$2\" >objh &&\n> -\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n> -\tgit -C \"$1\" config --add \\\n> -\t\t\"uploadpack.blobpackfileuri\" \\\n> -\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n> -\tcat objh\n> + objt=\"$1\"\n> + P=\"$2\"\n> +\tversion=\"$3\"\n> +\n> + oldc=\"uploadpack.blobpackfileuri\"\n> + newc=\"uploadpack.excludeobject\"\n> +\tconfigkey=\"\"\n> +\tif test \"$version\" = \"0\"\n> + then\n> + \tconfigkey=\"$oldc\"\n> + else\n> + \tconfigkey=\"$newc\"\n> +\tfi\n\nYou've got all sorts of mixed space/tab indent here.\n\n> + if test \"$objt\" = \"blob\"\n> + then\n> + git -C \"$P\" hash-object \"$3\" >objh &&\n> + git -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n> + git -C \"$P\" config --add \\\n> + \"$configkey\" \\\n> + \"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n> + cat objh\n> + elif test \"$objt\" = \"commit\" || test \"$objt\" = \"tag\"\n> + then\n> + echo \"$3\" >objh\n> +\t\tgit -C \"$2\" pack-objects --revs \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n> + git -C \"$P\" config --add \\\n> + \"$configkey\" \\\n> + \"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n> + cat objh\n> + else\n> + echo \"unsupported object type in configure_exclusion (got $objt)\"\n> + fi\n> }\n> \n> test_expect_success 'part of packfile response provided as URI' '\n> \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> -\trm -rf \"$P\" http_child log &&\n> +\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n> \n> \tgit init \"$P\" &&\n> \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n> @@ -843,10 +870,10 @@ test_expect_success 'part of packfile response provided as URI' '\n> \tgit -C \"$P\" add my-blob &&\n> \techo other-blob >\"$P/other-blob\" &&\n> \tgit -C \"$P\" add other-blob &&\n> -\tgit -C \"$P\" commit -m x &&\n> +\ttest_commit -C \"$P\" A &&\n> \n> -\tconfigure_exclusion \"$P\" my-blob >h &&\n> -\tconfigure_exclusion \"$P\" other-blob >h2 &&\n> +\tconfigure_exclusion blob \"$P\" my-blob 0 >h &&\n> +\tconfigure_exclusion blob \"$P\" other-blob 0 >h2 &&\n> \n> \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n> \tgit -c protocol.version=2 \\\n> @@ -881,18 +908,40 @@ test_expect_success 'part of packfile response provided as URI' '\n> \ttest_line_count = 6 filelist\n> '\n> \n> -test_expect_success 'packfile URIs with fetch instead of clone' '\n> +test_expect_success 'blobs packfile URIs with fetch instead of clone' '\n> \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n> -\trm -rf \"$P\" http_child log &&\n> +\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n> \n> \tgit init \"$P\" &&\n> \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n> \n> \techo my-blob >\"$P/my-blob\" &&\n> \tgit -C \"$P\" add my-blob &&\n> -\tgit -C \"$P\" commit -m x &&\n> +\ttest_commit -C \"$P\" A &&\n> +\n> +\tconfigure_exclusion blob \"$P\" my-blob >h &&\n> +\n> +\tgit init http_child &&\n> +\n> +\tGIT_TEST_SIDEBAND_ALL=1 \\\n> +\tgit -C http_child -c protocol.version=2 \\\n> +\t\t-c fetch.uriprotocols=http,https \\\n\nIsn't accepting http and https the default? Is this guarding against\nconfig leak from a previous test? Ditto some later changes.\n"},{"id":"431209","messageId":"xmqqbl6pj60y.fsf@gitster.g","threadId":"55638","inReplyTo":"91dce385f630f4741bc6bb1540ef3c65e95a113f.1627292425.git.dyroneteng@gmail.com","subject":"Re: [PATCH v3 1/3] packfile-uris: support for excluding commit objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-26T18:15:25Z","receivedAt":"2021-07-26T18:15:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Teng Long writes:\n\n> On the server, more sophisticated means of excluding objects should be\n> supported, such as commit object. This commit introduces a new\n> configuration `uploadpack.excludeobject` for this.\n\nPlease avoid adjectives that express subjective values, like\n\"sophisticated\". Readers will expect a lot more sophistication than\nyour code actually offers and will be disappointed (\"wow, that would\nbe wonderful if we can say 'exclude commits made by bots, and those\nolder than 3 months'---eh, you cannot do that? where is your\nsophistication then?\").\n\nPlease avoid \"should\" without first describing the background for\n\"why it should\". It would help if you briefly describe what we\ncurrently have and its limitation before this first paragraph\n(i.e. your \"we can already exclude only blob objects\" would become\nmajor part of the explanation, but you'd need to present in what\nsituations it would help to be able to exclude other types).\n\nThis commit is probalby doing too many things at once. For example,\nrefactoring like creation of match_packfile_uri_exclusions() helper\nfunction out of existing code (there probably are others) can and\nshould be done as separate preparatory steps before the API gets\nmodified (e.g. process-object callbacks gain an extra parameter) in\ntree-wide way.\n\nAnd by slimming the primary step that introduces the new feature,\nthere will be a space to also add documentation and test in the same\nstep, which would help reviewers. With the current structure of the\nseries, with a code dump in the first step with only a vague promiss\nof \"sophistication\" without documentation updates, reviewers cannot\neven tell how the \"commit object\" is used easily.\n\nThanks.\n\n> builtin/describe.c | 4 +-\n> builtin/pack-objects.c | 97 ++++++++++++++++++++++++------------------\n> builtin/rev-list.c | 2 +-\n> fetch-pack.c | 6 +++\n> list-objects.c | 37 +++++++++-------\n> list-objects.h | 2 +-\n> object.c | 15 +++++--\n> object.h | 4 ++\n> pack-bitmap.c | 8 ++--\n> reachable.c | 8 ++--\n> revision.c | 36 +++++++++++-----\n> revision.h | 4 ++\n> upload-pack.c | 7 +++\n> 13 files changed, 148 insertions(+), 82 deletions(-)\n>\n> diff --git a/builtin/describe.c b/builtin/describe.c\n> index 40482d8e9f..045da79b5c 100644\n> --- a/builtin/describe.c\n> +++ b/builtin/describe.c\n> @@ -485,9 +485,9 @@ static void process_commit(struct commit *commit, void *data)\n> \tpcd->current_commit = commit->object.oid;\n> }\n> \n> -static void process_object(struct object *obj, const char *path, void *data)\n> +static void process_object(struct object *obj, const char *path, void *show_data, void *carry_data)\n> {\n> -\tstruct process_commit_data *pcd = data;\n> +\tstruct process_commit_data *pcd = show_data;\n> \n> \tif (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) {\n> \t\treset_revision_walk();\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 6d13cd3e1a..154c98bcb6 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1188,6 +1188,24 @@ static int have_duplicate_entry(const struct object_id *oid,\n> \treturn 1;\n> }\n> \n> +static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n> +{\n> +\tint i;\n> +\tconst char *p;\n> +\n> +\tif (ex) {\n> +\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n> +\t\t\tif (skip_prefix(ex->uri,\n> +\t\t\t\t\turi_protocols.items[i].string,\n> +\t\t\t\t\t&p) &&\n> +\t\t\t *p == ':')\n> +\t\t\t\treturn 1;\n> +\n> +\t\t}\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> static int want_found_object(const struct object_id *oid, int exclude,\n> \t\t\t struct packed_git *p)\n> {\n> @@ -1293,7 +1311,8 @@ static int want_object_in_pack_one(struct packed_git *p,\n> static int want_object_in_pack(const struct object_id *oid,\n> \t\t\t int exclude,\n> \t\t\t struct packed_git **found_pack,\n> -\t\t\t off_t *found_offset)\n> +\t\t\t off_t *found_offset,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tint want;\n> \tstruct list_head *pos;\n> @@ -1333,21 +1352,13 @@ static int want_object_in_pack(const struct object_id *oid,\n> \t}\n> \n> \tif (uri_protocols.nr) {\n> -\t\tstruct configured_exclusion *ex =\n> -\t\t\toidmap_get(&configured_exclusions, oid);\n> -\t\tint i;\n> -\t\tconst char *p;\n> -\n> -\t\tif (ex) {\n> -\t\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n> -\t\t\t\tif (skip_prefix(ex->uri,\n> -\t\t\t\t\t\turi_protocols.items[i].string,\n> -\t\t\t\t\t\t&p) &&\n> -\t\t\t\t *p == ':') {\n> -\t\t\t\t\toidset_insert(&excluded_by_config, oid);\n> -\t\t\t\t\treturn 0;\n> -\t\t\t\t}\n> -\t\t\t}\n> +\t\tif (referred_commit) {\n> +\t\t\tif (oidmap_get(&configured_exclusions, &referred_commit->oid) && match_packfile_uri_exclusions(referred_ex))\n> +\t\t\t\treturn 0;\n> +\t\t}\n> +\t\tif (oidmap_get(&configured_exclusions, oid) && match_packfile_uri_exclusions(ex)) {\n> +\t\t\toidset_insert(&excluded_by_config, oid);\n> +\t\t\treturn 0;\n> \t\t}\n> \t}\n> \n> @@ -1384,7 +1395,8 @@ static const char no_closure_warning[] = N_(\n> );\n> \n> static int add_object_entry(const struct object_id *oid, enum object_type type,\n> -\t\t\t const char *name, int exclude)\n> +\t\t\t const char *name, int exclude,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tstruct packed_git *found_pack = NULL;\n> \toff_t found_offset = 0;\n> @@ -1394,7 +1406,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n> \tif (have_duplicate_entry(oid, exclude))\n> \t\treturn 0;\n> \n> -\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) {\n> +\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) {\n> \t\t/* The pack is missing an object, so it will not have closure */\n> \t\tif (write_bitmap_index) {\n> \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n> @@ -1420,7 +1432,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n> \tif (have_duplicate_entry(oid, 0))\n> \t\treturn 0;\n> \n> -\tif (!want_object_in_pack(oid, 0, &pack, &offset))\n> +\tif (!want_object_in_pack(oid, 0, &pack, &offset, NULL))\n> \t\treturn 0;\n> \n> \tcreate_object_entry(oid, type, name_hash, 0, 0, pack, offset);\n> @@ -1560,7 +1572,7 @@ static void add_pbase_object(struct tree_desc *tree,\n> \t\tif (name[cmplen] != '/') {\n> \t\t\tadd_object_entry(&entry.oid,\n> \t\t\t\t\t object_type(entry.mode),\n> -\t\t\t\t\t fullname, 1);\n> +\t\t\t\t\t fullname, 1, NULL);\n> \t\t\treturn;\n> \t\t}\n> \t\tif (S_ISDIR(entry.mode)) {\n> @@ -1628,7 +1640,7 @@ static void add_preferred_base_object(const char *name)\n> \tcmplen = name_cmp_len(name);\n> \tfor (it = pbase_tree; it; it = it->next) {\n> \t\tif (cmplen == 0) {\n> -\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1);\n> +\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL);\n> \t\t}\n> \t\telse {\n> \t\t\tstruct tree_desc tree;\n> @@ -2830,7 +2842,7 @@ static void add_tag_chain(const struct object_id *oid)\n> \t\t\tdie(_(\"unable to pack objects reachable from tag %s\"),\n> \t\t\t oid_to_hex(oid));\n> \n> -\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0);\n> +\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL);\n> \n> \t\tif (tag->tagged->type != OBJ_TAG)\n> \t\t\treturn;\n> @@ -2985,7 +2997,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n> \t\treturn 0;\n> \t}\n> -\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n> +\tif (!strcmp(k, \"uploadpack.excludeobject\") || !strcmp(k, \"uploadpack.blobpackfileuri\")) {\n> \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n> \t\tconst char *oid_end, *pack_end;\n> \t\t/*\n> @@ -2998,11 +3010,11 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t *oid_end != ' ' ||\n> \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n> \t\t *pack_end != ' ')\n> -\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n> -\t\t\t \"of the form ' ' (got '%s')\"), v);\n> + die(_(\"value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be \"\n> + \"of the form ' ' (got '%s')\"), v);\n> \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n> -\t\t\tdie(_(\"object already configured in another \"\n> -\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n> + die(_(\"object already configured by an earlier \"\n> + \"uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')\"), v);\n> \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n> \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n> \t\tex->uri = xstrdup(pack_end + 1);\n> @@ -3031,7 +3043,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t\treturn 0;\n> \n> \tofs = nth_packed_object_offset(p, pos);\n> -\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n> +\tif (!want_object_in_pack(oid, 0, &p, &ofs, NULL))\n> \t\treturn 0;\n> \n> \toi.typep = &type;\n> @@ -3059,7 +3071,7 @@ static void show_commit_pack_hint(struct commit *commit, void *_data)\n> }\n> \n> static void show_object_pack_hint(struct object *object, const char *name,\n> -\t\t\t\t void *_data)\n> +\t\t\t\t void *show_data, void *carry_data)\n> {\n> \tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n> \tif (!oe)\n> @@ -3224,7 +3236,7 @@ static void read_object_list_from_stdin(void)\n> \t\t\tdie(_(\"expected object ID, got garbage:\\n %s\"), line);\n> \n> \t\tadd_preferred_base_object(p + 1);\n> -\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n> +\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL);\n> \t}\n> }\n> \n> @@ -3233,7 +3245,7 @@ static void read_object_list_from_stdin(void)\n> \n> static void show_commit(struct commit *commit, void *data)\n> {\n> -\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0);\n> +\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n> \tcommit->object.flags |= OBJECT_ADDED;\n> \n> \tif (write_bitmap_index)\n> @@ -3243,10 +3255,11 @@ static void show_commit(struct commit *commit, void *data)\n> \t\tpropagate_island_marks(commit);\n> }\n> \n> -static void show_object(struct object *obj, const char *name, void *data)\n> +static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n> {\n> +\tstruct object *referred_commit = carry_data;\n> \tadd_preferred_base_object(name);\n> -\tadd_object_entry(&obj->oid, obj->type, name, 0);\n> +\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_commit);\n> \tobj->flags |= OBJECT_ADDED;\n> \n> \tif (use_delta_islands) {\n> @@ -3265,7 +3278,7 @@ static void show_object(struct object *obj, const char *name, void *data)\n> \t}\n> }\n> \n> -static void show_object__ma_allow_any(struct object *obj, const char *name, void *data)\n> +static void show_object__ma_allow_any(struct object *obj, const char *name, void *show_data, void *carry_data)\n> {\n> \tassert(arg_missing_action == MA_ALLOW_ANY);\n> \n> @@ -3276,10 +3289,10 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void\n> \tif (!has_object(the_repository, &obj->oid, 0))\n> \t\treturn;\n> \n> -\tshow_object(obj, name, data);\n> +\tshow_object(obj, name, show_data, carry_data);\n> }\n> \n> -static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data)\n> +static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *show_data, void *carry_data)\n> {\n> \tassert(arg_missing_action == MA_ALLOW_PROMISOR);\n> \n> @@ -3290,7 +3303,7 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,\n> \tif (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid))\n> \t\treturn;\n> \n> -\tshow_object(obj, name, data);\n> +\tshow_object(obj, name, show_data, carry_data);\n> }\n> \n> static int option_parse_missing_action(const struct option *opt,\n> @@ -3397,7 +3410,7 @@ static void add_objects_in_unpacked_packs(void)\n> \t\tQSORT(in_pack.array, in_pack.nr, ofscmp);\n> \t\tfor (i = 0; i < in_pack.nr; i++) {\n> \t\t\tstruct object *o = in_pack.array[i].object;\n> -\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0);\n> +\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0, NULL);\n> \t\t}\n> \t}\n> \tfree(in_pack.array);\n> @@ -3413,7 +3426,7 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n> \t\treturn 0;\n> \t}\n> \n> -\tadd_object_entry(oid, type, \"\", 0);\n> +\tadd_object_entry(oid, type, \"\", 0, NULL);\n> \treturn 0;\n> }\n> \n> @@ -3538,7 +3551,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n> \n> static void record_recent_object(struct object *obj,\n> \t\t\t\t const char *name,\n> -\t\t\t\t void *data)\n> +\t\t\t\t void *show_data,\n> +\t\t\t\t void *carry_data)\n> {\n> \toid_array_append(&recent_objects, &obj->oid);\n> }\n> @@ -3831,7 +3845,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \t\t\t N_(\"respect islands during delta compression\")),\n> \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n> \t\t\t\tN_(\"protocol\"),\n> -\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n> +\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject or \"\n> +\t\t\t\t \t\"uploadpack.blobpackfileuri with this protocol\")),\n> \t\tOPT_END(),\n> \t};\n> \n> diff --git a/builtin/rev-list.c b/builtin/rev-list.c\n> index b4d8ea0a35..1cad33d9e8 100644\n> --- a/builtin/rev-list.c\n> +++ b/builtin/rev-list.c\n> @@ -266,7 +266,7 @@ static int finish_object(struct object *obj, const char *name, void *cb_data)\n> \treturn 0;\n> }\n> \n> -static void show_object(struct object *obj, const char *name, void *cb_data)\n> +static void show_object(struct object *obj, const char *name, void *cb_data, void *carry_data)\n> {\n> \tstruct rev_list_info *info = cb_data;\n> \tstruct rev_info *revs = info->revs;\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 2318ebe680..39bb449586 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -23,6 +23,7 @@\n> #include \"fetch-negotiator.h\"\n> #include \"fsck.h\"\n> #include \"shallow.h\"\n> +#include \"strmap.h\"\n> \n> static int transfer_unpack_limit = -1;\n> static int fetch_unpack_limit = -1;\n> @@ -1576,6 +1577,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \tstruct string_list packfile_uris = STRING_LIST_INIT_DUP;\n> \tint i;\n> \tstruct strvec index_pack_args = STRVEC_INIT;\n> +\tstruct strset uris;\n> \n> \tnegotiator = &negotiator_alloc;\n> \tfetch_negotiator_init(r, negotiator);\n> @@ -1677,6 +1679,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t}\n> \t}\n> \n> +\tstrset_init(&uris);\n> \tfor (i = 0; i < packfile_uris.nr; i++) {\n> \t\tint j;\n> \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n> @@ -1684,6 +1687,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\tconst char *uri = packfile_uris.items[i].string +\n> \t\t\tthe_hash_algo->hexsz + 1;\n> \n> +\t\tif (!strset_add(&uris, uri))\n> +\t\t\tcontinue;\n> \t\tstrvec_push(&cmd.args, \"http-fetch\");\n> \t\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n> \t\t\t (int) the_hash_algo->hexsz,\n> @@ -1727,6 +1732,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\t\t\t\t get_object_directory(),\n> \t\t\t\t\t\t packname));\n> \t}\n> +\tstrset_clear(&uris);\n> \tstring_list_clear(&packfile_uris, 0);\n> \tstrvec_clear(&index_pack_args);\n> \n> diff --git a/list-objects.c b/list-objects.c\n> index e19589baa0..fa3156dc89 100644\n> --- a/list-objects.c\n> +++ b/list-objects.c\n> @@ -24,7 +24,8 @@ struct traversal_context {\n> static void process_blob(struct traversal_context *ctx,\n> \t\t\t struct blob *blob,\n> \t\t\t struct strbuf *path,\n> -\t\t\t const char *name)\n> +\t\t\t const char *name,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tstruct object *obj = &blob->object;\n> \tsize_t pathlen;\n> @@ -60,7 +61,7 @@ static void process_blob(struct traversal_context *ctx,\n> \tif (r & LOFR_MARK_SEEN)\n> \t\tobj->flags |= SEEN;\n> \tif (r & LOFR_DO_SHOW)\n> -\t\tctx->show_object(obj, path->buf, ctx->show_data);\n> +\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n> \tstrbuf_setlen(path, pathlen);\n> }\n> \n> @@ -97,11 +98,13 @@ static void process_gitlink(struct traversal_context *ctx,\n> static void process_tree(struct traversal_context *ctx,\n> \t\t\t struct tree *tree,\n> \t\t\t struct strbuf *base,\n> -\t\t\t const char *name);\n> +\t\t\t const char *name,\n> +\t\t\t struct object *referred_commit);\n> \n> static void process_tree_contents(struct traversal_context *ctx,\n> \t\t\t\t struct tree *tree,\n> -\t\t\t\t struct strbuf *base)\n> +\t\t\t\t struct strbuf *base,\n> +\t\t\t\t struct object *referred_commit)\n> {\n> \tstruct tree_desc desc;\n> \tstruct name_entry entry;\n> @@ -129,7 +132,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n> \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n> \t\t\t}\n> \t\t\tt->object.flags |= NOT_USER_GIVEN;\n> -\t\t\tprocess_tree(ctx, t, base, entry.path);\n> +\t\t\tprocess_tree(ctx, t, base, entry.path, referred_commit);\n> \t\t}\n> \t\telse if (S_ISGITLINK(entry.mode))\n> \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n> @@ -142,7 +145,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n> \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n> \t\t\t}\n> \t\t\tb->object.flags |= NOT_USER_GIVEN;\n> -\t\t\tprocess_blob(ctx, b, base, entry.path);\n> +\t\t\tprocess_blob(ctx, b, base, entry.path, referred_commit);\n> \t\t}\n> \t}\n> }\n> @@ -150,7 +153,8 @@ static void process_tree_contents(struct traversal_context *ctx,\n> static void process_tree(struct traversal_context *ctx,\n> \t\t\t struct tree *tree,\n> \t\t\t struct strbuf *base,\n> -\t\t\t const char *name)\n> +\t\t\t const char *name,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tstruct object *obj = &tree->object;\n> \tstruct rev_info *revs = ctx->revs;\n> @@ -191,14 +195,14 @@ static void process_tree(struct traversal_context *ctx,\n> \tif (r & LOFR_MARK_SEEN)\n> \t\tobj->flags |= SEEN;\n> \tif (r & LOFR_DO_SHOW)\n> -\t\tctx->show_object(obj, base->buf, ctx->show_data);\n> +\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n> \tif (base->len)\n> \t\tstrbuf_addch(base, '/');\n> \n> \tif (r & LOFR_SKIP_TREE)\n> \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n> \telse if (!failed_parse)\n> -\t\tprocess_tree_contents(ctx, tree, base);\n> +\t\tprocess_tree_contents(ctx, tree, base, referred_commit);\n> \n> \tr = list_objects_filter__filter_object(ctx->revs->repo,\n> \t\t\t\t\t LOFS_END_TREE, obj,\n> @@ -207,7 +211,7 @@ static void process_tree(struct traversal_context *ctx,\n> \tif (r & LOFR_MARK_SEEN)\n> \t\tobj->flags |= SEEN;\n> \tif (r & LOFR_DO_SHOW)\n> -\t\tctx->show_object(obj, base->buf, ctx->show_data);\n> +\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n> \n> \tstrbuf_setlen(base, baselen);\n> \tfree_tree_buffer(tree);\n> @@ -314,9 +318,9 @@ void mark_edges_uninteresting(struct rev_info *revs,\n> \t}\n> }\n> \n> -static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n> +static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit)\n> {\n> -\tadd_pending_object(revs, &tree->object, \"\");\n> +\tadd_pending_object_with_referred_commit(revs, &tree->object, \"\", referred_commit);\n> }\n> \n> static void traverse_trees_and_blobs(struct traversal_context *ctx,\n> @@ -329,23 +333,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n> \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n> \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n> \t\tstruct object *obj = pending->item;\n> +\t\tstruct object *referred_commit = pending->referred_commit;\n> \t\tconst char *name = pending->name;\n> \t\tconst char *path = pending->path;\n> \t\tif (obj->flags & (UNINTERESTING | SEEN))\n> \t\t\tcontinue;\n> \t\tif (obj->type == OBJ_TAG) {\n> \t\t\tobj->flags |= SEEN;\n> -\t\t\tctx->show_object(obj, name, ctx->show_data);\n> +\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n> \t\t\tcontinue;\n> \t\t}\n> \t\tif (!path)\n> \t\t\tpath = \"\";\n> \t\tif (obj->type == OBJ_TREE) {\n> -\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n> +\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_commit);\n> \t\t\tcontinue;\n> \t\t}\n> \t\tif (obj->type == OBJ_BLOB) {\n> -\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n> +\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_commit);\n> \t\t\tcontinue;\n> \t\t}\n> \t\tdie(\"unknown pending object %s (%s)\",\n> @@ -370,7 +375,7 @@ static void do_traverse(struct traversal_context *ctx)\n> \t\telse if (get_commit_tree(commit)) {\n> \t\t\tstruct tree *tree = get_commit_tree(commit);\n> \t\t\ttree->object.flags |= NOT_USER_GIVEN;\n> -\t\t\tadd_pending_tree(ctx->revs, tree);\n> +\t\t\tadd_pending_tree(ctx->revs, tree, &commit->object);\n> \t\t} else if (commit->object.parsed) {\n> \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n> \t\t\t oid_to_hex(&commit->object.oid));\n> diff --git a/list-objects.h b/list-objects.h\n> index a952680e46..ab946d34db 100644\n> --- a/list-objects.h\n> +++ b/list-objects.h\n> @@ -6,7 +6,7 @@ struct object;\n> struct rev_info;\n> \n> typedef void (*show_commit_fn)(struct commit *, void *);\n> -typedef void (*show_object_fn)(struct object *, const char *, void *);\n> +typedef void (*show_object_fn)(struct object *, const char *, void *, void *);\n> void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n> \n> typedef void (*show_edge_fn)(struct commit *);\n> diff --git a/object.c b/object.c\n> index 14188453c5..6b1ce2fcde 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -322,9 +322,10 @@ void object_list_free(struct object_list **list)\n> */\n> static char object_array_slopbuf[1];\n> \n> -void add_object_array_with_path(struct object *obj, const char *name,\n> -\t\t\t\tstruct object_array *array,\n> -\t\t\t\tunsigned mode, const char *path)\n> +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name,\n> +\t\t\t\t\t\t struct object_array *array,\n> +\t\t\t\t\t\t unsigned mode, const char *path,\n> +\t\t\t\t\t\t struct object *referred_commit)\n> {\n> \tunsigned nr = array->nr;\n> \tunsigned alloc = array->alloc;\n> @@ -339,6 +340,7 @@ void add_object_array_with_path(struct object *obj, const char *name,\n> \t}\n> \tentry = &objects[nr];\n> \tentry->item = obj;\n> +\tentry->referred_commit = referred_commit;\n> \tif (!name)\n> \t\tentry->name = NULL;\n> \telse if (!*name)\n> @@ -354,6 +356,13 @@ void add_object_array_with_path(struct object *obj, const char *name,\n> \tarray->nr = ++nr;\n> }\n> \n> +void add_object_array_with_path(struct object *obj, const char *name,\n> +\t\t\t\tstruct object_array *array,\n> +\t\t\t\tunsigned mode, const char *path)\n> +{\n> +\tadd_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL);\n> +}\n> +\n> void add_object_array(struct object *obj, const char *name, struct object_array *array)\n> {\n> \tadd_object_array_with_path(obj, name, array, S_IFINVALID, NULL);\n> diff --git a/object.h b/object.h\n> index 87a6da47c8..de9f15b97d 100644\n> --- a/object.h\n> +++ b/object.h\n> @@ -43,6 +43,7 @@ struct object_array {\n> \tunsigned int alloc;\n> \tstruct object_array_entry {\n> \t\tstruct object *item;\n> +\t\tstruct object *referred_commit;\n> \t\t/*\n> \t\t * name or NULL. If non-NULL, the memory pointed to\n> \t\t * is owned by this object *except* if it points at\n> @@ -157,6 +158,9 @@ void object_list_free(struct object_list **list);\n> /* Object array handling .. */\n> void add_object_array(struct object *obj, const char *name, struct object_array *array);\n> void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path);\n> +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array,\n> +\t\t\t\t\t\t unsigned mode, const char *path,\n> +\t\t\t\t\t\t struct object *referred_commit);\n> \n> /*\n> * Returns NULL if the array is empty. Otherwise, returns the last object\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 3ed15431cd..516eb235da 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -459,9 +459,9 @@ struct bitmap_show_data {\n> \tstruct bitmap *base;\n> };\n> \n> -static void show_object(struct object *object, const char *name, void *data_)\n> +static void show_object(struct object *object, const char *name, void *show_data, void *carry_data)\n> {\n> -\tstruct bitmap_show_data *data = data_;\n> +\tstruct bitmap_show_data *data = show_data;\n> \tint bitmap_pos;\n> \n> \tbitmap_pos = bitmap_position(data->bitmap_git, &object->oid);\n> @@ -1268,9 +1268,9 @@ struct bitmap_test_data {\n> };\n> \n> static void test_show_object(struct object *object, const char *name,\n> -\t\t\t void *data)\n> +\t\t\t void *show_data, void *carry_data)\n> {\n> -\tstruct bitmap_test_data *tdata = data;\n> +\tstruct bitmap_test_data *tdata = show_data;\n> \tint bitmap_pos;\n> \n> \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n> diff --git a/reachable.c b/reachable.c\n> index 77a60c70a5..ebd817c446 100644\n> --- a/reachable.c\n> +++ b/reachable.c\n> @@ -47,14 +47,14 @@ static int add_one_ref(const char *path, const struct object_id *oid,\n> * The traversal will have already marked us as SEEN, so we\n> * only need to handle any progress reporting here.\n> */\n> -static void mark_object(struct object *obj, const char *name, void *data)\n> +static void mark_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n> {\n> -\tupdate_progress(data);\n> +\tupdate_progress(show_data);\n> }\n> \n> -static void mark_commit(struct commit *c, void *data)\n> +static void mark_commit(struct commit *c, void *show_data)\n> {\n> -\tmark_object(&c->object, NULL, data);\n> +\tmark_object(&c->object, NULL, show_data, NULL);\n> }\n> \n> struct recent_data {\n> diff --git a/revision.c b/revision.c\n> index 4853c85d0b..da0ce0e3f2 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit)\n> \tcommit_stack_clear(&pending);\n> }\n> \n> -static void add_pending_object_with_path(struct rev_info *revs,\n> -\t\t\t\t\t struct object *obj,\n> -\t\t\t\t\t const char *name, unsigned mode,\n> -\t\t\t\t\t const char *path)\n> +static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs,\n> +\t\t\t\t\t\t\t struct object *obj,\n> +\t\t\t\t\t\t\t const char *name, unsigned mode,\n> +\t\t\t\t\t\t\t const char *path,\n> +\t\t\t\t\t\t\t struct object *referred_commit)\n> {\n> \tstruct interpret_branch_name_options options = { 0 };\n> \tif (!obj)\n> @@ -326,20 +327,36 @@ static void add_pending_object_with_path(struct rev_info *revs,\n> \t\tstrbuf_release(&buf);\n> \t\treturn; /* do not add the commit itself */\n> \t}\n> -\tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n> +\tadd_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit);\n> +}\n> +\n> +static void add_pending_object_with_path(struct rev_info *revs,\n> +\t\t\t\t\t struct object *obj,\n> +\t\t\t\t\t const char *name, unsigned mode,\n> +\t\t\t\t\t const char *path) {\n> +\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL);\n> }\n> \n> static void add_pending_object_with_mode(struct rev_info *revs,\n> \t\t\t\t\t struct object *obj,\n> -\t\t\t\t\t const char *name, unsigned mode)\n> +\t\t\t\t\t const char *name, unsigned mode,\n> +\t\t\t\t\t struct object *referred_commit)\n> {\n> -\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n> +\n> +\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit);\n> +}\n> +\n> +void add_pending_object_with_referred_commit(struct rev_info *revs,\n> +\t\t\t\t\t struct object *obj, const char *name,\n> +\t\t\t\t\t struct object *referred_commit)\n> +{\n> +\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit);\n> }\n> \n> void add_pending_object(struct rev_info *revs,\n> \t\t\tstruct object *obj, const char *name)\n> {\n> -\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID);\n> +\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL);\n> }\n> \n> void add_head_to_pending(struct rev_info *revs)\n> @@ -2764,7 +2781,6 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n> \t\t\tcontinue;\n> \t\t}\n> \n> -\n> \t\tif (handle_revision_arg(arg, revs, flags, revarg_opt)) {\n> \t\t\tint j;\n> \t\t\tif (seen_dashdash || *arg == '^')\n> @@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n> \t\tif (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc))\n> \t\t\tdiagnose_missing_default(revs->def);\n> \t\tobject = get_reference(revs, revs->def, &oid, 0);\n> -\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode);\n> +\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL);\n> \t}\n> \n> \t/* Did the user ask for any diff output? Run the diff! */\n> diff --git a/revision.h b/revision.h\n> index a24f72dcd1..8a632e3587 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -424,6 +424,10 @@ void show_object_with_name(FILE *, struct object *, const char *);\n> void add_pending_object(struct rev_info *revs,\n> \t\t\tstruct object *obj, const char *name);\n> \n> +void add_pending_object_with_referred_commit(struct rev_info *revs,\n> +\t\t\t\t\t struct object *obj, const char *name,\n> +\t\t\t\t\t struct object *referred_commit);\n> +\n> void add_pending_oid(struct rev_info *revs,\n> \t\t const char *name, const struct object_id *oid,\n> \t\t unsigned int flags);\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 5c1cd19612..d26fb351a3 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r,\n> \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n> \t\t\tfree(str);\n> \t\t}\n> +\n> +\t\tif (!repo_config_get_string(the_repository,\n> +\t\t\t\t\t \"uploadpack.excludeobject\",\n> +\t\t\t\t\t &str) && str) {\n> +\t\t\tstrbuf_addstr(value, \" packfile-uris\");\n> +\t\t\tfree(str);\n> +\t\t}\n> \t}\n> \n> \treturn 1;\n"},{"id":"431226","messageId":"60ff10f378513_3b8d2085d@natae.notmuch","threadId":"55638","inReplyTo":"xmqqbl6pj60y.fsf@gitster.g","subject":"Re: [PATCH v3 1/3] packfile-uris: support for excluding commit objects","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-07-26T19:45:55Z","receivedAt":"2021-07-26T19:46:00Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Junio C Hamano wrote:\n> Teng Long writes:\n> \n> > On the server, more sophisticated means of excluding objects should be\n> > supported, such as commit object. This commit introduces a new\n> > configuration `uploadpack.excludeobject` for this.\n> \n> Please avoid adjectives that express subjective values, like\n> \"sophisticated\".\n\nThe word \"sophisticated\" is not necessarily subjective, it can easily\nmean \"complex\".\n\nhttps://www.merriam-webster.com/dictionary/sophisticated\n\n-- \nFelipe Contreras\n"},{"id":"431236","messageId":"xmqqr1fkhk5w.fsf@gitster.g","threadId":"55638","inReplyTo":"01ab2cbb34e6eb5f9b421d34ac368ea4e266fcd1.1627292425.git.dyroneteng@gmail.com","subject":"Re: [PATCH v3 3/3] packfile-uri.txt: support for excluding commit objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-26T20:52:59Z","receivedAt":"2021-07-26T20:53:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Teng Long writes:\n\n> +++ b/Documentation/technical/packfile-uri.txt\n> @@ -35,13 +35,16 @@ include some sort of non-trivial implementation in the Minimum Viable Product,\n> at least so that we can test the client.\n> \n> This is the implementation: a feature, marked experimental, that allows the\n> +server to be configured by one or more entries with the format:\n> +\n> + uploadpack.excludeobject= \n> +\n> +Value is the key of entry, and the object type can be a blob\n> +or commit. Whenever the list of objects to be sent is assembled, all such\n> +objects are excluded, replaced with URIs. At the same time, for the old\n> +configuration `uploadpack.blobPackfileUri= ` is\n> +still compatible for now, but this configuration only supports the\n> +exclusion of blob objects.\n\nDo not hint deprecation and future removal with \"still\" and \"for\nnow\", before seeing a concensus that it should be deprecated and\nremoved.\n\nThe new thing, , deserves some explanation. What are\nthe acceptable values (yes/no? spatial/time/both? infinitely/limited?)\nand what do these values mean?\n\nWhy is this limited to only and ?\n\nThere isn't a fundamental reason why I shouldn't be able to say\n\"v2.32.0\" instead of ebf3c04b262aa27fbb97f8a0156c2347fecafafb (or\n\"v2.32.0~0\") to say \"I want anything reachable from v2.32.0 (in\nother words, that version and everything before it)\", is there?\n\nFor that matter, \"everything reachable from this tree object\" may\nalso be a reasonable way to specify which set of objects are\noffloaded to an out-of-band URI.\n\nThanks.\n"},{"id":"432433","messageId":"20210811014457.15318-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"xmqqbl6pj60y.fsf@gitster.g","subject":"Re: [PATCH v3 1/3] packfile-uris: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T01:44:57Z","receivedAt":"2021-08-11T01:45:06Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":">> Please avoid adjectives that express subjective values, like\n>> \"sophisticated\". Readers will expect a lot more sophistication than\n>> your code actually offers and will be disappointed (\"wow, that would\n>> be wonderful if we can say 'exclude commits made by bots, and those\n>> older than 3 months'---eh, you cannot do that? where is your\n>> sophistication then?\").\n\nI use it(\"sophisticated\") from \"packfile-uri.txt\", but i agree with you.\nDocumentation will also remove/replace the subjective word in the next\npatch.\n\n>> Please avoid \"should\" without first describing the background for\n>> \"why it should\". It would help if you briefly describe what we\n>> currently have and its limitation before this first paragraph\n>> (i.e. your \"we can already exclude only blob objects\" would become\n>> major part of the explanation, but you'd need to present in what\n>> situations it would help to be able to exclude other types).\n\nAgree.\nCommit message will be appended with backgroud description.\n\n>> This commit is probalby doing too many things at once. For example,\n>> refactoring like creation of match_packfile_uri_exclusions() helper\n>> function out of existing code (there probably are others) can and\n>> should be done as separate preparatory steps before the API gets\n>> modified (e.g. process-object callbacks gain an e xtra parameter) in\n>> tree-wide way.\n>>\n>> And by slimming the primary step that introduces the new feature,\n>> there will be a space to also add documentation and test in the same\n>> step, which would help reviewers. With the current structure of the\n>> series, with a code dump in the first step with only a vague promiss\n>> of \"sophistication\" without documentation updates, reviewers cannot\n>> even tell how the \"commit object\" is used easily.\n\nAgree.\nThe current commit will be splitted for more clear reponsibilities, \ndocumentation and tests ditto.\n\nThank you.\n"},{"id":"432434","messageId":"20210811014607.15419-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"87a6m9ru85.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 1/3] packfile-uris: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T01:46:07Z","receivedAt":"2021-08-11T01:46:17Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"\n\n>>This looks like a good cleanup, but should be split into another cleanup\n>>commit. It looks unrelated.\n\nAgree.\nWill split up in next patchset.\n\n>>You've got all sorts of mixed space/tab indent here.\n\nYes.\nWill fix wrong indents in next patchset.\n\n>>Isn't accepting http and https the default?\n\nI think it's NOT after I took a view at function `fetch_pack_config`.\nThis may be optimized in another patch I think and also will listen\nto some other suggestions.\n\nThank you.\n\t \n"},{"id":"432435","messageId":"20210811014742.15594-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"xmqqr1fkhk5w.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] packfile-uri.txt: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T01:47:42Z","receivedAt":"2021-08-11T01:47:51Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":">>Do not hint deprecation and future removal with \"still\" and \"for\n>>now\", before seeing a concensus that it should be deprecated and\n>>removed.\n\nAgree.\nWill remove.\n\n>>The new thing, , deserves some explanation. What are\n>>the acceptable values (yes/no? spatial/time/both? infinitely/limited?)\n>>and what do these values mean?\n\nMore expanation.\nI think over about and consider to remove it in next patch\nbecause offer a pack only contains commit or tree object individually\nmay not make much sense, so will remove(as default on tree\nand commit object) in next patch.\n\n>>Why is this limited to only and ?\n\nWill support tree but not tag(maybe furture work) in next patch.\n\n>>There isn't a fundamental reason why I shouldn't be able to say\n>>\"v2.32.0\" instead of ebf3c04b262aa27fbb97f8a0156c2347fecafafb (or\n>>\"v2.32.0~0\") to say \"I want anything reachable from v2.32.0 (in\n>>other words, that version and everything before it)\", is there?\n\n>>For that matter, \"everything reachable from this tree object\" may\n>>also be a reasonable way to specify which set of objects are\n>>offloaded to an out-of-band URI.\n\nAgree.\nWill provide more detailed instuctions in next patch.\n\nThank you.\n\n\n"},{"id":"432436","messageId":"20210811014833.15753-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"87czr5s162.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 0/3] packfile-uris: commit objects exclusio","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T01:48:33Z","receivedAt":"2021-08-11T01:48:41Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":">>It looks like you provided the wrong base for the --range-diff (likely\n>>master?), so it's not a diff against v2, just whatever you used as a\n>>base.\n\nAgree.\nSorry about that, the first use made a mistake. Maybe corrected next time :)\n\nThank you.\n"},{"id":"432456","messageId":"cover.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1627292424.git.dyroneteng@gmail.com","subject":"[PATCH v4 0/7] packfile-uris: commits and trees exclusion","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:47Z","receivedAt":"2021-08-11T07:46:14Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Changes since v4:\n\n* Remove subjective words in commit message.\n* Add background descriptions in commit message.\n* Big commit split.\n* Fix `t/t5702-protocol-v2.sh` indent problems.\n* Test commit split(cleanup and new test case).\n* Remove hint deprecation words in documentation.\n* Support tree object exclusion.\n* Add details about recursive exclusion of commits and trees in documentation.\n\nTeng Long (7):\n pack-objects.c: introduce new method `match_packfile_uri_exclusions`\n Add new parameter \"carry_data\" for \"show_object\" function\n packfile-uri: support for excluding commit objects\n packfile-uri: support for excluding tree objects\n packfile-uri.txt: support for excluding commits and trees\n t5702: replace with \"test_when_finished\" for cleanup\n t5702: support for excluding commit objects\n\n Documentation/technical/packfile-uri.txt | 32 ++-\n builtin/describe.c | 4 +-\n builtin/pack-objects.c | 104 +++++---\n builtin/rev-list.c | 2 +-\n list-objects.c | 41 +--\n list-objects.h | 2 +-\n object.c | 19 +-\n object.h | 15 +-\n pack-bitmap.c | 8 +-\n reachable.c | 8 +-\n revision.c | 34 ++-\n revision.h | 3 +\n t/t5702-protocol-v2.sh | 312 ++++++++++++++++++++---\n upload-pack.c | 7 +\n 14 files changed, 466 insertions(+), 125 deletions(-)\n\nRange-diff against v3:\n-: ---------- > 1: 73a5b4ccc1 pack-objects.c: introduce new method `match_packfile_uri_exclusions`\n-: ---------- > 2: bc8fea97e3 Add new parameter \"carry_data\" for \"show_object\" function\n1: f324359ec8 ! 3: f71b310842 packfile-uris: support for excluding commit objects\n @@ Metadata\n Author: Teng Long \n \n ## Commit message ##\n - packfile-uris: support for excluding commit objects\n + packfile-uri: support for excluding commit objects\n \n - On the server, more sophisticated means of excluding objects should be\n - supported, such as commit object. This commit introduces a new\n - configuration `uploadpack.excludeobject` for this.\n + Currently packfile-uri supports the exclusion of blob objects, but in\n + some scenarios, users may wish to exclude more types of objects, such as\n + commit and tree objects, not only because packfile itself supports\n + storing these object types, but also on the other hand, to make\n + configuration items maintainable and simpler.\n +\n + This commit is used to support the recursive exclusion of a commit\n + object, which means that if the exclusion of a commit is configured as\n + packfile-uri, the commit itself and all the objects it contains will\n + also be recursively excluded. In addition, to support this feature, a\n + new configuration `uploadpack.excludeobject` is introduced.\n \n The reason for bringing a new configuration is for two considerations.\n First, the old configuration supports a single object type (blob), which\n @@ Commit message\n \n Signed-off-by: Teng Long \n \n - ## builtin/describe.c ##\n -@@ builtin/describe.c: static void process_commit(struct commit *commit, void *data)\n - \tpcd->current_commit = commit->object.oid;\n - }\n - \n --static void process_object(struct object *obj, const char *path, void *data)\n -+static void process_object(struct object *obj, const char *path, void *show_data, void *carry_data)\n - {\n --\tstruct process_commit_data *pcd = data;\n -+\tstruct process_commit_data *pcd = show_data;\n - \n - \tif (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) {\n - \t\treset_revision_walk();\n -\n ## builtin/pack-objects.c ##\n -@@ builtin/pack-objects.c: static int have_duplicate_entry(const struct object_id *oid,\n - \treturn 1;\n - }\n - \n -+static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n -+{\n -+\tint i;\n -+\tconst char *p;\n -+\n -+\tif (ex) {\n -+\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n -+\t\t\tif (skip_prefix(ex->uri,\n -+\t\t\t\t\turi_protocols.items[i].string,\n -+\t\t\t\t\t&p) &&\n -+\t\t\t *p == ':')\n -+\t\t\t\treturn 1;\n -+\n -+\t\t}\n -+\t}\n -+\treturn 0;\n -+}\n -+\n - static int want_found_object(const struct object_id *oid, int exclude,\n - \t\t\t struct packed_git *p)\n - {\n @@ builtin/pack-objects.c: static int want_object_in_pack_one(struct packed_git *p,\n static int want_object_in_pack(const struct object_id *oid,\n \t\t\t int exclude,\n @@ builtin/pack-objects.c: static int want_object_in_pack_one(struct packed_git *p,\n \tstruct multi_pack_index *m;\n +\tstruct configured_exclusion *ex;\n +\tstruct configured_exclusion *referred_ex;\n -+\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n @@ builtin/pack-objects.c: static int want_object_in_pack(const struct object_id *o\n \tif (uri_protocols.nr) {\n -\t\tstruct configured_exclusion *ex =\n -\t\t\toidmap_get(&configured_exclusions, oid);\n --\t\tint i;\n --\t\tconst char *p;\n --\n --\t\tif (ex) {\n --\t\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n --\t\t\t\tif (skip_prefix(ex->uri,\n --\t\t\t\t\t\turi_protocols.items[i].string,\n --\t\t\t\t\t\t&p) &&\n --\t\t\t\t *p == ':') {\n --\t\t\t\t\toidset_insert(&excluded_by_config, oid);\n --\t\t\t\t\treturn 0;\n --\t\t\t\t}\n --\t\t\t}\n +\t\tif (referred_commit) {\n +\t\t\treferred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid);\n +\t\t\tif (referred_ex && match_packfile_uri_exclusions(referred_ex))\n +\t\t\t\treturn 0;\n +\t\t}\n -+\t\tex = oidmap_get(&configured_exclusions, oid);\n -+\t\tif (ex && match_packfile_uri_exclusions(ex)) {\n -+\t\t\toidset_insert(&excluded_by_config, oid);\n -+\t\t\treturn 0;\n - \t\t}\n - \t}\n \n ++\t\tex = oidmap_get(&configured_exclusions, oid);\n + \t\tif (ex && match_packfile_uri_exclusions(ex)) {\n + \t\t\toidset_insert(&excluded_by_config, oid);\n + \t\t\treturn 0;\n @@ builtin/pack-objects.c: static const char no_closure_warning[] = N_(\n );\n \n @@ builtin/pack-objects.c: static int git_pack_config(const char *k, const char *v,\n \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n \t\t *pack_end != ' ')\n -\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n --\t\t\t \"of the form ' ' (got '%s')\"), v);\n -+ die(_(\"value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be \"\n -+ \"of the form ' ' (got '%s')\"), v);\n ++\t\t\tdie(_(\"value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be \"\n + \t\t\t \"of the form ' ' (got '%s')\"), v);\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n -\t\t\tdie(_(\"object already configured in another \"\n -\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n -+ die(_(\"object already configured by an earlier \"\n -+ \"uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')\"), v);\n ++\t\t\tdie(_(\"object already configured by an earlier \"\n ++\t\t\t \"uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')\"), v);\n \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n \t\tex->uri = xstrdup(pack_end + 1);\n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n \t\treturn 0;\n \n \toi.typep = &type;\n -@@ builtin/pack-objects.c: static void show_commit_pack_hint(struct commit *commit, void *_data)\n - }\n - \n - static void show_object_pack_hint(struct object *object, const char *name,\n --\t\t\t\t void *_data)\n -+\t\t\t\t void *show_data, void *carry_data)\n - {\n - \tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n - \tif (!oe)\n @@ builtin/pack-objects.c: static void read_object_list_from_stdin(void)\n \t\t\tdie(_(\"expected object ID, got garbage:\\n %s\"), line);\n \n @@ builtin/pack-objects.c: static void read_object_list_from_stdin(void)\n \n \tif (write_bitmap_index)\n @@ builtin/pack-objects.c: static void show_commit(struct commit *commit, void *data)\n - \t\tpropagate_island_marks(commit);\n - }\n \n --static void show_object(struct object *obj, const char *name, void *data)\n -+static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n + static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n +\tstruct object *referred_commit = carry_data;\n \tadd_preferred_base_object(name);\n @@ builtin/pack-objects.c: static void show_commit(struct commit *commit, void *dat\n \tobj->flags |= OBJECT_ADDED;\n \n \tif (use_delta_islands) {\n -@@ builtin/pack-objects.c: static void show_object(struct object *obj, const char *name, void *data)\n - \t}\n - }\n - \n --static void show_object__ma_allow_any(struct object *obj, const char *name, void *data)\n -+static void show_object__ma_allow_any(struct object *obj, const char *name, void *show_data, void *carry_data)\n - {\n - \tassert(arg_missing_action == MA_ALLOW_ANY);\n - \n -@@ builtin/pack-objects.c: static void show_object__ma_allow_any(struct object *obj, const char *name, void\n - \tif (!has_object(the_repository, &obj->oid, 0))\n - \t\treturn;\n - \n --\tshow_object(obj, name, data);\n -+\tshow_object(obj, name, show_data, carry_data);\n - }\n - \n --static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data)\n -+static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *show_data, void *carry_data)\n - {\n - \tassert(arg_missing_action == MA_ALLOW_PROMISOR);\n - \n -@@ builtin/pack-objects.c: static void show_object__ma_allow_promisor(struct object *obj, const char *name,\n - \tif (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid))\n - \t\treturn;\n - \n --\tshow_object(obj, name, data);\n -+\tshow_object(obj, name, show_data, carry_data);\n - }\n - \n - static int option_parse_missing_action(const struct option *opt,\n @@ builtin/pack-objects.c: static void add_objects_in_unpacked_packs(void)\n \t\tQSORT(in_pack.array, in_pack.nr, ofscmp);\n \t\tfor (i = 0; i < in_pack.nr; i++) {\n @@ builtin/pack-objects.c: static int add_loose_object(const struct object_id *oid,\n \treturn 0;\n }\n \n -@@ builtin/pack-objects.c: static int get_object_list_from_bitmap(struct rev_info *revs)\n - \n - static void record_recent_object(struct object *obj,\n - \t\t\t\t const char *name,\n --\t\t\t\t void *data)\n -+\t\t\t\t void *show_data,\n -+\t\t\t\t void *carry_data)\n - {\n - \toid_array_append(&recent_objects, &obj->oid);\n - }\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"respect islands during delta compression\")),\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n -\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n +\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject or \"\n -+\t\t\t\t \t\"uploadpack.blobpackfileuri with this protocol\")),\n ++\t\t\t\t \"uploadpack.blobpackfileuri with this protocol\")),\n \t\tOPT_END(),\n \t};\n \n \n - ## builtin/rev-list.c ##\n -@@ builtin/rev-list.c: static int finish_object(struct object *obj, const char *name, void *cb_data)\n - \treturn 0;\n - }\n - \n --static void show_object(struct object *obj, const char *name, void *cb_data)\n -+static void show_object(struct object *obj, const char *name, void *cb_data, void *carry_data)\n - {\n - \tstruct rev_list_info *info = cb_data;\n - \tstruct rev_info *revs = info->revs;\n -\n - ## fetch-pack.c ##\n -@@\n - #include \"fetch-negotiator.h\"\n - #include \"fsck.h\"\n - #include \"shallow.h\"\n -+#include \"strmap.h\"\n - \n - static int transfer_unpack_limit = -1;\n - static int fetch_unpack_limit = -1;\n -@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n - \tstruct string_list packfile_uris = STRING_LIST_INIT_DUP;\n - \tint i;\n - \tstruct strvec index_pack_args = STRVEC_INIT;\n -+\tstruct strset uris;\n - \n - \tnegotiator = &negotiator_alloc;\n - \tfetch_negotiator_init(r, negotiator);\n -@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n - \t\t}\n - \t}\n - \n -+\tstrset_init(&uris);\n - \tfor (i = 0; i < packfile_uris.nr; i++) {\n - \t\tint j;\n - \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n -@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n - \t\tconst char *uri = packfile_uris.items[i].string +\n - \t\t\tthe_hash_algo->hexsz + 1;\n - \n -+\t\tif (!strset_add(&uris, uri))\n -+\t\t\tcontinue;\n - \t\tstrvec_push(&cmd.args, \"http-fetch\");\n - \t\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n - \t\t\t (int) the_hash_algo->hexsz,\n -@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n - \t\t\t\t\t\t get_object_directory(),\n - \t\t\t\t\t\t packname));\n - \t}\n -+\tstrset_clear(&uris);\n - \tstring_list_clear(&packfile_uris, 0);\n - \tstrvec_clear(&index_pack_args);\n - \n -\n ## list-objects.c ##\n @@ list-objects.c: struct traversal_context {\n static void process_blob(struct traversal_context *ctx,\n @@ list-objects.c: static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n --\t\tctx->show_object(obj, path->buf, ctx->show_data);\n +-\t\tctx->show_object(obj, path->buf, ctx->show_data, NULL);\n +\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n \tstrbuf_setlen(path, pathlen);\n }\n @@ list-objects.c: static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n --\t\tctx->show_object(obj, base->buf, ctx->show_data);\n +-\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n +\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n @@ list-objects.c: static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n --\t\tctx->show_object(obj, base->buf, ctx->show_data);\n +-\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n +\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \n \tstrbuf_setlen(base, baselen);\n @@ list-objects.c: void mark_edges_uninteresting(struct rev_info *revs,\n }\n \n -static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n -+static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit)\n ++static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit)\n {\n -\tadd_pending_object(revs, &tree->object, \"\");\n +\tadd_pending_object_with_referred_commit(revs, &tree->object, \"\", referred_commit);\n @@ list-objects.c: static void traverse_trees_and_blobs(struct traversal_context *c\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n --\t\t\tctx->show_object(obj, name, ctx->show_data);\n +-\t\t\tctx->show_object(obj, name, ctx->show_data, NULL);\n +\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n \t\t\tcontinue;\n \t\t}\n @@ list-objects.c: static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \n - ## list-objects.h ##\n -@@ list-objects.h: struct object;\n - struct rev_info;\n - \n - typedef void (*show_commit_fn)(struct commit *, void *);\n --typedef void (*show_object_fn)(struct object *, const char *, void *);\n -+typedef void (*show_object_fn)(struct object *, const char *, void *, void *);\n - void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n - \n - typedef void (*show_edge_fn)(struct commit *);\n -\n ## object.c ##\n @@ object.c: void object_list_free(struct object_list **list)\n */\n @@ object.c: void add_object_array_with_path(struct object *obj, const char *name,\n \n ## object.h ##\n @@ object.h: struct object_array {\n - \tunsigned int alloc;\n - \tstruct object_array_entry {\n - \t\tstruct object *item;\n + \t\tchar *name;\n + \t\tchar *path;\n + \t\tunsigned mode;\n +\t\tstruct object *referred_commit;\n - \t\t/*\n - \t\t * name or NULL. If non-NULL, the memory pointed to\n - \t\t * is owned by this object *except* if it points at\n + \t} *objects;\n + };\n + \n @@ object.h: void object_list_free(struct object_list **list);\n /* Object array handling .. */\n void add_object_array(struct object *obj, const char *name, struct object_array *array);\n void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path);\n +-\n +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array,\n -+\t\t\t\t\t\t unsigned mode, const char *path,\n -+\t\t\t\t\t\t struct object *referred_commit);\n - \n ++\t\t\t\t\t\t unsigned mode, const char *path, struct object *referred_commit);\n /*\n * Returns NULL if the array is empty. Otherwise, returns the last object\n -\n - ## pack-bitmap.c ##\n -@@ pack-bitmap.c: struct bitmap_show_data {\n - \tstruct bitmap *base;\n - };\n - \n --static void show_object(struct object *object, const char *name, void *data_)\n -+static void show_object(struct object *object, const char *name, void *show_data, void *carry_data)\n - {\n --\tstruct bitmap_show_data *data = data_;\n -+\tstruct bitmap_show_data *data = show_data;\n - \tint bitmap_pos;\n - \n - \tbitmap_pos = bitmap_position(data->bitmap_git, &object->oid);\n -@@ pack-bitmap.c: struct bitmap_test_data {\n - };\n - \n - static void test_show_object(struct object *object, const char *name,\n --\t\t\t void *data)\n -+\t\t\t void *show_data, void *carry_data)\n - {\n --\tstruct bitmap_test_data *tdata = data;\n -+\tstruct bitmap_test_data *tdata = show_data;\n - \tint bitmap_pos;\n - \n - \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n -\n - ## reachable.c ##\n -@@ reachable.c: static int add_one_ref(const char *path, const struct object_id *oid,\n - * The traversal will have already marked us as SEEN, so we\n - * only need to handle any progress reporting here.\n - */\n --static void mark_object(struct object *obj, const char *name, void *data)\n -+static void mark_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n - {\n --\tupdate_progress(data);\n -+\tupdate_progress(show_data);\n - }\n - \n --static void mark_commit(struct commit *c, void *data)\n -+static void mark_commit(struct commit *c, void *show_data)\n - {\n --\tmark_object(&c->object, NULL, data);\n -+\tmark_object(&c->object, NULL, show_data, NULL);\n - }\n - \n - struct recent_data {\n + * after removing its entry from the array. Other resources associated\n \n ## revision.c ##\n @@ revision.c: void mark_parents_uninteresting(struct commit *commit)\n @@ revision.c: static void add_pending_object_with_path(struct rev_info *revs,\n \t}\n -\tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n +\tadd_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit);\n -+}\n -+\n + }\n + \n +static void add_pending_object_with_path(struct rev_info *revs,\n +\t\t\t\t\t struct object *obj,\n +\t\t\t\t\t const char *name, unsigned mode,\n -+\t\t\t\t\t const char *path) {\n ++\t\t\t\t\t const char *path)\n ++{\n +\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL);\n - }\n - \n ++}\n static void add_pending_object_with_mode(struct rev_info *revs,\n \t\t\t\t\t struct object *obj,\n -\t\t\t\t\t const char *name, unsigned mode)\n +\t\t\t\t\t const char *name, unsigned mode,\n +\t\t\t\t\t struct object *referred_commit)\n - {\n --\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n -+\n ++{\n +\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit);\n +}\n +\n +void add_pending_object_with_referred_commit(struct rev_info *revs,\n +\t\t\t\t\t struct object *obj, const char *name,\n +\t\t\t\t\t struct object *referred_commit)\n -+{\n + {\n +-\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n +\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit);\n }\n \n @@ revision.c: static void add_pending_object_with_path(struct rev_info *revs,\n }\n \n void add_head_to_pending(struct rev_info *revs)\n -@@ revision.c: int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n - \t\t\tcontinue;\n - \t\t}\n - \n --\n - \t\tif (handle_revision_arg(arg, revs, flags, revarg_opt)) {\n - \t\t\tint j;\n - \t\t\tif (seen_dashdash || *arg == '^')\n @@ revision.c: int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tif (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc))\n \t\t\tdiagnose_missing_default(revs->def);\n @@ revision.c: int setup_revisions(int argc, const char **argv, struct rev_info *re\n \n ## revision.h ##\n @@ revision.h: void show_object_with_name(FILE *, struct object *, const char *);\n + */\n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name);\n - \n +void add_pending_object_with_referred_commit(struct rev_info *revs,\n +\t\t\t\t\t struct object *obj, const char *name,\n +\t\t\t\t\t struct object *referred_commit);\n -+\n + \n void add_pending_oid(struct rev_info *revs,\n \t\t const char *name, const struct object_id *oid,\n - \t\t unsigned int flags);\n \n ## upload-pack.c ##\n @@ upload-pack.c: int upload_pack_advertise(struct repository *r,\n2: 7ce99a0f31 < -: ---------- t5702: support for excluding commit objects\n-: ---------- > 4: bbb0413cc4 packfile-uri: support for excluding tree objects\n3: 36b9a92132 ! 5: 8e5bf4010c packfile-uri.txt: support for excluding commit objects\n @@ Metadata\n Author: Teng Long \n \n ## Commit message ##\n - packfile-uri.txt: support for excluding commit objects\n + packfile-uri.txt: support for excluding commits and trees\n \n Signed-off-by: Teng Long \n \n @@ Documentation/technical/packfile-uri.txt: include some sort of non-trivial imple\n -downloaded in this way only contain single blobs.\n +server to be configured by one or more entries with the format:\n +\n -+ uploadpack.excludeobject= \n ++ uploadpack.excludeobject= \n +\n -+Value is the key of entry, and the object type can be a blob\n -+or commit. Whenever the list of objects to be sent is assembled, all such\n -+objects are excluded, replaced with URIs. At the same time, for the old\n -+configuration `uploadpack.blobPackfileUri= ` is\n -+still compatible for now, but this configuration only supports the\n -+exclusion of blob objects.\n ++Value is the key of entry, and the object type can be a blob,\n ++tree, or commit. The exclusion of tree and commit is recursive by default,\n ++which means that when a tree or commit object is excluded, the object itself\n ++and all reachable objects of the object will be excluded recursively. Whenever\n ++the list of objects to be sent is assembled, all such objects are excluded,\n ++replaced with URIs.\n ++\n ++Configuration compatibility\n ++-------------\n ++\n ++The old configuration of packfile-uri:\n ++\n ++\t`uploadpack.blobPackfileUri= `\n ++\n ++For the old configuration is compatible with the new one, but it only\n ++supports the exclusion of blob objects.\n \n Client design\n -------------\n @@ Documentation/technical/packfile-uri.txt: The protocol design allows some evolut\n * On the client, resumption of clone. If a clone is interrupted, information\n could be recorded in the repository's config and a \"clone-resume\" command\n can resume the clone in progress. (Resumption of subsequent fetches is more\n +@@ Documentation/technical/packfile-uri.txt: There are some possible features that will require a change in protocol:\n + \n + * Additional HTTP headers (e.g. authentication)\n + * Byte range support\n +- * Different file formats referenced by URIs (e.g. raw object)\n ++ * Different file formats referenced by URIs (e.g. raw object)\n + \\ No newline at end of file\n-: ---------- > 6: f3b1cba7e1 t5702: replace with \"test_when_finished\" for cleanup\n-: ---------- > 7: 3b5f9732b8 t5702: support for excluding commit objects\n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432457","messageId":"73a5b4ccc11b8b0604dc3414129069ab41be45d0.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v4 1/7] pack-objects.c: introduce new method `match_packfile_uri_exclusions`","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:48Z","receivedAt":"2021-08-11T07:46:22Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n builtin/pack-objects.c | 33 +++++++++++++++++++++------------\n 1 file changed, 21 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6d13cd3e1a..31556e7396 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1188,6 +1188,24 @@ static int have_duplicate_entry(const struct object_id *oid,\n \treturn 1;\n }\n \n+static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n+{\n+\tint i;\n+\tconst char *p;\n+\n+\tif (ex) {\n+\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n+\t\t\tif (skip_prefix(ex->uri,\n+\t\t\t\t\turi_protocols.items[i].string,\n+\t\t\t\t\t&p) &&\n+\t\t\t *p == ':')\n+\t\t\t\treturn 1;\n+\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n static int want_found_object(const struct object_id *oid, int exclude,\n \t\t\t struct packed_git *p)\n {\n@@ -1335,19 +1353,10 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (uri_protocols.nr) {\n \t\tstruct configured_exclusion *ex =\n \t\t\toidmap_get(&configured_exclusions, oid);\n-\t\tint i;\n-\t\tconst char *p;\n \n-\t\tif (ex) {\n-\t\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n-\t\t\t\tif (skip_prefix(ex->uri,\n-\t\t\t\t\t\turi_protocols.items[i].string,\n-\t\t\t\t\t\t&p) &&\n-\t\t\t\t *p == ':') {\n-\t\t\t\t\toidset_insert(&excluded_by_config, oid);\n-\t\t\t\t\treturn 0;\n-\t\t\t\t}\n-\t\t\t}\n+\t\tif (ex && match_packfile_uri_exclusions(ex)) {\n+\t\t\toidset_insert(&excluded_by_config, oid);\n+\t\t\treturn 0;\n \t\t}\n \t}\n \n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432458","messageId":"bc8fea97e34b3ccb4b62bb4d48dac6e1c009236c.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v4 2/7] Add new parameter \"carry_data\" for \"show_object\" function","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:49Z","receivedAt":"2021-08-11T07:46:26Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"During the pack-objects process, \"show_object\" function will be called\nto find the object and show the process(\"show_object_fn\" in\n\"list-object.h\"), the function definition contains three parameters:\n\n\t1. struct object *obj(contains object type, flags, and oid).\n\t2. const char *name(the object name).\n\t3. void *show_data(function to show progress info).\n\nThis commit adds a new parameter: \"void *carry_data\", the reason is\nmainly based on scalability and performance considerations when showing\nan object, space for time, avoid costly temporary calculations in the\n\"show\" phase. For example, carry the ownership relationship between\nblob or tree object and the referred commit to avoid redundant and\nexpensive calculations.\n\nSigned-off-by: Teng Long \n---\n builtin/describe.c | 4 ++--\n builtin/pack-objects.c | 15 ++++++++-------\n builtin/rev-list.c | 2 +-\n list-objects.c | 8 ++++----\n list-objects.h | 2 +-\n pack-bitmap.c | 8 ++++----\n reachable.c | 8 ++++----\n 7 files changed, 24 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/describe.c b/builtin/describe.c\nindex 40482d8e9f..045da79b5c 100644\n--- a/builtin/describe.c\n+++ b/builtin/describe.c\n@@ -485,9 +485,9 @@ static void process_commit(struct commit *commit, void *data)\n \tpcd->current_commit = commit->object.oid;\n }\n \n-static void process_object(struct object *obj, const char *path, void *data)\n+static void process_object(struct object *obj, const char *path, void *show_data, void *carry_data)\n {\n-\tstruct process_commit_data *pcd = data;\n+\tstruct process_commit_data *pcd = show_data;\n \n \tif (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) {\n \t\treset_revision_walk();\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 31556e7396..5f9ec3566f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,7 +3068,7 @@ static void show_commit_pack_hint(struct commit *commit, void *_data)\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *_data)\n+\t\t\t\t void *show_data, void *carry_dataa)\n {\n \tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n \tif (!oe)\n@@ -3252,7 +3252,7 @@ static void show_commit(struct commit *commit, void *data)\n \t\tpropagate_island_marks(commit);\n }\n \n-static void show_object(struct object *obj, const char *name, void *data)\n+static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tadd_preferred_base_object(name);\n \tadd_object_entry(&obj->oid, obj->type, name, 0);\n@@ -3274,7 +3274,7 @@ static void show_object(struct object *obj, const char *name, void *data)\n \t}\n }\n \n-static void show_object__ma_allow_any(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_any(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tassert(arg_missing_action == MA_ALLOW_ANY);\n \n@@ -3285,10 +3285,10 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void\n \tif (!has_object(the_repository, &obj->oid, 0))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, show_data, carry_data);\n }\n \n-static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tassert(arg_missing_action == MA_ALLOW_PROMISOR);\n \n@@ -3299,7 +3299,7 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,\n \tif (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, show_data, carry_data);\n }\n \n static int option_parse_missing_action(const struct option *opt,\n@@ -3547,7 +3547,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \n static void record_recent_object(struct object *obj,\n \t\t\t\t const char *name,\n-\t\t\t\t void *data)\n+\t\t\t\t void *show_data,\n+\t\t\t\t void *carry_data)\n {\n \toid_array_append(&recent_objects, &obj->oid);\n }\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..1cad33d9e8 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -266,7 +266,7 @@ static int finish_object(struct object *obj, const char *name, void *cb_data)\n \treturn 0;\n }\n \n-static void show_object(struct object *obj, const char *name, void *cb_data)\n+static void show_object(struct object *obj, const char *name, void *cb_data, void *carry_data)\n {\n \tstruct rev_list_info *info = cb_data;\n \tstruct rev_info *revs = info->revs;\ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..427228a3ba 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -60,7 +60,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data, NULL);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -191,7 +191,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -207,7 +207,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -335,7 +335,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data, NULL);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e46..ab946d34db 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -6,7 +6,7 @@ struct object;\n struct rev_info;\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n-typedef void (*show_object_fn)(struct object *, const char *, void *);\n+typedef void (*show_object_fn)(struct object *, const char *, void *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3ed15431cd..516eb235da 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -459,9 +459,9 @@ struct bitmap_show_data {\n \tstruct bitmap *base;\n };\n \n-static void show_object(struct object *object, const char *name, void *data_)\n+static void show_object(struct object *object, const char *name, void *show_data, void *carry_data)\n {\n-\tstruct bitmap_show_data *data = data_;\n+\tstruct bitmap_show_data *data = show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(data->bitmap_git, &object->oid);\n@@ -1268,9 +1268,9 @@ struct bitmap_test_data {\n };\n \n static void test_show_object(struct object *object, const char *name,\n-\t\t\t void *data)\n+\t\t\t void *show_data, void *carry_data)\n {\n-\tstruct bitmap_test_data *tdata = data;\n+\tstruct bitmap_test_data *tdata = show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..521b39edef 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -47,14 +47,14 @@ static int add_one_ref(const char *path, const struct object_id *oid,\n * The traversal will have already marked us as SEEN, so we\n * only need to handle any progress reporting here.\n */\n-static void mark_object(struct object *obj, const char *name, void *data)\n+static void mark_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n-\tupdate_progress(data);\n+\tupdate_progress(show_data);\n }\n \n-static void mark_commit(struct commit *c, void *data)\n+static void mark_commit(struct commit *c, void *show_data)\n {\n-\tmark_object(&c->object, NULL, data);\n+\tmark_object(&c->object, NULL, show_data, NULL);\n }\n \n struct recent_data {\n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432459","messageId":"f71b3108428b6ba36667e5e8a311ab8ab973d10c.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v4 3/7] packfile-uri: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:50Z","receivedAt":"2021-08-11T07:46:28Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Currently packfile-uri supports the exclusion of blob objects, but in\nsome scenarios, users may wish to exclude more types of objects, such as\ncommit and tree objects, not only because packfile itself supports\nstoring these object types, but also on the other hand, to make\nconfiguration items maintainable and simpler.\n\nThis commit is used to support the recursive exclusion of a commit\nobject, which means that if the exclusion of a commit is configured as\npackfile-uri, the commit itself and all the objects it contains will\nalso be recursively excluded. In addition, to support this feature, a\nnew configuration `uploadpack.excludeobject` is introduced.\n\nThe reason for bringing a new configuration is for two considerations.\nFirst, the old configuration supports a single object type (blob), which\nlimits the use of this feature. Secondly, the name of the old\nconfiguration is not abstract enough, this make extension difficult. If\ndifferent object types use different configuration names, the\nconfiguration items will be bloated and difficult to maintain, so the\nnew configuration is more abstract in name and easy to extend.\n\nAlthough a new configuration has been introduced, the old one is\nstill available and compatible with the new configuration. The old\nconfiguration `uploadpack.blobpackfileuri` only supports excluding\nblobs. The new configuration `uploadpack.excludeobject` not only\nsupports excluding blob objects, but also supports excluding commit\nobjects, as well as recursively excluding tree objects and blob objects\nthey contain.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 50 +++++++++++++++++++++++++-----------------\n list-objects.c | 37 +++++++++++++++++--------------\n object.c | 15 ++++++++++---\n object.h | 4 +++-\n revision.c | 34 ++++++++++++++++++++--------\n revision.h | 3 +++\n upload-pack.c | 7 ++++++\n 7 files changed, 101 insertions(+), 49 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5f9ec3566f..63f3aed70a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1311,11 +1311,14 @@ static int want_object_in_pack_one(struct packed_git *p,\n static int want_object_in_pack(const struct object_id *oid,\n \t\t\t int exclude,\n \t\t\t struct packed_git **found_pack,\n-\t\t\t off_t *found_offset)\n+\t\t\t off_t *found_offset,\n+\t\t\t struct object *referred_commit)\n {\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n+\tstruct configured_exclusion *ex;\n+\tstruct configured_exclusion *referred_ex;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1351,9 +1354,13 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n-\t\tstruct configured_exclusion *ex =\n-\t\t\toidmap_get(&configured_exclusions, oid);\n+\t\tif (referred_commit) {\n+\t\t\treferred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid);\n+\t\t\tif (referred_ex && match_packfile_uri_exclusions(referred_ex))\n+\t\t\t\treturn 0;\n+\t\t}\n \n+\t\tex = oidmap_get(&configured_exclusions, oid);\n \t\tif (ex && match_packfile_uri_exclusions(ex)) {\n \t\t\toidset_insert(&excluded_by_config, oid);\n \t\t\treturn 0;\n@@ -1393,7 +1400,8 @@ static const char no_closure_warning[] = N_(\n );\n \n static int add_object_entry(const struct object_id *oid, enum object_type type,\n-\t\t\t const char *name, int exclude)\n+\t\t\t const char *name, int exclude,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct packed_git *found_pack = NULL;\n \toff_t found_offset = 0;\n@@ -1403,7 +1411,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tif (have_duplicate_entry(oid, exclude))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) {\n+\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) {\n \t\t/* The pack is missing an object, so it will not have closure */\n \t\tif (write_bitmap_index) {\n \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n@@ -1429,7 +1437,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \tif (have_duplicate_entry(oid, 0))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, 0, &pack, &offset))\n+\tif (!want_object_in_pack(oid, 0, &pack, &offset, NULL))\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, pack, offset);\n@@ -1569,7 +1577,7 @@ static void add_pbase_object(struct tree_desc *tree,\n \t\tif (name[cmplen] != '/') {\n \t\t\tadd_object_entry(&entry.oid,\n \t\t\t\t\t object_type(entry.mode),\n-\t\t\t\t\t fullname, 1);\n+\t\t\t\t\t fullname, 1, NULL);\n \t\t\treturn;\n \t\t}\n \t\tif (S_ISDIR(entry.mode)) {\n@@ -1637,7 +1645,7 @@ static void add_preferred_base_object(const char *name)\n \tcmplen = name_cmp_len(name);\n \tfor (it = pbase_tree; it; it = it->next) {\n \t\tif (cmplen == 0) {\n-\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1);\n+\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL);\n \t\t}\n \t\telse {\n \t\t\tstruct tree_desc tree;\n@@ -2839,7 +2847,7 @@ static void add_tag_chain(const struct object_id *oid)\n \t\t\tdie(_(\"unable to pack objects reachable from tag %s\"),\n \t\t\t oid_to_hex(oid));\n \n-\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0);\n+\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL);\n \n \t\tif (tag->tagged->type != OBJ_TAG)\n \t\t\treturn;\n@@ -2994,7 +3002,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n+\tif (!strcmp(k, \"uploadpack.excludeobject\") || !strcmp(k, \"uploadpack.blobpackfileuri\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n \t\tconst char *oid_end, *pack_end;\n \t\t/*\n@@ -3007,11 +3015,11 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t *oid_end != ' ' ||\n \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n \t\t *pack_end != ' ')\n-\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n+\t\t\tdie(_(\"value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be \"\n \t\t\t \"of the form ' ' (got '%s')\"), v);\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n-\t\t\tdie(_(\"object already configured in another \"\n-\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n+\t\t\tdie(_(\"object already configured by an earlier \"\n+\t\t\t \"uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')\"), v);\n \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n \t\tex->uri = xstrdup(pack_end + 1);\n@@ -3040,7 +3048,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tofs = nth_packed_object_offset(p, pos);\n-\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n+\tif (!want_object_in_pack(oid, 0, &p, &ofs, NULL))\n \t\treturn 0;\n \n \toi.typep = &type;\n@@ -3233,7 +3241,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(_(\"expected object ID, got garbage:\\n %s\"), line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL);\n \t}\n }\n \n@@ -3242,7 +3250,7 @@ static void read_object_list_from_stdin(void)\n \n static void show_commit(struct commit *commit, void *data)\n {\n-\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0);\n+\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n \tcommit->object.flags |= OBJECT_ADDED;\n \n \tif (write_bitmap_index)\n@@ -3254,8 +3262,9 @@ static void show_commit(struct commit *commit, void *data)\n \n static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n+\tstruct object *referred_commit = carry_data;\n \tadd_preferred_base_object(name);\n-\tadd_object_entry(&obj->oid, obj->type, name, 0);\n+\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_commit);\n \tobj->flags |= OBJECT_ADDED;\n \n \tif (use_delta_islands) {\n@@ -3406,7 +3415,7 @@ static void add_objects_in_unpacked_packs(void)\n \t\tQSORT(in_pack.array, in_pack.nr, ofscmp);\n \t\tfor (i = 0; i < in_pack.nr; i++) {\n \t\t\tstruct object *o = in_pack.array[i].object;\n-\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0);\n+\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0, NULL);\n \t\t}\n \t}\n \tfree(in_pack.array);\n@@ -3422,7 +3431,7 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t\treturn 0;\n \t}\n \n-\tadd_object_entry(oid, type, \"\", 0);\n+\tadd_object_entry(oid, type, \"\", 0, NULL);\n \treturn 0;\n }\n \n@@ -3841,7 +3850,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"respect islands during delta compression\")),\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n-\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject or \"\n+\t\t\t\t \"uploadpack.blobpackfileuri with this protocol\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/list-objects.c b/list-objects.c\nindex 427228a3ba..968d842ceb 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -24,7 +24,8 @@ struct traversal_context {\n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n@@ -60,7 +61,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data, NULL);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -97,11 +98,13 @@ static void process_gitlink(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name);\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit);\n \n static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t struct tree *tree,\n-\t\t\t\t struct strbuf *base)\n+\t\t\t\t struct strbuf *base,\n+\t\t\t\t struct object *referred_commit)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -129,7 +132,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t\tprocess_tree(ctx, t, base, entry.path, referred_commit);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n@@ -142,7 +145,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t\tprocess_blob(ctx, b, base, entry.path, referred_commit);\n \t\t}\n \t}\n }\n@@ -150,7 +153,8 @@ static void process_tree_contents(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n@@ -191,14 +195,14 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n-\t\tprocess_tree_contents(ctx, tree, base);\n+\t\tprocess_tree_contents(ctx, tree, base, referred_commit);\n \n \tr = list_objects_filter__filter_object(ctx->revs->repo,\n \t\t\t\t\t LOFS_END_TREE, obj,\n@@ -207,7 +211,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -314,9 +318,9 @@ void mark_edges_uninteresting(struct rev_info *revs,\n \t}\n }\n \n-static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n+static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit)\n {\n-\tadd_pending_object(revs, &tree->object, \"\");\n+\tadd_pending_object_with_referred_commit(revs, &tree->object, \"\", referred_commit);\n }\n \n static void traverse_trees_and_blobs(struct traversal_context *ctx,\n@@ -329,23 +333,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n+\t\tstruct object *referred_commit = pending->referred_commit;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data, NULL);\n+\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n@@ -370,7 +375,7 @@ static void do_traverse(struct traversal_context *ctx)\n \t\telse if (get_commit_tree(commit)) {\n \t\t\tstruct tree *tree = get_commit_tree(commit);\n \t\t\ttree->object.flags |= NOT_USER_GIVEN;\n-\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t\tadd_pending_tree(ctx->revs, tree, &commit->object);\n \t\t} else if (commit->object.parsed) {\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\ndiff --git a/object.c b/object.c\nindex 14188453c5..6b1ce2fcde 100644\n--- a/object.c\n+++ b/object.c\n@@ -322,9 +322,10 @@ void object_list_free(struct object_list **list)\n */\n static char object_array_slopbuf[1];\n \n-void add_object_array_with_path(struct object *obj, const char *name,\n-\t\t\t\tstruct object_array *array,\n-\t\t\t\tunsigned mode, const char *path)\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name,\n+\t\t\t\t\t\t struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path,\n+\t\t\t\t\t\t struct object *referred_commit)\n {\n \tunsigned nr = array->nr;\n \tunsigned alloc = array->alloc;\n@@ -339,6 +340,7 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \t}\n \tentry = &objects[nr];\n \tentry->item = obj;\n+\tentry->referred_commit = referred_commit;\n \tif (!name)\n \t\tentry->name = NULL;\n \telse if (!*name)\n@@ -354,6 +356,13 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \tarray->nr = ++nr;\n }\n \n+void add_object_array_with_path(struct object *obj, const char *name,\n+\t\t\t\tstruct object_array *array,\n+\t\t\t\tunsigned mode, const char *path)\n+{\n+\tadd_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL);\n+}\n+\n void add_object_array(struct object *obj, const char *name, struct object_array *array)\n {\n \tadd_object_array_with_path(obj, name, array, S_IFINVALID, NULL);\ndiff --git a/object.h b/object.h\nindex 87a6da47c8..d63819ab91 100644\n--- a/object.h\n+++ b/object.h\n@@ -52,6 +52,7 @@ struct object_array {\n \t\tchar *name;\n \t\tchar *path;\n \t\tunsigned mode;\n+\t\tstruct object *referred_commit;\n \t} *objects;\n };\n \n@@ -157,7 +158,8 @@ void object_list_free(struct object_list **list);\n /* Object array handling .. */\n void add_object_array(struct object *obj, const char *name, struct object_array *array);\n void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path);\n-\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path, struct object *referred_commit);\n /*\n * Returns NULL if the array is empty. Otherwise, returns the last object\n * after removing its entry from the array. Other resources associated\ndiff --git a/revision.c b/revision.c\nindex 4853c85d0b..65e0926d25 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit)\n \tcommit_stack_clear(&pending);\n }\n \n-static void add_pending_object_with_path(struct rev_info *revs,\n-\t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode,\n-\t\t\t\t\t const char *path)\n+static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t\t struct object *referred_commit)\n {\n \tstruct interpret_branch_name_options options = { 0 };\n \tif (!obj)\n@@ -326,20 +327,35 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n+\tadd_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit);\n }\n \n+static void add_pending_object_with_path(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t const char *path)\n+{\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL);\n+}\n static void add_pending_object_with_mode(struct rev_info *revs,\n \t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode)\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t struct object *referred_commit)\n+{\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit);\n+}\n+\n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct object *referred_commit)\n {\n-\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit);\n }\n \n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name)\n {\n-\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID);\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL);\n }\n \n void add_head_to_pending(struct rev_info *revs)\n@@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tif (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc))\n \t\t\tdiagnose_missing_default(revs->def);\n \t\tobject = get_reference(revs, revs->def, &oid, 0);\n-\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode);\n+\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL);\n \t}\n \n \t/* Did the user ask for any diff output? Run the diff! */\ndiff --git a/revision.h b/revision.h\nindex a24f72dcd1..f9c9628ed8 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -423,6 +423,9 @@ void show_object_with_name(FILE *, struct object *, const char *);\n */\n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name);\n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct object *referred_commit);\n \n void add_pending_oid(struct rev_info *revs,\n \t\t const char *name, const struct object_id *oid,\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5c1cd19612..d26fb351a3 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r,\n \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n \t\t\tfree(str);\n \t\t}\n+\n+\t\tif (!repo_config_get_string(the_repository,\n+\t\t\t\t\t \"uploadpack.excludeobject\",\n+\t\t\t\t\t &str) && str) {\n+\t\t\tstrbuf_addstr(value, \" packfile-uris\");\n+\t\t\tfree(str);\n+\t\t}\n \t}\n \n \treturn 1;\n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432460","messageId":"bbb0413cc41811d5a8cd6025fc546657e88e48c6.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v4 4/7] packfile-uri: support for excluding tree objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:51Z","receivedAt":"2021-08-11T07:46:35Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"This commit supports the use of `uploadpack.excludeobject` to exclude\ntree objects, which means that when a type object is configured as\npackfile-uri, the tree object itself and all objects contains will be\nrecursively excluded.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 30 ++++++++++++++++++++----------\n list-objects.c | 32 ++++++++++++++++++--------------\n object.c | 6 +++++-\n object.h | 13 ++++++++++++-\n 4 files changed, 55 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 63f3aed70a..4ff12ec525 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1312,13 +1312,14 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\t int exclude,\n \t\t\t struct packed_git **found_pack,\n \t\t\t off_t *found_offset,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n+\tstruct configured_exclusion *commit_ex;\n+\tstruct configured_exclusion *tree_ex;\n \tstruct configured_exclusion *ex;\n-\tstruct configured_exclusion *referred_ex;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1354,14 +1355,23 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n-\t\tif (referred_commit) {\n-\t\t\treferred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid);\n-\t\t\tif (referred_ex && match_packfile_uri_exclusions(referred_ex))\n+\t\tif (referred_objs && referred_objs->commit) {\n+\t\t\tcommit_ex = oidmap_get(&configured_exclusions, &referred_objs->commit->oid);\n+\t\t\tif (match_packfile_uri_exclusions(commit_ex))\n \t\t\t\treturn 0;\n \t\t}\n \n+\t\tif (referred_objs && referred_objs->trees) {\n+\t\t\tstruct object_list *p;\n+\t\t\tfor (p = referred_objs->trees; p; p = p->next) {\n+\t\t\t\ttree_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n+\t\t\t\tif (match_packfile_uri_exclusions(tree_ex))\n+\t\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t}\n+\n \t\tex = oidmap_get(&configured_exclusions, oid);\n-\t\tif (ex && match_packfile_uri_exclusions(ex)) {\n+\t\tif (match_packfile_uri_exclusions(ex)) {\n \t\t\toidset_insert(&excluded_by_config, oid);\n \t\t\treturn 0;\n \t\t}\n@@ -1401,7 +1411,7 @@ static const char no_closure_warning[] = N_(\n \n static int add_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t const char *name, int exclude,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct packed_git *found_pack = NULL;\n \toff_t found_offset = 0;\n@@ -1411,7 +1421,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tif (have_duplicate_entry(oid, exclude))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) {\n+\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_objs)) {\n \t\t/* The pack is missing an object, so it will not have closure */\n \t\tif (write_bitmap_index) {\n \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n@@ -3262,9 +3272,9 @@ static void show_commit(struct commit *commit, void *data)\n \n static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n-\tstruct object *referred_commit = carry_data;\n+\tstruct referred_objects *referred_objs = carry_data;\n \tadd_preferred_base_object(name);\n-\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_commit);\n+\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_objs);\n \tobj->flags |= OBJECT_ADDED;\n \n \tif (use_delta_islands) {\ndiff --git a/list-objects.c b/list-objects.c\nindex 968d842ceb..49f177cb56 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -25,7 +25,7 @@ static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n@@ -61,7 +61,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_objs);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -99,19 +99,22 @@ static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name,\n-\t\t\t struct object *referred_commit);\n+\t\t\t struct referred_objects *referred_objs);\n \n static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t struct tree *tree,\n \t\t\t\t struct strbuf *base,\n-\t\t\t\t struct object *referred_commit)\n+\t\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n \t\tall_entries_interesting : entry_not_interesting;\n+\tstruct referred_objects *referred_buf;\n \n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\treferred_buf = xmemdupz(referred_objs, sizeof(struct referred_objects));\n+\tobject_list_insert(&tree->object, &referred_buf->trees);\n \n \twhile (tree_entry(&desc, &entry)) {\n \t\tif (match != all_entries_interesting) {\n@@ -132,7 +135,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path, referred_commit);\n+\t\t\tprocess_tree(ctx, t, base, entry.path, referred_buf);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n@@ -145,16 +148,17 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path, referred_commit);\n+\t\t\tprocess_blob(ctx, b, base, entry.path, referred_buf);\n \t\t}\n \t}\n+\tfree(referred_buf);\n }\n \n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n@@ -195,14 +199,14 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_objs);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n-\t\tprocess_tree_contents(ctx, tree, base, referred_commit);\n+\t\tprocess_tree_contents(ctx, tree, base, referred_objs);\n \n \tr = list_objects_filter__filter_object(ctx->revs->repo,\n \t\t\t\t\t LOFS_END_TREE, obj,\n@@ -211,7 +215,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_objs);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -333,24 +337,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n-\t\tstruct object *referred_commit = pending->referred_commit;\n+\t\tstruct referred_objects *referred_objs = pending->referred_objects;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n+\t\t\tctx->show_object(obj, name, ctx->show_data, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_commit);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_commit);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\ndiff --git a/object.c b/object.c\nindex 6b1ce2fcde..69ba0baf95 100644\n--- a/object.c\n+++ b/object.c\n@@ -331,6 +331,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch\n \tunsigned alloc = array->alloc;\n \tstruct object_array_entry *objects = array->objects;\n \tstruct object_array_entry *entry;\n+\tstruct referred_objects *referred_objs = xmalloc(sizeof(struct referred_objects));\n \n \tif (nr >= alloc) {\n \t\talloc = (alloc + 32) * 2;\n@@ -338,9 +339,11 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch\n \t\tarray->alloc = alloc;\n \t\tarray->objects = objects;\n \t}\n+\treferred_objs->commit = referred_commit;\n+\treferred_objs->trees = NULL;\n \tentry = &objects[nr];\n \tentry->item = obj;\n-\tentry->referred_commit = referred_commit;\n+\tentry->referred_objects = referred_objs;\n \tif (!name)\n \t\tentry->name = NULL;\n \telse if (!*name)\n@@ -377,6 +380,7 @@ static void object_array_release_entry(struct object_array_entry *ent)\n \tif (ent->name != object_array_slopbuf)\n \t\tfree(ent->name);\n \tfree(ent->path);\n+\tfree(ent->referred_objects);\n }\n \n struct object *object_array_pop(struct object_array *array)\ndiff --git a/object.h b/object.h\nindex d63819ab91..3785546adf 100644\n--- a/object.h\n+++ b/object.h\n@@ -52,12 +52,23 @@ struct object_array {\n \t\tchar *name;\n \t\tchar *path;\n \t\tunsigned mode;\n-\t\tstruct object *referred_commit;\n+\t\t /*\n+\t\t * referred_objects or NULL. If non-NULL, it will\n+\t\t * temporary storage the referred commit and trees when\n+\t\t * traversing the specified object. Space for time,\n+\t\t * reduce related computing costs (such as packfile-uri\n+\t\t * exclusion), clean up when the traversal is over.\n+\t\t */\n+\t\tstruct referred_objects *referred_objects;\n \t} *objects;\n };\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n+struct referred_objects{\n+ struct object *commit;\n+ struct object_list *trees;\n+};\n /*\n * object flag allocation:\n * revision.h: 0---------10 15 23------26\n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432461","messageId":"8e5bf4010cb0d7e8b4f6c7285fe1f5365af955e8.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v4 5/7] packfile-uri.txt: support for excluding commits and trees","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:52Z","receivedAt":"2021-08-11T07:46:38Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n Documentation/technical/packfile-uri.txt | 32 ++++++++++++++++--------\n 1 file changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt\nindex f7eabc6c76..c3e4873956 100644\n--- a/Documentation/technical/packfile-uri.txt\n+++ b/Documentation/technical/packfile-uri.txt\n@@ -35,13 +35,26 @@ include some sort of non-trivial implementation in the Minimum Viable Product,\n at least so that we can test the client.\n \n This is the implementation: a feature, marked experimental, that allows the\n-server to be configured by one or more `uploadpack.blobPackfileUri=\n-` entries. Whenever the list of objects to be sent is assembled, all such\n-blobs are excluded, replaced with URIs. As noted in \"Future work\" below, the\n-server can evolve in the future to support excluding other objects (or other\n-implementations of servers could be made that support excluding other objects)\n-without needing a protocol change, so clients should not expect that packfiles\n-downloaded in this way only contain single blobs.\n+server to be configured by one or more entries with the format:\n+\n+ uploadpack.excludeobject= \n+\n+Value is the key of entry, and the object type can be a blob,\n+tree, or commit. The exclusion of tree and commit is recursive by default,\n+which means that when a tree or commit object is excluded, the object itself\n+and all reachable objects of the object will be excluded recursively. Whenever\n+the list of objects to be sent is assembled, all such objects are excluded,\n+replaced with URIs.\n+\n+Configuration compatibility\n+-------------\n+\n+The old configuration of packfile-uri:\n+\n+\t`uploadpack.blobPackfileUri= `\n+\n+For the old configuration is compatible with the new one, but it only\n+supports the exclusion of blob objects.\n \n Client design\n -------------\n@@ -65,9 +78,6 @@ The protocol design allows some evolution of the server and client without any\n need for protocol changes, so only a small-scoped design is included here to\n form the MVP. For example, the following can be done:\n \n- * On the server, more sophisticated means of excluding objects (e.g. by\n- specifying a commit to represent that commit and all objects that it\n- references).\n * On the client, resumption of clone. If a clone is interrupted, information\n could be recorded in the repository's config and a \"clone-resume\" command\n can resume the clone in progress. (Resumption of subsequent fetches is more\n@@ -78,4 +88,4 @@ There are some possible features that will require a change in protocol:\n \n * Additional HTTP headers (e.g. authentication)\n * Byte range support\n- * Different file formats referenced by URIs (e.g. raw object)\n+ * Different file formats referenced by URIs (e.g. raw object)\n\\ No newline at end of file\n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432462","messageId":"f3b1cba7e102b3132f04b27be48e532da7d448c9.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v4 6/7] t5702: replace with \"test_when_finished\" for cleanup","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:53Z","receivedAt":"2021-08-11T07:46:42Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 2e1243ca40..e6314b53b0 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -753,7 +753,7 @@ test_expect_success 'ls-remote with v2 http sends only one POST' '\n '\n \n test_expect_success 'push with http:// and a config of v2 does not request v2' '\n-\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \t# Till v2 for push is designed, make sure that if a client has\n \t# protocol.version configured to use v2, that the client instead falls\n \t# back and uses v0.\n@@ -776,7 +776,7 @@ test_expect_success 'push with http:// and a config of v2 does not request v2' '\n '\n \n test_expect_success 'when server sends \"ready\", expect DELIM' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -796,7 +796,7 @@ test_expect_success 'when server sends \"ready\", expect DELIM' '\n '\n \n test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -834,7 +834,7 @@ configure_exclusion () {\n \n test_expect_success 'part of packfile response provided as URI' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -883,7 +883,7 @@ test_expect_success 'part of packfile response provided as URI' '\n \n test_expect_success 'packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -904,7 +904,7 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -935,7 +935,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -959,7 +959,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -989,7 +989,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1015,7 +1015,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child err &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child err\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1038,4 +1038,4 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-test_done\n+test_done\n\\ No newline at end of file\n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432463","messageId":"3b5f9732b814a712094091027c8229988e2ba791.1628666093.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v4 7/7] t5702: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-11T07:45:54Z","receivedAt":"2021-08-11T07:46:46Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 292 +++++++++++++++++++++++++++++++++++++----\n 1 file changed, 266 insertions(+), 26 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex e6314b53b0..5ad52e0cee 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -824,17 +824,47 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n '\n \n configure_exclusion () {\n-\tgit -C \"$1\" hash-object \"$2\" >objh &&\n-\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n-\tgit -C \"$1\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n-\tcat objh\n+\tobjt=\"$1\"\n+\tP=\"$2\"\n+\toid=\"$3\"\n+\tversion=\"$4\"\n+\n+\toldc=\"uploadpack.blobpackfileuri\"\n+\tnewc=\"uploadpack.excludeobject\"\n+\tconfigkey=\"\"\n+\n+\tif test \"$version\" = \"old\"\n+\tthen\n+\t\tconfigkey=\"$oldc\"\n+\telse\n+\t\tconfigkey=\"$newc\"\n+\tfi\n+\n+\tif test \"$objt\" = \"blob\"\n+\tthen\n+\t\tgit -C \"$P\" hash-object \"$oid\" >objh &&\n+\t\tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n+\t\tgit -C \"$P\" config --add \\\n+\t\t\t\"$configkey\" \\\n+\t\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+\t\tcat objh\n+\telif test \"$objt\" = \"commit\" || test \"$objt\" = \"tree\" || test \"$objt\" = \"tag\"\n+\tthen\n+\t\techo \"$oid\" >objh\n+\t\tgit -C \"$P\" pack-objects --revs \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n+\t\tgit -C \"$P\" config --add \\\n+ \t\t\t\"$configkey\" \\\n+ \t\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+\t\tcat objh\n+\telse\n+\t\techo \"unsupported object type in configure_exclusion (got $objt)\"\n+\tfi\n }\n \n-test_expect_success 'part of packfile response provided as URI' '\n+part_of_packfile_response_verify() {\n+\tconfig=\"$1\"\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -843,10 +873,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\tconfigure_exclusion \"$P\" other-blob >h2 &&\n+\tconfigure_exclusion blob \"$P\" my-blob config >h &&\n+\tconfigure_exclusion blob \"$P\" other-blob config >h2 &&\n \n \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n \tgit -c protocol.version=2 \\\n@@ -879,9 +909,11 @@ test_expect_success 'part of packfile response provided as URI' '\n \tls http_child/.git/objects/pack/*.pack \\\n \t http_child/.git/objects/pack/*.idx >filelist &&\n \ttest_line_count = 6 filelist\n-'\n+}\n+\n+blobpackfileuri_fetch () {\n+\tconfig=\"$1\"\n \n-test_expect_success 'packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n@@ -890,9 +922,9 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob $config >h &&\n \n \tgit init http_child &&\n \n@@ -900,6 +932,215 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \tgit -C http_child -c protocol.version=2 \\\n \t\t-c fetch.uriprotocols=http,https \\\n \t\tfetch \"$HTTPD_URL/smart/http_parent\"\n+}\n+\n+test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): part of packfile response provided as URI' '\n+\tpart_of_packfile_response_verify old\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.excludeobject): part of packfile response provided as URI' '\n+\tpart_of_packfile_response_verify new\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): packfile URIs with fetch instead of clone' '\n+\tblobpackfileuri_fetch old\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.excludeobject): packfile URIs with fetch instead of clone' '\n+\tblobpackfileuri_fetch new\n+'\n+\n+test_expect_success 'tree-exclusion: part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\t# Dir struct\n+\t# \t.\n+\t# |-- A.t\n+\t# |-- my-tree\n+\t# | `-- my-blob\n+\t# `-- other-tree\n+\t# |-- other-blob\n+\t# `-- sub-tree\n+\t# `-- sub-blob\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/other-tree &&\n+\techo other-blob >\"$P\"/other-tree/other-blob &&\n+\tmkdir \"$P\"/other-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/other-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add other-tree &&\n+ \ttest_commit -C \"$P\" A &&\n+\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tothertreeh=$(git -C \"$P\" ls-tree HEAD other-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD other-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\totherblobh=$(git -C \"$P\" hash-object other-tree/other-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object other-tree/sub-tree/sub-blob) &&\n+\n+\tconfigure_exclusion tree \"$P\" \"$mytreeh\" config >h &&\n+\tconfigure_exclusion tree \"$P\" \"$othertreeh\" config >h2 &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\n+\t# Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles.\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 3 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi\n+\t\telif test_line_count = 2 out.objectlist\n+\t\tthen\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi\n+\t\telif test_line_count = 4 out.objectlist\n+\t\tthen\n+\t\t\tif grep $othertreeh out\n+\t\t\tthen\n+\t\t\t\t>othertreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $otherblobh out\n+\t\t\tthen\n+\t\t\t\t>otherblobhfound\n+\t\t\tfi\n+\t\t\tif grep $subtreeh out\n+\t\t\tthen\n+\t\t\t\t>subtreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $subblobh out\n+\t\t\tthen\n+\t\t\t\t>subblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f othertreehfound &&\n+\ttest -f otherblobhfound &&\n+\ttest -f subtreehfound &&\n+\ttest -f subblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\n+\t# Ensure that there are exactly 3 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 6 filelist\n+'\n+\n+test_expect_success 'commit-exclusion: part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/my-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/my-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\ttest_commit -C \"$P\" A &&\n+\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD my-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object my-tree/sub-tree/sub-blob) &&\n+\n+\tconfigure_exclusion commit \"$P\" \"$commith\" >h &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 7 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $subtreeh out\n+\t\t\tthen\n+\t\t\t\t>subtreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $subblobh out\n+\t\t\tthen\n+\t\t\t\t>subblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f subtreehfound &&\n+\ttest -f subblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\n+\t# Ensure that there are exactly 2 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 4 filelist\n '\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n@@ -913,9 +1154,9 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \t# Configure a URL for other-blob. Just reuse the hash of the object as\n \t# the hash of the packfile, since the hash does not matter for this\n \t# test as long as it is not the hash of the pack, and it is of the\n@@ -923,7 +1164,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" hash-object other-blob >objh &&\n \tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n \tgit -C \"$P\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n+\t\t\"uploadpack.excludeobject\" \\\n \t\t\"$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n \n \ttest_must_fail env GIT_TEST_SIDEBAND_ALL=1 \\\n@@ -942,9 +1183,8 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -976,9 +1216,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1000,7 +1240,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \tgit -C \"$P\" add .gitmodules &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1024,9 +1264,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n \techo \"path = include/foo\" >>\"$P/.gitmodules\" &&\n \techo \"url = git://example.com/git/lib.git\" >>\"$P/.gitmodules\" &&\n \tgit -C \"$P\" add .gitmodules &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n-- \n2.31.1.449.gb2aa5456a8.dirty\n\n"},{"id":"432471","messageId":"5c6739f2-da8a-ae68-188d-5bb2fbe76ad3@gmail.com","threadId":"55638","inReplyTo":"8e5bf4010cb0d7e8b4f6c7285fe1f5365af955e8.1628666093.git.dyroneteng@gmail.com","subject":"Re: [PATCH v4 5/7] packfile-uri.txt: support for excluding commits and trees","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-08-11T09:59:56Z","receivedAt":"2021-08-11T10:00:02Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 11/08/21 14.45, Teng Long wrote:\n> +The old configuration of packfile-uri:\n> +\n> +\t`uploadpack.blobPackfileUri= `\n> +\n> +For the old configuration is compatible with the new one, but it only\n> +supports the exclusion of blob objects.\n\nI think better say \"The old configuration of packfile-uri ... is \ncompatible with the new one, but it only supports exclusion of blobs\".\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"433693","messageId":"cover.1629805395.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1628666093.git.dyroneteng@gmail.com","subject":"[PATCH v5 00/14] packfile-uris: commits, trees and tags exclusion","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:44Z","receivedAt":"2021-08-25T02:22:10Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Changes since v5:\n\n* Support tag objects exclusion (configurable but will repeat download object\ndata in previous patch)\n* Tests and docs modification for tag exclusion feature\n* Abstraction of some reusable methods\n\nTeng Long (14):\n pack-objects.c: introduce new method `match_packfile_uri_exclusions`\n Add new parameter \"carry_data\" for \"show_object\" function\n packfile-uri: support for excluding commit objects\n packfile-uri: support for excluding tree objects\n packfile-uri.txt: support for excluding commits and trees\n t5702: replace with \"test_when_finished\" for cleanup\n t5702: support for excluding commit objects\n Add new parameter \"carry_data\" for \"show_commit function\n commit.h: add wrapped tags in commit struct\n object.h: add referred tags in `referred_objects` struct\n packfile-uri: support for excluding tag objects\n packfile-uri.txt: support for excluding tag objects\n t5702: add tag exclusion test case\n pack-objects.c: introduce `want_exclude_object` function\n\n Documentation/technical/packfile-uri.txt | 38 ++-\n builtin/describe.c | 8 +-\n builtin/pack-objects.c | 128 ++++++---\n builtin/rev-list.c | 6 +-\n bundle.c | 4 +-\n commit.h | 5 +\n list-objects.c | 50 ++--\n list-objects.h | 4 +-\n object.c | 20 +-\n object.h | 16 +-\n pack-bitmap.c | 14 +-\n reachable.c | 8 +-\n revision.c | 42 ++-\n revision.h | 3 +\n shallow.c | 4 +-\n t/t5702-protocol-v2.sh | 330 ++++++++++++++++++++---\n upload-pack.c | 7 +\n 17 files changed, 542 insertions(+), 145 deletions(-)\n\nRange-diff against v4:\n -: ---------- > 1: 73a5b4ccc1 pack-objects.c: introduce new method `match_packfile_uri_exclusions`\n -: ---------- > 2: bc8fea97e3 Add new parameter \"carry_data\" for \"show_object\" function\n -: ---------- > 3: f71b310842 packfile-uri: support for excluding commit objects\n -: ---------- > 4: bbb0413cc4 packfile-uri: support for excluding tree objects\n -: ---------- > 5: 8e5bf4010c packfile-uri.txt: support for excluding commits and trees\n -: ---------- > 6: f3b1cba7e1 t5702: replace with \"test_when_finished\" for cleanup\n -: ---------- > 7: 3b5f9732b8 t5702: support for excluding commit objects\n 1: 19f7670384 = 8: 19f7670384 Add new parameter \"carry_data\" for \"show_commit function\n 2: 011e5eaea3 = 9: 011e5eaea3 commit.h: add wrapped tags in commit struct\n 3: 824844499f = 10: 824844499f object.h: add referred tags in `referred_objects` struct\n 4: 43aa811b65 = 11: 43aa811b65 packfile-uri: support for excluding tag objects\n 5: c83db0055c = 12: c83db0055c packfile-uri.txt: support for excluding tag objects\n 6: 29a52b7a0d = 13: 29a52b7a0d t5702: add tag exclusion test case\n 7: 1a7c4c5894 = 14: 1a7c4c5894 pack-objects.c: introduce `want_exclude_object` function\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433694","messageId":"73a5b4ccc11b8b0604dc3414129069ab41be45d0.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 01/14] pack-objects.c: introduce new method `match_packfile_uri_exclusions`","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:45Z","receivedAt":"2021-08-25T02:22:11Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n builtin/pack-objects.c | 33 +++++++++++++++++++++------------\n 1 file changed, 21 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6d13cd3e1a..31556e7396 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1188,6 +1188,24 @@ static int have_duplicate_entry(const struct object_id *oid,\n \treturn 1;\n }\n \n+static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n+{\n+\tint i;\n+\tconst char *p;\n+\n+\tif (ex) {\n+\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n+\t\t\tif (skip_prefix(ex->uri,\n+\t\t\t\t\turi_protocols.items[i].string,\n+\t\t\t\t\t&p) &&\n+\t\t\t *p == ':')\n+\t\t\t\treturn 1;\n+\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n static int want_found_object(const struct object_id *oid, int exclude,\n \t\t\t struct packed_git *p)\n {\n@@ -1335,19 +1353,10 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (uri_protocols.nr) {\n \t\tstruct configured_exclusion *ex =\n \t\t\toidmap_get(&configured_exclusions, oid);\n-\t\tint i;\n-\t\tconst char *p;\n \n-\t\tif (ex) {\n-\t\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n-\t\t\t\tif (skip_prefix(ex->uri,\n-\t\t\t\t\t\turi_protocols.items[i].string,\n-\t\t\t\t\t\t&p) &&\n-\t\t\t\t *p == ':') {\n-\t\t\t\t\toidset_insert(&excluded_by_config, oid);\n-\t\t\t\t\treturn 0;\n-\t\t\t\t}\n-\t\t\t}\n+\t\tif (ex && match_packfile_uri_exclusions(ex)) {\n+\t\t\toidset_insert(&excluded_by_config, oid);\n+\t\t\treturn 0;\n \t\t}\n \t}\n \n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433695","messageId":"bc8fea97e34b3ccb4b62bb4d48dac6e1c009236c.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 02/14] Add new parameter \"carry_data\" for \"show_object\" function","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:46Z","receivedAt":"2021-08-25T02:22:14Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"During the pack-objects process, \"show_object\" function will be called\nto find the object and show the process(\"show_object_fn\" in\n\"list-object.h\"), the function definition contains three parameters:\n\n\t1. struct object *obj(contains object type, flags, and oid).\n\t2. const char *name(the object name).\n\t3. void *show_data(function to show progress info).\n\nThis commit adds a new parameter: \"void *carry_data\", the reason is\nmainly based on scalability and performance considerations when showing\nan object, space for time, avoid costly temporary calculations in the\n\"show\" phase. For example, carry the ownership relationship between\nblob or tree object and the referred commit to avoid redundant and\nexpensive calculations.\n\nSigned-off-by: Teng Long \n---\n builtin/describe.c | 4 ++--\n builtin/pack-objects.c | 15 ++++++++-------\n builtin/rev-list.c | 2 +-\n list-objects.c | 8 ++++----\n list-objects.h | 2 +-\n pack-bitmap.c | 8 ++++----\n reachable.c | 8 ++++----\n 7 files changed, 24 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/describe.c b/builtin/describe.c\nindex 40482d8e9f..045da79b5c 100644\n--- a/builtin/describe.c\n+++ b/builtin/describe.c\n@@ -485,9 +485,9 @@ static void process_commit(struct commit *commit, void *data)\n \tpcd->current_commit = commit->object.oid;\n }\n \n-static void process_object(struct object *obj, const char *path, void *data)\n+static void process_object(struct object *obj, const char *path, void *show_data, void *carry_data)\n {\n-\tstruct process_commit_data *pcd = data;\n+\tstruct process_commit_data *pcd = show_data;\n \n \tif (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) {\n \t\treset_revision_walk();\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 31556e7396..5f9ec3566f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3068,7 +3068,7 @@ static void show_commit_pack_hint(struct commit *commit, void *_data)\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *_data)\n+\t\t\t\t void *show_data, void *carry_dataa)\n {\n \tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n \tif (!oe)\n@@ -3252,7 +3252,7 @@ static void show_commit(struct commit *commit, void *data)\n \t\tpropagate_island_marks(commit);\n }\n \n-static void show_object(struct object *obj, const char *name, void *data)\n+static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tadd_preferred_base_object(name);\n \tadd_object_entry(&obj->oid, obj->type, name, 0);\n@@ -3274,7 +3274,7 @@ static void show_object(struct object *obj, const char *name, void *data)\n \t}\n }\n \n-static void show_object__ma_allow_any(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_any(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tassert(arg_missing_action == MA_ALLOW_ANY);\n \n@@ -3285,10 +3285,10 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void\n \tif (!has_object(the_repository, &obj->oid, 0))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, show_data, carry_data);\n }\n \n-static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n \tassert(arg_missing_action == MA_ALLOW_PROMISOR);\n \n@@ -3299,7 +3299,7 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,\n \tif (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, show_data, carry_data);\n }\n \n static int option_parse_missing_action(const struct option *opt,\n@@ -3547,7 +3547,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \n static void record_recent_object(struct object *obj,\n \t\t\t\t const char *name,\n-\t\t\t\t void *data)\n+\t\t\t\t void *show_data,\n+\t\t\t\t void *carry_data)\n {\n \toid_array_append(&recent_objects, &obj->oid);\n }\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..1cad33d9e8 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -266,7 +266,7 @@ static int finish_object(struct object *obj, const char *name, void *cb_data)\n \treturn 0;\n }\n \n-static void show_object(struct object *obj, const char *name, void *cb_data)\n+static void show_object(struct object *obj, const char *name, void *cb_data, void *carry_data)\n {\n \tstruct rev_list_info *info = cb_data;\n \tstruct rev_info *revs = info->revs;\ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..427228a3ba 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -60,7 +60,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data, NULL);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -191,7 +191,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -207,7 +207,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -335,7 +335,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tctx->show_object(obj, name, ctx->show_data, NULL);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e46..ab946d34db 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -6,7 +6,7 @@ struct object;\n struct rev_info;\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n-typedef void (*show_object_fn)(struct object *, const char *, void *);\n+typedef void (*show_object_fn)(struct object *, const char *, void *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3ed15431cd..516eb235da 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -459,9 +459,9 @@ struct bitmap_show_data {\n \tstruct bitmap *base;\n };\n \n-static void show_object(struct object *object, const char *name, void *data_)\n+static void show_object(struct object *object, const char *name, void *show_data, void *carry_data)\n {\n-\tstruct bitmap_show_data *data = data_;\n+\tstruct bitmap_show_data *data = show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(data->bitmap_git, &object->oid);\n@@ -1268,9 +1268,9 @@ struct bitmap_test_data {\n };\n \n static void test_show_object(struct object *object, const char *name,\n-\t\t\t void *data)\n+\t\t\t void *show_data, void *carry_data)\n {\n-\tstruct bitmap_test_data *tdata = data;\n+\tstruct bitmap_test_data *tdata = show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..521b39edef 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -47,14 +47,14 @@ static int add_one_ref(const char *path, const struct object_id *oid,\n * The traversal will have already marked us as SEEN, so we\n * only need to handle any progress reporting here.\n */\n-static void mark_object(struct object *obj, const char *name, void *data)\n+static void mark_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n-\tupdate_progress(data);\n+\tupdate_progress(show_data);\n }\n \n-static void mark_commit(struct commit *c, void *data)\n+static void mark_commit(struct commit *c, void *show_data)\n {\n-\tmark_object(&c->object, NULL, data);\n+\tmark_object(&c->object, NULL, show_data, NULL);\n }\n \n struct recent_data {\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433696","messageId":"f71b3108428b6ba36667e5e8a311ab8ab973d10c.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 03/14] packfile-uri: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:47Z","receivedAt":"2021-08-25T02:22:16Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Currently packfile-uri supports the exclusion of blob objects, but in\nsome scenarios, users may wish to exclude more types of objects, such as\ncommit and tree objects, not only because packfile itself supports\nstoring these object types, but also on the other hand, to make\nconfiguration items maintainable and simpler.\n\nThis commit is used to support the recursive exclusion of a commit\nobject, which means that if the exclusion of a commit is configured as\npackfile-uri, the commit itself and all the objects it contains will\nalso be recursively excluded. In addition, to support this feature, a\nnew configuration `uploadpack.excludeobject` is introduced.\n\nThe reason for bringing a new configuration is for two considerations.\nFirst, the old configuration supports a single object type (blob), which\nlimits the use of this feature. Secondly, the name of the old\nconfiguration is not abstract enough, this make extension difficult. If\ndifferent object types use different configuration names, the\nconfiguration items will be bloated and difficult to maintain, so the\nnew configuration is more abstract in name and easy to extend.\n\nAlthough a new configuration has been introduced, the old one is\nstill available and compatible with the new configuration. The old\nconfiguration `uploadpack.blobpackfileuri` only supports excluding\nblobs. The new configuration `uploadpack.excludeobject` not only\nsupports excluding blob objects, but also supports excluding commit\nobjects, as well as recursively excluding tree objects and blob objects\nthey contain.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 50 +++++++++++++++++++++++++-----------------\n list-objects.c | 37 +++++++++++++++++--------------\n object.c | 15 ++++++++++---\n object.h | 4 +++-\n revision.c | 34 ++++++++++++++++++++--------\n revision.h | 3 +++\n upload-pack.c | 7 ++++++\n 7 files changed, 101 insertions(+), 49 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5f9ec3566f..63f3aed70a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1311,11 +1311,14 @@ static int want_object_in_pack_one(struct packed_git *p,\n static int want_object_in_pack(const struct object_id *oid,\n \t\t\t int exclude,\n \t\t\t struct packed_git **found_pack,\n-\t\t\t off_t *found_offset)\n+\t\t\t off_t *found_offset,\n+\t\t\t struct object *referred_commit)\n {\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n+\tstruct configured_exclusion *ex;\n+\tstruct configured_exclusion *referred_ex;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1351,9 +1354,13 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n-\t\tstruct configured_exclusion *ex =\n-\t\t\toidmap_get(&configured_exclusions, oid);\n+\t\tif (referred_commit) {\n+\t\t\treferred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid);\n+\t\t\tif (referred_ex && match_packfile_uri_exclusions(referred_ex))\n+\t\t\t\treturn 0;\n+\t\t}\n \n+\t\tex = oidmap_get(&configured_exclusions, oid);\n \t\tif (ex && match_packfile_uri_exclusions(ex)) {\n \t\t\toidset_insert(&excluded_by_config, oid);\n \t\t\treturn 0;\n@@ -1393,7 +1400,8 @@ static const char no_closure_warning[] = N_(\n );\n \n static int add_object_entry(const struct object_id *oid, enum object_type type,\n-\t\t\t const char *name, int exclude)\n+\t\t\t const char *name, int exclude,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct packed_git *found_pack = NULL;\n \toff_t found_offset = 0;\n@@ -1403,7 +1411,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tif (have_duplicate_entry(oid, exclude))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) {\n+\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) {\n \t\t/* The pack is missing an object, so it will not have closure */\n \t\tif (write_bitmap_index) {\n \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n@@ -1429,7 +1437,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \tif (have_duplicate_entry(oid, 0))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, 0, &pack, &offset))\n+\tif (!want_object_in_pack(oid, 0, &pack, &offset, NULL))\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, pack, offset);\n@@ -1569,7 +1577,7 @@ static void add_pbase_object(struct tree_desc *tree,\n \t\tif (name[cmplen] != '/') {\n \t\t\tadd_object_entry(&entry.oid,\n \t\t\t\t\t object_type(entry.mode),\n-\t\t\t\t\t fullname, 1);\n+\t\t\t\t\t fullname, 1, NULL);\n \t\t\treturn;\n \t\t}\n \t\tif (S_ISDIR(entry.mode)) {\n@@ -1637,7 +1645,7 @@ static void add_preferred_base_object(const char *name)\n \tcmplen = name_cmp_len(name);\n \tfor (it = pbase_tree; it; it = it->next) {\n \t\tif (cmplen == 0) {\n-\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1);\n+\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL);\n \t\t}\n \t\telse {\n \t\t\tstruct tree_desc tree;\n@@ -2839,7 +2847,7 @@ static void add_tag_chain(const struct object_id *oid)\n \t\t\tdie(_(\"unable to pack objects reachable from tag %s\"),\n \t\t\t oid_to_hex(oid));\n \n-\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0);\n+\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL);\n \n \t\tif (tag->tagged->type != OBJ_TAG)\n \t\t\treturn;\n@@ -2994,7 +3002,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n+\tif (!strcmp(k, \"uploadpack.excludeobject\") || !strcmp(k, \"uploadpack.blobpackfileuri\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n \t\tconst char *oid_end, *pack_end;\n \t\t/*\n@@ -3007,11 +3015,11 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t *oid_end != ' ' ||\n \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n \t\t *pack_end != ' ')\n-\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n+\t\t\tdie(_(\"value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be \"\n \t\t\t \"of the form ' ' (got '%s')\"), v);\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n-\t\t\tdie(_(\"object already configured in another \"\n-\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n+\t\t\tdie(_(\"object already configured by an earlier \"\n+\t\t\t \"uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')\"), v);\n \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n \t\tex->uri = xstrdup(pack_end + 1);\n@@ -3040,7 +3048,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tofs = nth_packed_object_offset(p, pos);\n-\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n+\tif (!want_object_in_pack(oid, 0, &p, &ofs, NULL))\n \t\treturn 0;\n \n \toi.typep = &type;\n@@ -3233,7 +3241,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(_(\"expected object ID, got garbage:\\n %s\"), line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL);\n \t}\n }\n \n@@ -3242,7 +3250,7 @@ static void read_object_list_from_stdin(void)\n \n static void show_commit(struct commit *commit, void *data)\n {\n-\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0);\n+\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n \tcommit->object.flags |= OBJECT_ADDED;\n \n \tif (write_bitmap_index)\n@@ -3254,8 +3262,9 @@ static void show_commit(struct commit *commit, void *data)\n \n static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n+\tstruct object *referred_commit = carry_data;\n \tadd_preferred_base_object(name);\n-\tadd_object_entry(&obj->oid, obj->type, name, 0);\n+\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_commit);\n \tobj->flags |= OBJECT_ADDED;\n \n \tif (use_delta_islands) {\n@@ -3406,7 +3415,7 @@ static void add_objects_in_unpacked_packs(void)\n \t\tQSORT(in_pack.array, in_pack.nr, ofscmp);\n \t\tfor (i = 0; i < in_pack.nr; i++) {\n \t\t\tstruct object *o = in_pack.array[i].object;\n-\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0);\n+\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0, NULL);\n \t\t}\n \t}\n \tfree(in_pack.array);\n@@ -3422,7 +3431,7 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t\treturn 0;\n \t}\n \n-\tadd_object_entry(oid, type, \"\", 0);\n+\tadd_object_entry(oid, type, \"\", 0, NULL);\n \treturn 0;\n }\n \n@@ -3841,7 +3850,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"respect islands during delta compression\")),\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n-\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject or \"\n+\t\t\t\t \"uploadpack.blobpackfileuri with this protocol\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/list-objects.c b/list-objects.c\nindex 427228a3ba..968d842ceb 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -24,7 +24,8 @@ struct traversal_context {\n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n@@ -60,7 +61,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data, NULL);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -97,11 +98,13 @@ static void process_gitlink(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name);\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit);\n \n static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t struct tree *tree,\n-\t\t\t\t struct strbuf *base)\n+\t\t\t\t struct strbuf *base,\n+\t\t\t\t struct object *referred_commit)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -129,7 +132,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t\tprocess_tree(ctx, t, base, entry.path, referred_commit);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n@@ -142,7 +145,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t\tprocess_blob(ctx, b, base, entry.path, referred_commit);\n \t\t}\n \t}\n }\n@@ -150,7 +153,8 @@ static void process_tree_contents(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct object *referred_commit)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n@@ -191,14 +195,14 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n-\t\tprocess_tree_contents(ctx, tree, base);\n+\t\tprocess_tree_contents(ctx, tree, base, referred_commit);\n \n \tr = list_objects_filter__filter_object(ctx->revs->repo,\n \t\t\t\t\t LOFS_END_TREE, obj,\n@@ -207,7 +211,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -314,9 +318,9 @@ void mark_edges_uninteresting(struct rev_info *revs,\n \t}\n }\n \n-static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n+static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit)\n {\n-\tadd_pending_object(revs, &tree->object, \"\");\n+\tadd_pending_object_with_referred_commit(revs, &tree->object, \"\", referred_commit);\n }\n \n static void traverse_trees_and_blobs(struct traversal_context *ctx,\n@@ -329,23 +333,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n+\t\tstruct object *referred_commit = pending->referred_commit;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data, NULL);\n+\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_commit);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n@@ -370,7 +375,7 @@ static void do_traverse(struct traversal_context *ctx)\n \t\telse if (get_commit_tree(commit)) {\n \t\t\tstruct tree *tree = get_commit_tree(commit);\n \t\t\ttree->object.flags |= NOT_USER_GIVEN;\n-\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t\tadd_pending_tree(ctx->revs, tree, &commit->object);\n \t\t} else if (commit->object.parsed) {\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\ndiff --git a/object.c b/object.c\nindex 14188453c5..6b1ce2fcde 100644\n--- a/object.c\n+++ b/object.c\n@@ -322,9 +322,10 @@ void object_list_free(struct object_list **list)\n */\n static char object_array_slopbuf[1];\n \n-void add_object_array_with_path(struct object *obj, const char *name,\n-\t\t\t\tstruct object_array *array,\n-\t\t\t\tunsigned mode, const char *path)\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name,\n+\t\t\t\t\t\t struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path,\n+\t\t\t\t\t\t struct object *referred_commit)\n {\n \tunsigned nr = array->nr;\n \tunsigned alloc = array->alloc;\n@@ -339,6 +340,7 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \t}\n \tentry = &objects[nr];\n \tentry->item = obj;\n+\tentry->referred_commit = referred_commit;\n \tif (!name)\n \t\tentry->name = NULL;\n \telse if (!*name)\n@@ -354,6 +356,13 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \tarray->nr = ++nr;\n }\n \n+void add_object_array_with_path(struct object *obj, const char *name,\n+\t\t\t\tstruct object_array *array,\n+\t\t\t\tunsigned mode, const char *path)\n+{\n+\tadd_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL);\n+}\n+\n void add_object_array(struct object *obj, const char *name, struct object_array *array)\n {\n \tadd_object_array_with_path(obj, name, array, S_IFINVALID, NULL);\ndiff --git a/object.h b/object.h\nindex 87a6da47c8..d63819ab91 100644\n--- a/object.h\n+++ b/object.h\n@@ -52,6 +52,7 @@ struct object_array {\n \t\tchar *name;\n \t\tchar *path;\n \t\tunsigned mode;\n+\t\tstruct object *referred_commit;\n \t} *objects;\n };\n \n@@ -157,7 +158,8 @@ void object_list_free(struct object_list **list);\n /* Object array handling .. */\n void add_object_array(struct object *obj, const char *name, struct object_array *array);\n void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path);\n-\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path, struct object *referred_commit);\n /*\n * Returns NULL if the array is empty. Otherwise, returns the last object\n * after removing its entry from the array. Other resources associated\ndiff --git a/revision.c b/revision.c\nindex 4853c85d0b..65e0926d25 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit)\n \tcommit_stack_clear(&pending);\n }\n \n-static void add_pending_object_with_path(struct rev_info *revs,\n-\t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode,\n-\t\t\t\t\t const char *path)\n+static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t\t struct object *referred_commit)\n {\n \tstruct interpret_branch_name_options options = { 0 };\n \tif (!obj)\n@@ -326,20 +327,35 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n+\tadd_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit);\n }\n \n+static void add_pending_object_with_path(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t const char *path)\n+{\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL);\n+}\n static void add_pending_object_with_mode(struct rev_info *revs,\n \t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode)\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t struct object *referred_commit)\n+{\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit);\n+}\n+\n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct object *referred_commit)\n {\n-\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit);\n }\n \n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name)\n {\n-\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID);\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL);\n }\n \n void add_head_to_pending(struct rev_info *revs)\n@@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tif (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc))\n \t\t\tdiagnose_missing_default(revs->def);\n \t\tobject = get_reference(revs, revs->def, &oid, 0);\n-\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode);\n+\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL);\n \t}\n \n \t/* Did the user ask for any diff output? Run the diff! */\ndiff --git a/revision.h b/revision.h\nindex a24f72dcd1..f9c9628ed8 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -423,6 +423,9 @@ void show_object_with_name(FILE *, struct object *, const char *);\n */\n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name);\n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct object *referred_commit);\n \n void add_pending_oid(struct rev_info *revs,\n \t\t const char *name, const struct object_id *oid,\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5c1cd19612..d26fb351a3 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r,\n \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n \t\t\tfree(str);\n \t\t}\n+\n+\t\tif (!repo_config_get_string(the_repository,\n+\t\t\t\t\t \"uploadpack.excludeobject\",\n+\t\t\t\t\t &str) && str) {\n+\t\t\tstrbuf_addstr(value, \" packfile-uris\");\n+\t\t\tfree(str);\n+\t\t}\n \t}\n \n \treturn 1;\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433697","messageId":"bbb0413cc41811d5a8cd6025fc546657e88e48c6.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 04/14] packfile-uri: support for excluding tree objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:48Z","receivedAt":"2021-08-25T02:22:18Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"This commit supports the use of `uploadpack.excludeobject` to exclude\ntree objects, which means that when a type object is configured as\npackfile-uri, the tree object itself and all objects contains will be\nrecursively excluded.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 30 ++++++++++++++++++++----------\n list-objects.c | 32 ++++++++++++++++++--------------\n object.c | 6 +++++-\n object.h | 13 ++++++++++++-\n 4 files changed, 55 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 63f3aed70a..4ff12ec525 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1312,13 +1312,14 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\t int exclude,\n \t\t\t struct packed_git **found_pack,\n \t\t\t off_t *found_offset,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n+\tstruct configured_exclusion *commit_ex;\n+\tstruct configured_exclusion *tree_ex;\n \tstruct configured_exclusion *ex;\n-\tstruct configured_exclusion *referred_ex;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1354,14 +1355,23 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n-\t\tif (referred_commit) {\n-\t\t\treferred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid);\n-\t\t\tif (referred_ex && match_packfile_uri_exclusions(referred_ex))\n+\t\tif (referred_objs && referred_objs->commit) {\n+\t\t\tcommit_ex = oidmap_get(&configured_exclusions, &referred_objs->commit->oid);\n+\t\t\tif (match_packfile_uri_exclusions(commit_ex))\n \t\t\t\treturn 0;\n \t\t}\n \n+\t\tif (referred_objs && referred_objs->trees) {\n+\t\t\tstruct object_list *p;\n+\t\t\tfor (p = referred_objs->trees; p; p = p->next) {\n+\t\t\t\ttree_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n+\t\t\t\tif (match_packfile_uri_exclusions(tree_ex))\n+\t\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t}\n+\n \t\tex = oidmap_get(&configured_exclusions, oid);\n-\t\tif (ex && match_packfile_uri_exclusions(ex)) {\n+\t\tif (match_packfile_uri_exclusions(ex)) {\n \t\t\toidset_insert(&excluded_by_config, oid);\n \t\t\treturn 0;\n \t\t}\n@@ -1401,7 +1411,7 @@ static const char no_closure_warning[] = N_(\n \n static int add_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t const char *name, int exclude,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct packed_git *found_pack = NULL;\n \toff_t found_offset = 0;\n@@ -1411,7 +1421,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tif (have_duplicate_entry(oid, exclude))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) {\n+\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_objs)) {\n \t\t/* The pack is missing an object, so it will not have closure */\n \t\tif (write_bitmap_index) {\n \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n@@ -3262,9 +3272,9 @@ static void show_commit(struct commit *commit, void *data)\n \n static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n {\n-\tstruct object *referred_commit = carry_data;\n+\tstruct referred_objects *referred_objs = carry_data;\n \tadd_preferred_base_object(name);\n-\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_commit);\n+\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_objs);\n \tobj->flags |= OBJECT_ADDED;\n \n \tif (use_delta_islands) {\ndiff --git a/list-objects.c b/list-objects.c\nindex 968d842ceb..49f177cb56 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -25,7 +25,7 @@ static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n@@ -61,7 +61,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n+\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_objs);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -99,19 +99,22 @@ static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name,\n-\t\t\t struct object *referred_commit);\n+\t\t\t struct referred_objects *referred_objs);\n \n static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t struct tree *tree,\n \t\t\t\t struct strbuf *base,\n-\t\t\t\t struct object *referred_commit)\n+\t\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n \t\tall_entries_interesting : entry_not_interesting;\n+\tstruct referred_objects *referred_buf;\n \n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\treferred_buf = xmemdupz(referred_objs, sizeof(struct referred_objects));\n+\tobject_list_insert(&tree->object, &referred_buf->trees);\n \n \twhile (tree_entry(&desc, &entry)) {\n \t\tif (match != all_entries_interesting) {\n@@ -132,7 +135,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path, referred_commit);\n+\t\t\tprocess_tree(ctx, t, base, entry.path, referred_buf);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n@@ -145,16 +148,17 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path, referred_commit);\n+\t\t\tprocess_blob(ctx, b, base, entry.path, referred_buf);\n \t\t}\n \t}\n+\tfree(referred_buf);\n }\n \n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name,\n-\t\t\t struct object *referred_commit)\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n@@ -195,14 +199,14 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_objs);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n-\t\tprocess_tree_contents(ctx, tree, base, referred_commit);\n+\t\tprocess_tree_contents(ctx, tree, base, referred_objs);\n \n \tr = list_objects_filter__filter_object(ctx->revs->repo,\n \t\t\t\t\t LOFS_END_TREE, obj,\n@@ -211,7 +215,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n+\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_objs);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -333,24 +337,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n-\t\tstruct object *referred_commit = pending->referred_commit;\n+\t\tstruct referred_objects *referred_objs = pending->referred_objects;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n+\t\t\tctx->show_object(obj, name, ctx->show_data, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_commit);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_commit);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\ndiff --git a/object.c b/object.c\nindex 6b1ce2fcde..69ba0baf95 100644\n--- a/object.c\n+++ b/object.c\n@@ -331,6 +331,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch\n \tunsigned alloc = array->alloc;\n \tstruct object_array_entry *objects = array->objects;\n \tstruct object_array_entry *entry;\n+\tstruct referred_objects *referred_objs = xmalloc(sizeof(struct referred_objects));\n \n \tif (nr >= alloc) {\n \t\talloc = (alloc + 32) * 2;\n@@ -338,9 +339,11 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch\n \t\tarray->alloc = alloc;\n \t\tarray->objects = objects;\n \t}\n+\treferred_objs->commit = referred_commit;\n+\treferred_objs->trees = NULL;\n \tentry = &objects[nr];\n \tentry->item = obj;\n-\tentry->referred_commit = referred_commit;\n+\tentry->referred_objects = referred_objs;\n \tif (!name)\n \t\tentry->name = NULL;\n \telse if (!*name)\n@@ -377,6 +380,7 @@ static void object_array_release_entry(struct object_array_entry *ent)\n \tif (ent->name != object_array_slopbuf)\n \t\tfree(ent->name);\n \tfree(ent->path);\n+\tfree(ent->referred_objects);\n }\n \n struct object *object_array_pop(struct object_array *array)\ndiff --git a/object.h b/object.h\nindex d63819ab91..3785546adf 100644\n--- a/object.h\n+++ b/object.h\n@@ -52,12 +52,23 @@ struct object_array {\n \t\tchar *name;\n \t\tchar *path;\n \t\tunsigned mode;\n-\t\tstruct object *referred_commit;\n+\t\t /*\n+\t\t * referred_objects or NULL. If non-NULL, it will\n+\t\t * temporary storage the referred commit and trees when\n+\t\t * traversing the specified object. Space for time,\n+\t\t * reduce related computing costs (such as packfile-uri\n+\t\t * exclusion), clean up when the traversal is over.\n+\t\t */\n+\t\tstruct referred_objects *referred_objects;\n \t} *objects;\n };\n \n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n+struct referred_objects{\n+ struct object *commit;\n+ struct object_list *trees;\n+};\n /*\n * object flag allocation:\n * revision.h: 0---------10 15 23------26\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433698","messageId":"8e5bf4010cb0d7e8b4f6c7285fe1f5365af955e8.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 05/14] packfile-uri.txt: support for excluding commits and trees","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:49Z","receivedAt":"2021-08-25T02:22:22Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n Documentation/technical/packfile-uri.txt | 32 ++++++++++++++++--------\n 1 file changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt\nindex f7eabc6c76..c3e4873956 100644\n--- a/Documentation/technical/packfile-uri.txt\n+++ b/Documentation/technical/packfile-uri.txt\n@@ -35,13 +35,26 @@ include some sort of non-trivial implementation in the Minimum Viable Product,\n at least so that we can test the client.\n \n This is the implementation: a feature, marked experimental, that allows the\n-server to be configured by one or more `uploadpack.blobPackfileUri=\n-` entries. Whenever the list of objects to be sent is assembled, all such\n-blobs are excluded, replaced with URIs. As noted in \"Future work\" below, the\n-server can evolve in the future to support excluding other objects (or other\n-implementations of servers could be made that support excluding other objects)\n-without needing a protocol change, so clients should not expect that packfiles\n-downloaded in this way only contain single blobs.\n+server to be configured by one or more entries with the format:\n+\n+ uploadpack.excludeobject= \n+\n+Value is the key of entry, and the object type can be a blob,\n+tree, or commit. The exclusion of tree and commit is recursive by default,\n+which means that when a tree or commit object is excluded, the object itself\n+and all reachable objects of the object will be excluded recursively. Whenever\n+the list of objects to be sent is assembled, all such objects are excluded,\n+replaced with URIs.\n+\n+Configuration compatibility\n+-------------\n+\n+The old configuration of packfile-uri:\n+\n+\t`uploadpack.blobPackfileUri= `\n+\n+For the old configuration is compatible with the new one, but it only\n+supports the exclusion of blob objects.\n \n Client design\n -------------\n@@ -65,9 +78,6 @@ The protocol design allows some evolution of the server and client without any\n need for protocol changes, so only a small-scoped design is included here to\n form the MVP. For example, the following can be done:\n \n- * On the server, more sophisticated means of excluding objects (e.g. by\n- specifying a commit to represent that commit and all objects that it\n- references).\n * On the client, resumption of clone. If a clone is interrupted, information\n could be recorded in the repository's config and a \"clone-resume\" command\n can resume the clone in progress. (Resumption of subsequent fetches is more\n@@ -78,4 +88,4 @@ There are some possible features that will require a change in protocol:\n \n * Additional HTTP headers (e.g. authentication)\n * Byte range support\n- * Different file formats referenced by URIs (e.g. raw object)\n+ * Different file formats referenced by URIs (e.g. raw object)\n\\ No newline at end of file\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433699","messageId":"f3b1cba7e102b3132f04b27be48e532da7d448c9.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 06/14] t5702: replace with \"test_when_finished\" for cleanup","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:50Z","receivedAt":"2021-08-25T02:22:24Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 2e1243ca40..e6314b53b0 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -753,7 +753,7 @@ test_expect_success 'ls-remote with v2 http sends only one POST' '\n '\n \n test_expect_success 'push with http:// and a config of v2 does not request v2' '\n-\ttest_when_finished \"rm -f log\" &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \t# Till v2 for push is designed, make sure that if a client has\n \t# protocol.version configured to use v2, that the client instead falls\n \t# back and uses v0.\n@@ -776,7 +776,7 @@ test_expect_success 'push with http:// and a config of v2 does not request v2' '\n '\n \n test_expect_success 'when server sends \"ready\", expect DELIM' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -796,7 +796,7 @@ test_expect_success 'when server sends \"ready\", expect DELIM' '\n '\n \n test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n-\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\\\" http_child log\" &&\n \n \tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one &&\n@@ -834,7 +834,7 @@ configure_exclusion () {\n \n test_expect_success 'part of packfile response provided as URI' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -883,7 +883,7 @@ test_expect_success 'part of packfile response provided as URI' '\n \n test_expect_success 'packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -904,7 +904,7 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -935,7 +935,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -959,7 +959,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -989,7 +989,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1015,7 +1015,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child err &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child err\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -1038,4 +1038,4 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-test_done\n+test_done\n\\ No newline at end of file\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433700","messageId":"3b5f9732b814a712094091027c8229988e2ba791.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 07/14] t5702: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:51Z","receivedAt":"2021-08-25T02:22:28Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 292 +++++++++++++++++++++++++++++++++++++----\n 1 file changed, 266 insertions(+), 26 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex e6314b53b0..5ad52e0cee 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -824,17 +824,47 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n '\n \n configure_exclusion () {\n-\tgit -C \"$1\" hash-object \"$2\" >objh &&\n-\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n-\tgit -C \"$1\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n-\tcat objh\n+\tobjt=\"$1\"\n+\tP=\"$2\"\n+\toid=\"$3\"\n+\tversion=\"$4\"\n+\n+\toldc=\"uploadpack.blobpackfileuri\"\n+\tnewc=\"uploadpack.excludeobject\"\n+\tconfigkey=\"\"\n+\n+\tif test \"$version\" = \"old\"\n+\tthen\n+\t\tconfigkey=\"$oldc\"\n+\telse\n+\t\tconfigkey=\"$newc\"\n+\tfi\n+\n+\tif test \"$objt\" = \"blob\"\n+\tthen\n+\t\tgit -C \"$P\" hash-object \"$oid\" >objh &&\n+\t\tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n+\t\tgit -C \"$P\" config --add \\\n+\t\t\t\"$configkey\" \\\n+\t\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+\t\tcat objh\n+\telif test \"$objt\" = \"commit\" || test \"$objt\" = \"tree\" || test \"$objt\" = \"tag\"\n+\tthen\n+\t\techo \"$oid\" >objh\n+\t\tgit -C \"$P\" pack-objects --revs \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n+\t\tgit -C \"$P\" config --add \\\n+ \t\t\t\"$configkey\" \\\n+ \t\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+\t\tcat objh\n+\telse\n+\t\techo \"unsupported object type in configure_exclusion (got $objt)\"\n+\tfi\n }\n \n-test_expect_success 'part of packfile response provided as URI' '\n+part_of_packfile_response_verify() {\n+\tconfig=\"$1\"\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -843,10 +873,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\tconfigure_exclusion \"$P\" other-blob >h2 &&\n+\tconfigure_exclusion blob \"$P\" my-blob config >h &&\n+\tconfigure_exclusion blob \"$P\" other-blob config >h2 &&\n \n \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n \tgit -c protocol.version=2 \\\n@@ -879,9 +909,11 @@ test_expect_success 'part of packfile response provided as URI' '\n \tls http_child/.git/objects/pack/*.pack \\\n \t http_child/.git/objects/pack/*.idx >filelist &&\n \ttest_line_count = 6 filelist\n-'\n+}\n+\n+blobpackfileuri_fetch () {\n+\tconfig=\"$1\"\n \n-test_expect_success 'packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n@@ -890,9 +922,9 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob $config >h &&\n \n \tgit init http_child &&\n \n@@ -900,6 +932,215 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \tgit -C http_child -c protocol.version=2 \\\n \t\t-c fetch.uriprotocols=http,https \\\n \t\tfetch \"$HTTPD_URL/smart/http_parent\"\n+}\n+\n+test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): part of packfile response provided as URI' '\n+\tpart_of_packfile_response_verify old\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.excludeobject): part of packfile response provided as URI' '\n+\tpart_of_packfile_response_verify new\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): packfile URIs with fetch instead of clone' '\n+\tblobpackfileuri_fetch old\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.excludeobject): packfile URIs with fetch instead of clone' '\n+\tblobpackfileuri_fetch new\n+'\n+\n+test_expect_success 'tree-exclusion: part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\t# Dir struct\n+\t# \t.\n+\t# |-- A.t\n+\t# |-- my-tree\n+\t# | `-- my-blob\n+\t# `-- other-tree\n+\t# |-- other-blob\n+\t# `-- sub-tree\n+\t# `-- sub-blob\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/other-tree &&\n+\techo other-blob >\"$P\"/other-tree/other-blob &&\n+\tmkdir \"$P\"/other-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/other-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add other-tree &&\n+ \ttest_commit -C \"$P\" A &&\n+\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tothertreeh=$(git -C \"$P\" ls-tree HEAD other-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD other-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\totherblobh=$(git -C \"$P\" hash-object other-tree/other-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object other-tree/sub-tree/sub-blob) &&\n+\n+\tconfigure_exclusion tree \"$P\" \"$mytreeh\" config >h &&\n+\tconfigure_exclusion tree \"$P\" \"$othertreeh\" config >h2 &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\n+\t# Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles.\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 3 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi\n+\t\telif test_line_count = 2 out.objectlist\n+\t\tthen\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi\n+\t\telif test_line_count = 4 out.objectlist\n+\t\tthen\n+\t\t\tif grep $othertreeh out\n+\t\t\tthen\n+\t\t\t\t>othertreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $otherblobh out\n+\t\t\tthen\n+\t\t\t\t>otherblobhfound\n+\t\t\tfi\n+\t\t\tif grep $subtreeh out\n+\t\t\tthen\n+\t\t\t\t>subtreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $subblobh out\n+\t\t\tthen\n+\t\t\t\t>subblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f othertreehfound &&\n+\ttest -f otherblobhfound &&\n+\ttest -f subtreehfound &&\n+\ttest -f subblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\n+\t# Ensure that there are exactly 3 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 6 filelist\n+'\n+\n+test_expect_success 'commit-exclusion: part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/my-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/my-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\ttest_commit -C \"$P\" A &&\n+\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD my-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object my-tree/sub-tree/sub-blob) &&\n+\n+\tconfigure_exclusion commit \"$P\" \"$commith\" >h &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 7 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $subtreeh out\n+\t\t\tthen\n+\t\t\t\t>subtreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $subblobh out\n+\t\t\tthen\n+\t\t\t\t>subblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f subtreehfound &&\n+\ttest -f subblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\n+\t# Ensure that there are exactly 2 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 4 filelist\n '\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n@@ -913,9 +1154,9 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \t# Configure a URL for other-blob. Just reuse the hash of the object as\n \t# the hash of the packfile, since the hash does not matter for this\n \t# test as long as it is not the hash of the pack, and it is of the\n@@ -923,7 +1164,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" hash-object other-blob >objh &&\n \tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n \tgit -C \"$P\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n+\t\t\"uploadpack.excludeobject\" \\\n \t\t\"$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n \n \ttest_must_fail env GIT_TEST_SIDEBAND_ALL=1 \\\n@@ -942,9 +1183,8 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -976,9 +1216,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1000,7 +1240,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \tgit -C \"$P\" add .gitmodules &&\n \tgit -C \"$P\" commit -m x &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n@@ -1024,9 +1264,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n \techo \"path = include/foo\" >>\"$P/.gitmodules\" &&\n \techo \"url = git://example.com/git/lib.git\" >>\"$P/.gitmodules\" &&\n \tgit -C \"$P\" add .gitmodules &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433701","messageId":"19f767038486e53a3d12e38aea3bba51e6541e7d.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 08/14] Add new parameter \"carry_data\" for \"show_commit function","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:52Z","receivedAt":"2021-08-25T02:22:37Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n builtin/describe.c | 4 ++--\n builtin/pack-objects.c | 6 +++---\n builtin/rev-list.c | 4 ++--\n bundle.c | 4 ++--\n list-objects.c | 2 +-\n list-objects.h | 2 +-\n pack-bitmap.c | 6 +++---\n reachable.c | 2 +-\n shallow.c | 4 ++--\n 9 files changed, 17 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/describe.c b/builtin/describe.c\nindex 045da79b5c..8fb99bbda5 100644\n--- a/builtin/describe.c\n+++ b/builtin/describe.c\n@@ -479,9 +479,9 @@ struct process_commit_data {\n \tstruct rev_info *revs;\n };\n \n-static void process_commit(struct commit *commit, void *data)\n+static void process_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n-\tstruct process_commit_data *pcd = data;\n+\tstruct process_commit_data *pcd = show_data;\n \tpcd->current_commit = commit->object.oid;\n }\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4ff12ec525..d38b24e375 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3080,7 +3080,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit, void *_data)\n+static void show_commit_pack_hint(struct commit *commit, void *show_data, void *carry_data)\n {\n \t/* nothing to do; commits don't have a namehash */\n }\n@@ -3258,7 +3258,7 @@ static void read_object_list_from_stdin(void)\n /* Remember to update object flag allocation in object.h */\n #define OBJECT_ADDED (1u<<20)\n \n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n \tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n \tcommit->object.flags |= OBJECT_ADDED;\n@@ -3572,7 +3572,7 @@ static void record_recent_object(struct object *obj,\n \toid_array_append(&recent_objects, &obj->oid);\n }\n \n-static void record_recent_commit(struct commit *commit, void *data)\n+static void record_recent_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n \toid_array_append(&recent_objects, &commit->object.oid);\n }\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 1cad33d9e8..b5e7ba6e83 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -94,9 +94,9 @@ static off_t get_object_disk_usage(struct object *obj)\n }\n \n static void finish_commit(struct commit *commit);\n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n-\tstruct rev_list_info *info = data;\n+\tstruct rev_list_info *info = show_data;\n \tstruct rev_info *revs = info->revs;\n \n \tdisplay_progress(progress, ++progress_counter);\ndiff --git a/bundle.c b/bundle.c\nindex 693d619551..143e45ce0c 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -437,9 +437,9 @@ struct bundle_prerequisites_info {\n \tint fd;\n };\n \n-static void write_bundle_prerequisites(struct commit *commit, void *data)\n+static void write_bundle_prerequisites(struct commit *commit, void *show_data, void *carry_data)\n {\n-\tstruct bundle_prerequisites_info *bpi = data;\n+\tstruct bundle_prerequisites_info *bpi = show_data;\n \tstruct object *object;\n \tstruct pretty_print_context ctx = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/list-objects.c b/list-objects.c\nindex 49f177cb56..2e53a01458 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -384,7 +384,7 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data);\n+\t\tctx->show_commit(commit, ctx->show_data, NULL);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\ndiff --git a/list-objects.h b/list-objects.h\nindex ab946d34db..838b8c78c9 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -5,7 +5,7 @@ struct commit;\n struct object;\n struct rev_info;\n \n-typedef void (*show_commit_fn)(struct commit *, void *);\n+typedef void (*show_commit_fn)(struct commit *, void *, void *);\n typedef void (*show_object_fn)(struct object *, const char *, void *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 516eb235da..81cf14ef8e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -473,7 +473,7 @@ static void show_object(struct object *object, const char *name, void *show_data\n \tbitmap_set(data->base, bitmap_pos);\n }\n \n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n }\n \n@@ -1281,9 +1281,9 @@ static void test_show_object(struct object *object, const char *name,\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n }\n \n-static void test_show_commit(struct commit *commit, void *data)\n+static void test_show_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n-\tstruct bitmap_test_data *tdata = data;\n+\tstruct bitmap_test_data *tdata = show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(tdata->bitmap_git,\ndiff --git a/reachable.c b/reachable.c\nindex 521b39edef..d38c31ae45 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -52,7 +52,7 @@ static void mark_object(struct object *obj, const char *name, void *show_data, v\n \tupdate_progress(show_data);\n }\n \n-static void mark_commit(struct commit *c, void *show_data)\n+static void mark_commit(struct commit *c, void *show_data, void *carry_data)\n {\n \tmark_object(&c->object, NULL, show_data, NULL);\n }\ndiff --git a/shallow.c b/shallow.c\nindex 9ed18eb884..33f878565c 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -185,9 +185,9 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \treturn result;\n }\n \n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n-\tcommit_list_insert(commit, data);\n+\tcommit_list_insert(commit, show_data);\n }\n \n /*\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433702","messageId":"011e5eaea3f48eeacac1614f769e8fd809e0c093.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 09/14] commit.h: add wrapped tags in commit struct","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:53Z","receivedAt":"2021-08-25T02:22:38Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n commit.h | 5 +++++\n revision.c | 8 ++++++--\n 2 files changed, 11 insertions(+), 2 deletions(-)\n\ndiff --git a/commit.h b/commit.h\nindex df42eb434f..1e99e9ae8a 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -38,6 +38,11 @@ struct commit {\n \t */\n \tstruct tree *maybe_tree;\n \tunsigned int index;\n+\t/*\n+\t* wrapped tags or NULL. If the commit is peeled from tag(s),\n+\t* then save the wraps, otherwise will be NULL.\n+\t*/\n+\tstruct object_list *wraps;\n };\n \n extern int save_commit_buffer;\ndiff --git a/revision.c b/revision.c\nindex 65e0926d25..aecf493f46 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -416,14 +416,17 @@ static struct commit *handle_commit(struct rev_info *revs,\n \tconst char *path = entry->path;\n \tunsigned int mode = entry->mode;\n \tunsigned long flags = object->flags;\n-\n+\tstruct object_list *wraps = NULL;\n \t/*\n \t * Tag object? Look what it points to..\n \t */\n \twhile (object->type == OBJ_TAG) {\n \t\tstruct tag *tag = (struct tag *) object;\n-\t\tif (revs->tag_objects && !(flags & UNINTERESTING))\n+\t\tif (revs->tag_objects && !(flags & UNINTERESTING)) {\n+\t\t\tobject_list_insert(object, &wraps);\n \t\t\tadd_pending_object(revs, object, tag->tag);\n+\t\t}\n+\n \t\tobject = parse_object(revs->repo, get_tagged_oid(tag));\n \t\tif (!object) {\n \t\t\tif (revs->ignore_missing_links || (flags & UNINTERESTING))\n@@ -449,6 +452,7 @@ static struct commit *handle_commit(struct rev_info *revs,\n \t */\n \tif (object->type == OBJ_COMMIT) {\n \t\tstruct commit *commit = (struct commit *)object;\n+\t\tcommit->wraps = wraps;\n \n \t\tif (repo_parse_commit(revs->repo, commit) < 0)\n \t\t\tdie(\"unable to parse commit %s\", name);\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433703","messageId":"824844499f3761b9bdc7952a5ace3721f3c92432.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 10/14] object.h: add referred tags in `referred_objects` struct","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:54Z","receivedAt":"2021-08-25T02:22:40Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n object.c | 1 +\n object.h | 5 +++--\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 69ba0baf95..b25b2e2919 100644\n--- a/object.c\n+++ b/object.c\n@@ -341,6 +341,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch\n \t}\n \treferred_objs->commit = referred_commit;\n \treferred_objs->trees = NULL;\n+\treferred_objs->tags = NULL;\n \tentry = &objects[nr];\n \tentry->item = obj;\n \tentry->referred_objects = referred_objs;\ndiff --git a/object.h b/object.h\nindex 3785546adf..bd59eaa6d9 100644\n--- a/object.h\n+++ b/object.h\n@@ -54,8 +54,8 @@ struct object_array {\n \t\tunsigned mode;\n \t\t /*\n \t\t * referred_objects or NULL. If non-NULL, it will\n-\t\t * temporary storage the referred commit and trees when\n-\t\t * traversing the specified object. Space for time,\n+\t\t * temporary storage the referred commit, trees and tags\n+\t\t * when traversing the specified object. Space for time,\n \t\t * reduce related computing costs (such as packfile-uri\n \t\t * exclusion), clean up when the traversal is over.\n \t\t */\n@@ -68,6 +68,7 @@ struct object_array {\n struct referred_objects{\n struct object *commit;\n struct object_list *trees;\n+ struct object_list *tags;\n };\n /*\n * object flag allocation:\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433704","messageId":"43aa811b659651611c041db8e9d6c63a9aedfce3.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 11/14] packfile-uri: support for excluding tag objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:55Z","receivedAt":"2021-08-25T02:22:45Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"This commit supports the use of `uploadpack.excludeobject` to exclude\ntag objects, both lightweight tag and annotated tag are supported:\n\n-If a lightweight tag (such as a commit object) have been configured,\nthe mechanism of exclusion is the same as the commit object (the commit\nobject and all objects it contains will be recusively excluded).\n\n-If an annotated tag (created with -a, -s, or -u) have been configured,\nthe annotated tag and all the objects that it contains will be excluded,\n\nFor an example of the annotated tag:\n\n\tCreate an annotated tag from HEAD:\n\n\t\tgit tag -a A -m \"tag A description\"\n\n\tOutput the SHA () of tag \"A\" :\n\n\t\tgit rev-parse A^{object}\n\n\tDereference , output the SHA of commit:\n\n\t\tgit rev-parse A^{}\n\nIn the above case, when the tag object () is configured with\n`uploadpack.excludeobject` which means , the dereference\ncommit object (), and all the objects that \nrecursively contains (trees, blobs), will be excluded (using a packfile\nURI instead).\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 20 ++++++++++++++++++--\n list-objects.c | 9 ++++++++-\n 2 files changed, 26 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d38b24e375..d5e3f2c229 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1317,9 +1317,11 @@ static int want_object_in_pack(const struct object_id *oid,\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n+\tstruct configured_exclusion *tag_ex;\n \tstruct configured_exclusion *commit_ex;\n \tstruct configured_exclusion *tree_ex;\n \tstruct configured_exclusion *ex;\n+\tstruct object_list *p;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1355,14 +1357,27 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n+\t\tif (referred_objs && referred_objs->tags) {\n+\t\t\tfor (p = referred_objs->tags; p; p = p->next) {\n+\t\t\t\ttag_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n+\t\t\t\tif (match_packfile_uri_exclusions(tag_ex))\n+\t\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (referred_objs && referred_objs->commit) {\n \t\t\tcommit_ex = oidmap_get(&configured_exclusions, &referred_objs->commit->oid);\n \t\t\tif (match_packfile_uri_exclusions(commit_ex))\n \t\t\t\treturn 0;\n+\t\t\tstruct commit *commit = (struct commit*) referred_objs->commit;\n+\t\t\tfor (p = commit->wraps; p; p = p->next) {\n+\t\t\t\ttag_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n+\t\t\t\tif (match_packfile_uri_exclusions(tag_ex))\n+\t\t\t\t\treturn 0;\n+\t\t\t}\n \t\t}\n \n \t\tif (referred_objs && referred_objs->trees) {\n-\t\t\tstruct object_list *p;\n \t\t\tfor (p = referred_objs->trees; p; p = p->next) {\n \t\t\t\ttree_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n \t\t\t\tif (match_packfile_uri_exclusions(tree_ex))\n@@ -3260,7 +3275,8 @@ static void read_object_list_from_stdin(void)\n \n static void show_commit(struct commit *commit, void *show_data, void *carry_data)\n {\n-\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n+\tstruct referred_objects *referred_objs = carry_data;\n+\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, referred_objs);\n \tcommit->object.flags |= OBJECT_ADDED;\n \n \tif (write_bitmap_index)\ndiff --git a/list-objects.c b/list-objects.c\nindex 2e53a01458..52f38c9151 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -367,6 +367,7 @@ static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n+\tstruct referred_objects *referred_objs;\n \tstrbuf_init(&csp, PATH_MAX);\n \n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n@@ -384,7 +385,13 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data, NULL);\n+\t\treferred_objs = xmalloc(sizeof(struct referred_objects));\n+\t\treferred_objs->commit = NULL;\n+\t\treferred_objs->trees = NULL;\n+\t\treferred_objs->tags = commit->wraps;\n+\n+\t\tctx->show_commit(commit, ctx->show_data, referred_objs);\n+\t\tfree(referred_objs);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433705","messageId":"c83db0055cf71d4db6893379f5fb41958f586001.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 12/14] packfile-uri.txt: support for excluding tag objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:56Z","receivedAt":"2021-08-25T02:22:46Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n Documentation/technical/packfile-uri.txt | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt\nindex c3e4873956..bf5939cb8d 100644\n--- a/Documentation/technical/packfile-uri.txt\n+++ b/Documentation/technical/packfile-uri.txt\n@@ -39,12 +39,18 @@ server to be configured by one or more entries with the format:\n \n uploadpack.excludeobject= \n \n-Value is the key of entry, and the object type can be a blob,\n-tree, or commit. The exclusion of tree and commit is recursive by default,\n-which means that when a tree or commit object is excluded, the object itself\n-and all reachable objects of the object will be excluded recursively. Whenever\n-the list of objects to be sent is assembled, all such objects are excluded,\n-replaced with URIs.\n+Value is the key of entry, and the object type can be blob, tree,\n+commit, or tag. When an object is configured with `uploadpack.excludeobject` which\n+means that whenever the list of objects to be sent is assembled, the object (also\n+include the related objects in some cases, the following will introduce) will be\n+excluded, replaced with URIS. The mechanism for exclusion is as follows:\n+\n+\t* blob: exclude blob object.\n+\t* tree: exclude tree object, blobs that the tree list, and recursive into sub-trees.\n+\t* commit: exclude commit object, and recursively exclude all the reachable trees\n+\t (ditto tree exclusion) and blobs it contains.\n+\t* tag: exclude tag object itself, and the dereference commit (ditto commit exclusion)\n+\t if the tag is annotated.\n \n Configuration compatibility\n -------------\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433706","messageId":"29a52b7a0d8a2169d19aabed3a48e59dd9ae3d34.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 13/14] t5702: add tag exclusion test case","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:57Z","receivedAt":"2021-08-25T02:22:47Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 18 ++++++++++++++++++\n 1 file changed, 18 insertions(+)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 5ad52e0cee..09683ebfdc 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -1142,6 +1142,24 @@ test_expect_success 'commit-exclusion: part of packfile response provided as URI\n \t\thttp_child/.git/objects/pack/*.idx >filelist &&\n \ttest_line_count = 4 filelist\n '\n+test_expect_success 'tag-exclusion: part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+ \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n+\tgit -C \"$P\" tag -a -m \"annotated_tag\" tagA &&\n+\ttagObj=$(git -C \"$P\" rev-parse tagA) &&\n+\tconfigure_exclusion tag \"$P\" \"$tagObj\" >h2 &&\n+\tgit init http_child &&\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=`pwd`/log GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child \\\n+\t\t-c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch --tags \"$HTTPD_URL/smart/http_parent\"\n+'\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433707","messageId":"1a7c4c58946cc6aefe07021f366d95f174c7da21.1629805396.git.dyroneteng@gmail.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v5 14/14] pack-objects.c: introduce `want_exclude_object` function","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-08-25T02:21:58Z","receivedAt":"2021-08-25T02:22:51Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n builtin/pack-objects.c | 52 ++++++++++++++++++++----------------------\n 1 file changed, 25 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d5e3f2c229..503f8a5746 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1206,6 +1206,21 @@ static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n \treturn 0;\n }\n \n+static int want_exclude_object(struct object_list *objects)\n+{\n+\tstruct object_list *p;\n+\tstruct configured_exclusion *ex;\n+\n+\tif (!objects)\n+\t\treturn 0;\n+\tfor (p = objects; p; p = p->next) {\n+\t\tex = oidmap_get(&configured_exclusions, &p->item->oid);\n+\t\tif (match_packfile_uri_exclusions(ex))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static int want_found_object(const struct object_id *oid, int exclude,\n \t\t\t struct packed_git *p)\n {\n@@ -1317,11 +1332,8 @@ static int want_object_in_pack(const struct object_id *oid,\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n-\tstruct configured_exclusion *tag_ex;\n \tstruct configured_exclusion *commit_ex;\n-\tstruct configured_exclusion *tree_ex;\n \tstruct configured_exclusion *ex;\n-\tstruct object_list *p;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1357,41 +1369,27 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n-\t\tif (referred_objs && referred_objs->tags) {\n-\t\t\tfor (p = referred_objs->tags; p; p = p->next) {\n-\t\t\t\ttag_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n-\t\t\t\tif (match_packfile_uri_exclusions(tag_ex))\n+\t\tif (referred_objs) {\n+\t\t\tif (referred_objs->commit) {\n+\t\t\t\tstruct commit *commit = (struct commit *) referred_objs->commit;\n+\t\t\t\tcommit_ex = oidmap_get(&configured_exclusions, &commit->object.oid);\n+\t\t\t\tif (match_packfile_uri_exclusions(commit_ex))\n \t\t\t\t\treturn 0;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (referred_objs && referred_objs->commit) {\n-\t\t\tcommit_ex = oidmap_get(&configured_exclusions, &referred_objs->commit->oid);\n-\t\t\tif (match_packfile_uri_exclusions(commit_ex))\n-\t\t\t\treturn 0;\n-\t\t\tstruct commit *commit = (struct commit*) referred_objs->commit;\n-\t\t\tfor (p = commit->wraps; p; p = p->next) {\n-\t\t\t\ttag_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n-\t\t\t\tif (match_packfile_uri_exclusions(tag_ex))\n+\t\t\t\tif (want_exclude_object(commit->wraps))\n \t\t\t\t\treturn 0;\n \t\t\t}\n-\t\t}\n+\t\t\tif (referred_objs->tags && want_exclude_object(referred_objs->tags))\n+\t\t\t\treturn 0;\n \n-\t\tif (referred_objs && referred_objs->trees) {\n-\t\t\tfor (p = referred_objs->trees; p; p = p->next) {\n-\t\t\t\ttree_ex = oidmap_get(&configured_exclusions, &p->item->oid);\n-\t\t\t\tif (match_packfile_uri_exclusions(tree_ex))\n-\t\t\t\t\treturn 0;\n-\t\t\t}\n+\t\t\tif (referred_objs->trees && want_exclude_object(referred_objs->trees))\n+\t\t\t\treturn 0;\n \t\t}\n-\n \t\tex = oidmap_get(&configured_exclusions, oid);\n \t\tif (match_packfile_uri_exclusions(ex)) {\n \t\t\toidset_insert(&excluded_by_config, oid);\n \t\t\treturn 0;\n \t\t}\n \t}\n-\n \treturn 1;\n }\n \n-- \n2.31.1.456.gec51e24953\n\n"},{"id":"433791","messageId":"875yvtccd8.fsf@evledraar.gmail.com","threadId":"55638","inReplyTo":"f71b3108428b6ba36667e5e8a311ab8ab973d10c.1629805396.git.dyroneteng@gmail.com","subject":"Re: [PATCH v5 03/14] packfile-uri: support for excluding commit objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-25T23:49:06Z","receivedAt":"2021-08-25T23:51:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 25 2021, Teng Long wrote:\n\n> Currently packfile-uri supports the exclusion of blob objects, but in\n> some scenarios, users may wish to exclude more types of objects, such as\n> commit and tree objects, not only because packfile itself supports\n> storing these object types, but also on the other hand, to make\n> configuration items maintainable and simpler.\n>\n> This commit is used to support the recursive exclusion of a commit\n> object, which means that if the exclusion of a commit is configured as\n> packfile-uri, the commit itself and all the objects it contains will\n> also be recursively excluded. In addition, to support this feature, a\n> new configuration `uploadpack.excludeobject` is introduced.\n>\n> The reason for bringing a new configuration is for two considerations.\n> First, the old configuration supports a single object type (blob), which\n> limits the use of this feature. Secondly, the name of the old\n> configuration is not abstract enough, this make extension difficult. If\n> different object types use different configuration names, the\n> configuration items will be bloated and difficult to maintain, so the\n> new configuration is more abstract in name and easy to extend.\n>\n> Although a new configuration has been introduced, the old one is\n> still available and compatible with the new configuration. The old\n> configuration `uploadpack.blobpackfileuri` only supports excluding\n> blobs. The new configuration `uploadpack.excludeobject` not only\n> supports excluding blob objects, but also supports excluding commit\n> objects, as well as recursively excluding tree objects and blob objects\n> they contain.\n\nI was under the impression that with uploadpack.blobpackfileuri we\nalready supported excluding non-blobs, it was just unfortunately\nnamed. Perhaps I'm conflating that with the protocol payload for\npackfile-uri, which I know doesn't only support excluding blobs.\n\nWhat we didn't support at all was a way to have the server-side\nmechanism in git.git recursively exclude anything, which I think is what\nyou're adding here...\n\n> Signed-off-by: Teng Long \n> ---\n> builtin/pack-objects.c | 50 +++++++++++++++++++++++++-----------------\n> list-objects.c | 37 +++++++++++++++++--------------\n> object.c | 15 ++++++++++---\n> object.h | 4 +++-\n> revision.c | 34 ++++++++++++++++++++--------\n> revision.h | 3 +++\n> upload-pack.c | 7 ++++++\n> 7 files changed, 101 insertions(+), 49 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 5f9ec3566f..63f3aed70a 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1311,11 +1311,14 @@ static int want_object_in_pack_one(struct packed_git *p,\n> static int want_object_in_pack(const struct object_id *oid,\n> \t\t\t int exclude,\n> \t\t\t struct packed_git **found_pack,\n> -\t\t\t off_t *found_offset)\n> +\t\t\t off_t *found_offset,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tint want;\n> \tstruct list_head *pos;\n> \tstruct multi_pack_index *m;\n> +\tstruct configured_exclusion *ex;\n> +\tstruct configured_exclusion *referred_ex;\n> \n> \tif (!exclude && local && has_loose_object_nonlocal(oid))\n> \t\treturn 0;\n> @@ -1351,9 +1354,13 @@ static int want_object_in_pack(const struct object_id *oid,\n> \t}\n> \n> \tif (uri_protocols.nr) {\n> -\t\tstruct configured_exclusion *ex =\n> -\t\t\toidmap_get(&configured_exclusions, oid);\n> +\t\tif (referred_commit) {\n> +\t\t\treferred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid);\n> +\t\t\tif (referred_ex && match_packfile_uri_exclusions(referred_ex))\n> +\t\t\t\treturn 0;\n> +\t\t}\n> \n> +\t\tex = oidmap_get(&configured_exclusions, oid);\n> \t\tif (ex && match_packfile_uri_exclusions(ex)) {\n> \t\t\toidset_insert(&excluded_by_config, oid);\n> \t\t\treturn 0;\n> @@ -1393,7 +1400,8 @@ static const char no_closure_warning[] = N_(\n> );\n> \n> static int add_object_entry(const struct object_id *oid, enum object_type type,\n> -\t\t\t const char *name, int exclude)\n> +\t\t\t const char *name, int exclude,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tstruct packed_git *found_pack = NULL;\n> \toff_t found_offset = 0;\n> @@ -1403,7 +1411,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n> \tif (have_duplicate_entry(oid, exclude))\n> \t\treturn 0;\n> \n> -\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) {\n> +\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) {\n> \t\t/* The pack is missing an object, so it will not have closure */\n> \t\tif (write_bitmap_index) {\n> \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n> @@ -1429,7 +1437,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n> \tif (have_duplicate_entry(oid, 0))\n> \t\treturn 0;\n> \n> -\tif (!want_object_in_pack(oid, 0, &pack, &offset))\n> +\tif (!want_object_in_pack(oid, 0, &pack, &offset, NULL))\n> \t\treturn 0;\n> \n> \tcreate_object_entry(oid, type, name_hash, 0, 0, pack, offset);\n> @@ -1569,7 +1577,7 @@ static void add_pbase_object(struct tree_desc *tree,\n> \t\tif (name[cmplen] != '/') {\n> \t\t\tadd_object_entry(&entry.oid,\n> \t\t\t\t\t object_type(entry.mode),\n> -\t\t\t\t\t fullname, 1);\n> +\t\t\t\t\t fullname, 1, NULL);\n> \t\t\treturn;\n> \t\t}\n> \t\tif (S_ISDIR(entry.mode)) {\n> @@ -1637,7 +1645,7 @@ static void add_preferred_base_object(const char *name)\n> \tcmplen = name_cmp_len(name);\n> \tfor (it = pbase_tree; it; it = it->next) {\n> \t\tif (cmplen == 0) {\n> -\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1);\n> +\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL);\n> \t\t}\n> \t\telse {\n> \t\t\tstruct tree_desc tree;\n> @@ -2839,7 +2847,7 @@ static void add_tag_chain(const struct object_id *oid)\n> \t\t\tdie(_(\"unable to pack objects reachable from tag %s\"),\n> \t\t\t oid_to_hex(oid));\n> \n> -\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0);\n> +\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL);\n> \n> \t\tif (tag->tagged->type != OBJ_TAG)\n> \t\t\treturn;\n> @@ -2994,7 +3002,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n> \t\treturn 0;\n> \t}\n> -\tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n> +\tif (!strcmp(k, \"uploadpack.excludeobject\") || !strcmp(k, \"uploadpack.blobpackfileuri\")) {\n> \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n> \t\tconst char *oid_end, *pack_end;\n> \t\t/*\n> @@ -3007,11 +3015,11 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t *oid_end != ' ' ||\n> \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n> \t\t *pack_end != ' ')\n> -\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n> +\t\t\tdie(_(\"value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be \"\n> \t\t\t \"of the form ' ' (got '%s')\"), v);\n> \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n> -\t\t\tdie(_(\"object already configured in another \"\n> -\t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n> +\t\t\tdie(_(\"object already configured by an earlier \"\n> +\t\t\t \"uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')\"), v);\n> \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n> \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n> \t\tex->uri = xstrdup(pack_end + 1);\n> @@ -3040,7 +3048,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t\treturn 0;\n> \n> \tofs = nth_packed_object_offset(p, pos);\n> -\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n> +\tif (!want_object_in_pack(oid, 0, &p, &ofs, NULL))\n> \t\treturn 0;\n> \n> \toi.typep = &type;\n> @@ -3233,7 +3241,7 @@ static void read_object_list_from_stdin(void)\n> \t\t\tdie(_(\"expected object ID, got garbage:\\n %s\"), line);\n> \n> \t\tadd_preferred_base_object(p + 1);\n> -\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n> +\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL);\n> \t}\n> }\n> \n> @@ -3242,7 +3250,7 @@ static void read_object_list_from_stdin(void)\n> \n> static void show_commit(struct commit *commit, void *data)\n> {\n> -\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0);\n> +\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n> \tcommit->object.flags |= OBJECT_ADDED;\n> \n> \tif (write_bitmap_index)\n> @@ -3254,8 +3262,9 @@ static void show_commit(struct commit *commit, void *data)\n> \n> static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data)\n> {\n> +\tstruct object *referred_commit = carry_data;\n> \tadd_preferred_base_object(name);\n> -\tadd_object_entry(&obj->oid, obj->type, name, 0);\n> +\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_commit);\n> \tobj->flags |= OBJECT_ADDED;\n> \n> \tif (use_delta_islands) {\n> @@ -3406,7 +3415,7 @@ static void add_objects_in_unpacked_packs(void)\n> \t\tQSORT(in_pack.array, in_pack.nr, ofscmp);\n> \t\tfor (i = 0; i < in_pack.nr; i++) {\n> \t\t\tstruct object *o = in_pack.array[i].object;\n> -\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0);\n> +\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0, NULL);\n> \t\t}\n> \t}\n> \tfree(in_pack.array);\n> @@ -3422,7 +3431,7 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n> \t\treturn 0;\n> \t}\n> \n> -\tadd_object_entry(oid, type, \"\", 0);\n> +\tadd_object_entry(oid, type, \"\", 0, NULL);\n> \treturn 0;\n> }\n> \n> @@ -3841,7 +3850,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \t\t\t N_(\"respect islands during delta compression\")),\n> \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n> \t\t\t\tN_(\"protocol\"),\n> -\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n> +\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject or \"\n> +\t\t\t\t \"uploadpack.blobpackfileuri with this protocol\")),\n> \t\tOPT_END(),\n> \t};\n> \n> diff --git a/list-objects.c b/list-objects.c\n> index 427228a3ba..968d842ceb 100644\n> --- a/list-objects.c\n> +++ b/list-objects.c\n> @@ -24,7 +24,8 @@ struct traversal_context {\n> static void process_blob(struct traversal_context *ctx,\n> \t\t\t struct blob *blob,\n> \t\t\t struct strbuf *path,\n> -\t\t\t const char *name)\n> +\t\t\t const char *name,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tstruct object *obj = &blob->object;\n> \tsize_t pathlen;\n> @@ -60,7 +61,7 @@ static void process_blob(struct traversal_context *ctx,\n> \tif (r & LOFR_MARK_SEEN)\n> \t\tobj->flags |= SEEN;\n> \tif (r & LOFR_DO_SHOW)\n> -\t\tctx->show_object(obj, path->buf, ctx->show_data, NULL);\n> +\t\tctx->show_object(obj, path->buf, ctx->show_data, referred_commit);\n> \tstrbuf_setlen(path, pathlen);\n> }\n> \n> @@ -97,11 +98,13 @@ static void process_gitlink(struct traversal_context *ctx,\n> static void process_tree(struct traversal_context *ctx,\n> \t\t\t struct tree *tree,\n> \t\t\t struct strbuf *base,\n> -\t\t\t const char *name);\n> +\t\t\t const char *name,\n> +\t\t\t struct object *referred_commit);\n> \n> static void process_tree_contents(struct traversal_context *ctx,\n> \t\t\t\t struct tree *tree,\n> -\t\t\t\t struct strbuf *base)\n> +\t\t\t\t struct strbuf *base,\n> +\t\t\t\t struct object *referred_commit)\n> {\n> \tstruct tree_desc desc;\n> \tstruct name_entry entry;\n> @@ -129,7 +132,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n> \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n> \t\t\t}\n> \t\t\tt->object.flags |= NOT_USER_GIVEN;\n> -\t\t\tprocess_tree(ctx, t, base, entry.path);\n> +\t\t\tprocess_tree(ctx, t, base, entry.path, referred_commit);\n> \t\t}\n> \t\telse if (S_ISGITLINK(entry.mode))\n> \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n> @@ -142,7 +145,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n> \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n> \t\t\t}\n> \t\t\tb->object.flags |= NOT_USER_GIVEN;\n> -\t\t\tprocess_blob(ctx, b, base, entry.path);\n> +\t\t\tprocess_blob(ctx, b, base, entry.path, referred_commit);\n> \t\t}\n> \t}\n> }\n> @@ -150,7 +153,8 @@ static void process_tree_contents(struct traversal_context *ctx,\n> static void process_tree(struct traversal_context *ctx,\n> \t\t\t struct tree *tree,\n> \t\t\t struct strbuf *base,\n> -\t\t\t const char *name)\n> +\t\t\t const char *name,\n> +\t\t\t struct object *referred_commit)\n> {\n> \tstruct object *obj = &tree->object;\n> \tstruct rev_info *revs = ctx->revs;\n> @@ -191,14 +195,14 @@ static void process_tree(struct traversal_context *ctx,\n> \tif (r & LOFR_MARK_SEEN)\n> \t\tobj->flags |= SEEN;\n> \tif (r & LOFR_DO_SHOW)\n> -\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n> +\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n> \tif (base->len)\n> \t\tstrbuf_addch(base, '/');\n> \n> \tif (r & LOFR_SKIP_TREE)\n> \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n> \telse if (!failed_parse)\n> -\t\tprocess_tree_contents(ctx, tree, base);\n> +\t\tprocess_tree_contents(ctx, tree, base, referred_commit);\n> \n> \tr = list_objects_filter__filter_object(ctx->revs->repo,\n> \t\t\t\t\t LOFS_END_TREE, obj,\n> @@ -207,7 +211,7 @@ static void process_tree(struct traversal_context *ctx,\n> \tif (r & LOFR_MARK_SEEN)\n> \t\tobj->flags |= SEEN;\n> \tif (r & LOFR_DO_SHOW)\n> -\t\tctx->show_object(obj, base->buf, ctx->show_data, NULL);\n> +\t\tctx->show_object(obj, base->buf, ctx->show_data, referred_commit);\n> \n> \tstrbuf_setlen(base, baselen);\n> \tfree_tree_buffer(tree);\n> @@ -314,9 +318,9 @@ void mark_edges_uninteresting(struct rev_info *revs,\n> \t}\n> }\n> \n> -static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n> +static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit)\n> {\n> -\tadd_pending_object(revs, &tree->object, \"\");\n> +\tadd_pending_object_with_referred_commit(revs, &tree->object, \"\", referred_commit);\n> }\n> \n> static void traverse_trees_and_blobs(struct traversal_context *ctx,\n> @@ -329,23 +333,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n> \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n> \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n> \t\tstruct object *obj = pending->item;\n> +\t\tstruct object *referred_commit = pending->referred_commit;\n> \t\tconst char *name = pending->name;\n> \t\tconst char *path = pending->path;\n> \t\tif (obj->flags & (UNINTERESTING | SEEN))\n> \t\t\tcontinue;\n> \t\tif (obj->type == OBJ_TAG) {\n> \t\t\tobj->flags |= SEEN;\n> -\t\t\tctx->show_object(obj, name, ctx->show_data, NULL);\n> +\t\t\tctx->show_object(obj, name, ctx->show_data, referred_commit);\n> \t\t\tcontinue;\n> \t\t}\n> \t\tif (!path)\n> \t\t\tpath = \"\";\n> \t\tif (obj->type == OBJ_TREE) {\n> -\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n> +\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_commit);\n> \t\t\tcontinue;\n> \t\t}\n> \t\tif (obj->type == OBJ_BLOB) {\n> -\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n> +\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_commit);\n> \t\t\tcontinue;\n> \t\t}\n> \t\tdie(\"unknown pending object %s (%s)\",\n> @@ -370,7 +375,7 @@ static void do_traverse(struct traversal_context *ctx)\n> \t\telse if (get_commit_tree(commit)) {\n> \t\t\tstruct tree *tree = get_commit_tree(commit);\n> \t\t\ttree->object.flags |= NOT_USER_GIVEN;\n> -\t\t\tadd_pending_tree(ctx->revs, tree);\n> +\t\t\tadd_pending_tree(ctx->revs, tree, &commit->object);\n> \t\t} else if (commit->object.parsed) {\n> \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n> \t\t\t oid_to_hex(&commit->object.oid));\n> diff --git a/object.c b/object.c\n> index 14188453c5..6b1ce2fcde 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -322,9 +322,10 @@ void object_list_free(struct object_list **list)\n> */\n> static char object_array_slopbuf[1];\n> \n> -void add_object_array_with_path(struct object *obj, const char *name,\n> -\t\t\t\tstruct object_array *array,\n> -\t\t\t\tunsigned mode, const char *path)\n> +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name,\n> +\t\t\t\t\t\t struct object_array *array,\n> +\t\t\t\t\t\t unsigned mode, const char *path,\n> +\t\t\t\t\t\t struct object *referred_commit)\n> {\n> \tunsigned nr = array->nr;\n> \tunsigned alloc = array->alloc;\n> @@ -339,6 +340,7 @@ void add_object_array_with_path(struct object *obj, const char *name,\n> \t}\n> \tentry = &objects[nr];\n> \tentry->item = obj;\n> +\tentry->referred_commit = referred_commit;\n> \tif (!name)\n> \t\tentry->name = NULL;\n> \telse if (!*name)\n> @@ -354,6 +356,13 @@ void add_object_array_with_path(struct object *obj, const char *name,\n> \tarray->nr = ++nr;\n> }\n> \n> +void add_object_array_with_path(struct object *obj, const char *name,\n> +\t\t\t\tstruct object_array *array,\n> +\t\t\t\tunsigned mode, const char *path)\n> +{\n> +\tadd_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL);\n> +}\n> +\n> void add_object_array(struct object *obj, const char *name, struct object_array *array)\n> {\n> \tadd_object_array_with_path(obj, name, array, S_IFINVALID, NULL);\n> diff --git a/object.h b/object.h\n> index 87a6da47c8..d63819ab91 100644\n> --- a/object.h\n> +++ b/object.h\n> @@ -52,6 +52,7 @@ struct object_array {\n> \t\tchar *name;\n> \t\tchar *path;\n> \t\tunsigned mode;\n> +\t\tstruct object *referred_commit;\n> \t} *objects;\n> };\n> \n> @@ -157,7 +158,8 @@ void object_list_free(struct object_list **list);\n> /* Object array handling .. */\n> void add_object_array(struct object *obj, const char *name, struct object_array *array);\n> void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path);\n> -\n> +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array,\n> +\t\t\t\t\t\t unsigned mode, const char *path, struct object *referred_commit);\n> /*\n> * Returns NULL if the array is empty. Otherwise, returns the last object\n> * after removing its entry from the array. Other resources associated\n> diff --git a/revision.c b/revision.c\n> index 4853c85d0b..65e0926d25 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit)\n> \tcommit_stack_clear(&pending);\n> }\n> \n> -static void add_pending_object_with_path(struct rev_info *revs,\n> -\t\t\t\t\t struct object *obj,\n> -\t\t\t\t\t const char *name, unsigned mode,\n> -\t\t\t\t\t const char *path)\n> +static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs,\n> +\t\t\t\t\t\t\t struct object *obj,\n> +\t\t\t\t\t\t\t const char *name, unsigned mode,\n> +\t\t\t\t\t\t\t const char *path,\n> +\t\t\t\t\t\t\t struct object *referred_commit)\n> {\n> \tstruct interpret_branch_name_options options = { 0 };\n> \tif (!obj)\n> @@ -326,20 +327,35 @@ static void add_pending_object_with_path(struct rev_info *revs,\n> \t\tstrbuf_release(&buf);\n> \t\treturn; /* do not add the commit itself */\n> \t}\n> -\tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n> +\tadd_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit);\n> }\n> \n> +static void add_pending_object_with_path(struct rev_info *revs,\n> +\t\t\t\t\t struct object *obj,\n> +\t\t\t\t\t const char *name, unsigned mode,\n> +\t\t\t\t\t const char *path)\n> +{\n> +\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL);\n> +}\n> static void add_pending_object_with_mode(struct rev_info *revs,\n> \t\t\t\t\t struct object *obj,\n> -\t\t\t\t\t const char *name, unsigned mode)\n> +\t\t\t\t\t const char *name, unsigned mode,\n> +\t\t\t\t\t struct object *referred_commit)\n> +{\n> +\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit);\n> +}\n> +\n> +void add_pending_object_with_referred_commit(struct rev_info *revs,\n> +\t\t\t\t\t struct object *obj, const char *name,\n> +\t\t\t\t\t struct object *referred_commit)\n> {\n> -\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n> +\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit);\n> }\n> \n> void add_pending_object(struct rev_info *revs,\n> \t\t\tstruct object *obj, const char *name)\n> {\n> -\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID);\n> +\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL);\n> }\n> \n> void add_head_to_pending(struct rev_info *revs)\n> @@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n> \t\tif (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc))\n> \t\t\tdiagnose_missing_default(revs->def);\n> \t\tobject = get_reference(revs, revs->def, &oid, 0);\n> -\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode);\n> +\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL);\n> \t}\n> \n> \t/* Did the user ask for any diff output? Run the diff! */\n> diff --git a/revision.h b/revision.h\n> index a24f72dcd1..f9c9628ed8 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -423,6 +423,9 @@ void show_object_with_name(FILE *, struct object *, const char *);\n> */\n> void add_pending_object(struct rev_info *revs,\n> \t\t\tstruct object *obj, const char *name);\n> +void add_pending_object_with_referred_commit(struct rev_info *revs,\n> +\t\t\t\t\t struct object *obj, const char *name,\n> +\t\t\t\t\t struct object *referred_commit);\n> \n> void add_pending_oid(struct rev_info *revs,\n> \t\t const char *name, const struct object_id *oid,\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 5c1cd19612..d26fb351a3 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r,\n> \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n> \t\t\tfree(str);\n> \t\t}\n> +\n> +\t\tif (!repo_config_get_string(the_repository,\n> +\t\t\t\t\t \"uploadpack.excludeobject\",\n> +\t\t\t\t\t &str) && str) {\n> +\t\t\tstrbuf_addstr(value, \" packfile-uris\");\n> +\t\t\tfree(str);\n> +\t\t}\n> \t}\n> \n> \treturn 1;\n\n"},{"id":"433792","messageId":"8735qxccay.fsf@evledraar.gmail.com","threadId":"55638","inReplyTo":"8e5bf4010cb0d7e8b4f6c7285fe1f5365af955e8.1629805396.git.dyroneteng@gmail.com","subject":"Re: [PATCH v5 05/14] packfile-uri.txt: support for excluding commits and trees","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-25T23:52:11Z","receivedAt":"2021-08-25T23:52:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 25 2021, Teng Long wrote:\n\n> Client design\n> -------------\n> @@ -65,9 +78,6 @@ The protocol design allows some evolution of the server and client without any\n> need for protocol changes, so only a small-scoped design is included here to\n> form the MVP. For example, the following can be done:\n> \n> - * On the server, more sophisticated means of excluding objects (e.g. by\n> - specifying a commit to represent that commit and all objects that it\n> - references).\n> * On the client, resumption of clone. If a clone is interrupted, information\n> could be recorded in the repository's config and a \"clone-resume\" command\n> can resume the clone in progress. (Resumption of subsequent fetches is more\n> @@ -78,4 +88,4 @@ There are some possible features that will require a change in protocol:\n> \n> * Additional HTTP headers (e.g. authentication)\n> * Byte range support\n> - * Different file formats referenced by URIs (e.g. raw object)\n> + * Different file formats referenced by URIs (e.g. raw object)\n> \\ No newline at end of file\n\nNewline churn?\n"},{"id":"433793","messageId":"87zgt5axjv.fsf@evledraar.gmail.com","threadId":"55638","inReplyTo":"f3b1cba7e102b3132f04b27be48e532da7d448c9.1629805396.git.dyroneteng@gmail.com","subject":"Re: [PATCH v5 06/14] t5702: replace with \"test_when_finished\" for cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-25T23:55:36Z","receivedAt":"2021-08-25T23:56:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 25 2021, Teng Long wrote:\n\nThanks, much needed cleanup. I have an unsubmitted patch to do pretty\nmuch this, plus some: https://github.com/avar/git/commit/27b3543c6ed\n\nYou might find the difference between the two interesting..\n\n> -test_done\n> +test_done\n> \\ No newline at end of file\n\nMore newline churn.\n"},{"id":"433794","messageId":"87wno9axfs.fsf@evledraar.gmail.com","threadId":"55638","inReplyTo":"011e5eaea3f48eeacac1614f769e8fd809e0c093.1629805396.git.dyroneteng@gmail.com","subject":"Re: [PATCH v5 09/14] commit.h: add wrapped tags in commit struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-25T23:58:23Z","receivedAt":"2021-08-25T23:58:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 25 2021, Teng Long wrote:\n\n> Signed-off-by: Teng Long \n> ---\n> commit.h | 5 +++++\n> revision.c | 8 ++++++--\n> 2 files changed, 11 insertions(+), 2 deletions(-)\n>\n> diff --git a/commit.h b/commit.h\n> index df42eb434f..1e99e9ae8a 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -38,6 +38,11 @@ struct commit {\n> \t */\n> \tstruct tree *maybe_tree;\n> \tunsigned int index;\n> +\t/*\n> +\t* wrapped tags or NULL. If the commit is peeled from tag(s),\n> +\t* then save the wraps, otherwise will be NULL.\n> +\t*/\n> +\tstruct object_list *wraps;\n> };\n> \n> extern int save_commit_buffer;\n> diff --git a/revision.c b/revision.c\n> index 65e0926d25..aecf493f46 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -416,14 +416,17 @@ static struct commit *handle_commit(struct rev_info *revs,\n> \tconst char *path = entry->path;\n> \tunsigned int mode = entry->mode;\n> \tunsigned long flags = object->flags;\n> -\n> +\tstruct object_list *wraps = NULL;\n> \t/*\n> \t * Tag object? Look what it points to..\n> \t */\n> \twhile (object->type == OBJ_TAG) {\n> \t\tstruct tag *tag = (struct tag *) object;\n> -\t\tif (revs->tag_objects && !(flags & UNINTERESTING))\n> +\t\tif (revs->tag_objects && !(flags & UNINTERESTING)) {\n> +\t\t\tobject_list_insert(object, &wraps);\n> \t\t\tadd_pending_object(revs, object, tag->tag);\n> +\t\t}\n> +\n> \t\tobject = parse_object(revs->repo, get_tagged_oid(tag));\n> \t\tif (!object) {\n> \t\t\tif (revs->ignore_missing_links || (flags & UNINTERESTING))\n> @@ -449,6 +452,7 @@ static struct commit *handle_commit(struct rev_info *revs,\n> \t */\n> \tif (object->type == OBJ_COMMIT) {\n> \t\tstruct commit *commit = (struct commit *)object;\n> +\t\tcommit->wraps = wraps;\n> \n> \t\tif (repo_parse_commit(revs->repo, commit) < 0)\n> \t\t\tdie(\"unable to parse commit %s\", name);\n\nCan't we store this info on the side between these two static functions\nsomehow, instead of adding this \"wraps\" to all commit structs?\n"},{"id":"433870","messageId":"xmqqlf4oc4u1.fsf@gitster.g","threadId":"55638","inReplyTo":"bc8fea97e34b3ccb4b62bb4d48dac6e1c009236c.1629805396.git.dyroneteng@gmail.com","subject":"Re: [PATCH v5 02/14] Add new parameter \"carry_data\" for \"show_object\" function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-26T20:45:58Z","receivedAt":"2021-08-26T20:46:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Teng Long writes:\n\n> Subject: Re: [PATCH v5 02/14] Add new parameter \"carry_data\" for \"show_object\" function\n\nSince this lacks : prefix, \"git shortlog\" readers will have a\nhard time guessing which show_object() function this commit is\nabout.\n\n> During the pack-objects process, \"show_object\" function will be called\n> to find the object and show the process(\"show_object_fn\" in\n> \"list-object.h\"), the function definition contains three parameters:\n>\n> \t1. struct object *obj(contains object type, flags, and oid).\n> \t2. const char *name(the object name).\n> \t3. void *show_data(function to show progress info).\n>\n> This commit adds a new parameter: \"void *carry_data\", the reason is\n> mainly based on scalability and performance considerations when showing\n> an object, space for time, avoid costly temporary calculations in the\n> \"show\" phase. For example, carry the ownership relationship between\n> blob or tree object and the referred commit to avoid redundant and\n> expensive calculations.\n\nThe above explains what we want to carry around extra data for\n(i.e. compute something in one place, and use it later somewhere\nelse)\n\nBut it does not quite explain why we need another parameter to do\nso, which involves changing the function signature of many\nfunctions, instead of making show_data to point at a new structure\ntype that holds the original data show_data used to carry plus\nanother single void * member (or the set of members you'd be\ncarrying into these functions using this new parameter).\n\nI also find \"carry_data\" a meaningless name for the parameter. All\nin-parameters into functions are used to carry some data into it\nafter all. The existing \"show_data\" at least makes a bit more\nsense; it contains data necessary for showing the object in these\ncode paths. If the purpose this new thing was introduced is to\ncache ownership relationship data, perhaps ownership_cache would be\na more descriptive and understandable name (be it a new parameter to\nadded to many functions, or a member to the new structure that\nreplaces show_data).\n\nThanks.\n"},{"id":"433871","messageId":"xmqqfsuwc4cl.fsf@gitster.g","threadId":"55638","inReplyTo":"f71b3108428b6ba36667e5e8a311ab8ab973d10c.1629805396.git.dyroneteng@gmail.com","subject":"Re: [PATCH v5 03/14] packfile-uri: support for excluding commit objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-26T20:56:26Z","receivedAt":"2021-08-26T20:56:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Teng Long writes:\n\n> This commit is used to support the recursive exclusion of a commit\n> object, which means that if the exclusion of a commit is configured as\n> packfile-uri, the commit itself and all the objects it contains will\n> also be recursively excluded.\n\nWhenever you say \"which means\", e.g. \"We do X, which means we do Y\",\nthink twice to see if you do not even have to say X. In this\nparticular sentence, I think you can simplify the description\ngreatly and you do not even have to use the word \"recursive\". \n\nAlso, because this \"exclusion by a commit\" does not work like the\nusual \"reachability\" relationship Git users are familiar with, it\nwould help to highlight what is special that is done here to the\nreaders.\n\nTaking the above together, along the lines of ...\n\n When a commit is specified to be excluded as packfile-uri,\n exclude all trees and blobs contained in its top-level tree, as\n well as the commit itself, but not the ancestors of the commit\n and objects that are reachable by them.\n\nor something like that, perhaps.\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 5f9ec3566f..63f3aed70a 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1311,11 +1311,14 @@ static int want_object_in_pack_one(struct packed_git *p,\n> static int want_object_in_pack(const struct object_id *oid,\n> \t\t\t int exclude,\n> \t\t\t struct packed_git **found_pack,\n> -\t\t\t off_t *found_offset)\n> +\t\t\t off_t *found_offset,\n> +\t\t\t struct object *referred_commit)\n\nAs the caller is limited to pass commit and no other types of\nobject, the new parameter should be of type \"struct commit\", no?\n"},{"id":"434513","messageId":"20210902110855.75687-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"xmqqlf4oc4u1.fsf@gitster.g","subject":"Re: [PATCH v5 02/14] Add new parameter \"carry_data\" for \"show_object\" function","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-09-02T11:08:55Z","receivedAt":"2021-09-02T11:09:07Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"> Since this lacks : prefix, \"git shortlog\" readers will have a\n> hard time guessing which show_object() function this commit is\n> about.\n\n\nSorry for the late reply.\nAgree, it will be fixed in next patch.\n\n> But it does not quite explain why we need another parameter to do\n> so, which involves changing the function signature of many\n> functions, instead of making show_data to point at a new structure\n> type that holds the original data show_data used to carry plus\n> another single void * member (or the set of members you'd be\n> carrying into these functions using this new parameter).\n> \n> I also find \"carry_data\" a meaningless name for the parameter. All\n> in-parameters into functions are used to carry some data into it\n> after all. The existing \"show_data\" at least makes a bit more\n> sense; it contains data necessary for showing the object in these\n> code paths. If the purpose this new thing was introduced is to\n> cache ownership relationship data, perhaps ownership_cache would be\n> a more descriptive and understandable name (be it a new parameter to\n> added to many functions, or a member to the new structure that\n> replaces show_data).\n\nAgree.\n\nI think \"show_data to point at a new structure\" is a better idea. I will\nfollow the idea and optimize the related code. By the way, the naming\nabout \"ownership_cache\" will be introduced in the next patch too, unless I\ncan find a better one t(-_-t).\n\nThanks.\n"},{"id":"434514","messageId":"20210902112355.76244-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"8735qxccay.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 05/14] packfile-uri.txt: support for excluding commits and trees","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-09-02T11:23:55Z","receivedAt":"2021-09-02T11:24:06Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"> Newline churn?\n\nAgree.\n\nI have no impression why the NEWLINE was deleted, will be fixed\nin the next patch.\n\nThank you.\n"},{"id":"434515","messageId":"20210902113740.76680-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"87zgt5axjv.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 06/14] t5702: replace with \"test_when_finished\" for cleanup","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-09-02T11:37:40Z","receivedAt":"2021-09-02T11:37:52Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"\n\n> Thanks, much needed cleanup. I have an unsubmitted patch to do pretty\n> much this, plus some: https://github.com/avar/git/commit/27b3543c6ed\n>\n> You might find the difference between the two interesting..\n\nCool.\n\nI had took a look at the commit. In comparision, I ignored some places that need\nto cleanup, I don't know if I got your meaning?\n\nIf so, how to deal with this situation better, continue my work or rebase your commit\nto the current patchset?\n\n>> -test_done\n>> +test_done\n>> \\ No newline at end of file\n>\n> More newline churn.\n\nWill be fixed in the next patch.\n\nThank you.\n"},{"id":"434517","messageId":"20210902121709.77198-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"87wno9axfs.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 09/14] commit.h: add wrapped tags in commit struct","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-09-02T12:17:09Z","receivedAt":"2021-09-02T12:17:23Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"\nÆvar Arnfjörð Bjarmason wrote:\n\n> Can't we store this info on the side between these two static functions\n> somehow, instead of adding this \"wraps\" to all commit structs?\n>\t \n\nThanks very much and I have some doubts.\n\n> ... instead of adding this \"wraps\" to all commit structs?\nI think \"adding this \"wraps\" to all commit struct\" is an easy but a little\nrough indeed. I didn't know if this is okay at the time. So I pushed the patch,\nhoping to ask some different opinions.\n\n> Can't we store this info on the side between these two static functions...\n\nDo you mean to use static storage to share the \"wraps\", or other way? I want to make\nsure that I understand your opinion accurately.\n\nThank you.\n"},{"id":"434518","messageId":"20210902122638.77513-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"875yvtccd8.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 03/14] packfile-uri: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-09-02T12:26:38Z","receivedAt":"2021-09-02T12:26:52Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n\n> I was under the impression that with uploadpack.blobpackfileuri we\n> already supported excluding non-blobs, it was just unfortunately\n> named. Perhaps I'm conflating that with the protocol payload for\n> packfile-uri, which I know doesn't only support excluding blobs.\n>\n> What we didn't support at all was a way to have the server-side\n> mechanism in git.git recursively exclude anything, which I think is what\n> you're adding here...\n>\n\nAgree.\n\nYou are absolutely right(ಥ_ಥ ). The description of the commit is inaccurate, I will\nfix this problem in the next patch.\n\nThank you.\n"},{"id":"434519","messageId":"CAOLTT8Qm2wGMU0RuX+OrzZt78zSmzxHNoh0NYUFuVKJSAZJ1Ng@mail.gmail.com","threadId":"55638","inReplyTo":"011e5eaea3f48eeacac1614f769e8fd809e0c093.1629805396.git.dyroneteng@gmail.com","subject":"Re: [PATCH v5 09/14] commit.h: add wrapped tags in commit struct","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-09-02T12:39:56Z","receivedAt":"2021-09-02T12:40:11Z","isPatch":true,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Teng Long 于2021年8月25日周三 上午10:24写道：\n>\n> Signed-off-by: Teng Long \n> ---\n> commit.h | 5 +++++\n> revision.c | 8 ++++++--\n> 2 files changed, 11 insertions(+), 2 deletions(-)\n>\n> diff --git a/commit.h b/commit.h\n> index df42eb434f..1e99e9ae8a 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -38,6 +38,11 @@ struct commit {\n> */\n> struct tree *maybe_tree;\n> unsigned int index;\n> + /*\n> + * wrapped tags or NULL. If the commit is peeled from tag(s),\n> + * then save the wraps, otherwise will be NULL.\n> + */\n> + struct object_list *wraps;\n> };\n>\n> extern int save_commit_buffer;\n> diff --git a/revision.c b/revision.c\n> index 65e0926d25..aecf493f46 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -416,14 +416,17 @@ static struct commit *handle_commit(struct rev_info *revs,\n> const char *path = entry->path;\n> unsigned int mode = entry->mode;\n> unsigned long flags = object->flags;\n> -\n> + struct object_list *wraps = NULL;\n> /*\n> * Tag object? Look what it points to..\n> */\n> while (object->type == OBJ_TAG) {\n> struct tag *tag = (struct tag *) object;\n> - if (revs->tag_objects && !(flags & UNINTERESTING))\n> + if (revs->tag_objects && !(flags & UNINTERESTING)) {\n> + object_list_insert(object, &wraps);\n> add_pending_object(revs, object, tag->tag);\n> + }\n> +\n> object = parse_object(revs->repo, get_tagged_oid(tag));\n> if (!object) {\n> if (revs->ignore_missing_links || (flags & UNINTERESTING))\n> @@ -449,6 +452,7 @@ static struct commit *handle_commit(struct rev_info *revs,\n> */\n> if (object->type == OBJ_COMMIT) {\n> struct commit *commit = (struct commit *)object;\n> + commit->wraps = wraps;\n>\n> if (repo_parse_commit(revs->repo, commit) < 0)\n> die(\"unable to parse commit %s\", name);\n> --\n> 2.31.1.456.gec51e24953\n>\n\n/*\n * The size of this struct matters in full repo walk operations like\n * 'git clone' or 'git gc'. Consider using commit-slab to attach data\n * to a commit instead of adding new fields here.\n */\nstruct commit {\n struct object object;\n timestamp_t date;\n struct commit_list *parents;\n\n /*\n * If the commit is loaded from the commit-graph file, then this\n * member may be NULL. Only access it through repo_get_commit_tree()\n * or get_commit_tree_oid().\n */\n struct tree *maybe_tree;\n unsigned int index;\n};\n\nAccording to the instructions above, I wonder if you should use \"commit_slab\" to\nstore part of the data related to the commit object instead of\nmodifying the member\nof struct commit itself?\n\nSee:\nhttps://github.com/git/git/blob/master/commit-slab.h\nhttps://github.com/git/git/blob/master/commit-slab-impl.h\nhttps://github.com/git/git/blob/master/commit-slab-decl.h\nhttps://lore.kernel.org/git/CAOLTT8Q8BEKCVwPDypW1w66P9_xP7QC0T-CnLqamqAL4haGzwA@mail.gmail.com/\n\nThanks.\n--\nZheNing Hu\n"},{"id":"434521","messageId":"20210902125102.78016-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"xmqqfsuwc4cl.fsf@gitster.g","subject":"Re: [PATCH v5 03/14] packfile-uri: support for excluding commit objects","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-09-02T12:51:02Z","receivedAt":"2021-09-02T12:51:13Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"\nJunio C Hamano wrote:\n\n> Whenever you say \"which means\", e.g. \"We do X, which means we do Y\",\n> think twice to see if you do not even have to say X. In this\n> particular sentence, I think you can simplify the description\n> greatly and you do not even have to use the word \"recursive\". \n\nForgive my English for another accident.\n\nWill be fixed in the next patch.\n\n> Also, because this \"exclusion by a commit\" does not work like the\n> usual \"reachability\" relationship Git users are familiar with, it\n> would help to highlight what is special that is done here to the\n> readers.\n> \n> Taking the above together, along the lines of ...\n> \n> When a commit is specified to be excluded as packfile-uri,\n> exclude all trees and blobs contained in its top-level tree, as\n> well as the commit itself, but not the ancestors of the commit\n> and objects that are reachable by them.\n> \n> or something like that, perhaps.\n\nAgree.\n\nI will use your suggestion in the next patch (may be slightly modified).\n\nIn the next patch, I originally planned to introduce the exclusion of the\ncommit and with all it's ancestor objects. The problems in the current patch\nwill also be fixed in the next patch.\n\nThanks.\n"},{"id":"434522","messageId":"20210902130139.78316-1-dyroneteng@gmail.com","threadId":"55638","inReplyTo":"CAOLTT8Qm2wGMU0RuX+OrzZt78zSmzxHNoh0NYUFuVKJSAZJ1Ng@mail.gmail.com","subject":"Re: [PATCH v5 09/14] commit.h: add wrapped tags in commit struct","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-09-02T13:01:39Z","receivedAt":"2021-09-02T13:01:52Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"\nZheNing Hu wrote:\n\n> /*\n> * The size of this struct matters in full repo walk operations like\n> * 'git clone' or 'git gc'. Consider using commit-slab to attach data\n> * to a commit instead of adding new fields here.\n> */\n> struct commit {\n> struct object object;\n> timestamp_t date;\n> struct commit_list *parents;\n> \n> /*\n> * If the commit is loaded from the commit-graph file, then this\n> * member may be NULL. Only access it through repo_get_commit_tree()\n> * or get_commit_tree_oid().\n> */\n> struct tree *maybe_tree;\n> unsigned int index;\n> };\n> \n> According to the instructions above, I wonder if you should use \"commit_slab\" to\n> store part of the data related to the commit object instead of\n> modifying the member\n> of struct commit itself?\n> \n> See:\n> https://github.com/git/git/blob/master/commit-slab.h\n> https://github.com/git/git/blob/master/commit-slab-impl.h\n> https://github.com/git/git/blob/master/commit-slab-decl.h\n> https://lore.kernel.org/git/CAOLTT8Q8BEKCVwPDypW1w66P9_xP7QC0T-CnLqamqAL4haGzwA@mail.gmail.com/\n\nAwesome!\n\nMaybe it's what I really need now, I will make a try.\n\nThanks(比心).\n"},{"id":"439019","messageId":"cover.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1629805395.git.dyroneteng@gmail.com","subject":"[PATCH v6 00/12] packfile-uri: support excluding multiple object types","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:24Z","receivedAt":"2021-10-19T11:39:03Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"\nAbout This Patch:\n-----------------\nThis patch has almost no reuse of previous patch's commits, because this\none redesigned the exclusion for various object types and reoganized the\ncommits. \n\nCommit (1): objects.c: introduce `exclude_level` enum\n\nThis commit introduce a new enum named `exclude_level` in object.c, it\ncontains three enumerated values: \"ET_SELF\", \"ET_INCLUDE\" and\n\"ET_REACHABLE\". The commit only makes the definitions, without any\nimplementations.\n\nCommit (2): Introduce function `match_packfile_uri_exclusions`\n\nThis one move the codes used for matching the URI protocols from\n`want_found_object` to new function `match_packfile_uri_exclusions`. The\npurpose is to improve code readability related to the feature and make\npreparation for further extension about the matching and exclusion for\nmultiple object types.\n\nCommit (3): Replace `show_data` with structure `show_info`\n\nThere is no feature related codes in this commit. This commit modified\nthe parameters in function `show_object` by replacing `show_data` with a\nnew structure `show_info`, this is another way of implementation about\nthe previous commit[1] in patch v5.\n\nCommit (4): Introduce `uploadpack.excludeobject` configuration\n\nThis commit introduce a new and backward-compatible configuration named\n`uploadpack.excludeobject`, the diff between old and new: \n\n\tuploadpack.blobPackfileUri= \n uploadpack.excludeobject= \n\nThe is correspond to the exclusion scope of the given object,\nand it's mentioned as `exclude_value` in object.c by \"Commit (1)\".\n\nCommit (6,8,10): Implementations for excluding commits, trees and tags\n\nCommit (7,9,11): Tests for excluding excluding commits, trees and tags\n\nCommit (12): Corresponding documentation modifications\n\nbundle-uri And packfile-uri \n----------------------------\n\nÆvar Arnfjörð Bjarmason post a new feature patchset[2] named \"bundle-uri\",\nit supports to let client download the full or incremental bundles\ndirectly without any negotiations with the server (full clone first and\nin a MVP progress now). I'm sorry if I misleading the meanings and\nplease point it out.\n\nIn \"packfile-uri\", different with \"bundle-uri\", happens in the span of\npacking objects during git-upload-pack on the server. This is an\nexperimental feature, originally designed as a CDN for large BLOB\nobjects, but it's not yet fully functional. I'm currently working on it\nfor making it support other types of objects.\n\nI also noticed that some opinions about the two features are mentioned\nin the patch[2], so I cc to Stolee in this patch, I hope I did not\ndisturb you.\n\n\n[1] https://public-inbox.org/git/xmqqlf4oc4u1.fsf@gitster.g/\n[2] https://lore.kernel.org/git/RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com/\n\nTeng Long (12):\n objects.c: introduce `exclude_level` enum\n Introduce function `match_packfile_uri_exclusions`\n Replace `show_data` with structure `show_info`\n Introduce `uploadpack.excludeobject` configuration\n t5702: test cases for `uploadpack.excludeobject`\n packfile-uri: support for excluding commits\n t5702: test cases for excluding commits\n packfile-uri: support for excluding trees\n t5702: test cases for excluding trees\n packfile-uri: support for excluding tags\n t5702: test cases for excluding tags\n packfile-uri.txt: support multiple object types\n\n Documentation/technical/packfile-uri.txt | 74 +-\n builtin/describe.c | 9 +-\n builtin/pack-objects.c | 236 +++++--\n builtin/rev-list.c | 11 +-\n bundle.c | 5 +-\n list-objects.c | 74 +-\n list-objects.h | 11 +-\n object.c | 21 +-\n object.h | 16 +-\n pack-bitmap.c | 16 +-\n reachable.c | 11 +-\n revision.c | 47 +-\n revision.h | 12 +\n shallow.c | 4 +-\n t/t5702-protocol-v2.sh | 856 +++++++++++++++++++++--\n upload-pack.c | 7 +\n 16 files changed, 1253 insertions(+), 157 deletions(-)\n\nRange-diff against v5:\n 1: 3a885678c9 = 1: 3a885678c9 objects.c: introduce `exclude_level` enum\n 2: 36426b4d9f = 2: 36426b4d9f Introduce function `match_packfile_uri_exclusions`\n 3: dced036f89 = 3: dced036f89 Replace `show_data` with structure `show_info`\n 4: b1d779b26a = 4: b1d779b26a Introduce `uploadpack.excludeobject` configuration\n 5: f643db3c71 = 5: f643db3c71 t5702: test cases for `uploadpack.excludeobject`\n 6: c29efeac21 = 6: c29efeac21 packfile-uri: support for excluding commits\n 7: ca72efd22e = 7: ca72efd22e t5702: test cases for excluding commits\n 8: c7a885ebec = 8: c7a885ebec packfile-uri: support for excluding trees\n 9: 5fc79a9a32 = 9: 5fc79a9a32 t5702: test cases for excluding trees\n10: 16c41c40a2 = 10: 16c41c40a2 packfile-uri: support for excluding tags\n11: 171ece533b = 11: 171ece533b t5702: test cases for excluding tags\n12: 79fc2c23cf = 12: 79fc2c23cf packfile-uri.txt: support multiple object types\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439020","messageId":"3a885678c9cee6ed4a70a7c88c15a7fa1b332150.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 01/12] objects.c: introduce `exclude_level` enum","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:25Z","receivedAt":"2021-10-19T11:39:12Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nCurrently packfile-uri supports the exclusion of blob objects, but\nin some scenarios, users may wish to exclude more types of objects,\nsuch as commits, trees and tags, the difference with blob is they are\nmore complicated. In addition to the meaning of a single object itself,\nit may also represent a collection of the objects that it includes\n(trees and blob in a root-tree) or reaches (ancestors of a commit).\nThe exclusion range is designed by an enum named `exclude_level` in\n\"pack-objects.c\" that enumerate three values, \"ET_SELF\", \"ET_INCLUDE\"\nand \"ET_REACHABLE\".\n\nHere are some explanations for their differences:\n\n- Scene 1: \"ET_SELF\" for excluding object itself.\n\nReason to support the \"ET_SELF\" is because the definition of\nexclusion-level should be better to be consistent, no matter what\nthe object type is. Excluding a single object itself has meaning for\nblobs, because blob is the smallest granularity among object types, and\nindeed, sometimes there are some frequently used big-size blobs in\nrepository.\n\nIf you want to exclude more ownership or reachable objects, it can be\nconsidered to use \"ET_INCLUDE\" or \"ET_REACHABLE\".\n\n- Scene 2: \"ET_INCLUDE\" for excluding object itself and objects it\ncontains.\n\nWhen a commit is specified to be excluded as packfile-uri,\nmore offen, it's hoped to exclude the trees and blobs contained in\nits top-level tree, as well as the commit itself, but not the\nancestors of the commit. This applies to scenarios where we want to\nexclude a specified non-blob object that includes some big-size\nobjects.\n\nCommit, tag and tree are suitable for this scenario. When a tag\nis specified, it will exclude the dereference commit, and all trees and\nblobs contained in its top-tree, as well as the tag itself if it's not a\nlightweight one.\n\n- Scene 3: \"ET_REACHABLE\" for excluding object itself, all the objects\nit contains, and its ancestors.\n\nFor further exclusion range, the ancestors are needed to excluded\ntogether, for example, the clone scenes.\n\nThis commit only defines the `exclude_level` enum type. The implementations\nof non-blob object types will be added in subsequent commits.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 44 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6d13cd3e1a..73b92a0c90 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -127,9 +127,52 @@ enum missing_action {\n };\n static enum missing_action arg_missing_action;\n static show_object_fn fn_show_object;\n+/*\n+ * exclude_level defines how to make the exclusion when the object matches\n+ * a packfile-uri (uploadpack.excludeobject or uploadpack.blobpackfileuri)\n+ * configuration.\n+ *\n+ * - ET_SELF;\n+ * This type means only the object itself will be excluded, and all other\n+ * objects it includes or reachable will not. For example, if object type is:\n+ * \t- BLOB: The blob object will be excluded\n+ * \t- TREE: The tree object will be excluded, the sub-trees and blobs it\n+ * \t includes will not be excluded.\n+ * \t- COMMIT: The commit object will be excluded, all the trees and blobs\n+ * \t that be included in its top-level tree will not be excluded.\n+ * \t- TAG: TAG object will be excluded, the referrenced commit object will\n+ * \t not be excluded.\n+ * - ET_INCLUDE;\n+ * This type means that not only the object itself will be excluded, but\n+ * also the objects it includes. For example, if object type is:\n+ * \t- BLOB: Same with 'ET_SELF'\n+ * \t- TREE: The tree object, and also the sub-trees and blobs that\n+ * \t the object includes will be excluded.\n+ * \t- COMMIT: The commit object, and also all the trees and blobs\n+ * \t contained in its top-level tree will be excluded.\n+ * \t- TAG: The TAG object will be excluded, and also the referrenced\n+ * \t commit will be excluded (the referrenced commit exclusion will\n+ * \t treat as a 'ET_INCLUDE' way).\n+ * - ET_REACHABLE;\n+ * This type means that not only the object and all the objects it includes\n+ * will be excluded, but also the reachable objects. For exmple, if object\n+ * type is:\n+ * \t- BLOB: Same with 'ET_INCLUDE'\n+ * \t- TREE: Same with 'ET_INCLUDE'\n+ * \t- COMMIT: The Objects in the case of 'ET_INCLUDE' will be excluded,\n+ * \t and also the ancestors of the commit will be excluded.\n+ * \t- TAG: The Objects in the case of 'ET_INCLUDE' will be excluded, and\n+ * \t also the ancestors of the referrenced commit will be excluded.\n+ */\n \n+enum exclude_level {\n+ ET_SELF,\n+ ET_INCLUDE,\n+ ET_REACHABLE,\n+};\n struct configured_exclusion {\n \tstruct oidmap_entry e;\n+\tint level;\n \tchar *pack_hash_hex;\n \tchar *uri;\n };\n@@ -3003,6 +3046,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n \t\t\tdie(_(\"object already configured in another \"\n \t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n+\t\tex->level = ET_SELF;\n \t\tex->pack_hash_hex = xcalloc(1, pack_end - oid_end);\n \t\tmemcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1);\n \t\tex->uri = xstrdup(pack_end + 1);\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439021","messageId":"36426b4d9fc868f144277472605a4384e39e159c.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 02/12] Introduce function `match_packfile_uri_exclusions`","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:26Z","receivedAt":"2021-10-19T11:39:15Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nThe matching codes now placed in function `want_object_in_pack`, move it\nto a new function `match_packfile_uri_exclusions` to prevent subsequent\nmodifications caused by its continuing expansion.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 33 +++++++++++++++++++++------------\n 1 file changed, 21 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 73b92a0c90..17053dc85a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1231,6 +1231,24 @@ static int have_duplicate_entry(const struct object_id *oid,\n \treturn 1;\n }\n \n+static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n+{\n+\tint i;\n+\tconst char *p;\n+\n+\tif (ex) {\n+\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n+\t\t\tif (skip_prefix(ex->uri,\n+\t\t\t\t\turi_protocols.items[i].string,\n+\t\t\t\t\t&p) &&\n+\t\t\t *p == ':')\n+\t\t\t\treturn 1;\n+\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n static int want_found_object(const struct object_id *oid, int exclude,\n \t\t\t struct packed_git *p)\n {\n@@ -1378,19 +1396,10 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (uri_protocols.nr) {\n \t\tstruct configured_exclusion *ex =\n \t\t\toidmap_get(&configured_exclusions, oid);\n-\t\tint i;\n-\t\tconst char *p;\n \n-\t\tif (ex) {\n-\t\t\tfor (i = 0; i < uri_protocols.nr; i++) {\n-\t\t\t\tif (skip_prefix(ex->uri,\n-\t\t\t\t\t\turi_protocols.items[i].string,\n-\t\t\t\t\t\t&p) &&\n-\t\t\t\t *p == ':') {\n-\t\t\t\t\toidset_insert(&excluded_by_config, oid);\n-\t\t\t\t\treturn 0;\n-\t\t\t\t}\n-\t\t\t}\n+\t\tif (ex && match_packfile_uri_exclusions(ex)) {\n+\t\t\toidset_insert(&excluded_by_config, oid);\n+\t\t\treturn 0;\n \t\t}\n \t}\n \n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439022","messageId":"dced036f89b44cf5b719287bbb82727ea4cc9543.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 03/12] Replace `show_data` with structure `show_info`","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:27Z","receivedAt":"2021-10-19T11:39:21Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nDuring the pack-objects process, \"show_object\" function will be called\nto find and show the object the function contains three parameters:\n\n\t1. struct object *obj\n\t2. const char *name\n\t3. void *show_data\n\nThis commit replace \"show_data\" with with a new structure `show_info`,\nit has two members:\n\n\t1. void *show_data\n 2. void *show_cache\n\nThe `show_data` in `show_info `is the same purpose with the original\none, is the callback when showing the objects.\n\nThe `show_cache` is the cache that ownership the relationship data.\nIt stores around the extra data for showing the objects, for example,\ncarry the ownership relationship between blob or tree objects and the\nreferred commit to avoid redundant and expensive calculations.\n\nSigned-off-by: Teng Long \n---\n builtin/describe.c | 9 +++++----\n builtin/pack-objects.c | 23 +++++++++++++----------\n builtin/rev-list.c | 11 ++++++-----\n bundle.c | 5 +++--\n list-objects.c | 29 +++++++++++++++++++++++------\n list-objects.h | 11 ++++++++---\n pack-bitmap.c | 16 +++++++++-------\n reachable.c | 11 +++++++----\n shallow.c | 4 ++--\n 9 files changed, 76 insertions(+), 43 deletions(-)\n\ndiff --git a/builtin/describe.c b/builtin/describe.c\nindex 40482d8e9f..a6a9e64fb0 100644\n--- a/builtin/describe.c\n+++ b/builtin/describe.c\n@@ -479,15 +479,16 @@ struct process_commit_data {\n \tstruct rev_info *revs;\n };\n \n-static void process_commit(struct commit *commit, void *data)\n+static void process_commit(struct commit *commit, struct show_info *info)\n {\n-\tstruct process_commit_data *pcd = data;\n+\tstruct process_commit_data *pcd = info->show_data;\n \tpcd->current_commit = commit->object.oid;\n }\n \n-static void process_object(struct object *obj, const char *path, void *data)\n+static void process_object(struct object *obj, const char *path,\n+\t\t\t struct show_info *info)\n {\n-\tstruct process_commit_data *pcd = data;\n+\tstruct process_commit_data *pcd = info->show_data;\n \n \tif (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) {\n \t\treset_revision_walk();\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 17053dc85a..a41a0a3ea7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3106,13 +3106,13 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit, void *_data)\n+static void show_commit_pack_hint(struct commit *commit, struct show_info *info)\n {\n \t/* nothing to do; commits don't have a namehash */\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *_data)\n+\t\t\t\t struct show_info *info)\n {\n \tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n \tif (!oe)\n@@ -3284,7 +3284,7 @@ static void read_object_list_from_stdin(void)\n /* Remember to update object flag allocation in object.h */\n #define OBJECT_ADDED (1u<<20)\n \n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, struct show_info *info)\n {\n \tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0);\n \tcommit->object.flags |= OBJECT_ADDED;\n@@ -3296,7 +3296,8 @@ static void show_commit(struct commit *commit, void *data)\n \t\tpropagate_island_marks(commit);\n }\n \n-static void show_object(struct object *obj, const char *name, void *data)\n+static void show_object(struct object *obj, const char *name,\n+\t\t\tstruct show_info *info)\n {\n \tadd_preferred_base_object(name);\n \tadd_object_entry(&obj->oid, obj->type, name, 0);\n@@ -3318,7 +3319,8 @@ static void show_object(struct object *obj, const char *name, void *data)\n \t}\n }\n \n-static void show_object__ma_allow_any(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_any(struct object *obj, const char *name,\n+\t\t\t\t struct show_info *info)\n {\n \tassert(arg_missing_action == MA_ALLOW_ANY);\n \n@@ -3329,10 +3331,11 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void\n \tif (!has_object(the_repository, &obj->oid, 0))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, info);\n }\n \n-static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data)\n+static void show_object__ma_allow_promisor(struct object *obj, const char *name,\n+\t\t\t\t\t struct show_info *info)\n {\n \tassert(arg_missing_action == MA_ALLOW_PROMISOR);\n \n@@ -3343,7 +3346,7 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,\n \tif (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid))\n \t\treturn;\n \n-\tshow_object(obj, name, data);\n+\tshow_object(obj, name, info);\n }\n \n static int option_parse_missing_action(const struct option *opt,\n@@ -3591,12 +3594,12 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \n static void record_recent_object(struct object *obj,\n \t\t\t\t const char *name,\n-\t\t\t\t void *data)\n+\t\t\t\t struct show_info *info)\n {\n \toid_array_append(&recent_objects, &obj->oid);\n }\n \n-static void record_recent_commit(struct commit *commit, void *data)\n+static void record_recent_commit(struct commit *commit, struct show_info *info)\n {\n \toid_array_append(&recent_objects, &commit->object.oid);\n }\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..116d8d39f1 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -94,9 +94,9 @@ static off_t get_object_disk_usage(struct object *obj)\n }\n \n static void finish_commit(struct commit *commit);\n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, struct show_info *s_info)\n {\n-\tstruct rev_list_info *info = data;\n+\tstruct rev_list_info *info = s_info->show_data;\n \tstruct rev_info *revs = info->revs;\n \n \tdisplay_progress(progress, ++progress_counter);\n@@ -266,12 +266,13 @@ static int finish_object(struct object *obj, const char *name, void *cb_data)\n \treturn 0;\n }\n \n-static void show_object(struct object *obj, const char *name, void *cb_data)\n+static void show_object(struct object *obj, const char *name,\n+\t\t\tstruct show_info *s_info)\n {\n-\tstruct rev_list_info *info = cb_data;\n+\tstruct rev_list_info *info = s_info->show_data;\n \tstruct rev_info *revs = info->revs;\n \n-\tif (finish_object(obj, name, cb_data))\n+\tif (finish_object(obj, name, info))\n \t\treturn;\n \tdisplay_progress(progress, ++progress_counter);\n \tif (show_disk_usage)\ndiff --git a/bundle.c b/bundle.c\nindex 693d619551..d4c71ee2b4 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -437,9 +437,9 @@ struct bundle_prerequisites_info {\n \tint fd;\n };\n \n-static void write_bundle_prerequisites(struct commit *commit, void *data)\n+static void write_bundle_prerequisites(struct commit *commit, struct show_info *info)\n {\n-\tstruct bundle_prerequisites_info *bpi = data;\n+\tstruct bundle_prerequisites_info *bpi = info->show_data;\n \tstruct object *object;\n \tstruct pretty_print_context ctx = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -530,6 +530,7 @@ int create_bundle(struct repository *r, const char *path,\n \t\tdie(\"revision walk setup failed\");\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n+\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n \ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..bffce67dd8 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -29,6 +29,10 @@ static void process_blob(struct traversal_context *ctx,\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n \tenum list_objects_filter_result r;\n+\tstruct show_info show_info;\n+\n+\tshow_info.show_data = ctx->show_data;\n+\tshow_info.show_cache = NULL;\n \n \tif (!ctx->revs->blob_objects)\n \t\treturn;\n@@ -60,7 +64,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n+\t\tctx->show_object(obj, path->buf, &show_info);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -154,10 +158,14 @@ static void process_tree(struct traversal_context *ctx,\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n+\tstruct show_info show_info;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r;\n \tint failed_parse;\n \n+\tshow_info.show_data = ctx->show_data;\n+\tshow_info.show_cache = NULL;\n+\n \tif (!revs->tree_objects)\n \t\treturn;\n \tif (!obj)\n@@ -191,7 +199,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tctx->show_object(obj, base->buf, &show_info);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -207,8 +215,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n-\n+\t\tctx->show_object(obj, base->buf, &show_info);\n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n }\n@@ -322,8 +329,12 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\t\t\t struct strbuf *base)\n {\n+\tstruct show_info show_info;\n \tint i;\n \n+\tshow_info.show_data = ctx->show_data;\n+\tshow_info.show_cache = NULL;\n+\n \tassert(base->len == 0);\n \n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n@@ -335,7 +346,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tctx->show_object(obj, name, &show_info);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n@@ -358,8 +369,13 @@ static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n+\tstruct show_info show_info;\n \tstrbuf_init(&csp, PATH_MAX);\n \n+\n+\tshow_info.show_data = ctx->show_data;\n+\tshow_info.show_cache = NULL;\n+\n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n@@ -375,7 +391,8 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data);\n+\n+\t\tctx->show_commit(commit, &show_info);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e46..c7b61e9e10 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -5,9 +5,14 @@ struct commit;\n struct object;\n struct rev_info;\n \n-typedef void (*show_commit_fn)(struct commit *, void *);\n-typedef void (*show_object_fn)(struct object *, const char *, void *);\n-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n+struct show_info {\n+ void *show_data; /* the data necessary for showing the object */\n+ void *show_cache; /* the cache ownership relationship data for showing the object */\n+};\n+\n+typedef void (*show_commit_fn)(struct commit *, struct show_info *);\n+typedef void (*show_object_fn)(struct object *, const char *, struct show_info *);\n+void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *show_data);\n \n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *revs,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3ed15431cd..0dcfa5b50e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -459,9 +459,10 @@ struct bitmap_show_data {\n \tstruct bitmap *base;\n };\n \n-static void show_object(struct object *object, const char *name, void *data_)\n+static void show_object(struct object *object, const char *name,\n+\t\t\tstruct show_info *info)\n {\n-\tstruct bitmap_show_data *data = data_;\n+\tstruct bitmap_show_data *data = info->show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(data->bitmap_git, &object->oid);\n@@ -473,7 +474,7 @@ static void show_object(struct object *object, const char *name, void *data_)\n \tbitmap_set(data->base, bitmap_pos);\n }\n \n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, struct show_info *info)\n {\n }\n \n@@ -628,6 +629,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n+\n \t\ttraverse_commit_list_filtered(filter, revs,\n \t\t\t\t\t show_commit, show_object,\n \t\t\t\t\t &show_data, NULL);\n@@ -1268,9 +1270,9 @@ struct bitmap_test_data {\n };\n \n static void test_show_object(struct object *object, const char *name,\n-\t\t\t void *data)\n+\t\t\t struct show_info *info)\n {\n-\tstruct bitmap_test_data *tdata = data;\n+\tstruct bitmap_test_data *tdata = info->show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid);\n@@ -1281,9 +1283,9 @@ static void test_show_object(struct object *object, const char *name,\n \tdisplay_progress(tdata->prg, ++tdata->seen);\n }\n \n-static void test_show_commit(struct commit *commit, void *data)\n+static void test_show_commit(struct commit *commit, struct show_info *info)\n {\n-\tstruct bitmap_test_data *tdata = data;\n+\tstruct bitmap_test_data *tdata = info->show_data;\n \tint bitmap_pos;\n \n \tbitmap_pos = bitmap_position(tdata->bitmap_git,\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..4f78954c62 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -47,14 +47,15 @@ static int add_one_ref(const char *path, const struct object_id *oid,\n * The traversal will have already marked us as SEEN, so we\n * only need to handle any progress reporting here.\n */\n-static void mark_object(struct object *obj, const char *name, void *data)\n+static void mark_object(struct object *obj, const char *name,\n+\t\t\tstruct show_info *info)\n {\n-\tupdate_progress(data);\n+\tupdate_progress(info->show_data);\n }\n \n-static void mark_commit(struct commit *c, void *data)\n+static void mark_commit(struct commit *c, struct show_info *info)\n {\n-\tmark_object(&c->object, NULL, data);\n+\tmark_object(&c->object, NULL, info);\n }\n \n struct recent_data {\n@@ -230,6 +231,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \t\treturn;\n \t}\n \n+\n \t/*\n \t * Set up the revision walk - this will move all commits\n \t * from the pending list to the commit walking list.\n@@ -244,6 +246,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \t\t\tdie(\"unable to mark recent objects\");\n \t\tif (prepare_revision_walk(revs))\n \t\t\tdie(\"revision walk setup failed\");\n+\n \t\ttraverse_commit_list(revs, mark_commit, mark_object, &cp);\n \t}\n \ndiff --git a/shallow.c b/shallow.c\nindex 9ed18eb884..ab1e49eba4 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -185,9 +185,9 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \treturn result;\n }\n \n-static void show_commit(struct commit *commit, void *data)\n+static void show_commit(struct commit *commit, struct show_info *info)\n {\n-\tcommit_list_insert(commit, data);\n+\tcommit_list_insert(commit, info->show_data);\n }\n \n /*\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439023","messageId":"b1d779b26a2f5d646b09131645b503d2e82a29ec.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 04/12] Introduce `uploadpack.excludeobject` configuration","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:28Z","receivedAt":"2021-10-19T11:39:24Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nBackground:\n\nThe `uploadpack.blobpackfileuri` is made for the \"packfile-uri\" feature,\nthe feature is similar to a CDN cache, it supports the client to download\n pack file by a URI directly, without or reducing the server load when make\npacking and transporting.\n\nReasons:\n\nFirst reason, `uploadpack.blobpackfileuri` supports to exclude a single\nobject, but actually the object type can be not only a blob, so the name\nis inaccurate currently.\n\nSecondly, the name of the old configuration is not abstract enough, this\nmake the furthur extension difficult. If do not change its name, to let\ndifferent object types use different configuration names, the\nconfiguration items will be bloated and difficult to maintain, so the\nnew configuration is more abstract in name.\n\nConfiguation format diff:\n\n\told: uploadpack.blobPackfileUri= \n\tnew: uploadpack.excludeobject= \n\nThe new configuration `uploadpack.excludeobject` not only supports to\nexclude a single object itself, but also to exclude related objects with\nit at once, the scope of exclusion is determined by the object type and\nthe specified `` value in the entry.\n\n-Compatibility:\n\nAlthough a new configuration has been introduced, the old one is\navailable to use and compatible with the new configuration.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 33 ++++++++++++++++++++++++++++++++-\n upload-pack.c | 7 +++++++\n 2 files changed, 39 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a41a0a3ea7..75461483c0 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3061,6 +3061,36 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tex->uri = xstrdup(pack_end + 1);\n \t\toidmap_put(&configured_exclusions, ex);\n \t}\n+\tif (!strcmp(k, \"uploadpack.excludeobject\")) {\n+\t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n+\t\tconst char *oid_end, *pack_end, *type_end;\n+\t\tstruct object_id pack_hash;\n+\t\tchar type[2];\n+\t\tint level;\n+\n+\t\tif (parse_oid_hex(v, &ex->e.oid, &oid_end) ||\n+\t\t *oid_end != ' ' ||\n+\t\t !strlcpy(type, oid_end + 1, sizeof(type)) ||\n+\t\t parse_oid_hex(oid_end + 3, &pack_hash, &pack_end) ||\n+\t\t *pack_end != ' ')\n+\t\t\tdie(_(\"value of uploadpack.excludeobject must be \"\n+\t\t\t \"of the form ' ' (got '%s')\"), v);\n+\t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n+\t\t\tdie(_(\"object already configured by an earlier \"\n+\t\t\t \"uploadpack.excludeobject (got '%s')\"), v);\n+\n+\t\tlevel = atoi(type);\n+\t\tif (level < ET_SELF || level > ET_REACHABLE) {\n+\t\t\tdie(_(\"value of must be 0 or 1 or 2 (got '%s')\"), v);\n+\t\t}\n+\t\tex->level = level;\n+\t\ttype_end = oid_end + 2;\n+\t\tex->pack_hash_hex = xcalloc(1, pack_end - type_end);\n+\t\tmemcpy(ex->pack_hash_hex, type_end + 1, pack_end - type_end - 1);\n+\t\tex->uri = xstrdup(pack_end + 1);\n+\t\toidmap_put(&configured_exclusions, ex);\n+\t}\n+\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -3887,7 +3917,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"respect islands during delta compression\")),\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n-\t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\t\t\tN_(\"exclude any configured uploadpack.excludeobject or \"\n+\t\t\t\t\t \"uploadpack.blobpackfileuri with this protocol\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5c1cd19612..d26fb351a3 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r,\n \t\t\tstrbuf_addstr(value, \" packfile-uris\");\n \t\t\tfree(str);\n \t\t}\n+\n+\t\tif (!repo_config_get_string(the_repository,\n+\t\t\t\t\t \"uploadpack.excludeobject\",\n+\t\t\t\t\t &str) && str) {\n+\t\t\tstrbuf_addstr(value, \" packfile-uris\");\n+\t\t\tfree(str);\n+\t\t}\n \t}\n \n \treturn 1;\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439024","messageId":"f643db3c71f3e6425326b8877586dd8cff544769.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 05/12] t5702: test cases for `uploadpack.excludeobject`","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:29Z","receivedAt":"2021-10-19T11:39:27Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"This commit expends the function `configure_exclusion` to support new\nexcluding type: commit, tree and tag.\n\nSigned-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 146 +++++++++++++++++++++++++++--------------\n 1 file changed, 98 insertions(+), 48 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 2e1243ca40..ccd3678311 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -824,17 +824,63 @@ test_expect_success 'when server does not send \"ready\", expect FLUSH' '\n '\n \n configure_exclusion () {\n-\tgit -C \"$1\" hash-object \"$2\" >objh &&\n-\tgit -C \"$1\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n-\tgit -C \"$1\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n-\tcat objh\n+\tobjt=\"$1\"\n+\tP=\"$2\"\n+\toid=\"$3\"\n+\tversion=\"$4\"\n+\texcluding_type=\"$5\"\n+\n+\toldc=\"uploadpack.blobpackfileuri\"\n+\tnewc=\"uploadpack.excludeobject\"\n+\tconfigkey=\"\"\n+\n+\tif test \"$version\" = \"old\"\n+\tthen\n+\t\tconfigkey=\"$oldc\"\n+\telse\n+\t\tconfigkey=\"$newc\"\n+\tfi\n+\n+\tif test \"$objt\" = \"blob\"\n+\tthen\n+\t\texcluding_type=\"0\"\n+\t\tgit -C \"$P\" hash-object \"$oid\" >objh &&\n+\t\tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n+\t\tif test \"$version\" = \"old\"\n+\t\tthen\n+\t\t\tgit -C \"$P\" config --add \\\n+ \t\t\t\"$configkey\" \\\n+ \t\t\t\"$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\"\n+ else\n+ \tgit -C \"$P\" config --add \\\n+\t\t\t\t\t\t\"$configkey\" \\\n+\t\t\t\t\t\t\"$(cat objh) $excluding_type $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\"\n+\t\tfi &&\n+\t\tcat objh\n+\telif test \"$objt\" = \"commit\" || test \"$objt\" = \"tree\" || test \"$objt\" = \"tag\"\n+\tthen\n+\t\techo \"$oid\" >objh &&\n+\t\tif test \"$excluding_type\" = \"0\"\n+\t\tthen\n+\t\t\tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n+\t\telse\n+\t\t\tgit -C \"$P\" pack-objects --revs \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh\n+\t\tfi &&\n+\n+\t\tgit -C \"$P\" config --add \\\n+ \t\t\t\"$configkey\" \\\n+ \t\t\t\"$(cat objh) $excluding_type $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n+\t\tcat objh\n+\telse\n+\t\techo \"unsupported object type in configure_exclusion (got $objt)\"\n+\tfi\n }\n \n-test_expect_success 'part of packfile response provided as URI' '\n+part_of_packfile_response_verify() {\n+\n+\tconfig=\"$1\" &&\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n@@ -843,10 +889,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\tconfigure_exclusion \"$P\" other-blob >h2 &&\n+\tconfigure_exclusion blob \"$P\" my-blob \"$config\" >h &&\n+\tconfigure_exclusion blob \"$P\" other-blob \"$config\" >h2 &&\n \n \tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n \tgit -c protocol.version=2 \\\n@@ -879,20 +925,22 @@ test_expect_success 'part of packfile response provided as URI' '\n \tls http_child/.git/objects/pack/*.pack \\\n \t http_child/.git/objects/pack/*.idx >filelist &&\n \ttest_line_count = 6 filelist\n-'\n+}\n+\n+blobpackfileuri_fetch () {\n+\tconfig=\"$1\"\n \n-test_expect_success 'packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n+\ttest_commit -C \"$P\" A &&\n \n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\tconfigure_exclusion blob \"$P\" my-blob $config >h &&\n \n \tgit init http_child &&\n \n@@ -900,12 +948,28 @@ test_expect_success 'packfile URIs with fetch instead of clone' '\n \tgit -C http_child -c protocol.version=2 \\\n \t\t-c fetch.uriprotocols=http,https \\\n \t\tfetch \"$HTTPD_URL/smart/http_parent\"\n+}\n+\n+test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): part of packfile response provided as URI' '\n+\trm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log &&\n+\tpart_of_packfile_response_verify old\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.excludeobject): part of packfile response provided as URI' '\n+\tpart_of_packfile_response_verify new\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): packfile URIs with fetch instead of clone' '\n+\tblobpackfileuri_fetch old\n+'\n+\n+test_expect_success 'blob-exclusion (using uploadpack.excludeobject): packfile URIs with fetch instead of clone' '\n+\tblobpackfileuri_fetch new\n '\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n-\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n@@ -913,9 +977,8 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" add my-blob &&\n \techo other-blob >\"$P/other-blob\" &&\n \tgit -C \"$P\" add other-blob &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" my-blob >h &&\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \t# Configure a URL for other-blob. Just reuse the hash of the object as\n \t# the hash of the packfile, since the hash does not matter for this\n \t# test as long as it is not the hash of the pack, and it is of the\n@@ -923,9 +986,8 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tgit -C \"$P\" hash-object other-blob >objh &&\n \tgit -C \"$P\" pack-objects \"$HTTPD_DOCUMENT_ROOT_PATH/mypack\" packh &&\n \tgit -C \"$P\" config --add \\\n-\t\t\"uploadpack.blobpackfileuri\" \\\n-\t\t\"$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n-\n+\t\t\"uploadpack.excludeobject\" \\\n+\t\t\"$(cat objh) 0 $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack\" &&\n \ttest_must_fail env GIT_TEST_SIDEBAND_ALL=1 \\\n \t\tgit -c protocol.version=2 \\\n \t\t-c fetch.uriprotocols=http,https \\\n@@ -935,17 +997,14 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n-\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n \t\t-c fetch.uriprotocols=http,https \\\n@@ -959,8 +1018,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child log &&\n-\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n@@ -976,10 +1034,8 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n \techo my-blob >\"$P/my-blob\" &&\n \tgit -C \"$P\" add my-blob &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" my-blob >h &&\n-\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" my-blob >h &&\n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n \t\t-c fetch.uriprotocols=http,https \\\n@@ -989,8 +1045,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object'\n \n test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child &&\n-\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child\" &&\n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n@@ -999,9 +1054,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \techo \"url = git://example.com/git/lib.git\" >>\"$P/.gitmodules\" &&\n \tgit -C \"$P\" add .gitmodules &&\n \tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n-\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \tgit -c protocol.version=2 -c transfer.fsckobjects=1 \\\n \t\t-c fetch.uriprotocols=http,https \\\n@@ -1015,8 +1068,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n-\trm -rf \"$P\" http_child err &&\n-\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child err\" &&\n \tgit init \"$P\" &&\n \tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n \n@@ -1024,10 +1076,8 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n \techo \"path = include/foo\" >>\"$P/.gitmodules\" &&\n \techo \"url = git://example.com/git/lib.git\" >>\"$P/.gitmodules\" &&\n \tgit -C \"$P\" add .gitmodules &&\n-\tgit -C \"$P\" commit -m x &&\n-\n-\tconfigure_exclusion \"$P\" .gitmodules >h &&\n-\n+\ttest_commit -C \"$P\" A &&\n+\tconfigure_exclusion blob \"$P\" .gitmodules >h &&\n \tsane_unset GIT_TEST_SIDEBAND_ALL &&\n \ttest_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \\\n \t\t-c fetch.uriprotocols=http,https \\\n@@ -1038,4 +1088,4 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul\n # DO NOT add non-httpd-specific tests here, because the last part of this\n # test script is only executed when httpd is available and enabled.\n \n-test_done\n+test_done\n\\ No newline at end of file\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439025","messageId":"c29efeac21a6770324a58488b0d53709778c83de.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 06/12] packfile-uri: support for excluding commits","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:30Z","receivedAt":"2021-10-19T11:40:00Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nThis commit work on the exlusion of commit objects, and the\nIts excluding `level` can be configured as \"ET_SELF\", \"ET_INCLUDE\"\nor \"ET_REACHABLE\".\n\nExclusion scope on different level:\n\n1. When a commit is specified to be excluded with level \"ET_SELF\", only the\ncommit object itself will be excluded.\n\n2. When it's specified to be excluded with level \"ET_INCLUDE\", exclude\nall trees and blobs contained in its top-level tree, as well as the\ncommit itself.\n\n3. When it is specified with level \"ET_REACHABLE\", exclude its\nancestors, as well as the objects need to be excluded under the level\n\"ET_INCLUDED\".\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 83 +++++++++++++++++++++++++++++++++---------\n list-objects.c | 37 +++++++++++--------\n object.c | 19 ++++++++--\n object.h | 14 ++++++-\n revision.c | 34 ++++++++++++-----\n revision.h | 3 ++\n 6 files changed, 144 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 75461483c0..e7b27ef443 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -36,6 +36,7 @@\n #include \"trace2.h\"\n #include \"shallow.h\"\n #include \"promisor-remote.h\"\n+#include \"commit-reach.h\"\n \n #define IN_PACK(obj) oe_in_pack(&to_pack, obj)\n #define SIZE(obj) oe_size(&to_pack, obj)\n@@ -1354,11 +1355,14 @@ static int want_object_in_pack_one(struct packed_git *p,\n static int want_object_in_pack(const struct object_id *oid,\n \t\t\t int exclude,\n \t\t\t struct packed_git **found_pack,\n-\t\t\t off_t *found_offset)\n+\t\t\t off_t *found_offset,\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tint want;\n \tstruct list_head *pos;\n \tstruct multi_pack_index *m;\n+\tstruct configured_exclusion *ex;\n+\tstruct configured_exclusion *commit_ex;\n \n \tif (!exclude && local && has_loose_object_nonlocal(oid))\n \t\treturn 0;\n@@ -1394,9 +1398,16 @@ static int want_object_in_pack(const struct object_id *oid,\n \t}\n \n \tif (uri_protocols.nr) {\n-\t\tstruct configured_exclusion *ex =\n-\t\t\toidmap_get(&configured_exclusions, oid);\n+\t\tif (referred_objs) {\n+\t\t\tstruct commit *commit = referred_objs->commit;\n+\t\t\tif (commit) {\n+\t\t\t\tcommit_ex = oidmap_get(&configured_exclusions, &commit->object.oid);\n+\t\t\t\tif (match_packfile_uri_exclusions(commit_ex) && commit_ex->level > ET_SELF)\n+\t\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t}\n \n+\t\tex = oidmap_get(&configured_exclusions, oid);\n \t\tif (ex && match_packfile_uri_exclusions(ex)) {\n \t\t\toidset_insert(&excluded_by_config, oid);\n \t\t\treturn 0;\n@@ -1436,7 +1447,8 @@ static const char no_closure_warning[] = N_(\n );\n \n static int add_object_entry(const struct object_id *oid, enum object_type type,\n-\t\t\t const char *name, int exclude)\n+\t\t\t const char *name, int exclude,\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct packed_git *found_pack = NULL;\n \toff_t found_offset = 0;\n@@ -1446,7 +1458,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \tif (have_duplicate_entry(oid, exclude))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) {\n+\tif (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_objs)) {\n \t\t/* The pack is missing an object, so it will not have closure */\n \t\tif (write_bitmap_index) {\n \t\t\tif (write_bitmap_index != WRITE_BITMAP_QUIET)\n@@ -1472,7 +1484,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid,\n \tif (have_duplicate_entry(oid, 0))\n \t\treturn 0;\n \n-\tif (!want_object_in_pack(oid, 0, &pack, &offset))\n+\tif (!want_object_in_pack(oid, 0, &pack, &offset, NULL))\n \t\treturn 0;\n \n \tcreate_object_entry(oid, type, name_hash, 0, 0, pack, offset);\n@@ -1612,7 +1624,7 @@ static void add_pbase_object(struct tree_desc *tree,\n \t\tif (name[cmplen] != '/') {\n \t\t\tadd_object_entry(&entry.oid,\n \t\t\t\t\t object_type(entry.mode),\n-\t\t\t\t\t fullname, 1);\n+\t\t\t\t\t fullname, 1, NULL);\n \t\t\treturn;\n \t\t}\n \t\tif (S_ISDIR(entry.mode)) {\n@@ -1680,7 +1692,7 @@ static void add_preferred_base_object(const char *name)\n \tcmplen = name_cmp_len(name);\n \tfor (it = pbase_tree; it; it = it->next) {\n \t\tif (cmplen == 0) {\n-\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1);\n+\t\t\tadd_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL);\n \t\t}\n \t\telse {\n \t\t\tstruct tree_desc tree;\n@@ -2882,7 +2894,7 @@ static void add_tag_chain(const struct object_id *oid)\n \t\t\tdie(_(\"unable to pack objects reachable from tag %s\"),\n \t\t\t oid_to_hex(oid));\n \n-\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0);\n+\t\tadd_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL);\n \n \t\tif (tag->tagged->type != OBJ_TAG)\n \t\t\treturn;\n@@ -3050,8 +3062,6 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t *oid_end != ' ' ||\n \t\t parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) ||\n \t\t *pack_end != ' ')\n-\t\t\tdie(_(\"value of uploadpack.blobpackfileuri must be \"\n-\t\t\t \"of the form ' ' (got '%s')\"), v);\n \t\tif (oidmap_get(&configured_exclusions, &ex->e.oid))\n \t\t\tdie(_(\"object already configured in another \"\n \t\t\t \"uploadpack.blobpackfileuri (got '%s')\"), v);\n@@ -3114,7 +3124,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tofs = nth_packed_object_offset(p, pos);\n-\tif (!want_object_in_pack(oid, 0, &p, &ofs))\n+\tif (!want_object_in_pack(oid, 0, &p, &ofs, NULL))\n \t\treturn 0;\n \n \toi.typep = &type;\n@@ -3307,7 +3317,7 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(_(\"expected object ID, got garbage:\\n %s\"), line);\n \n \t\tadd_preferred_base_object(p + 1);\n-\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0);\n+\t\tadd_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL);\n \t}\n }\n \n@@ -3316,7 +3326,7 @@ static void read_object_list_from_stdin(void)\n \n static void show_commit(struct commit *commit, struct show_info *info)\n {\n-\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0);\n+\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n \tcommit->object.flags |= OBJECT_ADDED;\n \n \tif (write_bitmap_index)\n@@ -3329,8 +3339,9 @@ static void show_commit(struct commit *commit, struct show_info *info)\n static void show_object(struct object *obj, const char *name,\n \t\t\tstruct show_info *info)\n {\n+\tstruct referred_objects *referred_objs = info->show_cache;\n \tadd_preferred_base_object(name);\n-\tadd_object_entry(&obj->oid, obj->type, name, 0);\n+\tadd_object_entry(&obj->oid, obj->type, name, 0, referred_objs);\n \tobj->flags |= OBJECT_ADDED;\n \n \tif (use_delta_islands) {\n@@ -3483,7 +3494,7 @@ static void add_objects_in_unpacked_packs(void)\n \t\tQSORT(in_pack.array, in_pack.nr, ofscmp);\n \t\tfor (i = 0; i < in_pack.nr; i++) {\n \t\t\tstruct object *o = in_pack.array[i].object;\n-\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0);\n+\t\t\tadd_object_entry(&o->oid, o->type, \"\", 0, NULL);\n \t\t}\n \t}\n \tfree(in_pack.array);\n@@ -3499,7 +3510,7 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t\treturn 0;\n \t}\n \n-\tadd_object_entry(oid, type, \"\", 0);\n+\tadd_object_entry(oid, type, \"\", 0, NULL);\n \treturn 0;\n }\n \n@@ -3665,6 +3676,42 @@ static void mark_bitmap_preferred_tips(void)\n \t}\n }\n \n+static void reuse_exclusion_packfile(struct rev_info *revs)\n+{\n+\tstruct commit *commit;\n+\tstruct oidmap_iter iter;\n+\tstruct configured_exclusion *ex;\n+\tstruct object_id ex_oid;\n+\tstruct commit *ex_commit;\n+\tstruct commit_list *list = revs->commits;\n+\tstruct commit_list *newlist = NULL;\n+\tstruct commit_list **p = &newlist;\n+\n+\tif (revs->limited)\n+\t\treturn;\n+\twhile (list) {\n+\t\tcommit = pop_commit(&list);\n+\t\tif (commit_list_contains(commit, newlist)) {\n+\t\t\tcontinue;\n+\t\t}\n+\t\tp = commit_list_append(commit, p);\n+\t\toidmap_iter_init(&configured_exclusions, &iter);\n+\t\twhile ((ex = oidmap_iter_next(&iter)) && ex->level == ET_REACHABLE) {\n+\t\t\tex_oid = ex->e.oid;\n+\t\t\tex_commit = lookup_commit_reference(the_repository, &ex_oid);\n+\t\t\tif (!ex_commit)\n+\t\t\t\tdie(\"Not a valid commit name %s\", oid_to_hex(&ex_oid));\n+\t\t\tif (!in_merge_bases(ex_commit, commit))\n+\t\t\t\tcontinue;\n+\t\t\toidset_insert(&excluded_by_config, &ex_oid);\n+\t\t\tex_commit->object.flags |= UNINTERESTING;\n+\t\t\tp = commit_list_append(ex_commit, p);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\trevs->commits = newlist;\n+}\n+\n static void get_object_list(int ac, const char **av)\n {\n \tstruct rev_info revs;\n@@ -3726,6 +3773,8 @@ static void get_object_list(int ac, const char **av)\n \t\tdie(_(\"revision walk setup failed\"));\n \tmark_edges_uninteresting(&revs, show_edge, sparse);\n \n+\treuse_exclusion_packfile(&revs);\n+\n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n \ttraverse_commit_list_filtered(&filter_options, &revs,\ndiff --git a/list-objects.c b/list-objects.c\nindex bffce67dd8..b32213ecf1 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -24,7 +24,8 @@ struct traversal_context {\n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct referred_objects *referred_obj)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n@@ -32,7 +33,7 @@ static void process_blob(struct traversal_context *ctx,\n \tstruct show_info show_info;\n \n \tshow_info.show_data = ctx->show_data;\n-\tshow_info.show_cache = NULL;\n+\tshow_info.show_cache = referred_obj;\n \n \tif (!ctx->revs->blob_objects)\n \t\treturn;\n@@ -101,11 +102,13 @@ static void process_gitlink(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name);\n+\t\t\t const char *name,\n+\t\t\t struct referred_objects *referred_objs);\n \n static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t struct tree *tree,\n-\t\t\t\t struct strbuf *base)\n+\t\t\t\t struct strbuf *base,\n+\t\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n@@ -133,7 +136,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path);\n+\t\t\tprocess_tree(ctx, t, base, entry.path, referred_objs);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n@@ -146,7 +149,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path);\n+\t\t\tprocess_blob(ctx, b, base, entry.path, referred_objs);\n \t\t}\n \t}\n }\n@@ -154,7 +157,8 @@ static void process_tree_contents(struct traversal_context *ctx,\n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n-\t\t\t const char *name)\n+\t\t\t const char *name,\n+\t\t\t struct referred_objects *referred_objs)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n@@ -163,9 +167,6 @@ static void process_tree(struct traversal_context *ctx,\n \tenum list_objects_filter_result r;\n \tint failed_parse;\n \n-\tshow_info.show_data = ctx->show_data;\n-\tshow_info.show_cache = NULL;\n-\n \tif (!revs->tree_objects)\n \t\treturn;\n \tif (!obj)\n@@ -196,6 +197,9 @@ static void process_tree(struct traversal_context *ctx,\n \t\t\t\t\t LOFS_BEGIN_TREE, obj,\n \t\t\t\t\t base->buf, &base->buf[baselen],\n \t\t\t\t\t ctx->filter);\n+\tshow_info.show_cache = referred_objs;\n+\tshow_info.show_data = ctx->show_data;\n+\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n@@ -206,7 +210,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n-\t\tprocess_tree_contents(ctx, tree, base);\n+\t\tprocess_tree_contents(ctx, tree, base, referred_objs);\n \n \tr = list_objects_filter__filter_object(ctx->revs->repo,\n \t\t\t\t\t LOFS_END_TREE, obj,\n@@ -321,9 +325,9 @@ void mark_edges_uninteresting(struct rev_info *revs,\n \t}\n }\n \n-static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n+static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct commit *referred_commit)\n {\n-\tadd_pending_object(revs, &tree->object, \"\");\n+\tadd_pending_object_with_referred_commit(revs, &tree->object, \"\", referred_commit);\n }\n \n static void traverse_trees_and_blobs(struct traversal_context *ctx,\n@@ -340,6 +344,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n+\t\tstruct referred_objects *referred_objs = pending->referred_objects;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n@@ -352,11 +357,11 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path);\n+\t\t\tprocess_tree(ctx, (struct tree *)obj, base, path, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n-\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path);\n+\t\t\tprocess_blob(ctx, (struct blob *)obj, base, path, referred_objs);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n@@ -386,7 +391,7 @@ static void do_traverse(struct traversal_context *ctx)\n \t\telse if (get_commit_tree(commit)) {\n \t\t\tstruct tree *tree = get_commit_tree(commit);\n \t\t\ttree->object.flags |= NOT_USER_GIVEN;\n-\t\t\tadd_pending_tree(ctx->revs, tree);\n+\t\t\tadd_pending_tree(ctx->revs, tree, commit);\n \t\t} else if (commit->object.parsed) {\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\ndiff --git a/object.c b/object.c\nindex 14188453c5..f86b52c4d6 100644\n--- a/object.c\n+++ b/object.c\n@@ -322,14 +322,18 @@ void object_list_free(struct object_list **list)\n */\n static char object_array_slopbuf[1];\n \n-void add_object_array_with_path(struct object *obj, const char *name,\n-\t\t\t\tstruct object_array *array,\n-\t\t\t\tunsigned mode, const char *path)\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name,\n+\t\t\t\t\t\t struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path,\n+\t\t\t\t\t\t struct commit *referred_commit)\n {\n \tunsigned nr = array->nr;\n \tunsigned alloc = array->alloc;\n \tstruct object_array_entry *objects = array->objects;\n \tstruct object_array_entry *entry;\n+\tstruct referred_objects *referred_objs;\n+\treferred_objs = xmalloc(sizeof(struct referred_objects));\n+\treferred_objs->commit = referred_commit;\n \n \tif (nr >= alloc) {\n \t\talloc = (alloc + 32) * 2;\n@@ -339,6 +343,7 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \t}\n \tentry = &objects[nr];\n \tentry->item = obj;\n+\tentry->referred_objects = referred_objs;\n \tif (!name)\n \t\tentry->name = NULL;\n \telse if (!*name)\n@@ -354,6 +359,13 @@ void add_object_array_with_path(struct object *obj, const char *name,\n \tarray->nr = ++nr;\n }\n \n+void add_object_array_with_path(struct object *obj, const char *name,\n+\t\t\t\tstruct object_array *array,\n+\t\t\t\tunsigned mode, const char *path)\n+{\n+\tadd_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL);\n+}\n+\n void add_object_array(struct object *obj, const char *name, struct object_array *array)\n {\n \tadd_object_array_with_path(obj, name, array, S_IFINVALID, NULL);\n@@ -368,6 +380,7 @@ static void object_array_release_entry(struct object_array_entry *ent)\n \tif (ent->name != object_array_slopbuf)\n \t\tfree(ent->name);\n \tfree(ent->path);\n+\tfree(ent->referred_objects);\n }\n \n struct object *object_array_pop(struct object_array *array)\ndiff --git a/object.h b/object.h\nindex 87a6da47c8..4db0ecc3f2 100644\n--- a/object.h\n+++ b/object.h\n@@ -52,9 +52,20 @@ struct object_array {\n \t\tchar *name;\n \t\tchar *path;\n \t\tunsigned mode;\n+\t\t/*\n+\t\t* referred_objects or NULL. If non-NULL, it will\n+\t\t* temporary storage the referred objects when\n+\t\t* traversing the specified object. Space for time,\n+\t\t* reduce related computing costs (such as packfile-uri\n+\t\t* exclusion), clean up when the traversal is over.\n+\t\t*/\n+\t\tstruct referred_objects *referred_objects;\n \t} *objects;\n };\n \n+struct referred_objects{\n+ struct commit *commit;\n+};\n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n /*\n@@ -157,7 +168,8 @@ void object_list_free(struct object_list **list);\n /* Object array handling .. */\n void add_object_array(struct object *obj, const char *name, struct object_array *array);\n void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path);\n-\n+void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array,\n+\t\t\t\t\t\t unsigned mode, const char *path, struct commit *referred_commit);\n /*\n * Returns NULL if the array is empty. Otherwise, returns the last object\n * after removing its entry from the array. Other resources associated\ndiff --git a/revision.c b/revision.c\nindex 4853c85d0b..89a8b311ea 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit)\n \tcommit_stack_clear(&pending);\n }\n \n-static void add_pending_object_with_path(struct rev_info *revs,\n-\t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode,\n-\t\t\t\t\t const char *path)\n+static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t\t\t const char *path,\n+\t\t\t\t\t\t\t struct commit *referred_commit)\n {\n \tstruct interpret_branch_name_options options = { 0 };\n \tif (!obj)\n@@ -326,20 +327,35 @@ static void add_pending_object_with_path(struct rev_info *revs,\n \t\tstrbuf_release(&buf);\n \t\treturn; /* do not add the commit itself */\n \t}\n-\tadd_object_array_with_path(obj, name, &revs->pending, mode, path);\n+\tadd_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit);\n }\n \n+static void add_pending_object_with_path(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj,\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t const char *path)\n+{\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL);\n+}\n static void add_pending_object_with_mode(struct rev_info *revs,\n \t\t\t\t\t struct object *obj,\n-\t\t\t\t\t const char *name, unsigned mode)\n+\t\t\t\t\t const char *name, unsigned mode,\n+\t\t\t\t\t struct commit *referred_commit)\n+{\n+\tadd_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit);\n+}\n+\n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct commit *referred_commit)\n {\n-\tadd_pending_object_with_path(revs, obj, name, mode, NULL);\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit);\n }\n \n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name)\n {\n-\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID);\n+\tadd_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL);\n }\n \n void add_head_to_pending(struct rev_info *revs)\n@@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tif (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc))\n \t\t\tdiagnose_missing_default(revs->def);\n \t\tobject = get_reference(revs, revs->def, &oid, 0);\n-\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode);\n+\t\tadd_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL);\n \t}\n \n \t/* Did the user ask for any diff output? Run the diff! */\ndiff --git a/revision.h b/revision.h\nindex a24f72dcd1..b2e0c0b9b7 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -423,6 +423,9 @@ void show_object_with_name(FILE *, struct object *, const char *);\n */\n void add_pending_object(struct rev_info *revs,\n \t\t\tstruct object *obj, const char *name);\n+void add_pending_object_with_referred_commit(struct rev_info *revs,\n+\t\t\t\t\t struct object *obj, const char *name,\n+\t\t\t\t\t struct commit *referred_commit);\n \n void add_pending_oid(struct rev_info *revs,\n \t\t const char *name, const struct object_id *oid,\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439026","messageId":"ca72efd22ef6a11d8cdbf48912818ea7edbcf73f.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 07/12] t5702: test cases for excluding commits","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:31Z","receivedAt":"2021-10-19T11:40:01Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 300 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 300 insertions(+)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex ccd3678311..6e323253f7 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -967,6 +967,306 @@ test_expect_success 'blob-exclusion (using uploadpack.excludeobject): packfile U\n \tblobpackfileuri_fetch new\n '\n \n+test_expect_success 'commit-exclusion(excluding_type=ET_SELF): part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\texcluding_type=\"0\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/my-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/my-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\ttest_commit -C \"$P\" A &&\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD my-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object my-tree/sub-tree/sub-blob) &&\n+\tconfigure_exclusion commit \"$P\" \"$commith\" new \"$excluding_type\" >h &&\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child\n+'\n+\n+test_expect_success 'commit-exclusion(excluding_type=ET_INCLUDE): part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\texcluding_type=\"1\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/my-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/my-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\ttest_commit -C \"$P\" A &&\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD my-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object my-tree/sub-tree/sub-blob) &&\n+\tconfigure_exclusion commit \"$P\" \"$commith\" new \"$excluding_type\" >h &&\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 7 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $subtreeh out\n+\t\t\tthen\n+\t\t\t\t>subtreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $subblobh out\n+\t\t\tthen\n+\t\t\t\t>subblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f subtreehfound &&\n+\ttest -f subblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\t# Ensure that there are exactly 2 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 4 filelist\n+'\n+\n+test_expect_success 'commit-exclusion(excluding_type=ET_REACHABLE): hitten a full packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+ \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\texcluding_type=\"2\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\ttest_commit -C \"$P\" A &&\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+ \tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+ \tah=$(git -C \"$P\" hash-object A.t) &&\n+ \tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\tconfigure_exclusion commit \"$P\" \"$commith\" new \"$excluding_type\" >h &&\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 5 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi\n+\t\telif test_line_count = 0 out.objectlist\n+\t\tthen\n+\t\t\t\t>emptypackfound\n+\t\tfi\n+\tdone &&\n+\ttest -f emptypackfound &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\t# Ensure that there are exactly 2 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 4 filelist\n+'\n+\n+test_expect_success 'commit-exclusion(excluding_type=ET_REACHABLE): part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+ \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\texcluding_type=\"2\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\ttest_commit -C \"$P\" A &&\n+\tmkdir \"$P\"/other-tree &&\n+\techo other-blob >\"$P\"/other-tree/other-blob &&\n+\tgit -C \"$P\" add other-tree &&\n+\ttest_commit -C \"$P\" B &&\n+ ah=$(git -C \"$P\" hash-object A.t) &&\n+ bh=$(git -C \"$P\" hash-object B.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\totherblobh=$(git -C \"$P\" hash-object other-tree/other-blob) &&\n+\trm -rf \"$P\"/my-tree \"$P\"/other-tree \"$P\"/A.t \"$P\"/B.t &&\n+\tmkdir \"$P\"/another-tree &&\n+\techo another-blob >\"$P\"/another-tree/another-blob &&\n+\tgit -C \"$P\" add . &&\n+\ttest_commit -C \"$P\" C &&\n+ \tcommitAh=$(git -C \"$P\" rev-parse A) &&\n+ \tcommitBh=$(git -C \"$P\" rev-parse B) &&\n+ \tcommitCh=$(git -C \"$P\" rev-parse C) &&\n+ \troottreeAh=$(git -C \"$P\" rev-parse A:) &&\n+ \troottreeBh=$(git -C \"$P\" rev-parse B:) &&\n+ \troottreeCh=$(git -C \"$P\" rev-parse C:) &&\n+ mytreeh=$(git -C \"$P\" ls-tree A my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+ othertreeh=$(git -C \"$P\" ls-tree B other-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+ anothertreeh=$(git -C \"$P\" ls-tree C another-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+ ch=$(git -C \"$P\" hash-object C.t) &&\n+\tanotherblobh=$(git -C \"$P\" hash-object another-tree/another-blob) &&\n+\tconfigure_exclusion commit \"$P\" \"$commitBh\" new \"$excluding_type\" >h &&\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 5 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commitCh out\n+\t\t\tthen\n+\t\t\t\t>commitChfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeCh out\n+\t\t\tthen\n+\t\t\t\t>roottreeChfound\n+\t\t\tfi &&\n+\t\t\tif grep $anothertreeh out\n+\t\t\tthen\n+\t\t\t\t>anothertreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $anotherblobh out\n+\t\t\tthen\n+\t\t\t\t>anotherblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $ch out\n+\t\t\tthen\n+\t\t\t\t>chfound\n+\t\t\tfi\n+\t\telif test_line_count = 10 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commitAh out\n+\t\t\tthen\n+\t\t\t\t>commitAhfound\n+\t\t\tfi &&\n+\t\t\tif grep $commitBh out\n+\t\t\tthen\n+\t\t\t\t>commitBhfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeAh out\n+\t\t\tthen\n+\t\t\t\t>roottreeAhfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeBh out\n+\t\t\tthen\n+\t\t\t\t>roottreeBhfound\n+\t\t\tfi &&\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $othertreeh out\n+\t\t\tthen\n+\t\t\t\t>othertreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $otherblobh out\n+\t\t\tthen\n+\t\t\t\t>otherblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $bh out\n+\t\t\tthen\n+\t\t\t\t>bhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f commitChfound &&\n+\ttest -f roottreeChfound &&\n+\ttest -f anothertreehfound &&\n+\ttest -f anotherblobhfound &&\n+\ttest -f chfound &&\n+\ttest -f commitAhfound &&\n+\ttest -f commitBhfound &&\n+\ttest -f roottreeAhfound &&\n+\ttest -f roottreeBhfound &&\n+\ttest -f mytreehfound &&\n+\ttest -f othertreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f otherblobhfound &&\n+\ttest -f ahfound &&\n+\ttest -f bhfound &&\n+\t# Ensure that there are exactly 2 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 4 filelist\n+'\n+\n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439027","messageId":"c7a885ebec703f383f5935517170b4ab755b5411.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 08/12] packfile-uri: support for excluding trees","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:32Z","receivedAt":"2021-10-19T11:40:06Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nThis commit introduce the exclusion of tree objects. The exclusion range\nis designed by an enum named `exclude_level` in \"pack-objects.c\" that\nenumerate three values, \"ET_SELF\", \"ET_INCLUDE\" and \"ET_REACHABLE\".\n\nExclusion scope on different level:\n\n1. When a tree is specified to be excluded with level \"ET_SELF\",\nonly the tree object itself will be excluded.\n\n2. When a tree specified to be excluded with level \"ET_INCLUDE\",\nexclude the tree itself, as well as all the trees and blobs it contains.\n\n3. When it is specified with level \"ET_REACHABLE\", the excluding scope\nis as same as level \"ET_INCLUDE\", because tree do not have any\nancestors.\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 18 ++++++++++++++++++\n list-objects.c | 8 ++++++--\n object.c | 1 +\n object.h | 1 +\n 4 files changed, 26 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e7b27ef443..6713e734fb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1250,6 +1250,21 @@ static int match_packfile_uri_exclusions(struct configured_exclusion *ex)\n \treturn 0;\n }\n \n+static int want_exclude_object(struct object_list *objects)\n+{\n+\tstruct object_list *p;\n+\tstruct configured_exclusion *ex;\n+\n+\tif (!objects)\n+\t\treturn 0;\n+\tfor (p = objects; p; p = p->next) {\n+\t\tex = oidmap_get(&configured_exclusions, &p->item->oid);\n+\t\tif (match_packfile_uri_exclusions(ex) && ex->level > ET_SELF)\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static int want_found_object(const struct object_id *oid, int exclude,\n \t\t\t struct packed_git *p)\n {\n@@ -1400,11 +1415,14 @@ static int want_object_in_pack(const struct object_id *oid,\n \tif (uri_protocols.nr) {\n \t\tif (referred_objs) {\n \t\t\tstruct commit *commit = referred_objs->commit;\n+\t\t\tstruct object_list *trees = referred_objs->trees;\n \t\t\tif (commit) {\n \t\t\t\tcommit_ex = oidmap_get(&configured_exclusions, &commit->object.oid);\n \t\t\t\tif (match_packfile_uri_exclusions(commit_ex) && commit_ex->level > ET_SELF)\n \t\t\t\t\treturn 0;\n \t\t\t}\n+\t\t\tif (want_exclude_object(trees))\n+\t\t\t\treturn 0;\n \t\t}\n \n \t\tex = oidmap_get(&configured_exclusions, oid);\ndiff --git a/list-objects.c b/list-objects.c\nindex b32213ecf1..40292e2cc8 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -114,8 +114,11 @@ static void process_tree_contents(struct traversal_context *ctx,\n \tstruct name_entry entry;\n \tenum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ?\n \t\tall_entries_interesting : entry_not_interesting;\n+\tstruct referred_objects *referred_buf;\n \n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\treferred_buf = xmemdupz(referred_objs, sizeof(struct referred_objects));\n+\tobject_list_insert(&tree->object, &referred_buf->trees);\n \n \twhile (tree_entry(&desc, &entry)) {\n \t\tif (match != all_entries_interesting) {\n@@ -136,7 +139,7 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tt->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_tree(ctx, t, base, entry.path, referred_objs);\n+\t\t\tprocess_tree(ctx, t, base, entry.path, referred_buf);\n \t\t}\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(ctx, entry.oid.hash,\n@@ -149,9 +152,10 @@ static void process_tree_contents(struct traversal_context *ctx,\n \t\t\t\t entry.path, oid_to_hex(&tree->object.oid));\n \t\t\t}\n \t\t\tb->object.flags |= NOT_USER_GIVEN;\n-\t\t\tprocess_blob(ctx, b, base, entry.path, referred_objs);\n+\t\t\tprocess_blob(ctx, b, base, entry.path, referred_buf);\n \t\t}\n \t}\n+\tfree(referred_buf);\n }\n \n static void process_tree(struct traversal_context *ctx,\ndiff --git a/object.c b/object.c\nindex f86b52c4d6..895068cbc2 100644\n--- a/object.c\n+++ b/object.c\n@@ -334,6 +334,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch\n \tstruct referred_objects *referred_objs;\n \treferred_objs = xmalloc(sizeof(struct referred_objects));\n \treferred_objs->commit = referred_commit;\n+\treferred_objs->trees = NULL;\n \n \tif (nr >= alloc) {\n \t\talloc = (alloc + 32) * 2;\ndiff --git a/object.h b/object.h\nindex 4db0ecc3f2..618d674249 100644\n--- a/object.h\n+++ b/object.h\n@@ -65,6 +65,7 @@ struct object_array {\n \n struct referred_objects{\n struct commit *commit;\n+ struct object_list *trees;\n };\n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439028","messageId":"5fc79a9a3213a17f1648dda9ee17d5fd82b0f979.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 09/12] t5702: test cases for excluding trees","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:33Z","receivedAt":"2021-10-19T11:40:07Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 213 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 213 insertions(+)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 6e323253f7..762a9c2505 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -1267,6 +1267,219 @@ test_expect_success 'commit-exclusion(excluding_type=ET_REACHABLE): part of pack\n \ttest_line_count = 4 filelist\n '\n \n+test_expect_success 'tree-exclusion(excluding_type=ET_SELF): part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\texcluding_type=\"0\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\t# Dir struct\n+\t# \t.\n+\t# |-- A.t\n+\t# |-- my-tree\n+\t# | `-- my-blob\n+\t# `-- other-tree\n+\t# |-- other-blob\n+\t# `-- sub-tree\n+\t# `-- sub-blob\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/other-tree &&\n+\techo other-blob >\"$P\"/other-tree/other-blob &&\n+\tmkdir \"$P\"/other-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/other-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add other-tree &&\n+ \ttest_commit -C \"$P\" A &&\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tothertreeh=$(git -C \"$P\" ls-tree HEAD other-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD other-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\totherblobh=$(git -C \"$P\" hash-object other-tree/other-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object other-tree/sub-tree/sub-blob) &&\n+\n+\tconfigure_exclusion tree \"$P\" \"$mytreeh\" new 0 >h &&\n+\tconfigure_exclusion tree \"$P\" \"$othertreeh\" new 0 >h2 &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+# \tEnsure that my-tree and other-tree and theirs complementary set are in separate packfiles.\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 1 out.objectlist\n+\t\tthen\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $othertreeh out\n+\t\t\tthen\n+\t\t\t\t>othertreehfound\n+\t\t\tfi\n+\t\telif test_line_count = 7 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $subtreeh out\n+\t\t\tthen\n+\t\t\t\t>subtreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $otherblobh out\n+\t\t\tthen\n+\t\t\t\t>otherblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $subblobh out\n+\t\t\tthen\n+\t\t\t\t>subblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f othertreehfound &&\n+\ttest -f otherblobhfound &&\n+\ttest -f subtreehfound &&\n+\ttest -f subblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\t# Ensure that there are exactly 3 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 6 filelist\n+'\n+\n+test_expect_success 'tree-exclusion(excluding_type=ET_INCLUDE), part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\ttest_when_finished \"rm -rf \\\"$P\\\" http_child log *found\" &&\n+\texcluding_type=\"1\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\t# Dir struct\n+\t# \t.\n+\t# |-- A.t\n+\t# |-- my-tree\n+\t# | `-- my-blob\n+\t# `-- other-tree\n+\t# |-- other-blob\n+\t# `-- sub-tree\n+\t# `-- sub-blob\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\tmkdir \"$P\"/other-tree &&\n+\techo other-blob >\"$P\"/other-tree/other-blob &&\n+\tmkdir \"$P\"/other-tree/sub-tree &&\n+\techo sub-blob >\"$P\"/other-tree/sub-tree/sub-blob &&\n+\tgit -C \"$P\" add other-tree &&\n+ \ttest_commit -C \"$P\" A &&\n+\n+ \tcommith=$(git -C \"$P\" rev-parse A) &&\n+ \troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tothertreeh=$(git -C \"$P\" ls-tree HEAD other-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tsubtreeh=$(git -C \"$P\" ls-tree HEAD other-tree/sub-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\totherblobh=$(git -C \"$P\" hash-object other-tree/other-blob) &&\n+\tsubblobh=$(git -C \"$P\" hash-object other-tree/sub-tree/sub-blob) &&\n+\n+\tconfigure_exclusion tree \"$P\" \"$mytreeh\" new $excluding_type >h &&\n+\tconfigure_exclusion tree \"$P\" \"$othertreeh\" new $excluding_type >h2 &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\t# Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles.\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 3 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi\n+\t\telif test_line_count = 2 out.objectlist\n+\t\tthen\n+\t\t\tif grep $mytreeh out\n+\t\t\tthen\n+\t\t\t\t>mytreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi\n+\t\telif test_line_count = 4 out.objectlist\n+\t\tthen\n+\t\t\tif grep $othertreeh out\n+\t\t\tthen\n+\t\t\t\t>othertreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $otherblobh out\n+\t\t\tthen\n+\t\t\t\t>otherblobhfound\n+\t\t\tfi &&\n+\t\t\tif grep $subtreeh out\n+\t\t\tthen\n+\t\t\t\t>subtreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $subblobh out\n+\t\t\tthen\n+\t\t\t\t>subblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f mytreehfound &&\n+\ttest -f myblobhfound &&\n+\ttest -f othertreehfound &&\n+\ttest -f otherblobhfound &&\n+\ttest -f subtreehfound &&\n+\ttest -f subblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\t# Ensure that there are exactly 3 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 6 filelist\n+'\n+\n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439029","messageId":"16c41c40a26e72d64959a332fbf7dc1c9794e70c.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 10/12] packfile-uri: support for excluding tags","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:34Z","receivedAt":"2021-10-19T11:40:11Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"From: Teng Long \n\nThis commit brings the tags exclusion feature of packfile-uri. The\nexcluding level of tag is supported with \"ET_SELF\", \"ET_INCLUDE\"\nand \"ET_REACHABLE\".\n\nExclusion scope on different level:\n\n1. When a tag is specified to be excluded with level \"ET_SELF\",\nonly the tag object itself will be excluded.\n\n2. When it's specified to be excluded with level \"ET_INCLUDE\",\nexclude the referenced commit, and all trees and blobs contained in its\ntop-level. If it's a annotated tag, the tag object will be excluded\ntoo.\n\n3. When it is specified with level \"ET_REACHABLE\", exclude the ancestors\nof the referenced commit, as well as the objects need to be excluded\nunder the level \"ET_INCLUDED\".\n\nSigned-off-by: Teng Long \n---\n builtin/pack-objects.c | 6 +++++-\n list-objects.c | 20 ++++++++++++++++----\n object.c | 1 +\n object.h | 1 +\n revision.c | 13 ++++++++++++-\n revision.h | 9 +++++++++\n 6 files changed, 44 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6713e734fb..073c3815a1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1416,6 +1416,9 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\tif (referred_objs) {\n \t\t\tstruct commit *commit = referred_objs->commit;\n \t\t\tstruct object_list *trees = referred_objs->trees;\n+\t\t\tstruct object_list *tags = referred_objs->tags;\n+\t\t\tif (want_exclude_object(tags))\n+\t\t\t\treturn 0;\n \t\t\tif (commit) {\n \t\t\t\tcommit_ex = oidmap_get(&configured_exclusions, &commit->object.oid);\n \t\t\t\tif (match_packfile_uri_exclusions(commit_ex) && commit_ex->level > ET_SELF)\n@@ -3344,7 +3347,8 @@ static void read_object_list_from_stdin(void)\n \n static void show_commit(struct commit *commit, struct show_info *info)\n {\n-\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL);\n+\tstruct referred_objects *referred_objs = info->show_cache;\n+\tadd_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, referred_objs);\n \tcommit->object.flags |= OBJECT_ADDED;\n \n \tif (write_bitmap_index)\ndiff --git a/list-objects.c b/list-objects.c\nindex 40292e2cc8..2e241e8707 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -348,9 +348,10 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \tfor (i = 0; i < ctx->revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = ctx->revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n-\t\tstruct referred_objects *referred_objs = pending->referred_objects;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n+\t\tstruct referred_objects *referred_objs = pending->referred_objects;\n+\t\tstruct commit_wraps_entry *cw_entry = NULL;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n@@ -358,6 +359,10 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\t\tctx->show_object(obj, name, &show_info);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (referred_objs->commit)\n+\t\t\tcw_entry = oidmap_get(ctx->revs->commit_wraps, &referred_objs->commit->object.oid);\n+\t\tif (cw_entry)\n+\t\t\treferred_objs->tags = cw_entry->wraps;\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n@@ -378,12 +383,14 @@ static void do_traverse(struct traversal_context *ctx)\n {\n \tstruct commit *commit;\n \tstruct strbuf csp; /* callee's scratch pad */\n-\tstruct show_info show_info;\n+\tstruct commit_wraps_entry *entry;\n+\tstruct oidmap *commit_wraps = ctx->revs->commit_wraps;\n+\tstruct referred_objects referred_objs = { NULL, NULL, NULL };\n+\tstruct show_info show_info = { ctx->show_data , NULL };\n \tstrbuf_init(&csp, PATH_MAX);\n \n-\n \tshow_info.show_data = ctx->show_data;\n-\tshow_info.show_cache = NULL;\n+\tshow_info.show_cache = &referred_objs;\n \n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n \t\t/*\n@@ -401,6 +408,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n \n+\t\tif (commit_wraps) {\n+\t\t\tentry = oidmap_get(commit_wraps, &commit->object.oid);\n+\t\t\treferred_objs.tags = entry ? entry->wraps : NULL;\n+\t\t}\n+\n \t\tctx->show_commit(commit, &show_info);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\ndiff --git a/object.c b/object.c\nindex 895068cbc2..167cc87ec9 100644\n--- a/object.c\n+++ b/object.c\n@@ -335,6 +335,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch\n \treferred_objs = xmalloc(sizeof(struct referred_objects));\n \treferred_objs->commit = referred_commit;\n \treferred_objs->trees = NULL;\n+\treferred_objs->tags = NULL;\n \n \tif (nr >= alloc) {\n \t\talloc = (alloc + 32) * 2;\ndiff --git a/object.h b/object.h\nindex 618d674249..42a2178531 100644\n--- a/object.h\n+++ b/object.h\n@@ -66,6 +66,7 @@ struct object_array {\n struct referred_objects{\n struct commit *commit;\n struct object_list *trees;\n+ struct object_list *tags;\n };\n #define OBJECT_ARRAY_INIT { 0, 0, NULL }\n \ndiff --git a/revision.c b/revision.c\nindex 89a8b311ea..d16f9bbbb9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -416,14 +416,17 @@ static struct commit *handle_commit(struct rev_info *revs,\n \tconst char *path = entry->path;\n \tunsigned int mode = entry->mode;\n \tunsigned long flags = object->flags;\n+\tstruct object_list *wraps = NULL;\n \n \t/*\n \t * Tag object? Look what it points to..\n \t */\n \twhile (object->type == OBJ_TAG) {\n \t\tstruct tag *tag = (struct tag *) object;\n-\t\tif (revs->tag_objects && !(flags & UNINTERESTING))\n+\t\tif (revs->tag_objects && !(flags & UNINTERESTING)) {\n \t\t\tadd_pending_object(revs, object, tag->tag);\n+\t\t\tobject_list_insert(object, &wraps);\n+\t\t}\n \t\tobject = parse_object(revs->repo, get_tagged_oid(tag));\n \t\tif (!object) {\n \t\t\tif (revs->ignore_missing_links || (flags & UNINTERESTING))\n@@ -449,6 +452,14 @@ static struct commit *handle_commit(struct rev_info *revs,\n \t */\n \tif (object->type == OBJ_COMMIT) {\n \t\tstruct commit *commit = (struct commit *)object;\n+\t\tstruct oidmap *commit_wraps = malloc(sizeof(struct oidmap));\n+\t\tstruct commit_wraps_entry *cw_entry = xmalloc(sizeof(struct commit_wraps_entry));\n+\n+\t\toidmap_init(commit_wraps, 0);\n+\t\tcw_entry->e.oid = object->oid;\n+\t\tcw_entry->wraps = wraps;\n+\t\toidmap_put(commit_wraps, cw_entry);\n+\t\trevs->commit_wraps = commit_wraps;\n \n \t\tif (repo_parse_commit(revs->repo, commit) < 0)\n \t\t\tdie(\"unable to parse commit %s\", name);\ndiff --git a/revision.h b/revision.h\nindex b2e0c0b9b7..a633ea2174 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,6 +8,7 @@\n #include \"pretty.h\"\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n+#include \"oidmap.h\"\n \n /**\n * The revision walking API offers functions to build a list of revisions\n@@ -64,6 +65,11 @@ struct bloom_key;\n struct bloom_filter_settings;\n define_shared_commit_slab(revision_sources, char *);\n \n+struct commit_wraps_entry {\n+ struct oidmap_entry e;\n+ struct object_list *wraps;\n+};\n+\n struct rev_cmdline_info {\n \tunsigned int nr;\n \tunsigned int alloc;\n@@ -321,6 +327,9 @@ struct rev_info {\n \n \t/* misc. flags related to '--no-kept-objects' */\n \tunsigned keep_pack_cache_flags;\n+\n+\t/* The commit_wraps caches the referred wrapped objects(such as tags) of a commit */\n+\tstruct oidmap *commit_wraps;\n };\n \n int ref_excluded(struct string_list *, const char *path);\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439030","messageId":"171ece533b4aecf270d93f523086a257dd3edb32.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 11/12] t5702: test cases for excluding tags","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:35Z","receivedAt":"2021-10-19T11:40:11Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n t/t5702-protocol-v2.sh | 197 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 197 insertions(+)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 762a9c2505..f91ae62fd8 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -1480,6 +1480,203 @@ test_expect_success 'tree-exclusion(excluding_type=ET_INCLUDE), part of packfile\n \ttest_line_count = 6 filelist\n '\n \n+test_expect_success 'tag-exclusion(excluding_type=ET_SELF): part of packfile response provided as URI' '\n+ \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n+\tgit -C \"$P\" tag -a -m \"annotated_tag\" tagA &&\n+\n+\ttagh=$(git -C \"$P\" rev-parse tagA) &&\n+\tcommith=$(git -C \"$P\" rev-parse A) &&\n+\troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-blob) &&\n+\n+\tconfigure_exclusion tag \"$P\" \"$tagh\" new 0 >h2 &&\n+\tgit init http_child &&\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=`pwd`/log GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child \\\n+\t\t-c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch --tags \"$HTTPD_URL/smart/http_parent\" &&\n+\n+\t# Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles.\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 1 out.objectlist\n+\t\tthen\n+\t\t\tif grep $tagh out\n+\t\t\tthen\n+\t\t\t\t>taghfound\n+\t\t\tfi\n+\n+\t\telif test_line_count = 4 out.objectlist\n+\t\tthen\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f myblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\ttest -f taghfound &&\n+\t# Ensure that there are exactly 3 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 4 filelist\n+'\n+\n+test_expect_success 'tag-exclusion(excluding_type=ET_INCLUDE): part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+ \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\techo my-blob >\"$P/my-blob\" &&\n+\tgit -C \"$P\" add my-blob &&\n+\ttest_commit -C \"$P\" A &&\n+\tgit -C \"$P\" tag -a -m \"annotated_tag\" tagA &&\n+\n+\ttagh=$(git -C \"$P\" rev-parse tagA) &&\n+\tcommith=$(git -C \"$P\" rev-parse A) &&\n+\troottreeh=$(git -C \"$P\" rev-parse A:) &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-blob) &&\n+\n+\tconfigure_exclusion tag \"$P\" \"$tagh\" new 1 >h2 &&\n+\n+\tgit init http_child &&\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=`pwd`/log GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -C http_child \\\n+\t\t-c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tfetch --tags \"$HTTPD_URL/smart/http_parent\" &&\n+\n+\t# Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles.\n+\tfor idx in http_child/.git/objects/pack/*.idx\n+\tdo\n+\t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+\t\t{\n+\t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+\t\t} >out.objectlist &&\n+\t\tif test_line_count = 5 out.objectlist\n+\t\tthen\n+\t\t\tif grep $tagh out\n+\t\t\tthen\n+\t\t\t\t>taghfound\n+\t\t\tfi &&\n+\t\t\tif grep $commith out\n+\t\t\tthen\n+\t\t\t\t>commithfound\n+\t\t\tfi &&\n+\t\t\tif grep $roottreeh out\n+\t\t\tthen\n+\t\t\t\t>roottreehfound\n+\t\t\tfi &&\n+\t\t\tif grep $ah out\n+\t\t\tthen\n+\t\t\t\t>ahfound\n+\t\t\tfi &&\n+\t\t\tif grep $myblobh out\n+\t\t\tthen\n+\t\t\t\t>myblobhfound\n+\t\t\tfi\n+\t\tfi\n+\tdone &&\n+\ttest -f myblobhfound &&\n+\ttest -f commithfound &&\n+\ttest -f roottreehfound &&\n+\ttest -f ahfound &&\n+\ttest -f taghfound &&\n+\t# Ensure that there are exactly 3 packfiles with associated .idx\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 4 filelist\n+'\n+\n+test_expect_success 'tag-exclusion(excluding_type=ET_REACHABLE): part of packfile response provided as URI' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+ \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\n+\tmkdir \"$P\"/my-tree &&\n+\techo my-blob >\"$P\"/my-tree/my-blob &&\n+\tgit -C \"$P\" add my-tree &&\n+\ttest_commit -C \"$P\" A &&\n+\n+\tmkdir \"$P\"/other-tree &&\n+\techo other-blob >\"$P\"/other-tree/other-blob &&\n+\tgit -C \"$P\" add other-tree &&\n+\ttest_commit -C \"$P\" B &&\n+\n+\tgit -C \"$P\" tag -a -m \"tag X\" tagX &&\n+\n+\ttagh=$(git -C \"$P\" rev-parse tagX) &&\n+ \tcommitAh=$(git -C \"$P\" rev-parse A) &&\n+ \tcommitBh=$(git -C \"$P\" rev-parse B) &&\n+ \troottreeAh=$(git -C \"$P\" rev-parse A:) &&\n+\troottreeBh=$(git -C \"$P\" rev-parse B:) &&\n+\tmytreeh=$(git -C \"$P\" ls-tree HEAD my-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tothertreeh=$(git -C \"$P\" ls-tree HEAD other-tree | sed -ne \"s/.*\\($OID_REGEX\\).*/\\1/p\") &&\n+\tah=$(git -C \"$P\" hash-object A.t) &&\n+\tbh=$(git -C \"$P\" hash-object B.t) &&\n+\tmyblobh=$(git -C \"$P\" hash-object my-tree/my-blob) &&\n+\totherblobh=$(git -C \"$P\" hash-object other-tree/other-blob) &&\n+\n+\tconfigure_exclusion tag \"$P\" \"$tagh\" new 2 >h &&\n+\n+\tGIT_TRACE=1 GIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TEST_SIDEBAND_ALL=1 \\\n+\tgit -c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\n+\t# Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles.\n+ \tfor idx in http_child/.git/objects/pack/*.idx\n+ \tdo\n+ \t\tgit verify-pack --object-format=$(test_oid algo) --verbose $idx >out &&\n+ \t\t{\n+ \t\t\tgrep \"^[0-9a-f]\\{16,\\} \" out || :\n+ \t\t} >out.objectlist &&\n+ \t\tif test_line_count = 11 out.objectlist\n+ \t\tthen\n+ \t\t\t\t>fullpackfound\n+ \t\telif test_line_count = 0 out.objectlist\n+ \t\tthen\n+\t\t\t\t\t>emptypackfound\n+ \t\tfi\n+ \tdone &&\n+ \ttest -f fullpackfound &&\n+ \ttest -f emptypackfound &&\n+ \t# Ensure that there are exactly 3 packfiles with associated .idx\n+ \tls http_child/.git/objects/pack/*.pack \\\n+ \t\thttp_child/.git/objects/pack/*.idx >filelist &&\n+ \ttest_line_count = 4 filelist\n+'\n+\n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \ttest_when_finished \"rm -rf \\\"$P\\\" http_child log\" &&\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"},{"id":"439031","messageId":"79fc2c23cf053cb525d818b6b47e24c49b57f672.1634634814.git.tenglong@alibaba-inc.com","threadId":"55638","inReplyTo":"cover.1634634814.git.tenglong@alibaba-inc.com","subject":"[PATCH v6 12/12] packfile-uri.txt: support multiple object types","fromName":"Teng Long","fromEmail":"dyroneteng@gmail.com","sentAt":"2021-10-19T11:38:36Z","receivedAt":"2021-10-19T11:40:14Z","isPatch":true,"sender":{"key":"dyroneteng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/7803958?v=4"},"body":"Signed-off-by: Teng Long \n---\n Documentation/technical/packfile-uri.txt | 74 ++++++++++++++++++++----\n 1 file changed, 63 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt\nindex f7eabc6c76..0e6d7c8dda 100644\n--- a/Documentation/technical/packfile-uri.txt\n+++ b/Documentation/technical/packfile-uri.txt\n@@ -34,14 +34,69 @@ having it advertise `packfile-uris`, tolerating the client sending\n include some sort of non-trivial implementation in the Minimum Viable Product,\n at least so that we can test the client.\n \n-This is the implementation: a feature, marked experimental, that allows the\n-server to be configured by one or more `uploadpack.blobPackfileUri=\n-` entries. Whenever the list of objects to be sent is assembled, all such\n-blobs are excluded, replaced with URIs. As noted in \"Future work\" below, the\n-server can evolve in the future to support excluding other objects (or other\n-implementations of servers could be made that support excluding other objects)\n-without needing a protocol change, so clients should not expect that packfiles\n-downloaded in this way only contain single blobs.\n+This is the implementation: a feature, marked experimental, that allows\n+the server to be configured by one or more entries with the format:\n+\n+ uploadpack.excludeobject= \n+\n+Value `` is the key of entry, and the object type can be\n+blob, tree, commit, or tag. Value of entry has three parts,\n+`` is used to identify the packfile which contains the given\n+`` object, and `` is the URI to download the packfile by\n+client. For example, When a blob is configured with `uploadpack.excludeobject`\n+that means whenever the blob to be send is assembled, the object will\n+be excluded.\n+\n+In addition to excluding a single object like blob, sometimes it's\n+hoped to exclude not only the object itself, but also all the related\n+objects with it, like all the objects a tree contains or the ancestors\n+that a commit can reach. In these cases, the `` is designed to\n+distinguish the scope of exclusion, it supports three levels:\n+\n+- Level 0: Excluding a single object itself, without any objects that\n+ have a relationship with it. \n+\n+- Level 1: Excluding object itself, and objects it contains.\n+\n+- Level 2: Excluding object itself, the objects it contains, and the\n+ ancestors it can reach.\n+\n+If `` is configured as 0, only the object itself will be\n+excluded, no matter what the object type is. It is a common scenario\n+for large size blobs, but it does much not sense for other object types\n+(e.g. download a singe commit without downloading the blobs and tree\n+in it).\n+\n+If `` is configured as 1, not only the single object but also all\n+the objects in it will be excluded. This applies to scenarios where\n+it's wanted to exclude a specified non-blob object that includes some\n+lage size objects.\n+\n+- If is a blob, the result is the same as level 0, because blob\n+contains nothing just itself.\n+\n+- If is a tree, the tree itself, and all blobs and trees\n+ in it will be excluded.\n+\n+- If is a commit, the commit itself, the referenced\n+ root-tree, and all blobs and trees in the root-tree will be excluded.\n+\n+- If is a tag, the tag itself, the dereferenced commit\n+ and all trees and blobs contained in its root-tree will be excluded.\n+\n+If `` is configured as 2, not only the objects in the scope of\n+level 1 , but also the reachable ancestors will be excluded if\n+`` is commit or tag.\n+\n+Configuration compatibility\n+---------------------------\n+\n+The old configuration of packfile-uri:\n+\n+ uploadpack.blobPackfileUri= \n+\n+For the old configuration is compatible with the new one, but it only\n+supports the exclusion of blob objects.\n \n Client design\n -------------\n@@ -65,9 +120,6 @@ The protocol design allows some evolution of the server and client without any\n need for protocol changes, so only a small-scoped design is included here to\n form the MVP. For example, the following can be done:\n \n- * On the server, more sophisticated means of excluding objects (e.g. by\n- specifying a commit to represent that commit and all objects that it\n- references).\n * On the client, resumption of clone. If a clone is interrupted, information\n could be recorded in the repository's config and a \"clone-resume\" command\n can resume the clone in progress. (Resumption of subsequent fetches is more\n-- \n2.31.1.453.g945ddc3a74.dirty\n\n"}]}