{"thread":{"id":"58706","subject":"[PATCH 1/2] connected: allow supplying different view of reachable objects","startedAt":"2022-10-28T14:42:57Z","lastAt":"2022-11-17T21:26:21Z","messageCount":88,"participants":["Patrick Steinhardt","Ævar Arnfjörð Bjarmason","Junio C Hamano","Taylor Blau","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"465945","messageId":"a32e3d6146dd41af36f525a744d6cc099b42d6fb.1666967670.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"[PATCH 1/2] connected: allow supplying different view of reachable objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-10-28T14:42:23Z","receivedAt":"2022-10-28T14:42:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The connectivity check is executed via git-receive-pack(1) to verify\nthat a client has provided all references that are required to satisfy a\nset of reference updates. What the connectivity check does is to walk\nthe object graph with all reference tips as starting points while all\npreexisting reference tips are marked as uninteresting.\n\nPreexisting references are currently marked uninteresting by passing\n`--not --all` to git-rev-list(1). Some users of the connectivity check\nmay have a better picture of which objects should be regarded as\nuninteresting though, e.g. by reusing information from the reference\nadvertisement when serving a push.\n\nAdd a new field to `struct check_connected_options` that allows callers\nto replace the `--not --all` logic with their own set of object IDs they\nregard as uninteresting.\n\nSigned-off-by: Patrick Steinhardt \n---\n connected.c | 9 ++++++++-\n connected.h | 7 +++++++\n 2 files changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/connected.c b/connected.c\nindex 74a20cb32e..2a4c4e0025 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -98,7 +98,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \tstrvec_push(&rev_list.args, \"--stdin\");\n \tif (has_promisor_remote())\n \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n-\tif (!opt->is_deepening_fetch) {\n+\tif (!opt->is_deepening_fetch && !opt->reachable_oids_fn) {\n \t\tstrvec_push(&rev_list.args, \"--not\");\n \t\tstrvec_push(&rev_list.args, \"--all\");\n \t}\n@@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \n \trev_list_in = xfdopen(rev_list.in, \"w\");\n \n+\tif (opt->reachable_oids_fn) {\n+\t\tconst struct object_id *reachable_oid;\n+\t\twhile ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)\n+\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(reachable_oid)) < 0)\n+\t\t\t\tbreak;\n+\t}\n+\n \tdo {\n \t\t/*\n \t\t * If index-pack already checked that:\ndiff --git a/connected.h b/connected.h\nindex 6e59c92aa3..f09c7d7884 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -46,6 +46,13 @@ struct check_connected_options {\n \t * during a fetch.\n \t */\n \tunsigned is_deepening_fetch : 1;\n+\n+\t/*\n+\t * If non-NULL, use this iterator to determine the set of reachable\n+\t * objects instead of marking all references as unreachable.\n+\t */\n+\toid_iterate_fn reachable_oids_fn;\n+\tvoid *reachable_oids_data;\n };\n \n #define CHECK_CONNECTED_INIT { 0 }\n-- \n2.38.1\n\n"},{"id":"465946","messageId":"cover.1666967670.git.ps@pks.im","threadId":"58706","inReplyTo":null,"subject":"[PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-10-28T14:42:19Z","receivedAt":"2022-10-28T14:42:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series improves the connectivity check done by stateful\ngit-receive-pack(1) to only consider references as reachable that have\nbeen advertised to the client. This has two advantages:\n\n - A client shouldn't assume objects to exist that have not been part\n of the reference advertisement. But if it excluded an object from\n the packfile that is reachable via any ref that is excluded from\n the reference advertisement due to `transfer.hideRefs` we'd have\n accepted the push anyway. I'd argue that this is a bug in the\n current implementation.\n\n - Second, by using advertised refs as inputs instead of `git\n rev-list --not --all` we avoid looking up all refs that are\n irrelevant to the current push. This can be a huge performance\n improvement in repos that have a huge amount of internal, hidden\n refs. In one of our repos with 7m refs, of which 6.8m are hidden,\n this speeds up pushes from ~30s to ~4.5s.\n\nOne downside is that we need to pass in the object IDs that were part of\nthe reference advertisement via the standard input, which is seemingly\nslower than reading them from the refdb. I'm discussing this in the\nsecond commit.\n\nPatrick\n\nPatrick Steinhardt (2):\n connected: allow supplying different view of reachable objects\n receive-pack: use advertised reference tips to inform connectivity\n check\n\n builtin/receive-pack.c | 31 ++++++++++++++++++++++---------\n connected.c | 9 ++++++++-\n connected.h | 7 +++++++\n 3 files changed, 37 insertions(+), 10 deletions(-)\n\n-- \n2.38.1\n\n"},{"id":"465947","messageId":"006e89f384be1227b922fb6fdc8755ae84cac587.1666967670.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"[PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-10-28T14:42:27Z","receivedAt":"2022-10-28T14:43:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When serving a push, git-receive-pack(1) needs to verify that the\npackfile sent by the client contains all objects that are required by\nthe updated references. This connectivity check works by marking all\npreexisting references as uninteresting and using the new reference tips\nas starting point for a graph walk.\n\nThis strategy has the major downside that it will not require any object\nto be sent by the client that is reachable by any of the repositories'\nreferences. While that sounds like it would be indeed what we are after\nwith the connectivity check, it is arguably not. The administrator that\nmanages the server-side Git repository may have configured certain refs\nto be hidden during the reference advertisement via `transfer.hideRefs`\nor `receivepack.hideRefs`. Whatever the reason, the result is that the\nclient shouldn't expect that any of those hidden references exists on\nthe remote side, and neither should they assume any of the pointed-to\nobjects to exist except if referenced by any visible reference. But\nbecause we treat _all_ local refs as uninteresting in the connectivity\ncheck, a client is free to send a packfile that references objects that\nare only reachable via a hidden reference on the server-side, and we\nwill gladly accept it.\n\nBesides the correctness issue there is also a performance issue. Git\nforges tend to do internal bookkeeping to keep alive sets of objects for\ninternal use or make them easy to find via certain references. These\nreferences are typically hidden away from the user so that they are\nneither advertised nor writeable. At GitLab, we have one particular\nrepository that contains a total of 7 million references, of which 6.8\nmillion are indeed internal references. With the current connectivity\ncheck we are forced to load all these references in order to mark them\nas uninteresting, and this alone takes around 15 seconds to compute.\n\nWe can fix both of these issues by changing the logic for stateful\ninvocations of git-receive-pack(1) where the reference advertisement and\npackfile negotiation are served by the same process. Instead of marking\nall preexisting references as unreachable, we will only mark those that\nwe have announced to the client.\n\nBesides the stated fix to correctness this also provides a huge boost to\nperformance in the repository mentioned above. Pushing a new commit into\nthis repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\nas it is configured in Gitaly leads to an almost 7.5-fold speedup:\n\n Benchmark 1: main\n Time (mean ± σ): 29.902 s ± 0.105 s [User: 29.176 s, System: 1.052 s]\n Range (min … max): 29.781 s … 29.969 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 4.033 s ± 0.088 s [User: 4.071 s, System: 0.374 s]\n Range (min … max): 3.953 s … 4.128 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 7.42 ± 0.16 times faster than 'main'\n\nUnfortunately, this change comes with a performance hit when refs are\nnot hidden. Executed in the same repository:\n\n Benchmark 1: main\n Time (mean ± σ): 45.780 s ± 0.507 s [User: 46.908 s, System: 4.838 s]\n Range (min … max): 45.453 s … 46.364 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 49.886 s ± 0.282 s [User: 51.168 s, System: 5.015 s]\n Range (min … max): 49.589 s … 50.149 s 3 runs\n\n Summary\n 'main' ran\n 1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'\n\nThis is probably caused by the overhead of reachable tips being passed\nin via git-rev-list(1)'s standard input, which seems to be slower than\nreading the references from disk.\n\nIt is debatable what to do about this. If this were only about improving\nperformance then it would be trivial to make the new logic depend on\nwhether or not `transfer.hideRefs` has been configured in the repo. But\nas explained this is also about correctness, even though this can be\nconsidered an edge case. Furthermore, this slowdown is really only\nnoticeable in outliers like the above repository with an unreasonable\namount of refs. The same benchmark in linux-stable.git with about\n4500 references shows no measurable difference:\n\n Benchmark 1: main\n Time (mean ± σ): 375.4 ms ± 25.4 ms [User: 312.2 ms, System: 155.7 ms]\n Range (min … max): 324.2 ms … 492.9 ms 50 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 374.9 ms ± 36.9 ms [User: 311.6 ms, System: 158.2 ms]\n Range (min … max): 319.2 ms … 583.1 ms 50 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 1.00 ± 0.12 times faster than 'main'\n\nLet's keep this as-is for the time being and accept the performance hit.\nIt is arguably extremely noticeable to a user if a push now performs 7.5\ntimes faster than before, but a lot less so in case an already-slow push\nbecomes about 10% slower.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 31 ++++++++++++++++++++++---------\n 1 file changed, 22 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 44bcea3a5b..50794539c6 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -326,13 +326,10 @@ static void show_one_alternate_ref(const struct object_id *oid,\n \tshow_ref(\".have\", oid);\n }\n \n-static void write_head_info(void)\n+static void write_head_info(struct oidset *announced_objects)\n {\n-\tstatic struct oidset seen = OIDSET_INIT;\n-\n-\tfor_each_ref(show_ref_cb, &seen);\n-\tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n-\toidset_clear(&seen);\n+\tfor_each_ref(show_ref_cb, announced_objects);\n+\tfor_each_alternate_ref(show_one_alternate_ref, announced_objects);\n \tif (!sent_capabilities)\n \t\tshow_ref(\"capabilities^{}\", null_oid());\n \n@@ -1896,12 +1893,20 @@ static void execute_commands_atomic(struct command *commands,\n \tstrbuf_release(&err);\n }\n \n+static const struct object_id *iterate_announced_oids(void *cb_data)\n+{\n+\tstruct oidset_iter *iter = cb_data;\n+\treturn oidset_iter_next(iter);\n+}\n+\n static void execute_commands(struct command *commands,\n \t\t\t const char *unpacker_error,\n \t\t\t struct shallow_info *si,\n-\t\t\t const struct string_list *push_options)\n+\t\t\t const struct string_list *push_options,\n+\t\t\t struct oidset *announced_oids)\n {\n \tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n+\tstruct oidset_iter announced_oids_iter;\n \tstruct command *cmd;\n \tstruct iterate_data data;\n \tstruct async muxer;\n@@ -1928,6 +1933,12 @@ static void execute_commands(struct command *commands,\n \topt.err_fd = err_fd;\n \topt.progress = err_fd && !quiet;\n \topt.env = tmp_objdir_env(tmp_objdir);\n+\tif (oidset_size(announced_oids) != 0) {\n+\t\toidset_iter_init(announced_oids, &announced_oids_iter);\n+\t\topt.reachable_oids_fn = iterate_announced_oids;\n+\t\topt.reachable_oids_data = &announced_oids_iter;\n+\t}\n+\n \tif (check_connected(iterate_receive_command_list, &data, &opt))\n \t\tset_connectivity_errors(commands, si);\n \n@@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n {\n \tint advertise_refs = 0;\n \tstruct command *commands;\n+\tstruct oidset announced_oids = OIDSET_INIT;\n \tstruct oid_array shallow = OID_ARRAY_INIT;\n \tstruct oid_array ref = OID_ARRAY_INIT;\n \tstruct shallow_info si;\n@@ -2524,7 +2536,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (advertise_refs || !stateless_rpc) {\n-\t\twrite_head_info();\n+\t\twrite_head_info(&announced_oids);\n \t}\n \tif (advertise_refs)\n \t\treturn 0;\n@@ -2554,7 +2566,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t\tuse_keepalive = KEEPALIVE_ALWAYS;\n \t\texecute_commands(commands, unpack_status, &si,\n-\t\t\t\t &push_options);\n+\t\t\t\t &push_options, &announced_oids);\n \t\tif (pack_lockfile)\n \t\t\tunlink_or_warn(pack_lockfile);\n \t\tsigchain_push(SIGPIPE, SIG_IGN);\n@@ -2591,6 +2603,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n+\toidset_clear(&announced_oids);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\n-- \n2.38.1\n\n"},{"id":"465948","messageId":"221028.86bkpw805n.gmgdl@evledraar.gmail.com","threadId":"58706","inReplyTo":"a32e3d6146dd41af36f525a744d6cc099b42d6fb.1666967670.git.ps@pks.im","subject":"Re: [PATCH 1/2] connected: allow supplying different view of reachable objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-28T14:54:56Z","receivedAt":"2022-10-28T14:59:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 28 2022, Patrick Steinhardt wrote:\n\n> @@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n> \n> \trev_list_in = xfdopen(rev_list.in, \"w\");\n> \n> +\tif (opt->reachable_oids_fn) {\n> +\t\tconst struct object_id *reachable_oid;\n> +\t\twhile ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)\n> +\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(reachable_oid)) < 0)\n> +\t\t\t\tbreak;\n\nJust a style nit, we tend to avoid != NULL, != 0 etc. comparisons. I see\nconnected.c has some of that already, but for new code let's just check\ntruthiness.\n\nAlso for such a small scope a shorter variable name helps us stay at the\nusual column limits:\n\n\tif (opt->reachable_oids_fn) {\n\t\tconst struct object_id *oid;\n\t\twhile ((oid = opt->reachable_oids_fn(opt->reachable_oids_data)))\n\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(oid)) < 0)\n\t\t\t\tbreak;\n\nThe fprintf() return value checking seemed a bit odd, not because we\nshouldn't do it, but because we usually don't bother. For other\nreviewers: We have that form already in connected.c, so at least locally\nwe're not being diligently careful, only to have it undone by adjacent\ncode...\n\nLooks good!\n"},{"id":"465953","messageId":"221028.867d0k7yny.gmgdl@evledraar.gmail.com","threadId":"58706","inReplyTo":"006e89f384be1227b922fb6fdc8755ae84cac587.1666967670.git.ps@pks.im","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-28T15:01:58Z","receivedAt":"2022-10-28T15:31:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 28 2022, Patrick Steinhardt wrote:\n\n> When serving a push, git-receive-pack(1) needs to verify that the\n> packfile sent by the client contains all objects that are required by\n> the updated references. This connectivity check works by marking all\n> preexisting references as uninteresting and using the new reference tips\n> as starting point for a graph walk.\n>\n> This strategy has the major downside that it will not require any object\n> to be sent by the client that is reachable by any of the repositories'\n> references. While that sounds like it would be indeed what we are after\n> with the connectivity check, it is arguably not. The administrator that\n> manages the server-side Git repository may have configured certain refs\n> to be hidden during the reference advertisement via `transfer.hideRefs`\n> or `receivepack.hideRefs`. Whatever the reason, the result is that the\n> client shouldn't expect that any of those hidden references exists on\n> the remote side, and neither should they assume any of the pointed-to\n> objects to exist except if referenced by any visible reference. But\n> because we treat _all_ local refs as uninteresting in the connectivity\n> check, a client is free to send a packfile that references objects that\n> are only reachable via a hidden reference on the server-side, and we\n> will gladly accept it.\n>\n> Besides the correctness issue there is also a performance issue. Git\n> forges tend to do internal bookkeeping to keep alive sets of objects for\n> internal use or make them easy to find via certain references. These\n> references are typically hidden away from the user so that they are\n> neither advertised nor writeable. At GitLab, we have one particular\n> repository that contains a total of 7 million references, of which 6.8\n> million are indeed internal references. With the current connectivity\n> check we are forced to load all these references in order to mark them\n> as uninteresting, and this alone takes around 15 seconds to compute.\n>\n> We can fix both of these issues by changing the logic for stateful\n> invocations of git-receive-pack(1) where the reference advertisement and\n> packfile negotiation are served by the same process. Instead of marking\n> all preexisting references as unreachable, we will only mark those that\n> we have announced to the client.\n>\n> Besides the stated fix to correctness this also provides a huge boost to\n> performance in the repository mentioned above. Pushing a new commit into\n> this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\n> as it is configured in Gitaly leads to an almost 7.5-fold speedup:\n\nReally well explained.\n\n> Benchmark 1: main\n> Time (mean ± σ): 29.902 s ± 0.105 s [User: 29.176 s, System: 1.052 s]\n> Range (min … max): 29.781 s … 29.969 s 3 runs\n>\n> Benchmark 2: pks-connectivity-check-hide-refs\n> Time (mean ± σ): 4.033 s ± 0.088 s [User: 4.071 s, System: 0.374 s]\n> Range (min … max): 3.953 s … 4.128 s 3 runs\n>\n> Summary\n> 'pks-connectivity-check-hide-refs' ran\n> 7.42 ± 0.16 times faster than 'main'\n\nAnd impressive, thanks!\n\n> Unfortunately, this change comes with a performance hit when refs are\n> not hidden. Executed in the same repository:\n>\n> Benchmark 1: main\n> Time (mean ± σ): 45.780 s ± 0.507 s [User: 46.908 s, System: 4.838 s]\n> Range (min … max): 45.453 s … 46.364 s 3 runs\n>\n> Benchmark 2: pks-connectivity-check-hide-refs\n> Time (mean ± σ): 49.886 s ± 0.282 s [User: 51.168 s, System: 5.015 s]\n> Range (min … max): 49.589 s … 50.149 s 3 runs\n>\n> Summary\n> 'main' ran\n> 1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'\n>\n> This is probably caused by the overhead of reachable tips being passed\n> in via git-rev-list(1)'s standard input, which seems to be slower than\n> reading the references from disk.\n>\n> It is debatable what to do about this. If this were only about improving\n> performance then it would be trivial to make the new logic depend on\n> whether or not `transfer.hideRefs` has been configured in the repo. But\n> as explained this is also about correctness, even though this can be\n> considered an edge case. Furthermore, this slowdown is really only\n> noticeable in outliers like the above repository with an unreasonable\n> amount of refs. The same benchmark in linux-stable.git with about\n> 4500 references shows no measurable difference:\n\nDo we have a test that would start failing if we changed the behavior?\nPerhaps such a test is peeking too much behind the curtain, but if it's\neasy come up with one I think it would be most welcome to have it\nalongside this. to have exposes\n\n> -static void write_head_info(void)\n> +static void write_head_info(struct oidset *announced_objects)\n> {\n> -\tstatic struct oidset seen = OIDSET_INIT;\n> -\n> -\tfor_each_ref(show_ref_cb, &seen);\n> -\tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n> -\toidset_clear(&seen);\n> +\tfor_each_ref(show_ref_cb, announced_objects);\n> +\tfor_each_alternate_ref(show_one_alternate_ref, announced_objects);\n> \tif (!sent_capabilities)\n> \t\tshow_ref(\"capabilities^{}\", null_oid());\n\nNit: The variable rename stands out slightly,\ni.e. s/&seen/announced_objects/ not s/&seen/seen/, especially as:\n\n> static void execute_commands(struct command *commands,\n> \t\t\t const char *unpacker_error,\n> \t\t\t struct shallow_info *si,\n> -\t\t\t const struct string_list *push_options)\n> +\t\t\t const struct string_list *push_options,\n> +\t\t\t struct oidset *announced_oids)\n\nHere we have the same variable, but now it's *_oids, not *objects.\n\n> +\tif (oidset_size(announced_oids) != 0) {\n\nNit as before: The \"!= 0\" can go here.\n\n> @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> {\n> \tint advertise_refs = 0;\n> \tstruct command *commands;\n> +\tstruct oidset announced_oids = OIDSET_INIT;\n> \tstruct oid_array shallow = OID_ARRAY_INIT;\n> \tstruct oid_array ref = OID_ARRAY_INIT;\n> \tstruct shallow_info si;\n> @@ -2524,7 +2536,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> \t}\n> \n> \tif (advertise_refs || !stateless_rpc) {\n> -\t\twrite_head_info();\n> +\t\twrite_head_info(&announced_oids);\n> \t}\n> \tif (advertise_refs)\n> \t\treturn 0;\n\nThis introduces a memory leak to the function., We probably have other\nones in code it calls, but from a quick eyeballing not in the function\nitself.\n\nSquashing in / combining it with this should do it, as it never returns\nnon-zero (except for calling die()):\n\t\n\tdiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n\tindex 44bcea3a5b3..8d5c2fbef1c 100644\n\t--- a/builtin/receive-pack.c\n\t+++ b/builtin/receive-pack.c\n\t@@ -2527,7 +2527,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n\t \t\twrite_head_info();\n\t \t}\n\t \tif (advertise_refs)\n\t-\t\treturn 0;\n\t+\t\tgoto cleanup;\n\t \n\t \tpacket_reader_init(&reader, 0, NULL, 0,\n\t \t\t\t PACKET_READ_CHOMP_NEWLINE |\n\t@@ -2587,6 +2587,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n\t \t\t\tupdate_server_info(0);\n\t \t\tclear_shallow_info(&si);\n\t \t}\n\t+cleanup:\n\t \tif (use_sideband)\n\t \t\tpacket_flush(1);\n\t \toid_array_clear(&shallow);\n\n> @@ -2591,6 +2603,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> \t\tpacket_flush(1);\n> \toid_array_clear(&shallow);\n> \toid_array_clear(&ref);\n> +\toidset_clear(&announced_oids);\n> \tfree((void *)push_cert_nonce);\n> \treturn 0;\n> }\n\nWe'll then properly reach this new oidset_clear()> The oid_array_clear()\nare all for variables we're populating after we're past htat \"if\n(advertise_refs)\".\n\nI think if you're re-rolling this sqashing 1/2 and 2/2 together would be\nan improvement. The 1/2 is tiny, and it's an API that's not used until\nthis 1/2. I found myself going back & forth more than helped in\nreviewing this.\n\nGgoing back a bit this:\n\n> +static const struct object_id *iterate_announced_oids(void *cb_data)\n> +{\n> +\tstruct oidset_iter *iter = cb_data;\n> +\treturn oidset_iter_next(iter);\n> +}\n> +\n\nIs just used as (from 1/2):\n\n> +\tif (opt->reachable_oids_fn) {\n> +\t\tconst struct object_id *reachable_oid;\n> +\t\twhile ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)\n> +\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(reachable_oid)) < 0)\n> +\t\t\t\tbreak;\n> +\t}\n\nAfter doing above:\n\n> +\tif (oidset_size(announced_oids) != 0) {\n> +\t\toidset_iter_init(announced_oids, &announced_oids_iter);\n> +\t\topt.reachable_oids_fn = iterate_announced_oids;\n> +\t\topt.reachable_oids_data = &announced_oids_iter;\n> +\t}\n\nBut I don't see the reason for the indirection, but maybe I'm missing\nsomething obvious.\n\nWhy not just pass the oidset itself and have connected.c iterate through\nit, rather than going thorugh this callback / data indirection?\n\n"},{"id":"465958","messageId":"xmqq8rkzkikw.fsf@gitster.g","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-28T16:40:15Z","receivedAt":"2022-10-28T16:40:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this patch series improves the connectivity check done by stateful\n> git-receive-pack(1) to only consider references as reachable that have\n> been advertised to the client. This has two advantages:\n>\n> - A client shouldn't assume objects to exist that have not been part\n> of the reference advertisement. But if it excluded an object from\n> the packfile that is reachable via any ref that is excluded from\n> the reference advertisement due to `transfer.hideRefs` we'd have\n> accepted the push anyway. I'd argue that this is a bug in the\n> current implementation.\n\nI agree that it is bad to accept an incoming pack that depends on an\nobject that is *only* reachable by a hidden ref, but what you said\nabove is a bit stronger than that. Use transfer.hideRefs does not\nhave to be to hide objects (e.g. I could hide the tip of the\nbranches that holds tips of older maintenance tracks, just to\nunclutter, and giving a pack that depends on older parts of history\nis just fine).\n\nLet's let it pass, as the cover letter material won't become part of\nthe permanent history ;-)\n\n> - Second, by using advertised refs as inputs instead of `git\n> rev-list --not --all` we avoid looking up all refs that are\n> irrelevant to the current push. This can be a huge performance\n> improvement in repos that have a huge amount of internal, hidden\n> refs. In one of our repos with 7m refs, of which 6.8m are hidden,\n> this speeds up pushes from ~30s to ~4.5s.\n\nYes.\n\n> One downside is that we need to pass in the object IDs that were part of\n> the reference advertisement via the standard input, which is seemingly\n> slower than reading them from the refdb. I'm discussing this in the\n> second commit.\n\nInteresting.\n"},{"id":"465963","messageId":"xmqqr0yrizqm.fsf@gitster.g","threadId":"58706","inReplyTo":"a32e3d6146dd41af36f525a744d6cc099b42d6fb.1666967670.git.ps@pks.im","subject":"Re: [PATCH 1/2] connected: allow supplying different view of reachable objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-28T18:12:33Z","receivedAt":"2022-10-28T18:12:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> diff --git a/connected.c b/connected.c\n> index 74a20cb32e..2a4c4e0025 100644\n> --- a/connected.c\n> +++ b/connected.c\n> @@ -98,7 +98,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n> \tstrvec_push(&rev_list.args, \"--stdin\");\n> \tif (has_promisor_remote())\n> \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n> -\tif (!opt->is_deepening_fetch) {\n> +\tif (!opt->is_deepening_fetch && !opt->reachable_oids_fn) {\n> \t\tstrvec_push(&rev_list.args, \"--not\");\n> \t\tstrvec_push(&rev_list.args, \"--all\");\n> \t}\n> @@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n> \n> \trev_list_in = xfdopen(rev_list.in, \"w\");\n> \n> +\tif (opt->reachable_oids_fn) {\n> +\t\tconst struct object_id *reachable_oid;\n> +\t\twhile ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)\n> +\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(reachable_oid)) < 0)\n> +\t\t\t\tbreak;\n> +\t}\n\nIt is good that these individual negative references are fed from\nthe standard input, not on the command line, as they can be many.\n\nIn the original code without the reachable_oids_fn, we refrain from\nexcluding when the is_deepening_fetch bit is set, but here we do not\npay attention to the bit at all. Is that sensible, and if so why?\n"},{"id":"466066","messageId":"Y17HO9c9fJweCrPt@nand.local","threadId":"58706","inReplyTo":"xmqqr0yrizqm.fsf@gitster.g","subject":"Re: [PATCH 1/2] connected: allow supplying different view of reachable objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-10-30T18:49:31Z","receivedAt":"2022-10-30T18:49:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Oct 28, 2022 at 11:12:33AM -0700, Junio C Hamano wrote:\n> In the original code without the reachable_oids_fn, we refrain from\n> excluding when the is_deepening_fetch bit is set, but here we do not\n> pay attention to the bit at all. Is that sensible, and if so why?\n\nI was wondering the same thing. Thanks for asking.\n\nThanks,\nTaylor\n"},{"id":"466069","messageId":"Y17L0IjELU5QlOPL@nand.local","threadId":"58706","inReplyTo":"006e89f384be1227b922fb6fdc8755ae84cac587.1666967670.git.ps@pks.im","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-10-30T19:09:04Z","receivedAt":"2022-10-30T19:11:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:\n> This strategy has the major downside that it will not require any object\n> to be sent by the client that is reachable by any of the repositories'\n> references. While that sounds like it would be indeed what we are after\n> with the connectivity check, it is arguably not. The administrator that\n> manages the server-side Git repository may have configured certain refs\n> to be hidden during the reference advertisement via `transfer.hideRefs`\n> or `receivepack.hideRefs`. Whatever the reason, the result is that the\n> client shouldn't expect that any of those hidden references exists on\n> the remote side, and neither should they assume any of the pointed-to\n> objects to exist except if referenced by any visible reference. But\n> because we treat _all_ local refs as uninteresting in the connectivity\n> check, a client is free to send a packfile that references objects that\n> are only reachable via a hidden reference on the server-side, and we\n> will gladly accept it.\n\nYou mention below that this is a correctness issue, but I am not sure\nthat I agree.\n\nThe existing behavior is a little strange, I agree, but your argument\nrelies on an assumption that the history on hidden refs is not part of\nthe reachable set, which is not the case. Any part of the repository\nthat is reachable from _any_ reference, hidden or not, is reachable by\ndefinition.\n\nSo it's perfectly fine to consider objects on hidden refs to be in the\nuninteresting set, because they are reachable. It's odd from the\nclient's perspective, but I do not see a path to repository corruption\nwith thee existing behavior.\n\n> Besides the stated fix to correctness this also provides a huge boost to\n> performance in the repository mentioned above. Pushing a new commit into\n> this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\n> as it is configured in Gitaly leads to an almost 7.5-fold speedup:\n\nNice, here we expect a pretty good speed-up, and indeed...\n\n> Summary\n> 'pks-connectivity-check-hide-refs' ran\n> 7.42 ± 0.16 times faster than 'main'\n\n...that's exactly what we get. Good.\n\n> @@ -1928,6 +1933,12 @@ static void execute_commands(struct command *commands,\n> \topt.err_fd = err_fd;\n> \topt.progress = err_fd && !quiet;\n> \topt.env = tmp_objdir_env(tmp_objdir);\n> +\tif (oidset_size(announced_oids) != 0) {\n\nI'm nitpicking, but this would be preferable as \"if (oidset_size(announced_oids))\"\nwithout the \"!= 0\".\n\n> +\t\toidset_iter_init(announced_oids, &announced_oids_iter);\n> +\t\topt.reachable_oids_fn = iterate_announced_oids;\n> +\t\topt.reachable_oids_data = &announced_oids_iter;\n> +\t}\n\nWhy do we see a slowdown when there there aren't any hidden references?\nOr am I misunderstanding your patch message which instead means \"we see\na slow-down when there are no hidden references [since we still must\nstore and enumerate all advertised references]\"?\n\nIf the latter, could we avoid invoking the new machinery altogether? In\nother words, shouldn't receive-pack only set the reachable_oids_fn() to\nenumerate advertised references only when the set of advertised\nreferences differs from the behavior of `--not --all`?\n\n> \tif (check_connected(iterate_receive_command_list, &data, &opt))\n> \t\tset_connectivity_errors(commands, si);\n>\n> @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> {\n> \tint advertise_refs = 0;\n> \tstruct command *commands;\n> +\tstruct oidset announced_oids = OIDSET_INIT;\n\nThis looks like trading one problem for another. In your above example,\nwe now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the\nend of the world, but it feels like an avoidable problem.\n\nCould we enumerate the references in a callback to for_each_ref() and\nonly emit ones which aren't hidden? Storing these and then recalling\nthem after the fact is worth avoiding.\n\nThanks,\nTaylor\n"},{"id":"466086","messageId":"Y1/JOGihHtMWv240@ncase","threadId":"58706","inReplyTo":"xmqqr0yrizqm.fsf@gitster.g","subject":"Re: [PATCH 1/2] connected: allow supplying different view of reachable objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-10-31T13:10:16Z","receivedAt":"2022-10-31T13:10:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 28, 2022 at 11:12:33AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > diff --git a/connected.c b/connected.c\n> > index 74a20cb32e..2a4c4e0025 100644\n> > --- a/connected.c\n> > +++ b/connected.c\n> > @@ -98,7 +98,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n> > \tstrvec_push(&rev_list.args, \"--stdin\");\n> > \tif (has_promisor_remote())\n> > \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n> > -\tif (!opt->is_deepening_fetch) {\n> > +\tif (!opt->is_deepening_fetch && !opt->reachable_oids_fn) {\n> > \t\tstrvec_push(&rev_list.args, \"--not\");\n> > \t\tstrvec_push(&rev_list.args, \"--all\");\n> > \t}\n> > @@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n> > \n> > \trev_list_in = xfdopen(rev_list.in, \"w\");\n> > \n> > +\tif (opt->reachable_oids_fn) {\n> > +\t\tconst struct object_id *reachable_oid;\n> > +\t\twhile ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)\n> > +\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(reachable_oid)) < 0)\n> > +\t\t\t\tbreak;\n> > +\t}\n> \n> It is good that these individual negative references are fed from\n> the standard input, not on the command line, as they can be many.\n> \n> In the original code without the reachable_oids_fn, we refrain from\n> excluding when the is_deepening_fetch bit is set, but here we do not\n> pay attention to the bit at all. Is that sensible, and if so why?\n\nHm, good point. On a deepening fetch the commits that were the previous\nboundary will likely get replaced by new commits that are at a deeper\npoint in history, so they cannot be used as a well-defined boundary.\nInstead, we do a complete graph-walk that doesn't stop at any previously\nknown commits at all. At least that's how I understand the code, the\nexplanation is likely a bit fuzzy.\n\nI guess we should thus also pay attention to `is_deepening_fetch` here.\nAs this means that `is_deepening_fetch` and `reachable_oids_fn` are\nmutually exclusive I'm inclined to go even further and `die()` if both\nare set at the same time. We only adapt git-receive-pack(1) anyway, so\nwe should never run into this situation for now.\n\nPatrick\n"},{"id":"466091","messageId":"Y1/Z7q+56Mu+rmAX@ncase","threadId":"58706","inReplyTo":"221028.867d0k7yny.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-10-31T14:21:34Z","receivedAt":"2022-10-31T14:21:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 28, 2022 at 05:01:58PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> On Fri, Oct 28 2022, Patrick Steinhardt wrote:\n[sinp]\n> > Unfortunately, this change comes with a performance hit when refs are\n> > not hidden. Executed in the same repository:\n> >\n> > Benchmark 1: main\n> > Time (mean ± σ): 45.780 s ± 0.507 s [User: 46.908 s, System: 4.838 s]\n> > Range (min … max): 45.453 s … 46.364 s 3 runs\n> >\n> > Benchmark 2: pks-connectivity-check-hide-refs\n> > Time (mean ± σ): 49.886 s ± 0.282 s [User: 51.168 s, System: 5.015 s]\n> > Range (min … max): 49.589 s … 50.149 s 3 runs\n> >\n> > Summary\n> > 'main' ran\n> > 1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'\n> >\n> > This is probably caused by the overhead of reachable tips being passed\n> > in via git-rev-list(1)'s standard input, which seems to be slower than\n> > reading the references from disk.\n> >\n> > It is debatable what to do about this. If this were only about improving\n> > performance then it would be trivial to make the new logic depend on\n> > whether or not `transfer.hideRefs` has been configured in the repo. But\n> > as explained this is also about correctness, even though this can be\n> > considered an edge case. Furthermore, this slowdown is really only\n> > noticeable in outliers like the above repository with an unreasonable\n> > amount of refs. The same benchmark in linux-stable.git with about\n> > 4500 references shows no measurable difference:\n> \n> Do we have a test that would start failing if we changed the behavior?\n> Perhaps such a test is peeking too much behind the curtain, but if it's\n> easy come up with one I think it would be most welcome to have it\n> alongside this. to have exposes\n\nWe have tests that verify that we indeed detect missing objects in\nt5504. But what we're lacking is tests that verify that we stop walking\nat the boundary of preexisting objects, and I honestly wouldn't quite\nknow how to do that as there is no functional difference, but really\nonly a performance issue if we overwalked.\n\n> > -static void write_head_info(void)\n> > +static void write_head_info(struct oidset *announced_objects)\n> > {\n> > -\tstatic struct oidset seen = OIDSET_INIT;\n> > -\n> > -\tfor_each_ref(show_ref_cb, &seen);\n> > -\tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n> > -\toidset_clear(&seen);\n> > +\tfor_each_ref(show_ref_cb, announced_objects);\n> > +\tfor_each_alternate_ref(show_one_alternate_ref, announced_objects);\n> > \tif (!sent_capabilities)\n> > \t\tshow_ref(\"capabilities^{}\", null_oid());\n> \n> Nit: The variable rename stands out slightly,\n> i.e. s/&seen/announced_objects/ not s/&seen/seen/, especially as:\n> \n> > static void execute_commands(struct command *commands,\n> > \t\t\t const char *unpacker_error,\n> > \t\t\t struct shallow_info *si,\n> > -\t\t\t const struct string_list *push_options)\n> > +\t\t\t const struct string_list *push_options,\n> > +\t\t\t struct oidset *announced_oids)\n> \n> Here we have the same variable, but now it's *_oids, not *objects.\n\nHm. I think that `announced_oids` is easier to understand compared to\n`seen`, so I'd prefer to keep the rename. But I'll definitely make this\nconsistent so we use `announced_oids` in both places.\n\n[snip]\n> > +static const struct object_id *iterate_announced_oids(void *cb_data)\n> > +{\n> > +\tstruct oidset_iter *iter = cb_data;\n> > +\treturn oidset_iter_next(iter);\n> > +}\n> > +\n> \n> Is just used as (from 1/2):\n> \n> > +\tif (opt->reachable_oids_fn) {\n> > +\t\tconst struct object_id *reachable_oid;\n> > +\t\twhile ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)\n> > +\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(reachable_oid)) < 0)\n> > +\t\t\t\tbreak;\n> > +\t}\n> \n> After doing above:\n> \n> > +\tif (oidset_size(announced_oids) != 0) {\n> > +\t\toidset_iter_init(announced_oids, &announced_oids_iter);\n> > +\t\topt.reachable_oids_fn = iterate_announced_oids;\n> > +\t\topt.reachable_oids_data = &announced_oids_iter;\n> > +\t}\n> \n> But I don't see the reason for the indirection, but maybe I'm missing\n> something obvious.\n> \n> Why not just pass the oidset itself and have connected.c iterate through\n> it, rather than going thorugh this callback / data indirection?\n\nThis is done to stay consistent with the way new tips are passed in via\nthe `oid_iterate_fn`. I'm happy to change callers to just directly pass\na `struct oidset *` though.\n\nPatrick\n"},{"id":"466095","messageId":"Y1/fm1prlAs3U1NE@ncase","threadId":"58706","inReplyTo":"Y17L0IjELU5QlOPL@nand.local","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-10-31T14:45:47Z","receivedAt":"2022-10-31T14:46:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:\n> On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:\n> > This strategy has the major downside that it will not require any object\n> > to be sent by the client that is reachable by any of the repositories'\n> > references. While that sounds like it would be indeed what we are after\n> > with the connectivity check, it is arguably not. The administrator that\n> > manages the server-side Git repository may have configured certain refs\n> > to be hidden during the reference advertisement via `transfer.hideRefs`\n> > or `receivepack.hideRefs`. Whatever the reason, the result is that the\n> > client shouldn't expect that any of those hidden references exists on\n> > the remote side, and neither should they assume any of the pointed-to\n> > objects to exist except if referenced by any visible reference. But\n> > because we treat _all_ local refs as uninteresting in the connectivity\n> > check, a client is free to send a packfile that references objects that\n> > are only reachable via a hidden reference on the server-side, and we\n> > will gladly accept it.\n> \n> You mention below that this is a correctness issue, but I am not sure\n> that I agree.\n> \n> The existing behavior is a little strange, I agree, but your argument\n> relies on an assumption that the history on hidden refs is not part of\n> the reachable set, which is not the case. Any part of the repository\n> that is reachable from _any_ reference, hidden or not, is reachable by\n> definition.\n> \n> So it's perfectly fine to consider objects on hidden refs to be in the\n> uninteresting set, because they are reachable. It's odd from the\n> client's perspective, but I do not see a path to repository corruption\n> with thee existing behavior.\n\nIndeed, I'm not trying to say that this can lead to repository\ncorruption. If at all you can argue that this is more security-related.\nSuppose an object is not reachable from any public reference and that\n`allowAnySHA1InWant=false`. Then you could make these hidden objects\nreachable by sending a packfile with an object that references the\nhidden object. It naturally requires you to somehow know about the\nobject ID, so I don't think this is a critical issue.\n\nBut security-related or not, I think it is safe to say that any packfile\nsent by a client that does not contain objects required for the updated\nreference that the client cannot know to exist on the server-side must\nbe generated by buggy code.\n\n[snip]\n> Why do we see a slowdown when there there aren't any hidden references?\n> Or am I misunderstanding your patch message which instead means \"we see\n> a slow-down when there are no hidden references [since we still must\n> store and enumerate all advertised references]\"?\n\nI have tried to dig down into the code of `revision.c` but ultimately\nreturned empty-handed. I _think_ that this is because of the different\npaths we use when reading revisions from stdin as we have to resolve the\nrevision to an OID first, which is more involved than taking the OIDs as\nreturned by the reference backend. I have tried to short-circuit this\nlogic in case the revision read from stdin is exactly `hash_algo->hexsz`\nlong so that we try to parse it as an OID directly instead of trying to\ndo any of the magic that is required to resolve a revision. But this\nonly speed things up by a small margin.\n\nAnother assumption was that this is overhead caused by using stdin\ninstead of reading data from a file, but flame graphs didn't support\nthis theory, either.\n\n> If the latter, could we avoid invoking the new machinery altogether? In\n> other words, shouldn't receive-pack only set the reachable_oids_fn() to\n> enumerate advertised references only when the set of advertised\n> references differs from the behavior of `--not --all`?\n\nYeah, I was taking a \"wait for feedback and see\" stance on this. We can\neasily make the logic conditional on whether there are any hidden refs\nat all.\n\n> > \tif (check_connected(iterate_receive_command_list, &data, &opt))\n> > \t\tset_connectivity_errors(commands, si);\n> >\n> > @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> > {\n> > \tint advertise_refs = 0;\n> > \tstruct command *commands;\n> > +\tstruct oidset announced_oids = OIDSET_INIT;\n> \n> This looks like trading one problem for another. In your above example,\n> we now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the\n> end of the world, but it feels like an avoidable problem.\n\nWe store these references in an `oidset` before this patch set already,\nbut yes, the lifetime is longer now. But note that this set stores the\nannounced objects, not the hidden ones. So we don't store 6.8m OIDs, but\nonly the 250k announced ones.\n\n> Could we enumerate the references in a callback to for_each_ref() and\n> only emit ones which aren't hidden? Storing these and then recalling\n> them after the fact is worth avoiding.\n\nSorry, I don't quite get what you're proposing. `for_each_ref()` already\ndoes exactly that: it stores every reference that is not hidden in the\nabove `oidset`. This is the exact set of advertised references, which in\nmy example repository would be about 250k. This information is used by\ngit-receive-pack(1) to avoid announcing the same object twice, and now\nit's also used to inform the connectivity check to use these objects as\nthe set of already-reachable objects.\n\nPatrick\n"},{"id":"466098","messageId":"221031.86h6zk6m42.gmgdl@evledraar.gmail.com","threadId":"58706","inReplyTo":"Y1/Z7q+56Mu+rmAX@ncase","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-31T15:36:09Z","receivedAt":"2022-10-31T15:37:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 31 2022, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> On Fri, Oct 28, 2022 at 05:01:58PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> On Fri, Oct 28 2022, Patrick Steinhardt wrote:\n> [sinp]\n>> > Unfortunately, this change comes with a performance hit when refs are\n>> > not hidden. Executed in the same repository:\n>> >\n>> > Benchmark 1: main\n>> > Time (mean ± σ): 45.780 s ± 0.507 s [User: 46.908 s, System: 4.838 s]\n>> > Range (min … max): 45.453 s … 46.364 s 3 runs\n>> >\n>> > Benchmark 2: pks-connectivity-check-hide-refs\n>> > Time (mean ± σ): 49.886 s ± 0.282 s [User: 51.168 s, System: 5.015 s]\n>> > Range (min … max): 49.589 s … 50.149 s 3 runs\n>> >\n>> > Summary\n>> > 'main' ran\n>> > 1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'\n>> >\n>> > This is probably caused by the overhead of reachable tips being passed\n>> > in via git-rev-list(1)'s standard input, which seems to be slower than\n>> > reading the references from disk.\n>> >\n>> > It is debatable what to do about this. If this were only about improving\n>> > performance then it would be trivial to make the new logic depend on\n>> > whether or not `transfer.hideRefs` has been configured in the repo. But\n>> > as explained this is also about correctness, even though this can be\n>> > considered an edge case. Furthermore, this slowdown is really only\n>> > noticeable in outliers like the above repository with an unreasonable\n>> > amount of refs. The same benchmark in linux-stable.git with about\n>> > 4500 references shows no measurable difference:\n>> \n>> Do we have a test that would start failing if we changed the behavior?\n>> Perhaps such a test is peeking too much behind the curtain, but if it's\n>> easy come up with one I think it would be most welcome to have it\n>> alongside this. to have exposes\n>\n> We have tests that verify that we indeed detect missing objects in\n> t5504. But what we're lacking is tests that verify that we stop walking\n> at the boundary of preexisting objects, and I honestly wouldn't quite\n> know how to do that as there is no functional difference, but really\n> only a performance issue if we overwalked.\n>\n>> > -static void write_head_info(void)\n>> > +static void write_head_info(struct oidset *announced_objects)\n>> > {\n>> > -\tstatic struct oidset seen = OIDSET_INIT;\n>> > -\n>> > -\tfor_each_ref(show_ref_cb, &seen);\n>> > -\tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n>> > -\toidset_clear(&seen);\n>> > +\tfor_each_ref(show_ref_cb, announced_objects);\n>> > +\tfor_each_alternate_ref(show_one_alternate_ref, announced_objects);\n>> > \tif (!sent_capabilities)\n>> > \t\tshow_ref(\"capabilities^{}\", null_oid());\n>> \n>> Nit: The variable rename stands out slightly,\n>> i.e. s/&seen/announced_objects/ not s/&seen/seen/, especially as:\n>> \n>> > static void execute_commands(struct command *commands,\n>> > \t\t\t const char *unpacker_error,\n>> > \t\t\t struct shallow_info *si,\n>> > -\t\t\t const struct string_list *push_options)\n>> > +\t\t\t const struct string_list *push_options,\n>> > +\t\t\t struct oidset *announced_oids)\n>> \n>> Here we have the same variable, but now it's *_oids, not *objects.\n>\n> Hm. I think that `announced_oids` is easier to understand compared to\n> `seen`, so I'd prefer to keep the rename. But I'll definitely make this\n> consistent so we use `announced_oids` in both places.\n\nSounds good, we'll need to look at the diff lines in either case (as\nwe're converting it to a pointer), so changing the name while at it is\nfine...\n\n> [snip]\n>> > +static const struct object_id *iterate_announced_oids(void *cb_data)\n>> > +{\n>> > +\tstruct oidset_iter *iter = cb_data;\n>> > +\treturn oidset_iter_next(iter);\n>> > +}\n>> > +\n>> \n>> Is just used as (from 1/2):\n>> \n>> > +\tif (opt->reachable_oids_fn) {\n>> > +\t\tconst struct object_id *reachable_oid;\n>> > +\t\twhile ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)\n>> > +\t\t\tif (fprintf(rev_list_in, \"^%s\\n\", oid_to_hex(reachable_oid)) < 0)\n>> > +\t\t\t\tbreak;\n>> > +\t}\n>> \n>> After doing above:\n>> \n>> > +\tif (oidset_size(announced_oids) != 0) {\n>> > +\t\toidset_iter_init(announced_oids, &announced_oids_iter);\n>> > +\t\topt.reachable_oids_fn = iterate_announced_oids;\n>> > +\t\topt.reachable_oids_data = &announced_oids_iter;\n>> > +\t}\n>> \n>> But I don't see the reason for the indirection, but maybe I'm missing\n>> something obvious.\n>> \n>> Why not just pass the oidset itself and have connected.c iterate through\n>> it, rather than going thorugh this callback / data indirection?\n>\n> This is done to stay consistent with the way new tips are passed in via\n> the `oid_iterate_fn`. I'm happy to change callers to just directly pass\n> a `struct oidset *` though.\n\n*nod*, makes sense, no need to change it. Just wondering...\n"},{"id":"466160","messageId":"Y2BzY0kqyt3doMXT@nand.local","threadId":"58706","inReplyTo":"Y1/JOGihHtMWv240@ncase","subject":"Re: [PATCH 1/2] connected: allow supplying different view of reachable objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-01T01:16:19Z","receivedAt":"2022-11-01T01:16:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 31, 2022 at 02:10:16PM +0100, Patrick Steinhardt wrote:\n> I guess we should thus also pay attention to `is_deepening_fetch` here.\n> As this means that `is_deepening_fetch` and `reachable_oids_fn` are\n> mutually exclusive I'm inclined to go even further and `die()` if both\n> are set at the same time. We only adapt git-receive-pack(1) anyway, so\n> we should never run into this situation for now.\n\nYes, I agree.\n\nThanks,\nTaylor\n"},{"id":"466161","messageId":"Y2B2KeYX5X9vp+cf@nand.local","threadId":"58706","inReplyTo":"Y1/fm1prlAs3U1NE@ncase","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-01T01:28:09Z","receivedAt":"2022-11-01T01:28:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:\n> On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:\n> > On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:\n> > > This strategy has the major downside that it will not require any object\n> > > to be sent by the client that is reachable by any of the repositories'\n> > > references. While that sounds like it would be indeed what we are after\n> > > with the connectivity check, it is arguably not. The administrator that\n> > > manages the server-side Git repository may have configured certain refs\n> > > to be hidden during the reference advertisement via `transfer.hideRefs`\n> > > or `receivepack.hideRefs`. Whatever the reason, the result is that the\n> > > client shouldn't expect that any of those hidden references exists on\n> > > the remote side, and neither should they assume any of the pointed-to\n> > > objects to exist except if referenced by any visible reference. But\n> > > because we treat _all_ local refs as uninteresting in the connectivity\n> > > check, a client is free to send a packfile that references objects that\n> > > are only reachable via a hidden reference on the server-side, and we\n> > > will gladly accept it.\n> >\n> > You mention below that this is a correctness issue, but I am not sure\n> > that I agree.\n> >\n> > The existing behavior is a little strange, I agree, but your argument\n> > relies on an assumption that the history on hidden refs is not part of\n> > the reachable set, which is not the case. Any part of the repository\n> > that is reachable from _any_ reference, hidden or not, is reachable by\n> > definition.\n> >\n> > So it's perfectly fine to consider objects on hidden refs to be in the\n> > uninteresting set, because they are reachable. It's odd from the\n> > client's perspective, but I do not see a path to repository corruption\n> > with thee existing behavior.\n>\n> Indeed, I'm not trying to say that this can lead to repository\n> corruption.\n\nI definitely agree with that. I have thought about this on-and-off since\nyou sent the topic, and I am pretty certain that there is no path to\nrepository corruption with the existing behavior. It would be worth\nupdating the commit message to make this clearer.\n\n> But security-related or not, I think it is safe to say that any packfile\n> sent by a client that does not contain objects required for the updated\n> reference that the client cannot know to exist on the server-side must\n> be generated by buggy code.\n\nMaybe, though I think it's fine to let clients send us smaller packfiles\nif they have some a-priori knowledge that the server has objects that it\nisn't advertising. And that can all happen without buggy code. So it's\nweird, but there isn't anything wrong with letting it happen.\n\n> [snip]\n> > Why do we see a slowdown when there there aren't any hidden references?\n> > Or am I misunderstanding your patch message which instead means \"we see\n> > a slow-down when there are no hidden references [since we still must\n> > store and enumerate all advertised references]\"?\n>\n> I have tried to dig down into the code of `revision.c` but ultimately\n> returned empty-handed. I _think_ that this is because of the different\n> paths we use when reading revisions from stdin as we have to resolve the\n> revision to an OID first, which is more involved than taking the OIDs as\n> returned by the reference backend. I have tried to short-circuit this\n> logic in case the revision read from stdin is exactly `hash_algo->hexsz`\n> long so that we try to parse it as an OID directly instead of trying to\n> do any of the magic that is required to resolve a revision. But this\n> only speed things up by a small margin.\n>\n> Another assumption was that this is overhead caused by using stdin\n> instead of reading data from a file, but flame graphs didn't support\n> this theory, either.\n>\n> > If the latter, could we avoid invoking the new machinery altogether? In\n> > other words, shouldn't receive-pack only set the reachable_oids_fn() to\n> > enumerate advertised references only when the set of advertised\n> > references differs from the behavior of `--not --all`?\n>\n> Yeah, I was taking a \"wait for feedback and see\" stance on this. We can\n> easily make the logic conditional on whether there are any hidden refs\n> at all.\n\nYeah, I think that this would be preferable. I'm surprised that your\ndata doesn't support the idea that the slowdown is caused by reading\nfrom stdin instead of parsing `--not --all`. I'd be curious to see what\nyou have tried so far.\n\nI'm almost certain that forcing rev-list to chew through a bunch of data\non stdin that is basically equivalent to saying `--not --all` is going\nto be the source of the slowdown.\n\n> > > \tif (check_connected(iterate_receive_command_list, &data, &opt))\n> > > \t\tset_connectivity_errors(commands, si);\n> > >\n> > > @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> > > {\n> > > \tint advertise_refs = 0;\n> > > \tstruct command *commands;\n> > > +\tstruct oidset announced_oids = OIDSET_INIT;\n> >\n> > This looks like trading one problem for another. In your above example,\n> > we now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the\n> > end of the world, but it feels like an avoidable problem.\n>\n> We store these references in an `oidset` before this patch set already,\n> but yes, the lifetime is longer now. But note that this set stores the\n> announced objects, not the hidden ones. So we don't store 6.8m OIDs, but\n> only the 250k announced ones.\n\nHmm, OK. I wonder, could rev-list be given a `--refs=`\nargument which is equal to the advertised references? Or something that\nimplies \"give me all of the references which aren't hidden?\"\n\nIf we call that maybe `--visible-refs=receive` (to imply the references\n*not* in receive.hideRefs), then our connectivity check would become:\n\n git rev-list --stdin --not --visible-refs=receive\n\nor something like that. Not having to extend the lifetime of these OIDs\nin an oidset would be worthwhile. Besides, having a new rev-list option\nis cool, too ;-).\n\n> > Could we enumerate the references in a callback to for_each_ref() and\n> > only emit ones which aren't hidden? Storing these and then recalling\n> > them after the fact is worth avoiding.\n>\n> Sorry, I don't quite get what you're proposing. `for_each_ref()` already\n> does exactly that: it stores every reference that is not hidden in the\n> above `oidset`. This is the exact set of advertised references, which in\n> my example repository would be about 250k. This information is used by\n> git-receive-pack(1) to avoid announcing the same object twice, and now\n> it's also used to inform the connectivity check to use these objects as\n> the set of already-reachable objects.\n\nYes, ignore me ;-).\n\nThanks,\nTaylor\n"},{"id":"466162","messageId":"Y2B2sttQ2fDF/ocf@nand.local","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-01T01:30:26Z","receivedAt":"2022-11-01T01:30:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Oct 28, 2022 at 04:42:19PM +0200, Patrick Steinhardt wrote:\n> this patch series improves the connectivity check done by stateful\n> git-receive-pack(1) to only consider references as reachable that have\n> been advertised to the client. This has two advantages:\n\nA third advantage which I didn't see mentioned here is that we have a\nbetter chance of being helped by reachability bitmaps here. Because we\naren't likely to cover hidden references with bitmaps, we would be\npaying a significant cost to build up a complete picture of what's\nreachable from --all.\n\nThat will inevitably involve a lot of traversing from unbitmapped\nreference tips (i.e. the hidden ones) down to either a random bitmap\nsprinkled throughout history, or the root of that line of history.\n\nBut if we limited the negated side of our connectivity check to just the\nnon-hidden references, then we will likely end up with more full bitmap\ncoverage of those tips, without having to do any additional traversal.\n\nSo that is definitely something worth experimenting with.\n\nThanks,\nTaylor\n"},{"id":"466180","messageId":"Y2DI0OL1bXhPS/JD@ncase","threadId":"58706","inReplyTo":"Y2B2KeYX5X9vp+cf@nand.local","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-01T07:20:48Z","receivedAt":"2022-11-01T07:20:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 31, 2022 at 09:28:09PM -0400, Taylor Blau wrote:\n> On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:\n> > On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:\n> > > On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:\n> > > > This strategy has the major downside that it will not require any object\n> > > > to be sent by the client that is reachable by any of the repositories'\n> > > > references. While that sounds like it would be indeed what we are after\n> > > > with the connectivity check, it is arguably not. The administrator that\n> > > > manages the server-side Git repository may have configured certain refs\n> > > > to be hidden during the reference advertisement via `transfer.hideRefs`\n> > > > or `receivepack.hideRefs`. Whatever the reason, the result is that the\n> > > > client shouldn't expect that any of those hidden references exists on\n> > > > the remote side, and neither should they assume any of the pointed-to\n> > > > objects to exist except if referenced by any visible reference. But\n> > > > because we treat _all_ local refs as uninteresting in the connectivity\n> > > > check, a client is free to send a packfile that references objects that\n> > > > are only reachable via a hidden reference on the server-side, and we\n> > > > will gladly accept it.\n> > >\n> > > You mention below that this is a correctness issue, but I am not sure\n> > > that I agree.\n> > >\n> > > The existing behavior is a little strange, I agree, but your argument\n> > > relies on an assumption that the history on hidden refs is not part of\n> > > the reachable set, which is not the case. Any part of the repository\n> > > that is reachable from _any_ reference, hidden or not, is reachable by\n> > > definition.\n> > >\n> > > So it's perfectly fine to consider objects on hidden refs to be in the\n> > > uninteresting set, because they are reachable. It's odd from the\n> > > client's perspective, but I do not see a path to repository corruption\n> > > with thee existing behavior.\n> >\n> > Indeed, I'm not trying to say that this can lead to repository\n> > corruption.\n> \n> I definitely agree with that. I have thought about this on-and-off since\n> you sent the topic, and I am pretty certain that there is no path to\n> repository corruption with the existing behavior. It would be worth\n> updating the commit message to make this clearer.\n\nFair enough, I can try to do that.\n\n> > But security-related or not, I think it is safe to say that any packfile\n> > sent by a client that does not contain objects required for the updated\n> > reference that the client cannot know to exist on the server-side must\n> > be generated by buggy code.\n> \n> Maybe, though I think it's fine to let clients send us smaller packfiles\n> if they have some a-priori knowledge that the server has objects that it\n> isn't advertising. And that can all happen without buggy code. So it's\n> weird, but there isn't anything wrong with letting it happen.\n\nWell, I don't see how to achieve both at the same time though: we can\neither limit the set of uninteresting tips to what we have announced to\nthe client, or we allow clients to omit objects that have not been\nannounced. These are mutually exclusive.\n\nSo if we take the stance that it was fine to send packfiles that omit\nhidden objects and that this is something we want to continue to support\nthen this patch series probably becomes moot. Doing the proposed\noptimization means that we also tighten the rules here.\n\n> > [snip]\n> > > Why do we see a slowdown when there there aren't any hidden references?\n> > > Or am I misunderstanding your patch message which instead means \"we see\n> > > a slow-down when there are no hidden references [since we still must\n> > > store and enumerate all advertised references]\"?\n> >\n> > I have tried to dig down into the code of `revision.c` but ultimately\n> > returned empty-handed. I _think_ that this is because of the different\n> > paths we use when reading revisions from stdin as we have to resolve the\n> > revision to an OID first, which is more involved than taking the OIDs as\n> > returned by the reference backend. I have tried to short-circuit this\n> > logic in case the revision read from stdin is exactly `hash_algo->hexsz`\n> > long so that we try to parse it as an OID directly instead of trying to\n> > do any of the magic that is required to resolve a revision. But this\n> > only speed things up by a small margin.\n> >\n> > Another assumption was that this is overhead caused by using stdin\n> > instead of reading data from a file, but flame graphs didn't support\n> > this theory, either.\n> >\n> > > If the latter, could we avoid invoking the new machinery altogether? In\n> > > other words, shouldn't receive-pack only set the reachable_oids_fn() to\n> > > enumerate advertised references only when the set of advertised\n> > > references differs from the behavior of `--not --all`?\n> >\n> > Yeah, I was taking a \"wait for feedback and see\" stance on this. We can\n> > easily make the logic conditional on whether there are any hidden refs\n> > at all.\n> \n> Yeah, I think that this would be preferable. I'm surprised that your\n> data doesn't support the idea that the slowdown is caused by reading\n> from stdin instead of parsing `--not --all`. I'd be curious to see what\n> you have tried so far.\n> \n> I'm almost certain that forcing rev-list to chew through a bunch of data\n> on stdin that is basically equivalent to saying `--not --all` is going\n> to be the source of the slowdown.\n\nI was basically benchmarking these two commands in the repository with\n7m refs:\n\n # old-style connectivity check\n $ git commit-tree HEAD^{tree} -m commit -p HEAD >newtip\n $ git rev-list --objects --quiet --stdin --not --all oldandnewtips\n $ git rev-list --objects --quiet --stdin > > > \tif (check_connected(iterate_receive_command_list, &data, &opt))\n> > > > \t\tset_connectivity_errors(commands, si);\n> > > >\n> > > > @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> > > > {\n> > > > \tint advertise_refs = 0;\n> > > > \tstruct command *commands;\n> > > > +\tstruct oidset announced_oids = OIDSET_INIT;\n> > >\n> > > This looks like trading one problem for another. In your above example,\n> > > we now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the\n> > > end of the world, but it feels like an avoidable problem.\n> >\n> > We store these references in an `oidset` before this patch set already,\n> > but yes, the lifetime is longer now. But note that this set stores the\n> > announced objects, not the hidden ones. So we don't store 6.8m OIDs, but\n> > only the 250k announced ones.\n> \n> Hmm, OK. I wonder, could rev-list be given a `--refs=`\n> argument which is equal to the advertised references? Or something that\n> implies \"give me all of the references which aren't hidden?\"\n> \n> If we call that maybe `--visible-refs=receive` (to imply the references\n> *not* in receive.hideRefs), then our connectivity check would become:\n> \n> git rev-list --stdin --not --visible-refs=receive\n> \n> or something like that. Not having to extend the lifetime of these OIDs\n> in an oidset would be worthwhile. Besides, having a new rev-list option\n> is cool, too ;-).\n\nThat is indeed an interesting idea. It would likely both fix the issue\nof extended memory lifetime of the announced OIDs as well as fixing the\nslowdown caused by using `--stdin` for them.\n\nPatrick\n"},{"id":"466186","messageId":"Y2DYqZRlO7emYu8/@coredump.intra.peff.net","threadId":"58706","inReplyTo":"Y1/fm1prlAs3U1NE@ncase","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-01T08:28:25Z","receivedAt":"2022-11-01T08:29:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:\n\n> Indeed, I'm not trying to say that this can lead to repository\n> corruption. If at all you can argue that this is more security-related.\n> Suppose an object is not reachable from any public reference and that\n> `allowAnySHA1InWant=false`. Then you could make these hidden objects\n> reachable by sending a packfile with an object that references the\n> hidden object. It naturally requires you to somehow know about the\n> object ID, so I don't think this is a critical issue.\n\nI'd have to double check, but isn't this all moot with the v2 protocol\nanyway? I didn't think it even respected allowAnySHA1InWant.\n\nEven that aside, there are other tricks you can do. E.g., pushing up an\nobject which claims to be a delta of X (though you have the trick then\nof figuring out how to reference it), or pushing up a test object then\nfetching it back while claiming to have X, treating the server as an\noracle which may give you a delta against X.\n\nIn short, I don't think it's worth considering unreachable objects in a\nrepository to be considered secret.\n\n> > Why do we see a slowdown when there there aren't any hidden references?\n> > Or am I misunderstanding your patch message which instead means \"we see\n> > a slow-down when there are no hidden references [since we still must\n> > store and enumerate all advertised references]\"?\n> \n> I have tried to dig down into the code of `revision.c` but ultimately\n> returned empty-handed. I _think_ that this is because of the different\n> paths we use when reading revisions from stdin as we have to resolve the\n> revision to an OID first, which is more involved than taking the OIDs as\n> returned by the reference backend. I have tried to short-circuit this\n> logic in case the revision read from stdin is exactly `hash_algo->hexsz`\n> long so that we try to parse it as an OID directly instead of trying to\n> do any of the magic that is required to resolve a revision. But this\n> only speed things up by a small margin.\n> \n> Another assumption was that this is overhead caused by using stdin\n> instead of reading data from a file, but flame graphs didn't support\n> this theory, either.\n\nCertainly read_revisions_from_stdin() will allocate a bunch of extra\ndata per item it reads (via add_rev_cmdline() and add_pending), which is\ngoing to be way more than parsing or stdin overhead.\n\nMuch worse is that it will call get_reference(), which is very keen to\nactually open and parse the object (as you know, since you added\ncommit-graph handling there). That might have gotten a bit better in\nv2.38.0 if you have any references to blobs (as we'd now skip the extra\nhash check).\n\nOf course the original \"rev-list --not --all\" would suffer from the same\nthing, so some of that may not explain any difference between the two.\n\n-Peff\n"},{"id":"466191","messageId":"Y2DgJi6foPyBhycU@coredump.intra.peff.net","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-01T09:00:22Z","receivedAt":"2022-11-01T09:00:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 28, 2022 at 04:42:19PM +0200, Patrick Steinhardt wrote:\n\n> - A client shouldn't assume objects to exist that have not been part\n> of the reference advertisement. But if it excluded an object from\n> the packfile that is reachable via any ref that is excluded from\n> the reference advertisement due to `transfer.hideRefs` we'd have\n> accepted the push anyway. I'd argue that this is a bug in the\n> current implementation.\n\nLike others, I don't think this is a bug exactly. We'd never introduce a\ncorruption. We're just more lenient with clients than we need to be.\n\nBut I don't think your scheme changes that. In a sense, the tips used by\n\"rev-list --not --all\" are really an optimization. We will walk the\nhistory from the to-be-updated ref tips all the way down to the roots if\nwe have to. So imagine that I have object X which is not referenced at\nall (neither hidden nor visible ref). We obviously do not advertise it\nto the client, but let's further imagine that a client sends us a pack\nwith X..Y, and a request to update some ref to Y.\n\nBoth before and after your code, if rev-list is able to walk down from Y\nuntil either we hit all roots or all UNINTERESTING commits, it will be\nsatisfied. So as long as the receiving repo actually has all of the\nhistory leading up to X, it will allow the push, regardless of your\npatch.\n\nIf we wanted to stop being lenient, we'd have to actually check that\nevery object we traverse is either reachable, or came from the\njust-pushed pack.\n\n\nThere's also a subtle timing issue here. Our connectivity check happens\nafter we've finished receiving the pack. So not only are we including\nhidden refs, but we are using the ref state at the end of the push\n(after receiving and processing the incoming pack), rather than the\nbeginning.\n\nFrom the same \"leniency\" lens this seems like the wrong thing. But as\nabove, it doesn't matter in practice, because these tips are really an\noptimization to tell rev-list that it can stop traversing.\n\nIf you think of the connectivity check less as \"did the client try to\ncheat\" and more as \"is it OK to update these refs without introducing a\ncorruption\", then it makes sense that you'd want to do read the inputs\nto the check as close to the ref update as possible, because it shrinks\nthe race window which could introduce corruption.\n\nImagine a situation like this:\n\n 0. We advertise to client that we have commit X.\n\n 1. Client starts pushing up a pack with X..Y and asks to update some\n branch to Y.\n\n 2. Meanwhile, the branch with X is deleted, and X is pruned.\n\n 3. Server finishes receiving the pack. All looks good, and then we\n start a connectivity check.\n\nIn the current code, that check starts with the current ref state (with\nX deleted) as a given, and makes sure that we have the objects we need\nto update the refs. After your patches, it would take X as a given, and\nstop traversing when we see it.\n\nThat same race exists before your patch, but it's between the time of\n\"rev-list --not --all\" running and the ref update. After your patch,\nit's between the advertisement and the ref update, which can be a long\ntime (hours or even days, if the client is very slow).\n\nIn practice I'm not sure how big a deal this is. If we feed the\nnow-pruned X to rev-list, it may notice that X went away, though we've\nbeen reducing the number of checks there in the name of efficiency\n(e.g., if it's still in the commit graph, we'd say \"OK, good enough\"\nthese days, even if we don't have it on disk anymore).\n\nBut it feels like a wrong direction to make that race longer if there's\nno need to.\n\nSo all that said...\n\n> - Second, by using advertised refs as inputs instead of `git\n> rev-list --not --all` we avoid looking up all refs that are\n> irrelevant to the current push. This can be a huge performance\n> improvement in repos that have a huge amount of internal, hidden\n> refs. In one of our repos with 7m refs, of which 6.8m are hidden,\n> this speeds up pushes from ~30s to ~4.5s.\n\nI like the general direction here of avoiding the hidden refs. The\nclient _shouldn't_ have been using them, so we can optimistically assume\nthey're useless (and in the case of races or other weirdness, rev-list\njust ends up traversing a bit further).\n\nBut we can split the two ideas in your series:\n\n 1. Feed the advertised tips from receive-pack to rev-list.\n\n 2. Avoid traversing from the hidden tips.\n\nDoing (1) gets you (2) for free. But if we don't want to do (1), and I\ndon't think we do, we can get (2) by just teaching rev-list to narrow\nthe check.\n\nI see some discussion in the other part of the thread, and we may need a\nnew rev-list option to do this, as mentioned there. However, you _might_\nbe able to do it the existing --exclude mechanism. I.e., something like:\n\n rev-list --stdin --not --exclude 'refs/hidden/*' --all\n\nThe gotchas are:\n\n - I'm not 100% sure that --exclude globbing and transfer.hideRefs\n syntax are compatible. You'd want to check.\n\n - these would have to come on the command line (at least with the\n current code). Probably nobody has enough hiderefs patterns for that\n to be a problem (and remember we are passing the glob pattern here,\n not the 6.8M refs themselves). But it could bite somebody in a\n pathological case.\n\n-Peff\n"},{"id":"466196","messageId":"Y2EH3+l62S55EJOZ@ncase","threadId":"58706","inReplyTo":"Y2DgJi6foPyBhycU@coredump.intra.peff.net","subject":"Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-01T11:49:51Z","receivedAt":"2022-11-01T11:52:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 01, 2022 at 05:00:22AM -0400, Jeff King wrote:\n> On Fri, Oct 28, 2022 at 04:42:19PM +0200, Patrick Steinhardt wrote:\n> \n> > - A client shouldn't assume objects to exist that have not been part\n> > of the reference advertisement. But if it excluded an object from\n> > the packfile that is reachable via any ref that is excluded from\n> > the reference advertisement due to `transfer.hideRefs` we'd have\n> > accepted the push anyway. I'd argue that this is a bug in the\n> > current implementation.\n> \n> Like others, I don't think this is a bug exactly. We'd never introduce a\n> corruption. We're just more lenient with clients than we need to be.\n> \n> But I don't think your scheme changes that. In a sense, the tips used by\n> \"rev-list --not --all\" are really an optimization. We will walk the\n> history from the to-be-updated ref tips all the way down to the roots if\n> we have to. So imagine that I have object X which is not referenced at\n> all (neither hidden nor visible ref). We obviously do not advertise it\n> to the client, but let's further imagine that a client sends us a pack\n> with X..Y, and a request to update some ref to Y.\n> \n> Both before and after your code, if rev-list is able to walk down from Y\n> until either we hit all roots or all UNINTERESTING commits, it will be\n> satisfied. So as long as the receiving repo actually has all of the\n> history leading up to X, it will allow the push, regardless of your\n> patch.\n\nOh, right! Now I see where my thinko was, which means both you and\nTaylor are correct. I somehow assumed that we'd fail the connectivity\ncheck in that case, but all it means is that we now potentially walk\nmore objects than we'd have done if we used `--not --all`.\n\n> If we wanted to stop being lenient, we'd have to actually check that\n> every object we traverse is either reachable, or came from the\n> just-pushed pack.\n\nYes, indeed.\n\n> There's also a subtle timing issue here. Our connectivity check happens\n> after we've finished receiving the pack. So not only are we including\n> hidden refs, but we are using the ref state at the end of the push\n> (after receiving and processing the incoming pack), rather than the\n> beginning.\n> \n> From the same \"leniency\" lens this seems like the wrong thing. But as\n> above, it doesn't matter in practice, because these tips are really an\n> optimization to tell rev-list that it can stop traversing.\n> \n> If you think of the connectivity check less as \"did the client try to\n> cheat\" and more as \"is it OK to update these refs without introducing a\n> corruption\", then it makes sense that you'd want to do read the inputs\n> to the check as close to the ref update as possible, because it shrinks\n> the race window which could introduce corruption.\n\nAgreed.\n\n> Imagine a situation like this:\n> \n> 0. We advertise to client that we have commit X.\n> \n> 1. Client starts pushing up a pack with X..Y and asks to update some\n> branch to Y.\n> \n> 2. Meanwhile, the branch with X is deleted, and X is pruned.\n> \n> 3. Server finishes receiving the pack. All looks good, and then we\n> start a connectivity check.\n> \n> In the current code, that check starts with the current ref state (with\n> X deleted) as a given, and makes sure that we have the objects we need\n> to update the refs. After your patches, it would take X as a given, and\n> stop traversing when we see it.\n> \n> That same race exists before your patch, but it's between the time of\n> \"rev-list --not --all\" running and the ref update. After your patch,\n> it's between the advertisement and the ref update, which can be a long\n> time (hours or even days, if the client is very slow).\n> \n> In practice I'm not sure how big a deal this is. If we feed the\n> now-pruned X to rev-list, it may notice that X went away, though we've\n> been reducing the number of checks there in the name of efficiency\n> (e.g., if it's still in the commit graph, we'd say \"OK, good enough\"\n> these days, even if we don't have it on disk anymore).\n> \n> But it feels like a wrong direction to make that race longer if there's\n> no need to.\n\nGood point.\n\n> So all that said...\n> \n> > - Second, by using advertised refs as inputs instead of `git\n> > rev-list --not --all` we avoid looking up all refs that are\n> > irrelevant to the current push. This can be a huge performance\n> > improvement in repos that have a huge amount of internal, hidden\n> > refs. In one of our repos with 7m refs, of which 6.8m are hidden,\n> > this speeds up pushes from ~30s to ~4.5s.\n> \n> I like the general direction here of avoiding the hidden refs. The\n> client _shouldn't_ have been using them, so we can optimistically assume\n> they're useless (and in the case of races or other weirdness, rev-list\n> just ends up traversing a bit further).\n> \n> But we can split the two ideas in your series:\n> \n> 1. Feed the advertised tips from receive-pack to rev-list.\n> \n> 2. Avoid traversing from the hidden tips.\n> \n> Doing (1) gets you (2) for free. But if we don't want to do (1), and I\n> don't think we do, we can get (2) by just teaching rev-list to narrow\n> the check.\n> \n> I see some discussion in the other part of the thread, and we may need a\n> new rev-list option to do this, as mentioned there. However, you _might_\n> be able to do it the existing --exclude mechanism. I.e., something like:\n> \n> rev-list --stdin --not --exclude 'refs/hidden/*' --all\n\nYeah, Taylor proposed to add a new `--visible-refs=receive` option that\nlets git-rev-list(1) automatically add all references that are visible\nwhen paying attention to `receive.hideRefs`. I actually like this idea\nand will likely have a look at how easy or hard it is to implement.\n\n> The gotchas are:\n> \n> - I'm not 100% sure that --exclude globbing and transfer.hideRefs\n> syntax are compatible. You'd want to check.\n> \n> - these would have to come on the command line (at least with the\n> current code). Probably nobody has enough hiderefs patterns for that\n> to be a problem (and remember we are passing the glob pattern here,\n> not the 6.8M refs themselves). But it could bite somebody in a\n> pathological case.\n> \n> -Peff\n\nWell, we can avoid these gotchas if we used `--visible-refs`.\n\nPatrick\n"},{"id":"466197","messageId":"Y2EIxp+YM0Bee79v@ncase","threadId":"58706","inReplyTo":"Y2DI0OL1bXhPS/JD@ncase","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-01T11:53:42Z","receivedAt":"2022-11-01T11:54:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 01, 2022 at 08:20:48AM +0100, Patrick Steinhardt wrote:\n> On Mon, Oct 31, 2022 at 09:28:09PM -0400, Taylor Blau wrote:\n> > On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:\n> > > On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:\n> > > > On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:\n> > > > > This strategy has the major downside that it will not require any object\n> > > > > to be sent by the client that is reachable by any of the repositories'\n> > > > > references. While that sounds like it would be indeed what we are after\n> > > > > with the connectivity check, it is arguably not. The administrator that\n> > > > > manages the server-side Git repository may have configured certain refs\n> > > > > to be hidden during the reference advertisement via `transfer.hideRefs`\n> > > > > or `receivepack.hideRefs`. Whatever the reason, the result is that the\n> > > > > client shouldn't expect that any of those hidden references exists on\n> > > > > the remote side, and neither should they assume any of the pointed-to\n> > > > > objects to exist except if referenced by any visible reference. But\n> > > > > because we treat _all_ local refs as uninteresting in the connectivity\n> > > > > check, a client is free to send a packfile that references objects that\n> > > > > are only reachable via a hidden reference on the server-side, and we\n> > > > > will gladly accept it.\n> > > >\n> > > > You mention below that this is a correctness issue, but I am not sure\n> > > > that I agree.\n> > > >\n> > > > The existing behavior is a little strange, I agree, but your argument\n> > > > relies on an assumption that the history on hidden refs is not part of\n> > > > the reachable set, which is not the case. Any part of the repository\n> > > > that is reachable from _any_ reference, hidden or not, is reachable by\n> > > > definition.\n> > > >\n> > > > So it's perfectly fine to consider objects on hidden refs to be in the\n> > > > uninteresting set, because they are reachable. It's odd from the\n> > > > client's perspective, but I do not see a path to repository corruption\n> > > > with thee existing behavior.\n> > >\n> > > Indeed, I'm not trying to say that this can lead to repository\n> > > corruption.\n> > \n> > I definitely agree with that. I have thought about this on-and-off since\n> > you sent the topic, and I am pretty certain that there is no path to\n> > repository corruption with the existing behavior. It would be worth\n> > updating the commit message to make this clearer.\n> \n> Fair enough, I can try to do that.\n> \n> > > But security-related or not, I think it is safe to say that any packfile\n> > > sent by a client that does not contain objects required for the updated\n> > > reference that the client cannot know to exist on the server-side must\n> > > be generated by buggy code.\n> > \n> > Maybe, though I think it's fine to let clients send us smaller packfiles\n> > if they have some a-priori knowledge that the server has objects that it\n> > isn't advertising. And that can all happen without buggy code. So it's\n> > weird, but there isn't anything wrong with letting it happen.\n> \n> Well, I don't see how to achieve both at the same time though: we can\n> either limit the set of uninteresting tips to what we have announced to\n> the client, or we allow clients to omit objects that have not been\n> announced. These are mutually exclusive.\n> \n> So if we take the stance that it was fine to send packfiles that omit\n> hidden objects and that this is something we want to continue to support\n> then this patch series probably becomes moot. Doing the proposed\n> optimization means that we also tighten the rules here.\n\nI'm wrong and you're right: we can do the optimization to limit the refs\nwe use but still let clients send objects that are hidden. I didn't take\ninto account that this is merely an optimization that we stop walking at\nreachable tips. I'll reword the commit message when having another go\nand will likely do something along the lines of your proposed new\n`--visible-refs` option in v2 of this series.\n\nPatrick\n"},{"id":"466277","messageId":"Y2HCQAGfY+xsDzKe@nand.local","threadId":"58706","inReplyTo":"Y2EIxp+YM0Bee79v@ncase","subject":"Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-02T01:05:04Z","receivedAt":"2022-11-02T01:05:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 01, 2022 at 12:53:42PM +0100, Patrick Steinhardt wrote:\n> > > Maybe, though I think it's fine to let clients send us smaller packfiles\n> > > if they have some a-priori knowledge that the server has objects that it\n> > > isn't advertising. And that can all happen without buggy code. So it's\n> > > weird, but there isn't anything wrong with letting it happen.\n> >\n> > Well, I don't see how to achieve both at the same time though: we can\n> > either limit the set of uninteresting tips to what we have announced to\n> > the client, or we allow clients to omit objects that have not been\n> > announced. These are mutually exclusive.\n> >\n> > So if we take the stance that it was fine to send packfiles that omit\n> > hidden objects and that this is something we want to continue to support\n> > then this patch series probably becomes moot. Doing the proposed\n> > optimization means that we also tighten the rules here.\n>\n> I'm wrong and you're right: we can do the optimization to limit the refs\n> we use but still let clients send objects that are hidden. I didn't take\n> into account that this is merely an optimization that we stop walking at\n> reachable tips. I'll reword the commit message when having another go\n> and will likely do something along the lines of your proposed new\n> `--visible-refs` option in v2 of this series.\n\nI wasn't necessarily advocating for a behavior change in this series,\nmore pointing out that the situation you said can only happen with buggy\ncode doesn't actually require a bug in practice.\n\nThanks,\nTaylor\n"},{"id":"466404","messageId":"cover.1667485737.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"[PATCH v2 0/3] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-03T14:37:25Z","receivedAt":"2022-11-03T14:37:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that tries to improve\nperformance of the connectivity check by only considering preexisting\nrefs as uninteresting that could actually have been advertised to the\nclient.\n\nThis version uses the same idea as v1, but it's basically rewritten\nbased on Taylor's idea of adding a `git rev-list --visible-refs=`\nswitch. This fixes two major concerns:\n\n 1. The performance regression in repositories is now gone when there\n are no hidden refs. Previously, there was a 10% regression in one\n specific benchmark that was caused by reading advertised tips\n via stdin in git-rev-list(1).\n\n 2. It fixes Jeffs concerns around a TOCTOU-style race with very slow\n clients. It could in theory happen that a push takes multiple\n days. With the previous idea of reusing advertised refs for the\n connectivity check, the end result would be that we perform the\n check with a set of refs that has been generated when the push\n started.\n\nThe series is structured as following:\n\n - Patch 1 does some preliminary cleanups to de-globalize the set of\n parsed hidden refs so that we can use the existing functions\n easily for the new `--visible-refs=` option.\n\n - Patch 2 adds the new `--visible-refs=` option to git-rev-list(1).\n\n - Patch 3 converts git-receive-pack(1) to use `--visible-refs=`\n instead of `--all`.\n\nOverall the performance improvement isn't quite as strong as before:\nwe're only 4.5x faster compared to 6.5x in our repo. But I guess that's\nstill a good-enough improvement, doubly so that there are no downsides\nfor repos anymore that ain't got any hidden refs.\n\nPatrick\n\nPatrick Steinhardt (3):\n refs: get rid of global list of hidden refs\n revision: add new parameter to specify all visible refs\n receive-pack: only use visible refs for connectivity check\n\n Documentation/rev-list-options.txt | 15 +++--\n builtin/receive-pack.c | 10 ++-\n builtin/rev-list.c | 1 +\n builtin/rev-parse.c | 1 +\n connected.c | 5 +-\n connected.h | 6 ++\n ls-refs.c | 13 ++--\n refs.c | 14 ++--\n refs.h | 5 +-\n revision.c | 34 +++++++++-\n t/t6021-rev-list-visible-refs.sh | 102 +++++++++++++++++++++++++++++\n upload-pack.c | 30 +++++----\n 12 files changed, 196 insertions(+), 40 deletions(-)\n create mode 100755 t/t6021-rev-list-visible-refs.sh\n\n-- \n2.38.1\n\n"},{"id":"466405","messageId":"ef619ae6dac84da5375b97c297014f14916e538b.1667485737.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667485737.git.ps@pks.im","subject":"[PATCH v2 1/3] refs: get rid of global list of hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-03T14:37:28Z","receivedAt":"2022-11-03T14:37:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to add a new argument to git-rev-list(1) that allows it to\nadd all references that are visible when taking `transfer.hideRefs` et\nal into account. This will require us to potentially parse multiple sets\nof hidden refs, which is not easily possible right now as there is only\na single, global instance of the list of parsed hidden refs.\n\nRefactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both\ntake the list of hidden references as input and adjust callers to keep a\nlocal list, instead. This allows us to easily use multiple hidden-ref\nlists. Furthermore, it allows us to properly free this list before we\nexit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 8 +++++---\n ls-refs.c | 13 +++++++++----\n refs.c | 14 ++++----------\n refs.h | 5 +++--\n upload-pack.c | 30 ++++++++++++++++++------------\n 5 files changed, 39 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 44bcea3a5b..1f3efc58fb 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)\n \n static int receive_pack_config(const char *var, const char *value, void *cb)\n {\n-\tint status = parse_hide_refs_config(var, value, \"receive\");\n+\tint status = parse_hide_refs_config(var, value, \"receive\", &hidden_refs);\n \n \tif (status)\n \t\treturn status;\n@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,\n \tstruct oidset *seen = data;\n \tconst char *path = strip_namespace(path_full);\n \n-\tif (ref_is_hidden(path, path_full))\n+\tif (ref_is_hidden(path, path_full, &hidden_refs))\n \t\treturn 0;\n \n \t/*\n@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)\n \t\tstrbuf_setlen(&refname_full, prefix_len);\n \t\tstrbuf_addstr(&refname_full, cmd->ref_name);\n \n-\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf))\n+\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))\n \t\t\tcontinue;\n \t\tif (is_null_oid(&cmd->new_oid))\n \t\t\tcmd->error_string = \"deny deleting a hidden ref\";\n@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n+\tstring_list_clear(&hidden_refs, 1);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex fa0d01b47c..ae89f850e9 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -6,6 +6,7 @@\n #include \"ls-refs.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"string-list.h\"\n \n static int config_read;\n static int advertise_unborn;\n@@ -73,6 +74,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n+\tstruct string_list hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \n \tstrbuf_reset(&data->buf);\n \n-\tif (ref_is_hidden(refname_nons, refname))\n+\tif (ref_is_hidden(refname_nons, refname, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (!ref_match(&data->prefixes, refname_nons))\n@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)\n }\n \n static int ls_refs_config(const char *var, const char *value,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *cb_data)\n {\n+\tstruct ls_refs_data *data = cb_data;\n \t/*\n \t * We only serve fetches over v2 for now, so respect only \"uploadpack\"\n \t * config. This may need to eventually be expanded to \"receive\", but we\n \t * don't yet know how that information will be passed to ls-refs.\n \t */\n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n int ls_refs(struct repository *r, struct packet_reader *request)\n@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n+\tstring_list_init_dup(&data.hidden_refs);\n \n \tensure_config_read();\n-\tgit_config(ls_refs_config, NULL);\n+\tgit_config(ls_refs_config, &data);\n \n \twhile (packet_reader_read(request) == PACKET_READ_NORMAL) {\n \t\tconst char *arg = request->line;\n@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n+\tstring_list_clear(&data.hidden_refs, 1);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 1491ae937e..f1711e2e9f 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n \t\t\t\t\t refname, strict);\n }\n \n-static struct string_list *hide_refs;\n-\n-int parse_hide_refs_config(const char *var, const char *value, const char *section)\n+int parse_hide_refs_config(const char *var, const char *value, const char *section,\n+\t\t\t struct string_list *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tif (!hide_refs) {\n-\t\t\tCALLOC_ARRAY(hide_refs, 1);\n-\t\t\thide_refs->strdup_strings = 1;\n-\t\t}\n \t\tstring_list_append(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n-int ref_is_hidden(const char *refname, const char *refname_full)\n+int ref_is_hidden(const char *refname, const char *refname_full,\n+\t\t const struct string_list *hide_refs)\n {\n \tint i;\n \n-\tif (!hide_refs)\n-\t\treturn 0;\n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n \t\tconst char *match = hide_refs->items[i].string;\n \t\tconst char *subject;\ndiff --git a/refs.h b/refs.h\nindex 8958717a17..3266fd8f57 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,\n \t const struct object_id *new_oid, const struct object_id *old_oid,\n \t unsigned int flags, enum action_on_err onerr);\n \n-int parse_hide_refs_config(const char *var, const char *value, const char *);\n+int parse_hide_refs_config(const char *var, const char *value, const char *,\n+\t\t\t struct string_list *);\n \n /*\n * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);\n * the ref is outside that namespace, the first parameter is NULL. The second\n * parameter always points to the full ref name.\n */\n-int ref_is_hidden(const char *, const char *);\n+int ref_is_hidden(const char *, const char *, const struct string_list *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0b8311bd68..9db17f8787 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -62,6 +62,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n+\tstruct string_list hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n+\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n \tmemset(data, 0, sizeof(*data));\n \tdata->symref = symref;\n \tdata->wanted_refs = wanted_refs;\n+\tdata->hidden_refs = hidden_refs;\n \tdata->want_obj = want_obj;\n \tdata->have_obj = have_obj;\n \tdata->haves = haves;\n@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n+\tstring_list_clear(&data->hidden_refs, 1);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * Checking for reachable shallows requires that our refs be\n \t\t * marked with OUR_REF.\n \t\t */\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, data);\n+\t\tfor_each_namespaced_ref(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid)\n+\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n-\tif (ref_is_hidden(refname, refname_full)) {\n+\tif (ref_is_hidden(refname, refname_full, hidden_refs)) {\n \t\to->flags |= HIDDEN_REF;\n \t\treturn 1;\n \t}\n@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,\n }\n \n static int check_ref(const char *refname_full, const struct object_id *oid,\n-\t\t int flag UNUSED, void *cb_data UNUSED)\n+\t\t int flag UNUSED, void *cb_data)\n {\n \tconst char *refname = strip_namespace(refname_full);\n+\tstruct upload_pack_data *data = cb_data;\n \n-\tmark_our_ref(refname, refname_full, oid);\n+\tmark_our_ref(refname, refname_full, oid, &data->hidden_refs);\n \treturn 0;\n }\n \n@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \tstruct object_id peeled;\n \tstruct upload_pack_data *data = cb_data;\n \n-\tif (mark_our_ref(refname_nons, refname, oid))\n+\tif (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (capabilities) {\n@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)\n \tif (parse_object_filter_config(var, value, data) < 0)\n \t\treturn -1;\n \n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)\n@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, &data);\n+\t\tfor_each_namespaced_ref(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t struct string_list *wanted_refs,\n+\t\t\t struct string_list *hidden_refs,\n \t\t\t struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\tstruct strbuf refname = STRBUF_INIT;\n \n \t\tstrbuf_addf(&refname, \"%s%s\", get_git_namespace(), refname_nons);\n-\t\tif (ref_is_hidden(refname_nons, refname.buf) ||\n+\t\tif (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||\n \t\t read_ref(refname.buf, &oid)) {\n \t\t\tpacket_writer_error(writer, \"unknown ref %s\", refname_nons);\n \t\t\tdie(\"unknown ref %s\", refname_nons);\n@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (data->allow_ref_in_want &&\n \t\t parse_want_ref(&data->writer, arg, &data->wanted_refs,\n-\t\t\t\t &data->want_obj))\n+\t\t\t\t &data->hidden_refs, &data->want_obj))\n \t\t\tcontinue;\n \t\t/* process have line */\n \t\tif (parse_have(arg, &data->haves))\n-- \n2.38.1\n\n"},{"id":"466406","messageId":"3ccd8fc0e35407e5c9ff896165f122b10598e0e2.1667485737.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667485737.git.ps@pks.im","subject":"[PATCH v2 2/3] revision: add new parameter to specify all visible refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-03T14:37:32Z","receivedAt":"2022-11-03T14:37:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Users can optionally hide refs from remote users in git-upload-pack(1),\ngit-receive-pack(1) and others via the `transfer.hideRefs`, but there is\nnot an easy way to obtain the list of all visible or hidden refs right\nnow. We'll require just that though for a performance improvement in our\nconnectivity check.\n\nAdd a new pseudo-ref `--visible-refs=` that pretends as if all refs have\nbeen added to the command line that are not hidden. The pseudo-ref\nrequiers either one of \"transfer\", \"uploadpack\" or \"receive\" as argument\nto pay attention to `transfer.hideRefs`, `uploadpack.hideRefs` or\n`receive.hideRefs`, respectively.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 15 +++--\n builtin/rev-list.c | 1 +\n builtin/rev-parse.c | 1 +\n revision.c | 34 +++++++++-\n t/t6021-rev-list-visible-refs.sh | 102 +++++++++++++++++++++++++++++\n 5 files changed, 145 insertions(+), 8 deletions(-)\n create mode 100755 t/t6021-rev-list-visible-refs.sh\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 1837509566..a0e34b0e2b 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -180,14 +180,19 @@ endif::git-log[]\n \tis automatically prepended if missing. If pattern lacks '?', '{asterisk}',\n \tor '[', '/{asterisk}' at the end is implied.\n \n+--visible-refs=[transfer|receive|uploadpack]::\n+\tPretend as if all the refs that have not been hidden via either one of\n+\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` are\n+\tlisted on the command line.\n+\n --exclude=::\n \n \tDo not include refs matching '' that the next `--all`,\n-\t`--branches`, `--tags`, `--remotes`, or `--glob` would otherwise\n-\tconsider. Repetitions of this option accumulate exclusion patterns\n-\tup to the next `--all`, `--branches`, `--tags`, `--remotes`, or\n-\t`--glob` option (other options or arguments do not clear\n-\taccumulated patterns).\n+\t`--branches`, `--tags`, `--remotes`, `--glob` or `--visible-refs` would\n+\totherwise consider. Repetitions of this option accumulate exclusion\n+\tpatterns up to the next `--all`, `--branches`, `--tags`, `--remotes`,\n+\t`--glob` or `--visible-refs` option (other options or arguments do not\n+\tclear accumulated patterns).\n +\n The patterns given should not begin with `refs/heads`, `refs/tags`, or\n `refs/remotes` when applied to `--branches`, `--tags`, or `--remotes`,\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3acd93f71e..f719286cf8 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -38,6 +38,7 @@ static const char rev_list_usage[] =\n \" --tags\\n\"\n \" --remotes\\n\"\n \" --stdin\\n\"\n+\" --visible-refs=[transfer|receive|uploadpack]\\n\"\n \" --quiet\\n\"\n \" ordering output:\\n\"\n \" --topo-order\\n\"\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 8f61050bde..31617bf3d5 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -77,6 +77,7 @@ static int is_rev_argument(const char *arg)\n \t\t\"--topo-order\",\n \t\t\"--date-order\",\n \t\t\"--unpacked\",\n+\t\t\"--visible-refs=\",\n \t\tNULL\n \t};\n \tconst char **p = rev_args;\ndiff --git a/revision.c b/revision.c\nindex 0760e78936..ef9e2947af 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"object-store.h\"\n #include \"tag.h\"\n #include \"blob.h\"\n@@ -1523,6 +1524,8 @@ struct all_refs_cb {\n \tstruct rev_info *all_revs;\n \tconst char *name_for_errormsg;\n \tstruct worktree *wt;\n+\tstruct string_list hidden_refs;\n+\tconst char *hidden_refs_section;\n };\n \n int ref_excluded(struct string_list *ref_excludes, const char *path)\n@@ -1542,11 +1545,13 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \t\t\t int flag UNUSED,\n \t\t\t void *cb_data)\n {\n+\tconst char *stripped_path = strip_namespace(path);\n \tstruct all_refs_cb *cb = cb_data;\n \tstruct object *object;\n \n-\tif (ref_excluded(cb->all_revs->ref_excludes, path))\n-\t return 0;\n+\tif (ref_excluded(cb->all_revs->ref_excludes, path) ||\n+\t ref_is_hidden(stripped_path, path, &cb->hidden_refs))\n+\t\treturn 0;\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n \tadd_rev_cmdline(cb->all_revs, object, path, REV_CMD_REF, cb->all_flags);\n@@ -1561,6 +1566,7 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,\n \tcb->all_flags = flags;\n \trevs->rev_input_given = 1;\n \tcb->wt = NULL;\n+\tstring_list_init_dup(&cb->hidden_refs);\n }\n \n void clear_ref_exclusion(struct string_list **ref_excludes_p)\n@@ -1596,6 +1602,13 @@ static void handle_refs(struct ref_store *refs,\n \tfor_each(refs, handle_one_ref, &cb);\n }\n \n+static int hide_refs_config(const char *var, const char *value, void *cb_data)\n+{\n+\tstruct all_refs_cb *cb = cb_data;\n+\treturn parse_hide_refs_config(var, value, cb->hidden_refs_section,\n+\t\t\t\t &cb->hidden_refs);\n+}\n+\n static void handle_one_reflog_commit(struct object_id *oid, void *cb_data)\n {\n \tstruct all_refs_cb *cb = cb_data;\n@@ -2225,7 +2238,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t !strcmp(arg, \"--bisect\") || starts_with(arg, \"--glob=\") ||\n \t !strcmp(arg, \"--indexed-objects\") ||\n \t !strcmp(arg, \"--alternate-refs\") ||\n-\t starts_with(arg, \"--exclude=\") ||\n+\t starts_with(arg, \"--exclude=\") || starts_with(arg, \"--visible-refs=\") ||\n \t starts_with(arg, \"--branches=\") || starts_with(arg, \"--tags=\") ||\n \t starts_with(arg, \"--remotes=\") || starts_with(arg, \"--no-walk=\"))\n \t{\n@@ -2759,6 +2772,21 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\tparse_list_objects_filter(&revs->filter, arg);\n \t} else if (!strcmp(arg, (\"--no-filter\"))) {\n \t\tlist_objects_filter_set_no_filter(&revs->filter);\n+\t} else if (skip_prefix(arg, \"--visible-refs=\", &arg)) {\n+\t\tstruct all_refs_cb cb;\n+\n+\t\tif (strcmp(arg, \"transfer\") && strcmp(arg, \"receive\") &&\n+\t\t strcmp(arg, \"uploadpack\"))\n+\t\t\tdie(_(\"unsupported section for --visible-refs: %s\"), arg);\n+\n+\t\tinit_all_refs_cb(&cb, revs, *flags);\n+\t\tcb.hidden_refs_section = arg;\n+\t\tgit_config(hide_refs_config, &cb);\n+\n+\t\trefs_for_each_ref(refs, handle_one_ref, &cb);\n+\n+\t\tstring_list_clear(&cb.hidden_refs, 1);\n+\t\tclear_ref_exclusion(&revs->ref_excludes);\n \t} else {\n \t\treturn 0;\n \t}\ndiff --git a/t/t6021-rev-list-visible-refs.sh b/t/t6021-rev-list-visible-refs.sh\nnew file mode 100755\nindex 0000000000..9e12384dcf\n--- /dev/null\n+++ b/t/t6021-rev-list-visible-refs.sh\n@@ -0,0 +1,102 @@\n+#!/bin/sh\n+\n+test_description='git rev-list --visible-refs test'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit_bulk --id=commit --ref=refs/heads/main 1 &&\n+\tCOMMIT=$(git rev-parse refs/heads/main) &&\n+\ttest_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&\n+\tTAG=$(git rev-parse refs/tags/lightweight) &&\n+\ttest_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&\n+\tHIDDEN=$(git rev-parse refs/hidden/commit)\n+'\n+\n+test_expect_success 'invalid section' '\n+\techo \"fatal: unsupported section for --visible-refs: unsupported\" >expected &&\n+\ttest_must_fail git rev-list --visible-refs=unsupported 2>err &&\n+\ttest_cmp expected err\n+'\n+\n+test_expect_success '--visible-refs without hiddenRefs' '\n+\tgit rev-list --visible-refs=transfer >out &&\n+\tcat >expected <<-EOF &&\n+\t$HIDDEN\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success 'hidden via transfer.hideRefs' '\n+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --visible-refs=transfer >out &&\n+\tcat >expected <<-EOF &&\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--all --not --visible-refs=transfer without hidden refs' '\n+\tgit rev-list --all --not --visible-refs=transfer >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success '--all --not --visible-refs=transfer with hidden ref' '\n+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --all --not --visible-refs=transfer >out &&\n+\tcat >expected <<-EOF &&\n+\t$HIDDEN\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--visible-refs with --exclude' '\n+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --visible-refs=transfer >out &&\n+\tcat >expected <<-EOF &&\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"hidden via $section.hideRefs\" '\n+\t\tgit -c receive.hideRefs=refs/hidden/ rev-list --visible-refs=receive >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"--visible-refs=$section respects transfer.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --visible-refs=$section >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"--visible-refs=transfer ignores $section.hideRefs\" '\n+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --visible-refs=transfer >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"--visible-refs=$section respects both transfer.hideRefs and $section.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --visible-refs=$section >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+done\n+\n+test_done\n-- \n2.38.1\n\n"},{"id":"466407","messageId":"71c3d791aa60d0024793aa72e7d94d59d3e337d6.1667485737.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667485737.git.ps@pks.im","subject":"[PATCH v2 3/3] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-03T14:37:36Z","receivedAt":"2022-11-03T14:37:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When serving a push, git-receive-pack(1) needs to verify that the\npackfile sent by the client contains all objects that are required by\nthe updated references. This connectivity check works by marking all\npreexisting references as uninteresting and using the new reference tips\nas starting point for a graph walk.\n\nMarking all preexisting references as uninteresting can be a problem\nwhen it comes to performance. Git forges tend to do internal bookkeeping\nto keep alive sets of objects for internal use or make them easy to find\nvia certain references. These references are typically hidden away from\nthe user so that they are neither advertised nor writeable. At GitLab,\nwe have one particular repository that contains a total of 7 million\nreferences, of which 6.8 million are indeed internal references. With\nthe current connectivity check we are forced to load all these\nreferences in order to mark them as uninteresting, and this alone takes\naround 15 seconds to compute.\n\nWe can optimize this by only taking into account the set of visible refs\nwhen marking objects as uninteresting. This means that we may now walk\nmore objects until we hit any object that is marked as uninteresting.\nBut it is rather unlikely that clients send objects that make large\nparts of objects reachable that have previously only ever been hidden,\nwhereas the common case is to push incremental changes that build on top\nof the visible object graph.\n\nThis provides a huge boost to performance in the mentioned repository,\nwhere the vast majority of its refs hidden. Pushing a new commit into\nthis repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\nas it is configured in Gitaly leads to an almost 4.5-fold speedup:\n\n Benchmark 1: main\n Time (mean ± σ): 29.475 s ± 0.248 s [User: 28.812 s, System: 1.006 s]\n Range (min … max): 29.189 s … 29.636 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 6.657 s ± 0.027 s [User: 6.664 s, System: 0.355 s]\n Range (min … max): 6.629 s … 6.682 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 4.43 ± 0.04 times faster than 'main'\n\nAs we mostly go through the same codepaths even in the case where there\nare no hidden refs at all compared to the code before there is no change\nin performance when no refs are hidden:\n\n Benchmark 1: main\n Time (mean ± σ): 48.688 s ± 1.535 s [User: 49.579 s, System: 5.058 s]\n Range (min … max): 47.518 s … 50.425 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 48.147 s ± 0.716 s [User: 48.800 s, System: 5.297 s]\n Range (min … max): 47.694 s … 48.973 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 1.01 ± 0.04 times faster than 'main'\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 2 ++\n connected.c | 5 ++++-\n connected.h | 6 ++++++\n 3 files changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1f3efc58fb..be1c1d2702 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,\n \topt.err_fd = err_fd;\n \topt.progress = err_fd && !quiet;\n \topt.env = tmp_objdir_env(tmp_objdir);\n+\topt.visible_refs_section = \"receive\";\n+\n \tif (check_connected(iterate_receive_command_list, &data, &opt))\n \t\tset_connectivity_errors(commands, si);\n \ndiff --git a/connected.c b/connected.c\nindex 74a20cb32e..c64501f755 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -100,7 +100,10 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n \tif (!opt->is_deepening_fetch) {\n \t\tstrvec_push(&rev_list.args, \"--not\");\n-\t\tstrvec_push(&rev_list.args, \"--all\");\n+\t\tif (opt->visible_refs_section)\n+\t\t\tstrvec_pushf(&rev_list.args, \"--visible-refs=%s\", opt->visible_refs_section);\n+\t\telse\n+\t\t\tstrvec_push(&rev_list.args, \"--all\");\n \t}\n \tstrvec_push(&rev_list.args, \"--quiet\");\n \tstrvec_push(&rev_list.args, \"--alternate-refs\");\ndiff --git a/connected.h b/connected.h\nindex 6e59c92aa3..d8396e5d55 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -46,6 +46,12 @@ struct check_connected_options {\n \t * during a fetch.\n \t */\n \tunsigned is_deepening_fetch : 1;\n+\n+\t/*\n+\t * If not NULL, use `--visible-refs=$section` instead of `--not --all`\n+\t * as the set of already-reachable references.\n+\t */\n+\tconst char *visible_refs_section;\n };\n \n #define CHECK_CONNECTED_INIT { 0 }\n-- \n2.38.1\n\n"},{"id":"466571","messageId":"Y2WxAwNuKTOJEhz0@nand.local","threadId":"58706","inReplyTo":"cover.1667485737.git.ps@pks.im","subject":"Re: [PATCH v2 0/3] receive-pack: only use visible refs for connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-05T00:40:35Z","receivedAt":"2022-11-05T00:40:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 03, 2022 at 03:37:25PM +0100, Patrick Steinhardt wrote:\n> Hi,\n>\n> this is the second version of my patch series that tries to improve\n> performance of the connectivity check by only considering preexisting\n> refs as uninteresting that could actually have been advertised to the\n> client.\n\nThis version was delightful to read, and I don't have any concerns with\nthe approach or implementation.\n\nI would appreciate another set of reviewer eyes on the topic, but if\nnot, I am comfortable starting to merge this down as-is.\n\nThanks,\nTaylor\n"},{"id":"466577","messageId":"Y2ZbDXYb1jGqSfTj@coredump.intra.peff.net","threadId":"58706","inReplyTo":"3ccd8fc0e35407e5c9ff896165f122b10598e0e2.1667485737.git.ps@pks.im","subject":"Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-05T12:46:05Z","receivedAt":"2022-11-05T12:46:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 03, 2022 at 03:37:32PM +0100, Patrick Steinhardt wrote:\n\n> Users can optionally hide refs from remote users in git-upload-pack(1),\n> git-receive-pack(1) and others via the `transfer.hideRefs`, but there is\n> not an easy way to obtain the list of all visible or hidden refs right\n> now. We'll require just that though for a performance improvement in our\n> connectivity check.\n> \n> Add a new pseudo-ref `--visible-refs=` that pretends as if all refs have\n> been added to the command line that are not hidden. The pseudo-ref\n> requiers either one of \"transfer\", \"uploadpack\" or \"receive\" as argument\n> to pay attention to `transfer.hideRefs`, `uploadpack.hideRefs` or\n> `receive.hideRefs`, respectively.\n\nThanks for re-working this. I think it's a sensible path forward for the\nproblem you're facing.\n\nThere were two parts of the implementation that surprised me a bit.\nThese might just be nits, but because this is a new user-facing plumbing\noption that will be hard to change later, we should make sure it fits in\nwith the adjacent features.\n\nThe two things I saw were:\n\n 1. The mutual-exclusion selection between \"transfer\", \"uploadpack\",\n and \"receive\" is not how those options work in their respective\n programs. The \"transfer.hideRefs\" variable is respected for either\n program. So whichever program you are running, it will always look\n at both \"transfer\" and itself (\"uploadpack\" or \"receive\"). Another\n way to think of it is that the \"section\" argument to\n parse_hide_refs_config() is not a config section so much as a\n profile. And the profiles are:\n\n - uploadpack: respect transfer.hideRefs and uploadpack.hideRefs\n - receive: respect transfer.hideRefs and receive.hideRefs\n\n So it does not make sense to ask for \"transfer\" as a section; each\n of the modes is already looking at transfer.hideRefs.\n\n In theory if this option was \"look just at $section.hideRefs\", it\n could be more flexible to separate out the two. But that makes it\n more of a pain to use (for normal use, you have to specify both\n \"transfer\" and \"receive\"). And that is not what your patch\n implements anyway; because it relies on parse_hide_refs_config(),\n it is always adding in \"transfer\" under the hood (which is why your\n final patch is correct to just say \"--visible-refs=receive\" without\n specifying \"transfer\" as well).\n\n 2. Your \"--visible-refs\" implies \"--all\", because it's really \"all\n refs minus the hidden ones\". That's convenient for the intended\n caller, but not as flexible as it could be. If it were instead\n treated the way \"--exclude\" is, as a modifier for the next\n iteration option, then you do a few extra things:\n\n a. Combine multiple exclusions in a single iteration:\n\n git rev-list --exclude-hidden=receive \\\n\t --exclude-hidden=upload \\\n\t\t\t --all\n\n That excludes both types in a single iteration. Whereas if you\n\t did:\n\n\t git rev-list --visible-refs=receive \\\n\t --visible-refs=upload\n\n\t that will do _two_ iterations, and end up with the union of\n\t the sets. Equivalent to:\n\n\t git rev-list --exclude-hidden=receive --all \\\n\t --exclude-hidden=upload --all\n\n b. Do exclusions on smaller sets than --all:\n\n git rev-list --exclude-hidden=receive \\\n\t --branches\n\n\t which would show just the branches that we'd advertise.\n\n Now I don't have a particular use case for either of those things.\n But they're plausible things to want in the long run, and they fit\n in nicely with the existing ref-selection scheme of rev-list. They\n do make your call from check_connected() slightly longer, but it is\n pretty negligible. It's \"--exclude-hidden=receive --all\" instead of\n \"--visible-refs=hidden\".\n\nSo looking at the patch itself, if you wanted to take my suggestions:\n\n> +--visible-refs=[transfer|receive|uploadpack]::\n> +\tPretend as if all the refs that have not been hidden via either one of\n> +\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` are\n> +\tlisted on the command line.\n\nThis would drop \"transfer\" as a mode, and explain that the argument is\n\"hide the refs that receive-pack would use\", etc.\n\nLikewise, the name would switch and pick up explanation similar to\n--exclude below about how it affects the next --all, etc.\n\n> @@ -1542,11 +1545,13 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n> \t\t\t int flag UNUSED,\n> \t\t\t void *cb_data)\n> {\n> +\tconst char *stripped_path = strip_namespace(path);\n> \tstruct all_refs_cb *cb = cb_data;\n> \tstruct object *object;\n> \n> -\tif (ref_excluded(cb->all_revs->ref_excludes, path))\n> -\t return 0;\n> +\tif (ref_excluded(cb->all_revs->ref_excludes, path) ||\n> +\t ref_is_hidden(stripped_path, path, &cb->hidden_refs))\n> +\t\treturn 0;\n\nThis would stay the same. We'd still exclude hidden refs during the\niteration.\n\n> @@ -2759,6 +2772,21 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n> \t\tparse_list_objects_filter(&revs->filter, arg);\n> \t} else if (!strcmp(arg, (\"--no-filter\"))) {\n> \t\tlist_objects_filter_set_no_filter(&revs->filter);\n> +\t} else if (skip_prefix(arg, \"--visible-refs=\", &arg)) {\n> +\t\tstruct all_refs_cb cb;\n> +\n> +\t\tif (strcmp(arg, \"transfer\") && strcmp(arg, \"receive\") &&\n> +\t\t strcmp(arg, \"uploadpack\"))\n> +\t\t\tdie(_(\"unsupported section for --visible-refs: %s\"), arg);\n> +\n> +\t\tinit_all_refs_cb(&cb, revs, *flags);\n> +\t\tcb.hidden_refs_section = arg;\n> +\t\tgit_config(hide_refs_config, &cb);\n> +\n> +\t\trefs_for_each_ref(refs, handle_one_ref, &cb);\n> +\n> +\t\tstring_list_clear(&cb.hidden_refs, 1);\n> +\t\tclear_ref_exclusion(&revs->ref_excludes);\n\nAnd here we'd do the same git_config() call, but drop the\nrefs_for_each_ref() call. We'd clear the hidden_refs field in all the\nplaces that call clear_ref_exclusion() now.\n\nIn fact, you could argue that all of this should just be folded into\nclear_ref_exclusion() and ref_excluded(), since from the perspective of\nthe iterating code, they are all part of the same feature. I don't mind\nleaving it separate from the perspective of rev-list, though I think\nif you did so, it would all Just Work for \"rev-parse\", too (I doubt\nanybody cares in practice, but it's probably better to keep it\nconsistent).\n\n-Peff\n"},{"id":"466580","messageId":"Y2Zci0z39Q0/sMHY@coredump.intra.peff.net","threadId":"58706","inReplyTo":"cover.1667485737.git.ps@pks.im","subject":"Re: [PATCH v2 0/3] receive-pack: only use visible refs for connectivity check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-05T12:52:27Z","receivedAt":"2022-11-05T12:52:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 03, 2022 at 03:37:25PM +0100, Patrick Steinhardt wrote:\n\n> Overall the performance improvement isn't quite as strong as before:\n> we're only 4.5x faster compared to 6.5x in our repo. But I guess that's\n> still a good-enough improvement, doubly so that there are no downsides\n> for repos anymore that ain't got any hidden refs.\n\nJust a guess, but the extra time is probably spent because \"rev-list\"\nhas to iterate over your kajillion refs saying \"nope, not this one\".\n\nI wonder if there is a way to carve up the ref namespace early, in a\ntrie-like way, similar to how for_each_fullref_in_prefixes() skips\nstraight to the interesting part. It's a harder problem, because we are\nexcluding rather than including entries based on our patterns. But it\nseems like you'd be able to notice that we are on \"refs/foo/bar\", that\nall of \"refs/foo\" is excluded, and jump ahead to the end of the\n\"refs/foo\" section by binary-searching for the end within the\npacked-refs file.\n\nAnd that would speed up both this rev-list, but also the initial\nadvertisement that receive-pack does, because it's also iterating all of\nthese saying \"nope, it's hidden\". And assuming these are all internal\nrefs you hide from users, upload-pack would benefit, too.\n\nI don't think that should be part of this series, but it may be an\ninteresting future direction.\n\n-Peff\n"},{"id":"466581","messageId":"Y2ZdKL9QG+fNw3ME@coredump.intra.peff.net","threadId":"58706","inReplyTo":"3ccd8fc0e35407e5c9ff896165f122b10598e0e2.1667485737.git.ps@pks.im","subject":"Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-05T12:55:04Z","receivedAt":"2022-11-05T12:55:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 03, 2022 at 03:37:32PM +0100, Patrick Steinhardt wrote:\n\n> diff --git a/t/t6021-rev-list-visible-refs.sh b/t/t6021-rev-list-visible-refs.sh\n> new file mode 100755\n> index 0000000000..9e12384dcf\n\nOh, I forgot to mention in my earlier response: these tests mostly look\ngood to me, but it may be worth having a few that cover the namespaces\nfeature. I _think_ your code is doing the right thing (because it calls\nstrip_namespace() as appropriate), but I have to admit I am generally\nconfused about how the namespace feature works in the first place. So\nit's probably good to figure out how it is supposed to interact here and\nmake sure it is covered by tests.\n\n-Peff\n"},{"id":"466582","messageId":"Y2ZdScjauDUKOG+z@coredump.intra.peff.net","threadId":"58706","inReplyTo":"Y2WxAwNuKTOJEhz0@nand.local","subject":"Re: [PATCH v2 0/3] receive-pack: only use visible refs for connectivity check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-05T12:55:37Z","receivedAt":"2022-11-05T12:55:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 04, 2022 at 08:40:35PM -0400, Taylor Blau wrote:\n\n> On Thu, Nov 03, 2022 at 03:37:25PM +0100, Patrick Steinhardt wrote:\n> > Hi,\n> >\n> > this is the second version of my patch series that tries to improve\n> > performance of the connectivity check by only considering preexisting\n> > refs as uninteresting that could actually have been advertised to the\n> > client.\n> \n> This version was delightful to read, and I don't have any concerns with\n> the approach or implementation.\n> \n> I would appreciate another set of reviewer eyes on the topic, but if\n> not, I am comfortable starting to merge this down as-is.\n\nI like the overall direction, but I had some suggestions for the\ninterface in patch 2. Hopefully Patrick agrees, and we'll see a v2.\n\n-Peff\n"},{"id":"466642","messageId":"Y2i/uJPm5KxqAdkE@ncase","threadId":"58706","inReplyTo":"Y2ZbDXYb1jGqSfTj@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T08:20:08Z","receivedAt":"2022-11-07T08:20:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Nov 05, 2022 at 08:46:05AM -0400, Jeff King wrote:\n> On Thu, Nov 03, 2022 at 03:37:32PM +0100, Patrick Steinhardt wrote:\n> \n> > Users can optionally hide refs from remote users in git-upload-pack(1),\n> > git-receive-pack(1) and others via the `transfer.hideRefs`, but there is\n> > not an easy way to obtain the list of all visible or hidden refs right\n> > now. We'll require just that though for a performance improvement in our\n> > connectivity check.\n> > \n> > Add a new pseudo-ref `--visible-refs=` that pretends as if all refs have\n> > been added to the command line that are not hidden. The pseudo-ref\n> > requiers either one of \"transfer\", \"uploadpack\" or \"receive\" as argument\n> > to pay attention to `transfer.hideRefs`, `uploadpack.hideRefs` or\n> > `receive.hideRefs`, respectively.\n> \n> Thanks for re-working this. I think it's a sensible path forward for the\n> problem you're facing.\n> \n> There were two parts of the implementation that surprised me a bit.\n> These might just be nits, but because this is a new user-facing plumbing\n> option that will be hard to change later, we should make sure it fits in\n> with the adjacent features.\n> \n> The two things I saw were:\n> \n> 1. The mutual-exclusion selection between \"transfer\", \"uploadpack\",\n> and \"receive\" is not how those options work in their respective\n> programs. The \"transfer.hideRefs\" variable is respected for either\n> program. So whichever program you are running, it will always look\n> at both \"transfer\" and itself (\"uploadpack\" or \"receive\"). Another\n> way to think of it is that the \"section\" argument to\n> parse_hide_refs_config() is not a config section so much as a\n> profile. And the profiles are:\n> \n> - uploadpack: respect transfer.hideRefs and uploadpack.hideRefs\n> - receive: respect transfer.hideRefs and receive.hideRefs\n> \n> So it does not make sense to ask for \"transfer\" as a section; each\n> of the modes is already looking at transfer.hideRefs.\n> \n> In theory if this option was \"look just at $section.hideRefs\", it\n> could be more flexible to separate out the two. But that makes it\n> more of a pain to use (for normal use, you have to specify both\n> \"transfer\" and \"receive\"). And that is not what your patch\n> implements anyway; because it relies on parse_hide_refs_config(),\n> it is always adding in \"transfer\" under the hood (which is why your\n> final patch is correct to just say \"--visible-refs=receive\" without\n> specifying \"transfer\" as well).\n\nYup, I'm aware of this. And as you say, the current implementation\nalready handles this alright for both `receive` and `uploadpack` as we\nrely on `parse_hide_refs_config()`, which knows to look at both\n`transfer.hideRefs` and `$section.hideRefs`. But I don't see a reason\nwhy we shouldn't allow users to ask \"What is the set of hidden refs that\nare shared by `uploadpack` and `receive`?\", which is exactly what\n`--visible-refs=transfer` does.\n\nThe implementation is not really explicit about this as we cheat a\nlittle bit here by passing \"transfer\" as a section to the parsing\nfunction. So what it does right now is to basically check for the same\nsection twice, once via the hard-coded \"transfer.hideRefs\" and once for\nthe \"$section.hideRefs\" with `section == \"transfer\"`. But I didn't see\nmuch of a point in making this more explicit.\n\nI might update the commit message and/or documentation though to point\nthis out.\n\n> 2. Your \"--visible-refs\" implies \"--all\", because it's really \"all\n> refs minus the hidden ones\". That's convenient for the intended\n> caller, but not as flexible as it could be. If it were instead\n> treated the way \"--exclude\" is, as a modifier for the next\n> iteration option, then you do a few extra things:\n> \n> a. Combine multiple exclusions in a single iteration:\n> \n> git rev-list --exclude-hidden=receive \\\n> \t --exclude-hidden=upload \\\n> \t\t\t --all\n> \n> That excludes both types in a single iteration. Whereas if you\n> \t did:\n> \n> \t git rev-list --visible-refs=receive \\\n> \t --visible-refs=upload\n> \n> \t that will do _two_ iterations, and end up with the union of\n> \t the sets. Equivalent to:\n> \n> \t git rev-list --exclude-hidden=receive --all \\\n> \t --exclude-hidden=upload --all\n> \n> b. Do exclusions on smaller sets than --all:\n> \n> git rev-list --exclude-hidden=receive \\\n> \t --branches\n> \n> \t which would show just the branches that we'd advertise.\n> \n> Now I don't have a particular use case for either of those things.\n> But they're plausible things to want in the long run, and they fit\n> in nicely with the existing ref-selection scheme of rev-list. They\n> do make your call from check_connected() slightly longer, but it is\n> pretty negligible. It's \"--exclude-hidden=receive --all\" instead of\n> \"--visible-refs=hidden\".\n\nFair enough. I guess that the usecase where you e.g. only hide a subset\nof branches via `hideRefs` is going to be rare, so in most cases you\ndon't gain much by modelling this so that you can `--exclude-hidden\n--branches`. But as you rightfully point out, modelling it that way fits\nneatly with the existing `--exclude` switch and is overall more\nflexible. So there's not much of a reason to not do so.\n\nI'll send a v3 and include your suggestion, thanks.\n\nPatrick\n"},{"id":"466648","messageId":"cover.1667823042.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"[PATCH v3 0/6] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T12:16:19Z","receivedAt":"2022-11-07T12:17:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that tries to improve\nperformance of the connectivity check by only considering preexisting\nrefs as uninteresting that could actually have been advertised to the\nclient.\n\nIn v2 of this series, we introduced a new option `--visible-refs=` that\nmostly acted as if `--all` was given, but its output was restricted to\nonly visible refs. As Peff rightly points out though, this is less\nflexible than it needs to be. This new version instead introduces a new\noption `--exclude-hidden=` that can be combined with `--all`, `--glob`,\n`--branches` and so on to provide a more flexible interface.\n\nThe patch series is structured as following:\n\n - Patch 1-3 refactor multiple different parts of refs.c and\n revision.c so that they are more readily reusable.\n\n - Patch 4-5 implement `--exclude-hidden=` for git-rev-list(1) and\n git-rev-parse(1).\n\n - Patch 6 starts using that option in the connectivity check.\n\nPatrick\n\n\nPatrick Steinhardt (6):\n refs: get rid of global list of hidden refs\n revision: move together exclusion-related functions\n revision: introduce struct to handle exclusions\n revision: add new parameter to exclude hidden refs\n revparse: add `--exclude-hidden=` option\n receive-pack: only use visible refs for connectivity check\n\n Documentation/git-rev-parse.txt | 7 ++\n Documentation/rev-list-options.txt | 7 ++\n builtin/receive-pack.c | 10 +-\n builtin/rev-list.c | 1 +\n builtin/rev-parse.c | 12 ++-\n connected.c | 3 +\n connected.h | 7 ++\n ls-refs.c | 13 ++-\n refs.c | 14 +--\n refs.h | 5 +-\n revision.c | 118 +++++++++++++--------\n revision.h | 29 ++++--\n t/t6018-rev-list-glob.sh | 8 ++\n t/t6021-rev-list-exclude-hidden.sh | 159 +++++++++++++++++++++++++++++\n upload-pack.c | 30 +++---\n 15 files changed, 341 insertions(+), 82 deletions(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\n-- \n2.38.1\n\n"},{"id":"466649","messageId":"a6dcc99ca9526d6a05d5d5aa1e6a8ff14af991da.1667823042.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667823042.git.ps@pks.im","subject":"[PATCH v3 2/6] revision: move together exclusion-related functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T12:16:27Z","receivedAt":"2022-11-07T12:17:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move together the definitions of functions that handle exclusions of\nrefs so that related functionality sits in a single place, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 52 ++++++++++++++++++++++++++--------------------------\n 1 file changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 0760e78936..be755670e2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-struct all_refs_cb {\n-\tint all_flags;\n-\tint warned_bad_reflog;\n-\tstruct rev_info *all_revs;\n-\tconst char *name_for_errormsg;\n-\tstruct worktree *wt;\n-};\n-\n int ref_excluded(struct string_list *ref_excludes, const char *path)\n {\n \tstruct string_list_item *item;\n@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)\n \treturn 0;\n }\n \n+void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+{\n+\tif (*ref_excludes_p) {\n+\t\tstring_list_clear(*ref_excludes_p, 0);\n+\t\tfree(*ref_excludes_p);\n+\t}\n+\t*ref_excludes_p = NULL;\n+}\n+\n+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+{\n+\tif (!*ref_excludes_p) {\n+\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n+\t\t(*ref_excludes_p)->strdup_strings = 1;\n+\t}\n+\tstring_list_append(*ref_excludes_p, exclude);\n+}\n+\n+struct all_refs_cb {\n+\tint all_flags;\n+\tint warned_bad_reflog;\n+\tstruct rev_info *all_revs;\n+\tconst char *name_for_errormsg;\n+\tstruct worktree *wt;\n+};\n+\n static int handle_one_ref(const char *path, const struct object_id *oid,\n \t\t\t int flag UNUSED,\n \t\t\t void *cb_data)\n@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,\n \tcb->wt = NULL;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n-{\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n-}\n-\n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n-{\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n-}\n-\n static void handle_refs(struct ref_store *refs,\n \t\t\tstruct rev_info *revs, unsigned flags,\n \t\t\tint (*for_each)(struct ref_store *, each_ref_fn, void *))\n-- \n2.38.1\n\n"},{"id":"466651","messageId":"3741f0a389ec28e683722547b2e898db08763424.1667823042.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667823042.git.ps@pks.im","subject":"[PATCH v3 1/6] refs: get rid of global list of hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T12:16:23Z","receivedAt":"2022-11-07T12:17:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to add a new argument to git-rev-list(1) that allows it to\nadd all references that are visible when taking `transfer.hideRefs` et\nal into account. This will require us to potentially parse multiple sets\nof hidden refs, which is not easily possible right now as there is only\na single, global instance of the list of parsed hidden refs.\n\nRefactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both\ntake the list of hidden references as input and adjust callers to keep a\nlocal list, instead. This allows us to easily use multiple hidden-ref\nlists. Furthermore, it allows us to properly free this list before we\nexit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 8 +++++---\n ls-refs.c | 13 +++++++++----\n refs.c | 14 ++++----------\n refs.h | 5 +++--\n upload-pack.c | 30 ++++++++++++++++++------------\n 5 files changed, 39 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 44bcea3a5b..1f3efc58fb 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)\n \n static int receive_pack_config(const char *var, const char *value, void *cb)\n {\n-\tint status = parse_hide_refs_config(var, value, \"receive\");\n+\tint status = parse_hide_refs_config(var, value, \"receive\", &hidden_refs);\n \n \tif (status)\n \t\treturn status;\n@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,\n \tstruct oidset *seen = data;\n \tconst char *path = strip_namespace(path_full);\n \n-\tif (ref_is_hidden(path, path_full))\n+\tif (ref_is_hidden(path, path_full, &hidden_refs))\n \t\treturn 0;\n \n \t/*\n@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)\n \t\tstrbuf_setlen(&refname_full, prefix_len);\n \t\tstrbuf_addstr(&refname_full, cmd->ref_name);\n \n-\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf))\n+\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))\n \t\t\tcontinue;\n \t\tif (is_null_oid(&cmd->new_oid))\n \t\t\tcmd->error_string = \"deny deleting a hidden ref\";\n@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n+\tstring_list_clear(&hidden_refs, 1);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex fa0d01b47c..ae89f850e9 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -6,6 +6,7 @@\n #include \"ls-refs.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"string-list.h\"\n \n static int config_read;\n static int advertise_unborn;\n@@ -73,6 +74,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n+\tstruct string_list hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \n \tstrbuf_reset(&data->buf);\n \n-\tif (ref_is_hidden(refname_nons, refname))\n+\tif (ref_is_hidden(refname_nons, refname, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (!ref_match(&data->prefixes, refname_nons))\n@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)\n }\n \n static int ls_refs_config(const char *var, const char *value,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *cb_data)\n {\n+\tstruct ls_refs_data *data = cb_data;\n \t/*\n \t * We only serve fetches over v2 for now, so respect only \"uploadpack\"\n \t * config. This may need to eventually be expanded to \"receive\", but we\n \t * don't yet know how that information will be passed to ls-refs.\n \t */\n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n int ls_refs(struct repository *r, struct packet_reader *request)\n@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n+\tstring_list_init_dup(&data.hidden_refs);\n \n \tensure_config_read();\n-\tgit_config(ls_refs_config, NULL);\n+\tgit_config(ls_refs_config, &data);\n \n \twhile (packet_reader_read(request) == PACKET_READ_NORMAL) {\n \t\tconst char *arg = request->line;\n@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n+\tstring_list_clear(&data.hidden_refs, 1);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 1491ae937e..f1711e2e9f 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n \t\t\t\t\t refname, strict);\n }\n \n-static struct string_list *hide_refs;\n-\n-int parse_hide_refs_config(const char *var, const char *value, const char *section)\n+int parse_hide_refs_config(const char *var, const char *value, const char *section,\n+\t\t\t struct string_list *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tif (!hide_refs) {\n-\t\t\tCALLOC_ARRAY(hide_refs, 1);\n-\t\t\thide_refs->strdup_strings = 1;\n-\t\t}\n \t\tstring_list_append(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n-int ref_is_hidden(const char *refname, const char *refname_full)\n+int ref_is_hidden(const char *refname, const char *refname_full,\n+\t\t const struct string_list *hide_refs)\n {\n \tint i;\n \n-\tif (!hide_refs)\n-\t\treturn 0;\n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n \t\tconst char *match = hide_refs->items[i].string;\n \t\tconst char *subject;\ndiff --git a/refs.h b/refs.h\nindex 8958717a17..3266fd8f57 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,\n \t const struct object_id *new_oid, const struct object_id *old_oid,\n \t unsigned int flags, enum action_on_err onerr);\n \n-int parse_hide_refs_config(const char *var, const char *value, const char *);\n+int parse_hide_refs_config(const char *var, const char *value, const char *,\n+\t\t\t struct string_list *);\n \n /*\n * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);\n * the ref is outside that namespace, the first parameter is NULL. The second\n * parameter always points to the full ref name.\n */\n-int ref_is_hidden(const char *, const char *);\n+int ref_is_hidden(const char *, const char *, const struct string_list *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0b8311bd68..9db17f8787 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -62,6 +62,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n+\tstruct string_list hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n+\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n \tmemset(data, 0, sizeof(*data));\n \tdata->symref = symref;\n \tdata->wanted_refs = wanted_refs;\n+\tdata->hidden_refs = hidden_refs;\n \tdata->want_obj = want_obj;\n \tdata->have_obj = have_obj;\n \tdata->haves = haves;\n@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n+\tstring_list_clear(&data->hidden_refs, 1);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * Checking for reachable shallows requires that our refs be\n \t\t * marked with OUR_REF.\n \t\t */\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, data);\n+\t\tfor_each_namespaced_ref(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid)\n+\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n-\tif (ref_is_hidden(refname, refname_full)) {\n+\tif (ref_is_hidden(refname, refname_full, hidden_refs)) {\n \t\to->flags |= HIDDEN_REF;\n \t\treturn 1;\n \t}\n@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,\n }\n \n static int check_ref(const char *refname_full, const struct object_id *oid,\n-\t\t int flag UNUSED, void *cb_data UNUSED)\n+\t\t int flag UNUSED, void *cb_data)\n {\n \tconst char *refname = strip_namespace(refname_full);\n+\tstruct upload_pack_data *data = cb_data;\n \n-\tmark_our_ref(refname, refname_full, oid);\n+\tmark_our_ref(refname, refname_full, oid, &data->hidden_refs);\n \treturn 0;\n }\n \n@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \tstruct object_id peeled;\n \tstruct upload_pack_data *data = cb_data;\n \n-\tif (mark_our_ref(refname_nons, refname, oid))\n+\tif (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (capabilities) {\n@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)\n \tif (parse_object_filter_config(var, value, data) < 0)\n \t\treturn -1;\n \n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)\n@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, &data);\n+\t\tfor_each_namespaced_ref(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t struct string_list *wanted_refs,\n+\t\t\t struct string_list *hidden_refs,\n \t\t\t struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\tstruct strbuf refname = STRBUF_INIT;\n \n \t\tstrbuf_addf(&refname, \"%s%s\", get_git_namespace(), refname_nons);\n-\t\tif (ref_is_hidden(refname_nons, refname.buf) ||\n+\t\tif (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||\n \t\t read_ref(refname.buf, &oid)) {\n \t\t\tpacket_writer_error(writer, \"unknown ref %s\", refname_nons);\n \t\t\tdie(\"unknown ref %s\", refname_nons);\n@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (data->allow_ref_in_want &&\n \t\t parse_want_ref(&data->writer, arg, &data->wanted_refs,\n-\t\t\t\t &data->want_obj))\n+\t\t\t\t &data->hidden_refs, &data->want_obj))\n \t\t\tcontinue;\n \t\t/* process have line */\n \t\tif (parse_have(arg, &data->haves))\n-- \n2.38.1\n\n"},{"id":"466650","messageId":"9d154495593606532ecec62f29c9c34dc2da8917.1667823042.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667823042.git.ps@pks.im","subject":"[PATCH v3 6/6] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T12:16:43Z","receivedAt":"2022-11-07T12:17:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When serving a push, git-receive-pack(1) needs to verify that the\npackfile sent by the client contains all objects that are required by\nthe updated references. This connectivity check works by marking all\npreexisting references as uninteresting and using the new reference tips\nas starting point for a graph walk.\n\nMarking all preexisting references as uninteresting can be a problem\nwhen it comes to performance. Git forges tend to do internal bookkeeping\nto keep alive sets of objects for internal use or make them easy to find\nvia certain references. These references are typically hidden away from\nthe user so that they are neither advertised nor writeable. At GitLab,\nwe have one particular repository that contains a total of 7 million\nreferences, of which 6.8 million are indeed internal references. With\nthe current connectivity check we are forced to load all these\nreferences in order to mark them as uninteresting, and this alone takes\naround 15 seconds to compute.\n\nWe can optimize this by only taking into account the set of visible refs\nwhen marking objects as uninteresting. This means that we may now walk\nmore objects until we hit any object that is marked as uninteresting.\nBut it is rather unlikely that clients send objects that make large\nparts of objects reachable that have previously only ever been hidden,\nwhereas the common case is to push incremental changes that build on top\nof the visible object graph.\n\nThis provides a huge boost to performance in the mentioned repository,\nwhere the vast majority of its refs hidden. Pushing a new commit into\nthis repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\nas it is configured in Gitaly leads to a 4.5-fold speedup:\n\n Benchmark 1: main\n Time (mean ± σ): 30.977 s ± 0.157 s [User: 30.226 s, System: 1.083 s]\n Range (min … max): 30.796 s … 31.071 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 6.799 s ± 0.063 s [User: 6.803 s, System: 0.354 s]\n Range (min … max): 6.729 s … 6.850 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 4.56 ± 0.05 times faster than 'main'\n\nAs we mostly go through the same codepaths even in the case where there\nare no hidden refs at all compared to the code before there is no change\nin performance when no refs are hidden:\n\n Benchmark 1: main\n Time (mean ± σ): 48.188 s ± 0.432 s [User: 49.326 s, System: 5.009 s]\n Range (min … max): 47.706 s … 48.539 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 48.027 s ± 0.500 s [User: 48.934 s, System: 5.025 s]\n Range (min … max): 47.504 s … 48.500 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 1.00 ± 0.01 times faster than 'main'\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 2 ++\n connected.c | 3 +++\n connected.h | 7 +++++++\n 3 files changed, 12 insertions(+)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1f3efc58fb..77ab40f123 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,\n \topt.err_fd = err_fd;\n \topt.progress = err_fd && !quiet;\n \topt.env = tmp_objdir_env(tmp_objdir);\n+\topt.exclude_hidden_refs_section = \"receive\";\n+\n \tif (check_connected(iterate_receive_command_list, &data, &opt))\n \t\tset_connectivity_errors(commands, si);\n \ndiff --git a/connected.c b/connected.c\nindex 74a20cb32e..4f6388eed7 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n \tif (!opt->is_deepening_fetch) {\n \t\tstrvec_push(&rev_list.args, \"--not\");\n+\t\tif (opt->exclude_hidden_refs_section)\n+\t\t\tstrvec_pushf(&rev_list.args, \"--exclude-hidden=%s\",\n+\t\t\t\t opt->exclude_hidden_refs_section);\n \t\tstrvec_push(&rev_list.args, \"--all\");\n \t}\n \tstrvec_push(&rev_list.args, \"--quiet\");\ndiff --git a/connected.h b/connected.h\nindex 6e59c92aa3..16b2c84f2e 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -46,6 +46,13 @@ struct check_connected_options {\n \t * during a fetch.\n \t */\n \tunsigned is_deepening_fetch : 1;\n+\n+\t/*\n+\t * If not NULL, use `--exclude-hidden=$section` to exclude all refs\n+\t * hidden via the `$section.hideRefs` config from the set of\n+\t * already-reachable refs.\n+\t */\n+\tconst char *exclude_hidden_refs_section;\n };\n \n #define CHECK_CONNECTED_INIT { 0 }\n-- \n2.38.1\n\n"},{"id":"466652","messageId":"2a6a67df1d470bf790025d55095c237ddc6a6bd6.1667823042.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667823042.git.ps@pks.im","subject":"[PATCH v3 3/6] revision: introduce struct to handle exclusions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T12:16:31Z","receivedAt":"2022-11-07T12:17:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions that handle exclusion of refs work on a single string\nlist. We're about to add a second mechanism for excluding refs though,\nand it makes sense to reuse much of the same architecture for both kinds\nof exclusion.\n\nIntroduce a new `struct ref_exclusions` that encapsulates all the logic\nrelated to excluding refs.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/rev-parse.c | 8 ++++----\n revision.c | 47 ++++++++++++++++++++-------------------------\n revision.h | 22 +++++++++++++++------\n 3 files changed, 41 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 8f61050bde..7fa5b6991b 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -39,7 +39,7 @@ static int abbrev_ref_strict;\n static int output_sq;\n \n static int stuck_long;\n-static struct string_list *ref_excludes;\n+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;\n \n /*\n * Some arguments are relevant \"revision\" arguments,\n@@ -198,7 +198,7 @@ static int show_default(void)\n static int show_reference(const char *refname, const struct object_id *oid,\n \t\t\t int flag UNUSED, void *cb_data UNUSED)\n {\n-\tif (ref_excluded(ref_excludes, refname))\n+\tif (ref_excluded(&ref_excludes, refname))\n \t\treturn 0;\n \tshow_rev(NORMAL, oid, refname);\n \treturn 0;\n@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)\n \t\tfor_each_glob_ref_in(show_reference, pattern, prefix, NULL);\n \telse\n \t\tfor_each_ref_in(prefix, show_reference, NULL);\n-\tclear_ref_exclusion(&ref_excludes);\n+\tclear_ref_exclusions(&ref_excludes);\n }\n \n enum format_type {\n@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--all\")) {\n \t\t\t\tfor_each_ref(show_reference, NULL);\n-\t\t\t\tclear_ref_exclusion(&ref_excludes);\n+\t\t\t\tclear_ref_exclusions(&ref_excludes);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (skip_prefix(arg, \"--disambiguate=\", &arg)) {\ndiff --git a/revision.c b/revision.c\nindex be755670e2..e5eaaa24ba 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,35 +1517,29 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-int ref_excluded(struct string_list *ref_excludes, const char *path)\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n \tstruct string_list_item *item;\n-\n-\tif (!ref_excludes)\n-\t\treturn 0;\n-\tfor_each_string_list_item(item, ref_excludes) {\n+\tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+void init_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n+\tstring_list_init_dup(&exclusions->excluded_refs);\n }\n \n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n+\tstring_list_clear(&exclusions->excluded_refs, 0);\n+}\n+\n+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n+{\n+\tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n struct all_refs_cb {\n@@ -1563,7 +1557,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \tstruct all_refs_cb *cb = cb_data;\n \tstruct object *object;\n \n-\tif (ref_excluded(cb->all_revs->ref_excludes, path))\n+\tif (ref_excluded(&cb->all_revs->ref_excludes, path))\n \t return 0;\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n@@ -1901,6 +1895,7 @@ void repo_init_revisions(struct repository *r,\n \n \tinit_display_notes(&revs->notes_opt);\n \tlist_objects_filter_init(&revs->filter);\n+\tinit_ref_exclusions(&revs->ref_excludes);\n }\n \n static void add_pending_commit_list(struct rev_info *revs,\n@@ -2689,10 +2684,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\t\tother_head_refs(handle_one_ref, &cb);\n \t\t}\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--branches\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--bisect\")) {\n \t\tread_bisect_terms(&term_bad, &term_good);\n \t\thandle_refs(refs, revs, *flags, for_each_bad_bisect_ref);\n@@ -2701,15 +2696,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->bisect = 1;\n \t} else if (!strcmp(arg, \"--tags\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--remotes\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref(handle_one_ref, optarg, &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t\treturn argcount;\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n@@ -2718,17 +2713,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--reflog\")) {\n \t\tadd_reflogs_to_pending(revs, *flags);\n \t} else if (!strcmp(arg, \"--indexed-objects\")) {\ndiff --git a/revision.h b/revision.h\nindex afe1b77985..87d6824c55 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -81,6 +81,14 @@ struct rev_cmdline_info {\n \t} *rev;\n };\n \n+struct ref_exclusions {\n+\t/*\n+\t * Excluded refs is a list of wildmatch patterns. If any of the\n+\t * patterns matches, the reference will be excluded.\n+\t */\n+\tstruct string_list excluded_refs;\n+};\n+\n struct oidset;\n struct topo_walk_info;\n \n@@ -103,7 +111,7 @@ struct rev_info {\n \tstruct list_objects_filter_options filter;\n \n \t/* excluding from --branches, --refs, etc. expansion */\n-\tstruct string_list *ref_excludes;\n+\tstruct ref_exclusions ref_excludes;\n \n \t/* Basic information */\n \tconst char *prefix;\n@@ -439,12 +447,14 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)\n void show_object_with_name(FILE *, struct object *, const char *);\n \n /**\n- * Helpers to check if a \"struct string_list\" item matches with\n- * wildmatch().\n+ * Helpers to check if a reference should be excluded.\n */\n-int ref_excluded(struct string_list *, const char *path);\n-void clear_ref_exclusion(struct string_list **);\n-void add_ref_exclusion(struct string_list **, const char *exclude);\n+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }\n+\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n+void init_ref_exclusions(struct ref_exclusions *);\n+void clear_ref_exclusions(struct ref_exclusions *);\n+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n \n /**\n * This function can be used if you want to add commit objects as revision\n-- \n2.38.1\n\n"},{"id":"466653","messageId":"68a5e563045f1b3e6db7304206d95ebebcacbe6f.1667823042.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667823042.git.ps@pks.im","subject":"[PATCH v3 5/6] revparse: add `--exclude-hidden=` option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T12:16:39Z","receivedAt":"2022-11-07T12:17:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new `--exclude-hidden=` option that is similar to the one we just\nadded to git-rev-list(1). Given a seciton name `transfer`, `uploadpack`\nor `receive` as argument, it causes us to exclude all references that\nwould be hidden by the respective `$seciton.hideRefs` configuration.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/git-rev-parse.txt | 7 +++++++\n builtin/rev-parse.c | 4 ++++\n t/t6018-rev-list-glob.sh | 8 ++++++++\n 3 files changed, 19 insertions(+)\n\ndiff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt\nindex 6b8ca085aa..a016cb5abe 100644\n--- a/Documentation/git-rev-parse.txt\n+++ b/Documentation/git-rev-parse.txt\n@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[transfer|receive|uploadpack]::\n+\tDo not include refs that have been hidden via either one of\n+\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n+\tthe next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would\n+\totherwise consider. This option is cleared when seeing one of these\n+\tpseudo-refs.\n+\n --disambiguate=::\n \tShow every object whose name begins with the given prefix.\n \tThe must be at least 4 hexadecimal digits long to\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 7fa5b6991b..49730c7a23 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -895,6 +895,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tadd_ref_exclusion(&ref_excludes, arg);\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (skip_prefix(arg, \"--exclude-hidden=\", &arg)) {\n+\t\t\t\texclude_hidden_refs(&ref_excludes, arg);\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-toplevel\")) {\n \t\t\t\tconst char *work_tree = get_git_work_tree();\n \t\t\t\tif (work_tree)\ndiff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh\nindex e1abc5c2b3..f92616de12 100755\n--- a/t/t6018-rev-list-glob.sh\n+++ b/t/t6018-rev-list-glob.sh\n@@ -187,6 +187,14 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '\n \tcompare rev-parse \"--exclude=upstream/x --remotes=upstream/*\" \"upstream/one upstream/two\"\n '\n \n+test_expect_success 'rev-parse --exclude-hidden= with --all' '\n+\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--exclude-hidden=transfer --all\" \"--branches --tags\"\n+'\n+\n+test_expect_success 'rev-parse --exclude-hidden= with --all' '\n+\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude-hidden=transfer --all\" \"--exclude=refs/heads/subspace/* --all\"\n+'\n+\n test_expect_success 'rev-list --exclude=glob with --branches=glob' '\n \tcompare rev-list \"--exclude=subspace-* --branches=sub*\" \"subspace/one subspace/two\"\n '\n-- \n2.38.1\n\n"},{"id":"466654","messageId":"de7c1aa210c2df9bdbbb6c19f44f72c37f56c5da.1667823042.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667823042.git.ps@pks.im","subject":"[PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-07T12:16:35Z","receivedAt":"2022-11-07T12:17:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Users can optionally hide refs from remote users in git-upload-pack(1),\ngit-receive-pack(1) and others via the `transfer.hideRefs`, but there is\nnot an easy way to obtain the list of all visible or hidden refs right\nnow. We'll require just that though for a performance improvement in our\nconnectivity check.\n\nAdd a new option `--exclude-hidden=` that excludes any hidden refs from\nthe next pseudo-ref like `--all` or `--branches`.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 7 ++\n builtin/rev-list.c | 1 +\n revision.c | 43 +++++++-\n revision.h | 9 +-\n t/t6021-rev-list-exclude-hidden.sh | 159 +++++++++++++++++++++++++++++\n 5 files changed, 217 insertions(+), 2 deletions(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 1837509566..a178956613 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[transfer|receive|uploadpack]::\n+\tDo not include refs that have been hidden via either one of\n+\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n+\tthe next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would\n+\totherwise consider. This option is cleared when seeing one of these\n+\tpseudo-refs.\n+\n --reflog::\n \tPretend as if all objects mentioned by reflogs are listed on the\n \tcommand line as ``.\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3acd93f71e..9eace06385 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -38,6 +38,7 @@ static const char rev_list_usage[] =\n \" --tags\\n\"\n \" --remotes\\n\"\n \" --stdin\\n\"\n+\" --exclude-hidden=[transfer|receive|uploadpack]\\n\"\n \" --quiet\\n\"\n \" ordering output:\\n\"\n \" --topo-order\\n\"\ndiff --git a/revision.c b/revision.c\nindex e5eaaa24ba..45652f9b0b 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"object-store.h\"\n #include \"tag.h\"\n #include \"blob.h\"\n@@ -1519,22 +1520,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \n int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n+\tconst char *stripped_path = strip_namespace(path);\n \tstruct string_list_item *item;\n+\n \tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n+\n+\tif (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))\n+\t\treturn 1;\n+\n \treturn 0;\n }\n \n void init_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_init_dup(&exclusions->excluded_refs);\n+\tstring_list_init_dup(&exclusions->hidden_refs);\n }\n \n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n+\tstring_list_clear(&exclusions->hidden_refs, 1);\n }\n \n void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n@@ -1542,6 +1551,35 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n \tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n+struct exclude_hidden_refs_cb {\n+\tstruct ref_exclusions *exclusions;\n+\tconst char *section;\n+};\n+\n+static int hide_refs_config(const char *var, const char *value, void *cb_data)\n+{\n+\tstruct exclude_hidden_refs_cb *cb = cb_data;\n+\treturn parse_hide_refs_config(var, value, cb->section,\n+\t\t\t\t &cb->exclusions->hidden_refs);\n+}\n+\n+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)\n+{\n+\tstruct exclude_hidden_refs_cb cb;\n+\n+\tif (strcmp(section, \"transfer\") && strcmp(section, \"receive\") &&\n+\t strcmp(section, \"uploadpack\"))\n+\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n+\n+\tif (exclusions->hidden_refs.nr)\n+\t\tdie(_(\"--exclude-hidden= passed more than once\"));\n+\n+\tcb.exclusions = exclusions;\n+\tcb.section = section;\n+\n+\tgit_config(hide_refs_config, &cb);\n+}\n+\n struct all_refs_cb {\n \tint all_flags;\n \tint warned_bad_reflog;\n@@ -2220,7 +2258,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t !strcmp(arg, \"--bisect\") || starts_with(arg, \"--glob=\") ||\n \t !strcmp(arg, \"--indexed-objects\") ||\n \t !strcmp(arg, \"--alternate-refs\") ||\n-\t starts_with(arg, \"--exclude=\") ||\n+\t starts_with(arg, \"--exclude=\") || starts_with(arg, \"--exclude-hidden=\") ||\n \t starts_with(arg, \"--branches=\") || starts_with(arg, \"--tags=\") ||\n \t starts_with(arg, \"--remotes=\") || starts_with(arg, \"--no-walk=\"))\n \t{\n@@ -2709,6 +2747,9 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n \t\treturn argcount;\n+\t} else if ((argcount = parse_long_opt(\"exclude-hidden\", argv, &optarg))) {\n+\t\texclude_hidden_refs(&revs->ref_excludes, optarg);\n+\t\treturn argcount;\n \t} else if (skip_prefix(arg, \"--branches=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\ndiff --git a/revision.h b/revision.h\nindex 87d6824c55..fef5e063d1 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -87,6 +87,12 @@ struct ref_exclusions {\n \t * patterns matches, the reference will be excluded.\n \t */\n \tstruct string_list excluded_refs;\n+\n+\t/*\n+\t * Hidden refs is a list of patterns that is to be hidden via\n+\t * `ref_is_hidden()`.\n+\t */\n+\tstruct string_list hidden_refs;\n };\n \n struct oidset;\n@@ -449,12 +455,13 @@ void show_object_with_name(FILE *, struct object *, const char *);\n /**\n * Helpers to check if a reference should be excluded.\n */\n-#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }\n+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }\n \n int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n void init_ref_exclusions(struct ref_exclusions *);\n void clear_ref_exclusions(struct ref_exclusions *);\n void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n+void exclude_hidden_refs(struct ref_exclusions *, const char *section);\n \n /**\n * This function can be used if you want to add commit objects as revision\ndiff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh\nnew file mode 100755\nindex 0000000000..d08fc2da93\n--- /dev/null\n+++ b/t/t6021-rev-list-exclude-hidden.sh\n@@ -0,0 +1,159 @@\n+#!/bin/sh\n+\n+test_description='git rev-list --exclude-hidden test'\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit_bulk --id=commit --ref=refs/heads/main 1 &&\n+\tCOMMIT=$(git rev-parse refs/heads/main) &&\n+\ttest_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&\n+\tTAG=$(git rev-parse refs/tags/lightweight) &&\n+\ttest_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&\n+\tHIDDEN=$(git rev-parse refs/hidden/commit) &&\n+\ttest_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&\n+\tNAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)\n+'\n+\n+test_expect_success 'invalid section' '\n+\techo \"fatal: unsupported section for hidden refs: unsupported\" >expected &&\n+\ttest_must_fail git rev-list --exclude-hidden=unsupported 2>err &&\n+\ttest_cmp expected err\n+'\n+\n+test_expect_success 'passed multiple times' '\n+\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n+\ttest_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --exclude-hidden=transfer 2>err &&\n+\ttest_cmp expected err\n+'\n+\n+test_expect_success '--exclude-hidden without hiddenRefs' '\n+\tgit rev-list --exclude-hidden=transfer --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$NAMESPACE\n+\t$HIDDEN\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success 'hidden via transfer.hideRefs' '\n+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$NAMESPACE\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--all --exclude-hidden=transfer --not --all without hidden refs' '\n+\tgit rev-list --all --exclude-hidden=transfer --not --all >out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success '--all --exclude-hidden=transfer --not --all with hidden ref' '\n+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=transfer --not --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$HIDDEN\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--exclude-hidden with --exclude' '\n+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=transfer --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$NAMESPACE\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--exclude-hidden is reset' '\n+\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=transfer --all --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$NAMESPACE\n+\t$HIDDEN\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--exclude-hidden operates on stripped refs by default' '\n+\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=transfer --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$HIDDEN\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--exclude-hidden does not hide namespace by default' '\n+\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$NAMESPACE\n+\t$HIDDEN\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+test_expect_success '--exclude-hidden= may operate on unstripped refs' '\n+\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&\n+\tcat >expected <<-EOF &&\n+\t$HIDDEN\n+\t$TAG\n+\t$COMMIT\n+\tEOF\n+\ttest_cmp expected out\n+'\n+\n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"hidden via $section.hideRefs\" '\n+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"--exclude-hidden=$section respects transfer.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"--exclude-hidden=transfer ignores $section.hideRefs\" '\n+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"--exclude-hidden=$section respects both transfer.hideRefs and $section.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+done\n+\n+test_done\n-- \n2.38.1\n\n"},{"id":"466656","messageId":"221107.86tu3ax5b6.gmgdl@evledraar.gmail.com","threadId":"58706","inReplyTo":"2a6a67df1d470bf790025d55095c237ddc6a6bd6.1667823042.git.ps@pks.im","subject":"Re: [PATCH v3 3/6] revision: introduce struct to handle exclusions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-07T12:51:51Z","receivedAt":"2022-11-07T13:28:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 07 2022, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> The functions that handle exclusion of refs work on a single string\n> list. We're about to add a second mechanism for excluding refs though,\n> and it makes sense to reuse much of the same architecture for both kinds\n> of exclusion.\n>\n> Introduce a new `struct ref_exclusions` that encapsulates all the logic\n> related to excluding refs.\n\nI think it's a good change, but probably worth mentioning tha we're\nmoving the \"excluded_refs\" from being malloc'd to the \"struct\nstring_list\" being embedded in this new \"struct ref_exclusions\".\n\nThat change isn't necessary for hoisting it into a container struct, but\ndoes make things nicer down the line.\n\n> \tstruct string_list_item *item;\n> -\n> -\tif (!ref_excludes)\n> -\t\treturn 0;\n> -\tfor_each_string_list_item(item, ref_excludes) {\n> +\tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n> \t\tif (!wildmatch(item->string, path, 0))\n> \t\t\treturn 1;\n\nE.g. because here we don't care about the distinction between NULL and\n!list->nr anymore, it *does* matter in some cases, but it's always nice\nto be able to clearly distinguish the cases where we don't, such as this\none....\n\n> -void clear_ref_exclusion(struct string_list **ref_excludes_p)\n> +void init_ref_exclusions(struct ref_exclusions *exclusions)\n> {\n> -\tif (*ref_excludes_p) {\n> -\t\tstring_list_clear(*ref_excludes_p, 0);\n> -\t\tfree(*ref_excludes_p);\n> -\t}\n> -\t*ref_excludes_p = NULL;\n\n...and this becomes much nicer.\n\nAside: There's some churn, and this diff is worse for the\nrename-while-at-it of \"clear_ref_exclusion\" to \"add_ref_exclusion\", but\nthat's probably worth it to have the macro match the struct name etc.\n\n> +\tstring_list_init_dup(&exclusions->excluded_refs);\n\nOkey, so this is partly my fault for not following up on f196c1e908d\n(revisions API users: use release_revisions() needing REV_INFO_INIT,\n2022-04-13) :); But here:\n\nIf we keep this *_init() function don't duplicate what you're adding to\nthe macro, just init this in terms of the macro. See the two-line\nexamples in:\n\n\tgit grep -W memcpy.*blank\n\nBut here (and this is the part that's mostly me) as we don't malloc this\nanymore you're only needing to keep this init function for\nrepo_init_revisions().\n\nSo, probably too big a digression for a \"while at it\", but FWIW this on\ntop of your topic would do:\n\t\n\t revision.c | 10 ++--------\n\t revision.h | 10 +++++++---\n\t 2 files changed, 9 insertions(+), 11 deletions(-)\n\t\n\tdiff --git a/revision.c b/revision.c\n\tindex 45652f9b0bb..cf352d1fa43 100644\n\t--- a/revision.c\n\t+++ b/revision.c\n\t@@ -1534,12 +1534,6 @@ int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n\t \treturn 0;\n\t }\n\t \n\t-void init_ref_exclusions(struct ref_exclusions *exclusions)\n\t-{\n\t-\tstring_list_init_dup(&exclusions->excluded_refs);\n\t-\tstring_list_init_dup(&exclusions->hidden_refs);\n\t-}\n\t-\n\t void clear_ref_exclusions(struct ref_exclusions *exclusions)\n\t {\n\t \tstring_list_clear(&exclusions->excluded_refs, 0);\n\t@@ -1897,7 +1891,8 @@ void repo_init_revisions(struct repository *r,\n\t \t\t\t struct rev_info *revs,\n\t \t\t\t const char *prefix)\n\t {\n\t-\tmemset(revs, 0, sizeof(*revs));\n\t+\tstruct rev_info blank = REV_INFO_INIT;\n\t+\tmemcpy(revs, &blank, sizeof(*revs));\n\t \n\t \trevs->repo = r;\n\t \trevs->abbrev = DEFAULT_ABBREV;\n\t@@ -1933,7 +1928,6 @@ void repo_init_revisions(struct repository *r,\n\t \n\t \tinit_display_notes(&revs->notes_opt);\n\t \tlist_objects_filter_init(&revs->filter);\n\t-\tinit_ref_exclusions(&revs->ref_excludes);\n\t }\n\t \n\t static void add_pending_commit_list(struct rev_info *revs,\n\tdiff --git a/revision.h b/revision.h\n\tindex fef5e063d16..75b8ecc307b 100644\n\t--- a/revision.h\n\t+++ b/revision.h\n\t@@ -94,6 +94,10 @@ struct ref_exclusions {\n\t \t */\n\t \tstruct string_list hidden_refs;\n\t };\n\t+#define REF_EXCLUSIONS_INIT { \\\n\t+\t.excluded_refs = STRING_LIST_INIT_DUP, \\\n\t+\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n\t+}\n\t \n\t struct oidset;\n\t struct topo_walk_info;\n\t@@ -371,7 +375,9 @@ struct rev_info {\n\t * called before release_revisions() the \"struct rev_info\" can be left\n\t * uninitialized.\n\t */\n\t-#define REV_INFO_INIT { 0 }\n\t+#define REV_INFO_INIT { \\\n\t+\t.ref_excludes = REF_EXCLUSIONS_INIT, \\\n\t+}\n\t \n\t /**\n\t * Initialize a rev_info structure with default values. The third parameter may\n\t@@ -455,10 +461,8 @@ void show_object_with_name(FILE *, struct object *, const char *);\n\t /**\n\t * Helpers to check if a reference should be excluded.\n\t */\n\t-#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }\n\t \n\t int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n\t-void init_ref_exclusions(struct ref_exclusions *);\n\t void clear_ref_exclusions(struct ref_exclusions *);\n\t void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n\t void exclude_hidden_refs(struct ref_exclusions *, const char *section);\n\nBut I'll submit that cleanup seperately, but for now let's not duplicate\nyour REF_EXCLUSIONS_INIT macro here in init_ref_exclusions(), just have\nthe function do what the macro is doing, now that we don't need the\nmalloc.\n\n> -void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n> +void clear_ref_exclusions(struct ref_exclusions *exclusions)\n> {\n> -\tif (!*ref_excludes_p) {\n> -\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n> -\t\t(*ref_excludes_p)->strdup_strings = 1;\n> -\t}\n> -\tstring_list_append(*ref_excludes_p, exclude);\n> +\tstring_list_clear(&exclusions->excluded_refs, 0);\n\nAlso nicer.\n\n> static void add_pending_commit_list(struct rev_info *revs,\n> @@ -2689,10 +2684,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n> \t\t\tinit_all_refs_cb(&cb, revs, *flags);\n> \t\t\tother_head_refs(handle_one_ref, &cb);\n> \t\t}\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \t} else if (!strcmp(arg, \"--branches\")) {\n> \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \t} else if (!strcmp(arg, \"--bisect\")) {\n> \t\tread_bisect_terms(&term_bad, &term_good);\n> \t\thandle_refs(refs, revs, *flags, for_each_bad_bisect_ref);\n> @@ -2701,15 +2696,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n> \t\trevs->bisect = 1;\n> \t} else if (!strcmp(arg, \"--tags\")) {\n> \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \t} else if (!strcmp(arg, \"--remotes\")) {\n> \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n> \t\tstruct all_refs_cb cb;\n> \t\tinit_all_refs_cb(&cb, revs, *flags);\n> \t\tfor_each_glob_ref(handle_one_ref, optarg, &cb);\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \t\treturn argcount;\n> \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n> \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n> @@ -2718,17 +2713,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n> \t\tstruct all_refs_cb cb;\n> \t\tinit_all_refs_cb(&cb, revs, *flags);\n> \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n> \t\tstruct all_refs_cb cb;\n> \t\tinit_all_refs_cb(&cb, revs, *flags);\n> \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n> \t\tstruct all_refs_cb cb;\n> \t\tinit_all_refs_cb(&cb, revs, *flags);\n> \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n> -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> +\t\tclear_ref_exclusions(&revs->ref_excludes);\n\nThe churn I mentioned with the renaming, so maybe worth doing that as a\n\"prep\" commit?\n\n> +struct ref_exclusions {\n> +\t/*\n> +\t * Excluded refs is a list of wildmatch patterns. If any of the\n> +\t * patterns matches, the reference will be excluded.\n> +\t */\n> +\tstruct string_list excluded_refs;\n> +};\n\nPer the above POC diff though, please move...\n\n> struct oidset;\n> struct topo_walk_info;\n> \n> @@ -103,7 +111,7 @@ struct rev_info {\n> \tstruct list_objects_filter_options filter;\n> \n> \t/* excluding from --branches, --refs, etc. expansion */\n> -\tstruct string_list *ref_excludes;\n> +\tstruct ref_exclusions ref_excludes;\n> \n> \t/* Basic information */\n> \tconst char *prefix;\n> @@ -439,12 +447,14 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)\n> void show_object_with_name(FILE *, struct object *, const char *);\n> \n> /**\n> - * Helpers to check if a \"struct string_list\" item matches with\n> - * wildmatch().\n> + * Helpers to check if a reference should be excluded.\n> */\n> -int ref_excluded(struct string_list *, const char *path);\n> -void clear_ref_exclusion(struct string_list **);\n> -void add_ref_exclusion(struct string_list **, const char *exclude);\n> +#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }\n\n...this macro to right after declaring the struct, which is what we\nusually do, and will help in adding it to \"REV_INFO_INIT\" sooner than\nlater.\n\nAlso, at the end of your series this end up being overly long, so per\nthe diff-above (which is tot he end of the series), let's start by\nline-wrapping it:\n\n\t#define ..._INIT { \\\n \t.member = ..._INIT, \\\n\t}\n"},{"id":"466657","messageId":"221107.86pmdyx4me.gmgdl@evledraar.gmail.com","threadId":"58706","inReplyTo":"de7c1aa210c2df9bdbbb6c19f44f72c37f56c5da.1667823042.git.ps@pks.im","subject":"Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-07T13:34:45Z","receivedAt":"2022-11-07T13:42:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 07 2022, Patrick Steinhardt wrote:\n\n> +--exclude-hidden=[transfer|receive|uploadpack]::\n> +\tDo not include refs that have been hidden via either one of\n> +\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n\nMaybe worth adding \"(see linkgit:git-config[1]) after listing the config\nvariables.\n\n> int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n> {\n> +\tconst char *stripped_path = strip_namespace(path);\n> \tstruct string_list_item *item;\n> +\n\nnit: stray whitespace in otherwise \"clean\" commit, but the post-image looks nicer, so...\n\n> void init_ref_exclusions(struct ref_exclusions *exclusions)\n> {\n> \tstring_list_init_dup(&exclusions->excluded_refs);\n> +\tstring_list_init_dup(&exclusions->hidden_refs);\n> }\n\nPer my comment on 3/6 we wouldn't need this when using the macro as a\nsource of truth.\n\n> void clear_ref_exclusions(struct ref_exclusions *exclusions)\n> {\n> \tstring_list_clear(&exclusions->excluded_refs, 0);\n> +\tstring_list_clear(&exclusions->hidden_refs, 1);\n> }\n\nHrm, I'l read on, but I don't see any use of \"util\" here at a glance,\nshould the \"1\" here be \"0\", or maybe I've just missed how it's used...\n\n> +\tif (strcmp(section, \"transfer\") && strcmp(section, \"receive\") &&\n> +\t strcmp(section, \"uploadpack\"))\n> +\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n> +\n> +\tif (exclusions->hidden_refs.nr)\n> +\t\tdie(_(\"--exclude-hidden= passed more than once\"));\n\nWe usually just ignore the first of --foo=bar --foo=baz and take \"baz\"\nin our CLI use. Is it better to die here than just clear the previous\none & continue?\n\n\n> -#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }\n> +#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }\n\n...the getting overly long line I mentioned in 3/6...\n\n> +TEST_PASSES_SANITIZE_LEAK=true\n\nThanks for adding this! :)\n"},{"id":"466678","messageId":"221107.86eduevgk3.gmgdl@evledraar.gmail.com","threadId":"58706","inReplyTo":"221107.86pmdyx4me.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-07T17:07:01Z","receivedAt":"2022-11-07T17:08:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 07 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> On Mon, Nov 07 2022, Patrick Steinhardt wrote:\n\n>> +TEST_PASSES_SANITIZE_LEAK=true\n>\n> Thanks for adding this! :)\n\nHrm, I spoke too soon :) This series adds new leaks, so it'll fail with\nthe linux-leaks job. I have the following local monkeypatch on top,\nwhich obviously doesn't address the root cause. The t6018 leak is new\ndue to the new tests you added.\n\ndiff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh\nindex f92616de12d..54221588dd0 100755\n--- a/t/t6018-rev-list-glob.sh\n+++ b/t/t6018-rev-list-glob.sh\n@@ -5,7 +5,6 @@ test_description='rev-list/rev-parse --glob'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n-TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n commit () {\ndiff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh\nindex d08fc2da93d..908b9dba611 100755\n--- a/t/t6021-rev-list-exclude-hidden.sh\n+++ b/t/t6021-rev-list-exclude-hidden.sh\n@@ -2,7 +2,6 @@\n \n test_description='git rev-list --exclude-hidden test'\n \n-TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n"},{"id":"466775","messageId":"Y2mpefFJspp4QnAz@nand.local","threadId":"58706","inReplyTo":"de7c1aa210c2df9bdbbb6c19f44f72c37f56c5da.1667823042.git.ps@pks.im","subject":"Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-08T00:57:29Z","receivedAt":"2022-11-08T00:57:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Nov 07, 2022 at 01:16:35PM +0100, Patrick Steinhardt wrote:\n> @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n> or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n> explicitly.\n>\n> +--exclude-hidden=[transfer|receive|uploadpack]::\n> +\tDo not include refs that have been hidden via either one of\n> +\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n> +\tthe next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would\n> +\totherwise consider. This option is cleared when seeing one of these\n> +\tpseudo-refs.\n> +\n\nHmm. I thought that part of the motivation behind this round was to drop\nthe 'transfer' group, since it's implied by '--exclude-hidden=receive\n--exclude-hidden-uploadpack', no?\n\nThanks,\nTaylor\n"},{"id":"466776","messageId":"Y2mp3MZNehiTBKbE@nand.local","threadId":"58706","inReplyTo":"cover.1667823042.git.ps@pks.im","subject":"Re: [PATCH v3 0/6] receive-pack: only use visible refs for connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-08T00:59:08Z","receivedAt":"2022-11-08T00:59:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Nov 07, 2022 at 01:16:19PM +0100, Patrick Steinhardt wrote:\n> Patrick Steinhardt (6):\n> refs: get rid of global list of hidden refs\n> revision: move together exclusion-related functions\n> revision: introduce struct to handle exclusions\n> revision: add new parameter to exclude hidden refs\n> revparse: add `--exclude-hidden=` option\n> receive-pack: only use visible refs for connectivity check\n\nThanks for the updated round. This version is looking pretty good,\nthough it looks like Ævar and I had a few minor recommendations. I am\ndefinitely puzzled by seeing '--exclude-hidden=transfer', though.\n\nThanks,\nTaylor\n"},{"id":"466793","messageId":"Y2oQb12WwEh0bpxW@ncase","threadId":"58706","inReplyTo":"Y2mpefFJspp4QnAz@nand.local","subject":"Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T08:16:47Z","receivedAt":"2022-11-08T08:17:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 07, 2022 at 07:57:29PM -0500, Taylor Blau wrote:\n> On Mon, Nov 07, 2022 at 01:16:35PM +0100, Patrick Steinhardt wrote:\n> > @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n> > or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n> > explicitly.\n> >\n> > +--exclude-hidden=[transfer|receive|uploadpack]::\n> > +\tDo not include refs that have been hidden via either one of\n> > +\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n> > +\tthe next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would\n> > +\totherwise consider. This option is cleared when seeing one of these\n> > +\tpseudo-refs.\n> > +\n> \n> Hmm. I thought that part of the motivation behind this round was to drop\n> the 'transfer' group, since it's implied by '--exclude-hidden=receive\n> --exclude-hidden-uploadpack', no?\n> \n> Thanks,\n> Taylor\n\nI didn't quite see the point in not providing the `transfer` group so\nthat users can ask for only the set of refs that are hidden by both\n`uploadpack` and `receive`. But given that you're the second person\nasking for it to be dropped now and given that I don't really have a\nplausible usecase for this I'll drop it in the next version.\n\nPatrick\n"},{"id":"466796","messageId":"Y2odS7H8lmom/8C3@ncase","threadId":"58706","inReplyTo":"221107.86tu3ax5b6.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 3/6] revision: introduce struct to handle exclusions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T09:11:39Z","receivedAt":"2022-11-08T09:11:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 07, 2022 at 01:51:51PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Nov 07 2022, Patrick Steinhardt wrote:\n[snip]\n> > +\tstring_list_init_dup(&exclusions->excluded_refs);\n> \n> Okey, so this is partly my fault for not following up on f196c1e908d\n> (revisions API users: use release_revisions() needing REV_INFO_INIT,\n> 2022-04-13) :); But here:\n> \n> If we keep this *_init() function don't duplicate what you're adding to\n> the macro, just init this in terms of the macro. See the two-line\n> examples in:\n> \n> \tgit grep -W memcpy.*blank\n\nMakes sense.\n\n> But here (and this is the part that's mostly me) as we don't malloc this\n> anymore you're only needing to keep this init function for\n> repo_init_revisions().\n> \n> So, probably too big a digression for a \"while at it\", but FWIW this on\n> top of your topic would do:\n> \t\n> \t revision.c | 10 ++--------\n> \t revision.h | 10 +++++++---\n> \t 2 files changed, 9 insertions(+), 11 deletions(-)\n> \t\n> \tdiff --git a/revision.c b/revision.c\n> \tindex 45652f9b0bb..cf352d1fa43 100644\n> \t--- a/revision.c\n> \t+++ b/revision.c\n> \t@@ -1534,12 +1534,6 @@ int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n> \t \treturn 0;\n> \t }\n> \t \n> \t-void init_ref_exclusions(struct ref_exclusions *exclusions)\n> \t-{\n> \t-\tstring_list_init_dup(&exclusions->excluded_refs);\n> \t-\tstring_list_init_dup(&exclusions->hidden_refs);\n> \t-}\n> \t-\n> \t void clear_ref_exclusions(struct ref_exclusions *exclusions)\n> \t {\n> \t \tstring_list_clear(&exclusions->excluded_refs, 0);\n> \t@@ -1897,7 +1891,8 @@ void repo_init_revisions(struct repository *r,\n> \t \t\t\t struct rev_info *revs,\n> \t \t\t\t const char *prefix)\n> \t {\n> \t-\tmemset(revs, 0, sizeof(*revs));\n> \t+\tstruct rev_info blank = REV_INFO_INIT;\n> \t+\tmemcpy(revs, &blank, sizeof(*revs));\n> \t \n> \t \trevs->repo = r;\n> \t \trevs->abbrev = DEFAULT_ABBREV;\n> \t@@ -1933,7 +1928,6 @@ void repo_init_revisions(struct repository *r,\n> \t \n> \t \tinit_display_notes(&revs->notes_opt);\n> \t \tlist_objects_filter_init(&revs->filter);\n> \t-\tinit_ref_exclusions(&revs->ref_excludes);\n> \t }\n> \t \n> \t static void add_pending_commit_list(struct rev_info *revs,\n> \tdiff --git a/revision.h b/revision.h\n> \tindex fef5e063d16..75b8ecc307b 100644\n> \t--- a/revision.h\n> \t+++ b/revision.h\n> \t@@ -94,6 +94,10 @@ struct ref_exclusions {\n> \t \t */\n> \t \tstruct string_list hidden_refs;\n> \t };\n> \t+#define REF_EXCLUSIONS_INIT { \\\n> \t+\t.excluded_refs = STRING_LIST_INIT_DUP, \\\n> \t+\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n> \t+}\n> \t \n> \t struct oidset;\n> \t struct topo_walk_info;\n> \t@@ -371,7 +375,9 @@ struct rev_info {\n> \t * called before release_revisions() the \"struct rev_info\" can be left\n> \t * uninitialized.\n> \t */\n> \t-#define REV_INFO_INIT { 0 }\n> \t+#define REV_INFO_INIT { \\\n> \t+\t.ref_excludes = REF_EXCLUSIONS_INIT, \\\n> \t+}\n> \t \n> \t /**\n> \t * Initialize a rev_info structure with default values. The third parameter may\n> \t@@ -455,10 +461,8 @@ void show_object_with_name(FILE *, struct object *, const char *);\n> \t /**\n> \t * Helpers to check if a reference should be excluded.\n> \t */\n> \t-#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }\n> \t \n> \t int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n> \t-void init_ref_exclusions(struct ref_exclusions *);\n> \t void clear_ref_exclusions(struct ref_exclusions *);\n> \t void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n> \t void exclude_hidden_refs(struct ref_exclusions *, const char *section);\n> \n> But I'll submit that cleanup seperately, but for now let's not duplicate\n> your REF_EXCLUSIONS_INIT macro here in init_ref_exclusions(), just have\n> the function do what the macro is doing, now that we don't need the\n> malloc.\n\nGreat, thanks.\n\n[snip]\n> > static void add_pending_commit_list(struct rev_info *revs,\n> > @@ -2689,10 +2684,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n> > \t\t\tinit_all_refs_cb(&cb, revs, *flags);\n> > \t\t\tother_head_refs(handle_one_ref, &cb);\n> > \t\t}\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> > \t} else if (!strcmp(arg, \"--branches\")) {\n> > \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> > \t} else if (!strcmp(arg, \"--bisect\")) {\n> > \t\tread_bisect_terms(&term_bad, &term_good);\n> > \t\thandle_refs(refs, revs, *flags, for_each_bad_bisect_ref);\n> > @@ -2701,15 +2696,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n> > \t\trevs->bisect = 1;\n> > \t} else if (!strcmp(arg, \"--tags\")) {\n> > \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> > \t} else if (!strcmp(arg, \"--remotes\")) {\n> > \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> > \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n> > \t\tstruct all_refs_cb cb;\n> > \t\tinit_all_refs_cb(&cb, revs, *flags);\n> > \t\tfor_each_glob_ref(handle_one_ref, optarg, &cb);\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> > \t\treturn argcount;\n> > \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n> > \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n> > @@ -2718,17 +2713,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n> > \t\tstruct all_refs_cb cb;\n> > \t\tinit_all_refs_cb(&cb, revs, *flags);\n> > \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> > \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n> > \t\tstruct all_refs_cb cb;\n> > \t\tinit_all_refs_cb(&cb, revs, *flags);\n> > \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> > \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n> > \t\tstruct all_refs_cb cb;\n> > \t\tinit_all_refs_cb(&cb, revs, *flags);\n> > \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n> > -\t\tclear_ref_exclusion(&revs->ref_excludes);\n> > +\t\tclear_ref_exclusions(&revs->ref_excludes);\n> \n> The churn I mentioned with the renaming, so maybe worth doing that as a\n> \"prep\" commit?\n\nHm. I don't think it's too bad, and it's weird to rename things already\nwithout any clear justification why that's visible from the diff. Like,\nthere is no `struct ref_exclusions` yet, why rename it?\n\nI'll retain this in a single commit if you don't mind, but amend the\ncommit message to explicitly mention the rename.\n\n> > +struct ref_exclusions {\n> > +\t/*\n> > +\t * Excluded refs is a list of wildmatch patterns. If any of the\n> > +\t * patterns matches, the reference will be excluded.\n> > +\t */\n> > +\tstruct string_list excluded_refs;\n> > +};\n> \n> Per the above POC diff though, please move...\n> \n> > struct oidset;\n> > struct topo_walk_info;\n> > \n> > @@ -103,7 +111,7 @@ struct rev_info {\n> > \tstruct list_objects_filter_options filter;\n> > \n> > \t/* excluding from --branches, --refs, etc. expansion */\n> > -\tstruct string_list *ref_excludes;\n> > +\tstruct ref_exclusions ref_excludes;\n> > \n> > \t/* Basic information */\n> > \tconst char *prefix;\n> > @@ -439,12 +447,14 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)\n> > void show_object_with_name(FILE *, struct object *, const char *);\n> > \n> > /**\n> > - * Helpers to check if a \"struct string_list\" item matches with\n> > - * wildmatch().\n> > + * Helpers to check if a reference should be excluded.\n> > */\n> > -int ref_excluded(struct string_list *, const char *path);\n> > -void clear_ref_exclusion(struct string_list **);\n> > -void add_ref_exclusion(struct string_list **, const char *exclude);\n> > +#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }\n> \n> ...this macro to right after declaring the struct, which is what we\n> usually do, and will help in adding it to \"REV_INFO_INIT\" sooner than\n> later.\n\nFair, will change.\n\n> Also, at the end of your series this end up being overly long, so per\n> the diff-above (which is tot he end of the series), let's start by\n> line-wrapping it:\n> \n> \t#define ..._INIT { \\\n> \t.member = ..._INIT, \\\n> \t}\n\nMakes sense.\n\nPatrick\n"},{"id":"466798","messageId":"Y2ofxE5qY3gnZLpe@ncase","threadId":"58706","inReplyTo":"221107.86pmdyx4me.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T09:22:12Z","receivedAt":"2022-11-08T09:22:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 07, 2022 at 02:34:45PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> On Mon, Nov 07 2022, Patrick Steinhardt wrote:\n[snip]\n> > +\tif (strcmp(section, \"transfer\") && strcmp(section, \"receive\") &&\n> > +\t strcmp(section, \"uploadpack\"))\n> > +\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n> > +\n> > +\tif (exclusions->hidden_refs.nr)\n> > +\t\tdie(_(\"--exclude-hidden= passed more than once\"));\n> \n> We usually just ignore the first of --foo=bar --foo=baz and take \"baz\"\n> in our CLI use. Is it better to die here than just clear the previous\n> one & continue?\n\nIt's something I was torn on. I ultimately chose to die though because\nof the difference between `--exclude` and `--exclude-hidden`: the former\none will happily add additional patterns, all of which will ultimately\nbe ignored. So as a user you might rightfully expect that the latter\nwill work the same: if both `--exclude-hidden=uploadpack` and\n`--exclude-hidden=receive` are specified, you might want to have both be\nignored.\n\nTo me it wasn't quite clear how to support multiple instances of\n`transfer.hideRefs` though as there is also the concept of un-hiding\nalready-hidden refs. So I wanted to avoid going into this discussion to\nmake the patch series a little bit smaller.\n\nBy dying instead of silently overriding the previous argument we retain\nthe ability to iterate on this at a later point though to implement\nabove behaviour, if the usecase ever arises.\n\nPatrick\n"},{"id":"466799","messageId":"Y2omBeBQl14IxqOA@ncase","threadId":"58706","inReplyTo":"221107.86eduevgk3.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T09:48:53Z","receivedAt":"2022-11-08T09:49:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 07, 2022 at 06:07:01PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Nov 07 2022, Ævar Arnfjörð Bjarmason wrote:\n> \n> > On Mon, Nov 07 2022, Patrick Steinhardt wrote:\n> \n> >> +TEST_PASSES_SANITIZE_LEAK=true\n> >\n> > Thanks for adding this! :)\n> \n> Hrm, I spoke too soon :) This series adds new leaks, so it'll fail with\n> the linux-leaks job. I have the following local monkeypatch on top,\n> which obviously doesn't address the root cause. The t6018 leak is new\n> due to the new tests you added.\n\nRight, I didn't know it was as easy to run tests with leak checking as\njust executing `make test SANITIZE=leak`. Anyway, I did that now and the\nissue is in fact in how the hidden refs are parsed because we already\n`xstrdup()` the config value as we need to modify it anyway.\n\nThe following patch fixes the issue:\n\ndiff --git a/refs.c b/refs.c\nindex f1711e2e9f..2c7e88b190 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1430,7 +1430,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tstring_list_append(hide_refs, ref);\n+\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n\nPatrick\n"},{"id":"466803","messageId":"cover.1667901510.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"[PATCH v4 0/6] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T10:03:35Z","receivedAt":"2022-11-08T10:04:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fourth version of my patch series that tries to improve\nperformance of the connectivity check by only considering preexisting\nrefs as uninteresting that could actually have been advertised to the\nclient.\n\nThis time around there are only incremental changes compared to v3, the\noverall implementation stays the same. Changes:\n\n - Fixed a pre-existing memory leak in how hidden refs are parsed so\n that tests now pass with TEST_PASSES_SANITIZE_LEAK=true.\n\n - Improved initialization of `struct ref_exclusions` to reuse the\n `REF_EXCLUSIONS_INIT` macro so we don't have to repeat the logic.\n\n - Dropped the `--exclude-hidden=transfer` option. Only `receive` and\n `uploadpack` are supported now.\n\nPatrick\n\nPatrick Steinhardt (6):\n refs: get rid of global list of hidden refs\n revision: move together exclusion-related functions\n revision: introduce struct to handle exclusions\n revision: add new parameter to exclude hidden refs\n rev-parse: add `--exclude-hidden=` option\n receive-pack: only use visible refs for connectivity check\n\n Documentation/git-rev-parse.txt | 7 ++\n Documentation/rev-list-options.txt | 7 ++\n builtin/receive-pack.c | 10 ++-\n builtin/rev-list.c | 1 +\n builtin/rev-parse.c | 12 ++-\n connected.c | 3 +\n connected.h | 7 ++\n ls-refs.c | 13 ++-\n refs.c | 16 ++--\n refs.h | 5 +-\n revision.c | 117 +++++++++++++++---------\n revision.h | 36 ++++++--\n t/t6018-rev-list-glob.sh | 11 +++\n t/t6021-rev-list-exclude-hidden.sh | 137 +++++++++++++++++++++++++++++\n upload-pack.c | 30 ++++---\n 15 files changed, 329 insertions(+), 83 deletions(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\nRange-diff against v3:\n1: 3741f0a389 ! 1: 34afe30d60 refs: get rid of global list of hidden refs\n @@ refs.c: int parse_hide_refs_config(const char *var, const char *value, const cha\n -\t\t\tCALLOC_ARRAY(hide_refs, 1);\n -\t\t\thide_refs->strdup_strings = 1;\n -\t\t}\n - \t\tstring_list_append(hide_refs, ref);\n +-\t\tstring_list_append(hide_refs, ref);\n ++\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n2: a6dcc99ca9 = 2: b4f21d0a80 revision: move together exclusion-related functions\n3: 2a6a67df1d ! 3: 265b292ed5 revision: introduce struct to handle exclusions\n @@ Commit message\n of exclusion.\n \n Introduce a new `struct ref_exclusions` that encapsulates all the logic\n - related to excluding refs.\n + related to excluding refs and move the `struct string_list` that holds\n + all wildmatch patterns of excluded refs into it. Rename functions that\n + operate on this struct to match its name.\n \n Signed-off-by: Patrick Steinhardt \n \n @@ revision.c: static void add_rev_cmdline_list(struct rev_info *revs,\n -\t\tfree(*ref_excludes_p);\n -\t}\n -\t*ref_excludes_p = NULL;\n -+\tstring_list_init_dup(&exclusions->excluded_refs);\n ++\tstruct ref_exclusions blank = REF_EXCLUSIONS_INIT;\n ++\tmemcpy(exclusions, &blank, sizeof(*exclusions));\n }\n \n -void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n @@ revision.h: struct rev_cmdline_info {\n +\t */\n +\tstruct string_list excluded_refs;\n +};\n ++\n ++/**\n ++ * Initialize a `struct ref_exclusions` with a macro.\n ++ */\n ++#define REF_EXCLUSIONS_INIT { \\\n ++\t.excluded_refs = STRING_LIST_INIT_DUP, \\\n ++}\n +\n struct oidset;\n struct topo_walk_info;\n @@ revision.h: void mark_trees_uninteresting_sparse(struct repository *r, struct oi\n -int ref_excluded(struct string_list *, const char *path);\n -void clear_ref_exclusion(struct string_list **);\n -void add_ref_exclusion(struct string_list **, const char *exclude);\n -+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }\n -+\n +int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n +void init_ref_exclusions(struct ref_exclusions *);\n +void clear_ref_exclusions(struct ref_exclusions *);\n4: de7c1aa210 ! 4: c7fa6698db revision: add new parameter to exclude hidden refs\n @@ Documentation/rev-list-options.txt: respectively, and they must begin with `refs\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n -+--exclude-hidden=[transfer|receive|uploadpack]::\n ++--exclude-hidden=[receive|uploadpack]::\n +\tDo not include refs that have been hidden via either one of\n -+\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n -+\tthe next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would\n -+\totherwise consider. This option is cleared when seeing one of these\n -+\tpseudo-refs.\n ++\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n ++\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n ++\twould otherwise consider. This option is cleared when seeing one of\n ++\tthese pseudo-refs.\n +\n --reflog::\n \tPretend as if all objects mentioned by reflogs are listed on the\n @@ builtin/rev-list.c: static const char rev_list_usage[] =\n \" --tags\\n\"\n \" --remotes\\n\"\n \" --stdin\\n\"\n -+\" --exclude-hidden=[transfer|receive|uploadpack]\\n\"\n ++\" --exclude-hidden=[receive|uploadpack]\\n\"\n \" --quiet\\n\"\n \" ordering output:\\n\"\n \" --topo-order\\n\"\n @@ revision.c: static void add_rev_cmdline_list(struct rev_info *revs,\n \treturn 0;\n }\n \n - void init_ref_exclusions(struct ref_exclusions *exclusions)\n - {\n - \tstring_list_init_dup(&exclusions->excluded_refs);\n -+\tstring_list_init_dup(&exclusions->hidden_refs);\n - }\n - \n +@@ revision.c: void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n -+\tstring_list_clear(&exclusions->hidden_refs, 1);\n ++\tstring_list_clear(&exclusions->hidden_refs, 0);\n }\n \n void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n @@ revision.c: void add_ref_exclusion(struct ref_exclusions *exclusions, const char\n +{\n +\tstruct exclude_hidden_refs_cb cb;\n +\n -+\tif (strcmp(section, \"transfer\") && strcmp(section, \"receive\") &&\n -+\t strcmp(section, \"uploadpack\"))\n ++\tif (strcmp(section, \"receive\") && strcmp(section, \"uploadpack\"))\n +\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n +\n +\tif (exclusions->hidden_refs.nr)\n @@ revision.h: struct ref_exclusions {\n +\tstruct string_list hidden_refs;\n };\n \n + /**\n +@@ revision.h: struct ref_exclusions {\n + */\n + #define REF_EXCLUSIONS_INIT { \\\n + \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n ++\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n + }\n + \n struct oidset;\n @@ revision.h: void show_object_with_name(FILE *, struct object *, const char *);\n /**\n * Helpers to check if a reference should be excluded.\n */\n --#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }\n -+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }\n - \n ++\n int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n void init_ref_exclusions(struct ref_exclusions *);\n void clear_ref_exclusions(struct ref_exclusions *);\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +\n +test_description='git rev-list --exclude-hidden test'\n +\n -+TEST_PASSES_SANITIZE_LEAK=true\n +. ./test-lib.sh\n +\n +test_expect_success 'setup' '\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +\ttest_cmp expected err\n +'\n +\n -+test_expect_success 'passed multiple times' '\n -+\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n -+\ttest_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --exclude-hidden=transfer 2>err &&\n -+\ttest_cmp expected err\n -+'\n -+\n -+test_expect_success '--exclude-hidden without hiddenRefs' '\n -+\tgit rev-list --exclude-hidden=transfer --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$NAMESPACE\n -+\t$HIDDEN\n -+\t$TAG\n -+\t$COMMIT\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n -+test_expect_success 'hidden via transfer.hideRefs' '\n -+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$NAMESPACE\n -+\t$TAG\n -+\t$COMMIT\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n -+test_expect_success '--all --exclude-hidden=transfer --not --all without hidden refs' '\n -+\tgit rev-list --all --exclude-hidden=transfer --not --all >out &&\n -+\ttest_must_be_empty out\n -+'\n -+\n -+test_expect_success '--all --exclude-hidden=transfer --not --all with hidden ref' '\n -+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=transfer --not --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$HIDDEN\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n -+test_expect_success '--exclude-hidden with --exclude' '\n -+\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=transfer --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$NAMESPACE\n -+\t$COMMIT\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n -+test_expect_success '--exclude-hidden is reset' '\n -+\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=transfer --all --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$NAMESPACE\n -+\t$HIDDEN\n -+\t$TAG\n -+\t$COMMIT\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n -+test_expect_success '--exclude-hidden operates on stripped refs by default' '\n -+\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=transfer --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$HIDDEN\n -+\t$TAG\n -+\t$COMMIT\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n -+test_expect_success '--exclude-hidden does not hide namespace by default' '\n -+\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$NAMESPACE\n -+\t$HIDDEN\n -+\t$TAG\n -+\t$COMMIT\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n -+test_expect_success '--exclude-hidden= may operate on unstripped refs' '\n -+\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&\n -+\tcat >expected <<-EOF &&\n -+\t$HIDDEN\n -+\t$TAG\n -+\t$COMMIT\n -+\tEOF\n -+\ttest_cmp expected out\n -+'\n -+\n +for section in receive uploadpack\n +do\n -+\ttest_expect_success \"hidden via $section.hideRefs\" '\n -+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n -+\t\tcat >expected <<-EOF &&\n -+\t\t$NAMESPACE\n -+\t\t$TAG\n -+\t\t$COMMIT\n -+\t\tEOF\n -+\t\ttest_cmp expected out\n ++\ttest_expect_success \"$section: passed multiple times\" '\n ++\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n ++\t\ttest_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n ++\t\ttest_cmp expected err\n +\t'\n +\n -+\ttest_expect_success \"--exclude-hidden=$section respects transfer.hideRefs\" '\n -+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n -+\t\tcat >expected <<-EOF &&\n -+\t\t$NAMESPACE\n -+\t\t$TAG\n -+\t\t$COMMIT\n -+\t\tEOF\n -+\t\ttest_cmp expected out\n -+\t'\n -+\n -+\ttest_expect_success \"--exclude-hidden=transfer ignores $section.hideRefs\" '\n -+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&\n ++\ttest_expect_success \"$section: without hiddenRefs\" '\n ++\t\tgit rev-list --exclude-hidden=$section --all >out &&\n +\t\tcat >expected <<-EOF &&\n +\t\t$NAMESPACE\n +\t\t$HIDDEN\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +\t\ttest_cmp expected out\n +\t'\n +\n -+\ttest_expect_success \"--exclude-hidden=$section respects both transfer.hideRefs and $section.hideRefs\" '\n ++\ttest_expect_success \"$section: hidden via transfer.hideRefs\" '\n ++\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$NAMESPACE\n ++\t\t$TAG\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: hidden via $section.hideRefs\" '\n ++\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$NAMESPACE\n ++\t\t$TAG\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: respects both transfer.hideRefs and $section.hideRefs\" '\n +\t\tgit -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n +\t\tcat >expected <<-EOF &&\n +\t\t$NAMESPACE\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +\t\tEOF\n +\t\ttest_cmp expected out\n +\t'\n ++\n ++\ttest_expect_success \"$section: negation without hidden refs marks everything as uninteresting\" '\n ++\t\tgit rev-list --all --exclude-hidden=$section --not --all >out &&\n ++\t\ttest_must_be_empty out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: negation with hidden refs marks them as interesting\" '\n ++\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$HIDDEN\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: hidden refs and excludes work together\" '\n ++\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$NAMESPACE\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: excluded hidden refs get reset\" '\n ++\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$NAMESPACE\n ++\t\t$HIDDEN\n ++\t\t$TAG\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: operates on stripped refs by default\" '\n ++\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$HIDDEN\n ++\t\t$TAG\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: does not hide namespace by default\" '\n ++\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$NAMESPACE\n ++\t\t$HIDDEN\n ++\t\t$TAG\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: can operate on unstripped refs\" '\n ++\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$HIDDEN\n ++\t\t$TAG\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n +done\n +\n +test_done\n5: 68a5e56304 ! 5: 79c5c64a80 revparse: add `--exclude-hidden=` option\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - revparse: add `--exclude-hidden=` option\n + rev-parse: add `--exclude-hidden=` option\n \n Add a new `--exclude-hidden=` option that is similar to the one we just\n - added to git-rev-list(1). Given a seciton name `transfer`, `uploadpack`\n - or `receive` as argument, it causes us to exclude all references that\n - would be hidden by the respective `$seciton.hideRefs` configuration.\n + added to git-rev-list(1). Given a seciton name `uploadpack` or `receive`\n + as argument, it causes us to exclude all references that would be hidden\n + by the respective `$section.hideRefs` configuration.\n \n Signed-off-by: Patrick Steinhardt \n \n @@ Documentation/git-rev-parse.txt: respectively, and they must begin with `refs/`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n -+--exclude-hidden=[transfer|receive|uploadpack]::\n ++--exclude-hidden=[receive|uploadpack]::\n +\tDo not include refs that have been hidden via either one of\n -+\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n -+\tthe next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would\n -+\totherwise consider. This option is cleared when seeing one of these\n -+\tpseudo-refs.\n ++\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n ++\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n ++\twould otherwise consider. This option is cleared when seeing one of\n ++\tthese pseudo-refs.\n +\n --disambiguate=::\n \tShow every object whose name begins with the given prefix.\n @@ t/t6018-rev-list-glob.sh: test_expect_success 'rev-parse --exclude=ref with --re\n \tcompare rev-parse \"--exclude=upstream/x --remotes=upstream/*\" \"upstream/one upstream/two\"\n '\n \n -+test_expect_success 'rev-parse --exclude-hidden= with --all' '\n -+\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--exclude-hidden=transfer --all\" \"--branches --tags\"\n -+'\n ++for section in receive uploadpack\n ++do\n ++\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n ++\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--exclude-hidden=$section --all\" \"--branches --tags\"\n ++\t'\n +\n -+test_expect_success 'rev-parse --exclude-hidden= with --all' '\n -+\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude-hidden=transfer --all\" \"--exclude=refs/heads/subspace/* --all\"\n -+'\n ++\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n ++\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude-hidden=$section --all\" \"--exclude=refs/heads/subspace/* --all\"\n ++\t'\n ++done\n +\n test_expect_success 'rev-list --exclude=glob with --branches=glob' '\n \tcompare rev-list \"--exclude=subspace-* --branches=sub*\" \"subspace/one subspace/two\"\n6: 9d15449559 = 6: 39b4741734 receive-pack: only use visible refs for connectivity check\n-- \n2.38.1\n\n"},{"id":"466804","messageId":"34afe30d60e4dbc7e50a590541440c20c27507a6.1667901510.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667901510.git.ps@pks.im","subject":"[PATCH v4 1/6] refs: get rid of global list of hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T10:03:39Z","receivedAt":"2022-11-08T10:04:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to add a new argument to git-rev-list(1) that allows it to\nadd all references that are visible when taking `transfer.hideRefs` et\nal into account. This will require us to potentially parse multiple sets\nof hidden refs, which is not easily possible right now as there is only\na single, global instance of the list of parsed hidden refs.\n\nRefactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both\ntake the list of hidden references as input and adjust callers to keep a\nlocal list, instead. This allows us to easily use multiple hidden-ref\nlists. Furthermore, it allows us to properly free this list before we\nexit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 8 +++++---\n ls-refs.c | 13 +++++++++----\n refs.c | 16 +++++-----------\n refs.h | 5 +++--\n upload-pack.c | 30 ++++++++++++++++++------------\n 5 files changed, 40 insertions(+), 32 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 44bcea3a5b..1f3efc58fb 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)\n \n static int receive_pack_config(const char *var, const char *value, void *cb)\n {\n-\tint status = parse_hide_refs_config(var, value, \"receive\");\n+\tint status = parse_hide_refs_config(var, value, \"receive\", &hidden_refs);\n \n \tif (status)\n \t\treturn status;\n@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,\n \tstruct oidset *seen = data;\n \tconst char *path = strip_namespace(path_full);\n \n-\tif (ref_is_hidden(path, path_full))\n+\tif (ref_is_hidden(path, path_full, &hidden_refs))\n \t\treturn 0;\n \n \t/*\n@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)\n \t\tstrbuf_setlen(&refname_full, prefix_len);\n \t\tstrbuf_addstr(&refname_full, cmd->ref_name);\n \n-\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf))\n+\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))\n \t\t\tcontinue;\n \t\tif (is_null_oid(&cmd->new_oid))\n \t\t\tcmd->error_string = \"deny deleting a hidden ref\";\n@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n+\tstring_list_clear(&hidden_refs, 1);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex fa0d01b47c..ae89f850e9 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -6,6 +6,7 @@\n #include \"ls-refs.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"string-list.h\"\n \n static int config_read;\n static int advertise_unborn;\n@@ -73,6 +74,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n+\tstruct string_list hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \n \tstrbuf_reset(&data->buf);\n \n-\tif (ref_is_hidden(refname_nons, refname))\n+\tif (ref_is_hidden(refname_nons, refname, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (!ref_match(&data->prefixes, refname_nons))\n@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)\n }\n \n static int ls_refs_config(const char *var, const char *value,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *cb_data)\n {\n+\tstruct ls_refs_data *data = cb_data;\n \t/*\n \t * We only serve fetches over v2 for now, so respect only \"uploadpack\"\n \t * config. This may need to eventually be expanded to \"receive\", but we\n \t * don't yet know how that information will be passed to ls-refs.\n \t */\n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n int ls_refs(struct repository *r, struct packet_reader *request)\n@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n+\tstring_list_init_dup(&data.hidden_refs);\n \n \tensure_config_read();\n-\tgit_config(ls_refs_config, NULL);\n+\tgit_config(ls_refs_config, &data);\n \n \twhile (packet_reader_read(request) == PACKET_READ_NORMAL) {\n \t\tconst char *arg = request->line;\n@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n+\tstring_list_clear(&data.hidden_refs, 1);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 1491ae937e..2c7e88b190 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n \t\t\t\t\t refname, strict);\n }\n \n-static struct string_list *hide_refs;\n-\n-int parse_hide_refs_config(const char *var, const char *value, const char *section)\n+int parse_hide_refs_config(const char *var, const char *value, const char *section,\n+\t\t\t struct string_list *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tif (!hide_refs) {\n-\t\t\tCALLOC_ARRAY(hide_refs, 1);\n-\t\t\thide_refs->strdup_strings = 1;\n-\t\t}\n-\t\tstring_list_append(hide_refs, ref);\n+\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n-int ref_is_hidden(const char *refname, const char *refname_full)\n+int ref_is_hidden(const char *refname, const char *refname_full,\n+\t\t const struct string_list *hide_refs)\n {\n \tint i;\n \n-\tif (!hide_refs)\n-\t\treturn 0;\n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n \t\tconst char *match = hide_refs->items[i].string;\n \t\tconst char *subject;\ndiff --git a/refs.h b/refs.h\nindex 8958717a17..3266fd8f57 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,\n \t const struct object_id *new_oid, const struct object_id *old_oid,\n \t unsigned int flags, enum action_on_err onerr);\n \n-int parse_hide_refs_config(const char *var, const char *value, const char *);\n+int parse_hide_refs_config(const char *var, const char *value, const char *,\n+\t\t\t struct string_list *);\n \n /*\n * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);\n * the ref is outside that namespace, the first parameter is NULL. The second\n * parameter always points to the full ref name.\n */\n-int ref_is_hidden(const char *, const char *);\n+int ref_is_hidden(const char *, const char *, const struct string_list *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0b8311bd68..9db17f8787 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -62,6 +62,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n+\tstruct string_list hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n+\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n \tmemset(data, 0, sizeof(*data));\n \tdata->symref = symref;\n \tdata->wanted_refs = wanted_refs;\n+\tdata->hidden_refs = hidden_refs;\n \tdata->want_obj = want_obj;\n \tdata->have_obj = have_obj;\n \tdata->haves = haves;\n@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n+\tstring_list_clear(&data->hidden_refs, 1);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * Checking for reachable shallows requires that our refs be\n \t\t * marked with OUR_REF.\n \t\t */\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, data);\n+\t\tfor_each_namespaced_ref(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid)\n+\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n-\tif (ref_is_hidden(refname, refname_full)) {\n+\tif (ref_is_hidden(refname, refname_full, hidden_refs)) {\n \t\to->flags |= HIDDEN_REF;\n \t\treturn 1;\n \t}\n@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,\n }\n \n static int check_ref(const char *refname_full, const struct object_id *oid,\n-\t\t int flag UNUSED, void *cb_data UNUSED)\n+\t\t int flag UNUSED, void *cb_data)\n {\n \tconst char *refname = strip_namespace(refname_full);\n+\tstruct upload_pack_data *data = cb_data;\n \n-\tmark_our_ref(refname, refname_full, oid);\n+\tmark_our_ref(refname, refname_full, oid, &data->hidden_refs);\n \treturn 0;\n }\n \n@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \tstruct object_id peeled;\n \tstruct upload_pack_data *data = cb_data;\n \n-\tif (mark_our_ref(refname_nons, refname, oid))\n+\tif (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (capabilities) {\n@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)\n \tif (parse_object_filter_config(var, value, data) < 0)\n \t\treturn -1;\n \n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)\n@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, &data);\n+\t\tfor_each_namespaced_ref(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t struct string_list *wanted_refs,\n+\t\t\t struct string_list *hidden_refs,\n \t\t\t struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\tstruct strbuf refname = STRBUF_INIT;\n \n \t\tstrbuf_addf(&refname, \"%s%s\", get_git_namespace(), refname_nons);\n-\t\tif (ref_is_hidden(refname_nons, refname.buf) ||\n+\t\tif (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||\n \t\t read_ref(refname.buf, &oid)) {\n \t\t\tpacket_writer_error(writer, \"unknown ref %s\", refname_nons);\n \t\t\tdie(\"unknown ref %s\", refname_nons);\n@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (data->allow_ref_in_want &&\n \t\t parse_want_ref(&data->writer, arg, &data->wanted_refs,\n-\t\t\t\t &data->want_obj))\n+\t\t\t\t &data->hidden_refs, &data->want_obj))\n \t\t\tcontinue;\n \t\t/* process have line */\n \t\tif (parse_have(arg, &data->haves))\n-- \n2.38.1\n\n"},{"id":"466807","messageId":"b4f21d0a80101306bde350365c8fc452715fac1b.1667901510.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667901510.git.ps@pks.im","subject":"[PATCH v4 2/6] revision: move together exclusion-related functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T10:03:43Z","receivedAt":"2022-11-08T10:04:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move together the definitions of functions that handle exclusions of\nrefs so that related functionality sits in a single place, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 52 ++++++++++++++++++++++++++--------------------------\n 1 file changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 0760e78936..be755670e2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-struct all_refs_cb {\n-\tint all_flags;\n-\tint warned_bad_reflog;\n-\tstruct rev_info *all_revs;\n-\tconst char *name_for_errormsg;\n-\tstruct worktree *wt;\n-};\n-\n int ref_excluded(struct string_list *ref_excludes, const char *path)\n {\n \tstruct string_list_item *item;\n@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)\n \treturn 0;\n }\n \n+void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+{\n+\tif (*ref_excludes_p) {\n+\t\tstring_list_clear(*ref_excludes_p, 0);\n+\t\tfree(*ref_excludes_p);\n+\t}\n+\t*ref_excludes_p = NULL;\n+}\n+\n+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+{\n+\tif (!*ref_excludes_p) {\n+\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n+\t\t(*ref_excludes_p)->strdup_strings = 1;\n+\t}\n+\tstring_list_append(*ref_excludes_p, exclude);\n+}\n+\n+struct all_refs_cb {\n+\tint all_flags;\n+\tint warned_bad_reflog;\n+\tstruct rev_info *all_revs;\n+\tconst char *name_for_errormsg;\n+\tstruct worktree *wt;\n+};\n+\n static int handle_one_ref(const char *path, const struct object_id *oid,\n \t\t\t int flag UNUSED,\n \t\t\t void *cb_data)\n@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,\n \tcb->wt = NULL;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n-{\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n-}\n-\n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n-{\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n-}\n-\n static void handle_refs(struct ref_store *refs,\n \t\t\tstruct rev_info *revs, unsigned flags,\n \t\t\tint (*for_each)(struct ref_store *, each_ref_fn, void *))\n-- \n2.38.1\n\n"},{"id":"466805","messageId":"79c5c64a803b67ccfecc57328e565a2fa9563507.1667901510.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667901510.git.ps@pks.im","subject":"[PATCH v4 5/6] rev-parse: add `--exclude-hidden=` option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T10:03:58Z","receivedAt":"2022-11-08T10:04:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new `--exclude-hidden=` option that is similar to the one we just\nadded to git-rev-list(1). Given a seciton name `uploadpack` or `receive`\nas argument, it causes us to exclude all references that would be hidden\nby the respective `$section.hideRefs` configuration.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/git-rev-parse.txt | 7 +++++++\n builtin/rev-parse.c | 4 ++++\n t/t6018-rev-list-glob.sh | 11 +++++++++++\n 3 files changed, 22 insertions(+)\n\ndiff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt\nindex 6b8ca085aa..393aa6e982 100644\n--- a/Documentation/git-rev-parse.txt\n+++ b/Documentation/git-rev-parse.txt\n@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[receive|uploadpack]::\n+\tDo not include refs that have been hidden via either one of\n+\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n+\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n+\twould otherwise consider. This option is cleared when seeing one of\n+\tthese pseudo-refs.\n+\n --disambiguate=::\n \tShow every object whose name begins with the given prefix.\n \tThe must be at least 4 hexadecimal digits long to\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 7fa5b6991b..49730c7a23 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -895,6 +895,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tadd_ref_exclusion(&ref_excludes, arg);\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (skip_prefix(arg, \"--exclude-hidden=\", &arg)) {\n+\t\t\t\texclude_hidden_refs(&ref_excludes, arg);\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-toplevel\")) {\n \t\t\t\tconst char *work_tree = get_git_work_tree();\n \t\t\t\tif (work_tree)\ndiff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh\nindex e1abc5c2b3..af0c55cbe7 100755\n--- a/t/t6018-rev-list-glob.sh\n+++ b/t/t6018-rev-list-glob.sh\n@@ -187,6 +187,17 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '\n \tcompare rev-parse \"--exclude=upstream/x --remotes=upstream/*\" \"upstream/one upstream/two\"\n '\n \n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--exclude-hidden=$section --all\" \"--branches --tags\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude-hidden=$section --all\" \"--exclude=refs/heads/subspace/* --all\"\n+\t'\n+done\n+\n test_expect_success 'rev-list --exclude=glob with --branches=glob' '\n \tcompare rev-list \"--exclude=subspace-* --branches=sub*\" \"subspace/one subspace/two\"\n '\n-- \n2.38.1\n\n"},{"id":"466806","messageId":"265b292ed5c2de19b7118dfe046d3d9d932e2e89.1667901510.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667901510.git.ps@pks.im","subject":"[PATCH v4 3/6] revision: introduce struct to handle exclusions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T10:03:47Z","receivedAt":"2022-11-08T10:04:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions that handle exclusion of refs work on a single string\nlist. We're about to add a second mechanism for excluding refs though,\nand it makes sense to reuse much of the same architecture for both kinds\nof exclusion.\n\nIntroduce a new `struct ref_exclusions` that encapsulates all the logic\nrelated to excluding refs and move the `struct string_list` that holds\nall wildmatch patterns of excluded refs into it. Rename functions that\noperate on this struct to match its name.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/rev-parse.c | 8 ++++----\n revision.c | 48 +++++++++++++++++++++------------------------\n revision.h | 27 +++++++++++++++++++------\n 3 files changed, 47 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 8f61050bde..7fa5b6991b 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -39,7 +39,7 @@ static int abbrev_ref_strict;\n static int output_sq;\n \n static int stuck_long;\n-static struct string_list *ref_excludes;\n+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;\n \n /*\n * Some arguments are relevant \"revision\" arguments,\n@@ -198,7 +198,7 @@ static int show_default(void)\n static int show_reference(const char *refname, const struct object_id *oid,\n \t\t\t int flag UNUSED, void *cb_data UNUSED)\n {\n-\tif (ref_excluded(ref_excludes, refname))\n+\tif (ref_excluded(&ref_excludes, refname))\n \t\treturn 0;\n \tshow_rev(NORMAL, oid, refname);\n \treturn 0;\n@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)\n \t\tfor_each_glob_ref_in(show_reference, pattern, prefix, NULL);\n \telse\n \t\tfor_each_ref_in(prefix, show_reference, NULL);\n-\tclear_ref_exclusion(&ref_excludes);\n+\tclear_ref_exclusions(&ref_excludes);\n }\n \n enum format_type {\n@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--all\")) {\n \t\t\t\tfor_each_ref(show_reference, NULL);\n-\t\t\t\tclear_ref_exclusion(&ref_excludes);\n+\t\t\t\tclear_ref_exclusions(&ref_excludes);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (skip_prefix(arg, \"--disambiguate=\", &arg)) {\ndiff --git a/revision.c b/revision.c\nindex be755670e2..fe3ec98f46 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,35 +1517,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-int ref_excluded(struct string_list *ref_excludes, const char *path)\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n \tstruct string_list_item *item;\n-\n-\tif (!ref_excludes)\n-\t\treturn 0;\n-\tfor_each_string_list_item(item, ref_excludes) {\n+\tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+void init_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n+\tstruct ref_exclusions blank = REF_EXCLUSIONS_INIT;\n+\tmemcpy(exclusions, &blank, sizeof(*exclusions));\n }\n \n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n+\tstring_list_clear(&exclusions->excluded_refs, 0);\n+}\n+\n+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n+{\n+\tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n struct all_refs_cb {\n@@ -1563,7 +1558,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \tstruct all_refs_cb *cb = cb_data;\n \tstruct object *object;\n \n-\tif (ref_excluded(cb->all_revs->ref_excludes, path))\n+\tif (ref_excluded(&cb->all_revs->ref_excludes, path))\n \t return 0;\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n@@ -1901,6 +1896,7 @@ void repo_init_revisions(struct repository *r,\n \n \tinit_display_notes(&revs->notes_opt);\n \tlist_objects_filter_init(&revs->filter);\n+\tinit_ref_exclusions(&revs->ref_excludes);\n }\n \n static void add_pending_commit_list(struct rev_info *revs,\n@@ -2689,10 +2685,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\t\tother_head_refs(handle_one_ref, &cb);\n \t\t}\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--branches\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--bisect\")) {\n \t\tread_bisect_terms(&term_bad, &term_good);\n \t\thandle_refs(refs, revs, *flags, for_each_bad_bisect_ref);\n@@ -2701,15 +2697,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->bisect = 1;\n \t} else if (!strcmp(arg, \"--tags\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--remotes\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref(handle_one_ref, optarg, &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t\treturn argcount;\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n@@ -2718,17 +2714,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--reflog\")) {\n \t\tadd_reflogs_to_pending(revs, *flags);\n \t} else if (!strcmp(arg, \"--indexed-objects\")) {\ndiff --git a/revision.h b/revision.h\nindex afe1b77985..5c8ab16047 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -81,6 +81,21 @@ struct rev_cmdline_info {\n \t} *rev;\n };\n \n+struct ref_exclusions {\n+\t/*\n+\t * Excluded refs is a list of wildmatch patterns. If any of the\n+\t * patterns matches, the reference will be excluded.\n+\t */\n+\tstruct string_list excluded_refs;\n+};\n+\n+/**\n+ * Initialize a `struct ref_exclusions` with a macro.\n+ */\n+#define REF_EXCLUSIONS_INIT { \\\n+\t.excluded_refs = STRING_LIST_INIT_DUP, \\\n+}\n+\n struct oidset;\n struct topo_walk_info;\n \n@@ -103,7 +118,7 @@ struct rev_info {\n \tstruct list_objects_filter_options filter;\n \n \t/* excluding from --branches, --refs, etc. expansion */\n-\tstruct string_list *ref_excludes;\n+\tstruct ref_exclusions ref_excludes;\n \n \t/* Basic information */\n \tconst char *prefix;\n@@ -439,12 +454,12 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)\n void show_object_with_name(FILE *, struct object *, const char *);\n \n /**\n- * Helpers to check if a \"struct string_list\" item matches with\n- * wildmatch().\n+ * Helpers to check if a reference should be excluded.\n */\n-int ref_excluded(struct string_list *, const char *path);\n-void clear_ref_exclusion(struct string_list **);\n-void add_ref_exclusion(struct string_list **, const char *exclude);\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n+void init_ref_exclusions(struct ref_exclusions *);\n+void clear_ref_exclusions(struct ref_exclusions *);\n+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n \n /**\n * This function can be used if you want to add commit objects as revision\n-- \n2.38.1\n\n"},{"id":"466808","messageId":"c7fa6698dbe7b0eb5e2b9c4725e9bdc6cdb22983.1667901510.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667901510.git.ps@pks.im","subject":"[PATCH v4 4/6] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T10:03:51Z","receivedAt":"2022-11-08T10:04:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Users can optionally hide refs from remote users in git-upload-pack(1),\ngit-receive-pack(1) and others via the `transfer.hideRefs`, but there is\nnot an easy way to obtain the list of all visible or hidden refs right\nnow. We'll require just that though for a performance improvement in our\nconnectivity check.\n\nAdd a new option `--exclude-hidden=` that excludes any hidden refs from\nthe next pseudo-ref like `--all` or `--branches`.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 7 ++\n builtin/rev-list.c | 1 +\n revision.c | 41 ++++++++-\n revision.h | 9 ++\n t/t6021-rev-list-exclude-hidden.sh | 137 +++++++++++++++++++++++++++++\n 5 files changed, 194 insertions(+), 1 deletion(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 1837509566..5b46781b35 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[receive|uploadpack]::\n+\tDo not include refs that have been hidden via either one of\n+\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n+\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n+\twould otherwise consider. This option is cleared when seeing one of\n+\tthese pseudo-refs.\n+\n --reflog::\n \tPretend as if all objects mentioned by reflogs are listed on the\n \tcommand line as ``.\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3acd93f71e..d42db0b0cc 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -38,6 +38,7 @@ static const char rev_list_usage[] =\n \" --tags\\n\"\n \" --remotes\\n\"\n \" --stdin\\n\"\n+\" --exclude-hidden=[receive|uploadpack]\\n\"\n \" --quiet\\n\"\n \" ordering output:\\n\"\n \" --topo-order\\n\"\ndiff --git a/revision.c b/revision.c\nindex fe3ec98f46..b726cfd255 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"object-store.h\"\n #include \"tag.h\"\n #include \"blob.h\"\n@@ -1519,11 +1520,17 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \n int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n+\tconst char *stripped_path = strip_namespace(path);\n \tstruct string_list_item *item;\n+\n \tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n+\n+\tif (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))\n+\t\treturn 1;\n+\n \treturn 0;\n }\n \n@@ -1536,6 +1543,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n+\tstring_list_clear(&exclusions->hidden_refs, 0);\n }\n \n void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n@@ -1543,6 +1551,34 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n \tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n+struct exclude_hidden_refs_cb {\n+\tstruct ref_exclusions *exclusions;\n+\tconst char *section;\n+};\n+\n+static int hide_refs_config(const char *var, const char *value, void *cb_data)\n+{\n+\tstruct exclude_hidden_refs_cb *cb = cb_data;\n+\treturn parse_hide_refs_config(var, value, cb->section,\n+\t\t\t\t &cb->exclusions->hidden_refs);\n+}\n+\n+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)\n+{\n+\tstruct exclude_hidden_refs_cb cb;\n+\n+\tif (strcmp(section, \"receive\") && strcmp(section, \"uploadpack\"))\n+\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n+\n+\tif (exclusions->hidden_refs.nr)\n+\t\tdie(_(\"--exclude-hidden= passed more than once\"));\n+\n+\tcb.exclusions = exclusions;\n+\tcb.section = section;\n+\n+\tgit_config(hide_refs_config, &cb);\n+}\n+\n struct all_refs_cb {\n \tint all_flags;\n \tint warned_bad_reflog;\n@@ -2221,7 +2257,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t !strcmp(arg, \"--bisect\") || starts_with(arg, \"--glob=\") ||\n \t !strcmp(arg, \"--indexed-objects\") ||\n \t !strcmp(arg, \"--alternate-refs\") ||\n-\t starts_with(arg, \"--exclude=\") ||\n+\t starts_with(arg, \"--exclude=\") || starts_with(arg, \"--exclude-hidden=\") ||\n \t starts_with(arg, \"--branches=\") || starts_with(arg, \"--tags=\") ||\n \t starts_with(arg, \"--remotes=\") || starts_with(arg, \"--no-walk=\"))\n \t{\n@@ -2710,6 +2746,9 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n \t\treturn argcount;\n+\t} else if ((argcount = parse_long_opt(\"exclude-hidden\", argv, &optarg))) {\n+\t\texclude_hidden_refs(&revs->ref_excludes, optarg);\n+\t\treturn argcount;\n \t} else if (skip_prefix(arg, \"--branches=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\ndiff --git a/revision.h b/revision.h\nindex 5c8ab16047..a96fefebf1 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -87,6 +87,12 @@ struct ref_exclusions {\n \t * patterns matches, the reference will be excluded.\n \t */\n \tstruct string_list excluded_refs;\n+\n+\t/*\n+\t * Hidden refs is a list of patterns that is to be hidden via\n+\t * `ref_is_hidden()`.\n+\t */\n+\tstruct string_list hidden_refs;\n };\n \n /**\n@@ -94,6 +100,7 @@ struct ref_exclusions {\n */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n }\n \n struct oidset;\n@@ -456,10 +463,12 @@ void show_object_with_name(FILE *, struct object *, const char *);\n /**\n * Helpers to check if a reference should be excluded.\n */\n+\n int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n void init_ref_exclusions(struct ref_exclusions *);\n void clear_ref_exclusions(struct ref_exclusions *);\n void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n+void exclude_hidden_refs(struct ref_exclusions *, const char *section);\n \n /**\n * This function can be used if you want to add commit objects as revision\ndiff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh\nnew file mode 100755\nindex 0000000000..4ab50e7f4f\n--- /dev/null\n+++ b/t/t6021-rev-list-exclude-hidden.sh\n@@ -0,0 +1,137 @@\n+#!/bin/sh\n+\n+test_description='git rev-list --exclude-hidden test'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit_bulk --id=commit --ref=refs/heads/main 1 &&\n+\tCOMMIT=$(git rev-parse refs/heads/main) &&\n+\ttest_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&\n+\tTAG=$(git rev-parse refs/tags/lightweight) &&\n+\ttest_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&\n+\tHIDDEN=$(git rev-parse refs/hidden/commit) &&\n+\ttest_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&\n+\tNAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)\n+'\n+\n+test_expect_success 'invalid section' '\n+\techo \"fatal: unsupported section for hidden refs: unsupported\" >expected &&\n+\ttest_must_fail git rev-list --exclude-hidden=unsupported 2>err &&\n+\ttest_cmp expected err\n+'\n+\n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"$section: passed multiple times\" '\n+\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n+\t\ttest_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n+\t\ttest_cmp expected err\n+\t'\n+\n+\ttest_expect_success \"$section: without hiddenRefs\" '\n+\t\tgit rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden via transfer.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden via $section.hideRefs\" '\n+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: respects both transfer.hideRefs and $section.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: negation without hidden refs marks everything as uninteresting\" '\n+\t\tgit rev-list --all --exclude-hidden=$section --not --all >out &&\n+\t\ttest_must_be_empty out\n+\t'\n+\n+\ttest_expect_success \"$section: negation with hidden refs marks them as interesting\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden refs and excludes work together\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: excluded hidden refs get reset\" '\n+\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: operates on stripped refs by default\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: does not hide namespace by default\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: can operate on unstripped refs\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+done\n+\n+test_done\n-- \n2.38.1\n\n"},{"id":"466809","messageId":"39b4741734f9b968430a57a9d7168fb5df398f73.1667901510.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1667901510.git.ps@pks.im","subject":"[PATCH v4 6/6] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T10:04:02Z","receivedAt":"2022-11-08T10:04:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When serving a push, git-receive-pack(1) needs to verify that the\npackfile sent by the client contains all objects that are required by\nthe updated references. This connectivity check works by marking all\npreexisting references as uninteresting and using the new reference tips\nas starting point for a graph walk.\n\nMarking all preexisting references as uninteresting can be a problem\nwhen it comes to performance. Git forges tend to do internal bookkeeping\nto keep alive sets of objects for internal use or make them easy to find\nvia certain references. These references are typically hidden away from\nthe user so that they are neither advertised nor writeable. At GitLab,\nwe have one particular repository that contains a total of 7 million\nreferences, of which 6.8 million are indeed internal references. With\nthe current connectivity check we are forced to load all these\nreferences in order to mark them as uninteresting, and this alone takes\naround 15 seconds to compute.\n\nWe can optimize this by only taking into account the set of visible refs\nwhen marking objects as uninteresting. This means that we may now walk\nmore objects until we hit any object that is marked as uninteresting.\nBut it is rather unlikely that clients send objects that make large\nparts of objects reachable that have previously only ever been hidden,\nwhereas the common case is to push incremental changes that build on top\nof the visible object graph.\n\nThis provides a huge boost to performance in the mentioned repository,\nwhere the vast majority of its refs hidden. Pushing a new commit into\nthis repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\nas it is configured in Gitaly leads to a 4.5-fold speedup:\n\n Benchmark 1: main\n Time (mean ± σ): 30.977 s ± 0.157 s [User: 30.226 s, System: 1.083 s]\n Range (min … max): 30.796 s … 31.071 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 6.799 s ± 0.063 s [User: 6.803 s, System: 0.354 s]\n Range (min … max): 6.729 s … 6.850 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 4.56 ± 0.05 times faster than 'main'\n\nAs we mostly go through the same codepaths even in the case where there\nare no hidden refs at all compared to the code before there is no change\nin performance when no refs are hidden:\n\n Benchmark 1: main\n Time (mean ± σ): 48.188 s ± 0.432 s [User: 49.326 s, System: 5.009 s]\n Range (min … max): 47.706 s … 48.539 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 48.027 s ± 0.500 s [User: 48.934 s, System: 5.025 s]\n Range (min … max): 47.504 s … 48.500 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 1.00 ± 0.01 times faster than 'main'\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 2 ++\n connected.c | 3 +++\n connected.h | 7 +++++++\n 3 files changed, 12 insertions(+)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1f3efc58fb..77ab40f123 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,\n \topt.err_fd = err_fd;\n \topt.progress = err_fd && !quiet;\n \topt.env = tmp_objdir_env(tmp_objdir);\n+\topt.exclude_hidden_refs_section = \"receive\";\n+\n \tif (check_connected(iterate_receive_command_list, &data, &opt))\n \t\tset_connectivity_errors(commands, si);\n \ndiff --git a/connected.c b/connected.c\nindex 74a20cb32e..4f6388eed7 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n \tif (!opt->is_deepening_fetch) {\n \t\tstrvec_push(&rev_list.args, \"--not\");\n+\t\tif (opt->exclude_hidden_refs_section)\n+\t\t\tstrvec_pushf(&rev_list.args, \"--exclude-hidden=%s\",\n+\t\t\t\t opt->exclude_hidden_refs_section);\n \t\tstrvec_push(&rev_list.args, \"--all\");\n \t}\n \tstrvec_push(&rev_list.args, \"--quiet\");\ndiff --git a/connected.h b/connected.h\nindex 6e59c92aa3..16b2c84f2e 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -46,6 +46,13 @@ struct check_connected_options {\n \t * during a fetch.\n \t */\n \tunsigned is_deepening_fetch : 1;\n+\n+\t/*\n+\t * If not NULL, use `--exclude-hidden=$section` to exclude all refs\n+\t * hidden via the `$section.hideRefs` config from the set of\n+\t * already-reachable refs.\n+\t */\n+\tconst char *exclude_hidden_refs_section;\n };\n \n #define CHECK_CONNECTED_INIT { 0 }\n-- \n2.38.1\n\n"},{"id":"466820","messageId":"221108.86leolsgze.gmgdl@evledraar.gmail.com","threadId":"58706","inReplyTo":"34afe30d60e4dbc7e50a590541440c20c27507a6.1667901510.git.ps@pks.im","subject":"Re: [PATCH v4 1/6] refs: get rid of global list of hidden refs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-11-08T13:36:04Z","receivedAt":"2022-11-08T13:39:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 08 2022, Patrick Steinhardt wrote:\n\n> @@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> \t\tpacket_flush(1);\n> \toid_array_clear(&shallow);\n> \toid_array_clear(&ref);\n> +\tstring_list_clear(&hidden_refs, 1);\n\nIn the v4 re-roll you got rid of the \"1\" for some other string_lists,\nbut is this one still needed, i.e. does it use \"util\"? At a glance it\ndoesn't seem so. There's another \"hidden_refs\" (maybe just semi-related)\nin 4/6 that doesn't use it when clearing.\n\n> diff --git a/refs.c b/refs.c\n> index 1491ae937e..2c7e88b190 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n> \t\t\t\t\t refname, strict);\n> }\n> \n> -static struct string_list *hide_refs;\n> -\n> -int parse_hide_refs_config(const char *var, const char *value, const char *section)\n> +int parse_hide_refs_config(const char *var, const char *value, const char *section,\n> +\t\t\t struct string_list *hide_refs)\n> {\n> \tconst char *key;\n> \tif (!strcmp(\"transfer.hiderefs\", var) ||\n> @@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n> \t\tlen = strlen(ref);\n> \t\twhile (len && ref[len - 1] == '/')\n> \t\t\tref[--len] = '\\0';\n> -\t\tif (!hide_refs) {\n> -\t\t\tCALLOC_ARRAY(hide_refs, 1);\n> -\t\t\thide_refs->strdup_strings = 1;\n> -\t\t}\n> -\t\tstring_list_append(hide_refs, ref);\n> +\t\tstring_list_append_nodup(hide_refs, ref);\n> \t}\n> \treturn 0;\n> }\n> \n\nAs before, this is all much nicer, thanks.\n"},{"id":"466838","messageId":"Y2pomzGndfhggXWz@coredump.intra.peff.net","threadId":"58706","inReplyTo":"Y2i/uJPm5KxqAdkE@ncase","subject":"Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-08T14:32:59Z","receivedAt":"2022-11-08T14:33:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 07, 2022 at 09:20:08AM +0100, Patrick Steinhardt wrote:\n\n> > 1. The mutual-exclusion selection between \"transfer\", \"uploadpack\",\n> > and \"receive\" is not how those options work in their respective\n> > [...]\n> \n> Yup, I'm aware of this. And as you say, the current implementation\n> already handles this alright for both `receive` and `uploadpack` as we\n> rely on `parse_hide_refs_config()`, which knows to look at both\n> `transfer.hideRefs` and `$section.hideRefs`. But I don't see a reason\n> why we shouldn't allow users to ask \"What is the set of hidden refs that\n> are shared by `uploadpack` and `receive`?\", which is exactly what\n> `--visible-refs=transfer` does.\n\nOK. I don't have a real problem with having that mode, as long as it is\ndocumented accurately. I do think, though, that it's not that likely to\nbe useful on its own (even less so than the hypotheticals I gave! ;) ),\nand it would be easy to add later in a backwards-compatible way. So my\ninstinct would be to leave it out for now, but I don't think it's\nhurting too much as-is (again, with a correct explanation in the\ndocumentation).\n\n> The implementation is not really explicit about this as we cheat a\n> little bit here by passing \"transfer\" as a section to the parsing\n> function. So what it does right now is to basically check for the same\n> section twice, once via the hard-coded \"transfer.hideRefs\" and once for\n> the \"$section.hideRefs\" with `section == \"transfer\"`. But I didn't see\n> much of a point in making this more explicit.\n\nYeah, I agree the implementation works OK here. It does a duplicate\nstring-comparison for the section, but the important thing is that it\ndoesn't add each entry to the list twice.\n\n> > Now I don't have a particular use case for either of those things.\n> > But they're plausible things to want in the long run, and they fit\n> > in nicely with the existing ref-selection scheme of rev-list. They\n> > do make your call from check_connected() slightly longer, but it is\n> > pretty negligible. It's \"--exclude-hidden=receive --all\" instead of\n> > \"--visible-refs=hidden\".\n> \n> Fair enough. I guess that the usecase where you e.g. only hide a subset\n> of branches via `hideRefs` is going to be rare, so in most cases you\n> don't gain much by modelling this so that you can `--exclude-hidden\n> --branches`. But as you rightfully point out, modelling it that way fits\n> neatly with the existing `--exclude` switch and is overall more\n> flexible. So there's not much of a reason to not do so.\n\nThanks. I agree these aren't incredibly likely cases to come up in\npractice. But unlike the \"transfer\" thing, it would be very hard to\nswitch techniques later without adding an awkward almost-the-same\noption.\n\n-Peff\n"},{"id":"466839","messageId":"Y2pq1q3VmXq6UEJk@coredump.intra.peff.net","threadId":"58706","inReplyTo":"Y2oQb12WwEh0bpxW@ncase","subject":"Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-08T14:42:30Z","receivedAt":"2022-11-08T14:42:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 08, 2022 at 09:16:47AM +0100, Patrick Steinhardt wrote:\n\n> > > +--exclude-hidden=[transfer|receive|uploadpack]::\n> > > +\tDo not include refs that have been hidden via either one of\n> > > +\t`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that\n> > > +\tthe next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would\n> > > +\totherwise consider. This option is cleared when seeing one of these\n> > > +\tpseudo-refs.\n> > > +\n> > \n> > Hmm. I thought that part of the motivation behind this round was to drop\n> > the 'transfer' group, since it's implied by '--exclude-hidden=receive\n> > --exclude-hidden-uploadpack', no?\n> > \n> > Thanks,\n> > Taylor\n> \n> I didn't quite see the point in not providing the `transfer` group so\n> that users can ask for only the set of refs that are hidden by both\n> `uploadpack` and `receive`. But given that you're the second person\n> asking for it to be dropped now and given that I don't really have a\n> plausible usecase for this I'll drop it in the next version.\n\nSorry, I'm a little slow on the review. I just left a message in\nresponse to v2 saying that I'm OK with it _if_ it's explained. But the\nexplanation above still seems misleading. Saying \"either one of\" implies\nthat they are mutually exclusive, but \"receive\" is really pulling from\n\"receive.hideRefs\" and \"transfer.hideRefs\".\n\nI think you'd need to lay out the rules. But if we just drop \"transfer\"\nthen that is simpler still (but the explanation probably still ought to\nbecome \"refs hidden by receive-pack\" and not mention receive.hideRefs\nexplicitly).\n\n-Peff\n"},{"id":"466841","messageId":"Y2prQqR2Ka3nHdWP@coredump.intra.peff.net","threadId":"58706","inReplyTo":"68a5e563045f1b3e6db7304206d95ebebcacbe6f.1667823042.git.ps@pks.im","subject":"Re: [PATCH v3 5/6] revparse: add `--exclude-hidden=` option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-08T14:44:18Z","receivedAt":"2022-11-08T14:46:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 07, 2022 at 01:16:39PM +0100, Patrick Steinhardt wrote:\n\n> Add a new `--exclude-hidden=` option that is similar to the one we just\n> added to git-rev-list(1). Given a seciton name `transfer`, `uploadpack`\n> or `receive` as argument, it causes us to exclude all references that\n> would be hidden by the respective `$seciton.hideRefs` configuration.\n\nThanks for adding this one in. I feel like rev-parse isn't used all that\nmuch these days, and in a sense, we could just let it fall behind what\nrev-list could do and probably nobody would care. But since it's only a\nfew lines, keeping parity with rev-list is nice to have.\n\n-Peff\n"},{"id":"466843","messageId":"Y2psfpS3rxg46SHV@ncase","threadId":"58706","inReplyTo":"221108.86leolsgze.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 1/6] refs: get rid of global list of hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-08T14:49:34Z","receivedAt":"2022-11-08T14:49:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 08, 2022 at 02:36:04PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Nov 08 2022, Patrick Steinhardt wrote:\n> \n> > @@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> > \t\tpacket_flush(1);\n> > \toid_array_clear(&shallow);\n> > \toid_array_clear(&ref);\n> > +\tstring_list_clear(&hidden_refs, 1);\n> \n> In the v4 re-roll you got rid of the \"1\" for some other string_lists,\n> but is this one still needed, i.e. does it use \"util\"? At a glance it\n> doesn't seem so. There's another \"hidden_refs\" (maybe just semi-related)\n> in 4/6 that doesn't use it when clearing.\n\nOh, right, I missed this one. Will wait a bit though for other feedback\nto come in before sending a v5 only with this one-line change.\n\nPatrick\n"},{"id":"466844","messageId":"Y2ps6Fpf52Ra68v3@coredump.intra.peff.net","threadId":"58706","inReplyTo":"34afe30d60e4dbc7e50a590541440c20c27507a6.1667901510.git.ps@pks.im","subject":"Re: [PATCH v4 1/6] refs: get rid of global list of hidden refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-08T14:51:20Z","receivedAt":"2022-11-08T14:51:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 08, 2022 at 11:03:39AM +0100, Patrick Steinhardt wrote:\n\n> -static struct string_list *hide_refs;\n> -\n> -int parse_hide_refs_config(const char *var, const char *value, const char *section)\n> +int parse_hide_refs_config(const char *var, const char *value, const char *section,\n> +\t\t\t struct string_list *hide_refs)\n> {\n> \tconst char *key;\n> \tif (!strcmp(\"transfer.hiderefs\", var) ||\n> @@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n> \t\tlen = strlen(ref);\n> \t\twhile (len && ref[len - 1] == '/')\n> \t\t\tref[--len] = '\\0';\n> -\t\tif (!hide_refs) {\n> -\t\t\tCALLOC_ARRAY(hide_refs, 1);\n> -\t\t\thide_refs->strdup_strings = 1;\n> -\t\t}\n> -\t\tstring_list_append(hide_refs, ref);\n> +\t\tstring_list_append_nodup(hide_refs, ref);\n> \t}\n> \treturn 0;\n> }\n\nThis nodup is definitely the right thing to be doing, but it's kind of\nhidden in here. AFAICT it is fixing an existing leak, because the\nprevious code always set strdup_strings, and we always made our own copy\nof \"ref\".\n\nProbably not worth a re-roll on its own, but I'd probably have pulled\nthat into its own commit.\n\nThe rest of the commit looks OK to me. Like Ævar, I'm confused by the\n\"free_util\" arguments to string_list_clear().\n\n-Peff\n"},{"id":"466846","messageId":"Y2pwt77e9KP5J3XD@coredump.intra.peff.net","threadId":"58706","inReplyTo":"c7fa6698dbe7b0eb5e2b9c4725e9bdc6cdb22983.1667901510.git.ps@pks.im","subject":"Re: [PATCH v4 4/6] revision: add new parameter to exclude hidden refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-08T15:07:35Z","receivedAt":"2022-11-08T15:07:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 08, 2022 at 11:03:51AM +0100, Patrick Steinhardt wrote:\n\n> --- a/Documentation/rev-list-options.txt\n> +++ b/Documentation/rev-list-options.txt\n> @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n> or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n> explicitly.\n> \n> +--exclude-hidden=[receive|uploadpack]::\n> +\tDo not include refs that have been hidden via either one of\n> +\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n> +\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n> +\twould otherwise consider. This option is cleared when seeing one of\n> +\tthese pseudo-refs.\n\nOK, so this one drops \"transfer\", which I think is good. But again, I\nthink the explanation here is subtly misleading, because it's not just\n\"hidden via those sections\", but \"hidden as receive-pack or upload-pack\nwould, respecting both those sections _and_ transfer\".\n\nI also wondered if we could simplify the explanation a bit by tying it\ninto the existing --exclude, and then we don't need that mouthful of\npseudo-ref options. So together something like:\n\n Do not include refs that would be hidden by `receive-pack` or\n `upload-pack`, by consulting the appropriate `receive.hideRefs` or\n `uploadpack.hideRefs`, along with `transfer.hideRefs`. Like\n `--exclude`, this option affects the next pseudo-ref option (`--all`,\n `--glob`, etc), and is cleared after processing them.\n\nBut then I read the next section in the --exclude docs, which says:\n\n The patterns given should not begin with refs/heads, refs/tags, or\n refs/remotes when applied to --branches, --tags, or --remotes,\n respectively, and they must begin with refs/ when applied to --glob or\n --all. If a trailing /* is intended, it must be given explicitly.\n\nYikes. So --all is going to process \"refs/heads/foo\", but --branches\nwill see just \"foo\". And that means that:\n\n git rev-list --exclude-hidden=receive --branches\n\nwill not work! Because receive.hideRefs is naming fully qualified refs,\nbut we'll pass unqualified ones to ref_is_hidden(). Even though that\nfunction takes a \"full\" refname argument, I think that has to do with\nnamespaces.\n\nI'm sure this _could_ be made to work, but I wonder if it is worth the\ntrouble. If it's not going to work, though, I think we'd want to detect\nthe situation and complain, at least for now. And likewise the\ndocumentation needs to make clear it only works with --all and --glob.\n\nSorry to have misled in my initial suggestion to turn --visible-refs\ninto --exclude-hidden. However, I do still stand by that suggestion.\nEven if we don't make it work with \"--branches\" now, the user-visible\nframework is still there, so it becomes a matter of extending the\nimplementation later, rather than re-designing the options.\n\n-Peff\n"},{"id":"466900","messageId":"Y2rGfjr9vPEx1JEs@nand.local","threadId":"58706","inReplyTo":"Y2pwt77e9KP5J3XD@coredump.intra.peff.net","subject":"Re: [PATCH v4 4/6] revision: add new parameter to exclude hidden refs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-08T21:13:34Z","receivedAt":"2022-11-08T21:13:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 08, 2022 at 10:07:35AM -0500, Jeff King wrote:\n> I'm sure this _could_ be made to work, but I wonder if it is worth the\n> trouble. If it's not going to work, though, I think we'd want to detect\n> the situation and complain, at least for now. And likewise the\n> documentation needs to make clear it only works with --all and --glob.\n>\n> Sorry to have misled in my initial suggestion to turn --visible-refs\n> into --exclude-hidden. However, I do still stand by that suggestion.\n> Even if we don't make it work with \"--branches\" now, the user-visible\n> framework is still there, so it becomes a matter of extending the\n> implementation later, rather than re-designing the options.\n\nGood catch. I agree completely, so hopefully we will see something to\nthis effect in the forthcoming v5.\n\nI'll hold off on merging this down until we see that version.\n\n\nThanks,\nTaylor\n"},{"id":"467121","messageId":"Y23iHhBYJclNzXn6@ncase","threadId":"58706","inReplyTo":"Y2pwt77e9KP5J3XD@coredump.intra.peff.net","subject":"Re: [PATCH v4 4/6] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T05:48:14Z","receivedAt":"2022-11-11T05:49:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 08, 2022 at 10:07:35AM -0500, Jeff King wrote:\n> On Tue, Nov 08, 2022 at 11:03:51AM +0100, Patrick Steinhardt wrote:\n> \n> > --- a/Documentation/rev-list-options.txt\n> > +++ b/Documentation/rev-list-options.txt\n> > @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n> > or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n> > explicitly.\n> > \n> > +--exclude-hidden=[receive|uploadpack]::\n> > +\tDo not include refs that have been hidden via either one of\n> > +\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n> > +\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n> > +\twould otherwise consider. This option is cleared when seeing one of\n> > +\tthese pseudo-refs.\n> \n> OK, so this one drops \"transfer\", which I think is good. But again, I\n> think the explanation here is subtly misleading, because it's not just\n> \"hidden via those sections\", but \"hidden as receive-pack or upload-pack\n> would, respecting both those sections _and_ transfer\".\n> \n> I also wondered if we could simplify the explanation a bit by tying it\n> into the existing --exclude, and then we don't need that mouthful of\n> pseudo-ref options. So together something like:\n> \n> Do not include refs that would be hidden by `receive-pack` or\n> `upload-pack`, by consulting the appropriate `receive.hideRefs` or\n> `uploadpack.hideRefs`, along with `transfer.hideRefs`. Like\n> `--exclude`, this option affects the next pseudo-ref option (`--all`,\n> `--glob`, etc), and is cleared after processing them.\n\nI'll try to come up with something, thanks.\n\n> But then I read the next section in the --exclude docs, which says:\n> \n> The patterns given should not begin with refs/heads, refs/tags, or\n> refs/remotes when applied to --branches, --tags, or --remotes,\n> respectively, and they must begin with refs/ when applied to --glob or\n> --all. If a trailing /* is intended, it must be given explicitly.\n> \n> Yikes. So --all is going to process \"refs/heads/foo\", but --branches\n> will see just \"foo\". And that means that:\n> \n> git rev-list --exclude-hidden=receive --branches\n> \n> will not work! Because receive.hideRefs is naming fully qualified refs,\n> but we'll pass unqualified ones to ref_is_hidden(). Even though that\n> function takes a \"full\" refname argument, I think that has to do with\n> namespaces.\n\nOh, good catch. I noticed that paragraph before, but somehow it didn't\nclick.\n\n> I'm sure this _could_ be made to work, but I wonder if it is worth the\n> trouble. If it's not going to work, though, I think we'd want to detect\n> the situation and complain, at least for now. And likewise the\n> documentation needs to make clear it only works with --all and --glob.\n> \n> Sorry to have misled in my initial suggestion to turn --visible-refs\n> into --exclude-hidden. However, I do still stand by that suggestion.\n> Even if we don't make it work with \"--branches\" now, the user-visible\n> framework is still there, so it becomes a matter of extending the\n> implementation later, rather than re-designing the options.\n\nNo worries, and I agree that it's ultimately still the better route to\nuse `--exclude-hidden=`. The things that don't work right now just go to\nshow that there is still some potential here and that it's overall the\nmore flexible design. And as you say, this is something we can build on\nat a later point if any such usecases come up.\n\nPatrick\n"},{"id":"467122","messageId":"cfab8ba1a20c8a178815723504b16a5f10b5c413.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"[PATCH v5 1/7] refs: fix memory leak when parsing hideRefs config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:49:54Z","receivedAt":"2022-11-11T06:50:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When parsing the hideRefs configuration, we first duplicate the config\nvalue so that we can modify it. We then subsequently append it to the\n`hide_refs` string list, which is initialized with `strdup_strings`\nenabled. As a consequence we again reallocate the string, but never\nfree the first duplicate and thus have a memory leak.\n\nWhile we never clean up the static `hide_refs` variable anyway, this is\nno excuse to make the leak worse by leaking every value twice. We are\nalso about to change the way this variable will be handled so that we do\nindeed start to clean it up. So let's fix the memory leak by using the\n`string_list_append_nodup()` so that we pass ownership of the allocated\nstring to `hide_refs`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 1491ae937e..a4ab264d74 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1435,7 +1435,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\t\tCALLOC_ARRAY(hide_refs, 1);\n \t\t\thide_refs->strdup_strings = 1;\n \t\t}\n-\t\tstring_list_append(hide_refs, ref);\n+\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n-- \n2.38.1\n\n"},{"id":"467123","messageId":"cover.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"[PATCH v5 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:49:49Z","receivedAt":"2022-11-11T06:50:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fifth version of my patch series that tries to improve\nperformance of the connectivity check by only considering preexisting\nrefs as uninteresting that could actually have been advertised to the\nclient.\n\nChanges compared to v4:\n\n - Split out the memory leak fix when parsing hidden refs into a\n separate commit 1/7.\n\n - Fixed calls to `string_list_clear()` to not free the `util` field\n of `hidden_refs`.\n\n - Updated the documentation of the new `--exclude-hidden=` option to\n hopefully be easier to understand.\n\n - We now return an error when `--exclude-hidden=` is used with\n either one of `--branches`, `--tags` or `--remotes`.\n\n - Fixed a bug where we didn't bail when `--exclude-hidden=` was\n passed multiple times when there are no hidden refs.\n\n - Extended test coverage.\n\nPatrick\n\nPatrick Steinhardt (7):\n refs: fix memory leak when parsing hideRefs config\n refs: get rid of global list of hidden refs\n revision: move together exclusion-related functions\n revision: introduce struct to handle exclusions\n revision: add new parameter to exclude hidden refs\n rev-parse: add `--exclude-hidden=` option\n receive-pack: only use visible refs for connectivity check\n\n Documentation/git-rev-parse.txt | 7 ++\n Documentation/rev-list-options.txt | 7 ++\n builtin/receive-pack.c | 10 +-\n builtin/rev-list.c | 1 +\n builtin/rev-parse.c | 18 +++-\n connected.c | 3 +\n connected.h | 7 ++\n ls-refs.c | 13 ++-\n refs.c | 16 +--\n refs.h | 5 +-\n revision.c | 131 +++++++++++++++--------\n revision.h | 43 ++++++--\n t/t6018-rev-list-glob.sh | 40 +++++++\n t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++\n upload-pack.c | 30 +++---\n 15 files changed, 411 insertions(+), 83 deletions(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\nRange-diff against v4:\n-: ---------- > 1: cfab8ba1a2 refs: fix memory leak when parsing hideRefs config\n1: 34afe30d60 ! 2: d8118c6dd8 refs: get rid of global list of hidden refs\n @@ builtin/receive-pack.c: int cmd_receive_pack(int argc, const char **argv, const\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n -+\tstring_list_clear(&hidden_refs, 1);\n ++\tstring_list_clear(&hidden_refs, 0);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\n @@ ls-refs.c: int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n -+\tstring_list_clear(&data.hidden_refs, 1);\n ++\tstring_list_clear(&data.hidden_refs, 0);\n \treturn 0;\n }\n \n @@ refs.c: int parse_hide_refs_config(const char *var, const char *value, const cha\n -\t\t\tCALLOC_ARRAY(hide_refs, 1);\n -\t\t\thide_refs->strdup_strings = 1;\n -\t\t}\n --\t\tstring_list_append(hide_refs, ref);\n -+\t\tstring_list_append_nodup(hide_refs, ref);\n + \t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n @@ upload-pack.c: static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n -+\tstring_list_clear(&data->hidden_refs, 1);\n ++\tstring_list_clear(&data->hidden_refs, 0);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n2: b4f21d0a80 = 3: 93a627fb7f revision: move together exclusion-related functions\n3: 265b292ed5 = 4: ad41ade332 revision: introduce struct to handle exclusions\n4: c7fa6698db ! 5: b5a4ce432a revision: add new parameter to exclude hidden refs\n @@ Documentation/rev-list-options.txt: respectively, and they must begin with `refs\n explicitly.\n \n +--exclude-hidden=[receive|uploadpack]::\n -+\tDo not include refs that have been hidden via either one of\n -+\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n -+\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n -+\twould otherwise consider. This option is cleared when seeing one of\n -+\tthese pseudo-refs.\n ++\tDo not include refs that would be hidden by `git-receive-pack` or\n ++\t`git-upload-pack` by consulting the appropriate `receive.hideRefs` or\n ++\t`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see\n ++\tlinkgit:git-config[1]). This option affects the next pseudo-ref option\n ++\t`--all` or `--glob` and is cleared after processing them.\n +\n --reflog::\n \tPretend as if all objects mentioned by reflogs are listed on the\n @@ revision.c: void init_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n +\tstring_list_clear(&exclusions->hidden_refs, 0);\n ++\texclusions->hidden_refs_configured = 0;\n }\n \n void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n @@ revision.c: void add_ref_exclusion(struct ref_exclusions *exclusions, const char\n +static int hide_refs_config(const char *var, const char *value, void *cb_data)\n +{\n +\tstruct exclude_hidden_refs_cb *cb = cb_data;\n ++\tcb->exclusions->hidden_refs_configured = 1;\n +\treturn parse_hide_refs_config(var, value, cb->section,\n +\t\t\t\t &cb->exclusions->hidden_refs);\n +}\n @@ revision.c: void add_ref_exclusion(struct ref_exclusions *exclusions, const char\n +\tif (strcmp(section, \"receive\") && strcmp(section, \"uploadpack\"))\n +\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n +\n -+\tif (exclusions->hidden_refs.nr)\n ++\tif (exclusions->hidden_refs_configured)\n +\t\tdie(_(\"--exclude-hidden= passed more than once\"));\n +\n +\tcb.exclusions = exclusions;\n @@ revision.c: static int handle_revision_opt(struct rev_info *revs, int argc, cons\n \t starts_with(arg, \"--branches=\") || starts_with(arg, \"--tags=\") ||\n \t starts_with(arg, \"--remotes=\") || starts_with(arg, \"--no-walk=\"))\n \t{\n +@@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,\n + \t\t}\n + \t\tclear_ref_exclusions(&revs->ref_excludes);\n + \t} else if (!strcmp(arg, \"--branches\")) {\n ++\t\tif (revs->ref_excludes.hidden_refs_configured)\n ++\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n + \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n + \t\tclear_ref_exclusions(&revs->ref_excludes);\n + \t} else if (!strcmp(arg, \"--bisect\")) {\n +@@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,\n + \t\t\t for_each_good_bisect_ref);\n + \t\trevs->bisect = 1;\n + \t} else if (!strcmp(arg, \"--tags\")) {\n ++\t\tif (revs->ref_excludes.hidden_refs_configured)\n ++\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n + \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n + \t\tclear_ref_exclusions(&revs->ref_excludes);\n + \t} else if (!strcmp(arg, \"--remotes\")) {\n ++\t\tif (revs->ref_excludes.hidden_refs_configured)\n ++\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n + \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n + \t\tclear_ref_exclusions(&revs->ref_excludes);\n + \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n @@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n @@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,\n +\t\treturn argcount;\n \t} else if (skip_prefix(arg, \"--branches=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n ++\t\tif (revs->ref_excludes.hidden_refs_configured)\n ++\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n + \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n + \t\tclear_ref_exclusions(&revs->ref_excludes);\n + \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n + \t\tstruct all_refs_cb cb;\n ++\t\tif (revs->ref_excludes.hidden_refs_configured)\n ++\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n + \t\tinit_all_refs_cb(&cb, revs, *flags);\n + \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n + \t\tclear_ref_exclusions(&revs->ref_excludes);\n + \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n + \t\tstruct all_refs_cb cb;\n ++\t\tif (revs->ref_excludes.hidden_refs_configured)\n ++\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n + \t\tinit_all_refs_cb(&cb, revs, *flags);\n + \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n + \t\tclear_ref_exclusions(&revs->ref_excludes);\n \n ## revision.h ##\n @@ revision.h: struct ref_exclusions {\n @@ revision.h: struct ref_exclusions {\n +\t * `ref_is_hidden()`.\n +\t */\n +\tstruct string_list hidden_refs;\n ++\n ++\t/*\n ++\t * Indicates whether hidden refs have been configured. This is to\n ++\t * distinguish between no hidden refs existing and hidden refs not\n ++\t * being parsed.\n ++\t */\n ++\tchar hidden_refs_configured;\n };\n \n /**\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +do\n +\ttest_expect_success \"$section: passed multiple times\" '\n +\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n -+\t\ttest_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n ++\t\ttest_must_fail git rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n +\t\ttest_cmp expected err\n +\t'\n +\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +\t\ttest_cmp expected out\n +\t'\n +\n ++\ttest_expect_success \"$section: excluded hidden refs can be used with multiple pseudo-refs\" '\n ++\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&\n ++\t\ttest_must_be_empty out\n ++\t'\n ++\n ++\ttest_expect_success \"$section: works with --glob\" '\n ++\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --glob=refs/h* >out &&\n ++\t\tcat >expected <<-EOF &&\n ++\t\t$COMMIT\n ++\t\tEOF\n ++\t\ttest_cmp expected out\n ++\t'\n ++\n +\ttest_expect_success \"$section: operates on stripped refs by default\" '\n +\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&\n +\t\tcat >expected <<-EOF &&\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +\t\tEOF\n +\t\ttest_cmp expected out\n +\t'\n ++\n ++\tfor pseudoopt in remotes branches tags\n ++\tdo\n ++\t\ttest_expect_success \"$section: fails with --$pseudoopt\" '\n ++\t\t\ttest_must_fail git rev-list --exclude-hidden=$section --$pseudoopt 2>err &&\n ++\t\t\ttest_i18ngrep \"error: --exclude-hidden cannot be used together with --$pseudoopt\" err\n ++\t\t'\n ++\n ++\t\ttest_expect_success \"$section: fails with --$pseudoopt=pattern\" '\n ++\t\t\ttest_must_fail git rev-list --exclude-hidden=$section --$pseudoopt=pattern 2>err &&\n ++\t\t\ttest_i18ngrep \"error: --exclude-hidden cannot be used together with --$pseudoopt\" err\n ++\t\t'\n ++\tdone\n +done\n +\n +test_done\n5: 79c5c64a80 ! 6: 2eeb25eef0 rev-parse: add `--exclude-hidden=` option\n @@ Documentation/git-rev-parse.txt: respectively, and they must begin with `refs/`\n explicitly.\n \n +--exclude-hidden=[receive|uploadpack]::\n -+\tDo not include refs that have been hidden via either one of\n -+\t`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])\n -+\tthat the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`\n -+\twould otherwise consider. This option is cleared when seeing one of\n -+\tthese pseudo-refs.\n ++\tDo not include refs that would be hidden by `git-receive-pack` or\n ++\t`git-upload-pack` by consulting the appropriate `receive.hideRefs` or\n ++\t`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see\n ++\tlinkgit:git-config[1]). This option affects the next pseudo-ref option\n ++\t`--all` or `--glob` and is cleared after processing them.\n +\n --disambiguate=::\n \tShow every object whose name begins with the given prefix.\n \tThe must be at least 4 hexadecimal digits long to\n \n ## builtin/rev-parse.c ##\n +@@ builtin/rev-parse.c: int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n + \t\t\t\tcontinue;\n + \t\t\t}\n + \t\t\tif (opt_with_value(arg, \"--branches\", &arg)) {\n ++\t\t\t\tif (ref_excludes.hidden_refs_configured)\n ++\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n + \t\t\t\thandle_ref_opt(arg, \"refs/heads/\");\n + \t\t\t\tcontinue;\n + \t\t\t}\n + \t\t\tif (opt_with_value(arg, \"--tags\", &arg)) {\n ++\t\t\t\tif (ref_excludes.hidden_refs_configured)\n ++\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n + \t\t\t\thandle_ref_opt(arg, \"refs/tags/\");\n + \t\t\t\tcontinue;\n + \t\t\t}\n +@@ builtin/rev-parse.c: int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n + \t\t\t\tcontinue;\n + \t\t\t}\n + \t\t\tif (opt_with_value(arg, \"--remotes\", &arg)) {\n ++\t\t\t\tif (ref_excludes.hidden_refs_configured)\n ++\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n + \t\t\t\thandle_ref_opt(arg, \"refs/remotes/\");\n + \t\t\t\tcontinue;\n + \t\t\t}\n @@ builtin/rev-parse.c: int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tadd_ref_exclusion(&ref_excludes, arg);\n \t\t\t\tcontinue;\n @@ t/t6018-rev-list-glob.sh: test_expect_success 'rev-parse --exclude=ref with --re\n +for section in receive uploadpack\n +do\n +\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n -+\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--exclude-hidden=$section --all\" \"--branches --tags\"\n ++\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--branches --tags\" \"--exclude-hidden=$section --all\"\n +\t'\n +\n +\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n -+\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude-hidden=$section --all\" \"--exclude=refs/heads/subspace/* --all\"\n ++\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude=refs/heads/subspace/* --all\" \"--exclude-hidden=$section --all\"\n +\t'\n ++\n ++\ttest_expect_success \"rev-parse --exclude-hidden=$section with --glob\" '\n ++\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude=refs/heads/subspace/* --glob=refs/heads/*\" \"--exclude-hidden=$section --glob=refs/heads/*\"\n ++\t'\n ++\n ++\ttest_expect_success \"rev-parse --exclude-hidden=$section can be passed once per pseudo-ref\" '\n ++\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--branches --tags --branches --tags\" \"--exclude-hidden=$section --all --exclude-hidden=$section --all\"\n ++\t'\n ++\n ++\ttest_expect_success \"rev-parse --exclude-hidden=$section can only be passed once per pseudo-ref\" '\n ++\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n ++\t\ttest_must_fail git rev-parse --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n ++\t\ttest_cmp expected err\n ++\t'\n ++\n ++\tfor pseudoopt in branches tags remotes\n ++\tdo\n ++\t\ttest_expect_success \"rev-parse --exclude-hidden=$section fails with --$pseudoopt\" '\n ++\t\t\techo \"error: --exclude-hidden cannot be used together with --$pseudoopt\" >expected &&\n ++\t\t\ttest_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt 2>err &&\n ++\t\t\ttest_cmp expected err\n ++\t\t'\n ++\n ++\t\ttest_expect_success \"rev-parse --exclude-hidden=$section fails with --$pseudoopt=pattern\" '\n ++\t\t\techo \"error: --exclude-hidden cannot be used together with --$pseudoopt\" >expected &&\n ++\t\t\ttest_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt=pattern 2>err &&\n ++\t\t\ttest_cmp expected err\n ++\t\t'\n ++\tdone\n +done\n +\n test_expect_success 'rev-list --exclude=glob with --branches=glob' '\n6: 39b4741734 = 7: f5f18f3939 receive-pack: only use visible refs for connectivity check\n-- \n2.38.1\n\n"},{"id":"467124","messageId":"d8118c6dd8a97f923e629490061d883557a92861.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"[PATCH v5 2/7] refs: get rid of global list of hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:49:58Z","receivedAt":"2022-11-11T06:50:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to add a new argument to git-rev-list(1) that allows it to\nadd all references that are visible when taking `transfer.hideRefs` et\nal into account. This will require us to potentially parse multiple sets\nof hidden refs, which is not easily possible right now as there is only\na single, global instance of the list of parsed hidden refs.\n\nRefactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both\ntake the list of hidden references as input and adjust callers to keep a\nlocal list, instead. This allows us to easily use multiple hidden-ref\nlists. Furthermore, it allows us to properly free this list before we\nexit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 8 +++++---\n ls-refs.c | 13 +++++++++----\n refs.c | 14 ++++----------\n refs.h | 5 +++--\n upload-pack.c | 30 ++++++++++++++++++------------\n 5 files changed, 39 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 44bcea3a5b..1e24b31a0a 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)\n \n static int receive_pack_config(const char *var, const char *value, void *cb)\n {\n-\tint status = parse_hide_refs_config(var, value, \"receive\");\n+\tint status = parse_hide_refs_config(var, value, \"receive\", &hidden_refs);\n \n \tif (status)\n \t\treturn status;\n@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,\n \tstruct oidset *seen = data;\n \tconst char *path = strip_namespace(path_full);\n \n-\tif (ref_is_hidden(path, path_full))\n+\tif (ref_is_hidden(path, path_full, &hidden_refs))\n \t\treturn 0;\n \n \t/*\n@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)\n \t\tstrbuf_setlen(&refname_full, prefix_len);\n \t\tstrbuf_addstr(&refname_full, cmd->ref_name);\n \n-\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf))\n+\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))\n \t\t\tcontinue;\n \t\tif (is_null_oid(&cmd->new_oid))\n \t\t\tcmd->error_string = \"deny deleting a hidden ref\";\n@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n+\tstring_list_clear(&hidden_refs, 0);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex fa0d01b47c..fb6769742c 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -6,6 +6,7 @@\n #include \"ls-refs.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"string-list.h\"\n \n static int config_read;\n static int advertise_unborn;\n@@ -73,6 +74,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n+\tstruct string_list hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \n \tstrbuf_reset(&data->buf);\n \n-\tif (ref_is_hidden(refname_nons, refname))\n+\tif (ref_is_hidden(refname_nons, refname, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (!ref_match(&data->prefixes, refname_nons))\n@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)\n }\n \n static int ls_refs_config(const char *var, const char *value,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *cb_data)\n {\n+\tstruct ls_refs_data *data = cb_data;\n \t/*\n \t * We only serve fetches over v2 for now, so respect only \"uploadpack\"\n \t * config. This may need to eventually be expanded to \"receive\", but we\n \t * don't yet know how that information will be passed to ls-refs.\n \t */\n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n int ls_refs(struct repository *r, struct packet_reader *request)\n@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n+\tstring_list_init_dup(&data.hidden_refs);\n \n \tensure_config_read();\n-\tgit_config(ls_refs_config, NULL);\n+\tgit_config(ls_refs_config, &data);\n \n \twhile (packet_reader_read(request) == PACKET_READ_NORMAL) {\n \t\tconst char *arg = request->line;\n@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n+\tstring_list_clear(&data.hidden_refs, 0);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex a4ab264d74..2c7e88b190 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n \t\t\t\t\t refname, strict);\n }\n \n-static struct string_list *hide_refs;\n-\n-int parse_hide_refs_config(const char *var, const char *value, const char *section)\n+int parse_hide_refs_config(const char *var, const char *value, const char *section,\n+\t\t\t struct string_list *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tif (!hide_refs) {\n-\t\t\tCALLOC_ARRAY(hide_refs, 1);\n-\t\t\thide_refs->strdup_strings = 1;\n-\t\t}\n \t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n-int ref_is_hidden(const char *refname, const char *refname_full)\n+int ref_is_hidden(const char *refname, const char *refname_full,\n+\t\t const struct string_list *hide_refs)\n {\n \tint i;\n \n-\tif (!hide_refs)\n-\t\treturn 0;\n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n \t\tconst char *match = hide_refs->items[i].string;\n \t\tconst char *subject;\ndiff --git a/refs.h b/refs.h\nindex 8958717a17..3266fd8f57 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,\n \t const struct object_id *new_oid, const struct object_id *old_oid,\n \t unsigned int flags, enum action_on_err onerr);\n \n-int parse_hide_refs_config(const char *var, const char *value, const char *);\n+int parse_hide_refs_config(const char *var, const char *value, const char *,\n+\t\t\t struct string_list *);\n \n /*\n * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);\n * the ref is outside that namespace, the first parameter is NULL. The second\n * parameter always points to the full ref name.\n */\n-int ref_is_hidden(const char *, const char *);\n+int ref_is_hidden(const char *, const char *, const struct string_list *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0b8311bd68..551f22ffa5 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -62,6 +62,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n+\tstruct string_list hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n+\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n \tmemset(data, 0, sizeof(*data));\n \tdata->symref = symref;\n \tdata->wanted_refs = wanted_refs;\n+\tdata->hidden_refs = hidden_refs;\n \tdata->want_obj = want_obj;\n \tdata->have_obj = have_obj;\n \tdata->haves = haves;\n@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n+\tstring_list_clear(&data->hidden_refs, 0);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * Checking for reachable shallows requires that our refs be\n \t\t * marked with OUR_REF.\n \t\t */\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, data);\n+\t\tfor_each_namespaced_ref(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid)\n+\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n-\tif (ref_is_hidden(refname, refname_full)) {\n+\tif (ref_is_hidden(refname, refname_full, hidden_refs)) {\n \t\to->flags |= HIDDEN_REF;\n \t\treturn 1;\n \t}\n@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,\n }\n \n static int check_ref(const char *refname_full, const struct object_id *oid,\n-\t\t int flag UNUSED, void *cb_data UNUSED)\n+\t\t int flag UNUSED, void *cb_data)\n {\n \tconst char *refname = strip_namespace(refname_full);\n+\tstruct upload_pack_data *data = cb_data;\n \n-\tmark_our_ref(refname, refname_full, oid);\n+\tmark_our_ref(refname, refname_full, oid, &data->hidden_refs);\n \treturn 0;\n }\n \n@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \tstruct object_id peeled;\n \tstruct upload_pack_data *data = cb_data;\n \n-\tif (mark_our_ref(refname_nons, refname, oid))\n+\tif (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (capabilities) {\n@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)\n \tif (parse_object_filter_config(var, value, data) < 0)\n \t\treturn -1;\n \n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)\n@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, &data);\n+\t\tfor_each_namespaced_ref(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t struct string_list *wanted_refs,\n+\t\t\t struct string_list *hidden_refs,\n \t\t\t struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\tstruct strbuf refname = STRBUF_INIT;\n \n \t\tstrbuf_addf(&refname, \"%s%s\", get_git_namespace(), refname_nons);\n-\t\tif (ref_is_hidden(refname_nons, refname.buf) ||\n+\t\tif (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||\n \t\t read_ref(refname.buf, &oid)) {\n \t\t\tpacket_writer_error(writer, \"unknown ref %s\", refname_nons);\n \t\t\tdie(\"unknown ref %s\", refname_nons);\n@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (data->allow_ref_in_want &&\n \t\t parse_want_ref(&data->writer, arg, &data->wanted_refs,\n-\t\t\t\t &data->want_obj))\n+\t\t\t\t &data->hidden_refs, &data->want_obj))\n \t\t\tcontinue;\n \t\t/* process have line */\n \t\tif (parse_have(arg, &data->haves))\n-- \n2.38.1\n\n"},{"id":"467125","messageId":"93a627fb7f79610f67a7eefb29aac6191ea201b0.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"[PATCH v5 3/7] revision: move together exclusion-related functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:50:02Z","receivedAt":"2022-11-11T06:50:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move together the definitions of functions that handle exclusions of\nrefs so that related functionality sits in a single place, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 52 ++++++++++++++++++++++++++--------------------------\n 1 file changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 0760e78936..be755670e2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-struct all_refs_cb {\n-\tint all_flags;\n-\tint warned_bad_reflog;\n-\tstruct rev_info *all_revs;\n-\tconst char *name_for_errormsg;\n-\tstruct worktree *wt;\n-};\n-\n int ref_excluded(struct string_list *ref_excludes, const char *path)\n {\n \tstruct string_list_item *item;\n@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)\n \treturn 0;\n }\n \n+void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+{\n+\tif (*ref_excludes_p) {\n+\t\tstring_list_clear(*ref_excludes_p, 0);\n+\t\tfree(*ref_excludes_p);\n+\t}\n+\t*ref_excludes_p = NULL;\n+}\n+\n+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+{\n+\tif (!*ref_excludes_p) {\n+\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n+\t\t(*ref_excludes_p)->strdup_strings = 1;\n+\t}\n+\tstring_list_append(*ref_excludes_p, exclude);\n+}\n+\n+struct all_refs_cb {\n+\tint all_flags;\n+\tint warned_bad_reflog;\n+\tstruct rev_info *all_revs;\n+\tconst char *name_for_errormsg;\n+\tstruct worktree *wt;\n+};\n+\n static int handle_one_ref(const char *path, const struct object_id *oid,\n \t\t\t int flag UNUSED,\n \t\t\t void *cb_data)\n@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,\n \tcb->wt = NULL;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n-{\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n-}\n-\n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n-{\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n-}\n-\n static void handle_refs(struct ref_store *refs,\n \t\t\tstruct rev_info *revs, unsigned flags,\n \t\t\tint (*for_each)(struct ref_store *, each_ref_fn, void *))\n-- \n2.38.1\n\n"},{"id":"467126","messageId":"ad41ade33216bc6b977b93a13557991bc02a7605.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"[PATCH v5 4/7] revision: introduce struct to handle exclusions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:50:06Z","receivedAt":"2022-11-11T06:50:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions that handle exclusion of refs work on a single string\nlist. We're about to add a second mechanism for excluding refs though,\nand it makes sense to reuse much of the same architecture for both kinds\nof exclusion.\n\nIntroduce a new `struct ref_exclusions` that encapsulates all the logic\nrelated to excluding refs and move the `struct string_list` that holds\nall wildmatch patterns of excluded refs into it. Rename functions that\noperate on this struct to match its name.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/rev-parse.c | 8 ++++----\n revision.c | 48 +++++++++++++++++++++------------------------\n revision.h | 27 +++++++++++++++++++------\n 3 files changed, 47 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 8f61050bde..7fa5b6991b 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -39,7 +39,7 @@ static int abbrev_ref_strict;\n static int output_sq;\n \n static int stuck_long;\n-static struct string_list *ref_excludes;\n+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;\n \n /*\n * Some arguments are relevant \"revision\" arguments,\n@@ -198,7 +198,7 @@ static int show_default(void)\n static int show_reference(const char *refname, const struct object_id *oid,\n \t\t\t int flag UNUSED, void *cb_data UNUSED)\n {\n-\tif (ref_excluded(ref_excludes, refname))\n+\tif (ref_excluded(&ref_excludes, refname))\n \t\treturn 0;\n \tshow_rev(NORMAL, oid, refname);\n \treturn 0;\n@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)\n \t\tfor_each_glob_ref_in(show_reference, pattern, prefix, NULL);\n \telse\n \t\tfor_each_ref_in(prefix, show_reference, NULL);\n-\tclear_ref_exclusion(&ref_excludes);\n+\tclear_ref_exclusions(&ref_excludes);\n }\n \n enum format_type {\n@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--all\")) {\n \t\t\t\tfor_each_ref(show_reference, NULL);\n-\t\t\t\tclear_ref_exclusion(&ref_excludes);\n+\t\t\t\tclear_ref_exclusions(&ref_excludes);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (skip_prefix(arg, \"--disambiguate=\", &arg)) {\ndiff --git a/revision.c b/revision.c\nindex be755670e2..fe3ec98f46 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,35 +1517,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-int ref_excluded(struct string_list *ref_excludes, const char *path)\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n \tstruct string_list_item *item;\n-\n-\tif (!ref_excludes)\n-\t\treturn 0;\n-\tfor_each_string_list_item(item, ref_excludes) {\n+\tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+void init_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n+\tstruct ref_exclusions blank = REF_EXCLUSIONS_INIT;\n+\tmemcpy(exclusions, &blank, sizeof(*exclusions));\n }\n \n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n+\tstring_list_clear(&exclusions->excluded_refs, 0);\n+}\n+\n+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n+{\n+\tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n struct all_refs_cb {\n@@ -1563,7 +1558,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \tstruct all_refs_cb *cb = cb_data;\n \tstruct object *object;\n \n-\tif (ref_excluded(cb->all_revs->ref_excludes, path))\n+\tif (ref_excluded(&cb->all_revs->ref_excludes, path))\n \t return 0;\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n@@ -1901,6 +1896,7 @@ void repo_init_revisions(struct repository *r,\n \n \tinit_display_notes(&revs->notes_opt);\n \tlist_objects_filter_init(&revs->filter);\n+\tinit_ref_exclusions(&revs->ref_excludes);\n }\n \n static void add_pending_commit_list(struct rev_info *revs,\n@@ -2689,10 +2685,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\t\tother_head_refs(handle_one_ref, &cb);\n \t\t}\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--branches\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--bisect\")) {\n \t\tread_bisect_terms(&term_bad, &term_good);\n \t\thandle_refs(refs, revs, *flags, for_each_bad_bisect_ref);\n@@ -2701,15 +2697,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->bisect = 1;\n \t} else if (!strcmp(arg, \"--tags\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--remotes\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref(handle_one_ref, optarg, &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t\treturn argcount;\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n@@ -2718,17 +2714,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--reflog\")) {\n \t\tadd_reflogs_to_pending(revs, *flags);\n \t} else if (!strcmp(arg, \"--indexed-objects\")) {\ndiff --git a/revision.h b/revision.h\nindex afe1b77985..5c8ab16047 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -81,6 +81,21 @@ struct rev_cmdline_info {\n \t} *rev;\n };\n \n+struct ref_exclusions {\n+\t/*\n+\t * Excluded refs is a list of wildmatch patterns. If any of the\n+\t * patterns matches, the reference will be excluded.\n+\t */\n+\tstruct string_list excluded_refs;\n+};\n+\n+/**\n+ * Initialize a `struct ref_exclusions` with a macro.\n+ */\n+#define REF_EXCLUSIONS_INIT { \\\n+\t.excluded_refs = STRING_LIST_INIT_DUP, \\\n+}\n+\n struct oidset;\n struct topo_walk_info;\n \n@@ -103,7 +118,7 @@ struct rev_info {\n \tstruct list_objects_filter_options filter;\n \n \t/* excluding from --branches, --refs, etc. expansion */\n-\tstruct string_list *ref_excludes;\n+\tstruct ref_exclusions ref_excludes;\n \n \t/* Basic information */\n \tconst char *prefix;\n@@ -439,12 +454,12 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)\n void show_object_with_name(FILE *, struct object *, const char *);\n \n /**\n- * Helpers to check if a \"struct string_list\" item matches with\n- * wildmatch().\n+ * Helpers to check if a reference should be excluded.\n */\n-int ref_excluded(struct string_list *, const char *path);\n-void clear_ref_exclusion(struct string_list **);\n-void add_ref_exclusion(struct string_list **, const char *exclude);\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n+void init_ref_exclusions(struct ref_exclusions *);\n+void clear_ref_exclusions(struct ref_exclusions *);\n+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n \n /**\n * This function can be used if you want to add commit objects as revision\n-- \n2.38.1\n\n"},{"id":"467127","messageId":"b5a4ce432a97b8af6cb854aaf4aecb12dda95612.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"[PATCH v5 5/7] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:50:10Z","receivedAt":"2022-11-11T06:50:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Users can optionally hide refs from remote users in git-upload-pack(1),\ngit-receive-pack(1) and others via the `transfer.hideRefs`, but there is\nnot an easy way to obtain the list of all visible or hidden refs right\nnow. We'll require just that though for a performance improvement in our\nconnectivity check.\n\nAdd a new option `--exclude-hidden=` that excludes any hidden refs from\nthe next pseudo-ref like `--all` or `--branches`.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 7 ++\n builtin/rev-list.c | 1 +\n revision.c | 55 +++++++++-\n revision.h | 16 +++\n t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++\n 5 files changed, 241 insertions(+), 1 deletion(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 1837509566..ff68e48406 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[receive|uploadpack]::\n+\tDo not include refs that would be hidden by `git-receive-pack` or\n+\t`git-upload-pack` by consulting the appropriate `receive.hideRefs` or\n+\t`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see\n+\tlinkgit:git-config[1]). This option affects the next pseudo-ref option\n+\t`--all` or `--glob` and is cleared after processing them.\n+\n --reflog::\n \tPretend as if all objects mentioned by reflogs are listed on the\n \tcommand line as ``.\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3acd93f71e..d42db0b0cc 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -38,6 +38,7 @@ static const char rev_list_usage[] =\n \" --tags\\n\"\n \" --remotes\\n\"\n \" --stdin\\n\"\n+\" --exclude-hidden=[receive|uploadpack]\\n\"\n \" --quiet\\n\"\n \" ordering output:\\n\"\n \" --topo-order\\n\"\ndiff --git a/revision.c b/revision.c\nindex fe3ec98f46..bc32fb819a 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"object-store.h\"\n #include \"tag.h\"\n #include \"blob.h\"\n@@ -1519,11 +1520,17 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \n int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n+\tconst char *stripped_path = strip_namespace(path);\n \tstruct string_list_item *item;\n+\n \tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n+\n+\tif (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))\n+\t\treturn 1;\n+\n \treturn 0;\n }\n \n@@ -1536,6 +1543,8 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n+\tstring_list_clear(&exclusions->hidden_refs, 0);\n+\texclusions->hidden_refs_configured = 0;\n }\n \n void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n@@ -1543,6 +1552,35 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n \tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n+struct exclude_hidden_refs_cb {\n+\tstruct ref_exclusions *exclusions;\n+\tconst char *section;\n+};\n+\n+static int hide_refs_config(const char *var, const char *value, void *cb_data)\n+{\n+\tstruct exclude_hidden_refs_cb *cb = cb_data;\n+\tcb->exclusions->hidden_refs_configured = 1;\n+\treturn parse_hide_refs_config(var, value, cb->section,\n+\t\t\t\t &cb->exclusions->hidden_refs);\n+}\n+\n+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)\n+{\n+\tstruct exclude_hidden_refs_cb cb;\n+\n+\tif (strcmp(section, \"receive\") && strcmp(section, \"uploadpack\"))\n+\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n+\n+\tif (exclusions->hidden_refs_configured)\n+\t\tdie(_(\"--exclude-hidden= passed more than once\"));\n+\n+\tcb.exclusions = exclusions;\n+\tcb.section = section;\n+\n+\tgit_config(hide_refs_config, &cb);\n+}\n+\n struct all_refs_cb {\n \tint all_flags;\n \tint warned_bad_reflog;\n@@ -2221,7 +2259,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t !strcmp(arg, \"--bisect\") || starts_with(arg, \"--glob=\") ||\n \t !strcmp(arg, \"--indexed-objects\") ||\n \t !strcmp(arg, \"--alternate-refs\") ||\n-\t starts_with(arg, \"--exclude=\") ||\n+\t starts_with(arg, \"--exclude=\") || starts_with(arg, \"--exclude-hidden=\") ||\n \t starts_with(arg, \"--branches=\") || starts_with(arg, \"--tags=\") ||\n \t starts_with(arg, \"--remotes=\") || starts_with(arg, \"--no-walk=\"))\n \t{\n@@ -2687,6 +2725,8 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t}\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--branches\")) {\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--bisect\")) {\n@@ -2696,9 +2736,13 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t\t for_each_good_bisect_ref);\n \t\trevs->bisect = 1;\n \t} else if (!strcmp(arg, \"--tags\")) {\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--remotes\")) {\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n@@ -2710,18 +2754,27 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n \t\treturn argcount;\n+\t} else if ((argcount = parse_long_opt(\"exclude-hidden\", argv, &optarg))) {\n+\t\texclude_hidden_refs(&revs->ref_excludes, optarg);\n+\t\treturn argcount;\n \t} else if (skip_prefix(arg, \"--branches=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\ndiff --git a/revision.h b/revision.h\nindex 5c8ab16047..adb810c2f8 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -87,6 +87,19 @@ struct ref_exclusions {\n \t * patterns matches, the reference will be excluded.\n \t */\n \tstruct string_list excluded_refs;\n+\n+\t/*\n+\t * Hidden refs is a list of patterns that is to be hidden via\n+\t * `ref_is_hidden()`.\n+\t */\n+\tstruct string_list hidden_refs;\n+\n+\t/*\n+\t * Indicates whether hidden refs have been configured. This is to\n+\t * distinguish between no hidden refs existing and hidden refs not\n+\t * being parsed.\n+\t */\n+\tchar hidden_refs_configured;\n };\n \n /**\n@@ -94,6 +107,7 @@ struct ref_exclusions {\n */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n }\n \n struct oidset;\n@@ -456,10 +470,12 @@ void show_object_with_name(FILE *, struct object *, const char *);\n /**\n * Helpers to check if a reference should be excluded.\n */\n+\n int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n void init_ref_exclusions(struct ref_exclusions *);\n void clear_ref_exclusions(struct ref_exclusions *);\n void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n+void exclude_hidden_refs(struct ref_exclusions *, const char *section);\n \n /**\n * This function can be used if you want to add commit objects as revision\ndiff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh\nnew file mode 100755\nindex 0000000000..018796d41c\n--- /dev/null\n+++ b/t/t6021-rev-list-exclude-hidden.sh\n@@ -0,0 +1,163 @@\n+#!/bin/sh\n+\n+test_description='git rev-list --exclude-hidden test'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit_bulk --id=commit --ref=refs/heads/main 1 &&\n+\tCOMMIT=$(git rev-parse refs/heads/main) &&\n+\ttest_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&\n+\tTAG=$(git rev-parse refs/tags/lightweight) &&\n+\ttest_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&\n+\tHIDDEN=$(git rev-parse refs/hidden/commit) &&\n+\ttest_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&\n+\tNAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)\n+'\n+\n+test_expect_success 'invalid section' '\n+\techo \"fatal: unsupported section for hidden refs: unsupported\" >expected &&\n+\ttest_must_fail git rev-list --exclude-hidden=unsupported 2>err &&\n+\ttest_cmp expected err\n+'\n+\n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"$section: passed multiple times\" '\n+\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n+\t\ttest_must_fail git rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n+\t\ttest_cmp expected err\n+\t'\n+\n+\ttest_expect_success \"$section: without hiddenRefs\" '\n+\t\tgit rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden via transfer.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden via $section.hideRefs\" '\n+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: respects both transfer.hideRefs and $section.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: negation without hidden refs marks everything as uninteresting\" '\n+\t\tgit rev-list --all --exclude-hidden=$section --not --all >out &&\n+\t\ttest_must_be_empty out\n+\t'\n+\n+\ttest_expect_success \"$section: negation with hidden refs marks them as interesting\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden refs and excludes work together\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: excluded hidden refs get reset\" '\n+\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: excluded hidden refs can be used with multiple pseudo-refs\" '\n+\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&\n+\t\ttest_must_be_empty out\n+\t'\n+\n+\ttest_expect_success \"$section: works with --glob\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --glob=refs/h* >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: operates on stripped refs by default\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: does not hide namespace by default\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: can operate on unstripped refs\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\tfor pseudoopt in remotes branches tags\n+\tdo\n+\t\ttest_expect_success \"$section: fails with --$pseudoopt\" '\n+\t\t\ttest_must_fail git rev-list --exclude-hidden=$section --$pseudoopt 2>err &&\n+\t\t\ttest_i18ngrep \"error: --exclude-hidden cannot be used together with --$pseudoopt\" err\n+\t\t'\n+\n+\t\ttest_expect_success \"$section: fails with --$pseudoopt=pattern\" '\n+\t\t\ttest_must_fail git rev-list --exclude-hidden=$section --$pseudoopt=pattern 2>err &&\n+\t\t\ttest_i18ngrep \"error: --exclude-hidden cannot be used together with --$pseudoopt\" err\n+\t\t'\n+\tdone\n+done\n+\n+test_done\n-- \n2.38.1\n\n"},{"id":"467128","messageId":"2eeb25eef0cdf6c5f9f01058326d8b0a9992c461.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"[PATCH v5 6/7] rev-parse: add `--exclude-hidden=` option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:50:14Z","receivedAt":"2022-11-11T06:50:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new `--exclude-hidden=` option that is similar to the one we just\nadded to git-rev-list(1). Given a seciton name `uploadpack` or `receive`\nas argument, it causes us to exclude all references that would be hidden\nby the respective `$section.hideRefs` configuration.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/git-rev-parse.txt | 7 ++++++\n builtin/rev-parse.c | 10 +++++++++\n t/t6018-rev-list-glob.sh | 40 +++++++++++++++++++++++++++++++++\n 3 files changed, 57 insertions(+)\n\ndiff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt\nindex 6b8ca085aa..bcd8069287 100644\n--- a/Documentation/git-rev-parse.txt\n+++ b/Documentation/git-rev-parse.txt\n@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[receive|uploadpack]::\n+\tDo not include refs that would be hidden by `git-receive-pack` or\n+\t`git-upload-pack` by consulting the appropriate `receive.hideRefs` or\n+\t`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see\n+\tlinkgit:git-config[1]). This option affects the next pseudo-ref option\n+\t`--all` or `--glob` and is cleared after processing them.\n+\n --disambiguate=::\n \tShow every object whose name begins with the given prefix.\n \tThe must be at least 4 hexadecimal digits long to\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 7fa5b6991b..b5666a03bd 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -876,10 +876,14 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (opt_with_value(arg, \"--branches\", &arg)) {\n+\t\t\t\tif (ref_excludes.hidden_refs_configured)\n+\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n \t\t\t\thandle_ref_opt(arg, \"refs/heads/\");\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (opt_with_value(arg, \"--tags\", &arg)) {\n+\t\t\t\tif (ref_excludes.hidden_refs_configured)\n+\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n \t\t\t\thandle_ref_opt(arg, \"refs/tags/\");\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -888,6 +892,8 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (opt_with_value(arg, \"--remotes\", &arg)) {\n+\t\t\t\tif (ref_excludes.hidden_refs_configured)\n+\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n \t\t\t\thandle_ref_opt(arg, \"refs/remotes/\");\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -895,6 +901,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tadd_ref_exclusion(&ref_excludes, arg);\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (skip_prefix(arg, \"--exclude-hidden=\", &arg)) {\n+\t\t\t\texclude_hidden_refs(&ref_excludes, arg);\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-toplevel\")) {\n \t\t\t\tconst char *work_tree = get_git_work_tree();\n \t\t\t\tif (work_tree)\ndiff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh\nindex e1abc5c2b3..aabf590dda 100755\n--- a/t/t6018-rev-list-glob.sh\n+++ b/t/t6018-rev-list-glob.sh\n@@ -187,6 +187,46 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '\n \tcompare rev-parse \"--exclude=upstream/x --remotes=upstream/*\" \"upstream/one upstream/two\"\n '\n \n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--branches --tags\" \"--exclude-hidden=$section --all\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude=refs/heads/subspace/* --all\" \"--exclude-hidden=$section --all\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --glob\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude=refs/heads/subspace/* --glob=refs/heads/*\" \"--exclude-hidden=$section --glob=refs/heads/*\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section can be passed once per pseudo-ref\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--branches --tags --branches --tags\" \"--exclude-hidden=$section --all --exclude-hidden=$section --all\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section can only be passed once per pseudo-ref\" '\n+\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n+\t\ttest_must_fail git rev-parse --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n+\t\ttest_cmp expected err\n+\t'\n+\n+\tfor pseudoopt in branches tags remotes\n+\tdo\n+\t\ttest_expect_success \"rev-parse --exclude-hidden=$section fails with --$pseudoopt\" '\n+\t\t\techo \"error: --exclude-hidden cannot be used together with --$pseudoopt\" >expected &&\n+\t\t\ttest_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt 2>err &&\n+\t\t\ttest_cmp expected err\n+\t\t'\n+\n+\t\ttest_expect_success \"rev-parse --exclude-hidden=$section fails with --$pseudoopt=pattern\" '\n+\t\t\techo \"error: --exclude-hidden cannot be used together with --$pseudoopt\" >expected &&\n+\t\t\ttest_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt=pattern 2>err &&\n+\t\t\ttest_cmp expected err\n+\t\t'\n+\tdone\n+done\n+\n test_expect_success 'rev-list --exclude=glob with --branches=glob' '\n \tcompare rev-list \"--exclude=subspace-* --branches=sub*\" \"subspace/one subspace/two\"\n '\n-- \n2.38.1\n\n"},{"id":"467129","messageId":"f5f18f3939276e4dd63c66d5fb2a72231f9a7a06.1668149149.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"[PATCH v5 7/7] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-11T06:50:18Z","receivedAt":"2022-11-11T06:50:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When serving a push, git-receive-pack(1) needs to verify that the\npackfile sent by the client contains all objects that are required by\nthe updated references. This connectivity check works by marking all\npreexisting references as uninteresting and using the new reference tips\nas starting point for a graph walk.\n\nMarking all preexisting references as uninteresting can be a problem\nwhen it comes to performance. Git forges tend to do internal bookkeeping\nto keep alive sets of objects for internal use or make them easy to find\nvia certain references. These references are typically hidden away from\nthe user so that they are neither advertised nor writeable. At GitLab,\nwe have one particular repository that contains a total of 7 million\nreferences, of which 6.8 million are indeed internal references. With\nthe current connectivity check we are forced to load all these\nreferences in order to mark them as uninteresting, and this alone takes\naround 15 seconds to compute.\n\nWe can optimize this by only taking into account the set of visible refs\nwhen marking objects as uninteresting. This means that we may now walk\nmore objects until we hit any object that is marked as uninteresting.\nBut it is rather unlikely that clients send objects that make large\nparts of objects reachable that have previously only ever been hidden,\nwhereas the common case is to push incremental changes that build on top\nof the visible object graph.\n\nThis provides a huge boost to performance in the mentioned repository,\nwhere the vast majority of its refs hidden. Pushing a new commit into\nthis repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\nas it is configured in Gitaly leads to a 4.5-fold speedup:\n\n Benchmark 1: main\n Time (mean ± σ): 30.977 s ± 0.157 s [User: 30.226 s, System: 1.083 s]\n Range (min … max): 30.796 s … 31.071 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 6.799 s ± 0.063 s [User: 6.803 s, System: 0.354 s]\n Range (min … max): 6.729 s … 6.850 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 4.56 ± 0.05 times faster than 'main'\n\nAs we mostly go through the same codepaths even in the case where there\nare no hidden refs at all compared to the code before there is no change\nin performance when no refs are hidden:\n\n Benchmark 1: main\n Time (mean ± σ): 48.188 s ± 0.432 s [User: 49.326 s, System: 5.009 s]\n Range (min … max): 47.706 s … 48.539 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 48.027 s ± 0.500 s [User: 48.934 s, System: 5.025 s]\n Range (min … max): 47.504 s … 48.500 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 1.00 ± 0.01 times faster than 'main'\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 2 ++\n connected.c | 3 +++\n connected.h | 7 +++++++\n 3 files changed, 12 insertions(+)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1e24b31a0a..a90af30363 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,\n \topt.err_fd = err_fd;\n \topt.progress = err_fd && !quiet;\n \topt.env = tmp_objdir_env(tmp_objdir);\n+\topt.exclude_hidden_refs_section = \"receive\";\n+\n \tif (check_connected(iterate_receive_command_list, &data, &opt))\n \t\tset_connectivity_errors(commands, si);\n \ndiff --git a/connected.c b/connected.c\nindex 74a20cb32e..4f6388eed7 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n \tif (!opt->is_deepening_fetch) {\n \t\tstrvec_push(&rev_list.args, \"--not\");\n+\t\tif (opt->exclude_hidden_refs_section)\n+\t\t\tstrvec_pushf(&rev_list.args, \"--exclude-hidden=%s\",\n+\t\t\t\t opt->exclude_hidden_refs_section);\n \t\tstrvec_push(&rev_list.args, \"--all\");\n \t}\n \tstrvec_push(&rev_list.args, \"--quiet\");\ndiff --git a/connected.h b/connected.h\nindex 6e59c92aa3..16b2c84f2e 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -46,6 +46,13 @@ struct check_connected_options {\n \t * during a fetch.\n \t */\n \tunsigned is_deepening_fetch : 1;\n+\n+\t/*\n+\t * If not NULL, use `--exclude-hidden=$section` to exclude all refs\n+\t * hidden via the `$section.hideRefs` config from the set of\n+\t * already-reachable refs.\n+\t */\n+\tconst char *exclude_hidden_refs_section;\n };\n \n #define CHECK_CONNECTED_INIT { 0 }\n-- \n2.38.1\n\n"},{"id":"467165","messageId":"Y27KL0Yg7nzdQ+HC@nand.local","threadId":"58706","inReplyTo":"cover.1668149149.git.ps@pks.im","subject":"Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-11T22:18:23Z","receivedAt":"2022-11-11T22:19:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Nov 11, 2022 at 07:49:49AM +0100, Patrick Steinhardt wrote:\n> Patrick Steinhardt (7):\n> refs: fix memory leak when parsing hideRefs config\n> refs: get rid of global list of hidden refs\n> revision: move together exclusion-related functions\n> revision: introduce struct to handle exclusions\n> revision: add new parameter to exclude hidden refs\n> rev-parse: add `--exclude-hidden=` option\n> receive-pack: only use visible refs for connectivity check\n\nThe new version is looking pretty good to my eyes, though I would like\nto hear from Peff, too, since he caught a few things that I missed in\nthe previous rounds.\n\nI have to say, the final patch is *really* nice ;-).\n\nThanks,\nTaylor\n"},{"id":"467316","messageId":"Y3PLwW7krP0eJS6+@coredump.intra.peff.net","threadId":"58706","inReplyTo":"Y27KL0Yg7nzdQ+HC@nand.local","subject":"Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-15T17:26:25Z","receivedAt":"2022-11-15T17:26:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 11, 2022 at 05:18:23PM -0500, Taylor Blau wrote:\n\n> On Fri, Nov 11, 2022 at 07:49:49AM +0100, Patrick Steinhardt wrote:\n> > Patrick Steinhardt (7):\n> > refs: fix memory leak when parsing hideRefs config\n> > refs: get rid of global list of hidden refs\n> > revision: move together exclusion-related functions\n> > revision: introduce struct to handle exclusions\n> > revision: add new parameter to exclude hidden refs\n> > rev-parse: add `--exclude-hidden=` option\n> > receive-pack: only use visible refs for connectivity check\n> \n> The new version is looking pretty good to my eyes, though I would like\n> to hear from Peff, too, since he caught a few things that I missed in\n> the previous rounds.\n\nThis looks good to me, too. There's a typo (s/seciton/section/) in the\ncommit message of patch 6, but definitely not worth a re-roll. :)\n\nI admit I didn't think _too_ deeply about the interaction with\nnamespaces, and whether there are any corner cases. I was happy to see\nthe tests there, and I assume in writing them that Patrick matched how\nreceive-pack, etc, behave.\n\n-Peff\n"},{"id":"467397","messageId":"Y3VUkKoKd8j7IW6T@nand.local","threadId":"58706","inReplyTo":"Y3PLwW7krP0eJS6+@coredump.intra.peff.net","subject":"Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-16T21:22:24Z","receivedAt":"2022-11-16T21:22:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 15, 2022 at 12:26:25PM -0500, Jeff King wrote:\n> On Fri, Nov 11, 2022 at 05:18:23PM -0500, Taylor Blau wrote:\n>\n> > On Fri, Nov 11, 2022 at 07:49:49AM +0100, Patrick Steinhardt wrote:\n> > > Patrick Steinhardt (7):\n> > > refs: fix memory leak when parsing hideRefs config\n> > > refs: get rid of global list of hidden refs\n> > > revision: move together exclusion-related functions\n> > > revision: introduce struct to handle exclusions\n> > > revision: add new parameter to exclude hidden refs\n> > > rev-parse: add `--exclude-hidden=` option\n> > > receive-pack: only use visible refs for connectivity check\n> >\n> > The new version is looking pretty good to my eyes, though I would like\n> > to hear from Peff, too, since he caught a few things that I missed in\n> > the previous rounds.\n>\n> This looks good to me, too. There's a typo (s/seciton/section/) in the\n> commit message of patch 6, but definitely not worth a re-roll. :)\n\nHmm. It looks like this is broken in CI when the default initial branch\nname is something other than \"master\":\n\n $ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main ./t6021-rev-list-exclude-hidden.sh -i --verbose-only=12 -x\n [...]\n expecting success of 6021.12 'receive: excluded hidden refs can be used with multiple pseudo-refs':\n git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&\n test_must_be_empty out\n\n + git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=receive --all --exclude-hidden=receive --all\n + test_must_be_empty out\n + test 1 -ne 1\n + test_path_is_file out\n + test 1 -ne 1\n + test -f out\n + test -s out\n + echo 'out' is not empty, it contains:\n 'out' is not empty, it contains:\n + cat out\n d2e88f5a45c63e4ec7e1fd303542944487abe89a\n + return 1\n error: last command exited with $?=1\n not ok 12 - receive: excluded hidden refs can be used with multiple pseudo-refs\n #\n #\t\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&\n #\t\t\ttest_must_be_empty out\n #\n 1..12\n\nI haven't looked too deeply at what is going on here, but let's make\nsure to resolve this before graduating the topic down (which I would\notherwise like to do in the next push-out, probably tomorrow or the next\nday).\n\nThanks,\nTaylor\n"},{"id":"467398","messageId":"Y3VeWvj6bTw1C3jP@coredump.intra.peff.net","threadId":"58706","inReplyTo":"Y3VUkKoKd8j7IW6T@nand.local","subject":"Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-16T22:04:10Z","receivedAt":"2022-11-16T22:04:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 16, 2022 at 04:22:24PM -0500, Taylor Blau wrote:\n\n> > This looks good to me, too. There's a typo (s/seciton/section/) in the\n> > commit message of patch 6, but definitely not worth a re-roll. :)\n> \n> Hmm. It looks like this is broken in CI when the default initial branch\n> name is something other than \"master\":\n> \n> $ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main ./t6021-rev-list-exclude-hidden.sh -i --verbose-only=12 -x\n> [...]\n> expecting success of 6021.12 'receive: excluded hidden refs can be used with multiple pseudo-refs':\n> git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&\n> test_must_be_empty out\n> \n> + git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=receive --all --exclude-hidden=receive --all\n> + test_must_be_empty out\n> + test 1 -ne 1\n> + test_path_is_file out\n> + test 1 -ne 1\n> + test -f out\n> + test -s out\n> + echo 'out' is not empty, it contains:\n> 'out' is not empty, it contains:\n> + cat out\n> d2e88f5a45c63e4ec7e1fd303542944487abe89a\n> + return 1\n> error: last command exited with $?=1\n> not ok 12 - receive: excluded hidden refs can be used with multiple pseudo-refs\n> #\n> #\t\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&\n> #\t\t\ttest_must_be_empty out\n> #\n> 1..12\n> \n> I haven't looked too deeply at what is going on here, but let's make\n> sure to resolve this before graduating the topic down (which I would\n> otherwise like to do in the next push-out, probably tomorrow or the next\n> day).\n\nThe issue is that some of the tests assume that hiding \"refs/\" should\nproduce no output from \"--exclude-hidden=receive --all\". But it will\nalso show HEAD, even if it points to a hidden ref (which I think is OK,\nand matches what receive-pack would do).\n\nBut because the setup uses \"main\" as one of the sample refs, HEAD may or\nmay not be valid, depending on what it points to (without setting\nGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME it points to master, which is\nunborn).\n\nSo the fix is just:\n\ndiff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh\nindex 018796d41c..1543a93fe0 100755\n--- a/t/t6021-rev-list-exclude-hidden.sh\n+++ b/t/t6021-rev-list-exclude-hidden.sh\n@@ -5,8 +5,8 @@ test_description='git rev-list --exclude-hidden test'\n . ./test-lib.sh\n \n test_expect_success 'setup' '\n-\ttest_commit_bulk --id=commit --ref=refs/heads/main 1 &&\n-\tCOMMIT=$(git rev-parse refs/heads/main) &&\n+\ttest_commit_bulk --id=commit --ref=refs/heads/foo 1 &&\n+\tCOMMIT=$(git rev-parse refs/heads/foo) &&\n \ttest_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&\n \tTAG=$(git rev-parse refs/tags/lightweight) &&\n \ttest_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&\n\nbut Patrick may want to select a more meaningful name. :)\n\nNotably I don't think we want to do the usual\n\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n\nat the top of the script. We really don't mean to depend on having a\nspecific branch that HEAD points to here.\n\n-Peff\n"},{"id":"467400","messageId":"Y3VlKPFCLkFj188G@nand.local","threadId":"58706","inReplyTo":"Y3VeWvj6bTw1C3jP@coredump.intra.peff.net","subject":"Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-16T22:33:12Z","receivedAt":"2022-11-16T22:33:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Nov 16, 2022 at 05:04:10PM -0500, Jeff King wrote:\n> > I haven't looked too deeply at what is going on here, but let's make\n> > sure to resolve this before graduating the topic down (which I would\n> > otherwise like to do in the next push-out, probably tomorrow or the next\n> > day).\n>\n> The issue is that some of the tests assume that hiding \"refs/\" should\n> produce no output from \"--exclude-hidden=receive --all\". But it will\n> also show HEAD, even if it points to a hidden ref (which I think is OK,\n> and matches what receive-pack would do).\n>\n> But because the setup uses \"main\" as one of the sample refs, HEAD may or\n> may not be valid, depending on what it points to (without setting\n> GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME it points to master, which is\n> unborn).\n>\n> So the fix is just:\n>\n> [...]\n\nMakes perfect sense, and thanks for looking into it.\n\nPatrick: it sounds like there was one typo in the earlier round which\nyou may want to pick up also, assuming you reroll this.\n\nThanks,\nTaylor\n"},{"id":"467415","messageId":"Y3XKfrnmecgaHUU1@ncase","threadId":"58706","inReplyTo":"Y3VlKPFCLkFj188G@nand.local","subject":"Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:45:34Z","receivedAt":"2022-11-17T05:45:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Nov 16, 2022 at 05:33:12PM -0500, Taylor Blau wrote:\n> On Wed, Nov 16, 2022 at 05:04:10PM -0500, Jeff King wrote:\n> > > I haven't looked too deeply at what is going on here, but let's make\n> > > sure to resolve this before graduating the topic down (which I would\n> > > otherwise like to do in the next push-out, probably tomorrow or the next\n> > > day).\n> >\n> > The issue is that some of the tests assume that hiding \"refs/\" should\n> > produce no output from \"--exclude-hidden=receive --all\". But it will\n> > also show HEAD, even if it points to a hidden ref (which I think is OK,\n> > and matches what receive-pack would do).\n> >\n> > But because the setup uses \"main\" as one of the sample refs, HEAD may or\n> > may not be valid, depending on what it points to (without setting\n> > GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME it points to master, which is\n> > unborn).\n> >\n> > So the fix is just:\n> >\n> > [...]\n> \n> Makes perfect sense, and thanks for looking into it.\n> \n> Patrick: it sounds like there was one typo in the earlier round which\n> you may want to pick up also, assuming you reroll this.\n> \n> Thanks,\n> Taylor\n\nThanks to both of you, I'll send out v6 in a bit.\n\nPatrick\n"},{"id":"467416","messageId":"cover.1668663795.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1666967670.git.ps@pks.im","subject":"[PATCH v6 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:46:35Z","receivedAt":"2022-11-17T05:46:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the sixth version of my patch series that tries to improve\nperformance of the connectivity check by only considering preexisting\nrefs as uninteresting that could actually have been advertised to the\nclient.\n\nThere are only two changes in this version compared to v5:\n\n - A fix to the test setup in commit 5/7 so that tests pass when\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main.\n\n - A typo fix in the commit message of patch 6/7.\n\nPatrick\n\nPatrick Steinhardt (7):\n refs: fix memory leak when parsing hideRefs config\n refs: get rid of global list of hidden refs\n revision: move together exclusion-related functions\n revision: introduce struct to handle exclusions\n revision: add new parameter to exclude hidden refs\n rev-parse: add `--exclude-hidden=` option\n receive-pack: only use visible refs for connectivity check\n\n Documentation/git-rev-parse.txt | 7 ++\n Documentation/rev-list-options.txt | 7 ++\n builtin/receive-pack.c | 10 +-\n builtin/rev-list.c | 1 +\n builtin/rev-parse.c | 18 +++-\n connected.c | 3 +\n connected.h | 7 ++\n ls-refs.c | 13 ++-\n refs.c | 16 +--\n refs.h | 5 +-\n revision.c | 131 +++++++++++++++--------\n revision.h | 43 ++++++--\n t/t6018-rev-list-glob.sh | 40 +++++++\n t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++\n upload-pack.c | 30 +++---\n 15 files changed, 411 insertions(+), 83 deletions(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\nRange-diff against v5:\n1: cfab8ba1a2 = 1: ef182e4330 refs: fix memory leak when parsing hideRefs config\n2: d8118c6dd8 = 2: 48913c1493 refs: get rid of global list of hidden refs\n3: 93a627fb7f = 3: 3827d6a2fc revision: move together exclusion-related functions\n4: ad41ade332 = 4: 805de80e64 revision: introduce struct to handle exclusions\n5: b5a4ce432a ! 5: d86a3342f6 revision: add new parameter to exclude hidden refs\n @@ t/t6021-rev-list-exclude-hidden.sh (new)\n +. ./test-lib.sh\n +\n +test_expect_success 'setup' '\n -+\ttest_commit_bulk --id=commit --ref=refs/heads/main 1 &&\n -+\tCOMMIT=$(git rev-parse refs/heads/main) &&\n ++\ttest_commit_bulk --id=commit --ref=refs/heads/branch 1 &&\n ++\tCOMMIT=$(git rev-parse refs/heads/branch) &&\n +\ttest_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&\n +\tTAG=$(git rev-parse refs/tags/lightweight) &&\n +\ttest_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&\n6: 2eeb25eef0 ! 6: f8b5eb5a7e rev-parse: add `--exclude-hidden=` option\n @@ Commit message\n rev-parse: add `--exclude-hidden=` option\n \n Add a new `--exclude-hidden=` option that is similar to the one we just\n - added to git-rev-list(1). Given a seciton name `uploadpack` or `receive`\n + added to git-rev-list(1). Given a section name `uploadpack` or `receive`\n as argument, it causes us to exclude all references that would be hidden\n by the respective `$section.hideRefs` configuration.\n \n7: f5f18f3939 = 7: a7eae80ff3 receive-pack: only use visible refs for connectivity check\n-- \n2.38.1\n\n"},{"id":"467417","messageId":"ef182e4330bfc80ac4eaf26f186ef15ac5530fde.1668663796.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"[PATCH v6 1/7] refs: fix memory leak when parsing hideRefs config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:46:39Z","receivedAt":"2022-11-17T05:46:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When parsing the hideRefs configuration, we first duplicate the config\nvalue so that we can modify it. We then subsequently append it to the\n`hide_refs` string list, which is initialized with `strdup_strings`\nenabled. As a consequence we again reallocate the string, but never\nfree the first duplicate and thus have a memory leak.\n\nWhile we never clean up the static `hide_refs` variable anyway, this is\nno excuse to make the leak worse by leaking every value twice. We are\nalso about to change the way this variable will be handled so that we do\nindeed start to clean it up. So let's fix the memory leak by using the\n`string_list_append_nodup()` so that we pass ownership of the allocated\nstring to `hide_refs`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/refs.c b/refs.c\nindex 1491ae937e..a4ab264d74 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1435,7 +1435,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\t\tCALLOC_ARRAY(hide_refs, 1);\n \t\t\thide_refs->strdup_strings = 1;\n \t\t}\n-\t\tstring_list_append(hide_refs, ref);\n+\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n-- \n2.38.1\n\n"},{"id":"467418","messageId":"48913c1493abfd297d3adb7a41dc3dcc1c5817f2.1668663796.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"[PATCH v6 2/7] refs: get rid of global list of hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:46:43Z","receivedAt":"2022-11-17T05:47:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to add a new argument to git-rev-list(1) that allows it to\nadd all references that are visible when taking `transfer.hideRefs` et\nal into account. This will require us to potentially parse multiple sets\nof hidden refs, which is not easily possible right now as there is only\na single, global instance of the list of parsed hidden refs.\n\nRefactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both\ntake the list of hidden references as input and adjust callers to keep a\nlocal list, instead. This allows us to easily use multiple hidden-ref\nlists. Furthermore, it allows us to properly free this list before we\nexit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 8 +++++---\n ls-refs.c | 13 +++++++++----\n refs.c | 14 ++++----------\n refs.h | 5 +++--\n upload-pack.c | 30 ++++++++++++++++++------------\n 5 files changed, 39 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 44bcea3a5b..1e24b31a0a 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)\n \n static int receive_pack_config(const char *var, const char *value, void *cb)\n {\n-\tint status = parse_hide_refs_config(var, value, \"receive\");\n+\tint status = parse_hide_refs_config(var, value, \"receive\", &hidden_refs);\n \n \tif (status)\n \t\treturn status;\n@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,\n \tstruct oidset *seen = data;\n \tconst char *path = strip_namespace(path_full);\n \n-\tif (ref_is_hidden(path, path_full))\n+\tif (ref_is_hidden(path, path_full, &hidden_refs))\n \t\treturn 0;\n \n \t/*\n@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)\n \t\tstrbuf_setlen(&refname_full, prefix_len);\n \t\tstrbuf_addstr(&refname_full, cmd->ref_name);\n \n-\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf))\n+\t\tif (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))\n \t\t\tcontinue;\n \t\tif (is_null_oid(&cmd->new_oid))\n \t\t\tcmd->error_string = \"deny deleting a hidden ref\";\n@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n+\tstring_list_clear(&hidden_refs, 0);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex fa0d01b47c..fb6769742c 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -6,6 +6,7 @@\n #include \"ls-refs.h\"\n #include \"pkt-line.h\"\n #include \"config.h\"\n+#include \"string-list.h\"\n \n static int config_read;\n static int advertise_unborn;\n@@ -73,6 +74,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n+\tstruct string_list hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \n \tstrbuf_reset(&data->buf);\n \n-\tif (ref_is_hidden(refname_nons, refname))\n+\tif (ref_is_hidden(refname_nons, refname, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (!ref_match(&data->prefixes, refname_nons))\n@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)\n }\n \n static int ls_refs_config(const char *var, const char *value,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *cb_data)\n {\n+\tstruct ls_refs_data *data = cb_data;\n \t/*\n \t * We only serve fetches over v2 for now, so respect only \"uploadpack\"\n \t * config. This may need to eventually be expanded to \"receive\", but we\n \t * don't yet know how that information will be passed to ls-refs.\n \t */\n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n int ls_refs(struct repository *r, struct packet_reader *request)\n@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n+\tstring_list_init_dup(&data.hidden_refs);\n \n \tensure_config_read();\n-\tgit_config(ls_refs_config, NULL);\n+\tgit_config(ls_refs_config, &data);\n \n \twhile (packet_reader_read(request) == PACKET_READ_NORMAL) {\n \t\tconst char *arg = request->line;\n@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n+\tstring_list_clear(&data.hidden_refs, 0);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex a4ab264d74..2c7e88b190 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n \t\t\t\t\t refname, strict);\n }\n \n-static struct string_list *hide_refs;\n-\n-int parse_hide_refs_config(const char *var, const char *value, const char *section)\n+int parse_hide_refs_config(const char *var, const char *value, const char *section,\n+\t\t\t struct string_list *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tif (!hide_refs) {\n-\t\t\tCALLOC_ARRAY(hide_refs, 1);\n-\t\t\thide_refs->strdup_strings = 1;\n-\t\t}\n \t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n-int ref_is_hidden(const char *refname, const char *refname_full)\n+int ref_is_hidden(const char *refname, const char *refname_full,\n+\t\t const struct string_list *hide_refs)\n {\n \tint i;\n \n-\tif (!hide_refs)\n-\t\treturn 0;\n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n \t\tconst char *match = hide_refs->items[i].string;\n \t\tconst char *subject;\ndiff --git a/refs.h b/refs.h\nindex 8958717a17..3266fd8f57 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,\n \t const struct object_id *new_oid, const struct object_id *old_oid,\n \t unsigned int flags, enum action_on_err onerr);\n \n-int parse_hide_refs_config(const char *var, const char *value, const char *);\n+int parse_hide_refs_config(const char *var, const char *value, const char *,\n+\t\t\t struct string_list *);\n \n /*\n * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);\n * the ref is outside that namespace, the first parameter is NULL. The second\n * parameter always points to the full ref name.\n */\n-int ref_is_hidden(const char *, const char *);\n+int ref_is_hidden(const char *, const char *, const struct string_list *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0b8311bd68..551f22ffa5 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -62,6 +62,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n+\tstruct string_list hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n+\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n \tmemset(data, 0, sizeof(*data));\n \tdata->symref = symref;\n \tdata->wanted_refs = wanted_refs;\n+\tdata->hidden_refs = hidden_refs;\n \tdata->want_obj = want_obj;\n \tdata->have_obj = have_obj;\n \tdata->haves = haves;\n@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n+\tstring_list_clear(&data->hidden_refs, 0);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * Checking for reachable shallows requires that our refs be\n \t\t * marked with OUR_REF.\n \t\t */\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, data);\n+\t\tfor_each_namespaced_ref(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid)\n+\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n-\tif (ref_is_hidden(refname, refname_full)) {\n+\tif (ref_is_hidden(refname, refname_full, hidden_refs)) {\n \t\to->flags |= HIDDEN_REF;\n \t\treturn 1;\n \t}\n@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,\n }\n \n static int check_ref(const char *refname_full, const struct object_id *oid,\n-\t\t int flag UNUSED, void *cb_data UNUSED)\n+\t\t int flag UNUSED, void *cb_data)\n {\n \tconst char *refname = strip_namespace(refname_full);\n+\tstruct upload_pack_data *data = cb_data;\n \n-\tmark_our_ref(refname, refname_full, oid);\n+\tmark_our_ref(refname, refname_full, oid, &data->hidden_refs);\n \treturn 0;\n }\n \n@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \tstruct object_id peeled;\n \tstruct upload_pack_data *data = cb_data;\n \n-\tif (mark_our_ref(refname_nons, refname, oid))\n+\tif (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))\n \t\treturn 0;\n \n \tif (capabilities) {\n@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)\n \tif (parse_object_filter_config(var, value, data) < 0)\n \t\treturn -1;\n \n-\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\", &data->hidden_refs);\n }\n \n static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)\n@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n+\t\thead_ref_namespaced(check_ref, &data);\n+\t\tfor_each_namespaced_ref(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t struct string_list *wanted_refs,\n+\t\t\t struct string_list *hidden_refs,\n \t\t\t struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\tstruct strbuf refname = STRBUF_INIT;\n \n \t\tstrbuf_addf(&refname, \"%s%s\", get_git_namespace(), refname_nons);\n-\t\tif (ref_is_hidden(refname_nons, refname.buf) ||\n+\t\tif (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||\n \t\t read_ref(refname.buf, &oid)) {\n \t\t\tpacket_writer_error(writer, \"unknown ref %s\", refname_nons);\n \t\t\tdie(\"unknown ref %s\", refname_nons);\n@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (data->allow_ref_in_want &&\n \t\t parse_want_ref(&data->writer, arg, &data->wanted_refs,\n-\t\t\t\t &data->want_obj))\n+\t\t\t\t &data->hidden_refs, &data->want_obj))\n \t\t\tcontinue;\n \t\t/* process have line */\n \t\tif (parse_have(arg, &data->haves))\n-- \n2.38.1\n\n"},{"id":"467419","messageId":"3827d6a2fcc9f3a14c6ebd2cca47d53d2d6cfa5f.1668663796.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"[PATCH v6 3/7] revision: move together exclusion-related functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:46:47Z","receivedAt":"2022-11-17T05:47:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move together the definitions of functions that handle exclusions of\nrefs so that related functionality sits in a single place, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 52 ++++++++++++++++++++++++++--------------------------\n 1 file changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 0760e78936..be755670e2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-struct all_refs_cb {\n-\tint all_flags;\n-\tint warned_bad_reflog;\n-\tstruct rev_info *all_revs;\n-\tconst char *name_for_errormsg;\n-\tstruct worktree *wt;\n-};\n-\n int ref_excluded(struct string_list *ref_excludes, const char *path)\n {\n \tstruct string_list_item *item;\n@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)\n \treturn 0;\n }\n \n+void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+{\n+\tif (*ref_excludes_p) {\n+\t\tstring_list_clear(*ref_excludes_p, 0);\n+\t\tfree(*ref_excludes_p);\n+\t}\n+\t*ref_excludes_p = NULL;\n+}\n+\n+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+{\n+\tif (!*ref_excludes_p) {\n+\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n+\t\t(*ref_excludes_p)->strdup_strings = 1;\n+\t}\n+\tstring_list_append(*ref_excludes_p, exclude);\n+}\n+\n+struct all_refs_cb {\n+\tint all_flags;\n+\tint warned_bad_reflog;\n+\tstruct rev_info *all_revs;\n+\tconst char *name_for_errormsg;\n+\tstruct worktree *wt;\n+};\n+\n static int handle_one_ref(const char *path, const struct object_id *oid,\n \t\t\t int flag UNUSED,\n \t\t\t void *cb_data)\n@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,\n \tcb->wt = NULL;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n-{\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n-}\n-\n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n-{\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n-}\n-\n static void handle_refs(struct ref_store *refs,\n \t\t\tstruct rev_info *revs, unsigned flags,\n \t\t\tint (*for_each)(struct ref_store *, each_ref_fn, void *))\n-- \n2.38.1\n\n"},{"id":"467420","messageId":"805de80e6461e5f918e4359970eb400b7437fb68.1668663796.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"[PATCH v6 4/7] revision: introduce struct to handle exclusions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:46:51Z","receivedAt":"2022-11-17T05:47:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The functions that handle exclusion of refs work on a single string\nlist. We're about to add a second mechanism for excluding refs though,\nand it makes sense to reuse much of the same architecture for both kinds\nof exclusion.\n\nIntroduce a new `struct ref_exclusions` that encapsulates all the logic\nrelated to excluding refs and move the `struct string_list` that holds\nall wildmatch patterns of excluded refs into it. Rename functions that\noperate on this struct to match its name.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/rev-parse.c | 8 ++++----\n revision.c | 48 +++++++++++++++++++++------------------------\n revision.h | 27 +++++++++++++++++++------\n 3 files changed, 47 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 8f61050bde..7fa5b6991b 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -39,7 +39,7 @@ static int abbrev_ref_strict;\n static int output_sq;\n \n static int stuck_long;\n-static struct string_list *ref_excludes;\n+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;\n \n /*\n * Some arguments are relevant \"revision\" arguments,\n@@ -198,7 +198,7 @@ static int show_default(void)\n static int show_reference(const char *refname, const struct object_id *oid,\n \t\t\t int flag UNUSED, void *cb_data UNUSED)\n {\n-\tif (ref_excluded(ref_excludes, refname))\n+\tif (ref_excluded(&ref_excludes, refname))\n \t\treturn 0;\n \tshow_rev(NORMAL, oid, refname);\n \treturn 0;\n@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)\n \t\tfor_each_glob_ref_in(show_reference, pattern, prefix, NULL);\n \telse\n \t\tfor_each_ref_in(prefix, show_reference, NULL);\n-\tclear_ref_exclusion(&ref_excludes);\n+\tclear_ref_exclusions(&ref_excludes);\n }\n \n enum format_type {\n@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t\tif (!strcmp(arg, \"--all\")) {\n \t\t\t\tfor_each_ref(show_reference, NULL);\n-\t\t\t\tclear_ref_exclusion(&ref_excludes);\n+\t\t\t\tclear_ref_exclusions(&ref_excludes);\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (skip_prefix(arg, \"--disambiguate=\", &arg)) {\ndiff --git a/revision.c b/revision.c\nindex be755670e2..fe3ec98f46 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1517,35 +1517,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \t}\n }\n \n-int ref_excluded(struct string_list *ref_excludes, const char *path)\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n \tstruct string_list_item *item;\n-\n-\tif (!ref_excludes)\n-\t\treturn 0;\n-\tfor_each_string_list_item(item, ref_excludes) {\n+\tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n }\n \n-void clear_ref_exclusion(struct string_list **ref_excludes_p)\n+void init_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (*ref_excludes_p) {\n-\t\tstring_list_clear(*ref_excludes_p, 0);\n-\t\tfree(*ref_excludes_p);\n-\t}\n-\t*ref_excludes_p = NULL;\n+\tstruct ref_exclusions blank = REF_EXCLUSIONS_INIT;\n+\tmemcpy(exclusions, &blank, sizeof(*exclusions));\n }\n \n-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n+void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n-\tif (!*ref_excludes_p) {\n-\t\tCALLOC_ARRAY(*ref_excludes_p, 1);\n-\t\t(*ref_excludes_p)->strdup_strings = 1;\n-\t}\n-\tstring_list_append(*ref_excludes_p, exclude);\n+\tstring_list_clear(&exclusions->excluded_refs, 0);\n+}\n+\n+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n+{\n+\tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n struct all_refs_cb {\n@@ -1563,7 +1558,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \tstruct all_refs_cb *cb = cb_data;\n \tstruct object *object;\n \n-\tif (ref_excluded(cb->all_revs->ref_excludes, path))\n+\tif (ref_excluded(&cb->all_revs->ref_excludes, path))\n \t return 0;\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n@@ -1901,6 +1896,7 @@ void repo_init_revisions(struct repository *r,\n \n \tinit_display_notes(&revs->notes_opt);\n \tlist_objects_filter_init(&revs->filter);\n+\tinit_ref_exclusions(&revs->ref_excludes);\n }\n \n static void add_pending_commit_list(struct rev_info *revs,\n@@ -2689,10 +2685,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\t\tother_head_refs(handle_one_ref, &cb);\n \t\t}\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--branches\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--bisect\")) {\n \t\tread_bisect_terms(&term_bad, &term_good);\n \t\thandle_refs(refs, revs, *flags, for_each_bad_bisect_ref);\n@@ -2701,15 +2697,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->bisect = 1;\n \t} else if (!strcmp(arg, \"--tags\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--remotes\")) {\n \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref(handle_one_ref, optarg, &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t\treturn argcount;\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n@@ -2718,17 +2714,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n-\t\tclear_ref_exclusion(&revs->ref_excludes);\n+\t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--reflog\")) {\n \t\tadd_reflogs_to_pending(revs, *flags);\n \t} else if (!strcmp(arg, \"--indexed-objects\")) {\ndiff --git a/revision.h b/revision.h\nindex afe1b77985..5c8ab16047 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -81,6 +81,21 @@ struct rev_cmdline_info {\n \t} *rev;\n };\n \n+struct ref_exclusions {\n+\t/*\n+\t * Excluded refs is a list of wildmatch patterns. If any of the\n+\t * patterns matches, the reference will be excluded.\n+\t */\n+\tstruct string_list excluded_refs;\n+};\n+\n+/**\n+ * Initialize a `struct ref_exclusions` with a macro.\n+ */\n+#define REF_EXCLUSIONS_INIT { \\\n+\t.excluded_refs = STRING_LIST_INIT_DUP, \\\n+}\n+\n struct oidset;\n struct topo_walk_info;\n \n@@ -103,7 +118,7 @@ struct rev_info {\n \tstruct list_objects_filter_options filter;\n \n \t/* excluding from --branches, --refs, etc. expansion */\n-\tstruct string_list *ref_excludes;\n+\tstruct ref_exclusions ref_excludes;\n \n \t/* Basic information */\n \tconst char *prefix;\n@@ -439,12 +454,12 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)\n void show_object_with_name(FILE *, struct object *, const char *);\n \n /**\n- * Helpers to check if a \"struct string_list\" item matches with\n- * wildmatch().\n+ * Helpers to check if a reference should be excluded.\n */\n-int ref_excluded(struct string_list *, const char *path);\n-void clear_ref_exclusion(struct string_list **);\n-void add_ref_exclusion(struct string_list **, const char *exclude);\n+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n+void init_ref_exclusions(struct ref_exclusions *);\n+void clear_ref_exclusions(struct ref_exclusions *);\n+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n \n /**\n * This function can be used if you want to add commit objects as revision\n-- \n2.38.1\n\n"},{"id":"467421","messageId":"d86a3342f61e2f44c976f379aea4a3463fb6fd98.1668663796.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"[PATCH v6 5/7] revision: add new parameter to exclude hidden refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:46:56Z","receivedAt":"2022-11-17T05:47:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Users can optionally hide refs from remote users in git-upload-pack(1),\ngit-receive-pack(1) and others via the `transfer.hideRefs`, but there is\nnot an easy way to obtain the list of all visible or hidden refs right\nnow. We'll require just that though for a performance improvement in our\nconnectivity check.\n\nAdd a new option `--exclude-hidden=` that excludes any hidden refs from\nthe next pseudo-ref like `--all` or `--branches`.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 7 ++\n builtin/rev-list.c | 1 +\n revision.c | 55 +++++++++-\n revision.h | 16 +++\n t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++\n 5 files changed, 241 insertions(+), 1 deletion(-)\n create mode 100755 t/t6021-rev-list-exclude-hidden.sh\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 1837509566..ff68e48406 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[receive|uploadpack]::\n+\tDo not include refs that would be hidden by `git-receive-pack` or\n+\t`git-upload-pack` by consulting the appropriate `receive.hideRefs` or\n+\t`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see\n+\tlinkgit:git-config[1]). This option affects the next pseudo-ref option\n+\t`--all` or `--glob` and is cleared after processing them.\n+\n --reflog::\n \tPretend as if all objects mentioned by reflogs are listed on the\n \tcommand line as ``.\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3acd93f71e..d42db0b0cc 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -38,6 +38,7 @@ static const char rev_list_usage[] =\n \" --tags\\n\"\n \" --remotes\\n\"\n \" --stdin\\n\"\n+\" --exclude-hidden=[receive|uploadpack]\\n\"\n \" --quiet\\n\"\n \" ordering output:\\n\"\n \" --topo-order\\n\"\ndiff --git a/revision.c b/revision.c\nindex fe3ec98f46..bc32fb819a 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"object-store.h\"\n #include \"tag.h\"\n #include \"blob.h\"\n@@ -1519,11 +1520,17 @@ static void add_rev_cmdline_list(struct rev_info *revs,\n \n int ref_excluded(const struct ref_exclusions *exclusions, const char *path)\n {\n+\tconst char *stripped_path = strip_namespace(path);\n \tstruct string_list_item *item;\n+\n \tfor_each_string_list_item(item, &exclusions->excluded_refs) {\n \t\tif (!wildmatch(item->string, path, 0))\n \t\t\treturn 1;\n \t}\n+\n+\tif (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))\n+\t\treturn 1;\n+\n \treturn 0;\n }\n \n@@ -1536,6 +1543,8 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n+\tstring_list_clear(&exclusions->hidden_refs, 0);\n+\texclusions->hidden_refs_configured = 0;\n }\n \n void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n@@ -1543,6 +1552,35 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)\n \tstring_list_append(&exclusions->excluded_refs, exclude);\n }\n \n+struct exclude_hidden_refs_cb {\n+\tstruct ref_exclusions *exclusions;\n+\tconst char *section;\n+};\n+\n+static int hide_refs_config(const char *var, const char *value, void *cb_data)\n+{\n+\tstruct exclude_hidden_refs_cb *cb = cb_data;\n+\tcb->exclusions->hidden_refs_configured = 1;\n+\treturn parse_hide_refs_config(var, value, cb->section,\n+\t\t\t\t &cb->exclusions->hidden_refs);\n+}\n+\n+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)\n+{\n+\tstruct exclude_hidden_refs_cb cb;\n+\n+\tif (strcmp(section, \"receive\") && strcmp(section, \"uploadpack\"))\n+\t\tdie(_(\"unsupported section for hidden refs: %s\"), section);\n+\n+\tif (exclusions->hidden_refs_configured)\n+\t\tdie(_(\"--exclude-hidden= passed more than once\"));\n+\n+\tcb.exclusions = exclusions;\n+\tcb.section = section;\n+\n+\tgit_config(hide_refs_config, &cb);\n+}\n+\n struct all_refs_cb {\n \tint all_flags;\n \tint warned_bad_reflog;\n@@ -2221,7 +2259,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t !strcmp(arg, \"--bisect\") || starts_with(arg, \"--glob=\") ||\n \t !strcmp(arg, \"--indexed-objects\") ||\n \t !strcmp(arg, \"--alternate-refs\") ||\n-\t starts_with(arg, \"--exclude=\") ||\n+\t starts_with(arg, \"--exclude=\") || starts_with(arg, \"--exclude-hidden=\") ||\n \t starts_with(arg, \"--branches=\") || starts_with(arg, \"--tags=\") ||\n \t starts_with(arg, \"--remotes=\") || starts_with(arg, \"--no-walk=\"))\n \t{\n@@ -2687,6 +2725,8 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t}\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--branches\")) {\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n \t\thandle_refs(refs, revs, *flags, refs_for_each_branch_ref);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--bisect\")) {\n@@ -2696,9 +2736,13 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\t\t for_each_good_bisect_ref);\n \t\trevs->bisect = 1;\n \t} else if (!strcmp(arg, \"--tags\")) {\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n \t\thandle_refs(refs, revs, *flags, refs_for_each_tag_ref);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (!strcmp(arg, \"--remotes\")) {\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n \t\thandle_refs(refs, revs, *flags, refs_for_each_remote_ref);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if ((argcount = parse_long_opt(\"glob\", argv, &optarg))) {\n@@ -2710,18 +2754,27 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t} else if ((argcount = parse_long_opt(\"exclude\", argv, &optarg))) {\n \t\tadd_ref_exclusion(&revs->ref_excludes, optarg);\n \t\treturn argcount;\n+\t} else if ((argcount = parse_long_opt(\"exclude-hidden\", argv, &optarg))) {\n+\t\texclude_hidden_refs(&revs->ref_excludes, optarg);\n+\t\treturn argcount;\n \t} else if (skip_prefix(arg, \"--branches=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/heads/\", &cb);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--tags=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/tags/\", &cb);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\n \t} else if (skip_prefix(arg, \"--remotes=\", &optarg)) {\n \t\tstruct all_refs_cb cb;\n+\t\tif (revs->ref_excludes.hidden_refs_configured)\n+\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n \t\tinit_all_refs_cb(&cb, revs, *flags);\n \t\tfor_each_glob_ref_in(handle_one_ref, optarg, \"refs/remotes/\", &cb);\n \t\tclear_ref_exclusions(&revs->ref_excludes);\ndiff --git a/revision.h b/revision.h\nindex 5c8ab16047..adb810c2f8 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -87,6 +87,19 @@ struct ref_exclusions {\n \t * patterns matches, the reference will be excluded.\n \t */\n \tstruct string_list excluded_refs;\n+\n+\t/*\n+\t * Hidden refs is a list of patterns that is to be hidden via\n+\t * `ref_is_hidden()`.\n+\t */\n+\tstruct string_list hidden_refs;\n+\n+\t/*\n+\t * Indicates whether hidden refs have been configured. This is to\n+\t * distinguish between no hidden refs existing and hidden refs not\n+\t * being parsed.\n+\t */\n+\tchar hidden_refs_configured;\n };\n \n /**\n@@ -94,6 +107,7 @@ struct ref_exclusions {\n */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n }\n \n struct oidset;\n@@ -456,10 +470,12 @@ void show_object_with_name(FILE *, struct object *, const char *);\n /**\n * Helpers to check if a reference should be excluded.\n */\n+\n int ref_excluded(const struct ref_exclusions *exclusions, const char *path);\n void init_ref_exclusions(struct ref_exclusions *);\n void clear_ref_exclusions(struct ref_exclusions *);\n void add_ref_exclusion(struct ref_exclusions *, const char *exclude);\n+void exclude_hidden_refs(struct ref_exclusions *, const char *section);\n \n /**\n * This function can be used if you want to add commit objects as revision\ndiff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh\nnew file mode 100755\nindex 0000000000..32b2b09413\n--- /dev/null\n+++ b/t/t6021-rev-list-exclude-hidden.sh\n@@ -0,0 +1,163 @@\n+#!/bin/sh\n+\n+test_description='git rev-list --exclude-hidden test'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit_bulk --id=commit --ref=refs/heads/branch 1 &&\n+\tCOMMIT=$(git rev-parse refs/heads/branch) &&\n+\ttest_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&\n+\tTAG=$(git rev-parse refs/tags/lightweight) &&\n+\ttest_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&\n+\tHIDDEN=$(git rev-parse refs/hidden/commit) &&\n+\ttest_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&\n+\tNAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)\n+'\n+\n+test_expect_success 'invalid section' '\n+\techo \"fatal: unsupported section for hidden refs: unsupported\" >expected &&\n+\ttest_must_fail git rev-list --exclude-hidden=unsupported 2>err &&\n+\ttest_cmp expected err\n+'\n+\n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"$section: passed multiple times\" '\n+\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n+\t\ttest_must_fail git rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n+\t\ttest_cmp expected err\n+\t'\n+\n+\ttest_expect_success \"$section: without hiddenRefs\" '\n+\t\tgit rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden via transfer.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden via $section.hideRefs\" '\n+\t\tgit -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: respects both transfer.hideRefs and $section.hideRefs\" '\n+\t\tgit -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: negation without hidden refs marks everything as uninteresting\" '\n+\t\tgit rev-list --all --exclude-hidden=$section --not --all >out &&\n+\t\ttest_must_be_empty out\n+\t'\n+\n+\ttest_expect_success \"$section: negation with hidden refs marks them as interesting\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: hidden refs and excludes work together\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: excluded hidden refs get reset\" '\n+\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: excluded hidden refs can be used with multiple pseudo-refs\" '\n+\t\tgit -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&\n+\t\ttest_must_be_empty out\n+\t'\n+\n+\ttest_expect_success \"$section: works with --glob\" '\n+\t\tgit -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --glob=refs/h* >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: operates on stripped refs by default\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: does not hide namespace by default\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$NAMESPACE\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\ttest_expect_success \"$section: can operate on unstripped refs\" '\n+\t\tGIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&\n+\t\tcat >expected <<-EOF &&\n+\t\t$HIDDEN\n+\t\t$TAG\n+\t\t$COMMIT\n+\t\tEOF\n+\t\ttest_cmp expected out\n+\t'\n+\n+\tfor pseudoopt in remotes branches tags\n+\tdo\n+\t\ttest_expect_success \"$section: fails with --$pseudoopt\" '\n+\t\t\ttest_must_fail git rev-list --exclude-hidden=$section --$pseudoopt 2>err &&\n+\t\t\ttest_i18ngrep \"error: --exclude-hidden cannot be used together with --$pseudoopt\" err\n+\t\t'\n+\n+\t\ttest_expect_success \"$section: fails with --$pseudoopt=pattern\" '\n+\t\t\ttest_must_fail git rev-list --exclude-hidden=$section --$pseudoopt=pattern 2>err &&\n+\t\t\ttest_i18ngrep \"error: --exclude-hidden cannot be used together with --$pseudoopt\" err\n+\t\t'\n+\tdone\n+done\n+\n+test_done\n-- \n2.38.1\n\n"},{"id":"467422","messageId":"f8b5eb5a7eaca22da803c4672058a706ffe719ce.1668663796.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"[PATCH v6 6/7] rev-parse: add `--exclude-hidden=` option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:47:00Z","receivedAt":"2022-11-17T05:48:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new `--exclude-hidden=` option that is similar to the one we just\nadded to git-rev-list(1). Given a section name `uploadpack` or `receive`\nas argument, it causes us to exclude all references that would be hidden\nby the respective `$section.hideRefs` configuration.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/git-rev-parse.txt | 7 ++++++\n builtin/rev-parse.c | 10 +++++++++\n t/t6018-rev-list-glob.sh | 40 +++++++++++++++++++++++++++++++++\n 3 files changed, 57 insertions(+)\n\ndiff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt\nindex 6b8ca085aa..bcd8069287 100644\n--- a/Documentation/git-rev-parse.txt\n+++ b/Documentation/git-rev-parse.txt\n@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`\n or `--all`. If a trailing '/{asterisk}' is intended, it must be given\n explicitly.\n \n+--exclude-hidden=[receive|uploadpack]::\n+\tDo not include refs that would be hidden by `git-receive-pack` or\n+\t`git-upload-pack` by consulting the appropriate `receive.hideRefs` or\n+\t`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see\n+\tlinkgit:git-config[1]). This option affects the next pseudo-ref option\n+\t`--all` or `--glob` and is cleared after processing them.\n+\n --disambiguate=::\n \tShow every object whose name begins with the given prefix.\n \tThe must be at least 4 hexadecimal digits long to\ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 7fa5b6991b..b5666a03bd 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -876,10 +876,14 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (opt_with_value(arg, \"--branches\", &arg)) {\n+\t\t\t\tif (ref_excludes.hidden_refs_configured)\n+\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --branches\"));\n \t\t\t\thandle_ref_opt(arg, \"refs/heads/\");\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (opt_with_value(arg, \"--tags\", &arg)) {\n+\t\t\t\tif (ref_excludes.hidden_refs_configured)\n+\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --tags\"));\n \t\t\t\thandle_ref_opt(arg, \"refs/tags/\");\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -888,6 +892,8 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tif (opt_with_value(arg, \"--remotes\", &arg)) {\n+\t\t\t\tif (ref_excludes.hidden_refs_configured)\n+\t\t\t\t\treturn error(_(\"--exclude-hidden cannot be used together with --remotes\"));\n \t\t\t\thandle_ref_opt(arg, \"refs/remotes/\");\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -895,6 +901,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\tadd_ref_exclusion(&ref_excludes, arg);\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (skip_prefix(arg, \"--exclude-hidden=\", &arg)) {\n+\t\t\t\texclude_hidden_refs(&ref_excludes, arg);\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--show-toplevel\")) {\n \t\t\t\tconst char *work_tree = get_git_work_tree();\n \t\t\t\tif (work_tree)\ndiff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh\nindex e1abc5c2b3..aabf590dda 100755\n--- a/t/t6018-rev-list-glob.sh\n+++ b/t/t6018-rev-list-glob.sh\n@@ -187,6 +187,46 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '\n \tcompare rev-parse \"--exclude=upstream/x --remotes=upstream/*\" \"upstream/one upstream/two\"\n '\n \n+for section in receive uploadpack\n+do\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--branches --tags\" \"--exclude-hidden=$section --all\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --all\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude=refs/heads/subspace/* --all\" \"--exclude-hidden=$section --all\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section with --glob\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/heads/subspace/ rev-parse\" \"--exclude=refs/heads/subspace/* --glob=refs/heads/*\" \"--exclude-hidden=$section --glob=refs/heads/*\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section can be passed once per pseudo-ref\" '\n+\t\tcompare \"-c transfer.hideRefs=refs/remotes/ rev-parse\" \"--branches --tags --branches --tags\" \"--exclude-hidden=$section --all --exclude-hidden=$section --all\"\n+\t'\n+\n+\ttest_expect_success \"rev-parse --exclude-hidden=$section can only be passed once per pseudo-ref\" '\n+\t\techo \"fatal: --exclude-hidden= passed more than once\" >expected &&\n+\t\ttest_must_fail git rev-parse --exclude-hidden=$section --exclude-hidden=$section 2>err &&\n+\t\ttest_cmp expected err\n+\t'\n+\n+\tfor pseudoopt in branches tags remotes\n+\tdo\n+\t\ttest_expect_success \"rev-parse --exclude-hidden=$section fails with --$pseudoopt\" '\n+\t\t\techo \"error: --exclude-hidden cannot be used together with --$pseudoopt\" >expected &&\n+\t\t\ttest_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt 2>err &&\n+\t\t\ttest_cmp expected err\n+\t\t'\n+\n+\t\ttest_expect_success \"rev-parse --exclude-hidden=$section fails with --$pseudoopt=pattern\" '\n+\t\t\techo \"error: --exclude-hidden cannot be used together with --$pseudoopt\" >expected &&\n+\t\t\ttest_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt=pattern 2>err &&\n+\t\t\ttest_cmp expected err\n+\t\t'\n+\tdone\n+done\n+\n test_expect_success 'rev-list --exclude=glob with --branches=glob' '\n \tcompare rev-list \"--exclude=subspace-* --branches=sub*\" \"subspace/one subspace/two\"\n '\n-- \n2.38.1\n\n"},{"id":"467423","messageId":"a7eae80ff32051b28124727c957b98d6cf5fa9ea.1668663796.git.ps@pks.im","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"[PATCH v6 7/7] receive-pack: only use visible refs for connectivity check","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-11-17T05:47:04Z","receivedAt":"2022-11-17T05:48:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When serving a push, git-receive-pack(1) needs to verify that the\npackfile sent by the client contains all objects that are required by\nthe updated references. This connectivity check works by marking all\npreexisting references as uninteresting and using the new reference tips\nas starting point for a graph walk.\n\nMarking all preexisting references as uninteresting can be a problem\nwhen it comes to performance. Git forges tend to do internal bookkeeping\nto keep alive sets of objects for internal use or make them easy to find\nvia certain references. These references are typically hidden away from\nthe user so that they are neither advertised nor writeable. At GitLab,\nwe have one particular repository that contains a total of 7 million\nreferences, of which 6.8 million are indeed internal references. With\nthe current connectivity check we are forced to load all these\nreferences in order to mark them as uninteresting, and this alone takes\naround 15 seconds to compute.\n\nWe can optimize this by only taking into account the set of visible refs\nwhen marking objects as uninteresting. This means that we may now walk\nmore objects until we hit any object that is marked as uninteresting.\nBut it is rather unlikely that clients send objects that make large\nparts of objects reachable that have previously only ever been hidden,\nwhereas the common case is to push incremental changes that build on top\nof the visible object graph.\n\nThis provides a huge boost to performance in the mentioned repository,\nwhere the vast majority of its refs hidden. Pushing a new commit into\nthis repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs\nas it is configured in Gitaly leads to a 4.5-fold speedup:\n\n Benchmark 1: main\n Time (mean ± σ): 30.977 s ± 0.157 s [User: 30.226 s, System: 1.083 s]\n Range (min … max): 30.796 s … 31.071 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 6.799 s ± 0.063 s [User: 6.803 s, System: 0.354 s]\n Range (min … max): 6.729 s … 6.850 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 4.56 ± 0.05 times faster than 'main'\n\nAs we mostly go through the same codepaths even in the case where there\nare no hidden refs at all compared to the code before there is no change\nin performance when no refs are hidden:\n\n Benchmark 1: main\n Time (mean ± σ): 48.188 s ± 0.432 s [User: 49.326 s, System: 5.009 s]\n Range (min … max): 47.706 s … 48.539 s 3 runs\n\n Benchmark 2: pks-connectivity-check-hide-refs\n Time (mean ± σ): 48.027 s ± 0.500 s [User: 48.934 s, System: 5.025 s]\n Range (min … max): 47.504 s … 48.500 s 3 runs\n\n Summary\n 'pks-connectivity-check-hide-refs' ran\n 1.00 ± 0.01 times faster than 'main'\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 2 ++\n connected.c | 3 +++\n connected.h | 7 +++++++\n 3 files changed, 12 insertions(+)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1e24b31a0a..a90af30363 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,\n \topt.err_fd = err_fd;\n \topt.progress = err_fd && !quiet;\n \topt.env = tmp_objdir_env(tmp_objdir);\n+\topt.exclude_hidden_refs_section = \"receive\";\n+\n \tif (check_connected(iterate_receive_command_list, &data, &opt))\n \t\tset_connectivity_errors(commands, si);\n \ndiff --git a/connected.c b/connected.c\nindex 74a20cb32e..4f6388eed7 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \t\tstrvec_push(&rev_list.args, \"--exclude-promisor-objects\");\n \tif (!opt->is_deepening_fetch) {\n \t\tstrvec_push(&rev_list.args, \"--not\");\n+\t\tif (opt->exclude_hidden_refs_section)\n+\t\t\tstrvec_pushf(&rev_list.args, \"--exclude-hidden=%s\",\n+\t\t\t\t opt->exclude_hidden_refs_section);\n \t\tstrvec_push(&rev_list.args, \"--all\");\n \t}\n \tstrvec_push(&rev_list.args, \"--quiet\");\ndiff --git a/connected.h b/connected.h\nindex 6e59c92aa3..16b2c84f2e 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -46,6 +46,13 @@ struct check_connected_options {\n \t * during a fetch.\n \t */\n \tunsigned is_deepening_fetch : 1;\n+\n+\t/*\n+\t * If not NULL, use `--exclude-hidden=$section` to exclude all refs\n+\t * hidden via the `$section.hideRefs` config from the set of\n+\t * already-reachable refs.\n+\t */\n+\tconst char *exclude_hidden_refs_section;\n };\n \n #define CHECK_CONNECTED_INIT { 0 }\n-- \n2.38.1\n\n"},{"id":"467448","messageId":"Y3ZNUuyWF2gDk2mA@coredump.intra.peff.net","threadId":"58706","inReplyTo":"cover.1668663795.git.ps@pks.im","subject":"Re: [PATCH v6 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2022-11-17T15:03:46Z","receivedAt":"2022-11-17T15:03:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 17, 2022 at 06:46:35AM +0100, Patrick Steinhardt wrote:\n\n> this is the sixth version of my patch series that tries to improve\n> performance of the connectivity check by only considering preexisting\n> refs as uninteresting that could actually have been advertised to the\n> client.\n> \n> There are only two changes in this version compared to v5:\n> \n> - A fix to the test setup in commit 5/7 so that tests pass when\n> GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main.\n> \n> - A typo fix in the commit message of patch 6/7.\n\nThanks. Looking at the range diff, this seems good to me!\n\n-Peff\n"},{"id":"467463","messageId":"Y3amjknVMzSMN4g6@nand.local","threadId":"58706","inReplyTo":"Y3ZNUuyWF2gDk2mA@coredump.intra.peff.net","subject":"Re: [PATCH v6 0/7] receive-pack: only use visible refs for connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-11-17T21:24:30Z","receivedAt":"2022-11-17T21:26:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 17, 2022 at 10:03:46AM -0500, Jeff King wrote:\n> On Thu, Nov 17, 2022 at 06:46:35AM +0100, Patrick Steinhardt wrote:\n>\n> > this is the sixth version of my patch series that tries to improve\n> > performance of the connectivity check by only considering preexisting\n> > refs as uninteresting that could actually have been advertised to the\n> > client.\n> >\n> > There are only two changes in this version compared to v5:\n> >\n> > - A fix to the test setup in commit 5/7 so that tests pass when\n> > GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main.\n> >\n> > - A typo fix in the commit message of patch 6/7.\n>\n> Thanks. Looking at the range diff, this seems good to me!\n\nYep, I concur. Let's make sure that it passes CI, too, and then start\nmerging this down. Thanks, both.\n\nThanks,\nTaylor\n"}]}