{"thread":{"id":"59715","subject":"[PATCH 00/15] refs: implement skip lists for packed backend","startedAt":"2023-05-08T21:59:45Z","lastAt":"2023-07-11T17:20:01Z","messageCount":149,"participants":["Taylor Blau","Junio C Hamano","Chris Torek","Patrick Steinhardt","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":15},"messages":[{"id":"476818","messageId":"cover.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":null,"subject":"[PATCH 00/15] refs: implement skip lists for packed backend","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T21:59:38Z","receivedAt":"2023-05-08T21:59:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements the concept of a skip list for the packed refs\nbackend. In situations where the caller has many references they want to\nexclude, the packed-refs iterator maintains a list of (start, end)\ntuples of regions to jump over (i.e. if `iter->pos == start`, jump\nforward to `end`).\n\nThis series implements that concept, and uses it to power a couple of\nnew things:\n\n  - `git for-each-ref --exclude`, which allows callers to specify\n    prefixes they wish to discard.\n\n    In a synthetic example, the naive implementation improved runtime\n    from ~820ms to enumerate all references and discard unwanted ones,\n    down to ~106ms with `--exclude`. By the end of the series, this time\n    drops to under ~5ms.\n\n  - `git receive-pack` uses the new skip-list machinery to avoid\n    visiting references hidden via a hideRefs rule.\n\n  - `git upload-pack` has an analogous optimization as above (though it\n    can't kick in quite as often for reasons described in that patch).\n\nThe series is laid out as follows:\n\n  - The first five patches are preparatory (various refs.c and\n    ref-filter cleanup).\n  - The sixth patch provides a naive implementation of `--exclude`.\n  - The next four patches prepare for and implement skip lists in the\n    packed-refs backend.\n  - The last five patches apply the optimization in `upload-pack` and\n    `receive-pack`.\n\nThe series is ordered so that any prefix of sections list above could be\nqueued independently. Thanks in advance for your review.\n\nJeff King (5):\n  refs.c: rename `ref_filter`\n  ref-filter.h: provide `REF_FILTER_INIT`\n  ref-filter: clear reachable list pointers after freeing\n  ref-filter: add ref_filter_clear()\n  ref-filter.c: parameterize match functions over patterns\n\nTaylor Blau (10):\n  builtin/for-each-ref.c: add `--exclude` option\n  refs: plumb `exclude_patterns` argument throughout\n  refs/packed-backend.c: refactor `find_reference_location()`\n  refs/packed-backend.c: implement skip lists to avoid excluded\n    pattern(s)\n  refs/packed-backend.c: add trace2 counters for skip list\n  revision.h: store hidden refs in a `strvec`\n  refs/packed-backend.c: ignore complicated hidden refs rules\n  refs.h: let `for_each_namespaced_ref()` take excluded patterns\n  upload-pack.c: avoid enumerating hidden refs where possible\n  builtin/receive-pack.c: avoid enumerating hidden references\n\n Documentation/git-for-each-ref.txt |   6 +\n builtin/branch.c                   |   4 +-\n builtin/for-each-ref.c             |   8 +-\n builtin/receive-pack.c             |   7 +-\n builtin/tag.c                      |   4 +-\n http-backend.c                     |   2 +-\n ls-refs.c                          |   8 +-\n ref-filter.c                       |  61 ++++++--\n ref-filter.h                       |  12 ++\n refs.c                             |  61 ++++----\n refs.h                             |  15 +-\n refs/debug.c                       |   5 +-\n refs/files-backend.c               |   5 +-\n refs/packed-backend.c              | 222 ++++++++++++++++++++++++++---\n refs/refs-internal.h               |   7 +-\n revision.c                         |   4 +-\n revision.h                         |   5 +-\n t/helper/test-reach.c              |   2 +-\n t/helper/test-ref-store.c          |  10 ++\n t/t1419-exclude-refs.sh            | 131 +++++++++++++++++\n t/t6300-for-each-ref.sh            |  35 +++++\n trace2.h                           |   2 +\n trace2/tr2_ctr.c                   |   5 +\n upload-pack.c                      |  43 ++++--\n 24 files changed, 559 insertions(+), 105 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\n-- \n2.40.1.477.g956c797dfc\n"},{"id":"476819","messageId":"2225f799410687ae72f6f64daaf93809e7c9b380.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 01/15] refs.c: rename `ref_filter`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T21:59:42Z","receivedAt":"2023-05-08T21:59:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nThe refs machinery has its own implementation of a `ref_filter` (used by\n`for-each-ref`), which is distinct from the `ref-filler.h` API (also\nused by `for-each-ref`, among other things).\n\nRename the one within refs.c to more clearly indicate its purpose.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex d2a98e1c21..b9b77d2eff 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,\n \t\t\t\t   oid, flags);\n }\n \n-/* The argument to filter_refs */\n-struct ref_filter {\n+/* The argument to for_each_filter_refs */\n+struct for_each_ref_filter {\n \tconst char *pattern;\n \tconst char *prefix;\n \teach_ref_fn *fn;\n@@ -409,10 +409,11 @@ int ref_exists(const char *refname)\n \treturn refs_ref_exists(get_main_ref_store(the_repository), refname);\n }\n \n-static int filter_refs(const char *refname, const struct object_id *oid,\n-\t\t\t   int flags, void *data)\n+static int for_each_filter_refs(const char *refname,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tint flags, void *data)\n {\n-\tstruct ref_filter *filter = (struct ref_filter *)data;\n+\tstruct for_each_ref_filter *filter = data;\n \n \tif (wildmatch(filter->pattern, refname, 0))\n \t\treturn 0;\n@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tconst char *prefix, void *cb_data)\n {\n \tstruct strbuf real_pattern = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct for_each_ref_filter filter;\n \tint ret;\n \n \tif (!prefix && !starts_with(pattern, \"refs/\"))\n@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tfilter.prefix = prefix;\n \tfilter.fn = fn;\n \tfilter.cb_data = cb_data;\n-\tret = for_each_ref(filter_refs, &filter);\n+\tret = for_each_ref(for_each_filter_refs, &filter);\n \n \tstrbuf_release(&real_pattern);\n \treturn ret;\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476820","messageId":"7fe8623f606d7abf20b39b24a849d4f8aad6ecec.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 03/15] ref-filter: clear reachable list pointers after freeing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T21:59:48Z","receivedAt":"2023-05-08T21:59:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nIn reach_filter(), we pop all commits from the reachable lists, leaving\nthem empty. But because we're operating on a list pointer that was\npassed by value, the original filter.reachable_from pointer is left\ndangling.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 10aab14f03..b1d5022a51 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2416,13 +2416,13 @@ void ref_array_clear(struct ref_array *array)\n #define EXCLUDE_REACHED 0\n #define INCLUDE_REACHED 1\n static void reach_filter(struct ref_array *array,\n-\t\t\t struct commit_list *check_reachable,\n+\t\t\t struct commit_list **check_reachable,\n \t\t\t int include_reached)\n {\n \tint i, old_nr;\n \tstruct commit **to_clear;\n \n-\tif (!check_reachable)\n+\tif (!*check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n@@ -2432,7 +2432,7 @@ static void reach_filter(struct ref_array *array,\n \t}\n \n \ttips_reachable_from_bases(the_repository,\n-\t\t\t\t  check_reachable,\n+\t\t\t\t  *check_reachable,\n \t\t\t\t  to_clear, array->nr,\n \t\t\t\t  UNINTERESTING);\n \n@@ -2453,8 +2453,8 @@ static void reach_filter(struct ref_array *array,\n \n \tclear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);\n \n-\twhile (check_reachable) {\n-\t\tstruct commit *merge_commit = pop_commit(&check_reachable);\n+\twhile (*check_reachable) {\n+\t\tstruct commit *merge_commit = pop_commit(check_reachable);\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n@@ -2551,8 +2551,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \tclear_contains_cache(&ref_cbdata.no_contains_cache);\n \n \t/*  Filters that need revision walking */\n-\treach_filter(array, filter->reachable_from, INCLUDE_REACHED);\n-\treach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);\n+\treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n+\treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n \n \tsave_commit_buffer = save_commit_buffer_orig;\n \treturn ret;\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476821","messageId":"ea1c7834db8c62d04344e824eb0848bad35afd1b.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 02/15] ref-filter.h: provide `REF_FILTER_INIT`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T21:59:45Z","receivedAt":"2023-05-08T22:00:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nProvide a sane initialization value for `struct ref_filter`, which in a\nsubsequent patch will be used to initialize a new field.\n\nIn the meantime, fix a case in test-reach.c where its `ref_filter` is\nnot even zero-initialized.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c       | 3 +--\n builtin/for-each-ref.c | 3 +--\n builtin/tag.c          | 3 +--\n ref-filter.h           | 3 +++\n t/helper/test-reach.c  | 2 +-\n 5 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 501c47657c..03bb8e414c 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -662,7 +662,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \tint reflog = 0, quiet = 0, icase = 0, force = 0,\n \t    recurse_submodules_explicit = 0;\n \tenum branch_track track;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstatic struct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -720,7 +720,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \n \tsetup_ref_filter_porcelain_msg();\n \n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.kind = FILTER_REFS_BRANCHES;\n \tfilter.abbrev = -1;\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 695fc8f4a5..99ccb73518 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tint maxcount = 0, icase = 0, omit_empty = 0;\n \tstruct ref_array array;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t};\n \n \tmemset(&array, 0, sizeof(array));\n-\tmemset(&filter, 0, sizeof(filter));\n \n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 1850a6a6fd..6b41bb7374 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -443,7 +443,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tstruct msg_arg msg = { .buf = STRBUF_INIT };\n \tstruct ref_transaction *transaction;\n \tstruct strbuf err = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -501,7 +501,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tgit_config(git_tag_config, &sorting_options);\n \n \tmemset(&opt, 0, sizeof(opt));\n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.lines = -1;\n \topt.sign = -1;\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 430701cfb7..a920f73b29 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,9 @@ struct ref_format {\n \tstruct string_list bases;\n };\n \n+#define REF_FILTER_INIT { \\\n+\t.points_at = OID_ARRAY_INIT, \\\n+}\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n \t.bases = STRING_LIST_INIT_DUP, \\\ndiff --git a/t/helper/test-reach.c b/t/helper/test-reach.c\nindex 5b6f217441..ef58f10c2d 100644\n--- a/t/helper/test-reach.c\n+++ b/t/helper/test-reach.c\n@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)\n \n \t\tprintf(\"%s(X,_,_,0,0):%d\\n\", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));\n \t} else if (!strcmp(av[1], \"commit_contains\")) {\n-\t\tstruct ref_filter filter;\n+\t\tstruct ref_filter filter = REF_FILTER_INIT;\n \t\tstruct contains_cache cache;\n \t\tinit_contains_cache(&cache);\n \n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476822","messageId":"c804ba3620476713bd0535a315876378149ad7dd.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 04/15] ref-filter: add ref_filter_clear()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T21:59:52Z","receivedAt":"2023-05-08T22:00:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nWe did not bother to clean up at all in branch/tag, and for-each-ref\nonly hit a few elements. So this is probably cleaning up leaks, but I\ndidn't check yet.\n\nNote that the reachable_from and unreachable_from lists should be\ncleaned as they are used. So this is just covering any case where we\nmight bail before running the reachability check.\n---\n builtin/branch.c       |  1 +\n builtin/for-each-ref.c |  3 +--\n builtin/tag.c          |  1 +\n ref-filter.c           | 16 ++++++++++++++++\n ref-filter.h           |  3 +++\n 5 files changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 03bb8e414c..c201f0cb0b 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -813,6 +813,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\tprint_columns(&output, colopts, NULL);\n \t\tstring_list_clear(&output, 0);\n \t\tref_sorting_release(sorting);\n+\t\tref_filter_clear(&filter);\n \t\treturn 0;\n \t} else if (edit_description) {\n \t\tconst char *branch_name;\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 99ccb73518..c01fa6fefe 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&err);\n \tstrbuf_release(&output);\n \tref_array_clear(&array);\n-\tfree_commit_list(filter.with_commit);\n-\tfree_commit_list(filter.no_commit);\n+\tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\n \tstrvec_clear(&vec);\n \treturn 0;\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 6b41bb7374..aab5e693fe 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -645,6 +645,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \n cleanup:\n \tref_sorting_release(sorting);\n+\tref_filter_clear(&filter);\n \tstrbuf_release(&buf);\n \tstrbuf_release(&ref);\n \tstrbuf_release(&reflog_msg);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex b1d5022a51..9ea92b9637 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2864,3 +2864,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)\n \n \treturn 0;\n }\n+\n+void ref_filter_init(struct ref_filter *filter)\n+{\n+\tstruct ref_filter blank = REF_FILTER_INIT;\n+\tmemcpy(filter, &blank, sizeof(blank));\n+}\n+\n+void ref_filter_clear(struct ref_filter *filter)\n+{\n+\toid_array_clear(&filter->points_at);\n+\tfree_commit_list(filter->with_commit);\n+\tfree_commit_list(filter->no_commit);\n+\tfree_commit_list(filter->reachable_from);\n+\tfree_commit_list(filter->unreachable_from);\n+\tref_filter_init(filter);\n+}\ndiff --git a/ref-filter.h b/ref-filter.h\nindex a920f73b29..160b807224 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,\n \t\t\t struct ref_format *format,\n \t\t\t struct ref_array *array);\n \n+void ref_filter_init(struct ref_filter *filter);\n+void ref_filter_clear(struct ref_filter *filter);\n+\n #endif /*  REF_FILTER_H  */\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476823","messageId":"c54000f5f5122c4ca3ac9b16828a8fd77050768c.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 05/15] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T21:59:55Z","receivedAt":"2023-05-08T22:00:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\n`match_pattern()` and `match_name_as_path()` both take a `struct\nref_filter *`, and then store a stack variable `patterns` pointing at\n`filter->patterns`.\n\nThe subsequent patch will add a new array of patterns to match over (the\nexcluded patterns, via a new `git for-each-ref --exclude` option),\ntreating the return value of these functions differently depending on\nwhich patterns are being used to match.\n\nTweak `match_pattern()` and `match_name_as_path()` to take an array of\npatterns to prepare for passing either in.\n\nCo-authored-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 9ea92b9637..6c5eed144f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2102,9 +2102,10 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n  * matches a pattern \"refs/heads/mas\") or a wildcard (e.g. the same ref\n  * matches \"refs/heads/mas*\", too).\n  */\n-static int match_pattern(const struct ref_filter *filter, const char *refname)\n+static int match_pattern(const struct ref_filter *filter,\n+\t\t\t const char **patterns,\n+\t\t\t const char *refname)\n {\n-\tconst char **patterns = filter->name_patterns;\n \tunsigned flags = 0;\n \n \tif (filter->ignore_case)\n@@ -2132,9 +2133,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n  * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n  * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n  */\n-static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n+static int match_name_as_path(const struct ref_filter *filter,\n+\t\t\t      const char **pattern,\n+\t\t\t      const char *refname)\n {\n-\tconst char **pattern = filter->name_patterns;\n \tint namelen = strlen(refname);\n \tunsigned flags = WM_PATHNAME;\n \n@@ -2163,8 +2165,8 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \tif (!*filter->name_patterns)\n \t\treturn 1; /* No pattern always matches */\n \tif (filter->match_as_path)\n-\t\treturn match_name_as_path(filter, refname);\n-\treturn match_pattern(filter, refname);\n+\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n+\treturn match_pattern(filter, filter->name_patterns, refname);\n }\n \n /*\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476824","messageId":"ea5c0ddc10cd484c2af8505b8078697896ff3bad.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 06/15] builtin/for-each-ref.c: add `--exclude` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T21:59:58Z","receivedAt":"2023-05-08T22:00:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When using `for-each-ref`, it is sometimes convenient for the caller to\nbe able to exclude certain parts of the references.\n\nFor example, if there are many `refs/__hidden__/*` references, the\ncaller may want to emit all references *except* the hidden ones.\nCurrently, the only way to do this is to post-process the output, like:\n\n    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'\n\nWhich is do-able, but requires processing a potentially large quantity\nof references.\n\nTeach `git for-each-ref` a new `--exclude=<pattern>` option, which\nexcludes references from the results if they match one or more excluded\npatterns.\n\nThis patch provides a naive implementation where the `ref_filter` still\nsees all references (including ones that it will discard) and is left to\ncheck whether each reference matches any excluded pattern(s) before\nemitting them.\n\nBy culling out references we know the caller doesn't care about, we can\navoid allocating memory for their storage, as well as spending time\nsorting the output (among other things). Even the naive implementation\nprovides a significant speed-up on a modified copy of linux.git (that\nhas a hidden ref pointing at each commit):\n\n    $ hyperfine \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/'\n    Benchmark 1: git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]\n      Range (min … max):   817.7 ms … 823.3 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/\n      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]\n      Range (min … max):   104.7 ms … 109.1 ms    27 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/' ran\n        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"'\n\nSubsequent patches will improve on this by avoiding visiting excluded\nsections of the `packed-refs` file in certain cases.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-for-each-ref.txt |  6 +++++\n builtin/for-each-ref.c             |  2 ++\n ref-filter.c                       | 13 +++++++++++\n ref-filter.h                       |  6 +++++\n t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++\n 5 files changed, 62 insertions(+)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 1e215d4e73..5743eb5def 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t\t   [--points-at=<object>]\n \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n+\t\t   [--exclude=<pattern> ...]\n \n DESCRIPTION\n -----------\n@@ -102,6 +103,11 @@ OPTIONS\n \tDo not print a newline after formatted refs where the format expands\n \tto the empty string.\n \n+--exclude=<pattern>::\n+\tIf one or more patterns are given, only refs which do not match\n+\tany excluded pattern(s) are shown. Matching is done using the\n+\tsame rules as `<pattern>` above.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex c01fa6fefe..449da61e11 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -14,6 +14,7 @@ static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [--points-at <object>]\"),\n \tN_(\"git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]\"),\n \tN_(\"git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]\"),\n+\tN_(\"git for-each-ref [--exclude=<pattern> ...]\"),\n \tNULL\n };\n \n@@ -47,6 +48,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n+\t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n \t\t\t     N_(\"object\"), N_(\"print only refs which points at the given object\"),\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 6c5eed144f..93dc9b331f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2169,6 +2169,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \treturn match_pattern(filter, filter->name_patterns, refname);\n }\n \n+static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n+{\n+\tif (!filter->exclude.nr)\n+\t\treturn 0;\n+\tif (filter->match_as_path)\n+\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n+\treturn match_pattern(filter, filter->exclude.v, refname);\n+}\n+\n /*\n  * This is the same as for_each_fullref_in(), but it tries to iterate\n  * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2336,6 +2345,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \tif (!filter_pattern_match(filter, refname))\n \t\treturn 0;\n \n+\tif (filter_exclude_match(filter, refname))\n+\t\treturn 0;\n+\n \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n \t\treturn 0;\n \n@@ -2875,6 +2887,7 @@ void ref_filter_init(struct ref_filter *filter)\n \n void ref_filter_clear(struct ref_filter *filter)\n {\n+\tstrvec_clear(&filter->exclude);\n \toid_array_clear(&filter->points_at);\n \tfree_commit_list(filter->with_commit);\n \tfree_commit_list(filter->no_commit);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 160b807224..1524bc463a 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -6,6 +6,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -59,6 +60,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\n \tstruct commit_list *no_commit;\n@@ -94,6 +96,7 @@ struct ref_format {\n \n #define REF_FILTER_INIT { \\\n \t.points_at = OID_ARRAY_INIT, \\\n+\t.exclude = STRVEC_INIT, \\\n }\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n@@ -112,6 +115,9 @@ struct ref_format {\n #define OPT_REF_SORT(var) \\\n \tOPT_STRING_LIST(0, \"sort\", (var), \\\n \t\t\tN_(\"key\"), N_(\"field name to sort on\"))\n+#define OPT_REF_FILTER_EXCLUDE(var) \\\n+\tOPT_STRVEC(0, \"exclude\", &(var)->exclude, \\\n+\t\t   N_(\"pattern\"), N_(\"exclude refs which match pattern\"))\n \n /*\n  * API for filtering a set of refs. Based on the type of refs the user\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 5c00607608..7e8d578522 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '\n \ttest_cmp expected actual\n '\n \n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with prefix exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=refs/tags/foo >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/foo/one\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with pattern exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=\"refs/tags/foo/t*\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n cat >expected <<\\EOF\n 'refs/heads/main'\n 'refs/remotes/origin/main'\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476825","messageId":"f437cd83e2d2177b1526673821b950f47814abc6.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 07/15] refs: plumb `exclude_patterns` argument throughout","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:01Z","receivedAt":"2023-05-08T22:00:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The subsequent patch will want to access an optional `excluded_patterns`\narray within refs/packed-backend.c. To do so, the refs subsystem needs\nto be updated to pass this value across a number of different locations.\n\nPrepare for a future patch by introducing this plumbing now, passing\nNULLs at top-level APIs in order to make that patch less noisy and more\neasily readable.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.co>\n---\n ls-refs.c             |  2 +-\n ref-filter.c          |  5 +++--\n refs.c                | 32 +++++++++++++++++++-------------\n refs.h                |  8 +++++++-\n refs/debug.c          |  5 +++--\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  5 +++--\n refs/refs-internal.h  |  7 ++++---\n revision.c            |  2 +-\n 9 files changed, 44 insertions(+), 27 deletions(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex b9f3e08ec3..c7ad39611a 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -192,7 +192,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  send_ref, &data);\n+\t\t\t\t\t  NULL, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 93dc9b331f..c8ced1104b 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2207,12 +2207,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n+\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t cb, cb_data);\n+\t\t\t\t\t\t NULL, cb, cb_data);\n }\n \n /*\ndiff --git a/refs.c b/refs.c\nindex b9b77d2eff..538bde644e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)\n \n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n+\t\tconst char *prefix,\n+\t\tconst char **exclude_patterns,\n+\t\tint trim,\n \t\tenum do_for_each_ref_flags flags)\n {\n \tstruct ref_iterator *iter;\n@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(\n \t\t}\n \t}\n \n-\titer = refs->be->iterator_begin(refs, prefix, flags);\n-\n+\titer = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);\n \t/*\n \t * `iterator_begin()` already takes care of prefix, but we\n \t * might need to do some trimming:\n@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);\n \n \treturn do_for_each_repo_ref_iterator(r, iter, fn, cb_data);\n }\n@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,\n }\n \n static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n+\t\t\t   const char **exclude_patterns,\n \t\t\t   each_ref_fn fn, int trim,\n \t\t\t   enum do_for_each_ref_flags flags, void *cb_data)\n {\n@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,\n+\t\t\t\t       flags);\n \n \treturn do_for_each_repo_ref_iterator(the_repository, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \n int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t       prefix, fn, 0, 0, cb_data);\n+\t\t\t       prefix, NULL, fn, 0, 0, cb_data);\n }\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);\n }\n \n int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\n \n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0,\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n \t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,\n int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \t\t\t\t      const char *namespace,\n \t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data)\n {\n \tstruct string_list prefixes = STRING_LIST_INIT_DUP;\n@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \n \tfor_each_string_list_item(prefix, &prefixes) {\n \t\tstrbuf_addstr(&buf, prefix->string);\n-\t\tret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);\n+\t\tret = refs_for_each_fullref_in(ref_store, buf.buf,\n+\t\t\t\t\t       exclude_patterns, fn, cb_data);\n \t\tif (ret)\n \t\t\tbreak;\n \t\tstrbuf_setlen(&buf, namespace_len);\n@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,\n \tstrbuf_addstr(&dirname, refname + dirname.len);\n \tstrbuf_addch(&dirname, '/');\n \n-\titer = refs_ref_iterator_begin(refs, dirname.buf, 0,\n+\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 123cfa4424..d672d636cf 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -338,6 +338,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);\n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data);\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n@@ -345,10 +346,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n  * iterate all refs in \"patterns\" by partitioning patterns into disjoint sets\n  * and iterating the longest-common prefix of each set.\n  *\n+ * references matching any pattern in \"exclude_patterns\" are omitted from the\n+ * result set on a best-effort basis.\n+ *\n  * callers should be prepared to ignore references that they did not ask for.\n  */\n int refs_for_each_fullref_in_prefixes(struct ref_store *refs,\n-\t\t\t\t      const char *namespace, const char **patterns,\n+\t\t\t\t      const char *namespace,\n+\t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data);\n \n /**\ndiff --git a/refs/debug.c b/refs/debug.c\nindex adc34c836f..8131133e99 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -228,11 +228,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \n static struct ref_iterator *\n debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n-\t\t\t unsigned int flags)\n+\t\t\t const char **exclude_patterns, unsigned int flags)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n \tstruct ref_iterator *res =\n-\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix, flags);\n+\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix,\n+\t\t\t\t\t\texclude_patterns, flags);\n \tstruct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));\n \tbase_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);\n \tditer->iter = res;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d0581ee41a..5fae864334 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -827,7 +827,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \n static struct ref_iterator *files_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct files_ref_store *refs;\n \tstruct ref_iterator *loose_iter, *packed_iter, *overlay_iter;\n@@ -872,7 +873,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * the packed and loose references.\n \t */\n \tpacked_iter = refs_ref_iterator_begin(\n-\t\t\trefs->packed_ref_store, prefix, 0,\n+\t\t\trefs->packed_ref_store, prefix, exclude_patterns, 0,\n \t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n \n \toverlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 34c0c4e20f..e54e78e540 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -923,7 +923,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n@@ -1148,7 +1149,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t * list of refs is exhausted, set iter to NULL. When the list\n \t * of updates is exhausted, leave i set to updates->nr.\n \t */\n-\titer = packed_ref_iterator_begin(&refs->base, \"\",\n+\titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n \tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n \t\titer = NULL;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a85d113123..28a11b9d61 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n  */\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n-\t\tenum do_for_each_ref_flags flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tint trim, enum do_for_each_ref_flags flags);\n \n /*\n  * A callback function used to instruct merge_ref_iterator how to\n@@ -570,7 +570,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,\n  */\n typedef struct ref_iterator *ref_iterator_begin_fn(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags);\n \n /* reflog functions */\n \ndiff --git a/revision.c b/revision.c\nindex b33cc1d106..89953592f9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,\n \tstruct strbuf bisect_refs = STRBUF_INIT;\n \tint status;\n \tstrbuf_addf(&bisect_refs, \"refs/bisect/%s\", term);\n-\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);\n+\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);\n \tstrbuf_release(&bisect_refs);\n \treturn status;\n }\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476826","messageId":"836a5665b7df065811edc678cb8e70004f7b7c49.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 08/15] refs/packed-backend.c: refactor `find_reference_location()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:05Z","receivedAt":"2023-05-08T22:00:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `find_reference_location()` is used to perform a\nbinary search-like function over the contents of a repository's\n`$GIT_DIR/packed-refs` file.\n\nThe search it implements is unlike a standard binary search in that the\nrecords it searches over are not of a fixed width, so the comparison\nmust locate the end of a record before comparing it.\n\nExtract the core routine of `find_reference_location()` in order to\nimplement a function in the following patch which will find the first\nlocation in the `packed-refs` file that *doesn't* match the given\npattern.\n\nThe behavior of `find_reference_location()` is unchanged.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c | 46 +++++++++++++++++++++++++------------------\n 1 file changed, 27 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex e54e78e540..98f96bf3ee 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -302,7 +302,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)\n  * Compare a snapshot record at `rec` to the specified NUL-terminated\n  * refname.\n  */\n-static int cmp_record_to_refname(const char *rec, const char *refname)\n+static int cmp_record_to_refname(const char *rec, const char *refname,\n+\t\t\t\t int start)\n {\n \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n \tconst char *r2 = refname;\n@@ -311,7 +312,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)\n \t\tif (*r1 == '\\n')\n \t\t\treturn *r2 ? -1 : 0;\n \t\tif (!*r2)\n-\t\t\treturn 1;\n+\t\t\treturn start ? 1 : -1;\n \t\tif (*r1 != *r2)\n \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n \t\tr1++;\n@@ -526,22 +527,9 @@ static int load_contents(struct snapshot *snapshot)\n \treturn 1;\n }\n \n-/*\n- * Find the place in `snapshot->buf` where the start of the record for\n- * `refname` starts. If `mustexist` is true and the reference doesn't\n- * exist, then return NULL. If `mustexist` is false and the reference\n- * doesn't exist, then return the point where that reference would be\n- * inserted, or `snapshot->eof` (which might be NULL) if it would be\n- * inserted at the end of the file. In the latter mode, `refname`\n- * doesn't have to be a proper reference name; for example, one could\n- * search for \"refs/replace/\" to find the start of any replace\n- * references.\n- *\n- * The record is sought using a binary search, so `snapshot->buf` must\n- * be sorted.\n- */\n-static const char *find_reference_location(struct snapshot *snapshot,\n-\t\t\t\t\t   const char *refname, int mustexist)\n+static const char *find_reference_location_1(struct snapshot *snapshot,\n+\t\t\t\t\t     const char *refname, int mustexist,\n+\t\t\t\t\t     int start)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -571,7 +559,7 @@ static const char *find_reference_location(struct snapshot *snapshot,\n \n \t\tmid = lo + (hi - lo) / 2;\n \t\trec = find_start_of_record(lo, mid);\n-\t\tcmp = cmp_record_to_refname(rec, refname);\n+\t\tcmp = cmp_record_to_refname(rec, refname, start);\n \t\tif (cmp < 0) {\n \t\t\tlo = find_end_of_record(mid, hi);\n \t\t} else if (cmp > 0) {\n@@ -587,6 +575,26 @@ static const char *find_reference_location(struct snapshot *snapshot,\n \t\treturn lo;\n }\n \n+/*\n+ * Find the place in `snapshot->buf` where the start of the record for\n+ * `refname` starts. If `mustexist` is true and the reference doesn't\n+ * exist, then return NULL. If `mustexist` is false and the reference\n+ * doesn't exist, then return the point where that reference would be\n+ * inserted, or `snapshot->eof` (which might be NULL) if it would be\n+ * inserted at the end of the file. In the latter mode, `refname`\n+ * doesn't have to be a proper reference name; for example, one could\n+ * search for \"refs/replace/\" to find the start of any replace\n+ * references.\n+ *\n+ * The record is sought using a binary search, so `snapshot->buf` must\n+ * be sorted.\n+ */\n+static const char *find_reference_location(struct snapshot *snapshot,\n+\t\t\t\t\t   const char *refname, int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n+}\n+\n /*\n  * Create a newly-allocated `snapshot` of the `packed-refs` file in\n  * its current state and return it. The return value will already have\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476827","messageId":"a39d1107c1f3e9fbd859a23aed72e63bbd394fa2.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:08Z","receivedAt":"2023-05-08T22:00:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When iterating through the `packed-refs` file in order to answer a query\nlike:\n\n    $ git for-each-ref --exclude=refs/__hidden__\n\nit would be useful to avoid walking over all of the entries in\n`refs/__hidden__/*` when possible, since we know that the ref-filter\ncode is going to throw them away anyways.\n\nIn certain circumstances, doing so is possible. The algorithm for doing\nso is as follows:\n\n  - For each excluded pattern, find the first record that matches it,\n    and the first pattern that *doesn't* match it (i.e. the location\n    you'd next want to consider when excluding that pattern).\n\n  - Sort the patterns by their starting location within the\n    `packed-refs` file.\n\n  - Construct a skip list of regions by combining adjacent and\n    overlapping regions from the previous step.\n\n  - When iterating through the `packed-refs` file, if `iter->pos` is\n    ever contained in one of the regions from the previous steps,\n    advance `iter->pos` past the end of that region, and continue\n    enumeration.\n\nNote that this optimization is only possible when none of the excluded\npattern(s) have special meta-characters in them. To see why this is the\ncase, consider the exclusion pattern \"refs/foo[a]\". In general, in order\nto find the location of the first record that matches this pattern, we\ncould only consider up to the first meta-character, \"refs/foo\". But this\ndoesn't work, since the excluded region we'd come up with would include\n\"refs/foobar\", even though it is not excluded.\n\nThere are a few other gotchas worth considering. First, note that the\nskip list is sorted, so once we skip past a region, we can avoid\nconsidering it (or any regions preceding it) again. The member\n`skip_pos` is used to track the first next-possible region to jump\nthrough.\n\nSecond, note that the exclusion list is best-effort, since we do not\nhandle loose references, and because of the meta-character issue above.\n\nIn repositories with a large number of hidden references, the speed-up\ncan be significant. Tests here are done with a copy of linux.git with a\nreference \"refs/pull/N\" pointing at every commit, as in:\n\n    $ git rev-list HEAD | awk '{ print \"create refs/pull/\" NR \" \" $0 }' |\n        git update-ref --stdin\n    $ git pack-refs --all\n\n, it is significantly faster to have `for-each-ref` skip over the\nexcluded references, as opposed to filtering them out after the fact:\n\n    $ hyperfine \\\n      'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n    Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n      Range (min … max):   800.0 ms … 807.7 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n      Range (min … max):     4.3 ms …   6.7 ms    422 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n      172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n\nUsing the skip list is fairly straightforward (see the changes to\n`refs/packed-backend.c::next_record()`), but constructing the list is\nnot. To ensure that the construction is correct, add a new suite of\ntests in t1419 covering various corner cases (overlapping regions,\npartially overlapping regions, adjacent regions, etc.).\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c              |   5 +-\n refs/packed-backend.c     | 150 +++++++++++++++++++++++++++++++++++++-\n t/helper/test-ref-store.c |  10 +++\n t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++++\n 4 files changed, 263 insertions(+), 3 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex c8ced1104b..56ebd332fa 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2208,12 +2208,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n \t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n+\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t NULL, cb, cb_data);\n+\t\t\t\t\t\t filter->exclude.v,\n+\t\t\t\t\t\t cb, cb_data);\n }\n \n /*\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 98f96bf3ee..137a4233f6 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -595,6 +595,21 @@ static const char *find_reference_location(struct snapshot *snapshot,\n \treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n }\n \n+/*\n+ * Find the place in `snapshot->buf` after the end of the record for\n+ * `refname`. In other words, find the location of first thing *after*\n+ * `refname`.\n+ *\n+ * Other semantics are identical to the ones in\n+ * `find_reference_location()`.\n+ */\n+static const char *find_reference_location_end(struct snapshot *snapshot,\n+\t\t\t\t\t       const char *refname,\n+\t\t\t\t\t       int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 0);\n+}\n+\n /*\n  * Create a newly-allocated `snapshot` of the `packed-refs` file in\n  * its current state and return it. The return value will already have\n@@ -786,6 +801,13 @@ struct packed_ref_iterator {\n \t/* The end of the part of the buffer that will be iterated over: */\n \tconst char *eof;\n \n+\tstruct skip_list_entry {\n+\t\tconst char *start;\n+\t\tconst char *end;\n+\t} *skip;\n+\tsize_t skip_nr, skip_alloc;\n+\tsize_t skip_pos;\n+\n \t/* Scratch space for current values: */\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n@@ -803,14 +825,34 @@ struct packed_ref_iterator {\n  */\n static int next_record(struct packed_ref_iterator *iter)\n {\n-\tconst char *p = iter->pos, *eol;\n+\tconst char *p, *eol;\n \n \tstrbuf_reset(&iter->refname_buf);\n \n+\t/*\n+\t * If iter->pos is contained within a skipped region, jump past\n+\t * it.\n+\t *\n+\t * Note that each skipped region is considered at most once,\n+\t * since they are ordered based on their starting position.\n+\t */\n+\twhile (iter->skip_pos < iter->skip_nr) {\n+\t\tstruct skip_list_entry *curr = &iter->skip[iter->skip_pos];\n+\t\tif (iter->pos < curr->start)\n+\t\t\tbreak; /* not to the next jump yet */\n+\n+\t\titer->skip_pos++;\n+\t\tif (iter->pos < curr->end) {\n+\t\t\titer->pos = curr->end;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n \tif (iter->pos == iter->eof)\n \t\treturn ITER_DONE;\n \n \titer->base.flags = REF_ISPACKED;\n+\tp = iter->pos;\n \n \tif (iter->eof - p < the_hash_algo->hexsz + 2 ||\n \t    parse_oid_hex(p, &iter->oid, &p) ||\n@@ -918,6 +960,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \tint ok = ITER_DONE;\n \n \tstrbuf_release(&iter->refname_buf);\n+\tfree(iter->skip);\n \trelease_snapshot(iter->snapshot);\n \tbase_ref_iterator_free(ref_iterator);\n \treturn ok;\n@@ -929,6 +972,108 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.abort = packed_ref_iterator_abort\n };\n \n+static int skip_list_entry_cmp(const void *va, const void *vb)\n+{\n+\tconst struct skip_list_entry *a = va;\n+\tconst struct skip_list_entry *b = vb;\n+\n+\tif (a->start < b->start)\n+\t\treturn -1;\n+\tif (a->start > b->start)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int has_glob_special(const char *str)\n+{\n+\tconst char *p;\n+\tfor (p = str; *p; p++) {\n+\t\tif (is_glob_special(*p))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static const char *ptr_max(const char *x, const char *y)\n+{\n+\tif (x > y)\n+\t\treturn x;\n+\treturn y;\n+}\n+\n+static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n+\t\t\t\t\tstruct snapshot *snapshot,\n+\t\t\t\t\tconst char **excluded_patterns)\n+{\n+\tsize_t i, j;\n+\tconst char **pattern;\n+\n+\tif (!excluded_patterns)\n+\t\treturn;\n+\n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\tstruct skip_list_entry *e;\n+\n+\t\t/*\n+\t\t * We can't feed any excludes with globs in them to the\n+\t\t * refs machinery.  It only understands prefix matching.\n+\t\t * We likewise can't even feed the string leading up to\n+\t\t * the first meta-character, as something like \"foo[a]\"\n+\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n+\t\t * would match that and mark it for exclusion).\n+\t\t */\n+\t\tif (has_glob_special(*pattern))\n+\t\t\tcontinue;\n+\n+\t\tALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);\n+\n+\t\te = &iter->skip[iter->skip_nr++];\n+\t\te->start = find_reference_location(snapshot, *pattern, 0);\n+\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n+\t}\n+\n+\tif (!iter->skip_nr) {\n+\t\t/*\n+\t\t * Every entry in exclude_patterns has a meta-character,\n+\t\t * nothing to do here.\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tQSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp);\n+\n+\t/*\n+\t * As an optimization, merge adjacent entries in the skip list\n+\t * to jump forwards as far as possible when entering a skipped\n+\t * region.\n+\t *\n+\t * For example, if we have two skipped regions:\n+\t *\n+\t *\t[[A, B], [B, C]]\n+\t *\n+\t * we want to combine that into a single entry jumping from A to\n+\t * C.\n+\t */\n+\tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n+\t\tstruct skip_list_entry *ours = &iter->skip[i];\n+\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n+\n+\t\tif (ours->start == ours->end) {\n+\t\t\t/* ignore empty regions (no matching entries) */\n+\t\t\tcontinue;\n+\t\t} else if (prev->end >= ours->start) {\n+\t\t\t/* overlapping regions extend the previous one */\n+\t\t\tprev->end = ptr_max(prev->end, ours->end);\n+\t\t} else {\n+\t\t\t/* otherwise, insert a new region */\n+\t\t\titer->skip[j++] = *ours;\n+\t\t}\n+\t}\n+\n+\titer->skip_nr = j;\n+\titer->skip_pos = 0;\n+}\n+\n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n \t\tconst char *prefix, const char **exclude_patterns,\n@@ -964,6 +1109,9 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);\n \n+\tif (exclude_patterns)\n+\t\tpopulate_excluded_skip_list(iter, snapshot, exclude_patterns);\n+\n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n \ndiff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\nindex 6d8f844e9c..2bff003f7c 100644\n--- a/t/helper/test-ref-store.c\n+++ b/t/helper/test-ref-store.c\n@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n \treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n }\n \n+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)\n+{\n+\tconst char *prefix = notnull(*argv++, \"prefix\");\n+\tconst char **exclude_patterns = argv;\n+\n+\treturn refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,\n+\t\t\t\t\tNULL);\n+}\n+\n static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\n {\n \tstruct object_id oid = *null_oid();\n@@ -307,6 +316,7 @@ static struct command commands[] = {\n \t{ \"delete-refs\", cmd_delete_refs },\n \t{ \"rename-ref\", cmd_rename_ref },\n \t{ \"for-each-ref\", cmd_for_each_ref },\n+\t{ \"for-each-ref--exclude\", cmd_for_each_ref__exclude },\n \t{ \"resolve-ref\", cmd_resolve_ref },\n \t{ \"verify-ref\", cmd_verify_ref },\n \t{ \"for-each-reflog\", cmd_for_each_reflog },\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nnew file mode 100755\nindex 0000000000..da5265a5a8\n--- /dev/null\n+++ b/t/t1419-exclude-refs.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='test exclude_patterns functionality in main ref store'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+for_each_ref__exclude () {\n+\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tcut -d ' ' -f 2 actual.raw\n+}\n+\n+for_each_ref () {\n+\tgit for-each-ref --format='%(refname)' \"$@\"\n+}\n+\n+test_expect_success 'setup' '\n+\ttest_commit --no-tag base &&\n+\tbase=\"$(git rev-parse HEAD)\" &&\n+\n+\tfor name in foo bar baz quux\n+\tdo\n+\t\tfor i in 1 2 3\n+\t\tdo\n+\t\t\techo \"create refs/heads/$name/$i $base\" || return 1\n+\t\tdone || return 1\n+\tdone >in &&\n+\techo \"delete refs/heads/main\" >>in &&\n+\n+\tgit update-ref --stdin <in &&\n+\tgit pack-refs --all\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '\n+\t# region in middle\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' '\n+\t# region at beginning\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' '\n+\t# region at end\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' '\n+\t# disjoint regions\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' '\n+\t# adjacent, non-overlapping regions\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' '\n+\t# overlapping region\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' '\n+\t# empty region\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '\n+\t# discards meta-characters\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476828","messageId":"5698c2794f8c780b4f8d8591ceb4475ec26d17c6.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 10/15] refs/packed-backend.c: add trace2 counters for skip list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:11Z","receivedAt":"2023-05-08T22:00:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The previous commit added low-level tests to ensure that the packed-refs\niterator did not enumerate excluded sections of the refspace.\n\nHowever, there was no guarantee that these sections weren't being\nvisited, only that they were being suppressed from the output. To harden\nthese tests, add a trace2 counter which tracks the number of regions\nskipped by the packed-refs iterator, and assert on its value.\n\nSuggested-by: Derrick Stolee <derrickstolee@github.com>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   |  2 ++\n t/t1419-exclude-refs.sh | 54 ++++++++++++++++++++++++++++-------------\n trace2.h                |  2 ++\n trace2/tr2_ctr.c        |  5 ++++\n 4 files changed, 46 insertions(+), 17 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 137a4233f6..ddfa9add14 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -11,6 +11,7 @@\n #include \"../chdir-notify.h\"\n #include \"../wrapper.h\"\n #include \"../write-or-die.h\"\n+#include \"../trace2.h\"\n \n enum mmap_strategy {\n \t/*\n@@ -844,6 +845,7 @@ static int next_record(struct packed_ref_iterator *iter)\n \t\titer->skip_pos++;\n \t\tif (iter->pos < curr->end) {\n \t\t\titer->pos = curr->end;\n+\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, 1);\n \t\t\tbreak;\n \t\t}\n \t}\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex da5265a5a8..051b5a54ce 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -9,7 +9,8 @@ TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n for_each_ref__exclude () {\n-\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tGIT_TRACE2_PERF=1 test-tool ref-store main \\\n+\t\tfor-each-ref--exclude \"$@\" >actual.raw\n \tcut -d ' ' -f 2 actual.raw\n }\n \n@@ -17,6 +18,17 @@ for_each_ref () {\n \tgit for-each-ref --format='%(refname)' \"$@\"\n }\n \n+assert_skips () {\n+\tlocal nr=\"$1\"\n+\tlocal trace=\"$2\"\n+\n+\tgrep -q \"name:skips_made value:$nr\" $trace\n+}\n+\n+assert_no_skips () {\n+\t! assert_skips \".*\" \"$1\"\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit --no-tag base &&\n \tbase=\"$(git rev-parse HEAD)\" &&\n@@ -36,66 +48,74 @@ test_expect_success 'setup' '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '\n \t# region in middle\n-\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_skips 1 perf\n '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' '\n \t# region at beginning\n-\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_skips 1 perf\n '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' '\n \t# region at end\n-\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_skips 1 perf\n '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' '\n \t# disjoint regions\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_skips 2 perf\n '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' '\n \t# adjacent, non-overlapping regions\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_skips 1 perf\n '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' '\n \t# overlapping region\n-\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_skips 1 perf\n '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' '\n \t# empty region\n-\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_skips\n '\n \n test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '\n \t# discards meta-characters\n-\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_skips\n '\n \n test_done\ndiff --git a/trace2.h b/trace2.h\nindex 4ced30c0db..6a116f60a9 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -551,6 +551,8 @@ enum trace2_counter_id {\n \tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n \tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n \n+\tTRACE2_COUNTER_ID_PACKED_REFS_SKIPS, /* counts number of skips */\n+\n \t/* Add additional counter definitions before here. */\n \tTRACE2_NUMBER_OF_COUNTERS\n };\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nindex b342d3b1a3..f7efbc7646 100644\n--- a/trace2/tr2_ctr.c\n+++ b/trace2/tr2_ctr.c\n@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER\n \t\t.name = \"test2\",\n \t\t.want_per_thread_events = 1,\n \t},\n+\t[TRACE2_COUNTER_ID_PACKED_REFS_SKIPS] = {\n+\t\t.category = \"packed-refs\",\n+\t\t.name = \"skips_made\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n \n \t/* Add additional metadata before here. */\n };\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476829","messageId":"dd5b34185cdb42eb43738aa837e811e42739b659.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 12/15] refs/packed-backend.c: ignore complicated hidden refs rules","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:19Z","receivedAt":"2023-05-08T22:00:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\nuse the new skip-list feature in the packed-refs iterator by ignoring\nreferences which are mentioned via its respective hideRefs lists.\n\nHowever, the packed-ref skip lists cannot handle un-hiding rules (that\nbegin with '!'), or namespace comparisons (that begin with '^'). Detect\nand avoid these cases by falling back to the normal enumeration without\na skip list when such patterns exist.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   | 19 +++++++++++++++++++\n t/t1419-exclude-refs.sh | 10 ++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex ddfa9add14..7f09201f35 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1013,6 +1013,25 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n \tif (!excluded_patterns)\n \t\treturn;\n \n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\t/*\n+\t\t * We also can't feed any excludes from hidden refs\n+\t\t * config sections, since later rules may override\n+\t\t * previous ones. For example, with rules \"refs/foo\" and\n+\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n+\t\t * everything underneath it), but the earlier exclusion\n+\t\t * would cause us to skip all of \"refs/foo\". We likewise\n+\t\t * don't implement the namespace stripping required for\n+\t\t * '^' rules.\n+\t\t *\n+\t\t * Both are possible to do, but complicated, so avoid\n+\t\t * populating the skip list at all if we see either of\n+\t\t * these patterns.\n+\t\t */\n+\t\tif (**pattern == '!' || **pattern == '^')\n+\t\t\treturn;\n+\t}\n+\n \tfor (pattern = excluded_patterns; *pattern; pattern++) {\n \t\tstruct skip_list_entry *e;\n \ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex 051b5a54ce..026e4414cd 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -118,4 +118,14 @@ test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '\n \tassert_no_skips\n '\n \n+test_expect_success 'for_each_ref__exclude(refs/heads/foo, !refs/heads/foo/1)' '\n+\t# discards complex hidden ref rules\n+\tfor_each_ref__exclude refs/heads refs/heads/foo \"!refs/heads/foo/1\" \\\n+\t\t>actual 2>perf &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual &&\n+\tassert_no_skips\n+'\n+\n test_done\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476830","messageId":"5b9814ad8c75029f836f051394ab387517ac3803.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 11/15] revision.h: store hidden refs in a `strvec`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:15Z","receivedAt":"2023-05-08T22:00:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, it will be convenient to have a 'const char **'\nof hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\netc.), instead of a `string_list`.\n\nConvert spots throughout the tree that store the list of hidden refs\nfrom a `string_list` to a `strvec`.\n\nNote that in `parse_hide_refs_config()` there is an ugly const-cast used\nto avoid an extra copy of each value before trimming any trailing slash\ncharacters. This could instead be written as:\n\n    ref = xstrdup(value);\n    len = strlen(ref);\n    while (len && ref[len - 1] == '/')\n            ref[--len] = '\\0';\n    strvec_push(hide_refs, ref);\n    free(ref);\n\nbut the double-copy (once when calling `xstrdup()`, and another via\n`strvec_push()`) is wasteful.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c |  4 ++--\n ls-refs.c              |  6 +++---\n refs.c                 | 11 ++++++-----\n refs.h                 |  4 ++--\n revision.c             |  2 +-\n revision.h             |  5 +++--\n upload-pack.c          | 10 +++++-----\n 7 files changed, 22 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex d22180435c..064df74715 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -89,7 +89,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+static struct strvec hidden_refs = STRVEC_INIT;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -2618,7 +2618,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n-\tstring_list_clear(&hidden_refs, 0);\n+\tstrvec_clear(&hidden_refs);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex c7ad39611a..d3d7e13e5a 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -71,7 +71,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -154,7 +154,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n-\tstring_list_init_dup(&data.hidden_refs);\n+\tstrvec_init(&data.hidden_refs);\n \n \tgit_config(ls_refs_config, &data);\n \n@@ -196,7 +196,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-\tstring_list_clear(&data.hidden_refs, 0);\n+\tstrvec_clear(&data.hidden_refs);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 538bde644e..ec4d5b9101 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n }\n \n int parse_hide_refs_config(const char *var, const char *value, const char *section,\n-\t\t\t   struct string_list *hide_refs)\n+\t\t\t   struct strvec *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n-\t\tref = xstrdup(value);\n+\n+\t\t/* drop const to remove trailing '/' characters */\n+\t\tref = (char *)strvec_push(hide_refs, value);\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n int ref_is_hidden(const char *refname, const char *refname_full,\n-\t\t  const struct string_list *hide_refs)\n+\t\t  const struct strvec *hide_refs)\n {\n \tint i;\n \n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n-\t\tconst char *match = hide_refs->items[i].string;\n+\t\tconst char *match = hide_refs->v[i];\n \t\tconst char *subject;\n \t\tint neg = 0;\n \t\tconst char *p;\ndiff --git a/refs.h b/refs.h\nindex d672d636cf..a7751a1fc9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -810,7 +810,7 @@ int update_ref(const char *msg, const char *refname,\n \t       unsigned int flags, enum action_on_err onerr);\n \n int parse_hide_refs_config(const char *var, const char *value, const char *,\n-\t\t\t   struct string_list *);\n+\t\t\t   struct strvec *);\n \n /*\n  * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -820,7 +820,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,\n  * the ref is outside that namespace, the first parameter is NULL. The second\n  * parameter always points to the full ref name.\n  */\n-int ref_is_hidden(const char *, const char *, const struct string_list *);\n+int ref_is_hidden(const char *, const char *, const struct strvec *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/revision.c b/revision.c\nindex 89953592f9..7c9367a266 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n-\tstring_list_clear(&exclusions->hidden_refs, 0);\n+\tstrvec_clear(&exclusions->hidden_refs);\n \texclusions->hidden_refs_configured = 0;\n }\n \ndiff --git a/revision.h b/revision.h\nindex e8f6de9684..30b5b5919d 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -9,6 +9,7 @@\n #include \"commit-slab-decl.h\"\n #include \"ident.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"strvec.h\"\n \n /**\n  * The revision walking API offers functions to build a list of revisions\n@@ -94,7 +95,7 @@ struct ref_exclusions {\n \t * Hidden refs is a list of patterns that is to be hidden via\n \t * `ref_is_hidden()`.\n \t */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \t/*\n \t * Indicates whether hidden refs have been configured. This is to\n@@ -109,7 +110,7 @@ struct ref_exclusions {\n  */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n-\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRVEC_INIT, \\\n }\n \n struct oidset;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 08633dc121..d77d58bdde 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -69,7 +69,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -126,7 +126,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n-\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+\tstruct strvec hidden_refs = STRVEC_INIT;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -161,7 +161,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n-\tstring_list_clear(&data->hidden_refs, 0);\n+\tstrvec_clear(&data->hidden_refs);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -1169,7 +1169,7 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n+\t\t\tconst struct object_id *oid, const struct strvec *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n@@ -1453,7 +1453,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t  struct string_list *wanted_refs,\n-\t\t\t  struct string_list *hidden_refs,\n+\t\t\t  struct strvec *hidden_refs,\n \t\t\t  struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476831","messageId":"c65b3dea815cee24785a5ec8cf568587a94299ee.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 13/15] refs.h: let `for_each_namespaced_ref()` take excluded patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:23Z","receivedAt":"2023-05-08T22:00:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The following commit will want to call `for_each_namespaced_ref()` with\na list of excluded patterns.\n\nWe could introduce a variant of that function, say,\n`for_each_namespaced_ref_exclude()` which takes the extra parameter, and\nreimplement the original function in terms of that. But all but one\ncaller (in `http-backend.c`) will supply the new parameter, so add the\nnew parameter to `for_each_namespaced_ref()` itself instead of\nintroducing a new function.\n\nFor now, supply NULL for the list of excluded patterns at all callers to\navoid changing behavior, which we will do in the subsequent commit.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n http-backend.c | 2 +-\n refs.c         | 5 +++--\n refs.h         | 3 ++-\n upload-pack.c  | 6 +++---\n 4 files changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex ac146d85c5..ad500683c8 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)\n \n \t} else {\n \t\tselect_getanyfile(hdr);\n-\t\tfor_each_namespaced_ref(show_text_ref, &buf);\n+\t\tfor_each_namespaced_ref(NULL, show_text_ref, &buf);\n \t\tsend_strbuf(hdr, \"text/plain\", &buf);\n \t}\n \tstrbuf_release(&buf);\ndiff --git a/refs.c b/refs.c\nindex ec4d5b9101..95a7db9563 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat\n \t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, exclude_patterns, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\ndiff --git a/refs.h b/refs.h\nindex a7751a1fc9..f23626beca 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -372,7 +372,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \t\t\t const char *prefix, void *cb_data);\n \n int head_ref_namespaced(each_ref_fn fn, void *cb_data);\n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data);\n \n /* can be used to learn about broken ref and symref */\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d77d58bdde..7c646ea5bd 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -854,7 +854,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(check_ref, data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1378,7 +1378,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(send_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n \t\t/*\n \t\t * fflush stdout before calling advertise_shallow_grafts because send_ref\n \t\t * uses stdio.\n@@ -1388,7 +1388,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(check_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476832","messageId":"44bbf85e73676b2c89a82c09f7d355122ce6e805.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 14/15] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:26Z","receivedAt":"2023-05-08T22:01:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as a previous commit, teach `upload-pack` to avoid\nenumerating hidden references where possible.\n\nNote, however, that there are certain cases where cannot avoid\nenumerating even hidden references, in particular when either of:\n\n  - `uploadpack.allowTipSHA1InWant`, or\n  - `uploadpack.allowReachableSHA1InWant`\n\nare set, corresponding to `ALLOW_TIP_SHA1` and `ALLOW_REACHABLE_SHA1`,\nrespectively.\n\nWhen either of these bits are set, upload-pack's `is_our_ref()` function\nneeds to consider the `HIDDEN_REF` bit of the referent's object flags.\nSo we must visit all references, including the hidden ones, in order to\nmark their referents with the `HIDDEN_REF` bit.\n\nWhen neither `ALLOW_TIP_SHA1` nor `ALLOW_REACHABLE_SHA1` are set, the\n`is_our_ref()` function considers only the `OUR_REF` bit, and not the\n`HIDDEN_REF` one. `OUR_REF` is applied via `mark_our_ref()`, and only\nto objects at the tips of non-hidden references, so we do not need to\nvisit hidden references in this case.\n\nWhen neither of those bits are set, `upload-pack` can potentially avoid\nenumerating a large number of references. In the same example as a\nprevious commit (linux.git with one hidden reference per commit,\n\"refs/pull/N\"):\n\n    $ printf 0000 >in\n    $ hyperfine --warmup=1 \\\n      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]\n      Range (min … max):   405.7 ms … 409.2 ms    10 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in\n      Time (mean ± σ):     406.5 ms ±   1.3 ms    [User: 356.5 ms, System: 49.9 ms]\n      Range (min … max):   404.6 ms … 408.8 ms    10 runs\n\n    Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):       4.7 ms ±   0.2 ms    [User: 0.7 ms, System: 3.9 ms]\n      Range (min … max):     4.3 ms …   6.1 ms    472 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran\n       86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in'\n       86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'\n\nAs above, we must visit every reference when\nuploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit\nfar fewer references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n upload-pack.c | 33 +++++++++++++++++++++++++++------\n 1 file changed, 27 insertions(+), 6 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7c646ea5bd..0162fffce0 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,\n \t}\n }\n \n+static int allow_hidden_refs(enum allow_uor allow_uor)\n+{\n+\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n+}\n+\n+static void for_each_namespaced_ref_1(each_ref_fn fn,\n+\t\t\t\t      struct upload_pack_data *data)\n+{\n+\t/*\n+\t * If `data->allow_uor` allows updating hidden refs, we need to\n+\t * mark all references (including hidden ones), to check in\n+\t * `is_our_ref()` below.\n+\t *\n+\t * Otherwise, we only care about whether each reference's object\n+\t * has the OUR_REF bit set or not, so do not need to visit\n+\t * hidden references.\n+\t */\n+\tif (allow_hidden_refs(data->allow_uor))\n+\t\tfor_each_namespaced_ref(NULL, fn, data);\n+\telse\n+\t\tfor_each_namespaced_ref(data->hidden_refs.v, fn, data);\n+}\n+\n static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n {\n-\tint allow_hidden_ref = (allow_uor &\n-\t\t\t\t(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n-\treturn o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);\n+\treturn o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);\n }\n \n /*\n@@ -854,7 +875,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n+\t\tfor_each_namespaced_ref_1(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1378,7 +1399,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n+\t\tfor_each_namespaced_ref_1(send_ref, &data);\n \t\t/*\n \t\t * fflush stdout before calling advertise_shallow_grafts because send_ref\n \t\t * uses stdio.\n@@ -1388,7 +1409,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n+\t\tfor_each_namespaced_ref_1(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.40.1.477.g956c797dfc\n\n"},{"id":"476833","messageId":"7d3383083db80cab6ee022508c4ec0dbec698b26.1683581621.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH 15/15] builtin/receive-pack.c: avoid enumerating hidden references","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:00:30Z","receivedAt":"2023-05-08T22:01:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that `refs_for_each_fullref_in()` has the ability to avoid\nenumerating references matching certain pattern(s), use that to avoid\nvisiting hidden refs when constructing the ref advertisement via\nreceive-pack.\n\nNote that since this exclusion is best-effort, we still need\n`show_ref_cb()` to check whether or not each reference is hidden or not\nbefore including it in the advertisement.\n\nAs was the case when applying this same optimization to `upload-pack`,\n`receive-pack`'s reference advertisement phase can proceed much quicker\nby avoiding enumerating references that will not be part of the\nadvertisement.\n\n(Below, we're still using linux.git with one hidden refs/pull/N ref per\ncommit):\n\n    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]\n      Range (min … max):    87.7 ms …  95.5 ms    31 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]\n      Range (min … max):     4.1 ms …   5.6 ms    508 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran\n       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 064df74715..b954bcf802 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -336,7 +336,8 @@ static void write_head_info(void)\n {\n \tstatic struct oidset seen = OIDSET_INIT;\n \n-\tfor_each_ref(show_ref_cb, &seen);\n+\trefs_for_each_fullref_in(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t hidden_refs.v, show_ref_cb, &seen);\n \tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n \toidset_clear(&seen);\n \tif (!sent_capabilities)\n-- \n2.40.1.477.g956c797dfc\n"},{"id":"476839","messageId":"xmqqr0rq4ful.fsf@gitster.g","threadId":"59715","inReplyTo":"c804ba3620476713bd0535a315876378149ad7dd.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 04/15] ref-filter: add ref_filter_clear()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-05-08T22:29:22Z","receivedAt":"2023-05-08T22:29:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> From: Jeff King <peff@peff.net>\n>\n> We did not bother to clean up at all in branch/tag, and for-each-ref\n> only hit a few elements. So this is probably cleaning up leaks, but I\n> didn't check yet.\n>\n> Note that the reachable_from and unreachable_from lists should be\n> cleaned as they are used. So this is just covering any case where we\n> might bail before running the reachability check.\n> ---\n\nNot signed-off?\n\n> +void ref_filter_clear(struct ref_filter *filter)\n> +{\n> +\toid_array_clear(&filter->points_at);\n> +\tfree_commit_list(filter->with_commit);\n> +\tfree_commit_list(filter->no_commit);\n> +\tfree_commit_list(filter->reachable_from);\n> +\tfree_commit_list(filter->unreachable_from);\n> +\tref_filter_init(filter);\n> +}\n\nAnd the previous step matters here---otherwise we will end up\nwalking two commit lists whose elements have all been popped\nin reach_filter().\n\nMakes sense.\n\n> +void ref_filter_init(struct ref_filter *filter)\n> +{\n> +\tstruct ref_filter blank = REF_FILTER_INIT;\n> +\tmemcpy(filter, &blank, sizeof(blank));\n> +}\n\n\nI wonder if structure assignment \"*filter = blank\" is easier to see\nbut I think we've seen this \"_INIT; memcpy()\" dance before.\n\n\n"},{"id":"476841","messageId":"ZFl4z/WD2j49a8bk@nand.local","threadId":"59715","inReplyTo":"xmqqr0rq4ful.fsf@gitster.g","subject":"Re: [PATCH 04/15] ref-filter: add ref_filter_clear()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-08T22:33:51Z","receivedAt":"2023-05-08T22:33:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 08, 2023 at 03:29:22PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > From: Jeff King <peff@peff.net>\n> >\n> > We did not bother to clean up at all in branch/tag, and for-each-ref\n> > only hit a few elements. So this is probably cleaning up leaks, but I\n> > didn't check yet.\n> >\n> > Note that the reachable_from and unreachable_from lists should be\n> > cleaned as they are used. So this is just covering any case where we\n> > might bail before running the reachability check.\n> > ---\n>\n> Not signed-off?\n\nOops. Sorry about that, this should be:\n\n    Signed-off-by: Jeff King <peff@peff.net>\n    Signed-off-by: Taylor Blau <me@ttaylorr.com>\n\n> I wonder if structure assignment \"*filter = blank\" is easier to see\n> but I think we've seen this \"_INIT; memcpy()\" dance before.\n\nYeah, this matches how many other _init() functions work (just looking\nthrough the output of `git grep -B3 memcpy | grep -A3 _init`).\n\nThanks,\nTaylor\n"},{"id":"476842","messageId":"xmqqmt2e4fia.fsf@gitster.g","threadId":"59715","inReplyTo":"c54000f5f5122c4ca3ac9b16828a8fd77050768c.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 05/15] ref-filter.c: parameterize match functions over patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-05-08T22:36:45Z","receivedAt":"2023-05-08T22:36:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> -static int match_pattern(const struct ref_filter *filter, const char *refname)\n> +static int match_pattern(const struct ref_filter *filter,\n> +\t\t\t const char **patterns,\n> +\t\t\t const char *refname)\n>  {\n> -\tconst char **patterns = filter->name_patterns;\n>  \tunsigned flags = 0;\n>  \n>  \tif (filter->ignore_case)\n> @@ -2132,9 +2133,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n>   * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n>   * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n>   */\n> -static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n> +static int match_name_as_path(const struct ref_filter *filter,\n> +\t\t\t      const char **pattern,\n> +\t\t\t      const char *refname)\n>  {\n> -\tconst char **pattern = filter->name_patterns;\n>  \tint namelen = strlen(refname);\n>  \tunsigned flags = WM_PATHNAME;\n>  \n\nThese hint that we'd eventually lose .name_patterns member from the\nstructure so that we can pass pattern array that is not necessarily\ntied to any instance of a filter?\n\n> @@ -2163,8 +2165,8 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n>  \tif (!*filter->name_patterns)\n>  \t\treturn 1; /* No pattern always matches */\n>  \tif (filter->match_as_path)\n> -\t\treturn match_name_as_path(filter, refname);\n> -\treturn match_pattern(filter, refname);\n> +\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n> +\treturn match_pattern(filter, filter->name_patterns, refname);\n\nAnd we are not there yet, so we hoist the use of .name_patterns\nmember one level up to the only caller?\n\nWithout seeing how it evolves, we can tell this does not make\nanything break, but we cannot tell how this helps anything (yet).\n\nLet's read on.\n\n"},{"id":"476844","messageId":"xmqqcz3a4den.fsf@gitster.g","threadId":"59715","inReplyTo":"ea5c0ddc10cd484c2af8505b8078697896ff3bad.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 06/15] builtin/for-each-ref.c: add `--exclude` option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-05-08T23:22:08Z","receivedAt":"2023-05-08T23:22:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\n> index c01fa6fefe..449da61e11 100644\n> --- a/builtin/for-each-ref.c\n> +++ b/builtin/for-each-ref.c\n> @@ -14,6 +14,7 @@ static char const * const for_each_ref_usage[] = {\n>  \tN_(\"git for-each-ref [--points-at <object>]\"),\n>  \tN_(\"git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]\"),\n>  \tN_(\"git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]\"),\n> +\tN_(\"git for-each-ref [--exclude=<pattern> ...]\"),\n>  \tNULL\n>  };\n\nI think the original is already wrong, but the easiest thing we can\ndo in order to avoid making it worse is to drop this hunk, as the\nexisting usage is this:\n\nstatic char const * const for_each_ref_usage[] = {\n\tN_(\"git for-each-ref [<options>] [<pattern>]\"),\n\tN_(\"git for-each-ref [--points-at <object>]\"),\n\tN_(\"git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]\"),\n\tN_(\"git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]\"),\n\tNULL\n};\n\nand this series merely adds a new \"--exclude=<pattern>\" as one of\nthe \"<options>\". \n\nAs we can see from the fact that for example\n\n $ git for-each-ref --no-merged next refs/heads/\\?\\?/\\*\n\nworks just fine, exactly the same thing can be said about the other\n--points-at/--merged/--no-merged/--contains/--no-contains options.\n\nThe SYNOPSIS section of the manual page is fine.\n\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 6c5eed144f..93dc9b331f 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2169,6 +2169,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n>  \treturn match_pattern(filter, filter->name_patterns, refname);\n>  }\n>  \n> +static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n> +{\n> +\tif (!filter->exclude.nr)\n> +\t\treturn 0;\n> +\tif (filter->match_as_path)\n> +\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n> +\treturn match_pattern(filter, filter->exclude.v, refname);\n> +}\n\nEarlier I made a comment about .name_patterns member becoming\nunnecessary, but I think what should need to happen is instead\nmatch_pattern() and match_name_as_path() to lose the \"filter\"\nparameter, and take a boolean \"ignore_case\" instead.\n\n>  struct ref_filter {\n>  \tconst char **name_patterns;\n> +\tstruct strvec exclude;\n\nAt some point after the dust settles, we may want to tweak\nname_patterns so that these two appear to complement each other more\nclearly, e.g. use \"struct strvec include\" to replace \"name_patterns\"\nor something.  But that is an unrelated tangent.\n\nIn any case, there wasn't anything surprising or unexpected in the\ncode.  Looking good.\n\n> diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\n> index 5c00607608..7e8d578522 100755\n> --- a/t/t6300-for-each-ref.sh\n> +++ b/t/t6300-for-each-ref.sh\n> @@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '\n>  \ttest_cmp expected actual\n>  '\n>  \n> +cat >expected <<\\EOF\n> +refs/tags/bar\n> +refs/tags/baz\n> +refs/tags/testtag\n> +EOF\n> +\n> +test_expect_success 'exercise patterns with prefix exclusions' '\n> +\tfor tag in foo/one foo/two foo/three bar baz\n> +\tdo\n> +\t\tgit tag \"$tag\" || return 1\n> +\tdone &&\n> +\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n> +\tgit for-each-ref --format=\"%(refname)\" \\\n> +\t\trefs/tags/ --exclude=refs/tags/foo >actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +cat >expected <<\\EOF\n> +refs/tags/bar\n> +refs/tags/baz\n> +refs/tags/foo/one\n> +refs/tags/testtag\n> +EOF\n> +\n> +test_expect_success 'exercise patterns with pattern exclusions' '\n> +\tfor tag in foo/one foo/two foo/three bar baz\n> +\tdo\n> +\t\tgit tag \"$tag\" || return 1\n> +\tdone &&\n> +\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n> +\tgit for-each-ref --format=\"%(refname)\" \\\n> +\t\trefs/tags/ --exclude=\"refs/tags/foo/t*\" >actual &&\n> +\ttest_cmp expected actual\n> +'\n\nThese are doing as Romans do, so I won't comment on the outdated\npattern of preparing the expectation outside the test script.  After\nthe dust settles, somebody needs to go in and clean it up.\n\nThanks.\n"},{"id":"476846","messageId":"xmqq5y924bt2.fsf@gitster.g","threadId":"59715","inReplyTo":"836a5665b7df065811edc678cb8e70004f7b7c49.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 08/15] refs/packed-backend.c: refactor `find_reference_location()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-05-08T23:56:41Z","receivedAt":"2023-05-08T23:56:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> The function `find_reference_location()` is used to perform a\n> binary search-like function over the contents of a repository's\n> `$GIT_DIR/packed-refs` file.\n>\n> The search it implements is unlike a standard binary search in that the\n> records it searches over are not of a fixed width, so the comparison\n> must locate the end of a record before comparing it.\n>\n> Extract the core routine of `find_reference_location()` in order to\n> implement a function in the following patch which will find the first\n> location in the `packed-refs` file that *doesn't* match the given\n> pattern.\n>\n> The behavior of `find_reference_location()` is unchanged.\n\nThe splitting out of this step is rather unfortunate in that the\nextra parameter \"start\" given to cmp_record_to_refname(), together\nwith a rather curious returning to \"-1\" when the parameter is false,\nare not justified at all by looking at the caller, because the only\ncaller of the function, find_reference_location_1() always passes\n\"start\" it got from its caller without modifying, and the sole\ncaller of that passes \"1\" to \"start\".  IOW, the behaviour of\ncmp_record_to_refname() when \"start\" is false can be any arbitrary\ngarbage (it could even be BUG(\"\")) and the claim \"the behaviour is\nunchanged\" will still be true.\n\nBut that does not help the readers all that much.\n\nSo, yes I can agree that this does not introduce any new bug, it is\na mysterious no-op, and why we want to pass different values in \"start\"\nin future steps in order to achieve what is not explained and leaves\nthe readers frustrated.\n\nI'll stop here for now and come back to the rest laster.\n\nThanks.\n\n"},{"id":"476850","messageId":"CAPx1Gvft0pXehRHpRXrinsTTSaQQkgOL8YaBgt54SHnfsFo2NQ@mail.gmail.com","threadId":"59715","inReplyTo":"a39d1107c1f3e9fbd859a23aed72e63bbd394fa2.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2023-05-09T00:10:52Z","receivedAt":"2023-05-09T00:11:08Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"I haven't read through any of this closely (don't have time for it now) but\nwant to mention one thing here:\n\nOn Mon, May 8, 2023 at 3:06 PM Taylor Blau <me@ttaylorr.com> wrote:\n>   - Construct a skip list of regions by combining adjacent and\n>     overlapping regions from the previous step.\n\nYou might want to add a note to the code that there is no\nrelationship here to the skip list data structure (see\nhttps://en.wikipedia.org/wiki/Skip_list).\n\nChris\n"},{"id":"476883","messageId":"ZFpjaUaDl7eUCh8N@ncase","threadId":"59715","inReplyTo":"c804ba3620476713bd0535a315876378149ad7dd.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 04/15] ref-filter: add ref_filter_clear()","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-05-09T15:14:49Z","receivedAt":"2023-05-09T15:15:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 08, 2023 at 05:59:52PM -0400, Taylor Blau wrote:\n> From: Jeff King <peff@peff.net>\n> \n> We did not bother to clean up at all in branch/tag, and for-each-ref\n> only hit a few elements. So this is probably cleaning up leaks, but I\n> didn't check yet.\n\nNit: it sounds like there still is the intent to check whether this does\nfix leaks. How about updating the commit message to either not mention\nthe intent or perform the check?\n\nPatrick\n"},{"id":"476884","messageId":"ZFpjcoylGSj7Palo@ncase","threadId":"59715","inReplyTo":"f437cd83e2d2177b1526673821b950f47814abc6.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 07/15] refs: plumb `exclude_patterns` argument throughout","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-05-09T15:14:58Z","receivedAt":"2023-05-09T15:15:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 08, 2023 at 06:00:01PM -0400, Taylor Blau wrote:\n> The subsequent patch will want to access an optional `excluded_patterns`\n> array within refs/packed-backend.c. To do so, the refs subsystem needs\n> to be updated to pass this value across a number of different locations.\n> \n> Prepare for a future patch by introducing this plumbing now, passing\n> NULLs at top-level APIs in order to make that patch less noisy and more\n> easily readable.\n\nIt might be worth mentioning in the commit message that the exclude\npatterns are supposed to be best-effort. In other words, any caller\nwould still need to do manual filtering if I understand correctly. And\nwhile this is indeed documented via `refs_for_each_fullref_in_prefixes`\nit is quite easy to miss this important little detail.\n\nPatrick\n"},{"id":"476885","messageId":"ZFpjn-vjP6H_v_KD@ncase","threadId":"59715","inReplyTo":"a39d1107c1f3e9fbd859a23aed72e63bbd394fa2.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-05-09T15:15:43Z","receivedAt":"2023-05-09T15:15:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 08, 2023 at 06:00:08PM -0400, Taylor Blau wrote:\n> When iterating through the `packed-refs` file in order to answer a query\n> like:\n> \n>     $ git for-each-ref --exclude=refs/__hidden__\n> \n> it would be useful to avoid walking over all of the entries in\n> `refs/__hidden__/*` when possible, since we know that the ref-filter\n> code is going to throw them away anyways.\n> \n> In certain circumstances, doing so is possible. The algorithm for doing\n> so is as follows:\n> \n>   - For each excluded pattern, find the first record that matches it,\n>     and the first pattern that *doesn't* match it (i.e. the location\n>     you'd next want to consider when excluding that pattern).\n> \n>   - Sort the patterns by their starting location within the\n>     `packed-refs` file.\n> \n>   - Construct a skip list of regions by combining adjacent and\n>     overlapping regions from the previous step.\n> \n>   - When iterating through the `packed-refs` file, if `iter->pos` is\n>     ever contained in one of the regions from the previous steps,\n>     advance `iter->pos` past the end of that region, and continue\n>     enumeration.\n> \n> Note that this optimization is only possible when none of the excluded\n> pattern(s) have special meta-characters in them. To see why this is the\n> case, consider the exclusion pattern \"refs/foo[a]\". In general, in order\n> to find the location of the first record that matches this pattern, we\n> could only consider up to the first meta-character, \"refs/foo\". But this\n> doesn't work, since the excluded region we'd come up with would include\n> \"refs/foobar\", even though it is not excluded.\n\nIs this generally true though? A naive implementation would iterate\nthrough all references and find the first reference that matches the\nexclusion regular exepression. From thereon we continue to iterate until\nwe find the first entry that doesn't match. This may cause us to end up\nwith a suboptimal skip list, but the skip list would still be valid.\n\nAs I said, this implementation would be naive as we're now forced to\niterate through all references starting at the beginning. I assume that\nyour implementation will instead use a binary search to locate the first\nentry that matches the exclusion pattern and the last pattern. But the\nway this paragraph is formulated makes it sound like this is a general\nfact, even though it is a fact that derives from the implementation.\n\nI of course don't propose to change the algorithm here, but instead to\nclarify where this restriction actually comes from and why the tradeoff\nmakes sense.\n\n[snip]\n> @@ -929,6 +972,108 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n>  \t.abort = packed_ref_iterator_abort\n>  };\n>  \n> +static int skip_list_entry_cmp(const void *va, const void *vb)\n> +{\n> +\tconst struct skip_list_entry *a = va;\n> +\tconst struct skip_list_entry *b = vb;\n> +\n> +\tif (a->start < b->start)\n> +\t\treturn -1;\n> +\tif (a->start > b->start)\n> +\t\treturn 1;\n> +\treturn 0;\n> +}\n> +\n> +static int has_glob_special(const char *str)\n> +{\n> +\tconst char *p;\n> +\tfor (p = str; *p; p++) {\n> +\t\tif (is_glob_special(*p))\n> +\t\t\treturn 1;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static const char *ptr_max(const char *x, const char *y)\n> +{\n> +\tif (x > y)\n> +\t\treturn x;\n> +\treturn y;\n> +}\n> +\n> +static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n> +\t\t\t\t\tstruct snapshot *snapshot,\n> +\t\t\t\t\tconst char **excluded_patterns)\n> +{\n> +\tsize_t i, j;\n> +\tconst char **pattern;\n> +\n> +\tif (!excluded_patterns)\n> +\t\treturn;\n> +\n> +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> +\t\tstruct skip_list_entry *e;\n> +\n> +\t\t/*\n> +\t\t * We can't feed any excludes with globs in them to the\n> +\t\t * refs machinery.  It only understands prefix matching.\n> +\t\t * We likewise can't even feed the string leading up to\n> +\t\t * the first meta-character, as something like \"foo[a]\"\n> +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> +\t\t * would match that and mark it for exclusion).\n> +\t\t */\n> +\t\tif (has_glob_special(*pattern))\n> +\t\t\tcontinue;\n> +\n> +\t\tALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);\n> +\n> +\t\te = &iter->skip[iter->skip_nr++];\n> +\t\te->start = find_reference_location(snapshot, *pattern, 0);\n> +\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n\nOne thing that makes this hard to reason about is that we don't\nexplicitly handle the case where the pattern doesn't match at all. So\nyou require a bunch of knowledge about what exactly the functions\n`find_reference_location()` and `find_reference_location_end()` do in\nthat case in order to determine whether we will end up doing the right\nthing.\n\nExplicitly handling this would give us some benefits:\n\n- It makes the code more obvious.\n\n- We can avoid an extra skip list entry for every non-matching\n  pattern.\n\n- We wouldn't have to perform binary search over the snapshot twice.\n\nMight be I'm missing something though.\n\n> +\t}\n> +\n> +\tif (!iter->skip_nr) {\n> +\t\t/*\n> +\t\t * Every entry in exclude_patterns has a meta-character,\n> +\t\t * nothing to do here.\n> +\t\t */\n> +\t\treturn;\n> +\t}\n> +\n> +\tQSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp);\n> +\n> +\t/*\n> +\t * As an optimization, merge adjacent entries in the skip list\n> +\t * to jump forwards as far as possible when entering a skipped\n> +\t * region.\n> +\t *\n> +\t * For example, if we have two skipped regions:\n> +\t *\n> +\t *\t[[A, B], [B, C]]\n> +\t *\n> +\t * we want to combine that into a single entry jumping from A to\n> +\t * C.\n> +\t */\n> +\tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n> +\t\tstruct skip_list_entry *ours = &iter->skip[i];\n> +\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n> +\n> +\t\tif (ours->start == ours->end) {\n> +\t\t\t/* ignore empty regions (no matching entries) */\n> +\t\t\tcontinue;\n> +\t\t} else if (prev->end >= ours->start) {\n> +\t\t\t/* overlapping regions extend the previous one */\n> +\t\t\tprev->end = ptr_max(prev->end, ours->end);\n> +\t\t} else {\n> +\t\t\t/* otherwise, insert a new region */\n> +\t\t\titer->skip[j++] = *ours;\n> +\t\t}\n> +\t}\n\nMh. Does this do the right thing in case we have multiple consecutive\noverlapping skip list entries? We always end up adjusting the immediate\npredecessor as we use `i - 1` to find `prev`. Shouldn't we instead start\nwith `j = 0` and assign `prev = &iter->skip[j]`?\n\n[snip]\n> diff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\n> new file mode 100755\n> index 0000000000..da5265a5a8\n> --- /dev/null\n> +++ b/t/t1419-exclude-refs.sh\n> @@ -0,0 +1,101 @@\n> +#!/bin/sh\n> +\n> +test_description='test exclude_patterns functionality in main ref store'\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +\n> +TEST_PASSES_SANITIZE_LEAK=true\n> +. ./test-lib.sh\n> +\n> +for_each_ref__exclude () {\n> +\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n> +\tcut -d ' ' -f 2 actual.raw\n> +}\n> +\n> +for_each_ref () {\n> +\tgit for-each-ref --format='%(refname)' \"$@\"\n> +}\n> +\n> +test_expect_success 'setup' '\n> +\ttest_commit --no-tag base &&\n> +\tbase=\"$(git rev-parse HEAD)\" &&\n> +\n> +\tfor name in foo bar baz quux\n> +\tdo\n> +\t\tfor i in 1 2 3\n> +\t\tdo\n> +\t\t\techo \"create refs/heads/$name/$i $base\" || return 1\n> +\t\tdone || return 1\n> +\tdone >in &&\n> +\techo \"delete refs/heads/main\" >>in &&\n> +\n> +\tgit update-ref --stdin <in &&\n> +\tgit pack-refs --all\n> +'\n> +\n> +test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '\n> +\t# region in middle\n> +\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n> +\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n\nNit: it might be a bit more readable if we put the comment into the test\ndescription instead of having an opaque description that mentions ref\nnames that don't have much of a meaning without reading the test itself.\n\nPatrick\n"},{"id":"476886","messageId":"ZFpjpsmCxDTUG-z3@ncase","threadId":"59715","inReplyTo":"44bbf85e73676b2c89a82c09f7d355122ce6e805.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 14/15] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-05-09T15:15:50Z","receivedAt":"2023-05-09T15:16:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 08, 2023 at 06:00:26PM -0400, Taylor Blau wrote:\n[snip]\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 7c646ea5bd..0162fffce0 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,\n>  \t}\n>  }\n>  \n> +static int allow_hidden_refs(enum allow_uor allow_uor)\n> +{\n> +\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n> +}\n> +\n> +static void for_each_namespaced_ref_1(each_ref_fn fn,\n> +\t\t\t\t      struct upload_pack_data *data)\n\nI know it's common practice in the Git project, but personally I tend to\nfight with functions that have a `_1` suffix. You simply cannot tell\nwhat the difference is to the non-suffixed variant without checking its\ndeclaration.\n\n`for_each_namespaced_ref_with_optional_hidden_refs()` is definitely a\nmouthful though, and I can't really think of something better either.\n\n> +{\n> +\t/*\n> +\t * If `data->allow_uor` allows updating hidden refs, we need to\n> +\t * mark all references (including hidden ones), to check in\n> +\t * `is_our_ref()` below.\n\nDoesn't this influence whether somebody can _fetch_ objects pointed to\nby the hidden refs instead of _updating_ them?\n\nPatrick\n"},{"id":"476910","messageId":"ZFqa9OYBr6l7UVg6@nand.local","threadId":"59715","inReplyTo":"ZFpjaUaDl7eUCh8N@ncase","subject":"Re: [PATCH 04/15] ref-filter: add ref_filter_clear()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T19:11:48Z","receivedAt":"2023-05-09T19:11:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 09, 2023 at 05:14:49PM +0200, Patrick Steinhardt wrote:\n> On Mon, May 08, 2023 at 05:59:52PM -0400, Taylor Blau wrote:\n> > From: Jeff King <peff@peff.net>\n> >\n> > We did not bother to clean up at all in branch/tag, and for-each-ref\n> > only hit a few elements. So this is probably cleaning up leaks, but I\n> > didn't check yet.\n>\n> Nit: it sounds like there still is the intent to check whether this does\n> fix leaks. How about updating the commit message to either not mention\n> the intent or perform the check?\n\nOops. Thanks for the reminder. Peff sent this patch to me while we were\nworking on this topic together, and I forgot to come back and actually\nperform this check.\n\nLuckily, it helps out quite a bit:\n\n   t/t0041-usage.sh                          |  1 +\n   t/t2016-checkout-patch.sh                 |  1 +\n   t/t3402-rebase-merge.sh                   |  1 +\n   t/t3407-rebase-abort.sh                   |  1 +\n   t/t4058-diff-duplicates.sh                |  2 ++\n   t/t5407-post-rewrite-hook.sh              |  1 +\n   t/t5811-proto-disable-git.sh              |  2 ++\n   t/t6001-rev-list-graft.sh                 |  1 +\n   t/t7008-filter-branch-null-sha1.sh        |  1 +\n   t/t7408-submodule-reference.sh            |  2 ++\n   t/t9502-gitweb-standalone-parse-output.sh |  1 +\n\n11 new leak-free tests! I'll take it ;-).\n\nThanks,\nTaylor\n"},{"id":"476919","messageId":"ZFqpZrwiKARsM53t@nand.local","threadId":"59715","inReplyTo":"xmqqmt2e4fia.fsf@gitster.g","subject":"Re: [PATCH 05/15] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T20:13:26Z","receivedAt":"2023-05-09T20:13:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 08, 2023 at 03:36:45PM -0700, Junio C Hamano wrote:\n> > @@ -2132,9 +2133,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n> >   * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n> >   * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n> >   */\n> > -static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n> > +static int match_name_as_path(const struct ref_filter *filter,\n> > +\t\t\t      const char **pattern,\n> > +\t\t\t      const char *refname)\n> >  {\n> > -\tconst char **pattern = filter->name_patterns;\n> >  \tint namelen = strlen(refname);\n> >  \tunsigned flags = WM_PATHNAME;\n> >\n>\n> These hint that we'd eventually lose .name_patterns member from the\n> structure so that we can pass pattern array that is not necessarily\n> tied to any instance of a filter?\n\nRight. We'll pass in the excluded patterns in the next commit.\n\n> And we are not there yet, so we hoist the use of .name_patterns\n> member one level up to the only caller?\n>\n> Without seeing how it evolves, we can tell this does not make\n> anything break, but we cannot tell how this helps anything (yet).\n\nHmm. I tried to allude to this in the patch message with the paragraph:\n\n    The subsequent patch will add a new array of patterns to match over (the\n    excluded patterns, via a new `git for-each-ref --exclude` option),\n    treating the return value of these functions differently depending on\n    which patterns are being used to match.\n\n    Tweak `match_pattern()` and `match_name_as_path()` to take an array of\n    patterns to prepare for passing either in.\n\n...but I'm happy to add detail or clarify it if you think that these\nparagraphs could use it.\n\nThanks,\nTaylor\n"},{"id":"476921","messageId":"ZFqrj/gdzkZBYF1V@nand.local","threadId":"59715","inReplyTo":"xmqqcz3a4den.fsf@gitster.g","subject":"Re: [PATCH 06/15] builtin/for-each-ref.c: add `--exclude` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T20:22:39Z","receivedAt":"2023-05-09T20:22:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 08, 2023 at 04:22:08PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\n> > index c01fa6fefe..449da61e11 100644\n> > --- a/builtin/for-each-ref.c\n> > +++ b/builtin/for-each-ref.c\n> > @@ -14,6 +14,7 @@ static char const * const for_each_ref_usage[] = {\n> >  \tN_(\"git for-each-ref [--points-at <object>]\"),\n> >  \tN_(\"git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]\"),\n> >  \tN_(\"git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]\"),\n> > +\tN_(\"git for-each-ref [--exclude=<pattern> ...]\"),\n> >  \tNULL\n> >  };\n>\n> I think the original is already wrong, but the easiest thing we can\n> do in order to avoid making it worse is to drop this hunk, as the\n> existing usage is this:\n>\n> static char const * const for_each_ref_usage[] = {\n> \tN_(\"git for-each-ref [<options>] [<pattern>]\"),\n> \tN_(\"git for-each-ref [--points-at <object>]\"),\n> \tN_(\"git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]\"),\n> \tN_(\"git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]\"),\n> \tNULL\n> };\n>\n> and this series merely adds a new \"--exclude=<pattern>\" as one of\n> the \"<options>\".\n>\n> As we can see from the fact that for example\n>\n>  $ git for-each-ref --no-merged next refs/heads/\\?\\?/\\*\n>\n> works just fine, exactly the same thing can be said about the other\n> --points-at/--merged/--no-merged/--contains/--no-contains options.\n>\n> The SYNOPSIS section of the manual page is fine.\n\nGood point, will tweak; thanks.\n\n> > @@ -2169,6 +2169,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n> >  \treturn match_pattern(filter, filter->name_patterns, refname);\n> >  }\n> >\n> > +static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n> > +{\n> > +\tif (!filter->exclude.nr)\n> > +\t\treturn 0;\n> > +\tif (filter->match_as_path)\n> > +\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n> > +\treturn match_pattern(filter, filter->exclude.v, refname);\n> > +}\n>\n> Earlier I made a comment about .name_patterns member becoming\n> unnecessary, but I think what should need to happen is instead\n> match_pattern() and match_name_as_path() to lose the \"filter\"\n> parameter, and take a boolean \"ignore_case\" instead.\n\nAgreed.\n\n> > +cat >expected <<\\EOF\n> > +refs/tags/bar\n> > +refs/tags/baz\n> > +refs/tags/foo/one\n> > +refs/tags/testtag\n> > +EOF\n> > +\n> > +test_expect_success 'exercise patterns with pattern exclusions' '\n> > +\tfor tag in foo/one foo/two foo/three bar baz\n> > +\tdo\n> > +\t\tgit tag \"$tag\" || return 1\n> > +\tdone &&\n> > +\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n> > +\tgit for-each-ref --format=\"%(refname)\" \\\n> > +\t\trefs/tags/ --exclude=\"refs/tags/foo/t*\" >actual &&\n> > +\ttest_cmp expected actual\n> > +'\n>\n> These are doing as Romans do, so I won't comment on the outdated\n> pattern of preparing the expectation outside the test script.  After\n> the dust settles, somebody needs to go in and clean it up.\n\nYeah, I figured that this series was already getting pretty long, but\nthat it would be expedient to propagate forward this pattern. But it\nshould be cleaned up. Let's tag it with #leftoverbits accordingly.\n\nThanks,\nTaylor\n"},{"id":"476922","messageId":"ZFqr0oxt4IZkbPaD@nand.local","threadId":"59715","inReplyTo":"ZFpjcoylGSj7Palo@ncase","subject":"Re: [PATCH 07/15] refs: plumb `exclude_patterns` argument throughout","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T20:23:46Z","receivedAt":"2023-05-09T20:23:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 09, 2023 at 05:14:58PM +0200, Patrick Steinhardt wrote:\n> On Mon, May 08, 2023 at 06:00:01PM -0400, Taylor Blau wrote:\n> > The subsequent patch will want to access an optional `excluded_patterns`\n> > array within refs/packed-backend.c. To do so, the refs subsystem needs\n> > to be updated to pass this value across a number of different locations.\n> >\n> > Prepare for a future patch by introducing this plumbing now, passing\n> > NULLs at top-level APIs in order to make that patch less noisy and more\n> > easily readable.\n>\n> It might be worth mentioning in the commit message that the exclude\n> patterns are supposed to be best-effort. In other words, any caller\n> would still need to do manual filtering if I understand correctly. And\n> while this is indeed documented via `refs_for_each_fullref_in_prefixes`\n> it is quite easy to miss this important little detail.\n\nGood suggestion, I updated this patch message accordingly.\n\nThanks,\nTaylor\n"},{"id":"476923","messageId":"ZFqtOPO5MMrG60S3@nand.local","threadId":"59715","inReplyTo":"xmqq5y924bt2.fsf@gitster.g","subject":"Re: [PATCH 08/15] refs/packed-backend.c: refactor `find_reference_location()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T20:29:44Z","receivedAt":"2023-05-09T20:29:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 08, 2023 at 04:56:41PM -0700, Junio C Hamano wrote:\n> So, yes I can agree that this does not introduce any new bug, it is\n> a mysterious no-op, and why we want to pass different values in \"start\"\n> in future steps in order to achieve what is not explained and leaves\n> the readers frustrated.\n\nRe-reading this patch again with your review in mind, I agree that the\nsplit is poorly placed.\n\nI modified this patch to just extract the core routine behind a helper\nfunction and avoided adding the \"start\" parameter here.\n\nThanks,\nTaylor\n"},{"id":"476925","messageId":"ZFqviJG0mnD+zaFt@nand.local","threadId":"59715","inReplyTo":"CAPx1Gvft0pXehRHpRXrinsTTSaQQkgOL8YaBgt54SHnfsFo2NQ@mail.gmail.com","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T20:39:36Z","receivedAt":"2023-05-09T20:39:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 08, 2023 at 05:10:52PM -0700, Chris Torek wrote:\n> On Mon, May 8, 2023 at 3:06 PM Taylor Blau <me@ttaylorr.com> wrote:\n> >   - Construct a skip list of regions by combining adjacent and\n> >     overlapping regions from the previous step.\n>\n> You might want to add a note to the code that there is no\n> relationship here to the skip list data structure (see\n> https://en.wikipedia.org/wiki/Skip_list).\n\nGood suggestion, thanks. I picked the name skip list for this concept\nsince we're skipping over excluded regions of the packed-refs file, but\nyou're right it has no relation to the skip list data structure.\n\nWill note in the patch message.\n\nThanks,\nTaylor\n"},{"id":"476927","messageId":"ZFqzW2vqcFJqqPXH@nand.local","threadId":"59715","inReplyTo":"ZFpjn-vjP6H_v_KD@ncase","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T20:55:55Z","receivedAt":"2023-05-09T20:57:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 09, 2023 at 05:15:43PM +0200, Patrick Steinhardt wrote:\n> On Mon, May 08, 2023 at 06:00:08PM -0400, Taylor Blau wrote:\n>\n> > Note that this optimization is only possible when none of the excluded\n> > pattern(s) have special meta-characters in them. To see why this is the\n> > case, consider the exclusion pattern \"refs/foo[a]\". In general, in order\n> > to find the location of the first record that matches this pattern, we\n> > could only consider up to the first meta-character, \"refs/foo\". But this\n> > doesn't work, since the excluded region we'd come up with would include\n> > \"refs/foobar\", even though it is not excluded.\n>\n> Is this generally true though? A naive implementation would iterate\n> through all references and find the first reference that matches the\n> exclusion regular exepression. From thereon we continue to iterate until\n> we find the first entry that doesn't match. This may cause us to end up\n> with a suboptimal skip list, but the skip list would still be valid.\n>\n> As I said, this implementation would be naive as we're now forced to\n> iterate through all references starting at the beginning. I assume that\n> your implementation will instead use a binary search to locate the first\n> entry that matches the exclusion pattern and the last pattern. But the\n> way this paragraph is formulated makes it sound like this is a general\n> fact, even though it is a fact that derives from the implementation.\n>\n> I of course don't propose to change the algorithm here, but instead to\n> clarify where this restriction actually comes from and why the tradeoff\n> makes sense.\n\nIn the example you include, it's possible. But consider something like:\n\n    $ git for-each-ref --exclude='refs/foo[ac]'\n\nThe region that matches that expression (\"refs/fooa\", \"refs/fooc\" and\neverything underneath them) does not have to appear as a continuous\nsingle region in the packed-refs file. If you have, say, \"refs/foobar\",\nthat will appear between the two regions you want to exclude.\n\nSo I think you *might* be able to do it in general, but at the very\nleast it would involve splitting each character class and finding the\nstart and end of any region(s) that it matches.\n\nEven so, you'd have to try and match each entry as you determine the\nwidth of the excluded region, at which point you're at par with\nenumerating them anyway and having the caller discard any entries it\ndoesn't want.\n\n> > +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> > +\t\tstruct skip_list_entry *e;\n> > +\n> > +\t\t/*\n> > +\t\t * We can't feed any excludes with globs in them to the\n> > +\t\t * refs machinery.  It only understands prefix matching.\n> > +\t\t * We likewise can't even feed the string leading up to\n> > +\t\t * the first meta-character, as something like \"foo[a]\"\n> > +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> > +\t\t * would match that and mark it for exclusion).\n> > +\t\t */\n> > +\t\tif (has_glob_special(*pattern))\n> > +\t\t\tcontinue;\n> > +\n> > +\t\tALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);\n> > +\n> > +\t\te = &iter->skip[iter->skip_nr++];\n> > +\t\te->start = find_reference_location(snapshot, *pattern, 0);\n> > +\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n>\n> One thing that makes this hard to reason about is that we don't\n> explicitly handle the case where the pattern doesn't match at all. So\n> you require a bunch of knowledge about what exactly the functions\n> `find_reference_location()` and `find_reference_location_end()` do in\n> that case in order to determine whether we will end up doing the right\n> thing.\n>\n> Explicitly handling this would give us some benefits:\n>\n> - It makes the code more obvious.\n>\n> - We can avoid an extra skip list entry for every non-matching\n>   pattern.\n>\n> - We wouldn't have to perform binary search over the snapshot twice.\n>\n> Might be I'm missing something though.\n\nWe handle it implicitly via the mustexist parameter that both\nfind_reference_location() and find_reference_location_end() take, which\nreturns the position that you would find a matching entry if one does\nnot already exist.\n\nYou're right that it would save you a second binary search, but that is\nlikely to be a vanishingly small cost compared to the actual traversal,\ndisk I/O, etc.\n\nI could imagine modifying the signature of both of those functions to be\nsomething like:\n\n    int find_reference_location(struct snapshot *, const char *pattern,\n                                int mustexist, char *out)\n\nWhich would propagate the result through `out`, and the return value\nwould be whether or not it found a matching entry. But that only really\nmakes sense when `mustexist` is set to 0, and it adds some verbosity\nthroughout.\n\nI think that it's totally possible to avoid the second search, but I'm\nnot sure that the cost of additional complexity and verbosity is worth\nthe benefit of avoiding one binary search (which will likely be\nresident, anyway).\n\nNote though that if e->start == e->end here we will discard the empty\nskip region below.\n\n> > +\tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n> > +\t\tstruct skip_list_entry *ours = &iter->skip[i];\n> > +\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n> > +\n> > +\t\tif (ours->start == ours->end) {\n> > +\t\t\t/* ignore empty regions (no matching entries) */\n> > +\t\t\tcontinue;\n> > +\t\t} else if (prev->end >= ours->start) {\n> > +\t\t\t/* overlapping regions extend the previous one */\n> > +\t\t\tprev->end = ptr_max(prev->end, ours->end);\n> > +\t\t} else {\n> > +\t\t\t/* otherwise, insert a new region */\n> > +\t\t\titer->skip[j++] = *ours;\n> > +\t\t}\n> > +\t}\n>\n> Mh. Does this do the right thing in case we have multiple consecutive\n> overlapping skip list entries? We always end up adjusting the immediate\n> predecessor as we use `i - 1` to find `prev`. Shouldn't we instead start\n> with `j = 0` and assign `prev = &iter->skip[j]`?\n\nGood catch. I think applying this on top should do the trick:\n\n--- 8< ---\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 137a4233f6..3b1337267a 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1054,9 +1054,9 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n \t * we want to combine that into a single entry jumping from A to\n \t * C.\n \t */\n-\tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n+\tfor (i = 0, j = 0; i < iter->skip_nr; i++) {\n \t\tstruct skip_list_entry *ours = &iter->skip[i];\n-\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n+\t\tstruct skip_list_entry *prev = &iter->skip[j];\n\n \t\tif (ours->start == ours->end) {\n \t\t\t/* ignore empty regions (no matching entries) */\n@@ -1066,7 +1066,7 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n \t\t\tprev->end = ptr_max(prev->end, ours->end);\n \t\t} else {\n \t\t\t/* otherwise, insert a new region */\n-\t\t\titer->skip[j++] = *ours;\n+\t\t\titer->skip[++j] = *ours;\n \t\t}\n \t}\n--- >8 ---\n\n> > +test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '\n> > +\t# region in middle\n> > +\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n> > +\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n> > +\n> > +\ttest_cmp expect actual\n> > +'\n>\n> Nit: it might be a bit more readable if we put the comment into the test\n> description instead of having an opaque description that mentions ref\n> names that don't have much of a meaning without reading the test itself.\n\nFair enough ;-).\n\nThanks,\nTaylor\n"},{"id":"476930","messageId":"ZFq31XquMwJrPEOK@nand.local","threadId":"59715","inReplyTo":"ZFqzW2vqcFJqqPXH@nand.local","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T21:15:01Z","receivedAt":"2023-05-09T21:15:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 09, 2023 at 04:55:55PM -0400, Taylor Blau wrote:\n> Good catch. I think applying this on top should do the trick:\n>\n> --- 8< ---\n> diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> index 137a4233f6..3b1337267a 100644\n> --- a/refs/packed-backend.c\n> +++ b/refs/packed-backend.c\n> @@ -1054,9 +1054,9 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n>  \t * we want to combine that into a single entry jumping from A to\n>  \t * C.\n>  \t */\n> -\tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n> +\tfor (i = 0, j = 0; i < iter->skip_nr; i++) {\n>  \t\tstruct skip_list_entry *ours = &iter->skip[i];\n> -\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n> +\t\tstruct skip_list_entry *prev = &iter->skip[j];\n>\n>  \t\tif (ours->start == ours->end) {\n>  \t\t\t/* ignore empty regions (no matching entries) */\n> @@ -1066,7 +1066,7 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n>  \t\t\tprev->end = ptr_max(prev->end, ours->end);\n>  \t\t} else {\n>  \t\t\t/* otherwise, insert a new region */\n> -\t\t\titer->skip[j++] = *ours;\n> +\t\t\titer->skip[++j] = *ours;\n>  \t\t}\n>  \t}\n> --- >8 ---\n\nOops, this is wrong. It should be something like this instead:\n\n--- 8< ---\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 137a4233f6..574f32d67f 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1007,6 +1007,7 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n {\n \tsize_t i, j;\n \tconst char **pattern;\n+\tstruct skip_list_entry *last_disjoint;\n\n \tif (!excluded_patterns)\n \t\treturn;\n@@ -1054,19 +1055,22 @@ static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n \t * we want to combine that into a single entry jumping from A to\n \t * C.\n \t */\n+\tlast_disjoint = iter->skip;\n+\n \tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n \t\tstruct skip_list_entry *ours = &iter->skip[i];\n-\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n\n \t\tif (ours->start == ours->end) {\n \t\t\t/* ignore empty regions (no matching entries) */\n \t\t\tcontinue;\n-\t\t} else if (prev->end >= ours->start) {\n+\t\t} else if (ours->start <= last_disjoint->end) {\n \t\t\t/* overlapping regions extend the previous one */\n-\t\t\tprev->end = ptr_max(prev->end, ours->end);\n+\t\t\tlast_disjoint->end = ptr_max(last_disjoint->end, ours->end);\n \t\t} else {\n \t\t\t/* otherwise, insert a new region */\n \t\t\titer->skip[j++] = *ours;\n+\t\t\tlast_disjoint = ours;\n+\n \t\t}\n \t}\n--- >8 ---\n\nThanks,\nTaylor\n"},{"id":"476932","messageId":"ZFq8aYppuhjlkChK@nand.local","threadId":"59715","inReplyTo":"ZFpjpsmCxDTUG-z3@ncase","subject":"Re: [PATCH 14/15] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-09T21:34:33Z","receivedAt":"2023-05-09T21:35:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 09, 2023 at 05:15:50PM +0200, Patrick Steinhardt wrote:\n> On Mon, May 08, 2023 at 06:00:26PM -0400, Taylor Blau wrote:\n> > @@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,\n> >  \t}\n> >  }\n> >\n> > +static int allow_hidden_refs(enum allow_uor allow_uor)\n> > +{\n> > +\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n> > +}\n> > +\n> > +static void for_each_namespaced_ref_1(each_ref_fn fn,\n> > +\t\t\t\t      struct upload_pack_data *data)\n>\n> I know it's common practice in the Git project, but personally I tend to\n> fight with functions that have a `_1` suffix. You simply cannot tell\n> what the difference is to the non-suffixed variant without checking its\n> declaration.\n>\n> `for_each_namespaced_ref_with_optional_hidden_refs()` is definitely a\n> mouthful though, and I can't really think of something better either.\n\nYeah, I know. It's not my favorite convention, either, but I also\ncouldn't come up with anything shorter ;-).\n\n> > +{\n> > +\t/*\n> > +\t * If `data->allow_uor` allows updating hidden refs, we need to\n> > +\t * mark all references (including hidden ones), to check in\n> > +\t * `is_our_ref()` below.\n>\n> Doesn't this influence whether somebody can _fetch_ objects pointed to\n> by the hidden refs instead of _updating_ them?\n\nOops, yes. Thanks for catching my obvious typo.\n\nThanks,\nTaylor\n"},{"id":"476936","messageId":"xmqqh6slt6nx.fsf@gitster.g","threadId":"59715","inReplyTo":"a39d1107c1f3e9fbd859a23aed72e63bbd394fa2.1683581621.git.me@ttaylorr.com","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-05-09T23:40:50Z","receivedAt":"2023-05-09T23:40:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> When iterating through the `packed-refs` file in order to answer a query\n> like:\n>\n>     $ git for-each-ref --exclude=refs/__hidden__\n>\n> it would be useful to avoid walking over all of the entries in\n> `refs/__hidden__/*` when possible, since we know that the ref-filter\n> code is going to throw them away anyways.\n>\n> In certain circumstances, doing so is possible. The algorithm for doing\n> so is as follows:\n>\n>   - For each excluded pattern, find the first record that matches it,\n>     and the first pattern that *doesn't* match it (i.e. the location\n>     you'd next want to consider when excluding that pattern).\n\nDo we find \"record\" and then \"pattern\", or is the latter a misspelt\n\"record\"?  I will assume it is the latter while reading the rest.\nIs the latter \"the record that does not match the pattern, whose\nrecord number is the smallest but yet larger than the first record\nthat matches the pattern?\"  That is we assume that the refs in the\npacked refs file are sorted and can be partitioned into three by\neach pattern: (1) refs before the first matching record---they do\nnot match the pattern. (2) refs at or after the first matching\nrecord that match.  (3) refs after (2) that do not match.\n\nI am not sure if that would work.  What if the pattern were refs/tags/[ad-z]*\nand there are packed tags refs/tags/{a,b,c,d,e}?  The pattern would partition\nthe refs into [a](matches), [b,c](does not match), [d,e](matches).\n\nPerhaps I am grossly misunderstanding what the above explanation\nsays.\n\n>   - Sort the patterns by their starting location within the\n>     `packed-refs` file.\n\nSo the idea is that the patterns are sorted by the first record they\nmatch, and after sorting these patterns, refs that are between the\nbeginning of the list of refs and the first record associated with\nthe first pattern will not match _any_ pattern in that list?\n\n>   - Construct a skip list of regions by combining adjacent and\n>     overlapping regions from the previous step.\n\n\"skip list of regions\" -> \"list of regions to skip\", I guess.\n\n>   - When iterating through the `packed-refs` file, if `iter->pos` is\n>     ever contained in one of the regions from the previous steps,\n>     advance `iter->pos` past the end of that region, and continue\n>     enumeration.\n>\n> Note that this optimization is only possible when none of the excluded\n> pattern(s) have special meta-characters in them.\n\nAh, so this is called \"patterns\" but it deals with literal patterns\nwithout globs?  Then I'll stop worrying about the refs/tags/[ad-z]\ncounter-example.\n\n> To see why this is the\n> case, consider the exclusion pattern \"refs/foo[a]\". In general, in order\n> to find the location of the first record that matches this pattern, we\n> could only consider up to the first meta-character, \"refs/foo\". But this\n> doesn't work, since the excluded region we'd come up with would include\n> \"refs/foobar\", even though it is not excluded.\n\nOK.\n\n> Using the skip list is fairly straightforward (see the changes to\n> `refs/packed-backend.c::next_record()`), but constructing the list is\n> not. To ensure that the construction is correct, add a new suite of\n> tests in t1419 covering various corner cases (overlapping regions,\n> partially overlapping regions, adjacent regions, etc.).\n\nSounds good.  Does this actually use the skip list data structure,\nor do they happen to share the same two words in their names, but\notherwise have nothing common with each other?  If the latter, we\nmay want to revise the explanation, data type names, and variable\nnames, to avoid confusion, as Chris pointed out earlier.\n\n> +static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n> +\t\t\t\t\tstruct snapshot *snapshot,\n> +\t\t\t\t\tconst char **excluded_patterns)\n> +{\n> +\tsize_t i, j;\n> +\tconst char **pattern;\n> +\n> +\tif (!excluded_patterns)\n> +\t\treturn;\n> +\n> +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> +\t\tstruct skip_list_entry *e;\n> +\n> +\t\t/*\n> +\t\t * We can't feed any excludes with globs in them to the\n> +\t\t * refs machinery.  It only understands prefix matching.\n> +\t\t * We likewise can't even feed the string leading up to\n> +\t\t * the first meta-character, as something like \"foo[a]\"\n> +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> +\t\t * would match that and mark it for exclusion).\n> +\t\t */\n> +\t\tif (has_glob_special(*pattern))\n> +\t\t\tcontinue;\n\nHmph.  I would have expected that a set of patterns with any one\npattern with glob would invalidate the whole skip optimization, but\nit is nice if we can salvage such a set and still optimize, if only\nfor literal patterns.  Interesting.\n\nAh, that is because we are dealing with ranges that cannot possibly\nmatch.  With the mention of \"first record that matches\" etc. in the\nearlier descriptoin, I somehow misled myself that we are dealing\nwith ranges that have interesting records.  So, a pattern with glob\ndoes not contribute any range to be skipped, but that is OK.\n\n> +\t\tALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);\n> +\n> +\t\te = &iter->skip[iter->skip_nr++];\n> +\t\te->start = find_reference_location(snapshot, *pattern, 0);\n> +\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n\nSo, iter->skip[] array has one range per pattern at most, but some\npatterns may not contribute any range to the list.\n\n> +\t}\n> +\n> +\tif (!iter->skip_nr) {\n> +\t\t/*\n> +\t\t * Every entry in exclude_patterns has a meta-character,\n> +\t\t * nothing to do here.\n> +\t\t */\n> +\t\treturn;\n> +\t}\n> +\n> +\tQSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp);\n> +\n> +\t/*\n> +\t * As an optimization, merge adjacent entries in the skip list\n> +\t * to jump forwards as far as possible when entering a skipped\n> +\t * region.\n> +\t *\n> +\t * For example, if we have two skipped regions:\n> +\t *\n> +\t *\t[[A, B], [B, C]]\n\nI am confused.  The first pattern may never match records in [A..B]\nrange, and the second pattern may never match records in [B..C]\nrange, but what does it mean to combine these two ranges?\n\n> +\t * we want to combine that into a single entry jumping from A to\n> +\t * C.\n> +\t */\n> +\tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n> +\t\tstruct skip_list_entry *ours = &iter->skip[i];\n> +\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n\n\n> +\t\tif (ours->start == ours->end) {\n> +\t\t\t/* ignore empty regions (no matching entries) */\n> +\t\t\tcontinue;\n> +\t\t} else if (prev->end >= ours->start) {\n> +\t\t\t/* overlapping regions extend the previous one */\n> +\t\t\tprev->end = ptr_max(prev->end, ours->end);\n> +\t\t} else {\n> +\t\t\t/* otherwise, insert a new region */\n> +\t\t\titer->skip[j++] = *ours;\n> +\t\t}\n\nNone of the {braces} seem needed, but OK.  \n\n> +\t}\n> +\n> +\titer->skip_nr = j;\n> +\titer->skip_pos = 0;\n> +}\n"},{"id":"476941","messageId":"ZFsBy6BacQNJ6LCo@nand.local","threadId":"59715","inReplyTo":"xmqqh6slt6nx.fsf@gitster.g","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-10T02:30:35Z","receivedAt":"2023-05-10T02:30:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 09, 2023 at 04:40:50PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > When iterating through the `packed-refs` file in order to answer a query\n> > like:\n> >\n> >     $ git for-each-ref --exclude=refs/__hidden__\n> >\n> > it would be useful to avoid walking over all of the entries in\n> > `refs/__hidden__/*` when possible, since we know that the ref-filter\n> > code is going to throw them away anyways.\n> >\n> > In certain circumstances, doing so is possible. The algorithm for doing\n> > so is as follows:\n> >\n> >   - For each excluded pattern, find the first record that matches it,\n> >     and the first pattern that *doesn't* match it (i.e. the location\n> >     you'd next want to consider when excluding that pattern).\n>\n> Do we find \"record\" and then \"pattern\", or is the latter a misspelt\n> \"record\"?  I will assume it is the latter while reading the rest.\n> Is the latter \"the record that does not match the pattern, whose\n> record number is the smallest but yet larger than the first record\n> that matches the pattern?\"  That is we assume that the refs in the\n> packed refs file are sorted and can be partitioned into three by\n> each pattern: (1) refs before the first matching record---they do\n> not match the pattern. (2) refs at or after the first matching\n> record that match.  (3) refs after (2) that do not match.\n>\n> I am not sure if that would work.  What if the pattern were refs/tags/[ad-z]*\n> and there are packed tags refs/tags/{a,b,c,d,e}?  The pattern would partition\n> the refs into [a](matches), [b,c](does not match), [d,e](matches).\n>\n> Perhaps I am grossly misunderstanding what the above explanation\n> says.\n\nSorry, you're right to assume the latter. This should read \"..., find\nthe record that matches it, and the first record that *doesn't* match.\"\n\nIn the example you give, we would ignore that whole pattern and\nenumerate everything (including entries that match 'refs/tags/[ad-z]*'),\nand the caller is expected to discard them.\n\n> >   - Sort the patterns by their starting location within the\n> >     `packed-refs` file.\n>\n> So the idea is that the patterns are sorted by the first record they\n> match, and after sorting these patterns, refs that are between the\n> beginning of the list of refs and the first record associated with\n> the first pattern will not match _any_ pattern in that list?\n\nMy patch could use some clarification here, since it is much easier to\ntreat the patterns as unsorted, and then sort the beginning of the range\nthat they match.\n\n> >   - Construct a skip list of regions by combining adjacent and\n> >     overlapping regions from the previous step.\n>\n> \"skip list of regions\" -> \"list of regions to skip\", I guess.\n\nThanks, will update.\n\n> > Using the skip list is fairly straightforward (see the changes to\n> > `refs/packed-backend.c::next_record()`), but constructing the list is\n> > not. To ensure that the construction is correct, add a new suite of\n> > tests in t1419 covering various corner cases (overlapping regions,\n> > partially overlapping regions, adjacent regions, etc.).\n>\n> Sounds good.  Does this actually use the skip list data structure,\n> or do they happen to share the same two words in their names, but\n> otherwise have nothing common with each other?  If the latter, we\n> may want to revise the explanation, data type names, and variable\n> names, to avoid confusion, as Chris pointed out earlier.\n\nThey have nothing to do with each other ;-). I made a note in this patch\nin the revised version I'll send in the next day or two to note the\ndistinction. But I'm fine with renaming the whole concept to \"jump list\"\nor something like that if you prefer.\n\n> > +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> > +\t\tstruct skip_list_entry *e;\n> > +\n> > +\t\t/*\n> > +\t\t * We can't feed any excludes with globs in them to the\n> > +\t\t * refs machinery.  It only understands prefix matching.\n> > +\t\t * We likewise can't even feed the string leading up to\n> > +\t\t * the first meta-character, as something like \"foo[a]\"\n> > +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> > +\t\t * would match that and mark it for exclusion).\n> > +\t\t */\n> > +\t\tif (has_glob_special(*pattern))\n> > +\t\t\tcontinue;\n>\n> Hmph.  I would have expected that a set of patterns with any one\n> pattern with glob would invalidate the whole skip optimization, but\n> it is nice if we can salvage such a set and still optimize, if only\n> for literal patterns.  Interesting.\n>\n> Ah, that is because we are dealing with ranges that cannot possibly\n> match.  With the mention of \"first record that matches\" etc. in the\n> earlier descriptoin, I somehow misled myself that we are dealing\n> with ranges that have interesting records.  So, a pattern with glob\n> does not contribute any range to be skipped, but that is OK.\n\nExactly right. This is all \"best effort\" anyway, since there are some\npatterns that we cannot construct a skip list entry for in the general\ncase. So it's fine if we enumerate some references that match one or\nmore of the excluded patterns, because the caller is expected to drop\nthose results themselves.\n\n> > +\t/*\n> > +\t * As an optimization, merge adjacent entries in the skip list\n> > +\t * to jump forwards as far as possible when entering a skipped\n> > +\t * region.\n> > +\t *\n> > +\t * For example, if we have two skipped regions:\n> > +\t *\n> > +\t *\t[[A, B], [B, C]]\n>\n> I am confused.  The first pattern may never match records in [A..B]\n> range, and the second pattern may never match records in [B..C]\n> range, but what does it mean to combine these two ranges?\n\nThe patterns would match all records in their respective regions, but\nsince they are excluded we want to jump over those references instead of\niterating (and then discarding them later).\n\nSo if we have a jump from A->B, and another from B->C, it would be fine\nto perform two jumps to get from A to C. But we can detect this case by\ncombining adjacent/overlapping regions.\n\n> > +\t\tif (ours->start == ours->end) {\n> > +\t\t\t/* ignore empty regions (no matching entries) */\n> > +\t\t\tcontinue;\n> > +\t\t} else if (prev->end >= ours->start) {\n> > +\t\t\t/* overlapping regions extend the previous one */\n> > +\t\t\tprev->end = ptr_max(prev->end, ours->end);\n> > +\t\t} else {\n> > +\t\t\t/* otherwise, insert a new region */\n> > +\t\t\titer->skip[j++] = *ours;\n> > +\t\t}\n>\n> None of the {braces} seem needed, but OK.\n\nYeah. I added the braces here intentionally to match the\nCodingGuidelines, which state that this case is an exception.\n\nThanks,\nTaylor\n"},{"id":"476951","messageId":"ZFtG4S4XBboztTap@ncase","threadId":"59715","inReplyTo":"ZFqzW2vqcFJqqPXH@nand.local","subject":"Re: [PATCH 09/15] refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-05-10T07:25:21Z","receivedAt":"2023-05-10T07:25:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 09, 2023 at 04:55:55PM -0400, Taylor Blau wrote:\n> On Tue, May 09, 2023 at 05:15:43PM +0200, Patrick Steinhardt wrote:\n> > On Mon, May 08, 2023 at 06:00:08PM -0400, Taylor Blau wrote:\n> >\n> > > Note that this optimization is only possible when none of the excluded\n> > > pattern(s) have special meta-characters in them. To see why this is the\n> > > case, consider the exclusion pattern \"refs/foo[a]\". In general, in order\n> > > to find the location of the first record that matches this pattern, we\n> > > could only consider up to the first meta-character, \"refs/foo\". But this\n> > > doesn't work, since the excluded region we'd come up with would include\n> > > \"refs/foobar\", even though it is not excluded.\n> >\n> > Is this generally true though? A naive implementation would iterate\n> > through all references and find the first reference that matches the\n> > exclusion regular exepression. From thereon we continue to iterate until\n> > we find the first entry that doesn't match. This may cause us to end up\n> > with a suboptimal skip list, but the skip list would still be valid.\n> >\n> > As I said, this implementation would be naive as we're now forced to\n> > iterate through all references starting at the beginning. I assume that\n> > your implementation will instead use a binary search to locate the first\n> > entry that matches the exclusion pattern and the last pattern. But the\n> > way this paragraph is formulated makes it sound like this is a general\n> > fact, even though it is a fact that derives from the implementation.\n> >\n> > I of course don't propose to change the algorithm here, but instead to\n> > clarify where this restriction actually comes from and why the tradeoff\n> > makes sense.\n> \n> In the example you include, it's possible. But consider something like:\n> \n>     $ git for-each-ref --exclude='refs/foo[ac]'\n> \n> The region that matches that expression (\"refs/fooa\", \"refs/fooc\" and\n> everything underneath them) does not have to appear as a continuous\n> single region in the packed-refs file. If you have, say, \"refs/foobar\",\n> that will appear between the two regions you want to exclude.\n> \n> So I think you *might* be able to do it in general, but at the very\n> least it would involve splitting each character class and finding the\n> start and end of any region(s) that it matches.\n> \n> Even so, you'd have to try and match each entry as you determine the\n> width of the excluded region, at which point you're at par with\n> enumerating them anyway and having the caller discard any entries it\n> doesn't want.\n\nAlternatively you could also do this on a best-effort basis and only\nfind the first matching region. But anyway, as said: I'm fine with the\nlimitations but think that we should document better where they come\nfrom. The current commit message sounds like the limitation is of\ngeneral nature even though it is in fact a conciously-chosen tradeoff\nthat allows us to make the implementation more efficient for most cases.\n\nPatrick\n"},{"id":"477293","messageId":"cover.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH v2 00/16] refs: implement jump lists for packed backend","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:07Z","receivedAt":"2023-05-15T19:23:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a reroll of my series to implement jump (née skip) lists for the\npacked refs backend.\n\nNot a ton has changed since last time, but some notable things that have\nchanged include:\n\n  - Renaming \"skip lists\" to \"jump lists\" to clarify that this\n    implementation does not use the skip list data structure.\n  - Patch reorganization, splitting out `find_reference_location_end()`\n    more sensibly, rewording patch messages, etc.\n  - Addresses feedback from Junio and Patrick Steinhardt's helpful\n    reviews.\n\nAs usual, a range-diff is included below for convenience.\n\nGiven that we are expecting -rc0 today, we should aim to not let review\nof this topic direct our attention away from testing the release\ncandidates. We can get more serious about it on the other side of 2.41.\n\nThanks in advance for another look.\n\nJeff King (5):\n  refs.c: rename `ref_filter`\n  ref-filter.h: provide `REF_FILTER_INIT`\n  ref-filter: clear reachable list pointers after freeing\n  ref-filter: add `ref_filter_clear()`\n  ref-filter.c: parameterize match functions over patterns\n\nTaylor Blau (11):\n  builtin/for-each-ref.c: add `--exclude` option\n  refs: plumb `exclude_patterns` argument throughout\n  refs/packed-backend.c: refactor `find_reference_location()`\n  refs/packed-backend.c: implement jump lists to avoid excluded\n    pattern(s)\n  refs/packed-backend.c: add trace2 counters for jump list\n  revision.h: store hidden refs in a `strvec`\n  refs/packed-backend.c: ignore complicated hidden refs rules\n  refs.h: let `for_each_namespaced_ref()` take excluded patterns\n  builtin/receive-pack.c: avoid enumerating hidden references\n  upload-pack.c: avoid enumerating hidden refs where possible\n  ls-refs.c: avoid enumerating hidden refs where possible\n\n Documentation/git-for-each-ref.txt |   6 +\n builtin/branch.c                   |   4 +-\n builtin/for-each-ref.c             |   7 +-\n builtin/receive-pack.c             |   7 +-\n builtin/tag.c                      |   4 +-\n http-backend.c                     |   2 +-\n ls-refs.c                          |   8 +-\n ref-filter.c                       |  63 ++++++--\n ref-filter.h                       |  12 ++\n refs.c                             |  61 ++++----\n refs.h                             |  15 +-\n refs/debug.c                       |   5 +-\n refs/files-backend.c               |   5 +-\n refs/packed-backend.c              | 226 ++++++++++++++++++++++++++---\n refs/refs-internal.h               |   7 +-\n revision.c                         |   4 +-\n revision.h                         |   5 +-\n t/helper/test-reach.c              |   2 +-\n t/helper/test-ref-store.c          |  10 ++\n t/t0041-usage.sh                   |   1 +\n t/t1419-exclude-refs.sh            | 131 +++++++++++++++++\n t/t3402-rebase-merge.sh            |   1 +\n t/t6300-for-each-ref.sh            |  35 +++++\n trace2.h                           |   2 +\n trace2/tr2_ctr.c                   |   5 +\n upload-pack.c                      |  43 ++++--\n 26 files changed, 565 insertions(+), 106 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\nRange-diff against v1:\n 1:  2225f79941 =  1:  6cac42e70e refs.c: rename `ref_filter`\n 2:  ea1c7834db =  2:  8dda7db738 ref-filter.h: provide `REF_FILTER_INIT`\n 3:  7fe8623f60 =  3:  bf21df783d ref-filter: clear reachable list pointers after freeing\n 4:  c804ba3620 !  4:  85ecb70957 ref-filter: add ref_filter_clear()\n    @@ Metadata\n     Author: Jeff King <peff@peff.net>\n     \n      ## Commit message ##\n    -    ref-filter: add ref_filter_clear()\n    +    ref-filter: add `ref_filter_clear()`\n     \n    -    We did not bother to clean up at all in branch/tag, and for-each-ref\n    -    only hit a few elements. So this is probably cleaning up leaks, but I\n    -    didn't check yet.\n    +    We did not bother to clean up at all in `git branch` or `git tag`, and\n    +    `git for-each-ref` only cleans up a couple of members.\n     \n    -    Note that the reachable_from and unreachable_from lists should be\n    +    Add and call `ref_filter_clear()` when cleaning up a `struct\n    +    ref_filter`. Running this patch (without any test changes) indicates a\n    +    couple of now leak-free tests. This was found by running:\n    +\n    +        $ make SANITIZE=leak\n    +        $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate\n    +\n    +    (Note that the `reachable_from` and `unreachable_from` lists should be\n         cleaned as they are used. So this is just covering any case where we\n    -    might bail before running the reachability check.\n    +    might bail before running the reachability check.)\n    +\n    +    Signed-off-by: Jeff King <peff@peff.net>\n    +    Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n      ## builtin/branch.c ##\n     @@ builtin/branch.c: int cmd_branch(int argc, const char **argv, const char *prefix)\n    @@ ref-filter.h: void filter_ahead_behind(struct repository *r,\n     +void ref_filter_clear(struct ref_filter *filter);\n     +\n      #endif /*  REF_FILTER_H  */\n    +\n    + ## t/t0041-usage.sh ##\n    +@@ t/t0041-usage.sh: test_description='Test commands behavior when given invalid argument value'\n    + GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n    + export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n    + \n    ++TEST_PASSES_SANITIZE_LEAK=true\n    + . ./test-lib.sh\n    + \n    + test_expect_success 'setup ' '\n    +\n    + ## t/t3402-rebase-merge.sh ##\n    +@@ t/t3402-rebase-merge.sh: test_description='git rebase --merge test'\n    + GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n    + export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n    + \n    ++TEST_PASSES_SANITIZE_LEAK=true\n    + . ./test-lib.sh\n    + \n    + T=\"A quick brown fox\n 5:  c54000f5f5 !  5:  385890b459 ref-filter.c: parameterize match functions over patterns\n    @@ Commit message\n         Tweak `match_pattern()` and `match_name_as_path()` to take an array of\n         patterns to prepare for passing either in.\n     \n    +    Once we start passing either in, `match_pattern()` will have little to\n    +    do with a particular `struct ref_filter *` instance. To clarify this,\n    +    drop it from the argument list, and replace it with the only bit of the\n    +    `ref_filter` that we care about (`filter->ignore_case`).\n    +\n         Co-authored-by: Taylor Blau <me@ttaylorr.com>\n         Signed-off-by: Jeff King <peff@peff.net>\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n    @@ ref-filter.c: static int get_ref_atom_value(struct ref_array_item *ref, int atom\n       * matches \"refs/heads/mas*\", too).\n       */\n     -static int match_pattern(const struct ref_filter *filter, const char *refname)\n    -+static int match_pattern(const struct ref_filter *filter,\n    -+\t\t\t const char **patterns,\n    -+\t\t\t const char *refname)\n    ++static int match_pattern(const char **patterns, const char *refname,\n    ++\t\t\t const int ignore_case)\n      {\n     -\tconst char **patterns = filter->name_patterns;\n      \tunsigned flags = 0;\n      \n    - \tif (filter->ignore_case)\n    +-\tif (filter->ignore_case)\n    ++\tif (ignore_case)\n    + \t\tflags |= WM_CASEFOLD;\n    + \n    + \t/*\n     @@ ref-filter.c: static int match_pattern(const struct ref_filter *filter, const char *refname)\n       * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n       * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n    @@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const c\n     -\t\treturn match_name_as_path(filter, refname);\n     -\treturn match_pattern(filter, refname);\n     +\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n    -+\treturn match_pattern(filter, filter->name_patterns, refname);\n    ++\treturn match_pattern(filter->name_patterns, refname,\n    ++\t\t\t     filter->ignore_case);\n      }\n      \n      /*\n 6:  ea5c0ddc10 !  6:  1a3371a0a7 builtin/for-each-ref.c: add `--exclude` option\n    @@ Documentation/git-for-each-ref.txt: OPTIONS\n      \n     \n      ## builtin/for-each-ref.c ##\n    -@@ builtin/for-each-ref.c: static char const * const for_each_ref_usage[] = {\n    - \tN_(\"git for-each-ref [--points-at <object>]\"),\n    - \tN_(\"git for-each-ref [--merged [<commit>]] [--no-merged [<commit>]]\"),\n    - \tN_(\"git for-each-ref [--contains [<commit>]] [--no-contains [<commit>]]\"),\n    -+\tN_(\"git for-each-ref [--exclude=<pattern> ...]\"),\n    - \tNULL\n    - };\n    - \n     @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n      \t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n      \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n    @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const\n     \n      ## ref-filter.c ##\n     @@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n    - \treturn match_pattern(filter, filter->name_patterns, refname);\n    + \t\t\t     filter->ignore_case);\n      }\n      \n     +static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n    @@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const c\n     +\t\treturn 0;\n     +\tif (filter->match_as_path)\n     +\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n    -+\treturn match_pattern(filter, filter->exclude.v, refname);\n    ++\treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n     +}\n     +\n      /*\n 7:  f437cd83e2 !  7:  aa05549b6e refs: plumb `exclude_patterns` argument throughout\n    @@ Commit message\n         refs: plumb `exclude_patterns` argument throughout\n     \n         The subsequent patch will want to access an optional `excluded_patterns`\n    -    array within refs/packed-backend.c. To do so, the refs subsystem needs\n    -    to be updated to pass this value across a number of different locations.\n    +    array within `refs/packed-backend.c` that will cull out certain\n    +    references matching any of the given patterns on a best-effort basis.\n    +\n    +    To do so, the refs subsystem needs to be updated to pass this value\n    +    across a number of different locations.\n     \n         Prepare for a future patch by introducing this plumbing now, passing\n         NULLs at top-level APIs in order to make that patch less noisy and more\n 8:  836a5665b7 !  8:  6002c568b5 refs/packed-backend.c: refactor `find_reference_location()`\n    @@ Commit message\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n      ## refs/packed-backend.c ##\n    -@@ refs/packed-backend.c: static int cmp_packed_ref_records(const void *v1, const void *v2)\n    -  * Compare a snapshot record at `rec` to the specified NUL-terminated\n    -  * refname.\n    -  */\n    --static int cmp_record_to_refname(const char *rec, const char *refname)\n    -+static int cmp_record_to_refname(const char *rec, const char *refname,\n    -+\t\t\t\t int start)\n    - {\n    - \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n    - \tconst char *r2 = refname;\n    -@@ refs/packed-backend.c: static int cmp_record_to_refname(const char *rec, const char *refname)\n    - \t\tif (*r1 == '\\n')\n    - \t\t\treturn *r2 ? -1 : 0;\n    - \t\tif (!*r2)\n    --\t\t\treturn 1;\n    -+\t\t\treturn start ? 1 : -1;\n    - \t\tif (*r1 != *r2)\n    - \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n    - \t\tr1++;\n     @@ refs/packed-backend.c: static int load_contents(struct snapshot *snapshot)\n      \treturn 1;\n      }\n    @@ refs/packed-backend.c: static int load_contents(struct snapshot *snapshot)\n     -static const char *find_reference_location(struct snapshot *snapshot,\n     -\t\t\t\t\t   const char *refname, int mustexist)\n     +static const char *find_reference_location_1(struct snapshot *snapshot,\n    -+\t\t\t\t\t     const char *refname, int mustexist,\n    -+\t\t\t\t\t     int start)\n    ++\t\t\t\t\t     const char *refname, int mustexist)\n      {\n      \t/*\n      \t * This is not *quite* a garden-variety binary search, because\n     @@ refs/packed-backend.c: static const char *find_reference_location(struct snapshot *snapshot,\n    - \n    - \t\tmid = lo + (hi - lo) / 2;\n    - \t\trec = find_start_of_record(lo, mid);\n    --\t\tcmp = cmp_record_to_refname(rec, refname);\n    -+\t\tcmp = cmp_record_to_refname(rec, refname, start);\n    - \t\tif (cmp < 0) {\n    - \t\t\tlo = find_end_of_record(mid, hi);\n    - \t\t} else if (cmp > 0) {\n    -@@ refs/packed-backend.c: static const char *find_reference_location(struct snapshot *snapshot,\n      \t\treturn lo;\n      }\n      \n    @@ refs/packed-backend.c: static const char *find_reference_location(struct snapsho\n     +static const char *find_reference_location(struct snapshot *snapshot,\n     +\t\t\t\t\t   const char *refname, int mustexist)\n     +{\n    -+\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n    ++\treturn find_reference_location_1(snapshot, refname, mustexist);\n     +}\n     +\n      /*\n 9:  a39d1107c1 !  9:  8c78f49a8d refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)\n    @@ Metadata\n     Author: Taylor Blau <me@ttaylorr.com>\n     \n      ## Commit message ##\n    -    refs/packed-backend.c: implement skip lists to avoid excluded pattern(s)\n    +    refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)\n     \n         When iterating through the `packed-refs` file in order to answer a query\n         like:\n    @@ Commit message\n         so is as follows:\n     \n           - For each excluded pattern, find the first record that matches it,\n    -        and the first pattern that *doesn't* match it (i.e. the location\n    +        and the first record that *doesn't* match it (i.e. the location\n             you'd next want to consider when excluding that pattern).\n     \n    -      - Sort the patterns by their starting location within the\n    -        `packed-refs` file.\n    +      - Sort the set of excluded regions from the previous step in ascending\n    +        order of the first location within the `packed-refs` file that\n    +        matches.\n     \n    -      - Construct a skip list of regions by combining adjacent and\n    -        overlapping regions from the previous step.\n    +      - Clean up the results from the previous step: discard empty regions,\n    +        and combine adjacent regions.\n     \n    -      - When iterating through the `packed-refs` file, if `iter->pos` is\n    -        ever contained in one of the regions from the previous steps,\n    -        advance `iter->pos` past the end of that region, and continue\n    -        enumeration.\n    +    Then when iterating through the `packed-refs` file, if `iter->pos` is\n    +    ever contained in one of the regions from the previous steps, advance\n    +    `iter->pos` past the end of that region, and continue enumeration.\n     \n    -    Note that this optimization is only possible when none of the excluded\n    -    pattern(s) have special meta-characters in them. To see why this is the\n    -    case, consider the exclusion pattern \"refs/foo[a]\". In general, in order\n    -    to find the location of the first record that matches this pattern, we\n    -    could only consider up to the first meta-character, \"refs/foo\". But this\n    -    doesn't work, since the excluded region we'd come up with would include\n    -    \"refs/foobar\", even though it is not excluded.\n    +    Note that we only perform this optimization when none of the excluded\n    +    pattern(s) have special meta-characters in them. For a pattern like\n    +    \"refs/foo[ac]\", the excluded regions (\"refs/fooa\", \"refs/fooc\", and\n    +    everything underneath them) are not connected. A future implementation\n    +    that handles this case may split the character class (pretending as if\n    +    two patterns were excluded: \"refs/fooa\", and \"refs/fooc\").\n     \n         There are a few other gotchas worth considering. First, note that the\n    -    skip list is sorted, so once we skip past a region, we can avoid\n    +    jump list is sorted, so once we jump past a region, we can avoid\n         considering it (or any regions preceding it) again. The member\n    -    `skip_pos` is used to track the first next-possible region to jump\n    +    `jump_pos` is used to track the first next-possible region to jump\n         through.\n     \n         Second, note that the exclusion list is best-effort, since we do not\n    @@ Commit message\n                 git update-ref --stdin\n             $ git pack-refs --all\n     \n    -    , it is significantly faster to have `for-each-ref` skip over the\n    +    , it is significantly faster to have `for-each-ref` jump over the\n         excluded references, as opposed to filtering them out after the fact:\n     \n             $ hyperfine \\\n    @@ Commit message\n               'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n               172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n     \n    -    Using the skip list is fairly straightforward (see the changes to\n    +    Using the jump list is fairly straightforward (see the changes to\n         `refs/packed-backend.c::next_record()`), but constructing the list is\n         not. To ensure that the construction is correct, add a new suite of\n         tests in t1419 covering various corner cases (overlapping regions,\n    @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n      /*\n     \n      ## refs/packed-backend.c ##\n    -@@ refs/packed-backend.c: static const char *find_reference_location(struct snapshot *snapshot,\n    - \treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n    +@@ refs/packed-backend.c: static int cmp_packed_ref_records(const void *v1, const void *v2)\n    +  * Compare a snapshot record at `rec` to the specified NUL-terminated\n    +  * refname.\n    +  */\n    +-static int cmp_record_to_refname(const char *rec, const char *refname)\n    ++static int cmp_record_to_refname(const char *rec, const char *refname,\n    ++\t\t\t\t int start)\n    + {\n    + \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n    + \tconst char *r2 = refname;\n    +@@ refs/packed-backend.c: static int cmp_record_to_refname(const char *rec, const char *refname)\n    + \t\tif (*r1 == '\\n')\n    + \t\t\treturn *r2 ? -1 : 0;\n    + \t\tif (!*r2)\n    +-\t\t\treturn 1;\n    ++\t\t\treturn start ? 1 : -1;\n    + \t\tif (*r1 != *r2)\n    + \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n    + \t\tr1++;\n    +@@ refs/packed-backend.c: static int load_contents(struct snapshot *snapshot)\n      }\n      \n    + static const char *find_reference_location_1(struct snapshot *snapshot,\n    +-\t\t\t\t\t     const char *refname, int mustexist)\n    ++\t\t\t\t\t     const char *refname, int mustexist,\n    ++\t\t\t\t\t     int start)\n    + {\n    + \t/*\n    + \t * This is not *quite* a garden-variety binary search, because\n    +@@ refs/packed-backend.c: static const char *find_reference_location_1(struct snapshot *snapshot,\n    + \n    + \t\tmid = lo + (hi - lo) / 2;\n    + \t\trec = find_start_of_record(lo, mid);\n    +-\t\tcmp = cmp_record_to_refname(rec, refname);\n    ++\t\tcmp = cmp_record_to_refname(rec, refname, start);\n    + \t\tif (cmp < 0) {\n    + \t\t\tlo = find_end_of_record(mid, hi);\n    + \t\t} else if (cmp > 0) {\n    +@@ refs/packed-backend.c: static const char *find_reference_location_1(struct snapshot *snapshot,\n    + static const char *find_reference_location(struct snapshot *snapshot,\n    + \t\t\t\t\t   const char *refname, int mustexist)\n    + {\n    +-\treturn find_reference_location_1(snapshot, refname, mustexist);\n    ++\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n    ++}\n    ++\n     +/*\n     + * Find the place in `snapshot->buf` after the end of the record for\n     + * `refname`. In other words, find the location of first thing *after*\n    @@ refs/packed-backend.c: static const char *find_reference_location(struct snapsho\n     +\t\t\t\t\t       int mustexist)\n     +{\n     +\treturn find_reference_location_1(snapshot, refname, mustexist, 0);\n    -+}\n    -+\n    + }\n    + \n      /*\n    -  * Create a newly-allocated `snapshot` of the `packed-refs` file in\n    -  * its current state and return it. The return value will already have\n     @@ refs/packed-backend.c: struct packed_ref_iterator {\n      \t/* The end of the part of the buffer that will be iterated over: */\n      \tconst char *eof;\n      \n    -+\tstruct skip_list_entry {\n    ++\tstruct jump_list_entry {\n     +\t\tconst char *start;\n     +\t\tconst char *end;\n    -+\t} *skip;\n    -+\tsize_t skip_nr, skip_alloc;\n    -+\tsize_t skip_pos;\n    ++\t} *jump;\n    ++\tsize_t jump_nr, jump_alloc;\n    ++\tsize_t jump_pos;\n     +\n      \t/* Scratch space for current values: */\n      \tstruct object_id oid, peeled;\n    @@ refs/packed-backend.c: struct packed_ref_iterator {\n     +\t * Note that each skipped region is considered at most once,\n     +\t * since they are ordered based on their starting position.\n     +\t */\n    -+\twhile (iter->skip_pos < iter->skip_nr) {\n    -+\t\tstruct skip_list_entry *curr = &iter->skip[iter->skip_pos];\n    ++\twhile (iter->jump_pos < iter->jump_nr) {\n    ++\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_pos];\n     +\t\tif (iter->pos < curr->start)\n     +\t\t\tbreak; /* not to the next jump yet */\n     +\n    -+\t\titer->skip_pos++;\n    ++\t\titer->jump_pos++;\n     +\t\tif (iter->pos < curr->end) {\n     +\t\t\titer->pos = curr->end;\n     +\t\t\tbreak;\n    @@ refs/packed-backend.c: static int packed_ref_iterator_abort(struct ref_iterator\n      \tint ok = ITER_DONE;\n      \n      \tstrbuf_release(&iter->refname_buf);\n    -+\tfree(iter->skip);\n    ++\tfree(iter->jump);\n      \trelease_snapshot(iter->snapshot);\n      \tbase_ref_iterator_free(ref_iterator);\n      \treturn ok;\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n      \t.abort = packed_ref_iterator_abort\n      };\n      \n    -+static int skip_list_entry_cmp(const void *va, const void *vb)\n    ++static int jump_list_entry_cmp(const void *va, const void *vb)\n     +{\n    -+\tconst struct skip_list_entry *a = va;\n    -+\tconst struct skip_list_entry *b = vb;\n    ++\tconst struct jump_list_entry *a = va;\n    ++\tconst struct jump_list_entry *b = vb;\n     +\n     +\tif (a->start < b->start)\n     +\t\treturn -1;\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\treturn y;\n     +}\n     +\n    -+static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n    ++static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n     +\t\t\t\t\tstruct snapshot *snapshot,\n     +\t\t\t\t\tconst char **excluded_patterns)\n     +{\n     +\tsize_t i, j;\n     +\tconst char **pattern;\n    ++\tstruct jump_list_entry *last_disjoint;\n     +\n     +\tif (!excluded_patterns)\n     +\t\treturn;\n     +\n     +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n    -+\t\tstruct skip_list_entry *e;\n    ++\t\tstruct jump_list_entry *e;\n     +\n     +\t\t/*\n     +\t\t * We can't feed any excludes with globs in them to the\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\t\tif (has_glob_special(*pattern))\n     +\t\t\tcontinue;\n     +\n    -+\t\tALLOC_GROW(iter->skip, iter->skip_nr + 1, iter->skip_alloc);\n    ++\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n     +\n    -+\t\te = &iter->skip[iter->skip_nr++];\n    ++\t\te = &iter->jump[iter->jump_nr++];\n     +\t\te->start = find_reference_location(snapshot, *pattern, 0);\n     +\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n     +\t}\n     +\n    -+\tif (!iter->skip_nr) {\n    ++\tif (!iter->jump_nr) {\n     +\t\t/*\n     +\t\t * Every entry in exclude_patterns has a meta-character,\n     +\t\t * nothing to do here.\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\t\treturn;\n     +\t}\n     +\n    -+\tQSORT(iter->skip, iter->skip_nr, skip_list_entry_cmp);\n    ++\tQSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);\n     +\n     +\t/*\n    -+\t * As an optimization, merge adjacent entries in the skip list\n    ++\t * As an optimization, merge adjacent entries in the jump list\n     +\t * to jump forwards as far as possible when entering a skipped\n     +\t * region.\n     +\t *\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\t * we want to combine that into a single entry jumping from A to\n     +\t * C.\n     +\t */\n    -+\tfor (i = 1, j = 1; i < iter->skip_nr; i++) {\n    -+\t\tstruct skip_list_entry *ours = &iter->skip[i];\n    -+\t\tstruct skip_list_entry *prev = &iter->skip[i - 1];\n    ++\tlast_disjoint = iter->jump;\n    ++\n    ++\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n    ++\t\tstruct jump_list_entry *ours = &iter->jump[i];\n     +\n     +\t\tif (ours->start == ours->end) {\n     +\t\t\t/* ignore empty regions (no matching entries) */\n     +\t\t\tcontinue;\n    -+\t\t} else if (prev->end >= ours->start) {\n    ++\t\t} else if (ours->start <= last_disjoint->end) {\n     +\t\t\t/* overlapping regions extend the previous one */\n    -+\t\t\tprev->end = ptr_max(prev->end, ours->end);\n    ++\t\t\tlast_disjoint->end = ptr_max(last_disjoint->end, ours->end);\n     +\t\t} else {\n     +\t\t\t/* otherwise, insert a new region */\n    -+\t\t\titer->skip[j++] = *ours;\n    ++\t\t\titer->jump[j++] = *ours;\n    ++\t\t\tlast_disjoint = ours;\n    ++\n     +\t\t}\n     +\t}\n     +\n    -+\titer->skip_nr = j;\n    -+\titer->skip_pos = 0;\n    ++\titer->jump_nr = j;\n    ++\titer->jump_pos = 0;\n     +}\n     +\n      static struct ref_iterator *packed_ref_iterator_begin(\n    @@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(\n      \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);\n      \n     +\tif (exclude_patterns)\n    -+\t\tpopulate_excluded_skip_list(iter, snapshot, exclude_patterns);\n    ++\t\tpopulate_excluded_jump_list(iter, snapshot, exclude_patterns);\n     +\n      \titer->snapshot = snapshot;\n      \tacquire_snapshot(snapshot);\n    @@ t/t1419-exclude-refs.sh (new)\n     +\tgit pack-refs --all\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '\n    -+\t# region in middle\n    ++test_expect_success 'excluded region in middle' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n     +\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n     +\n     +\ttest_cmp expect actual\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' '\n    -+\t# region at beginning\n    ++test_expect_success 'excluded region at beginning' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n     +\tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n     +\n     +\ttest_cmp expect actual\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' '\n    -+\t# region at end\n    ++test_expect_success 'excluded region at end' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n     +\tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n     +\n     +\ttest_cmp expect actual\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' '\n    -+\t# disjoint regions\n    ++test_expect_success 'disjoint excluded regions' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n     +\tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n     +\n     +\ttest_cmp expect actual\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' '\n    -+\t# adjacent, non-overlapping regions\n    ++test_expect_success 'adjacent, non-overlapping excluded regions' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n     +\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n     +\n     +\ttest_cmp expect actual\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' '\n    -+\t# overlapping region\n    ++test_expect_success 'overlapping excluded regions' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n     +\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n     +\n     +\ttest_cmp expect actual\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' '\n    -+\t# empty region\n    ++test_expect_success 'several overlapping excluded regions' '\n    ++\tfor_each_ref__exclude refs/heads \\\n    ++\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n    ++\tfor_each_ref refs/heads/quux >expect &&\n    ++\n    ++\ttest_cmp expect actual\n    ++'\n    ++\n    ++test_expect_success 'non-matching excluded section' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n     +\tfor_each_ref >expect &&\n     +\n     +\ttest_cmp expect actual\n     +'\n     +\n    -+test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '\n    -+\t# discards meta-characters\n    ++test_expect_success 'meta-characters are discarded' '\n     +\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n     +\tfor_each_ref >expect &&\n     +\n10:  5698c2794f ! 10:  5059f5dd42 refs/packed-backend.c: add trace2 counters for skip list\n    @@ Metadata\n     Author: Taylor Blau <me@ttaylorr.com>\n     \n      ## Commit message ##\n    -    refs/packed-backend.c: add trace2 counters for skip list\n    +    refs/packed-backend.c: add trace2 counters for jump list\n     \n         The previous commit added low-level tests to ensure that the packed-refs\n         iterator did not enumerate excluded sections of the refspace.\n    @@ refs/packed-backend.c\n      enum mmap_strategy {\n      \t/*\n     @@ refs/packed-backend.c: static int next_record(struct packed_ref_iterator *iter)\n    - \t\titer->skip_pos++;\n    + \t\titer->jump_pos++;\n      \t\tif (iter->pos < curr->end) {\n      \t\t\titer->pos = curr->end;\n    -+\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_SKIPS, 1);\n    ++\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);\n      \t\t\tbreak;\n      \t\t}\n      \t}\n    @@ t/t1419-exclude-refs.sh: for_each_ref () {\n      \tgit for-each-ref --format='%(refname)' \"$@\"\n      }\n      \n    -+assert_skips () {\n    ++assert_jumps () {\n     +\tlocal nr=\"$1\"\n     +\tlocal trace=\"$2\"\n     +\n    -+\tgrep -q \"name:skips_made value:$nr\" $trace\n    ++\tgrep -q \"name:jumps_made value:$nr\" $trace\n     +}\n     +\n    -+assert_no_skips () {\n    -+\t! assert_skips \".*\" \"$1\"\n    ++assert_no_jumps () {\n    ++\t! assert_jumps \".*\" \"$1\"\n     +}\n     +\n      test_expect_success 'setup' '\n      \ttest_commit --no-tag base &&\n      \tbase=\"$(git rev-parse HEAD)\" &&\n     @@ t/t1419-exclude-refs.sh: test_expect_success 'setup' '\n    + '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/foo/)' '\n    - \t# region in middle\n    + test_expect_success 'excluded region in middle' '\n     -\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n     +\tfor_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf &&\n      \tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_skips 1 perf\n    ++\tassert_jumps 1 perf\n      '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/bar/)' '\n    - \t# region at beginning\n    + test_expect_success 'excluded region at beginning' '\n     -\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n     +\tfor_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf &&\n      \tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_skips 1 perf\n    ++\tassert_jumps 1 perf\n      '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/quux/)' '\n    - \t# region at end\n    + test_expect_success 'excluded region at end' '\n     -\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n     +\tfor_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf &&\n      \tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_skips 1 perf\n    ++\tassert_jumps 1 perf\n      '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/quux/)' '\n    - \t# disjoint regions\n    + test_expect_success 'disjoint excluded regions' '\n     -\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n     +\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf &&\n      \tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_skips 2 perf\n    ++\tassert_jumps 2 perf\n      '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/bar/, refs/heads/baz/)' '\n    - \t# adjacent, non-overlapping regions\n    + test_expect_success 'adjacent, non-overlapping excluded regions' '\n     -\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n     +\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf &&\n      \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_skips 1 perf\n    ++\tassert_jumps 1 perf\n      '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/ba refs/heads/baz/)' '\n    - \t# overlapping region\n    + test_expect_success 'overlapping excluded regions' '\n     -\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n     +\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf &&\n      \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_skips 1 perf\n    ++\tassert_jumps 1 perf\n      '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/does/not/exist)' '\n    - \t# empty region\n    + test_expect_success 'several overlapping excluded regions' '\n    + \tfor_each_ref__exclude refs/heads \\\n    +-\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n    ++\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual 2>perf &&\n    + \tfor_each_ref refs/heads/quux >expect &&\n    + \n    +-\ttest_cmp expect actual\n    ++\ttest_cmp expect actual &&\n    ++\tassert_jumps 1 perf\n    + '\n    + \n    + test_expect_success 'non-matching excluded section' '\n     -\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n     +\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf &&\n      \tfor_each_ref >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_no_skips\n    ++\tassert_no_jumps\n      '\n      \n    - test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '\n    - \t# discards meta-characters\n    + test_expect_success 'meta-characters are discarded' '\n     -\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n     +\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual 2>perf &&\n      \tfor_each_ref >expect &&\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_no_skips\n    ++\tassert_no_jumps\n      '\n      \n      test_done\n    @@ trace2.h: enum trace2_counter_id {\n      \tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n      \tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n      \n    -+\tTRACE2_COUNTER_ID_PACKED_REFS_SKIPS, /* counts number of skips */\n    ++\tTRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */\n     +\n      \t/* Add additional counter definitions before here. */\n      \tTRACE2_NUMBER_OF_COUNTERS\n    @@ trace2/tr2_ctr.c: static struct tr2_counter_metadata tr2_counter_metadata[TRACE2\n      \t\t.name = \"test2\",\n      \t\t.want_per_thread_events = 1,\n      \t},\n    -+\t[TRACE2_COUNTER_ID_PACKED_REFS_SKIPS] = {\n    ++\t[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {\n     +\t\t.category = \"packed-refs\",\n    -+\t\t.name = \"skips_made\",\n    ++\t\t.name = \"jumps_made\",\n     +\t\t.want_per_thread_events = 0,\n     +\t},\n      \n11:  5b9814ad8c ! 11:  f765b50a84 revision.h: store hidden refs in a `strvec`\n    @@ revision.c: void init_ref_exclusions(struct ref_exclusions *exclusions)\n     \n      ## revision.h ##\n     @@\n    - #include \"commit-slab-decl.h\"\n    + #include \"decorate.h\"\n      #include \"ident.h\"\n      #include \"list-objects-filter-options.h\"\n     +#include \"strvec.h\"\n12:  dd5b34185c ! 12:  254bcc4361 refs/packed-backend.c: ignore complicated hidden refs rules\n    @@ Commit message\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n      ## refs/packed-backend.c ##\n    -@@ refs/packed-backend.c: static void populate_excluded_skip_list(struct packed_ref_iterator *iter,\n    +@@ refs/packed-backend.c: static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n      \tif (!excluded_patterns)\n      \t\treturn;\n      \n    @@ refs/packed-backend.c: static void populate_excluded_skip_list(struct packed_ref\n     +\t\t * '^' rules.\n     +\t\t *\n     +\t\t * Both are possible to do, but complicated, so avoid\n    -+\t\t * populating the skip list at all if we see either of\n    ++\t\t * populating the jump list at all if we see either of\n     +\t\t * these patterns.\n     +\t\t */\n     +\t\tif (**pattern == '!' || **pattern == '^')\n    @@ refs/packed-backend.c: static void populate_excluded_skip_list(struct packed_ref\n     +\t}\n     +\n      \tfor (pattern = excluded_patterns; *pattern; pattern++) {\n    - \t\tstruct skip_list_entry *e;\n    + \t\tstruct jump_list_entry *e;\n      \n     \n      ## t/t1419-exclude-refs.sh ##\n    -@@ t/t1419-exclude-refs.sh: test_expect_success 'for_each_ref__exclude(refs/heads/ba*)' '\n    - \tassert_no_skips\n    +@@ t/t1419-exclude-refs.sh: test_expect_success 'meta-characters are discarded' '\n    + \tassert_no_jumps\n      '\n      \n    -+test_expect_success 'for_each_ref__exclude(refs/heads/foo, !refs/heads/foo/1)' '\n    -+\t# discards complex hidden ref rules\n    ++test_expect_success 'complex hidden ref rules are discarded' '\n     +\tfor_each_ref__exclude refs/heads refs/heads/foo \"!refs/heads/foo/1\" \\\n     +\t\t>actual 2>perf &&\n     +\tfor_each_ref >expect &&\n     +\n     +\ttest_cmp expect actual &&\n    -+\tassert_no_skips\n    ++\tassert_no_jumps\n     +'\n     +\n      test_done\n13:  c65b3dea81 = 13:  50e7df7dc0 refs.h: let `for_each_namespaced_ref()` take excluded patterns\n15:  7d3383083d = 14:  f6a3a5a6ba builtin/receive-pack.c: avoid enumerating hidden references\n14:  44bbf85e73 ! 15:  2331fa7a4d upload-pack.c: avoid enumerating hidden refs where possible\n    @@ upload-pack.c: static int get_common_commits(struct upload_pack_data *data,\n     +\t\t\t\t      struct upload_pack_data *data)\n     +{\n     +\t/*\n    -+\t * If `data->allow_uor` allows updating hidden refs, we need to\n    ++\t * If `data->allow_uor` allows fetching hidden refs, we need to\n     +\t * mark all references (including hidden ones), to check in\n     +\t * `is_our_ref()` below.\n     +\t *\n -:  ---------- > 16:  2c6b89d64a ls-refs.c: avoid enumerating hidden refs where possible\n-- \n2.40.1.572.g5c4ab523ef\n"},{"id":"477294","messageId":"6cac42e70e9c1901761bd3068a9e22e60fe4deae.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 01/16] refs.c: rename `ref_filter`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:10Z","receivedAt":"2023-05-15T19:23:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nThe refs machinery has its own implementation of a `ref_filter` (used by\n`for-each-ref`), which is distinct from the `ref-filler.h` API (also\nused by `for-each-ref`, among other things).\n\nRename the one within refs.c to more clearly indicate its purpose.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex d2a98e1c21..b9b77d2eff 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,\n \t\t\t\t   oid, flags);\n }\n \n-/* The argument to filter_refs */\n-struct ref_filter {\n+/* The argument to for_each_filter_refs */\n+struct for_each_ref_filter {\n \tconst char *pattern;\n \tconst char *prefix;\n \teach_ref_fn *fn;\n@@ -409,10 +409,11 @@ int ref_exists(const char *refname)\n \treturn refs_ref_exists(get_main_ref_store(the_repository), refname);\n }\n \n-static int filter_refs(const char *refname, const struct object_id *oid,\n-\t\t\t   int flags, void *data)\n+static int for_each_filter_refs(const char *refname,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tint flags, void *data)\n {\n-\tstruct ref_filter *filter = (struct ref_filter *)data;\n+\tstruct for_each_ref_filter *filter = data;\n \n \tif (wildmatch(filter->pattern, refname, 0))\n \t\treturn 0;\n@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tconst char *prefix, void *cb_data)\n {\n \tstruct strbuf real_pattern = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct for_each_ref_filter filter;\n \tint ret;\n \n \tif (!prefix && !starts_with(pattern, \"refs/\"))\n@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tfilter.prefix = prefix;\n \tfilter.fn = fn;\n \tfilter.cb_data = cb_data;\n-\tret = for_each_ref(filter_refs, &filter);\n+\tret = for_each_ref(for_each_filter_refs, &filter);\n \n \tstrbuf_release(&real_pattern);\n \treturn ret;\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477295","messageId":"8dda7db73816c2e97ed6e195ac8ffb5321f649b7.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 02/16] ref-filter.h: provide `REF_FILTER_INIT`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:13Z","receivedAt":"2023-05-15T19:23:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nProvide a sane initialization value for `struct ref_filter`, which in a\nsubsequent patch will be used to initialize a new field.\n\nIn the meantime, fix a case in test-reach.c where its `ref_filter` is\nnot even zero-initialized.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c       | 3 +--\n builtin/for-each-ref.c | 3 +--\n builtin/tag.c          | 3 +--\n ref-filter.h           | 3 +++\n t/helper/test-reach.c  | 2 +-\n 5 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 501c47657c..03bb8e414c 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -662,7 +662,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \tint reflog = 0, quiet = 0, icase = 0, force = 0,\n \t    recurse_submodules_explicit = 0;\n \tenum branch_track track;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstatic struct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -720,7 +720,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \n \tsetup_ref_filter_porcelain_msg();\n \n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.kind = FILTER_REFS_BRANCHES;\n \tfilter.abbrev = -1;\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 695fc8f4a5..99ccb73518 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tint maxcount = 0, icase = 0, omit_empty = 0;\n \tstruct ref_array array;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t};\n \n \tmemset(&array, 0, sizeof(array));\n-\tmemset(&filter, 0, sizeof(filter));\n \n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 1850a6a6fd..6b41bb7374 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -443,7 +443,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tstruct msg_arg msg = { .buf = STRBUF_INIT };\n \tstruct ref_transaction *transaction;\n \tstruct strbuf err = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -501,7 +501,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tgit_config(git_tag_config, &sorting_options);\n \n \tmemset(&opt, 0, sizeof(opt));\n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.lines = -1;\n \topt.sign = -1;\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 430701cfb7..a920f73b29 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,9 @@ struct ref_format {\n \tstruct string_list bases;\n };\n \n+#define REF_FILTER_INIT { \\\n+\t.points_at = OID_ARRAY_INIT, \\\n+}\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n \t.bases = STRING_LIST_INIT_DUP, \\\ndiff --git a/t/helper/test-reach.c b/t/helper/test-reach.c\nindex 5b6f217441..ef58f10c2d 100644\n--- a/t/helper/test-reach.c\n+++ b/t/helper/test-reach.c\n@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)\n \n \t\tprintf(\"%s(X,_,_,0,0):%d\\n\", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));\n \t} else if (!strcmp(av[1], \"commit_contains\")) {\n-\t\tstruct ref_filter filter;\n+\t\tstruct ref_filter filter = REF_FILTER_INIT;\n \t\tstruct contains_cache cache;\n \t\tinit_contains_cache(&cache);\n \n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477296","messageId":"bf21df783d3f99099a4e0a53c9f813690f6ef785.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 03/16] ref-filter: clear reachable list pointers after freeing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:16Z","receivedAt":"2023-05-15T19:23:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nIn reach_filter(), we pop all commits from the reachable lists, leaving\nthem empty. But because we're operating on a list pointer that was\npassed by value, the original filter.reachable_from pointer is left\ndangling.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4991cd4f7a..048d277cbf 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2418,13 +2418,13 @@ void ref_array_clear(struct ref_array *array)\n #define EXCLUDE_REACHED 0\n #define INCLUDE_REACHED 1\n static void reach_filter(struct ref_array *array,\n-\t\t\t struct commit_list *check_reachable,\n+\t\t\t struct commit_list **check_reachable,\n \t\t\t int include_reached)\n {\n \tint i, old_nr;\n \tstruct commit **to_clear;\n \n-\tif (!check_reachable)\n+\tif (!*check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n@@ -2434,7 +2434,7 @@ static void reach_filter(struct ref_array *array,\n \t}\n \n \ttips_reachable_from_bases(the_repository,\n-\t\t\t\t  check_reachable,\n+\t\t\t\t  *check_reachable,\n \t\t\t\t  to_clear, array->nr,\n \t\t\t\t  UNINTERESTING);\n \n@@ -2455,8 +2455,8 @@ static void reach_filter(struct ref_array *array,\n \n \tclear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);\n \n-\twhile (check_reachable) {\n-\t\tstruct commit *merge_commit = pop_commit(&check_reachable);\n+\twhile (*check_reachable) {\n+\t\tstruct commit *merge_commit = pop_commit(check_reachable);\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n@@ -2553,8 +2553,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \tclear_contains_cache(&ref_cbdata.no_contains_cache);\n \n \t/*  Filters that need revision walking */\n-\treach_filter(array, filter->reachable_from, INCLUDE_REACHED);\n-\treach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);\n+\treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n+\treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n \n \tsave_commit_buffer = save_commit_buffer_orig;\n \treturn ret;\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477297","messageId":"85ecb70957d27911adc6d7a36dd08ba435684528.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 04/16] ref-filter: add `ref_filter_clear()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:19Z","receivedAt":"2023-05-15T19:23:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nWe did not bother to clean up at all in `git branch` or `git tag`, and\n`git for-each-ref` only cleans up a couple of members.\n\nAdd and call `ref_filter_clear()` when cleaning up a `struct\nref_filter`. Running this patch (without any test changes) indicates a\ncouple of now leak-free tests. This was found by running:\n\n    $ make SANITIZE=leak\n    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate\n\n(Note that the `reachable_from` and `unreachable_from` lists should be\ncleaned as they are used. So this is just covering any case where we\nmight bail before running the reachability check.)\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c        |  1 +\n builtin/for-each-ref.c  |  3 +--\n builtin/tag.c           |  1 +\n ref-filter.c            | 16 ++++++++++++++++\n ref-filter.h            |  3 +++\n t/t0041-usage.sh        |  1 +\n t/t3402-rebase-merge.sh |  1 +\n 7 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 03bb8e414c..c201f0cb0b 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -813,6 +813,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\tprint_columns(&output, colopts, NULL);\n \t\tstring_list_clear(&output, 0);\n \t\tref_sorting_release(sorting);\n+\t\tref_filter_clear(&filter);\n \t\treturn 0;\n \t} else if (edit_description) {\n \t\tconst char *branch_name;\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 99ccb73518..c01fa6fefe 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&err);\n \tstrbuf_release(&output);\n \tref_array_clear(&array);\n-\tfree_commit_list(filter.with_commit);\n-\tfree_commit_list(filter.no_commit);\n+\tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\n \tstrvec_clear(&vec);\n \treturn 0;\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 6b41bb7374..aab5e693fe 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -645,6 +645,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \n cleanup:\n \tref_sorting_release(sorting);\n+\tref_filter_clear(&filter);\n \tstrbuf_release(&buf);\n \tstrbuf_release(&ref);\n \tstrbuf_release(&reflog_msg);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 048d277cbf..d32f426898 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2866,3 +2866,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)\n \n \treturn 0;\n }\n+\n+void ref_filter_init(struct ref_filter *filter)\n+{\n+\tstruct ref_filter blank = REF_FILTER_INIT;\n+\tmemcpy(filter, &blank, sizeof(blank));\n+}\n+\n+void ref_filter_clear(struct ref_filter *filter)\n+{\n+\toid_array_clear(&filter->points_at);\n+\tfree_commit_list(filter->with_commit);\n+\tfree_commit_list(filter->no_commit);\n+\tfree_commit_list(filter->reachable_from);\n+\tfree_commit_list(filter->unreachable_from);\n+\tref_filter_init(filter);\n+}\ndiff --git a/ref-filter.h b/ref-filter.h\nindex a920f73b29..160b807224 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,\n \t\t\t struct ref_format *format,\n \t\t\t struct ref_array *array);\n \n+void ref_filter_init(struct ref_filter *filter);\n+void ref_filter_clear(struct ref_filter *filter);\n+\n #endif /*  REF_FILTER_H  */\ndiff --git a/t/t0041-usage.sh b/t/t0041-usage.sh\nindex c4fc34eb18..9ea974b0c6 100755\n--- a/t/t0041-usage.sh\n+++ b/t/t0041-usage.sh\n@@ -5,6 +5,7 @@ test_description='Test commands behavior when given invalid argument value'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'setup ' '\ndiff --git a/t/t3402-rebase-merge.sh b/t/t3402-rebase-merge.sh\nindex 79b0640c00..e9e03ca4b5 100755\n--- a/t/t3402-rebase-merge.sh\n+++ b/t/t3402-rebase-merge.sh\n@@ -8,6 +8,7 @@ test_description='git rebase --merge test'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n T=\"A quick brown fox\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477298","messageId":"385890b45960556cdffe5c09211baf4f8d3df0ad.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:22Z","receivedAt":"2023-05-15T19:23:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\n`match_pattern()` and `match_name_as_path()` both take a `struct\nref_filter *`, and then store a stack variable `patterns` pointing at\n`filter->patterns`.\n\nThe subsequent patch will add a new array of patterns to match over (the\nexcluded patterns, via a new `git for-each-ref --exclude` option),\ntreating the return value of these functions differently depending on\nwhich patterns are being used to match.\n\nTweak `match_pattern()` and `match_name_as_path()` to take an array of\npatterns to prepare for passing either in.\n\nOnce we start passing either in, `match_pattern()` will have little to\ndo with a particular `struct ref_filter *` instance. To clarify this,\ndrop it from the argument list, and replace it with the only bit of the\n`ref_filter` that we care about (`filter->ignore_case`).\n\nCo-authored-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 16 +++++++++-------\n 1 file changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex d32f426898..6d91c7cb0d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2104,12 +2104,12 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n  * matches a pattern \"refs/heads/mas\") or a wildcard (e.g. the same ref\n  * matches \"refs/heads/mas*\", too).\n  */\n-static int match_pattern(const struct ref_filter *filter, const char *refname)\n+static int match_pattern(const char **patterns, const char *refname,\n+\t\t\t const int ignore_case)\n {\n-\tconst char **patterns = filter->name_patterns;\n \tunsigned flags = 0;\n \n-\tif (filter->ignore_case)\n+\tif (ignore_case)\n \t\tflags |= WM_CASEFOLD;\n \n \t/*\n@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n  * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n  * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n  */\n-static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n+static int match_name_as_path(const struct ref_filter *filter,\n+\t\t\t      const char **pattern,\n+\t\t\t      const char *refname)\n {\n-\tconst char **pattern = filter->name_patterns;\n \tint namelen = strlen(refname);\n \tunsigned flags = WM_PATHNAME;\n \n@@ -2165,8 +2166,9 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \tif (!*filter->name_patterns)\n \t\treturn 1; /* No pattern always matches */\n \tif (filter->match_as_path)\n-\t\treturn match_name_as_path(filter, refname);\n-\treturn match_pattern(filter, refname);\n+\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n+\treturn match_pattern(filter->name_patterns, refname,\n+\t\t\t     filter->ignore_case);\n }\n \n /*\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477299","messageId":"1a3371a0a7511a7b47cffdf7094c51b027dd005b.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 06/16] builtin/for-each-ref.c: add `--exclude` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:25Z","receivedAt":"2023-05-15T19:23:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When using `for-each-ref`, it is sometimes convenient for the caller to\nbe able to exclude certain parts of the references.\n\nFor example, if there are many `refs/__hidden__/*` references, the\ncaller may want to emit all references *except* the hidden ones.\nCurrently, the only way to do this is to post-process the output, like:\n\n    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'\n\nWhich is do-able, but requires processing a potentially large quantity\nof references.\n\nTeach `git for-each-ref` a new `--exclude=<pattern>` option, which\nexcludes references from the results if they match one or more excluded\npatterns.\n\nThis patch provides a naive implementation where the `ref_filter` still\nsees all references (including ones that it will discard) and is left to\ncheck whether each reference matches any excluded pattern(s) before\nemitting them.\n\nBy culling out references we know the caller doesn't care about, we can\navoid allocating memory for their storage, as well as spending time\nsorting the output (among other things). Even the naive implementation\nprovides a significant speed-up on a modified copy of linux.git (that\nhas a hidden ref pointing at each commit):\n\n    $ hyperfine \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/'\n    Benchmark 1: git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]\n      Range (min … max):   817.7 ms … 823.3 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/\n      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]\n      Range (min … max):   104.7 ms … 109.1 ms    27 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/' ran\n        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"'\n\nSubsequent patches will improve on this by avoiding visiting excluded\nsections of the `packed-refs` file in certain cases.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-for-each-ref.txt |  6 +++++\n builtin/for-each-ref.c             |  1 +\n ref-filter.c                       | 13 +++++++++++\n ref-filter.h                       |  6 +++++\n t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++\n 5 files changed, 61 insertions(+)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 1e215d4e73..5743eb5def 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t\t   [--points-at=<object>]\n \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n+\t\t   [--exclude=<pattern> ...]\n \n DESCRIPTION\n -----------\n@@ -102,6 +103,11 @@ OPTIONS\n \tDo not print a newline after formatted refs where the format expands\n \tto the empty string.\n \n+--exclude=<pattern>::\n+\tIf one or more patterns are given, only refs which do not match\n+\tany excluded pattern(s) are shown. Matching is done using the\n+\tsame rules as `<pattern>` above.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex c01fa6fefe..3384987428 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -47,6 +47,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n+\t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n \t\t\t     N_(\"object\"), N_(\"print only refs which points at the given object\"),\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 6d91c7cb0d..d44418efb7 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2171,6 +2171,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \t\t\t     filter->ignore_case);\n }\n \n+static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n+{\n+\tif (!filter->exclude.nr)\n+\t\treturn 0;\n+\tif (filter->match_as_path)\n+\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n+\treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n+}\n+\n /*\n  * This is the same as for_each_fullref_in(), but it tries to iterate\n  * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2338,6 +2347,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \tif (!filter_pattern_match(filter, refname))\n \t\treturn 0;\n \n+\tif (filter_exclude_match(filter, refname))\n+\t\treturn 0;\n+\n \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n \t\treturn 0;\n \n@@ -2877,6 +2889,7 @@ void ref_filter_init(struct ref_filter *filter)\n \n void ref_filter_clear(struct ref_filter *filter)\n {\n+\tstrvec_clear(&filter->exclude);\n \toid_array_clear(&filter->points_at);\n \tfree_commit_list(filter->with_commit);\n \tfree_commit_list(filter->no_commit);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 160b807224..1524bc463a 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -6,6 +6,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -59,6 +60,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\n \tstruct commit_list *no_commit;\n@@ -94,6 +96,7 @@ struct ref_format {\n \n #define REF_FILTER_INIT { \\\n \t.points_at = OID_ARRAY_INIT, \\\n+\t.exclude = STRVEC_INIT, \\\n }\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n@@ -112,6 +115,9 @@ struct ref_format {\n #define OPT_REF_SORT(var) \\\n \tOPT_STRING_LIST(0, \"sort\", (var), \\\n \t\t\tN_(\"key\"), N_(\"field name to sort on\"))\n+#define OPT_REF_FILTER_EXCLUDE(var) \\\n+\tOPT_STRVEC(0, \"exclude\", &(var)->exclude, \\\n+\t\t   N_(\"pattern\"), N_(\"exclude refs which match pattern\"))\n \n /*\n  * API for filtering a set of refs. Based on the type of refs the user\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 5c00607608..7e8d578522 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '\n \ttest_cmp expected actual\n '\n \n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with prefix exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=refs/tags/foo >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/foo/one\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with pattern exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=\"refs/tags/foo/t*\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n cat >expected <<\\EOF\n 'refs/heads/main'\n 'refs/remotes/origin/main'\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477300","messageId":"aa05549b6ee6b2d3be01e66b8f49fb9ed4813575.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 07/16] refs: plumb `exclude_patterns` argument throughout","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:28Z","receivedAt":"2023-05-15T19:24:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The subsequent patch will want to access an optional `excluded_patterns`\narray within `refs/packed-backend.c` that will cull out certain\nreferences matching any of the given patterns on a best-effort basis.\n\nTo do so, the refs subsystem needs to be updated to pass this value\nacross a number of different locations.\n\nPrepare for a future patch by introducing this plumbing now, passing\nNULLs at top-level APIs in order to make that patch less noisy and more\neasily readable.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.co>\n---\n ls-refs.c             |  2 +-\n ref-filter.c          |  5 +++--\n refs.c                | 32 +++++++++++++++++++-------------\n refs.h                |  8 +++++++-\n refs/debug.c          |  5 +++--\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  5 +++--\n refs/refs-internal.h  |  7 ++++---\n revision.c            |  2 +-\n 9 files changed, 44 insertions(+), 27 deletions(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex f385938b64..6f490b2d9c 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  send_ref, &data);\n+\t\t\t\t\t  NULL, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex d44418efb7..717c3c4bcf 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n+\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t cb, cb_data);\n+\t\t\t\t\t\t NULL, cb, cb_data);\n }\n \n /*\ndiff --git a/refs.c b/refs.c\nindex b9b77d2eff..538bde644e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)\n \n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n+\t\tconst char *prefix,\n+\t\tconst char **exclude_patterns,\n+\t\tint trim,\n \t\tenum do_for_each_ref_flags flags)\n {\n \tstruct ref_iterator *iter;\n@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(\n \t\t}\n \t}\n \n-\titer = refs->be->iterator_begin(refs, prefix, flags);\n-\n+\titer = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);\n \t/*\n \t * `iterator_begin()` already takes care of prefix, but we\n \t * might need to do some trimming:\n@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);\n \n \treturn do_for_each_repo_ref_iterator(r, iter, fn, cb_data);\n }\n@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,\n }\n \n static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n+\t\t\t   const char **exclude_patterns,\n \t\t\t   each_ref_fn fn, int trim,\n \t\t\t   enum do_for_each_ref_flags flags, void *cb_data)\n {\n@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,\n+\t\t\t\t       flags);\n \n \treturn do_for_each_repo_ref_iterator(the_repository, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \n int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t       prefix, fn, 0, 0, cb_data);\n+\t\t\t       prefix, NULL, fn, 0, 0, cb_data);\n }\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);\n }\n \n int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\n \n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0,\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n \t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,\n int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \t\t\t\t      const char *namespace,\n \t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data)\n {\n \tstruct string_list prefixes = STRING_LIST_INIT_DUP;\n@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \n \tfor_each_string_list_item(prefix, &prefixes) {\n \t\tstrbuf_addstr(&buf, prefix->string);\n-\t\tret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);\n+\t\tret = refs_for_each_fullref_in(ref_store, buf.buf,\n+\t\t\t\t\t       exclude_patterns, fn, cb_data);\n \t\tif (ret)\n \t\t\tbreak;\n \t\tstrbuf_setlen(&buf, namespace_len);\n@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,\n \tstrbuf_addstr(&dirname, refname + dirname.len);\n \tstrbuf_addch(&dirname, '/');\n \n-\titer = refs_ref_iterator_begin(refs, dirname.buf, 0,\n+\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 123cfa4424..d672d636cf 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -338,6 +338,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);\n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data);\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n@@ -345,10 +346,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n  * iterate all refs in \"patterns\" by partitioning patterns into disjoint sets\n  * and iterating the longest-common prefix of each set.\n  *\n+ * references matching any pattern in \"exclude_patterns\" are omitted from the\n+ * result set on a best-effort basis.\n+ *\n  * callers should be prepared to ignore references that they did not ask for.\n  */\n int refs_for_each_fullref_in_prefixes(struct ref_store *refs,\n-\t\t\t\t      const char *namespace, const char **patterns,\n+\t\t\t\t      const char *namespace,\n+\t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data);\n \n /**\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 6f11e6de46..328f894177 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -229,11 +229,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \n static struct ref_iterator *\n debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n-\t\t\t unsigned int flags)\n+\t\t\t const char **exclude_patterns, unsigned int flags)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n \tstruct ref_iterator *res =\n-\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix, flags);\n+\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix,\n+\t\t\t\t\t\texclude_patterns, flags);\n \tstruct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));\n \tbase_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);\n \tditer->iter = res;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bca7b851c5..3bc3c57c05 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -829,7 +829,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \n static struct ref_iterator *files_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct files_ref_store *refs;\n \tstruct ref_iterator *loose_iter, *packed_iter, *overlay_iter;\n@@ -874,7 +875,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * the packed and loose references.\n \t */\n \tpacked_iter = refs_ref_iterator_begin(\n-\t\t\trefs->packed_ref_store, prefix, 0,\n+\t\t\trefs->packed_ref_store, prefix, exclude_patterns, 0,\n \t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n \n \toverlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 5b412a133b..176bd3905b 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -924,7 +924,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n@@ -1149,7 +1150,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t * list of refs is exhausted, set iter to NULL. When the list\n \t * of updates is exhausted, leave i set to updates->nr.\n \t */\n-\titer = packed_ref_iterator_begin(&refs->base, \"\",\n+\titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n \tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n \t\titer = NULL;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a85d113123..28a11b9d61 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n  */\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n-\t\tenum do_for_each_ref_flags flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tint trim, enum do_for_each_ref_flags flags);\n \n /*\n  * A callback function used to instruct merge_ref_iterator how to\n@@ -570,7 +570,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,\n  */\n typedef struct ref_iterator *ref_iterator_begin_fn(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags);\n \n /* reflog functions */\n \ndiff --git a/revision.c b/revision.c\nindex b33cc1d106..89953592f9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,\n \tstruct strbuf bisect_refs = STRBUF_INIT;\n \tint status;\n \tstrbuf_addf(&bisect_refs, \"refs/bisect/%s\", term);\n-\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);\n+\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);\n \tstrbuf_release(&bisect_refs);\n \treturn status;\n }\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477301","messageId":"6002c568b54a3ea2b87f4eed13d342f6e63ec420.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 08/16] refs/packed-backend.c: refactor `find_reference_location()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:31Z","receivedAt":"2023-05-15T19:24:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `find_reference_location()` is used to perform a\nbinary search-like function over the contents of a repository's\n`$GIT_DIR/packed-refs` file.\n\nThe search it implements is unlike a standard binary search in that the\nrecords it searches over are not of a fixed width, so the comparison\nmust locate the end of a record before comparing it.\n\nExtract the core routine of `find_reference_location()` in order to\nimplement a function in the following patch which will find the first\nlocation in the `packed-refs` file that *doesn't* match the given\npattern.\n\nThe behavior of `find_reference_location()` is unchanged.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c | 38 ++++++++++++++++++++++----------------\n 1 file changed, 22 insertions(+), 16 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 176bd3905b..33639f73e1 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -527,22 +527,8 @@ static int load_contents(struct snapshot *snapshot)\n \treturn 1;\n }\n \n-/*\n- * Find the place in `snapshot->buf` where the start of the record for\n- * `refname` starts. If `mustexist` is true and the reference doesn't\n- * exist, then return NULL. If `mustexist` is false and the reference\n- * doesn't exist, then return the point where that reference would be\n- * inserted, or `snapshot->eof` (which might be NULL) if it would be\n- * inserted at the end of the file. In the latter mode, `refname`\n- * doesn't have to be a proper reference name; for example, one could\n- * search for \"refs/replace/\" to find the start of any replace\n- * references.\n- *\n- * The record is sought using a binary search, so `snapshot->buf` must\n- * be sorted.\n- */\n-static const char *find_reference_location(struct snapshot *snapshot,\n-\t\t\t\t\t   const char *refname, int mustexist)\n+static const char *find_reference_location_1(struct snapshot *snapshot,\n+\t\t\t\t\t     const char *refname, int mustexist)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -588,6 +574,26 @@ static const char *find_reference_location(struct snapshot *snapshot,\n \t\treturn lo;\n }\n \n+/*\n+ * Find the place in `snapshot->buf` where the start of the record for\n+ * `refname` starts. If `mustexist` is true and the reference doesn't\n+ * exist, then return NULL. If `mustexist` is false and the reference\n+ * doesn't exist, then return the point where that reference would be\n+ * inserted, or `snapshot->eof` (which might be NULL) if it would be\n+ * inserted at the end of the file. In the latter mode, `refname`\n+ * doesn't have to be a proper reference name; for example, one could\n+ * search for \"refs/replace/\" to find the start of any replace\n+ * references.\n+ *\n+ * The record is sought using a binary search, so `snapshot->buf` must\n+ * be sorted.\n+ */\n+static const char *find_reference_location(struct snapshot *snapshot,\n+\t\t\t\t\t   const char *refname, int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist);\n+}\n+\n /*\n  * Create a newly-allocated `snapshot` of the `packed-refs` file in\n  * its current state and return it. The return value will already have\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477302","messageId":"8c78f49a8def4148c92cdd7efe4daa63fb47f641.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:33Z","receivedAt":"2023-05-15T19:24:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When iterating through the `packed-refs` file in order to answer a query\nlike:\n\n    $ git for-each-ref --exclude=refs/__hidden__\n\nit would be useful to avoid walking over all of the entries in\n`refs/__hidden__/*` when possible, since we know that the ref-filter\ncode is going to throw them away anyways.\n\nIn certain circumstances, doing so is possible. The algorithm for doing\nso is as follows:\n\n  - For each excluded pattern, find the first record that matches it,\n    and the first record that *doesn't* match it (i.e. the location\n    you'd next want to consider when excluding that pattern).\n\n  - Sort the set of excluded regions from the previous step in ascending\n    order of the first location within the `packed-refs` file that\n    matches.\n\n  - Clean up the results from the previous step: discard empty regions,\n    and combine adjacent regions.\n\nThen when iterating through the `packed-refs` file, if `iter->pos` is\never contained in one of the regions from the previous steps, advance\n`iter->pos` past the end of that region, and continue enumeration.\n\nNote that we only perform this optimization when none of the excluded\npattern(s) have special meta-characters in them. For a pattern like\n\"refs/foo[ac]\", the excluded regions (\"refs/fooa\", \"refs/fooc\", and\neverything underneath them) are not connected. A future implementation\nthat handles this case may split the character class (pretending as if\ntwo patterns were excluded: \"refs/fooa\", and \"refs/fooc\").\n\nThere are a few other gotchas worth considering. First, note that the\njump list is sorted, so once we jump past a region, we can avoid\nconsidering it (or any regions preceding it) again. The member\n`jump_pos` is used to track the first next-possible region to jump\nthrough.\n\nSecond, note that the exclusion list is best-effort, since we do not\nhandle loose references, and because of the meta-character issue above.\n\nIn repositories with a large number of hidden references, the speed-up\ncan be significant. Tests here are done with a copy of linux.git with a\nreference \"refs/pull/N\" pointing at every commit, as in:\n\n    $ git rev-list HEAD | awk '{ print \"create refs/pull/\" NR \" \" $0 }' |\n        git update-ref --stdin\n    $ git pack-refs --all\n\n, it is significantly faster to have `for-each-ref` jump over the\nexcluded references, as opposed to filtering them out after the fact:\n\n    $ hyperfine \\\n      'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n    Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n      Range (min … max):   800.0 ms … 807.7 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n      Range (min … max):     4.3 ms …   6.7 ms    422 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n      172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n\nUsing the jump list is fairly straightforward (see the changes to\n`refs/packed-backend.c::next_record()`), but constructing the list is\nnot. To ensure that the construction is correct, add a new suite of\ntests in t1419 covering various corner cases (overlapping regions,\npartially overlapping regions, adjacent regions, etc.).\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c              |   5 +-\n refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--\n t/helper/test-ref-store.c |  10 +++\n t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++\n 4 files changed, 274 insertions(+), 8 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 717c3c4bcf..ddc7f5204f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2210,12 +2210,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n \t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n+\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t NULL, cb, cb_data);\n+\t\t\t\t\t\t filter->exclude.v,\n+\t\t\t\t\t\t cb, cb_data);\n }\n \n /*\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 33639f73e1..67327e579c 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -303,7 +303,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)\n  * Compare a snapshot record at `rec` to the specified NUL-terminated\n  * refname.\n  */\n-static int cmp_record_to_refname(const char *rec, const char *refname)\n+static int cmp_record_to_refname(const char *rec, const char *refname,\n+\t\t\t\t int start)\n {\n \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n \tconst char *r2 = refname;\n@@ -312,7 +313,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)\n \t\tif (*r1 == '\\n')\n \t\t\treturn *r2 ? -1 : 0;\n \t\tif (!*r2)\n-\t\t\treturn 1;\n+\t\t\treturn start ? 1 : -1;\n \t\tif (*r1 != *r2)\n \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n \t\tr1++;\n@@ -528,7 +529,8 @@ static int load_contents(struct snapshot *snapshot)\n }\n \n static const char *find_reference_location_1(struct snapshot *snapshot,\n-\t\t\t\t\t     const char *refname, int mustexist)\n+\t\t\t\t\t     const char *refname, int mustexist,\n+\t\t\t\t\t     int start)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -558,7 +560,7 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n \n \t\tmid = lo + (hi - lo) / 2;\n \t\trec = find_start_of_record(lo, mid);\n-\t\tcmp = cmp_record_to_refname(rec, refname);\n+\t\tcmp = cmp_record_to_refname(rec, refname, start);\n \t\tif (cmp < 0) {\n \t\t\tlo = find_end_of_record(mid, hi);\n \t\t} else if (cmp > 0) {\n@@ -591,7 +593,22 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n static const char *find_reference_location(struct snapshot *snapshot,\n \t\t\t\t\t   const char *refname, int mustexist)\n {\n-\treturn find_reference_location_1(snapshot, refname, mustexist);\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n+}\n+\n+/*\n+ * Find the place in `snapshot->buf` after the end of the record for\n+ * `refname`. In other words, find the location of first thing *after*\n+ * `refname`.\n+ *\n+ * Other semantics are identical to the ones in\n+ * `find_reference_location()`.\n+ */\n+static const char *find_reference_location_end(struct snapshot *snapshot,\n+\t\t\t\t\t       const char *refname,\n+\t\t\t\t\t       int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 0);\n }\n \n /*\n@@ -785,6 +802,13 @@ struct packed_ref_iterator {\n \t/* The end of the part of the buffer that will be iterated over: */\n \tconst char *eof;\n \n+\tstruct jump_list_entry {\n+\t\tconst char *start;\n+\t\tconst char *end;\n+\t} *jump;\n+\tsize_t jump_nr, jump_alloc;\n+\tsize_t jump_pos;\n+\n \t/* Scratch space for current values: */\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n@@ -802,14 +826,34 @@ struct packed_ref_iterator {\n  */\n static int next_record(struct packed_ref_iterator *iter)\n {\n-\tconst char *p = iter->pos, *eol;\n+\tconst char *p, *eol;\n \n \tstrbuf_reset(&iter->refname_buf);\n \n+\t/*\n+\t * If iter->pos is contained within a skipped region, jump past\n+\t * it.\n+\t *\n+\t * Note that each skipped region is considered at most once,\n+\t * since they are ordered based on their starting position.\n+\t */\n+\twhile (iter->jump_pos < iter->jump_nr) {\n+\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_pos];\n+\t\tif (iter->pos < curr->start)\n+\t\t\tbreak; /* not to the next jump yet */\n+\n+\t\titer->jump_pos++;\n+\t\tif (iter->pos < curr->end) {\n+\t\t\titer->pos = curr->end;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n \tif (iter->pos == iter->eof)\n \t\treturn ITER_DONE;\n \n \titer->base.flags = REF_ISPACKED;\n+\tp = iter->pos;\n \n \tif (iter->eof - p < the_hash_algo->hexsz + 2 ||\n \t    parse_oid_hex(p, &iter->oid, &p) ||\n@@ -917,6 +961,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \tint ok = ITER_DONE;\n \n \tstrbuf_release(&iter->refname_buf);\n+\tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n \tbase_ref_iterator_free(ref_iterator);\n \treturn ok;\n@@ -928,6 +973,112 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.abort = packed_ref_iterator_abort\n };\n \n+static int jump_list_entry_cmp(const void *va, const void *vb)\n+{\n+\tconst struct jump_list_entry *a = va;\n+\tconst struct jump_list_entry *b = vb;\n+\n+\tif (a->start < b->start)\n+\t\treturn -1;\n+\tif (a->start > b->start)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int has_glob_special(const char *str)\n+{\n+\tconst char *p;\n+\tfor (p = str; *p; p++) {\n+\t\tif (is_glob_special(*p))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static const char *ptr_max(const char *x, const char *y)\n+{\n+\tif (x > y)\n+\t\treturn x;\n+\treturn y;\n+}\n+\n+static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n+\t\t\t\t\tstruct snapshot *snapshot,\n+\t\t\t\t\tconst char **excluded_patterns)\n+{\n+\tsize_t i, j;\n+\tconst char **pattern;\n+\tstruct jump_list_entry *last_disjoint;\n+\n+\tif (!excluded_patterns)\n+\t\treturn;\n+\n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\tstruct jump_list_entry *e;\n+\n+\t\t/*\n+\t\t * We can't feed any excludes with globs in them to the\n+\t\t * refs machinery.  It only understands prefix matching.\n+\t\t * We likewise can't even feed the string leading up to\n+\t\t * the first meta-character, as something like \"foo[a]\"\n+\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n+\t\t * would match that and mark it for exclusion).\n+\t\t */\n+\t\tif (has_glob_special(*pattern))\n+\t\t\tcontinue;\n+\n+\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n+\n+\t\te = &iter->jump[iter->jump_nr++];\n+\t\te->start = find_reference_location(snapshot, *pattern, 0);\n+\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n+\t}\n+\n+\tif (!iter->jump_nr) {\n+\t\t/*\n+\t\t * Every entry in exclude_patterns has a meta-character,\n+\t\t * nothing to do here.\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tQSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);\n+\n+\t/*\n+\t * As an optimization, merge adjacent entries in the jump list\n+\t * to jump forwards as far as possible when entering a skipped\n+\t * region.\n+\t *\n+\t * For example, if we have two skipped regions:\n+\t *\n+\t *\t[[A, B], [B, C]]\n+\t *\n+\t * we want to combine that into a single entry jumping from A to\n+\t * C.\n+\t */\n+\tlast_disjoint = iter->jump;\n+\n+\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n+\t\tstruct jump_list_entry *ours = &iter->jump[i];\n+\n+\t\tif (ours->start == ours->end) {\n+\t\t\t/* ignore empty regions (no matching entries) */\n+\t\t\tcontinue;\n+\t\t} else if (ours->start <= last_disjoint->end) {\n+\t\t\t/* overlapping regions extend the previous one */\n+\t\t\tlast_disjoint->end = ptr_max(last_disjoint->end, ours->end);\n+\t\t} else {\n+\t\t\t/* otherwise, insert a new region */\n+\t\t\titer->jump[j++] = *ours;\n+\t\t\tlast_disjoint = ours;\n+\n+\t\t}\n+\t}\n+\n+\titer->jump_nr = j;\n+\titer->jump_pos = 0;\n+}\n+\n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n \t\tconst char *prefix, const char **exclude_patterns,\n@@ -963,6 +1114,9 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);\n \n+\tif (exclude_patterns)\n+\t\tpopulate_excluded_jump_list(iter, snapshot, exclude_patterns);\n+\n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n \ndiff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\nindex 6d8f844e9c..2bff003f7c 100644\n--- a/t/helper/test-ref-store.c\n+++ b/t/helper/test-ref-store.c\n@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n \treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n }\n \n+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)\n+{\n+\tconst char *prefix = notnull(*argv++, \"prefix\");\n+\tconst char **exclude_patterns = argv;\n+\n+\treturn refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,\n+\t\t\t\t\tNULL);\n+}\n+\n static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\n {\n \tstruct object_id oid = *null_oid();\n@@ -307,6 +316,7 @@ static struct command commands[] = {\n \t{ \"delete-refs\", cmd_delete_refs },\n \t{ \"rename-ref\", cmd_rename_ref },\n \t{ \"for-each-ref\", cmd_for_each_ref },\n+\t{ \"for-each-ref--exclude\", cmd_for_each_ref__exclude },\n \t{ \"resolve-ref\", cmd_resolve_ref },\n \t{ \"verify-ref\", cmd_verify_ref },\n \t{ \"for-each-reflog\", cmd_for_each_reflog },\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nnew file mode 100755\nindex 0000000000..bc534c8ea1\n--- /dev/null\n+++ b/t/t1419-exclude-refs.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='test exclude_patterns functionality in main ref store'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+for_each_ref__exclude () {\n+\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tcut -d ' ' -f 2 actual.raw\n+}\n+\n+for_each_ref () {\n+\tgit for-each-ref --format='%(refname)' \"$@\"\n+}\n+\n+test_expect_success 'setup' '\n+\ttest_commit --no-tag base &&\n+\tbase=\"$(git rev-parse HEAD)\" &&\n+\n+\tfor name in foo bar baz quux\n+\tdo\n+\t\tfor i in 1 2 3\n+\t\tdo\n+\t\t\techo \"create refs/heads/$name/$i $base\" || return 1\n+\t\tdone || return 1\n+\tdone >in &&\n+\techo \"delete refs/heads/main\" >>in &&\n+\n+\tgit update-ref --stdin <in &&\n+\tgit pack-refs --all\n+'\n+\n+test_expect_success 'excluded region in middle' '\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at beginning' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at end' '\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'disjoint excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'adjacent, non-overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'several overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads \\\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'non-matching excluded section' '\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'meta-characters are discarded' '\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477303","messageId":"5059f5dd421c94164ce5aee19b8f920417aee27d.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 10/16] refs/packed-backend.c: add trace2 counters for jump list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:36Z","receivedAt":"2023-05-15T19:24:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The previous commit added low-level tests to ensure that the packed-refs\niterator did not enumerate excluded sections of the refspace.\n\nHowever, there was no guarantee that these sections weren't being\nvisited, only that they were being suppressed from the output. To harden\nthese tests, add a trace2 counter which tracks the number of regions\nskipped by the packed-refs iterator, and assert on its value.\n\nSuggested-by: Derrick Stolee <derrickstolee@github.com>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   |  2 ++\n t/t1419-exclude-refs.sh | 59 ++++++++++++++++++++++++++++-------------\n trace2.h                |  2 ++\n trace2/tr2_ctr.c        |  5 ++++\n 4 files changed, 49 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 67327e579c..7ba9fa2bb8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -12,6 +12,7 @@\n #include \"../chdir-notify.h\"\n #include \"../wrapper.h\"\n #include \"../write-or-die.h\"\n+#include \"../trace2.h\"\n \n enum mmap_strategy {\n \t/*\n@@ -845,6 +846,7 @@ static int next_record(struct packed_ref_iterator *iter)\n \t\titer->jump_pos++;\n \t\tif (iter->pos < curr->end) {\n \t\t\titer->pos = curr->end;\n+\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);\n \t\t\tbreak;\n \t\t}\n \t}\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex bc534c8ea1..350a7d2587 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -9,7 +9,8 @@ TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n for_each_ref__exclude () {\n-\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tGIT_TRACE2_PERF=1 test-tool ref-store main \\\n+\t\tfor-each-ref--exclude \"$@\" >actual.raw\n \tcut -d ' ' -f 2 actual.raw\n }\n \n@@ -17,6 +18,17 @@ for_each_ref () {\n \tgit for-each-ref --format='%(refname)' \"$@\"\n }\n \n+assert_jumps () {\n+\tlocal nr=\"$1\"\n+\tlocal trace=\"$2\"\n+\n+\tgrep -q \"name:jumps_made value:$nr\" $trace\n+}\n+\n+assert_no_jumps () {\n+\t! assert_jumps \".*\" \"$1\"\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit --no-tag base &&\n \tbase=\"$(git rev-parse HEAD)\" &&\n@@ -35,67 +47,76 @@ test_expect_success 'setup' '\n '\n \n test_expect_success 'excluded region in middle' '\n-\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at beginning' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at end' '\n-\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'disjoint excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 2 perf\n '\n \n test_expect_success 'adjacent, non-overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'several overlapping excluded regions' '\n \tfor_each_ref__exclude refs/heads \\\n-\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'non-matching excluded section' '\n-\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps\n '\n \n test_expect_success 'meta-characters are discarded' '\n-\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps\n '\n \n test_done\ndiff --git a/trace2.h b/trace2.h\nindex 4ced30c0db..9452e291f5 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -551,6 +551,8 @@ enum trace2_counter_id {\n \tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n \tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n \n+\tTRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */\n+\n \t/* Add additional counter definitions before here. */\n \tTRACE2_NUMBER_OF_COUNTERS\n };\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nindex b342d3b1a3..50570d0165 100644\n--- a/trace2/tr2_ctr.c\n+++ b/trace2/tr2_ctr.c\n@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER\n \t\t.name = \"test2\",\n \t\t.want_per_thread_events = 1,\n \t},\n+\t[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {\n+\t\t.category = \"packed-refs\",\n+\t\t.name = \"jumps_made\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n \n \t/* Add additional metadata before here. */\n };\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477304","messageId":"f765b50a8405f1653cbc2e79b375c979c4361550.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 11/16] revision.h: store hidden refs in a `strvec`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:39Z","receivedAt":"2023-05-15T19:24:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, it will be convenient to have a 'const char **'\nof hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\netc.), instead of a `string_list`.\n\nConvert spots throughout the tree that store the list of hidden refs\nfrom a `string_list` to a `strvec`.\n\nNote that in `parse_hide_refs_config()` there is an ugly const-cast used\nto avoid an extra copy of each value before trimming any trailing slash\ncharacters. This could instead be written as:\n\n    ref = xstrdup(value);\n    len = strlen(ref);\n    while (len && ref[len - 1] == '/')\n            ref[--len] = '\\0';\n    strvec_push(hide_refs, ref);\n    free(ref);\n\nbut the double-copy (once when calling `xstrdup()`, and another via\n`strvec_push()`) is wasteful.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c |  4 ++--\n ls-refs.c              |  6 +++---\n refs.c                 | 11 ++++++-----\n refs.h                 |  4 ++--\n revision.c             |  2 +-\n revision.h             |  5 +++--\n upload-pack.c          | 10 +++++-----\n 7 files changed, 22 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1a31a58367..1a8472eddc 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -90,7 +90,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+static struct strvec hidden_refs = STRVEC_INIT;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -2619,7 +2619,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n-\tstring_list_clear(&hidden_refs, 0);\n+\tstrvec_clear(&hidden_refs);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 6f490b2d9c..8c3181d051 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -72,7 +72,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -155,7 +155,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n-\tstring_list_init_dup(&data.hidden_refs);\n+\tstrvec_init(&data.hidden_refs);\n \n \tgit_config(ls_refs_config, &data);\n \n@@ -197,7 +197,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-\tstring_list_clear(&data.hidden_refs, 0);\n+\tstrvec_clear(&data.hidden_refs);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 538bde644e..ec4d5b9101 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n }\n \n int parse_hide_refs_config(const char *var, const char *value, const char *section,\n-\t\t\t   struct string_list *hide_refs)\n+\t\t\t   struct strvec *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n-\t\tref = xstrdup(value);\n+\n+\t\t/* drop const to remove trailing '/' characters */\n+\t\tref = (char *)strvec_push(hide_refs, value);\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n int ref_is_hidden(const char *refname, const char *refname_full,\n-\t\t  const struct string_list *hide_refs)\n+\t\t  const struct strvec *hide_refs)\n {\n \tint i;\n \n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n-\t\tconst char *match = hide_refs->items[i].string;\n+\t\tconst char *match = hide_refs->v[i];\n \t\tconst char *subject;\n \t\tint neg = 0;\n \t\tconst char *p;\ndiff --git a/refs.h b/refs.h\nindex d672d636cf..a7751a1fc9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -810,7 +810,7 @@ int update_ref(const char *msg, const char *refname,\n \t       unsigned int flags, enum action_on_err onerr);\n \n int parse_hide_refs_config(const char *var, const char *value, const char *,\n-\t\t\t   struct string_list *);\n+\t\t\t   struct strvec *);\n \n /*\n  * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -820,7 +820,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,\n  * the ref is outside that namespace, the first parameter is NULL. The second\n  * parameter always points to the full ref name.\n  */\n-int ref_is_hidden(const char *, const char *, const struct string_list *);\n+int ref_is_hidden(const char *, const char *, const struct strvec *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/revision.c b/revision.c\nindex 89953592f9..7c9367a266 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n-\tstring_list_clear(&exclusions->hidden_refs, 0);\n+\tstrvec_clear(&exclusions->hidden_refs);\n \texclusions->hidden_refs_configured = 0;\n }\n \ndiff --git a/revision.h b/revision.h\nindex 31828748dc..94f035fa22 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -10,6 +10,7 @@\n #include \"decorate.h\"\n #include \"ident.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"strvec.h\"\n \n /**\n  * The revision walking API offers functions to build a list of revisions\n@@ -95,7 +96,7 @@ struct ref_exclusions {\n \t * Hidden refs is a list of patterns that is to be hidden via\n \t * `ref_is_hidden()`.\n \t */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \t/*\n \t * Indicates whether hidden refs have been configured. This is to\n@@ -110,7 +111,7 @@ struct ref_exclusions {\n  */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n-\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRVEC_INIT, \\\n }\n \n struct oidset;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 08633dc121..d77d58bdde 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -69,7 +69,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -126,7 +126,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n-\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+\tstruct strvec hidden_refs = STRVEC_INIT;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -161,7 +161,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n-\tstring_list_clear(&data->hidden_refs, 0);\n+\tstrvec_clear(&data->hidden_refs);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -1169,7 +1169,7 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n+\t\t\tconst struct object_id *oid, const struct strvec *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n@@ -1453,7 +1453,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t  struct string_list *wanted_refs,\n-\t\t\t  struct string_list *hidden_refs,\n+\t\t\t  struct strvec *hidden_refs,\n \t\t\t  struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477305","messageId":"254bcc436124e70ae477d2668c22f2274e43e513.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 12/16] refs/packed-backend.c: ignore complicated hidden refs rules","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:42Z","receivedAt":"2023-05-15T19:24:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\nuse the new skip-list feature in the packed-refs iterator by ignoring\nreferences which are mentioned via its respective hideRefs lists.\n\nHowever, the packed-ref skip lists cannot handle un-hiding rules (that\nbegin with '!'), or namespace comparisons (that begin with '^'). Detect\nand avoid these cases by falling back to the normal enumeration without\na skip list when such patterns exist.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   | 19 +++++++++++++++++++\n t/t1419-exclude-refs.sh |  9 +++++++++\n 2 files changed, 28 insertions(+)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 7ba9fa2bb8..9ea6c07866 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1015,6 +1015,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n \tif (!excluded_patterns)\n \t\treturn;\n \n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\t/*\n+\t\t * We also can't feed any excludes from hidden refs\n+\t\t * config sections, since later rules may override\n+\t\t * previous ones. For example, with rules \"refs/foo\" and\n+\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n+\t\t * everything underneath it), but the earlier exclusion\n+\t\t * would cause us to skip all of \"refs/foo\". We likewise\n+\t\t * don't implement the namespace stripping required for\n+\t\t * '^' rules.\n+\t\t *\n+\t\t * Both are possible to do, but complicated, so avoid\n+\t\t * populating the jump list at all if we see either of\n+\t\t * these patterns.\n+\t\t */\n+\t\tif (**pattern == '!' || **pattern == '^')\n+\t\t\treturn;\n+\t}\n+\n \tfor (pattern = excluded_patterns; *pattern; pattern++) {\n \t\tstruct jump_list_entry *e;\n \ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex 350a7d2587..0e91e2f399 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -119,4 +119,13 @@ test_expect_success 'meta-characters are discarded' '\n \tassert_no_jumps\n '\n \n+test_expect_success 'complex hidden ref rules are discarded' '\n+\tfor_each_ref__exclude refs/heads refs/heads/foo \"!refs/heads/foo/1\" \\\n+\t\t>actual 2>perf &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps\n+'\n+\n test_done\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477306","messageId":"50e7df7dc0932a53682664a6334791d8f1ff31bb.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 13/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:45Z","receivedAt":"2023-05-15T19:24:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The following commit will want to call `for_each_namespaced_ref()` with\na list of excluded patterns.\n\nWe could introduce a variant of that function, say,\n`for_each_namespaced_ref_exclude()` which takes the extra parameter, and\nreimplement the original function in terms of that. But all but one\ncaller (in `http-backend.c`) will supply the new parameter, so add the\nnew parameter to `for_each_namespaced_ref()` itself instead of\nintroducing a new function.\n\nFor now, supply NULL for the list of excluded patterns at all callers to\navoid changing behavior, which we will do in the subsequent commit.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n http-backend.c | 2 +-\n refs.c         | 5 +++--\n refs.h         | 3 ++-\n upload-pack.c  | 6 +++---\n 4 files changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex ac146d85c5..ad500683c8 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)\n \n \t} else {\n \t\tselect_getanyfile(hdr);\n-\t\tfor_each_namespaced_ref(show_text_ref, &buf);\n+\t\tfor_each_namespaced_ref(NULL, show_text_ref, &buf);\n \t\tsend_strbuf(hdr, \"text/plain\", &buf);\n \t}\n \tstrbuf_release(&buf);\ndiff --git a/refs.c b/refs.c\nindex ec4d5b9101..95a7db9563 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat\n \t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, exclude_patterns, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\ndiff --git a/refs.h b/refs.h\nindex a7751a1fc9..f23626beca 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -372,7 +372,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \t\t\t const char *prefix, void *cb_data);\n \n int head_ref_namespaced(each_ref_fn fn, void *cb_data);\n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data);\n \n /* can be used to learn about broken ref and symref */\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d77d58bdde..7c646ea5bd 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -854,7 +854,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(check_ref, data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1378,7 +1378,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(send_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n \t\t/*\n \t\t * fflush stdout before calling advertise_shallow_grafts because send_ref\n \t\t * uses stdio.\n@@ -1388,7 +1388,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(check_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477307","messageId":"f6a3a5a6bafdb4c7c07f49092a9a15b482b4ee2c.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 14/16] builtin/receive-pack.c: avoid enumerating hidden references","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:48Z","receivedAt":"2023-05-15T19:24:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that `refs_for_each_fullref_in()` has the ability to avoid\nenumerating references matching certain pattern(s), use that to avoid\nvisiting hidden refs when constructing the ref advertisement via\nreceive-pack.\n\nNote that since this exclusion is best-effort, we still need\n`show_ref_cb()` to check whether or not each reference is hidden or not\nbefore including it in the advertisement.\n\nAs was the case when applying this same optimization to `upload-pack`,\n`receive-pack`'s reference advertisement phase can proceed much quicker\nby avoiding enumerating references that will not be part of the\nadvertisement.\n\n(Below, we're still using linux.git with one hidden refs/pull/N ref per\ncommit):\n\n    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]\n      Range (min … max):    87.7 ms …  95.5 ms    31 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]\n      Range (min … max):     4.1 ms …   5.6 ms    508 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran\n       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1a8472eddc..bd5bcc375f 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -337,7 +337,8 @@ static void write_head_info(void)\n {\n \tstatic struct oidset seen = OIDSET_INIT;\n \n-\tfor_each_ref(show_ref_cb, &seen);\n+\trefs_for_each_fullref_in(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t hidden_refs.v, show_ref_cb, &seen);\n \tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n \toidset_clear(&seen);\n \tif (!sent_capabilities)\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477308","messageId":"2331fa7a4d0ef0ed2445b25064e2b3270bed5d2b.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 15/16] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:51Z","receivedAt":"2023-05-15T19:24:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as a previous commit, teach `upload-pack` to avoid\nenumerating hidden references where possible.\n\nNote, however, that there are certain cases where cannot avoid\nenumerating even hidden references, in particular when either of:\n\n  - `uploadpack.allowTipSHA1InWant`, or\n  - `uploadpack.allowReachableSHA1InWant`\n\nare set, corresponding to `ALLOW_TIP_SHA1` and `ALLOW_REACHABLE_SHA1`,\nrespectively.\n\nWhen either of these bits are set, upload-pack's `is_our_ref()` function\nneeds to consider the `HIDDEN_REF` bit of the referent's object flags.\nSo we must visit all references, including the hidden ones, in order to\nmark their referents with the `HIDDEN_REF` bit.\n\nWhen neither `ALLOW_TIP_SHA1` nor `ALLOW_REACHABLE_SHA1` are set, the\n`is_our_ref()` function considers only the `OUR_REF` bit, and not the\n`HIDDEN_REF` one. `OUR_REF` is applied via `mark_our_ref()`, and only\nto objects at the tips of non-hidden references, so we do not need to\nvisit hidden references in this case.\n\nWhen neither of those bits are set, `upload-pack` can potentially avoid\nenumerating a large number of references. In the same example as a\nprevious commit (linux.git with one hidden reference per commit,\n\"refs/pull/N\"):\n\n    $ printf 0000 >in\n    $ hyperfine --warmup=1 \\\n      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]\n      Range (min … max):   405.7 ms … 409.2 ms    10 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in\n      Time (mean ± σ):     406.5 ms ±   1.3 ms    [User: 356.5 ms, System: 49.9 ms]\n      Range (min … max):   404.6 ms … 408.8 ms    10 runs\n\n    Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):       4.7 ms ±   0.2 ms    [User: 0.7 ms, System: 3.9 ms]\n      Range (min … max):     4.3 ms …   6.1 ms    472 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran\n       86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in'\n       86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'\n\nAs above, we must visit every reference when\nuploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit\nfar fewer references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n upload-pack.c | 33 +++++++++++++++++++++++++++------\n 1 file changed, 27 insertions(+), 6 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7c646ea5bd..6fa667bf25 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -601,11 +601,32 @@ static int get_common_commits(struct upload_pack_data *data,\n \t}\n }\n \n+static int allow_hidden_refs(enum allow_uor allow_uor)\n+{\n+\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n+}\n+\n+static void for_each_namespaced_ref_1(each_ref_fn fn,\n+\t\t\t\t      struct upload_pack_data *data)\n+{\n+\t/*\n+\t * If `data->allow_uor` allows fetching hidden refs, we need to\n+\t * mark all references (including hidden ones), to check in\n+\t * `is_our_ref()` below.\n+\t *\n+\t * Otherwise, we only care about whether each reference's object\n+\t * has the OUR_REF bit set or not, so do not need to visit\n+\t * hidden references.\n+\t */\n+\tif (allow_hidden_refs(data->allow_uor))\n+\t\tfor_each_namespaced_ref(NULL, fn, data);\n+\telse\n+\t\tfor_each_namespaced_ref(data->hidden_refs.v, fn, data);\n+}\n+\n static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n {\n-\tint allow_hidden_ref = (allow_uor &\n-\t\t\t\t(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n-\treturn o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);\n+\treturn o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);\n }\n \n /*\n@@ -854,7 +875,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n+\t\tfor_each_namespaced_ref_1(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1378,7 +1399,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n+\t\tfor_each_namespaced_ref_1(send_ref, &data);\n \t\t/*\n \t\t * fflush stdout before calling advertise_shallow_grafts because send_ref\n \t\t * uses stdio.\n@@ -1388,7 +1409,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n+\t\tfor_each_namespaced_ref_1(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.40.1.572.g5c4ab523ef\n\n"},{"id":"477309","messageId":"2c6b89d64a164bdeec34db13017f3228c6a73184.1684178576.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"[PATCH v2 16/16] ls-refs.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-05-15T19:23:53Z","receivedAt":"2023-05-15T19:24:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach `ls-refs` to avoid\nenumerating hidden references where possible.\n\nAs before, this is linux.git with one hidden reference per commit.\n\n    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'\n    Benchmark 1: git -c protocol.version=2 ls-remote .\n      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]\n      Range (min … max):    88.8 ms …  91.3 ms    32 runs\n\n    Benchmark 2: git.compile -c protocol.version=2 ls-remote .\n      Time (mean ± σ):       6.5 ms ±   0.1 ms    [User: 2.4 ms, System: 4.3 ms]\n      Range (min … max):     6.2 ms …   8.3 ms    397 runs\n\n    Summary\n      'git.compile -c protocol.version=2 ls-remote .' ran\n       13.85 ± 0.33 times faster than 'git -c protocol.version=2 ls-remote .'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ls-refs.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 8c3181d051..c9a723ba89 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  NULL, send_ref, &data);\n+\t\t\t\t\t  data.hidden_refs.v, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-- \n2.40.1.572.g5c4ab523ef\n"},{"id":"478084","messageId":"ZH7ZqjYqyvSOHY8u@ncase","threadId":"59715","inReplyTo":"f765b50a8405f1653cbc2e79b375c979c4361550.1684178576.git.me@ttaylorr.com","subject":"Re: [PATCH v2 11/16] revision.h: store hidden refs in a `strvec`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-06-06T07:00:58Z","receivedAt":"2023-06-06T07:01:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 15, 2023 at 03:23:39PM -0400, Taylor Blau wrote:\n> In subsequent commits, it will be convenient to have a 'const char **'\n> of hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\n> etc.), instead of a `string_list`.\n> \n> Convert spots throughout the tree that store the list of hidden refs\n> from a `string_list` to a `strvec`.\n> \n> Note that in `parse_hide_refs_config()` there is an ugly const-cast used\n> to avoid an extra copy of each value before trimming any trailing slash\n> characters. This could instead be written as:\n> \n>     ref = xstrdup(value);\n>     len = strlen(ref);\n>     while (len && ref[len - 1] == '/')\n>             ref[--len] = '\\0';\n>     strvec_push(hide_refs, ref);\n>     free(ref);\n> \n> but the double-copy (once when calling `xstrdup()`, and another via\n> `strvec_push()`) is wasteful.\n\nI guess the proper way to fix this would be to introduce something like\na `strvec_push_nodup()` function that takes ownership. And in fact this\nhelper exists already, but it's declared as static. So we could get\naround the ugly cast with a simple change to expose the helper function.\n\nPatrick\n"},{"id":"478085","messageId":"ZH7ZpB8eMBZN8q63@ncase","threadId":"59715","inReplyTo":"8c78f49a8def4148c92cdd7efe4daa63fb47f641.1684178576.git.me@ttaylorr.com","subject":"Re: [PATCH v2 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-06-06T07:00:52Z","receivedAt":"2023-06-06T07:01:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 15, 2023 at 03:23:33PM -0400, Taylor Blau wrote:\n> When iterating through the `packed-refs` file in order to answer a query\n> like:\n> \n>     $ git for-each-ref --exclude=refs/__hidden__\n> \n> it would be useful to avoid walking over all of the entries in\n> `refs/__hidden__/*` when possible, since we know that the ref-filter\n> code is going to throw them away anyways.\n> \n> In certain circumstances, doing so is possible. The algorithm for doing\n> so is as follows:\n> \n>   - For each excluded pattern, find the first record that matches it,\n>     and the first record that *doesn't* match it (i.e. the location\n>     you'd next want to consider when excluding that pattern).\n> \n>   - Sort the set of excluded regions from the previous step in ascending\n>     order of the first location within the `packed-refs` file that\n>     matches.\n> \n>   - Clean up the results from the previous step: discard empty regions,\n>     and combine adjacent regions.\n> \n> Then when iterating through the `packed-refs` file, if `iter->pos` is\n> ever contained in one of the regions from the previous steps, advance\n> `iter->pos` past the end of that region, and continue enumeration.\n> \n> Note that we only perform this optimization when none of the excluded\n> pattern(s) have special meta-characters in them. For a pattern like\n> \"refs/foo[ac]\", the excluded regions (\"refs/fooa\", \"refs/fooc\", and\n> everything underneath them) are not connected. A future implementation\n> that handles this case may split the character class (pretending as if\n> two patterns were excluded: \"refs/fooa\", and \"refs/fooc\").\n> \n> There are a few other gotchas worth considering. First, note that the\n> jump list is sorted, so once we jump past a region, we can avoid\n> considering it (or any regions preceding it) again. The member\n> `jump_pos` is used to track the first next-possible region to jump\n> through.\n> \n> Second, note that the exclusion list is best-effort, since we do not\n> handle loose references, and because of the meta-character issue above.\n> \n> In repositories with a large number of hidden references, the speed-up\n> can be significant. Tests here are done with a copy of linux.git with a\n> reference \"refs/pull/N\" pointing at every commit, as in:\n> \n>     $ git rev-list HEAD | awk '{ print \"create refs/pull/\" NR \" \" $0 }' |\n>         git update-ref --stdin\n>     $ git pack-refs --all\n> \n> , it is significantly faster to have `for-each-ref` jump over the\n> excluded references, as opposed to filtering them out after the fact:\n> \n>     $ hyperfine \\\n>       'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n>       'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n>     Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n>       Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n>       Range (min … max):   800.0 ms … 807.7 ms    10 runs\n> \n>     Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n>       Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n>       Range (min … max):     4.3 ms …   6.7 ms    422 runs\n> \n>     Summary\n>       'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n>       172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n> \n> Using the jump list is fairly straightforward (see the changes to\n> `refs/packed-backend.c::next_record()`), but constructing the list is\n> not. To ensure that the construction is correct, add a new suite of\n> tests in t1419 covering various corner cases (overlapping regions,\n> partially overlapping regions, adjacent regions, etc.).\n> \n> Co-authored-by: Jeff King <peff@peff.net>\n> Signed-off-by: Jeff King <peff@peff.net>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  ref-filter.c              |   5 +-\n>  refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--\n>  t/helper/test-ref-store.c |  10 +++\n>  t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++\n>  4 files changed, 274 insertions(+), 8 deletions(-)\n>  create mode 100755 t/t1419-exclude-refs.sh\n> \n> diff --git a/ref-filter.c b/ref-filter.c\n> index 717c3c4bcf..ddc7f5204f 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2210,12 +2210,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n>  \tif (!filter->name_patterns[0]) {\n>  \t\t/* no patterns; we have to look at everything */\n>  \t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> -\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n> +\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n>  \t}\n>  \n>  \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n>  \t\t\t\t\t\t NULL, filter->name_patterns,\n> -\t\t\t\t\t\t NULL, cb, cb_data);\n> +\t\t\t\t\t\t filter->exclude.v,\n> +\t\t\t\t\t\t cb, cb_data);\n>  }\n>  \n>  /*\n> diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> index 33639f73e1..67327e579c 100644\n> --- a/refs/packed-backend.c\n> +++ b/refs/packed-backend.c\n> @@ -303,7 +303,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)\n>   * Compare a snapshot record at `rec` to the specified NUL-terminated\n>   * refname.\n>   */\n> -static int cmp_record_to_refname(const char *rec, const char *refname)\n> +static int cmp_record_to_refname(const char *rec, const char *refname,\n> +\t\t\t\t int start)\n>  {\n>  \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n>  \tconst char *r2 = refname;\n> @@ -312,7 +313,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)\n>  \t\tif (*r1 == '\\n')\n>  \t\t\treturn *r2 ? -1 : 0;\n>  \t\tif (!*r2)\n> -\t\t\treturn 1;\n> +\t\t\treturn start ? 1 : -1;\n>  \t\tif (*r1 != *r2)\n>  \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n>  \t\tr1++;\n> @@ -528,7 +529,8 @@ static int load_contents(struct snapshot *snapshot)\n>  }\n>  \n>  static const char *find_reference_location_1(struct snapshot *snapshot,\n> -\t\t\t\t\t     const char *refname, int mustexist)\n> +\t\t\t\t\t     const char *refname, int mustexist,\n> +\t\t\t\t\t     int start)\n>  {\n>  \t/*\n>  \t * This is not *quite* a garden-variety binary search, because\n> @@ -558,7 +560,7 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n>  \n>  \t\tmid = lo + (hi - lo) / 2;\n>  \t\trec = find_start_of_record(lo, mid);\n> -\t\tcmp = cmp_record_to_refname(rec, refname);\n> +\t\tcmp = cmp_record_to_refname(rec, refname, start);\n>  \t\tif (cmp < 0) {\n>  \t\t\tlo = find_end_of_record(mid, hi);\n>  \t\t} else if (cmp > 0) {\n> @@ -591,7 +593,22 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n>  static const char *find_reference_location(struct snapshot *snapshot,\n>  \t\t\t\t\t   const char *refname, int mustexist)\n>  {\n> -\treturn find_reference_location_1(snapshot, refname, mustexist);\n> +\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n> +}\n> +\n> +/*\n> + * Find the place in `snapshot->buf` after the end of the record for\n> + * `refname`. In other words, find the location of first thing *after*\n> + * `refname`.\n> + *\n> + * Other semantics are identical to the ones in\n> + * `find_reference_location()`.\n> + */\n> +static const char *find_reference_location_end(struct snapshot *snapshot,\n> +\t\t\t\t\t       const char *refname,\n> +\t\t\t\t\t       int mustexist)\n> +{\n> +\treturn find_reference_location_1(snapshot, refname, mustexist, 0);\n>  }\n>  \n>  /*\n> @@ -785,6 +802,13 @@ struct packed_ref_iterator {\n>  \t/* The end of the part of the buffer that will be iterated over: */\n>  \tconst char *eof;\n>  \n> +\tstruct jump_list_entry {\n> +\t\tconst char *start;\n> +\t\tconst char *end;\n> +\t} *jump;\n> +\tsize_t jump_nr, jump_alloc;\n> +\tsize_t jump_pos;\n> \nNit: I had some trouble with `jump_pos` given that it sounds so similar\nto `iter->pos`, and thus you tend to think that they both apply to the\nposition in the packed-refs file. `jump_curr` or `jump_idx` might help\nto avoid this confusion.\n\n>  \t/* Scratch space for current values: */\n>  \tstruct object_id oid, peeled;\n>  \tstruct strbuf refname_buf;\n> @@ -802,14 +826,34 @@ struct packed_ref_iterator {\n>   */\n>  static int next_record(struct packed_ref_iterator *iter)\n>  {\n> -\tconst char *p = iter->pos, *eol;\n> +\tconst char *p, *eol;\n>  \n>  \tstrbuf_reset(&iter->refname_buf);\n>  \n> +\t/*\n> +\t * If iter->pos is contained within a skipped region, jump past\n> +\t * it.\n> +\t *\n> +\t * Note that each skipped region is considered at most once,\n> +\t * since they are ordered based on their starting position.\n> +\t */\n> +\twhile (iter->jump_pos < iter->jump_nr) {\n> +\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_pos];\n> +\t\tif (iter->pos < curr->start)\n> +\t\t\tbreak; /* not to the next jump yet */\n> +\n> +\t\titer->jump_pos++;\n> +\t\tif (iter->pos < curr->end) {\n> +\t\t\titer->pos = curr->end;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\n>  \tif (iter->pos == iter->eof)\n>  \t\treturn ITER_DONE;\n>  \n>  \titer->base.flags = REF_ISPACKED;\n> +\tp = iter->pos;\n>  \n>  \tif (iter->eof - p < the_hash_algo->hexsz + 2 ||\n>  \t    parse_oid_hex(p, &iter->oid, &p) ||\n> @@ -917,6 +961,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n>  \tint ok = ITER_DONE;\n>  \n>  \tstrbuf_release(&iter->refname_buf);\n> +\tfree(iter->jump);\n>  \trelease_snapshot(iter->snapshot);\n>  \tbase_ref_iterator_free(ref_iterator);\n>  \treturn ok;\n> @@ -928,6 +973,112 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n>  \t.abort = packed_ref_iterator_abort\n>  };\n>  \n> +static int jump_list_entry_cmp(const void *va, const void *vb)\n> +{\n> +\tconst struct jump_list_entry *a = va;\n> +\tconst struct jump_list_entry *b = vb;\n> +\n> +\tif (a->start < b->start)\n> +\t\treturn -1;\n> +\tif (a->start > b->start)\n> +\t\treturn 1;\n> +\treturn 0;\n> +}\n> +\n> +static int has_glob_special(const char *str)\n> +{\n> +\tconst char *p;\n> +\tfor (p = str; *p; p++) {\n> +\t\tif (is_glob_special(*p))\n> +\t\t\treturn 1;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static const char *ptr_max(const char *x, const char *y)\n> +{\n> +\tif (x > y)\n> +\t\treturn x;\n> +\treturn y;\n> +}\n> +\n> +static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n> +\t\t\t\t\tstruct snapshot *snapshot,\n> +\t\t\t\t\tconst char **excluded_patterns)\n> +{\n> +\tsize_t i, j;\n> +\tconst char **pattern;\n> +\tstruct jump_list_entry *last_disjoint;\n> +\n> +\tif (!excluded_patterns)\n> +\t\treturn;\n> +\n> +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> +\t\tstruct jump_list_entry *e;\n> +\n> +\t\t/*\n> +\t\t * We can't feed any excludes with globs in them to the\n> +\t\t * refs machinery.  It only understands prefix matching.\n> +\t\t * We likewise can't even feed the string leading up to\n> +\t\t * the first meta-character, as something like \"foo[a]\"\n> +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> +\t\t * would match that and mark it for exclusion).\n> +\t\t */\n> +\t\tif (has_glob_special(*pattern))\n> +\t\t\tcontinue;\n> +\n> +\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n> +\n> +\t\te = &iter->jump[iter->jump_nr++];\n> +\t\te->start = find_reference_location(snapshot, *pattern, 0);\n> +\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n\nNit: we could detect the non-matching case here already, which would\nallow us to skip an allocation. It's probably pre-mature optimization\nthough, so please feel free to ignore.\n\n> +\t}\n> +\n> +\tif (!iter->jump_nr) {\n> +\t\t/*\n> +\t\t * Every entry in exclude_patterns has a meta-character,\n> +\t\t * nothing to do here.\n> +\t\t */\n> +\t\treturn;\n> +\t}\n> +\n> +\tQSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);\n> +\n> +\t/*\n> +\t * As an optimization, merge adjacent entries in the jump list\n> +\t * to jump forwards as far as possible when entering a skipped\n> +\t * region.\n> +\t *\n> +\t * For example, if we have two skipped regions:\n> +\t *\n> +\t *\t[[A, B], [B, C]]\n> +\t *\n> +\t * we want to combine that into a single entry jumping from A to\n> +\t * C.\n> +\t */\n> +\tlast_disjoint = iter->jump;\n\nNit: if we initialized `j = 0`, then `last_disjoint` would always be\nequal to `iter->jump[j]`. We could then declare the variable inside of\nthe loop to make it a bit easier to understand.\n\nPatrick\n\n> +\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n> +\t\tstruct jump_list_entry *ours = &iter->jump[i];\n> +\n> +\t\tif (ours->start == ours->end) {\n> +\t\t\t/* ignore empty regions (no matching entries) */\n> +\t\t\tcontinue;\n> +\t\t} else if (ours->start <= last_disjoint->end) {\n> +\t\t\t/* overlapping regions extend the previous one */\n> +\t\t\tlast_disjoint->end = ptr_max(last_disjoint->end, ours->end);\n> +\t\t} else {\n> +\t\t\t/* otherwise, insert a new region */\n> +\t\t\titer->jump[j++] = *ours;\n> +\t\t\tlast_disjoint = ours;\n> +\n> +\t\t}\n> +\t}\n> +\n> +\titer->jump_nr = j;\n> +\titer->jump_pos = 0;\n> +}\n> +\n>  static struct ref_iterator *packed_ref_iterator_begin(\n>  \t\tstruct ref_store *ref_store,\n>  \t\tconst char *prefix, const char **exclude_patterns,\n> @@ -963,6 +1114,9 @@ static struct ref_iterator *packed_ref_iterator_begin(\n>  \tref_iterator = &iter->base;\n>  \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);\n>  \n> +\tif (exclude_patterns)\n> +\t\tpopulate_excluded_jump_list(iter, snapshot, exclude_patterns);\n> +\n>  \titer->snapshot = snapshot;\n>  \tacquire_snapshot(snapshot);\n>  \n> diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\n> index 6d8f844e9c..2bff003f7c 100644\n> --- a/t/helper/test-ref-store.c\n> +++ b/t/helper/test-ref-store.c\n> @@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n>  \treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n>  }\n>  \n> +static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)\n> +{\n> +\tconst char *prefix = notnull(*argv++, \"prefix\");\n> +\tconst char **exclude_patterns = argv;\n> +\n> +\treturn refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,\n> +\t\t\t\t\tNULL);\n> +}\n> +\n>  static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\n>  {\n>  \tstruct object_id oid = *null_oid();\n> @@ -307,6 +316,7 @@ static struct command commands[] = {\n>  \t{ \"delete-refs\", cmd_delete_refs },\n>  \t{ \"rename-ref\", cmd_rename_ref },\n>  \t{ \"for-each-ref\", cmd_for_each_ref },\n> +\t{ \"for-each-ref--exclude\", cmd_for_each_ref__exclude },\n>  \t{ \"resolve-ref\", cmd_resolve_ref },\n>  \t{ \"verify-ref\", cmd_verify_ref },\n>  \t{ \"for-each-reflog\", cmd_for_each_reflog },\n> diff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\n> new file mode 100755\n> index 0000000000..bc534c8ea1\n> --- /dev/null\n> +++ b/t/t1419-exclude-refs.sh\n> @@ -0,0 +1,101 @@\n> +#!/bin/sh\n> +\n> +test_description='test exclude_patterns functionality in main ref store'\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +\n> +TEST_PASSES_SANITIZE_LEAK=true\n> +. ./test-lib.sh\n> +\n> +for_each_ref__exclude () {\n> +\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n> +\tcut -d ' ' -f 2 actual.raw\n> +}\n> +\n> +for_each_ref () {\n> +\tgit for-each-ref --format='%(refname)' \"$@\"\n> +}\n> +\n> +test_expect_success 'setup' '\n> +\ttest_commit --no-tag base &&\n> +\tbase=\"$(git rev-parse HEAD)\" &&\n> +\n> +\tfor name in foo bar baz quux\n> +\tdo\n> +\t\tfor i in 1 2 3\n> +\t\tdo\n> +\t\t\techo \"create refs/heads/$name/$i $base\" || return 1\n> +\t\tdone || return 1\n> +\tdone >in &&\n> +\techo \"delete refs/heads/main\" >>in &&\n> +\n> +\tgit update-ref --stdin <in &&\n> +\tgit pack-refs --all\n> +'\n> +\n> +test_expect_success 'excluded region in middle' '\n> +\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n> +\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'excluded region at beginning' '\n> +\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n> +\tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'excluded region at end' '\n> +\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n> +\tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'disjoint excluded regions' '\n> +\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n> +\tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'adjacent, non-overlapping excluded regions' '\n> +\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n> +\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'overlapping excluded regions' '\n> +\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n> +\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'several overlapping excluded regions' '\n> +\tfor_each_ref__exclude refs/heads \\\n> +\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n> +\tfor_each_ref refs/heads/quux >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'non-matching excluded section' '\n> +\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n> +\tfor_each_ref >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'meta-characters are discarded' '\n> +\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n> +\tfor_each_ref >expect &&\n> +\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_done\n> -- \n> 2.40.1.572.g5c4ab523ef\n> \n"},{"id":"478086","messageId":"ZH7ZsIq5vQIstjAG@ncase","threadId":"59715","inReplyTo":"50e7df7dc0932a53682664a6334791d8f1ff31bb.1684178576.git.me@ttaylorr.com","subject":"Re: [PATCH v2 13/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-06-06T07:01:04Z","receivedAt":"2023-06-06T07:01:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 15, 2023 at 03:23:45PM -0400, Taylor Blau wrote:\n> The following commit will want to call `for_each_namespaced_ref()` with\n> a list of excluded patterns.\n\nThis statement isn't quite true as the following commit touches\ngit-receive-pack(1), which doesn't use `for_each_namespaced_ref_()`.\n\nPatrick\n"},{"id":"478087","messageId":"ZH7ZtmUrlyJ8H0aP@ncase","threadId":"59715","inReplyTo":"cover.1684178576.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/16] refs: implement jump lists for packed backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-06-06T07:01:10Z","receivedAt":"2023-06-06T07:01:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 15, 2023 at 03:23:07PM -0400, Taylor Blau wrote:\n> Here is a reroll of my series to implement jump (née skip) lists for the\n> packed refs backend.\n> \n> Not a ton has changed since last time, but some notable things that have\n> changed include:\n> \n>   - Renaming \"skip lists\" to \"jump lists\" to clarify that this\n>     implementation does not use the skip list data structure.\n>   - Patch reorganization, splitting out `find_reference_location_end()`\n>     more sensibly, rewording patch messages, etc.\n>   - Addresses feedback from Junio and Patrick Steinhardt's helpful\n>     reviews.\n> \n> As usual, a range-diff is included below for convenience.\n> \n> Given that we are expecting -rc0 today, we should aim to not let review\n> of this topic direct our attention away from testing the release\n> candidates. We can get more serious about it on the other side of 2.41.\n> \n> Thanks in advance for another look.\n\nI didn't have many comments in this round. Personally though I'd split\nup this patch series into two in order to land the individual parts\nfaster, where the first part introduces `git for-each-ref --exclude` and\nthe second part introduces the jump list for the packed-refs backend.\n\nEach of these have merit on their own, and especially the first part\nshould require less discussion. Furthermore, by splitting it up the\nreview becomes easier to manage as 16 patches does require quite a long\nattention span to handle.\n\nAnyway, this is just a suggestion from my sided, please feel free to\nignore.\n\nPatrick\n"},{"id":"478124","messageId":"cover.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH v3 00/16] refs: implement jump lists for packed backend","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:40:47Z","receivedAt":"2023-06-07T10:40:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a reroll of my series to implement jump (née skip) lists for the\npacked refs backend, based on top of the current 'master'.\n\nNothing substantive has changed since the last version, where review had\nstabilized. This version just resolves a couple of merge conflicts with\n633390bd08 (Merge branch 'bc/clone-empty-repo-via-protocol-v0',\n2023-05-19).\n\nAs usual, a range-diff is included below for convenience.\n\nThanks in advance for a hopefully final look ;-).\n\nJeff King (5):\n  refs.c: rename `ref_filter`\n  ref-filter.h: provide `REF_FILTER_INIT`\n  ref-filter: clear reachable list pointers after freeing\n  ref-filter: add `ref_filter_clear()`\n  ref-filter.c: parameterize match functions over patterns\n\nTaylor Blau (11):\n  builtin/for-each-ref.c: add `--exclude` option\n  refs: plumb `exclude_patterns` argument throughout\n  refs/packed-backend.c: refactor `find_reference_location()`\n  refs/packed-backend.c: implement jump lists to avoid excluded\n    pattern(s)\n  refs/packed-backend.c: add trace2 counters for jump list\n  revision.h: store hidden refs in a `strvec`\n  refs/packed-backend.c: ignore complicated hidden refs rules\n  refs.h: let `for_each_namespaced_ref()` take excluded patterns\n  builtin/receive-pack.c: avoid enumerating hidden references\n  upload-pack.c: avoid enumerating hidden refs where possible\n  ls-refs.c: avoid enumerating hidden refs where possible\n\n Documentation/git-for-each-ref.txt |   6 +\n builtin/branch.c                   |   4 +-\n builtin/for-each-ref.c             |   7 +-\n builtin/receive-pack.c             |   7 +-\n builtin/tag.c                      |   4 +-\n http-backend.c                     |   2 +-\n ls-refs.c                          |   8 +-\n ref-filter.c                       |  63 ++++++--\n ref-filter.h                       |  12 ++\n refs.c                             |  61 ++++----\n refs.h                             |  15 +-\n refs/debug.c                       |   5 +-\n refs/files-backend.c               |   5 +-\n refs/packed-backend.c              | 226 ++++++++++++++++++++++++++---\n refs/refs-internal.h               |   7 +-\n revision.c                         |   4 +-\n revision.h                         |   5 +-\n t/helper/test-reach.c              |   2 +-\n t/helper/test-ref-store.c          |  10 ++\n t/t0041-usage.sh                   |   1 +\n t/t1419-exclude-refs.sh            | 131 +++++++++++++++++\n t/t3402-rebase-merge.sh            |   1 +\n t/t6300-for-each-ref.sh            |  35 +++++\n trace2.h                           |   2 +\n trace2/tr2_ctr.c                   |   5 +\n upload-pack.c                      |  43 ++++--\n 26 files changed, 565 insertions(+), 106 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\nRange-diff against v2:\n 1:  6cac42e70e =  1:  afac948f04 refs.c: rename `ref_filter`\n 2:  8dda7db738 =  2:  b9336e3b77 ref-filter.h: provide `REF_FILTER_INIT`\n 3:  bf21df783d =  3:  fc28b5caaa ref-filter: clear reachable list pointers after freeing\n 4:  85ecb70957 =  4:  bc5356fe4b ref-filter: add `ref_filter_clear()`\n 5:  385890b459 =  5:  1988ca4c0a ref-filter.c: parameterize match functions over patterns\n 6:  1a3371a0a7 =  6:  60d85aa4ad builtin/for-each-ref.c: add `--exclude` option\n 7:  aa05549b6e =  7:  c4fe9a1893 refs: plumb `exclude_patterns` argument throughout\n 8:  6002c568b5 =  8:  9cab5e0699 refs/packed-backend.c: refactor `find_reference_location()`\n 9:  8c78f49a8d =  9:  8066858bf5 refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)\n10:  5059f5dd42 = 10:  3c045076a9 refs/packed-backend.c: add trace2 counters for jump list\n11:  f765b50a84 = 11:  0ff542eaef revision.h: store hidden refs in a `strvec`\n12:  254bcc4361 = 12:  ca006b2c3f refs/packed-backend.c: ignore complicated hidden refs rules\n13:  50e7df7dc0 ! 13:  cae703a425 refs.h: let `for_each_namespaced_ref()` take excluded patterns\n    @@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rp\n      \t\thead_ref_namespaced(send_ref, &data);\n     -\t\tfor_each_namespaced_ref(send_ref, &data);\n     +\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n    - \t\t/*\n    - \t\t * fflush stdout before calling advertise_shallow_grafts because send_ref\n    - \t\t * uses stdio.\n    + \t\tif (!data.sent_capabilities) {\n    + \t\t\tconst char *refname = \"capabilities^{}\";\n    + \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n     @@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rpc,\n      \t\tpacket_flush(1);\n      \t} else {\n14:  f6a3a5a6ba = 14:  1db10b76ea builtin/receive-pack.c: avoid enumerating hidden references\n15:  2331fa7a4d ! 15:  014243ebe6 upload-pack.c: avoid enumerating hidden refs where possible\n    @@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rp\n      \t\thead_ref_namespaced(send_ref, &data);\n     -\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n     +\t\tfor_each_namespaced_ref_1(send_ref, &data);\n    - \t\t/*\n    - \t\t * fflush stdout before calling advertise_shallow_grafts because send_ref\n    - \t\t * uses stdio.\n    + \t\tif (!data.sent_capabilities) {\n    + \t\t\tconst char *refname = \"capabilities^{}\";\n    + \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n     @@ upload-pack.c: void upload_pack(const int advertise_refs, const int stateless_rpc,\n      \t\tpacket_flush(1);\n      \t} else {\n16:  2c6b89d64a = 16:  e02fe93379 ls-refs.c: avoid enumerating hidden refs where possible\n-- \n2.41.0.16.g26cd413590\n"},{"id":"478125","messageId":"afac948f04dca5cd532ceea08fbc646184336311.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 01/16] refs.c: rename `ref_filter`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:40:53Z","receivedAt":"2023-06-07T10:41:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nThe refs machinery has its own implementation of a `ref_filter` (used by\n`for-each-ref`), which is distinct from the `ref-filler.h` API (also\nused by `for-each-ref`, among other things).\n\nRename the one within refs.c to more clearly indicate its purpose.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex d2a98e1c21..b9b77d2eff 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,\n \t\t\t\t   oid, flags);\n }\n \n-/* The argument to filter_refs */\n-struct ref_filter {\n+/* The argument to for_each_filter_refs */\n+struct for_each_ref_filter {\n \tconst char *pattern;\n \tconst char *prefix;\n \teach_ref_fn *fn;\n@@ -409,10 +409,11 @@ int ref_exists(const char *refname)\n \treturn refs_ref_exists(get_main_ref_store(the_repository), refname);\n }\n \n-static int filter_refs(const char *refname, const struct object_id *oid,\n-\t\t\t   int flags, void *data)\n+static int for_each_filter_refs(const char *refname,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tint flags, void *data)\n {\n-\tstruct ref_filter *filter = (struct ref_filter *)data;\n+\tstruct for_each_ref_filter *filter = data;\n \n \tif (wildmatch(filter->pattern, refname, 0))\n \t\treturn 0;\n@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tconst char *prefix, void *cb_data)\n {\n \tstruct strbuf real_pattern = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct for_each_ref_filter filter;\n \tint ret;\n \n \tif (!prefix && !starts_with(pattern, \"refs/\"))\n@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tfilter.prefix = prefix;\n \tfilter.fn = fn;\n \tfilter.cb_data = cb_data;\n-\tret = for_each_ref(filter_refs, &filter);\n+\tret = for_each_ref(for_each_filter_refs, &filter);\n \n \tstrbuf_release(&real_pattern);\n \treturn ret;\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478126","messageId":"b9336e3b778bff1bc944de3b713c7adf4aabf020.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 02/16] ref-filter.h: provide `REF_FILTER_INIT`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:40:58Z","receivedAt":"2023-06-07T10:41:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nProvide a sane initialization value for `struct ref_filter`, which in a\nsubsequent patch will be used to initialize a new field.\n\nIn the meantime, fix a case in test-reach.c where its `ref_filter` is\nnot even zero-initialized.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c       | 3 +--\n builtin/for-each-ref.c | 3 +--\n builtin/tag.c          | 3 +--\n ref-filter.h           | 3 +++\n t/helper/test-reach.c  | 2 +-\n 5 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex e6c2655af6..7891dec361 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -707,7 +707,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \tint reflog = 0, quiet = 0, icase = 0, force = 0,\n \t    recurse_submodules_explicit = 0;\n \tenum branch_track track;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstatic struct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -765,7 +765,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \n \tsetup_ref_filter_porcelain_msg();\n \n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.kind = FILTER_REFS_BRANCHES;\n \tfilter.abbrev = -1;\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 695fc8f4a5..99ccb73518 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tint maxcount = 0, icase = 0, omit_empty = 0;\n \tstruct ref_array array;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t};\n \n \tmemset(&array, 0, sizeof(array));\n-\tmemset(&filter, 0, sizeof(filter));\n \n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 1850a6a6fd..6b41bb7374 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -443,7 +443,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tstruct msg_arg msg = { .buf = STRBUF_INIT };\n \tstruct ref_transaction *transaction;\n \tstruct strbuf err = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -501,7 +501,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tgit_config(git_tag_config, &sorting_options);\n \n \tmemset(&opt, 0, sizeof(opt));\n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.lines = -1;\n \topt.sign = -1;\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 430701cfb7..a920f73b29 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,9 @@ struct ref_format {\n \tstruct string_list bases;\n };\n \n+#define REF_FILTER_INIT { \\\n+\t.points_at = OID_ARRAY_INIT, \\\n+}\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n \t.bases = STRING_LIST_INIT_DUP, \\\ndiff --git a/t/helper/test-reach.c b/t/helper/test-reach.c\nindex 5b6f217441..ef58f10c2d 100644\n--- a/t/helper/test-reach.c\n+++ b/t/helper/test-reach.c\n@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)\n \n \t\tprintf(\"%s(X,_,_,0,0):%d\\n\", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));\n \t} else if (!strcmp(av[1], \"commit_contains\")) {\n-\t\tstruct ref_filter filter;\n+\t\tstruct ref_filter filter = REF_FILTER_INIT;\n \t\tstruct contains_cache cache;\n \t\tinit_contains_cache(&cache);\n \n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478127","messageId":"fc28b5caaae2d72f147329a84e16aae416857565.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 03/16] ref-filter: clear reachable list pointers after freeing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:03Z","receivedAt":"2023-06-07T10:41:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nIn reach_filter(), we pop all commits from the reachable lists, leaving\nthem empty. But because we're operating on a list pointer that was\npassed by value, the original filter.reachable_from pointer is left\ndangling.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4991cd4f7a..048d277cbf 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2418,13 +2418,13 @@ void ref_array_clear(struct ref_array *array)\n #define EXCLUDE_REACHED 0\n #define INCLUDE_REACHED 1\n static void reach_filter(struct ref_array *array,\n-\t\t\t struct commit_list *check_reachable,\n+\t\t\t struct commit_list **check_reachable,\n \t\t\t int include_reached)\n {\n \tint i, old_nr;\n \tstruct commit **to_clear;\n \n-\tif (!check_reachable)\n+\tif (!*check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n@@ -2434,7 +2434,7 @@ static void reach_filter(struct ref_array *array,\n \t}\n \n \ttips_reachable_from_bases(the_repository,\n-\t\t\t\t  check_reachable,\n+\t\t\t\t  *check_reachable,\n \t\t\t\t  to_clear, array->nr,\n \t\t\t\t  UNINTERESTING);\n \n@@ -2455,8 +2455,8 @@ static void reach_filter(struct ref_array *array,\n \n \tclear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);\n \n-\twhile (check_reachable) {\n-\t\tstruct commit *merge_commit = pop_commit(&check_reachable);\n+\twhile (*check_reachable) {\n+\t\tstruct commit *merge_commit = pop_commit(check_reachable);\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n@@ -2553,8 +2553,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \tclear_contains_cache(&ref_cbdata.no_contains_cache);\n \n \t/*  Filters that need revision walking */\n-\treach_filter(array, filter->reachable_from, INCLUDE_REACHED);\n-\treach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);\n+\treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n+\treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n \n \tsave_commit_buffer = save_commit_buffer_orig;\n \treturn ret;\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478128","messageId":"bc5356fe4b368ddd6e100341707ddcc75979af75.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 04/16] ref-filter: add `ref_filter_clear()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:08Z","receivedAt":"2023-06-07T10:41:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nWe did not bother to clean up at all in `git branch` or `git tag`, and\n`git for-each-ref` only cleans up a couple of members.\n\nAdd and call `ref_filter_clear()` when cleaning up a `struct\nref_filter`. Running this patch (without any test changes) indicates a\ncouple of now leak-free tests. This was found by running:\n\n    $ make SANITIZE=leak\n    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate\n\n(Note that the `reachable_from` and `unreachable_from` lists should be\ncleaned as they are used. So this is just covering any case where we\nmight bail before running the reachability check.)\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c        |  1 +\n builtin/for-each-ref.c  |  3 +--\n builtin/tag.c           |  1 +\n ref-filter.c            | 16 ++++++++++++++++\n ref-filter.h            |  3 +++\n t/t0041-usage.sh        |  1 +\n t/t3402-rebase-merge.sh |  1 +\n 7 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 7891dec361..07ee874617 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -858,6 +858,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\tprint_columns(&output, colopts, NULL);\n \t\tstring_list_clear(&output, 0);\n \t\tref_sorting_release(sorting);\n+\t\tref_filter_clear(&filter);\n \t\treturn 0;\n \t} else if (edit_description) {\n \t\tconst char *branch_name;\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 99ccb73518..c01fa6fefe 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&err);\n \tstrbuf_release(&output);\n \tref_array_clear(&array);\n-\tfree_commit_list(filter.with_commit);\n-\tfree_commit_list(filter.no_commit);\n+\tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\n \tstrvec_clear(&vec);\n \treturn 0;\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 6b41bb7374..aab5e693fe 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -645,6 +645,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \n cleanup:\n \tref_sorting_release(sorting);\n+\tref_filter_clear(&filter);\n \tstrbuf_release(&buf);\n \tstrbuf_release(&ref);\n \tstrbuf_release(&reflog_msg);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 048d277cbf..d32f426898 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2866,3 +2866,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)\n \n \treturn 0;\n }\n+\n+void ref_filter_init(struct ref_filter *filter)\n+{\n+\tstruct ref_filter blank = REF_FILTER_INIT;\n+\tmemcpy(filter, &blank, sizeof(blank));\n+}\n+\n+void ref_filter_clear(struct ref_filter *filter)\n+{\n+\toid_array_clear(&filter->points_at);\n+\tfree_commit_list(filter->with_commit);\n+\tfree_commit_list(filter->no_commit);\n+\tfree_commit_list(filter->reachable_from);\n+\tfree_commit_list(filter->unreachable_from);\n+\tref_filter_init(filter);\n+}\ndiff --git a/ref-filter.h b/ref-filter.h\nindex a920f73b29..160b807224 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,\n \t\t\t struct ref_format *format,\n \t\t\t struct ref_array *array);\n \n+void ref_filter_init(struct ref_filter *filter);\n+void ref_filter_clear(struct ref_filter *filter);\n+\n #endif /*  REF_FILTER_H  */\ndiff --git a/t/t0041-usage.sh b/t/t0041-usage.sh\nindex c4fc34eb18..9ea974b0c6 100755\n--- a/t/t0041-usage.sh\n+++ b/t/t0041-usage.sh\n@@ -5,6 +5,7 @@ test_description='Test commands behavior when given invalid argument value'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'setup ' '\ndiff --git a/t/t3402-rebase-merge.sh b/t/t3402-rebase-merge.sh\nindex 79b0640c00..e9e03ca4b5 100755\n--- a/t/t3402-rebase-merge.sh\n+++ b/t/t3402-rebase-merge.sh\n@@ -8,6 +8,7 @@ test_description='git rebase --merge test'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n T=\"A quick brown fox\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478129","messageId":"1988ca4c0a79065a1893259da8abe63919e05cf3.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:13Z","receivedAt":"2023-06-07T10:41:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\n`match_pattern()` and `match_name_as_path()` both take a `struct\nref_filter *`, and then store a stack variable `patterns` pointing at\n`filter->patterns`.\n\nThe subsequent patch will add a new array of patterns to match over (the\nexcluded patterns, via a new `git for-each-ref --exclude` option),\ntreating the return value of these functions differently depending on\nwhich patterns are being used to match.\n\nTweak `match_pattern()` and `match_name_as_path()` to take an array of\npatterns to prepare for passing either in.\n\nOnce we start passing either in, `match_pattern()` will have little to\ndo with a particular `struct ref_filter *` instance. To clarify this,\ndrop it from the argument list, and replace it with the only bit of the\n`ref_filter` that we care about (`filter->ignore_case`).\n\nCo-authored-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 16 +++++++++-------\n 1 file changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex d32f426898..6d91c7cb0d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2104,12 +2104,12 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n  * matches a pattern \"refs/heads/mas\") or a wildcard (e.g. the same ref\n  * matches \"refs/heads/mas*\", too).\n  */\n-static int match_pattern(const struct ref_filter *filter, const char *refname)\n+static int match_pattern(const char **patterns, const char *refname,\n+\t\t\t const int ignore_case)\n {\n-\tconst char **patterns = filter->name_patterns;\n \tunsigned flags = 0;\n \n-\tif (filter->ignore_case)\n+\tif (ignore_case)\n \t\tflags |= WM_CASEFOLD;\n \n \t/*\n@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n  * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n  * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n  */\n-static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n+static int match_name_as_path(const struct ref_filter *filter,\n+\t\t\t      const char **pattern,\n+\t\t\t      const char *refname)\n {\n-\tconst char **pattern = filter->name_patterns;\n \tint namelen = strlen(refname);\n \tunsigned flags = WM_PATHNAME;\n \n@@ -2165,8 +2166,9 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \tif (!*filter->name_patterns)\n \t\treturn 1; /* No pattern always matches */\n \tif (filter->match_as_path)\n-\t\treturn match_name_as_path(filter, refname);\n-\treturn match_pattern(filter, refname);\n+\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n+\treturn match_pattern(filter->name_patterns, refname,\n+\t\t\t     filter->ignore_case);\n }\n \n /*\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478130","messageId":"60d85aa4ad10cf57c7694e6a07d6bf2d8f6984eb.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 06/16] builtin/for-each-ref.c: add `--exclude` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:18Z","receivedAt":"2023-06-07T10:41:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When using `for-each-ref`, it is sometimes convenient for the caller to\nbe able to exclude certain parts of the references.\n\nFor example, if there are many `refs/__hidden__/*` references, the\ncaller may want to emit all references *except* the hidden ones.\nCurrently, the only way to do this is to post-process the output, like:\n\n    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'\n\nWhich is do-able, but requires processing a potentially large quantity\nof references.\n\nTeach `git for-each-ref` a new `--exclude=<pattern>` option, which\nexcludes references from the results if they match one or more excluded\npatterns.\n\nThis patch provides a naive implementation where the `ref_filter` still\nsees all references (including ones that it will discard) and is left to\ncheck whether each reference matches any excluded pattern(s) before\nemitting them.\n\nBy culling out references we know the caller doesn't care about, we can\navoid allocating memory for their storage, as well as spending time\nsorting the output (among other things). Even the naive implementation\nprovides a significant speed-up on a modified copy of linux.git (that\nhas a hidden ref pointing at each commit):\n\n    $ hyperfine \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/'\n    Benchmark 1: git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]\n      Range (min … max):   817.7 ms … 823.3 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/\n      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]\n      Range (min … max):   104.7 ms … 109.1 ms    27 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/' ran\n        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"'\n\nSubsequent patches will improve on this by avoiding visiting excluded\nsections of the `packed-refs` file in certain cases.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-for-each-ref.txt |  6 +++++\n builtin/for-each-ref.c             |  1 +\n ref-filter.c                       | 13 +++++++++++\n ref-filter.h                       |  6 +++++\n t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++\n 5 files changed, 61 insertions(+)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 1e215d4e73..5743eb5def 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t\t   [--points-at=<object>]\n \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n+\t\t   [--exclude=<pattern> ...]\n \n DESCRIPTION\n -----------\n@@ -102,6 +103,11 @@ OPTIONS\n \tDo not print a newline after formatted refs where the format expands\n \tto the empty string.\n \n+--exclude=<pattern>::\n+\tIf one or more patterns are given, only refs which do not match\n+\tany excluded pattern(s) are shown. Matching is done using the\n+\tsame rules as `<pattern>` above.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex c01fa6fefe..3384987428 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -47,6 +47,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n+\t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n \t\t\t     N_(\"object\"), N_(\"print only refs which points at the given object\"),\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 6d91c7cb0d..d44418efb7 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2171,6 +2171,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \t\t\t     filter->ignore_case);\n }\n \n+static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n+{\n+\tif (!filter->exclude.nr)\n+\t\treturn 0;\n+\tif (filter->match_as_path)\n+\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n+\treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n+}\n+\n /*\n  * This is the same as for_each_fullref_in(), but it tries to iterate\n  * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2338,6 +2347,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \tif (!filter_pattern_match(filter, refname))\n \t\treturn 0;\n \n+\tif (filter_exclude_match(filter, refname))\n+\t\treturn 0;\n+\n \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n \t\treturn 0;\n \n@@ -2877,6 +2889,7 @@ void ref_filter_init(struct ref_filter *filter)\n \n void ref_filter_clear(struct ref_filter *filter)\n {\n+\tstrvec_clear(&filter->exclude);\n \toid_array_clear(&filter->points_at);\n \tfree_commit_list(filter->with_commit);\n \tfree_commit_list(filter->no_commit);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 160b807224..1524bc463a 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -6,6 +6,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -59,6 +60,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\n \tstruct commit_list *no_commit;\n@@ -94,6 +96,7 @@ struct ref_format {\n \n #define REF_FILTER_INIT { \\\n \t.points_at = OID_ARRAY_INIT, \\\n+\t.exclude = STRVEC_INIT, \\\n }\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n@@ -112,6 +115,9 @@ struct ref_format {\n #define OPT_REF_SORT(var) \\\n \tOPT_STRING_LIST(0, \"sort\", (var), \\\n \t\t\tN_(\"key\"), N_(\"field name to sort on\"))\n+#define OPT_REF_FILTER_EXCLUDE(var) \\\n+\tOPT_STRVEC(0, \"exclude\", &(var)->exclude, \\\n+\t\t   N_(\"pattern\"), N_(\"exclude refs which match pattern\"))\n \n /*\n  * API for filtering a set of refs. Based on the type of refs the user\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 5c00607608..7e8d578522 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '\n \ttest_cmp expected actual\n '\n \n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with prefix exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=refs/tags/foo >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/foo/one\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with pattern exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=\"refs/tags/foo/t*\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n cat >expected <<\\EOF\n 'refs/heads/main'\n 'refs/remotes/origin/main'\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478131","messageId":"c4fe9a1893c15c5aae4d3f305b248d0c3ac55222.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 07/16] refs: plumb `exclude_patterns` argument throughout","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:23Z","receivedAt":"2023-06-07T10:41:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The subsequent patch will want to access an optional `excluded_patterns`\narray within `refs/packed-backend.c` that will cull out certain\nreferences matching any of the given patterns on a best-effort basis.\n\nTo do so, the refs subsystem needs to be updated to pass this value\nacross a number of different locations.\n\nPrepare for a future patch by introducing this plumbing now, passing\nNULLs at top-level APIs in order to make that patch less noisy and more\neasily readable.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.co>\n---\n ls-refs.c             |  2 +-\n ref-filter.c          |  5 +++--\n refs.c                | 32 +++++++++++++++++++-------------\n refs.h                |  8 +++++++-\n refs/debug.c          |  5 +++--\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  5 +++--\n refs/refs-internal.h  |  7 ++++---\n revision.c            |  2 +-\n 9 files changed, 44 insertions(+), 27 deletions(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex f385938b64..6f490b2d9c 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  send_ref, &data);\n+\t\t\t\t\t  NULL, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex d44418efb7..717c3c4bcf 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n+\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t cb, cb_data);\n+\t\t\t\t\t\t NULL, cb, cb_data);\n }\n \n /*\ndiff --git a/refs.c b/refs.c\nindex b9b77d2eff..538bde644e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)\n \n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n+\t\tconst char *prefix,\n+\t\tconst char **exclude_patterns,\n+\t\tint trim,\n \t\tenum do_for_each_ref_flags flags)\n {\n \tstruct ref_iterator *iter;\n@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(\n \t\t}\n \t}\n \n-\titer = refs->be->iterator_begin(refs, prefix, flags);\n-\n+\titer = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);\n \t/*\n \t * `iterator_begin()` already takes care of prefix, but we\n \t * might need to do some trimming:\n@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);\n \n \treturn do_for_each_repo_ref_iterator(r, iter, fn, cb_data);\n }\n@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,\n }\n \n static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n+\t\t\t   const char **exclude_patterns,\n \t\t\t   each_ref_fn fn, int trim,\n \t\t\t   enum do_for_each_ref_flags flags, void *cb_data)\n {\n@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,\n+\t\t\t\t       flags);\n \n \treturn do_for_each_repo_ref_iterator(the_repository, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \n int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t       prefix, fn, 0, 0, cb_data);\n+\t\t\t       prefix, NULL, fn, 0, 0, cb_data);\n }\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);\n }\n \n int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\n \n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0,\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n \t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,\n int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \t\t\t\t      const char *namespace,\n \t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data)\n {\n \tstruct string_list prefixes = STRING_LIST_INIT_DUP;\n@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \n \tfor_each_string_list_item(prefix, &prefixes) {\n \t\tstrbuf_addstr(&buf, prefix->string);\n-\t\tret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);\n+\t\tret = refs_for_each_fullref_in(ref_store, buf.buf,\n+\t\t\t\t\t       exclude_patterns, fn, cb_data);\n \t\tif (ret)\n \t\t\tbreak;\n \t\tstrbuf_setlen(&buf, namespace_len);\n@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,\n \tstrbuf_addstr(&dirname, refname + dirname.len);\n \tstrbuf_addch(&dirname, '/');\n \n-\titer = refs_ref_iterator_begin(refs, dirname.buf, 0,\n+\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 123cfa4424..d672d636cf 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -338,6 +338,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);\n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data);\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n@@ -345,10 +346,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n  * iterate all refs in \"patterns\" by partitioning patterns into disjoint sets\n  * and iterating the longest-common prefix of each set.\n  *\n+ * references matching any pattern in \"exclude_patterns\" are omitted from the\n+ * result set on a best-effort basis.\n+ *\n  * callers should be prepared to ignore references that they did not ask for.\n  */\n int refs_for_each_fullref_in_prefixes(struct ref_store *refs,\n-\t\t\t\t      const char *namespace, const char **patterns,\n+\t\t\t\t      const char *namespace,\n+\t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data);\n \n /**\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 6f11e6de46..328f894177 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -229,11 +229,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \n static struct ref_iterator *\n debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n-\t\t\t unsigned int flags)\n+\t\t\t const char **exclude_patterns, unsigned int flags)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n \tstruct ref_iterator *res =\n-\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix, flags);\n+\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix,\n+\t\t\t\t\t\texclude_patterns, flags);\n \tstruct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));\n \tbase_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);\n \tditer->iter = res;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bca7b851c5..3bc3c57c05 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -829,7 +829,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \n static struct ref_iterator *files_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct files_ref_store *refs;\n \tstruct ref_iterator *loose_iter, *packed_iter, *overlay_iter;\n@@ -874,7 +875,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * the packed and loose references.\n \t */\n \tpacked_iter = refs_ref_iterator_begin(\n-\t\t\trefs->packed_ref_store, prefix, 0,\n+\t\t\trefs->packed_ref_store, prefix, exclude_patterns, 0,\n \t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n \n \toverlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 5b412a133b..176bd3905b 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -924,7 +924,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n@@ -1149,7 +1150,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t * list of refs is exhausted, set iter to NULL. When the list\n \t * of updates is exhausted, leave i set to updates->nr.\n \t */\n-\titer = packed_ref_iterator_begin(&refs->base, \"\",\n+\titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n \tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n \t\titer = NULL;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a85d113123..28a11b9d61 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n  */\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n-\t\tenum do_for_each_ref_flags flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tint trim, enum do_for_each_ref_flags flags);\n \n /*\n  * A callback function used to instruct merge_ref_iterator how to\n@@ -570,7 +570,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,\n  */\n typedef struct ref_iterator *ref_iterator_begin_fn(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags);\n \n /* reflog functions */\n \ndiff --git a/revision.c b/revision.c\nindex b33cc1d106..89953592f9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,\n \tstruct strbuf bisect_refs = STRBUF_INIT;\n \tint status;\n \tstrbuf_addf(&bisect_refs, \"refs/bisect/%s\", term);\n-\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);\n+\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);\n \tstrbuf_release(&bisect_refs);\n \treturn status;\n }\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478132","messageId":"9cab5e06996b370e106c965b0e6b92384a744f31.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 08/16] refs/packed-backend.c: refactor `find_reference_location()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:28Z","receivedAt":"2023-06-07T10:41:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `find_reference_location()` is used to perform a\nbinary search-like function over the contents of a repository's\n`$GIT_DIR/packed-refs` file.\n\nThe search it implements is unlike a standard binary search in that the\nrecords it searches over are not of a fixed width, so the comparison\nmust locate the end of a record before comparing it.\n\nExtract the core routine of `find_reference_location()` in order to\nimplement a function in the following patch which will find the first\nlocation in the `packed-refs` file that *doesn't* match the given\npattern.\n\nThe behavior of `find_reference_location()` is unchanged.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c | 38 ++++++++++++++++++++++----------------\n 1 file changed, 22 insertions(+), 16 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 176bd3905b..33639f73e1 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -527,22 +527,8 @@ static int load_contents(struct snapshot *snapshot)\n \treturn 1;\n }\n \n-/*\n- * Find the place in `snapshot->buf` where the start of the record for\n- * `refname` starts. If `mustexist` is true and the reference doesn't\n- * exist, then return NULL. If `mustexist` is false and the reference\n- * doesn't exist, then return the point where that reference would be\n- * inserted, or `snapshot->eof` (which might be NULL) if it would be\n- * inserted at the end of the file. In the latter mode, `refname`\n- * doesn't have to be a proper reference name; for example, one could\n- * search for \"refs/replace/\" to find the start of any replace\n- * references.\n- *\n- * The record is sought using a binary search, so `snapshot->buf` must\n- * be sorted.\n- */\n-static const char *find_reference_location(struct snapshot *snapshot,\n-\t\t\t\t\t   const char *refname, int mustexist)\n+static const char *find_reference_location_1(struct snapshot *snapshot,\n+\t\t\t\t\t     const char *refname, int mustexist)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -588,6 +574,26 @@ static const char *find_reference_location(struct snapshot *snapshot,\n \t\treturn lo;\n }\n \n+/*\n+ * Find the place in `snapshot->buf` where the start of the record for\n+ * `refname` starts. If `mustexist` is true and the reference doesn't\n+ * exist, then return NULL. If `mustexist` is false and the reference\n+ * doesn't exist, then return the point where that reference would be\n+ * inserted, or `snapshot->eof` (which might be NULL) if it would be\n+ * inserted at the end of the file. In the latter mode, `refname`\n+ * doesn't have to be a proper reference name; for example, one could\n+ * search for \"refs/replace/\" to find the start of any replace\n+ * references.\n+ *\n+ * The record is sought using a binary search, so `snapshot->buf` must\n+ * be sorted.\n+ */\n+static const char *find_reference_location(struct snapshot *snapshot,\n+\t\t\t\t\t   const char *refname, int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist);\n+}\n+\n /*\n  * Create a newly-allocated `snapshot` of the `packed-refs` file in\n  * its current state and return it. The return value will already have\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478133","messageId":"8066858bf59386eeec68f0f1e4247eeebb6482f7.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:33Z","receivedAt":"2023-06-07T10:41:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When iterating through the `packed-refs` file in order to answer a query\nlike:\n\n    $ git for-each-ref --exclude=refs/__hidden__\n\nit would be useful to avoid walking over all of the entries in\n`refs/__hidden__/*` when possible, since we know that the ref-filter\ncode is going to throw them away anyways.\n\nIn certain circumstances, doing so is possible. The algorithm for doing\nso is as follows:\n\n  - For each excluded pattern, find the first record that matches it,\n    and the first record that *doesn't* match it (i.e. the location\n    you'd next want to consider when excluding that pattern).\n\n  - Sort the set of excluded regions from the previous step in ascending\n    order of the first location within the `packed-refs` file that\n    matches.\n\n  - Clean up the results from the previous step: discard empty regions,\n    and combine adjacent regions.\n\nThen when iterating through the `packed-refs` file, if `iter->pos` is\never contained in one of the regions from the previous steps, advance\n`iter->pos` past the end of that region, and continue enumeration.\n\nNote that we only perform this optimization when none of the excluded\npattern(s) have special meta-characters in them. For a pattern like\n\"refs/foo[ac]\", the excluded regions (\"refs/fooa\", \"refs/fooc\", and\neverything underneath them) are not connected. A future implementation\nthat handles this case may split the character class (pretending as if\ntwo patterns were excluded: \"refs/fooa\", and \"refs/fooc\").\n\nThere are a few other gotchas worth considering. First, note that the\njump list is sorted, so once we jump past a region, we can avoid\nconsidering it (or any regions preceding it) again. The member\n`jump_pos` is used to track the first next-possible region to jump\nthrough.\n\nSecond, note that the exclusion list is best-effort, since we do not\nhandle loose references, and because of the meta-character issue above.\n\nIn repositories with a large number of hidden references, the speed-up\ncan be significant. Tests here are done with a copy of linux.git with a\nreference \"refs/pull/N\" pointing at every commit, as in:\n\n    $ git rev-list HEAD | awk '{ print \"create refs/pull/\" NR \" \" $0 }' |\n        git update-ref --stdin\n    $ git pack-refs --all\n\n, it is significantly faster to have `for-each-ref` jump over the\nexcluded references, as opposed to filtering them out after the fact:\n\n    $ hyperfine \\\n      'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n    Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n      Range (min … max):   800.0 ms … 807.7 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n      Range (min … max):     4.3 ms …   6.7 ms    422 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n      172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n\nUsing the jump list is fairly straightforward (see the changes to\n`refs/packed-backend.c::next_record()`), but constructing the list is\nnot. To ensure that the construction is correct, add a new suite of\ntests in t1419 covering various corner cases (overlapping regions,\npartially overlapping regions, adjacent regions, etc.).\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c              |   5 +-\n refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--\n t/helper/test-ref-store.c |  10 +++\n t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++\n 4 files changed, 274 insertions(+), 8 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 717c3c4bcf..ddc7f5204f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2210,12 +2210,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n \t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n+\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t NULL, cb, cb_data);\n+\t\t\t\t\t\t filter->exclude.v,\n+\t\t\t\t\t\t cb, cb_data);\n }\n \n /*\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 33639f73e1..67327e579c 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -303,7 +303,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)\n  * Compare a snapshot record at `rec` to the specified NUL-terminated\n  * refname.\n  */\n-static int cmp_record_to_refname(const char *rec, const char *refname)\n+static int cmp_record_to_refname(const char *rec, const char *refname,\n+\t\t\t\t int start)\n {\n \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n \tconst char *r2 = refname;\n@@ -312,7 +313,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)\n \t\tif (*r1 == '\\n')\n \t\t\treturn *r2 ? -1 : 0;\n \t\tif (!*r2)\n-\t\t\treturn 1;\n+\t\t\treturn start ? 1 : -1;\n \t\tif (*r1 != *r2)\n \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n \t\tr1++;\n@@ -528,7 +529,8 @@ static int load_contents(struct snapshot *snapshot)\n }\n \n static const char *find_reference_location_1(struct snapshot *snapshot,\n-\t\t\t\t\t     const char *refname, int mustexist)\n+\t\t\t\t\t     const char *refname, int mustexist,\n+\t\t\t\t\t     int start)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -558,7 +560,7 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n \n \t\tmid = lo + (hi - lo) / 2;\n \t\trec = find_start_of_record(lo, mid);\n-\t\tcmp = cmp_record_to_refname(rec, refname);\n+\t\tcmp = cmp_record_to_refname(rec, refname, start);\n \t\tif (cmp < 0) {\n \t\t\tlo = find_end_of_record(mid, hi);\n \t\t} else if (cmp > 0) {\n@@ -591,7 +593,22 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n static const char *find_reference_location(struct snapshot *snapshot,\n \t\t\t\t\t   const char *refname, int mustexist)\n {\n-\treturn find_reference_location_1(snapshot, refname, mustexist);\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n+}\n+\n+/*\n+ * Find the place in `snapshot->buf` after the end of the record for\n+ * `refname`. In other words, find the location of first thing *after*\n+ * `refname`.\n+ *\n+ * Other semantics are identical to the ones in\n+ * `find_reference_location()`.\n+ */\n+static const char *find_reference_location_end(struct snapshot *snapshot,\n+\t\t\t\t\t       const char *refname,\n+\t\t\t\t\t       int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 0);\n }\n \n /*\n@@ -785,6 +802,13 @@ struct packed_ref_iterator {\n \t/* The end of the part of the buffer that will be iterated over: */\n \tconst char *eof;\n \n+\tstruct jump_list_entry {\n+\t\tconst char *start;\n+\t\tconst char *end;\n+\t} *jump;\n+\tsize_t jump_nr, jump_alloc;\n+\tsize_t jump_pos;\n+\n \t/* Scratch space for current values: */\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n@@ -802,14 +826,34 @@ struct packed_ref_iterator {\n  */\n static int next_record(struct packed_ref_iterator *iter)\n {\n-\tconst char *p = iter->pos, *eol;\n+\tconst char *p, *eol;\n \n \tstrbuf_reset(&iter->refname_buf);\n \n+\t/*\n+\t * If iter->pos is contained within a skipped region, jump past\n+\t * it.\n+\t *\n+\t * Note that each skipped region is considered at most once,\n+\t * since they are ordered based on their starting position.\n+\t */\n+\twhile (iter->jump_pos < iter->jump_nr) {\n+\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_pos];\n+\t\tif (iter->pos < curr->start)\n+\t\t\tbreak; /* not to the next jump yet */\n+\n+\t\titer->jump_pos++;\n+\t\tif (iter->pos < curr->end) {\n+\t\t\titer->pos = curr->end;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n \tif (iter->pos == iter->eof)\n \t\treturn ITER_DONE;\n \n \titer->base.flags = REF_ISPACKED;\n+\tp = iter->pos;\n \n \tif (iter->eof - p < the_hash_algo->hexsz + 2 ||\n \t    parse_oid_hex(p, &iter->oid, &p) ||\n@@ -917,6 +961,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \tint ok = ITER_DONE;\n \n \tstrbuf_release(&iter->refname_buf);\n+\tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n \tbase_ref_iterator_free(ref_iterator);\n \treturn ok;\n@@ -928,6 +973,112 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.abort = packed_ref_iterator_abort\n };\n \n+static int jump_list_entry_cmp(const void *va, const void *vb)\n+{\n+\tconst struct jump_list_entry *a = va;\n+\tconst struct jump_list_entry *b = vb;\n+\n+\tif (a->start < b->start)\n+\t\treturn -1;\n+\tif (a->start > b->start)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int has_glob_special(const char *str)\n+{\n+\tconst char *p;\n+\tfor (p = str; *p; p++) {\n+\t\tif (is_glob_special(*p))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static const char *ptr_max(const char *x, const char *y)\n+{\n+\tif (x > y)\n+\t\treturn x;\n+\treturn y;\n+}\n+\n+static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n+\t\t\t\t\tstruct snapshot *snapshot,\n+\t\t\t\t\tconst char **excluded_patterns)\n+{\n+\tsize_t i, j;\n+\tconst char **pattern;\n+\tstruct jump_list_entry *last_disjoint;\n+\n+\tif (!excluded_patterns)\n+\t\treturn;\n+\n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\tstruct jump_list_entry *e;\n+\n+\t\t/*\n+\t\t * We can't feed any excludes with globs in them to the\n+\t\t * refs machinery.  It only understands prefix matching.\n+\t\t * We likewise can't even feed the string leading up to\n+\t\t * the first meta-character, as something like \"foo[a]\"\n+\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n+\t\t * would match that and mark it for exclusion).\n+\t\t */\n+\t\tif (has_glob_special(*pattern))\n+\t\t\tcontinue;\n+\n+\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n+\n+\t\te = &iter->jump[iter->jump_nr++];\n+\t\te->start = find_reference_location(snapshot, *pattern, 0);\n+\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n+\t}\n+\n+\tif (!iter->jump_nr) {\n+\t\t/*\n+\t\t * Every entry in exclude_patterns has a meta-character,\n+\t\t * nothing to do here.\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tQSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);\n+\n+\t/*\n+\t * As an optimization, merge adjacent entries in the jump list\n+\t * to jump forwards as far as possible when entering a skipped\n+\t * region.\n+\t *\n+\t * For example, if we have two skipped regions:\n+\t *\n+\t *\t[[A, B], [B, C]]\n+\t *\n+\t * we want to combine that into a single entry jumping from A to\n+\t * C.\n+\t */\n+\tlast_disjoint = iter->jump;\n+\n+\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n+\t\tstruct jump_list_entry *ours = &iter->jump[i];\n+\n+\t\tif (ours->start == ours->end) {\n+\t\t\t/* ignore empty regions (no matching entries) */\n+\t\t\tcontinue;\n+\t\t} else if (ours->start <= last_disjoint->end) {\n+\t\t\t/* overlapping regions extend the previous one */\n+\t\t\tlast_disjoint->end = ptr_max(last_disjoint->end, ours->end);\n+\t\t} else {\n+\t\t\t/* otherwise, insert a new region */\n+\t\t\titer->jump[j++] = *ours;\n+\t\t\tlast_disjoint = ours;\n+\n+\t\t}\n+\t}\n+\n+\titer->jump_nr = j;\n+\titer->jump_pos = 0;\n+}\n+\n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n \t\tconst char *prefix, const char **exclude_patterns,\n@@ -963,6 +1114,9 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);\n \n+\tif (exclude_patterns)\n+\t\tpopulate_excluded_jump_list(iter, snapshot, exclude_patterns);\n+\n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n \ndiff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\nindex 6d8f844e9c..2bff003f7c 100644\n--- a/t/helper/test-ref-store.c\n+++ b/t/helper/test-ref-store.c\n@@ -175,6 +175,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n \treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n }\n \n+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)\n+{\n+\tconst char *prefix = notnull(*argv++, \"prefix\");\n+\tconst char **exclude_patterns = argv;\n+\n+\treturn refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,\n+\t\t\t\t\tNULL);\n+}\n+\n static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\n {\n \tstruct object_id oid = *null_oid();\n@@ -307,6 +316,7 @@ static struct command commands[] = {\n \t{ \"delete-refs\", cmd_delete_refs },\n \t{ \"rename-ref\", cmd_rename_ref },\n \t{ \"for-each-ref\", cmd_for_each_ref },\n+\t{ \"for-each-ref--exclude\", cmd_for_each_ref__exclude },\n \t{ \"resolve-ref\", cmd_resolve_ref },\n \t{ \"verify-ref\", cmd_verify_ref },\n \t{ \"for-each-reflog\", cmd_for_each_reflog },\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nnew file mode 100755\nindex 0000000000..bc534c8ea1\n--- /dev/null\n+++ b/t/t1419-exclude-refs.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='test exclude_patterns functionality in main ref store'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+for_each_ref__exclude () {\n+\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tcut -d ' ' -f 2 actual.raw\n+}\n+\n+for_each_ref () {\n+\tgit for-each-ref --format='%(refname)' \"$@\"\n+}\n+\n+test_expect_success 'setup' '\n+\ttest_commit --no-tag base &&\n+\tbase=\"$(git rev-parse HEAD)\" &&\n+\n+\tfor name in foo bar baz quux\n+\tdo\n+\t\tfor i in 1 2 3\n+\t\tdo\n+\t\t\techo \"create refs/heads/$name/$i $base\" || return 1\n+\t\tdone || return 1\n+\tdone >in &&\n+\techo \"delete refs/heads/main\" >>in &&\n+\n+\tgit update-ref --stdin <in &&\n+\tgit pack-refs --all\n+'\n+\n+test_expect_success 'excluded region in middle' '\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at beginning' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at end' '\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'disjoint excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'adjacent, non-overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'several overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads \\\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'non-matching excluded section' '\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'meta-characters are discarded' '\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478134","messageId":"3c045076a95fab9d050a312114c9aca712eb7141.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 10/16] refs/packed-backend.c: add trace2 counters for jump list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:42Z","receivedAt":"2023-06-07T10:42:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The previous commit added low-level tests to ensure that the packed-refs\niterator did not enumerate excluded sections of the refspace.\n\nHowever, there was no guarantee that these sections weren't being\nvisited, only that they were being suppressed from the output. To harden\nthese tests, add a trace2 counter which tracks the number of regions\nskipped by the packed-refs iterator, and assert on its value.\n\nSuggested-by: Derrick Stolee <derrickstolee@github.com>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   |  2 ++\n t/t1419-exclude-refs.sh | 59 ++++++++++++++++++++++++++++-------------\n trace2.h                |  2 ++\n trace2/tr2_ctr.c        |  5 ++++\n 4 files changed, 49 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 67327e579c..7ba9fa2bb8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -12,6 +12,7 @@\n #include \"../chdir-notify.h\"\n #include \"../wrapper.h\"\n #include \"../write-or-die.h\"\n+#include \"../trace2.h\"\n \n enum mmap_strategy {\n \t/*\n@@ -845,6 +846,7 @@ static int next_record(struct packed_ref_iterator *iter)\n \t\titer->jump_pos++;\n \t\tif (iter->pos < curr->end) {\n \t\t\titer->pos = curr->end;\n+\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);\n \t\t\tbreak;\n \t\t}\n \t}\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex bc534c8ea1..350a7d2587 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -9,7 +9,8 @@ TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n for_each_ref__exclude () {\n-\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tGIT_TRACE2_PERF=1 test-tool ref-store main \\\n+\t\tfor-each-ref--exclude \"$@\" >actual.raw\n \tcut -d ' ' -f 2 actual.raw\n }\n \n@@ -17,6 +18,17 @@ for_each_ref () {\n \tgit for-each-ref --format='%(refname)' \"$@\"\n }\n \n+assert_jumps () {\n+\tlocal nr=\"$1\"\n+\tlocal trace=\"$2\"\n+\n+\tgrep -q \"name:jumps_made value:$nr\" $trace\n+}\n+\n+assert_no_jumps () {\n+\t! assert_jumps \".*\" \"$1\"\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit --no-tag base &&\n \tbase=\"$(git rev-parse HEAD)\" &&\n@@ -35,67 +47,76 @@ test_expect_success 'setup' '\n '\n \n test_expect_success 'excluded region in middle' '\n-\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at beginning' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at end' '\n-\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'disjoint excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 2 perf\n '\n \n test_expect_success 'adjacent, non-overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'several overlapping excluded regions' '\n \tfor_each_ref__exclude refs/heads \\\n-\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'non-matching excluded section' '\n-\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps\n '\n \n test_expect_success 'meta-characters are discarded' '\n-\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps\n '\n \n test_done\ndiff --git a/trace2.h b/trace2.h\nindex 4ced30c0db..9452e291f5 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -551,6 +551,8 @@ enum trace2_counter_id {\n \tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n \tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n \n+\tTRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */\n+\n \t/* Add additional counter definitions before here. */\n \tTRACE2_NUMBER_OF_COUNTERS\n };\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nindex b342d3b1a3..50570d0165 100644\n--- a/trace2/tr2_ctr.c\n+++ b/trace2/tr2_ctr.c\n@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER\n \t\t.name = \"test2\",\n \t\t.want_per_thread_events = 1,\n \t},\n+\t[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {\n+\t\t.category = \"packed-refs\",\n+\t\t.name = \"jumps_made\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n \n \t/* Add additional metadata before here. */\n };\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478135","messageId":"0ff542eaefd5142056b484901a3c22e2a590b07e.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 11/16] revision.h: store hidden refs in a `strvec`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:47Z","receivedAt":"2023-06-07T10:42:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, it will be convenient to have a 'const char **'\nof hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\netc.), instead of a `string_list`.\n\nConvert spots throughout the tree that store the list of hidden refs\nfrom a `string_list` to a `strvec`.\n\nNote that in `parse_hide_refs_config()` there is an ugly const-cast used\nto avoid an extra copy of each value before trimming any trailing slash\ncharacters. This could instead be written as:\n\n    ref = xstrdup(value);\n    len = strlen(ref);\n    while (len && ref[len - 1] == '/')\n            ref[--len] = '\\0';\n    strvec_push(hide_refs, ref);\n    free(ref);\n\nbut the double-copy (once when calling `xstrdup()`, and another via\n`strvec_push()`) is wasteful.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c |  4 ++--\n ls-refs.c              |  6 +++---\n refs.c                 | 11 ++++++-----\n refs.h                 |  4 ++--\n revision.c             |  2 +-\n revision.h             |  5 +++--\n upload-pack.c          | 10 +++++-----\n 7 files changed, 22 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1a31a58367..1a8472eddc 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -90,7 +90,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+static struct strvec hidden_refs = STRVEC_INIT;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -2619,7 +2619,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n-\tstring_list_clear(&hidden_refs, 0);\n+\tstrvec_clear(&hidden_refs);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 6f490b2d9c..8c3181d051 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -72,7 +72,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -155,7 +155,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n-\tstring_list_init_dup(&data.hidden_refs);\n+\tstrvec_init(&data.hidden_refs);\n \n \tgit_config(ls_refs_config, &data);\n \n@@ -197,7 +197,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-\tstring_list_clear(&data.hidden_refs, 0);\n+\tstrvec_clear(&data.hidden_refs);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 538bde644e..ec4d5b9101 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n }\n \n int parse_hide_refs_config(const char *var, const char *value, const char *section,\n-\t\t\t   struct string_list *hide_refs)\n+\t\t\t   struct strvec *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n-\t\tref = xstrdup(value);\n+\n+\t\t/* drop const to remove trailing '/' characters */\n+\t\tref = (char *)strvec_push(hide_refs, value);\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n int ref_is_hidden(const char *refname, const char *refname_full,\n-\t\t  const struct string_list *hide_refs)\n+\t\t  const struct strvec *hide_refs)\n {\n \tint i;\n \n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n-\t\tconst char *match = hide_refs->items[i].string;\n+\t\tconst char *match = hide_refs->v[i];\n \t\tconst char *subject;\n \t\tint neg = 0;\n \t\tconst char *p;\ndiff --git a/refs.h b/refs.h\nindex d672d636cf..a7751a1fc9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -810,7 +810,7 @@ int update_ref(const char *msg, const char *refname,\n \t       unsigned int flags, enum action_on_err onerr);\n \n int parse_hide_refs_config(const char *var, const char *value, const char *,\n-\t\t\t   struct string_list *);\n+\t\t\t   struct strvec *);\n \n /*\n  * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -820,7 +820,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,\n  * the ref is outside that namespace, the first parameter is NULL. The second\n  * parameter always points to the full ref name.\n  */\n-int ref_is_hidden(const char *, const char *, const struct string_list *);\n+int ref_is_hidden(const char *, const char *, const struct strvec *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/revision.c b/revision.c\nindex 89953592f9..7c9367a266 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n-\tstring_list_clear(&exclusions->hidden_refs, 0);\n+\tstrvec_clear(&exclusions->hidden_refs);\n \texclusions->hidden_refs_configured = 0;\n }\n \ndiff --git a/revision.h b/revision.h\nindex 31828748dc..94f035fa22 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -10,6 +10,7 @@\n #include \"decorate.h\"\n #include \"ident.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"strvec.h\"\n \n /**\n  * The revision walking API offers functions to build a list of revisions\n@@ -95,7 +96,7 @@ struct ref_exclusions {\n \t * Hidden refs is a list of patterns that is to be hidden via\n \t * `ref_is_hidden()`.\n \t */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \t/*\n \t * Indicates whether hidden refs have been configured. This is to\n@@ -110,7 +111,7 @@ struct ref_exclusions {\n  */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n-\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRVEC_INIT, \\\n }\n \n struct oidset;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d3312006a3..1a213ed775 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -69,7 +69,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -127,7 +127,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n-\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+\tstruct strvec hidden_refs = STRVEC_INIT;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -162,7 +162,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n-\tstring_list_clear(&data->hidden_refs, 0);\n+\tstrvec_clear(&data->hidden_refs);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -1170,7 +1170,7 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n+\t\t\tconst struct object_id *oid, const struct strvec *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n@@ -1465,7 +1465,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t  struct string_list *wanted_refs,\n-\t\t\t  struct string_list *hidden_refs,\n+\t\t\t  struct strvec *hidden_refs,\n \t\t\t  struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478136","messageId":"ca006b2c3f3474cdf7cfebbdb8575059e036d9cc.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 12/16] refs/packed-backend.c: ignore complicated hidden refs rules","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:52Z","receivedAt":"2023-06-07T10:42:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\nuse the new skip-list feature in the packed-refs iterator by ignoring\nreferences which are mentioned via its respective hideRefs lists.\n\nHowever, the packed-ref skip lists cannot handle un-hiding rules (that\nbegin with '!'), or namespace comparisons (that begin with '^'). Detect\nand avoid these cases by falling back to the normal enumeration without\na skip list when such patterns exist.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   | 19 +++++++++++++++++++\n t/t1419-exclude-refs.sh |  9 +++++++++\n 2 files changed, 28 insertions(+)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 7ba9fa2bb8..9ea6c07866 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1015,6 +1015,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n \tif (!excluded_patterns)\n \t\treturn;\n \n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\t/*\n+\t\t * We also can't feed any excludes from hidden refs\n+\t\t * config sections, since later rules may override\n+\t\t * previous ones. For example, with rules \"refs/foo\" and\n+\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n+\t\t * everything underneath it), but the earlier exclusion\n+\t\t * would cause us to skip all of \"refs/foo\". We likewise\n+\t\t * don't implement the namespace stripping required for\n+\t\t * '^' rules.\n+\t\t *\n+\t\t * Both are possible to do, but complicated, so avoid\n+\t\t * populating the jump list at all if we see either of\n+\t\t * these patterns.\n+\t\t */\n+\t\tif (**pattern == '!' || **pattern == '^')\n+\t\t\treturn;\n+\t}\n+\n \tfor (pattern = excluded_patterns; *pattern; pattern++) {\n \t\tstruct jump_list_entry *e;\n \ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex 350a7d2587..0e91e2f399 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -119,4 +119,13 @@ test_expect_success 'meta-characters are discarded' '\n \tassert_no_jumps\n '\n \n+test_expect_success 'complex hidden ref rules are discarded' '\n+\tfor_each_ref__exclude refs/heads refs/heads/foo \"!refs/heads/foo/1\" \\\n+\t\t>actual 2>perf &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps\n+'\n+\n test_done\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478137","messageId":"cae703a4253e5c1a7a99fa384ba65bcc3fc613a8.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 13/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:41:57Z","receivedAt":"2023-06-07T10:42:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The following commit will want to call `for_each_namespaced_ref()` with\na list of excluded patterns.\n\nWe could introduce a variant of that function, say,\n`for_each_namespaced_ref_exclude()` which takes the extra parameter, and\nreimplement the original function in terms of that. But all but one\ncaller (in `http-backend.c`) will supply the new parameter, so add the\nnew parameter to `for_each_namespaced_ref()` itself instead of\nintroducing a new function.\n\nFor now, supply NULL for the list of excluded patterns at all callers to\navoid changing behavior, which we will do in the subsequent commit.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n http-backend.c | 2 +-\n refs.c         | 5 +++--\n refs.h         | 3 ++-\n upload-pack.c  | 6 +++---\n 4 files changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex ac146d85c5..ad500683c8 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)\n \n \t} else {\n \t\tselect_getanyfile(hdr);\n-\t\tfor_each_namespaced_ref(show_text_ref, &buf);\n+\t\tfor_each_namespaced_ref(NULL, show_text_ref, &buf);\n \t\tsend_strbuf(hdr, \"text/plain\", &buf);\n \t}\n \tstrbuf_release(&buf);\ndiff --git a/refs.c b/refs.c\nindex ec4d5b9101..95a7db9563 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat\n \t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, exclude_patterns, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\ndiff --git a/refs.h b/refs.h\nindex a7751a1fc9..f23626beca 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -372,7 +372,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \t\t\t const char *prefix, void *cb_data);\n \n int head_ref_namespaced(each_ref_fn fn, void *cb_data);\n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data);\n \n /* can be used to learn about broken ref and symref */\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 1a213ed775..99d216938c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -855,7 +855,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(check_ref, data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1386,7 +1386,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(send_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n \t\tif (!data.sent_capabilities) {\n \t\t\tconst char *refname = \"capabilities^{}\";\n \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n@@ -1400,7 +1400,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(check_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478138","messageId":"1db10b76eaa4054fb0c065fe4e15449f07e3d1c5.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 14/16] builtin/receive-pack.c: avoid enumerating hidden references","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:42:02Z","receivedAt":"2023-06-07T10:42:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that `refs_for_each_fullref_in()` has the ability to avoid\nenumerating references matching certain pattern(s), use that to avoid\nvisiting hidden refs when constructing the ref advertisement via\nreceive-pack.\n\nNote that since this exclusion is best-effort, we still need\n`show_ref_cb()` to check whether or not each reference is hidden or not\nbefore including it in the advertisement.\n\nAs was the case when applying this same optimization to `upload-pack`,\n`receive-pack`'s reference advertisement phase can proceed much quicker\nby avoiding enumerating references that will not be part of the\nadvertisement.\n\n(Below, we're still using linux.git with one hidden refs/pull/N ref per\ncommit):\n\n    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]\n      Range (min … max):    87.7 ms …  95.5 ms    31 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]\n      Range (min … max):     4.1 ms …   5.6 ms    508 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran\n       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1a8472eddc..bd5bcc375f 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -337,7 +337,8 @@ static void write_head_info(void)\n {\n \tstatic struct oidset seen = OIDSET_INIT;\n \n-\tfor_each_ref(show_ref_cb, &seen);\n+\trefs_for_each_fullref_in(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t hidden_refs.v, show_ref_cb, &seen);\n \tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n \toidset_clear(&seen);\n \tif (!sent_capabilities)\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478139","messageId":"014243ebe6586313b38fe0eb2a57941b27131423.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 15/16] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:42:07Z","receivedAt":"2023-06-07T10:42:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as a previous commit, teach `upload-pack` to avoid\nenumerating hidden references where possible.\n\nNote, however, that there are certain cases where cannot avoid\nenumerating even hidden references, in particular when either of:\n\n  - `uploadpack.allowTipSHA1InWant`, or\n  - `uploadpack.allowReachableSHA1InWant`\n\nare set, corresponding to `ALLOW_TIP_SHA1` and `ALLOW_REACHABLE_SHA1`,\nrespectively.\n\nWhen either of these bits are set, upload-pack's `is_our_ref()` function\nneeds to consider the `HIDDEN_REF` bit of the referent's object flags.\nSo we must visit all references, including the hidden ones, in order to\nmark their referents with the `HIDDEN_REF` bit.\n\nWhen neither `ALLOW_TIP_SHA1` nor `ALLOW_REACHABLE_SHA1` are set, the\n`is_our_ref()` function considers only the `OUR_REF` bit, and not the\n`HIDDEN_REF` one. `OUR_REF` is applied via `mark_our_ref()`, and only\nto objects at the tips of non-hidden references, so we do not need to\nvisit hidden references in this case.\n\nWhen neither of those bits are set, `upload-pack` can potentially avoid\nenumerating a large number of references. In the same example as a\nprevious commit (linux.git with one hidden reference per commit,\n\"refs/pull/N\"):\n\n    $ printf 0000 >in\n    $ hyperfine --warmup=1 \\\n      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]\n      Range (min … max):   405.7 ms … 409.2 ms    10 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in\n      Time (mean ± σ):     406.5 ms ±   1.3 ms    [User: 356.5 ms, System: 49.9 ms]\n      Range (min … max):   404.6 ms … 408.8 ms    10 runs\n\n    Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):       4.7 ms ±   0.2 ms    [User: 0.7 ms, System: 3.9 ms]\n      Range (min … max):     4.3 ms …   6.1 ms    472 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran\n       86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in'\n       86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'\n\nAs above, we must visit every reference when\nuploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit\nfar fewer references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n upload-pack.c | 33 +++++++++++++++++++++++++++------\n 1 file changed, 27 insertions(+), 6 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 99d216938c..366a101d8d 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -602,11 +602,32 @@ static int get_common_commits(struct upload_pack_data *data,\n \t}\n }\n \n+static int allow_hidden_refs(enum allow_uor allow_uor)\n+{\n+\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n+}\n+\n+static void for_each_namespaced_ref_1(each_ref_fn fn,\n+\t\t\t\t      struct upload_pack_data *data)\n+{\n+\t/*\n+\t * If `data->allow_uor` allows fetching hidden refs, we need to\n+\t * mark all references (including hidden ones), to check in\n+\t * `is_our_ref()` below.\n+\t *\n+\t * Otherwise, we only care about whether each reference's object\n+\t * has the OUR_REF bit set or not, so do not need to visit\n+\t * hidden references.\n+\t */\n+\tif (allow_hidden_refs(data->allow_uor))\n+\t\tfor_each_namespaced_ref(NULL, fn, data);\n+\telse\n+\t\tfor_each_namespaced_ref(data->hidden_refs.v, fn, data);\n+}\n+\n static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n {\n-\tint allow_hidden_ref = (allow_uor &\n-\t\t\t\t(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n-\treturn o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);\n+\treturn o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);\n }\n \n /*\n@@ -855,7 +876,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n+\t\tfor_each_namespaced_ref_1(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1386,7 +1407,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n+\t\tfor_each_namespaced_ref_1(send_ref, &data);\n \t\tif (!data.sent_capabilities) {\n \t\t\tconst char *refname = \"capabilities^{}\";\n \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n@@ -1400,7 +1421,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n+\t\tfor_each_namespaced_ref_1(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.41.0.16.g26cd413590\n\n"},{"id":"478140","messageId":"e02fe9337953b8c742861320908a449fd20104d5.1686134440.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"[PATCH v3 16/16] ls-refs.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-07T10:42:11Z","receivedAt":"2023-06-07T10:42:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach `ls-refs` to avoid\nenumerating hidden references where possible.\n\nAs before, this is linux.git with one hidden reference per commit.\n\n    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'\n    Benchmark 1: git -c protocol.version=2 ls-remote .\n      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]\n      Range (min … max):    88.8 ms …  91.3 ms    32 runs\n\n    Benchmark 2: git.compile -c protocol.version=2 ls-remote .\n      Time (mean ± σ):       6.5 ms ±   0.1 ms    [User: 2.4 ms, System: 4.3 ms]\n      Range (min … max):     6.2 ms …   8.3 ms    397 runs\n\n    Summary\n      'git.compile -c protocol.version=2 ls-remote .' ran\n       13.85 ± 0.33 times faster than 'git -c protocol.version=2 ls-remote .'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ls-refs.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 8c3181d051..c9a723ba89 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  NULL, send_ref, &data);\n+\t\t\t\t\t  data.hidden_refs.v, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-- \n2.41.0.16.g26cd413590\n"},{"id":"478303","messageId":"xmqqzg548k8k.fsf@gitster.g","threadId":"59715","inReplyTo":"cover.1686134440.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/16] refs: implement jump lists for packed backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-12T21:05:15Z","receivedAt":"2023-06-12T21:10:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Here is a reroll of my series to implement jump (née skip) lists for the\n> packed refs backend, based on top of the current 'master'.\n\nHmph.  I kind of liked Patrick's suggestion to split this into two\nseries to make it easier for the earlier half to graduate faster,\nbut perhaps it didn't make much difference?  I certainly did not get\nthe impression that \"review had stabilized\".  During my review of\nthe initial round, for example, I lost steam in the middle because\nit was simply too long a series and didn't have a chance to give the\nremainder a proper review.  I do not know about others.\n\n"},{"id":"478342","messageId":"xmqqedmf3szq.fsf@gitster.g","threadId":"59715","inReplyTo":"afac948f04dca5cd532ceea08fbc646184336311.1686134440.git.me@ttaylorr.com","subject":"Re: [PATCH v3 01/16] refs.c: rename `ref_filter`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-13T22:19:37Z","receivedAt":"2023-06-13T22:19:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> -static int filter_refs(const char *refname, const struct object_id *oid,\n> -\t\t\t   int flags, void *data)\n> +static int for_each_filter_refs(const char *refname,\n> +\t\t\t\tconst struct object_id *oid,\n> +\t\t\t\tint flags, void *data)\n>  {\n> -\tstruct ref_filter *filter = (struct ref_filter *)data;\n> +\tstruct for_each_ref_filter *filter = data;\n\nNice to see that a trivial and obvious clean-up like this is\nsilently included ;-)\n"},{"id":"478344","messageId":"xmqq352v3s6n.fsf@gitster.g","threadId":"59715","inReplyTo":"1988ca4c0a79065a1893259da8abe63919e05cf3.1686134440.git.me@ttaylorr.com","subject":"Re: [PATCH v3 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-13T22:37:04Z","receivedAt":"2023-06-13T22:37:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Once we start passing either in, `match_pattern()` will have little to\n> do with a particular `struct ref_filter *` instance. To clarify this,\n> drop it from the argument list, and replace it with the only bit of the\n> `ref_filter` that we care about (`filter->ignore_case`).\n\nMakes sense.\n"},{"id":"478347","messageId":"xmqqsfav2alu.fsf@gitster.g","threadId":"59715","inReplyTo":"c4fe9a1893c15c5aae4d3f305b248d0c3ac55222.1686134440.git.me@ttaylorr.com","subject":"Re: [PATCH v3 07/16] refs: plumb `exclude_patterns` argument throughout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-13T23:42:05Z","receivedAt":"2023-06-13T23:45:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> diff --git a/ls-refs.c b/ls-refs.c\n> index f385938b64..6f490b2d9c 100644\n> --- a/ls-refs.c\n> +++ b/ls-refs.c\n> @@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n>  \t\tstrvec_push(&data.prefixes, \"\");\n>  \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n>  \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n> -\t\t\t\t\t  send_ref, &data);\n> +\t\t\t\t\t  NULL, send_ref, &data);\n\nOK.\n\n> diff --git a/ref-filter.c b/ref-filter.c\n> index d44418efb7..717c3c4bcf 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n>  \n>  \tif (!filter->name_patterns[0]) {\n>  \t\t/* no patterns; we have to look at everything */\n> -\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n> +\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> +\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n>  \t}\n\nIs this merely \"while at it\", or was there a reason why refs_*\nvariant must be used here?  It is curious that we do not teach the\nexclude_patterns to some functions like for_each_fullref_in() while\nadding exclude_patterns to others, making the API surface uneven.\n\n"},{"id":"478349","messageId":"xmqq352u3n3a.fsf@gitster.g","threadId":"59715","inReplyTo":"8066858bf59386eeec68f0f1e4247eeebb6482f7.1686134440.git.me@ttaylorr.com","subject":"Re: [PATCH v3 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-14T00:27:05Z","receivedAt":"2023-06-14T00:27:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> When iterating through the `packed-refs` file in order to answer a query\n> like:\n>\n>     $ git for-each-ref --exclude=refs/__hidden__\n>\n> it would be useful to avoid walking over all of the entries in\n> `refs/__hidden__/*` when possible, since we know that the ref-filter\n> code is going to throw them away anyways.\n>\n> In certain circumstances, doing so is possible. The algorithm for doing\n> so is as follows:\n>\n>   - For each excluded pattern, find the first record that matches it,\n>     and the first record that *doesn't* match it (i.e. the location\n>     you'd next want to consider when excluding that pattern).\n>\n>   - Sort the set of excluded regions from the previous step in ascending\n>     order of the first location within the `packed-refs` file that\n>     matches.\n>\n>   - Clean up the results from the previous step: discard empty regions,\n>     and combine adjacent regions.\n\nSay something like\n\n   The resulting list of regions that would never contain refs that\n   are not excluded is called the \"jump list\".\n\nhere, probably.  Otherwise the first reference to \"jump list\" we see\nlater feels a bit too abrupt.\n\n> Then when iterating through the `packed-refs` file, if `iter->pos` is\n> ever contained in one of the regions from the previous steps, advance\n> `iter->pos` past the end of that region, and continue enumeration.\n>\n> Note that we only perform this optimization when none of the excluded\n> pattern(s) have special meta-characters in them. For a pattern like\n> \"refs/foo[ac]\", the excluded regions (\"refs/fooa\", \"refs/fooc\", and\n> everything underneath them) are not connected. A future implementation\n> that handles this case may split the character class (pretending as if\n> two patterns were excluded: \"refs/fooa\", and \"refs/fooc\").\n\nMakes sense.\n\n> There are a few other gotchas worth considering. First, note that the\n> jump list is sorted, so once we jump past a region, we can avoid\n> considering it (or any regions preceding it) again. The member\n> `jump_pos` is used to track the first next-possible region to jump\n> through.\n>\n> Second, note that the exclusion list is best-effort, since we do not\n> handle loose references, and because of the meta-character issue above.\n\nI found this a bit misleading; a natural reading of \"exclusion list\"\nis that the phrase refers to the list of exclude patterns given from\nthe command line, and users would be upset if the processing of it\nis best effort.\n\nI think what you meant to say was that optimization to avoid full\nscan using the jump list does not aim for perfection, and entries\nthat are not skipped using the jump list may still be excluded by\nthe exclude patterns.\n\n> In repositories with a large number of hidden references, the speed-up\n\n\"hidden\" -> \"excluded\".  Your final objective to implement the\nfeature to exclude refs using patterns and optimize it using the\njump list data may be to implement \"hidden references\", but that\nhasn't be talked about in the above.  All we have been hearing was\nthat we are optimizing the walk over packed-refs using exclude\npatterns.\n\n> can be significant. Tests here are done with a copy of linux.git with a\n> reference \"refs/pull/N\" pointing at every commit, as in:\n> ...\n> Co-authored-by: Jeff King <peff@peff.net>\n> Signed-off-by: Jeff King <peff@peff.net>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  ref-filter.c              |   5 +-\n>  refs/packed-backend.c     | 166 ++++++++++++++++++++++++++++++++++++--\n>  t/helper/test-ref-store.c |  10 +++\n>  t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++\n>  4 files changed, 274 insertions(+), 8 deletions(-)\n>  create mode 100755 t/t1419-exclude-refs.sh\n\nNice.\n\n> @@ -785,6 +802,13 @@ struct packed_ref_iterator {\n>  \t/* The end of the part of the buffer that will be iterated over: */\n>  \tconst char *eof;\n>  \n> +\tstruct jump_list_entry {\n> +\t\tconst char *start;\n> +\t\tconst char *end;\n> +\t} *jump;\n> +\tsize_t jump_nr, jump_alloc;\n> +\tsize_t jump_pos;\n> +\n>  \t/* Scratch space for current values: */\n>  \tstruct object_id oid, peeled;\n>  \tstruct strbuf refname_buf;\n> @@ -802,14 +826,34 @@ struct packed_ref_iterator {\n>   */\n>  static int next_record(struct packed_ref_iterator *iter)\n>  {\n> -\tconst char *p = iter->pos, *eol;\n> +\tconst char *p, *eol;\n>  \n>  \tstrbuf_reset(&iter->refname_buf);\n>  \n> +\t/*\n> +\t * If iter->pos is contained within a skipped region, jump past\n> +\t * it.\n> +\t *\n> +\t * Note that each skipped region is considered at most once,\n> +\t * since they are ordered based on their starting position.\n> +\t */\n> +\twhile (iter->jump_pos < iter->jump_nr) {\n> +\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_pos];\n> +\t\tif (iter->pos < curr->start)\n> +\t\t\tbreak; /* not to the next jump yet */\n> +\n> +\t\titer->jump_pos++;\n> +\t\tif (iter->pos < curr->end) {\n> +\t\t\titer->pos = curr->end;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n\nQuite straight-forward.\n\n> +static const char *ptr_max(const char *x, const char *y)\n> +{\n> +\tif (x > y)\n> +\t\treturn x;\n> +\treturn y;\n> +}\n\nHopefully the compiler would inline the function without being told.\n\nThese pointers point into the same mmapped region of contiguous\nmemory that holds the contents of the packed-refs file, so\ncomparison between them is always defined.  Good.\n\nI wondered if\n\n\treturn (x > y) ? x : y;\n\nis easier to read, simply because it treats both cases more equally\n(in other words, as written, (x>y) appears more \"special\"), but that\nis minor.\n\n> +static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n> +\t\t\t\t\tstruct snapshot *snapshot,\n> +\t\t\t\t\tconst char **excluded_patterns)\n> +{\n> +\tsize_t i, j;\n> +\tconst char **pattern;\n> +\tstruct jump_list_entry *last_disjoint;\n> +\n> +\tif (!excluded_patterns)\n> +\t\treturn;\n> +\n> +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> +\t\tstruct jump_list_entry *e;\n> +\n> +\t\t/*\n> +\t\t * We can't feed any excludes with globs in them to the\n> +\t\t * refs machinery.  It only understands prefix matching.\n> +\t\t * We likewise can't even feed the string leading up to\n> +\t\t * the first meta-character, as something like \"foo[a]\"\n> +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> +\t\t * would match that and mark it for exclusion).\n> +\t\t */\n> +\t\tif (has_glob_special(*pattern))\n> +\t\t\tcontinue;\n\nOK.  When we have a \"literal\" exclude pattern and another \"glob\"\nexclude pattern, we can afford to ignore the \"glob\" one when\nbuilding the jump list and include only the \"literal\" one, because\nthe jump list is used only to skip over entries that obviously can\nnever be in the result, *and* --exclude are additive (i.e. being\non jump list because of the \"literal\" pattern is a reason enough to\nbe excluded from the result; matching or not matching the other\npatterns does not affect the fate of the ref that got excluded due\nto matching the \"literal\" pattern).\n\nMakes sense.\n\n> +\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n> +\n> +\t\te = &iter->jump[iter->jump_nr++];\n> +\t\te->start = find_reference_location(snapshot, *pattern, 0);\n> +\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n> +\t}\n> +\n> +\tif (!iter->jump_nr) {\n> +\t\t/*\n> +\t\t * Every entry in exclude_patterns has a meta-character,\n> +\t\t * nothing to do here.\n> +\t\t */\n> +\t\treturn;\n> +\t}\n\nOK.\n\nThanks.\n"},{"id":"478351","messageId":"xmqqttva28a7.fsf@gitster.g","threadId":"59715","inReplyTo":"3c045076a95fab9d050a312114c9aca712eb7141.1686134440.git.me@ttaylorr.com","subject":"Re: [PATCH v3 10/16] refs/packed-backend.c: add trace2 counters for jump list","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-14T00:32:16Z","receivedAt":"2023-06-14T00:32:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> +assert_jumps () {\n> +\tlocal nr=\"$1\"\n> +\tlocal trace=\"$2\"\n> +\n> +\tgrep -q \"name:jumps_made value:$nr\" $trace\n> +}\n\nYou may expect 1 hit and the test will pass if trace had 10 or 12 hits.\n\n"},{"id":"478352","messageId":"xmqqo7li27ww.fsf@gitster.g","threadId":"59715","inReplyTo":"ca006b2c3f3474cdf7cfebbdb8575059e036d9cc.1686134440.git.me@ttaylorr.com","subject":"Re: [PATCH v3 12/16] refs/packed-backend.c: ignore complicated hidden refs rules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-14T00:40:15Z","receivedAt":"2023-06-14T00:40:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\n> use the new skip-list feature in the packed-refs iterator by ignoring\n\n\"skip\" -> \"jump\".\n\n> references which are mentioned via its respective hideRefs lists.\n>\n> However, the packed-ref skip lists cannot handle un-hiding rules (that\n> begin with '!'), or namespace comparisons (that begin with '^'). Detect\n> and avoid these cases by falling back to the normal enumeration without\n> a skip list when such patterns exist.\n\n> +\t\t * We also can't feed any excludes from hidden refs\n> +\t\t * config sections, since later rules may override\n> +\t\t * previous ones. For example, with rules \"refs/foo\" and\n> +\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n> +\t\t * everything underneath it), but the earlier exclusion\n> +\t\t * would cause us to skip all of \"refs/foo\".\n\nGood observation.  The presence of !refs/foo/bar in hide list\nforbids us from adding refs/foo to the jump list, and it is the\nsimplest to disable the whole jump list business when we have such a\nfeature in use.\n\nMakes sense.  Thanks.\n"},{"id":"478540","messageId":"ZJGTFcYf7G05KHad@nand.local","threadId":"59715","inReplyTo":"xmqqsfav2alu.fsf@gitster.g","subject":"Re: [PATCH v3 07/16] refs: plumb `exclude_patterns` argument throughout","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T11:52:53Z","receivedAt":"2023-06-20T11:53:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 13, 2023 at 04:42:05PM -0700, Junio C Hamano wrote:\n> > diff --git a/ref-filter.c b/ref-filter.c\n> > index d44418efb7..717c3c4bcf 100644\n> > --- a/ref-filter.c\n> > +++ b/ref-filter.c\n> > @@ -2209,12 +2209,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> >\n> >  \tif (!filter->name_patterns[0]) {\n> >  \t\t/* no patterns; we have to look at everything */\n> > -\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n> > +\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> > +\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n> >  \t}\n>\n> Is this merely \"while at it\", or was there a reason why refs_*\n> variant must be used here?\n\nGood question. This changes later in the series (via\n\"refs/packed-backend.c: implement jump lists to avoid excluded\npattern(s)\") to pass the excluded patterns from the ref_filter.\n\nThere's no need to change it in this patch, since the functionality at\nthis point is equivalent in the pre- and post-image. I think this\nstaging is a consequence of having written much of this series before\ncommitting anything, and then trying to segment it out into meaningful\npatches after the fact.\n\n> It is curious that we do not teach the exclude_patterns to some\n> functions like for_each_fullref_in() while adding exclude_patterns to\n> others, making the API surface uneven.\n\nWe could plumb it through in more places, but my preference would be to\nmodify the API if/as new callers need to pass a list of excluded\npatterns. The API has an enormous amount of surface area (and many\nfunctions which take a ton of arguments), so I'd rather keep it small as\nlong as possible, even at the expense of some unevenness in its\ninterface.\n\nThanks,\nTaylor\n"},{"id":"478542","messageId":"ZJGV/bYDN9Dn4V8S@nand.local","threadId":"59715","inReplyTo":"xmqq352u3n3a.fsf@gitster.g","subject":"Re: [PATCH v3 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T12:05:17Z","receivedAt":"2023-06-20T12:05:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 13, 2023 at 05:27:05PM -0700, Junio C Hamano wrote:\n> >   - Clean up the results from the previous step: discard empty regions,\n> >     and combine adjacent regions.\n>\n> Say something like\n>\n>    The resulting list of regions that would never contain refs that\n>    are not excluded is called the \"jump list\".\n>\n> here, probably.  Otherwise the first reference to \"jump list\" we see\n> later feels a bit too abrupt.\n\nGood suggestion. I phrased it slightly differently in the version that\nI'll send shortly, but the spirit is the same.\n\n> > There are a few other gotchas worth considering. First, note that the\n> > jump list is sorted, so once we jump past a region, we can avoid\n> > considering it (or any regions preceding it) again. The member\n> > `jump_pos` is used to track the first next-possible region to jump\n> > through.\n> >\n> > Second, note that the exclusion list is best-effort, since we do not\n> > handle loose references, and because of the meta-character issue above.\n>\n> I found this a bit misleading; a natural reading of \"exclusion list\"\n> is that the phrase refers to the list of exclude patterns given from\n> the command line, and users would be upset if the processing of it\n> is best effort.\n>\n> I think what you meant to say was that optimization to avoid full\n> scan using the jump list does not aim for perfection, and entries\n> that are not skipped using the jump list may still be excluded by\n> the exclude patterns.\n\nYep, exactly. I think this was from an earlier version from before this\nwas called \"jump lists\", and I missed it when find-and-replacing\nthrough the patch messages. Thanks for spotting.\n\n> > In repositories with a large number of hidden references, the speed-up\n>\n> \"hidden\" -> \"excluded\".  Your final objective to implement the\n> feature to exclude refs using patterns and optimize it using the\n> jump list data may be to implement \"hidden references\", but that\n> hasn't be talked about in the above.  All we have been hearing was\n> that we are optimizing the walk over packed-refs using exclude\n> patterns.\n\nYep, thanks.\n\n> > +static const char *ptr_max(const char *x, const char *y)\n> > +{\n> > +\tif (x > y)\n> > +\t\treturn x;\n> > +\treturn y;\n> > +}\n>\n> Hopefully the compiler would inline the function without being told.\n>\n> These pointers point into the same mmapped region of contiguous\n> memory that holds the contents of the packed-refs file, so\n> comparison between them is always defined.  Good.\n>\n> I wondered if\n>\n> \treturn (x > y) ? x : y;\n>\n> is easier to read, simply because it treats both cases more equally\n> (in other words, as written, (x>y) appears more \"special\"), but that\n> is minor.\n\nYeah, I think that any reasonable compiler would almost certainly inline\nthis, especially at higher optimization levels. But I agree with your\nsuggestion nonetheless, thanks.\n\n> > +\t\t/*\n> > +\t\t * We can't feed any excludes with globs in them to the\n> > +\t\t * refs machinery.  It only understands prefix matching.\n> > +\t\t * We likewise can't even feed the string leading up to\n> > +\t\t * the first meta-character, as something like \"foo[a]\"\n> > +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> > +\t\t * would match that and mark it for exclusion).\n> > +\t\t */\n> > +\t\tif (has_glob_special(*pattern))\n> > +\t\t\tcontinue;\n>\n> OK.  When we have a \"literal\" exclude pattern and another \"glob\"\n> exclude pattern, we can afford to ignore the \"glob\" one when\n> building the jump list and include only the \"literal\" one, because\n> the jump list is used only to skip over entries that obviously can\n> never be in the result, *and* --exclude are additive (i.e. being\n> on jump list because of the \"literal\" pattern is a reason enough to\n> be excluded from the result; matching or not matching the other\n> patterns does not affect the fate of the ref that got excluded due\n> to matching the \"literal\" pattern).\n>\n> Makes sense.\n\nExactly.\n\nThanks,\nTaylor\n"},{"id":"478543","messageId":"ZJGW2NaQQhS+hirp@nand.local","threadId":"59715","inReplyTo":"xmqqttva28a7.fsf@gitster.g","subject":"Re: [PATCH v3 10/16] refs/packed-backend.c: add trace2 counters for jump list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T12:08:56Z","receivedAt":"2023-06-20T12:09:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 13, 2023 at 05:32:16PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > +assert_jumps () {\n> > +\tlocal nr=\"$1\"\n> > +\tlocal trace=\"$2\"\n> > +\n> > +\tgrep -q \"name:jumps_made value:$nr\" $trace\n> > +}\n>\n> You may expect 1 hit and the test will pass if trace had 10 or 12 hits.\n\nOops. Well spotted, thanks.\n\nThanks,\nTaylor\n"},{"id":"478544","messageId":"ZJGYS7D3fusfcou3@nand.local","threadId":"59715","inReplyTo":"ZH7ZpB8eMBZN8q63@ncase","subject":"Re: [PATCH v2 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T12:15:07Z","receivedAt":"2023-06-20T12:15:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 06, 2023 at 09:00:52AM +0200, Patrick Steinhardt wrote:\n> > @@ -785,6 +802,13 @@ struct packed_ref_iterator {\n> >  \t/* The end of the part of the buffer that will be iterated over: */\n> >  \tconst char *eof;\n> >\n> > +\tstruct jump_list_entry {\n> > +\t\tconst char *start;\n> > +\t\tconst char *end;\n> > +\t} *jump;\n> > +\tsize_t jump_nr, jump_alloc;\n> > +\tsize_t jump_pos;\n> >\n> Nit: I had some trouble with `jump_pos` given that it sounds so similar\n> to `iter->pos`, and thus you tend to think that they both apply to the\n> position in the packed-refs file. `jump_curr` or `jump_idx` might help\n> to avoid this confusion.\n\nVery fair, thanks for observing. I went with \"jump_cur\" (as a shorthand\nfor \"cursor\").\n\n> > +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> > +\t\tstruct jump_list_entry *e;\n> > +\n> > +\t\t/*\n> > +\t\t * We can't feed any excludes with globs in them to the\n> > +\t\t * refs machinery.  It only understands prefix matching.\n> > +\t\t * We likewise can't even feed the string leading up to\n> > +\t\t * the first meta-character, as something like \"foo[a]\"\n> > +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> > +\t\t * would match that and mark it for exclusion).\n> > +\t\t */\n> > +\t\tif (has_glob_special(*pattern))\n> > +\t\t\tcontinue;\n> > +\n> > +\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n> > +\n> > +\t\te = &iter->jump[iter->jump_nr++];\n> > +\t\te->start = find_reference_location(snapshot, *pattern, 0);\n> > +\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n>\n> Nit: we could detect the non-matching case here already, which would\n> allow us to skip an allocation. It's probably pre-mature optimization\n> though, so please feel free to ignore.\n\nProbably so, this allocation is so lightweight in comparison to all of\nthe other things that for-each-ref does throughout its execution that I\nthink it's probably negligible to shave off a few allocations.\n\n> > +\t}\n> > +\n> > +\tif (!iter->jump_nr) {\n> > +\t\t/*\n> > +\t\t * Every entry in exclude_patterns has a meta-character,\n> > +\t\t * nothing to do here.\n> > +\t\t */\n> > +\t\treturn;\n> > +\t}\n> > +\n> > +\tQSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);\n> > +\n> > +\t/*\n> > +\t * As an optimization, merge adjacent entries in the jump list\n> > +\t * to jump forwards as far as possible when entering a skipped\n> > +\t * region.\n> > +\t *\n> > +\t * For example, if we have two skipped regions:\n> > +\t *\n> > +\t *\t[[A, B], [B, C]]\n> > +\t *\n> > +\t * we want to combine that into a single entry jumping from A to\n> > +\t * C.\n> > +\t */\n> > +\tlast_disjoint = iter->jump;\n>\n> Nit: if we initialized `j = 0`, then `last_disjoint` would always be\n> equal to `iter->jump[j]`. We could then declare the variable inside of\n> the loop to make it a bit easier to understand.\n\nSure, though we would then need to assign `iter->jump_nr = j + 1`, which\nI think adds more confusion than inlining the variable is worth.\n\nThanks,\nTaylor\n"},{"id":"478545","messageId":"ZJGYs4hWjdMTSMm4@nand.local","threadId":"59715","inReplyTo":"ZH7ZqjYqyvSOHY8u@ncase","subject":"Re: [PATCH v2 11/16] revision.h: store hidden refs in a `strvec`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T12:16:51Z","receivedAt":"2023-06-20T12:16:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 06, 2023 at 09:00:58AM +0200, Patrick Steinhardt wrote:\n> On Mon, May 15, 2023 at 03:23:39PM -0400, Taylor Blau wrote:\n> > In subsequent commits, it will be convenient to have a 'const char **'\n> > of hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\n> > etc.), instead of a `string_list`.\n> >\n> > Convert spots throughout the tree that store the list of hidden refs\n> > from a `string_list` to a `strvec`.\n> >\n> > Note that in `parse_hide_refs_config()` there is an ugly const-cast used\n> > to avoid an extra copy of each value before trimming any trailing slash\n> > characters. This could instead be written as:\n> >\n> >     ref = xstrdup(value);\n> >     len = strlen(ref);\n> >     while (len && ref[len - 1] == '/')\n> >             ref[--len] = '\\0';\n> >     strvec_push(hide_refs, ref);\n> >     free(ref);\n> >\n> > but the double-copy (once when calling `xstrdup()`, and another via\n> > `strvec_push()`) is wasteful.\n>\n> I guess the proper way to fix this would be to introduce something like\n> a `strvec_push_nodup()` function that takes ownership. And in fact this\n> helper exists already, but it's declared as static. So we could get\n> around the ugly cast with a simple change to expose the helper function.\n\nWe could, but I'd prefer to explore doing so outside of this series,\nsince I think strvec_push_nodup() is a little bit of a footgun unless\nyou are thinking carefully about ownership. So making the function\npart of the exposed API may be controversial.\n\nThanks,\nTaylor\n"},{"id":"478546","messageId":"ZJGZDthfQTGsCDh/@nand.local","threadId":"59715","inReplyTo":"ZH7ZsIq5vQIstjAG@ncase","subject":"Re: [PATCH v2 13/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T12:18:22Z","receivedAt":"2023-06-20T12:18:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 06, 2023 at 09:01:04AM +0200, Patrick Steinhardt wrote:\n> On Mon, May 15, 2023 at 03:23:45PM -0400, Taylor Blau wrote:\n> > The following commit will want to call `for_each_namespaced_ref()` with\n> > a list of excluded patterns.\n>\n> This statement isn't quite true as the following commit touches\n> git-receive-pack(1), which doesn't use `for_each_namespaced_ref_()`.\n\nOops, well spotted. This was true when I wrote the series, but of course\nI decided to swap the order of patches 14 and 15 with one another,\nmaking this statement false. Good eyes ;-).\n\nThanks,\nTaylor\n"},{"id":"478547","messageId":"ZJGaESy7zuCqks6D@nand.local","threadId":"59715","inReplyTo":"ZH7ZtmUrlyJ8H0aP@ncase","subject":"Re: [PATCH v2 00/16] refs: implement jump lists for packed backend","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T12:22:41Z","receivedAt":"2023-06-20T12:22:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 06, 2023 at 09:01:10AM +0200, Patrick Steinhardt wrote:\n> On Mon, May 15, 2023 at 03:23:07PM -0400, Taylor Blau wrote:\n> > Here is a reroll of my series to implement jump (née skip) lists for the\n> > packed refs backend.\n> >\n> > Not a ton has changed since last time, but some notable things that have\n> > changed include:\n> >\n> >   - Renaming \"skip lists\" to \"jump lists\" to clarify that this\n> >     implementation does not use the skip list data structure.\n> >   - Patch reorganization, splitting out `find_reference_location_end()`\n> >     more sensibly, rewording patch messages, etc.\n> >   - Addresses feedback from Junio and Patrick Steinhardt's helpful\n> >     reviews.\n> >\n> > As usual, a range-diff is included below for convenience.\n> >\n> > Given that we are expecting -rc0 today, we should aim to not let review\n> > of this topic direct our attention away from testing the release\n> > candidates. We can get more serious about it on the other side of 2.41.\n> >\n> > Thanks in advance for another look.\n>\n> I didn't have many comments in this round. Personally though I'd split\n> up this patch series into two in order to land the individual parts\n> faster, where the first part introduces `git for-each-ref --exclude` and\n> the second part introduces the jump list for the packed-refs backend.\n\nThanks for reviewing, and for the good suggestion. I think that\nsplitting this series in two could be worthwhile, but I'm not sure I\nwant to make this change for v4.\n\nYou could imagine splitting the series there, where the first half would\nimplement the naive version of `for-each-ref --exclude`, and the second\nhalf would implement jump lists, and make upload- and receive-pack take\nadvantage of it.\n\nBut I think that makes the first half trivial and leaves all of the\ncomplexity in this series to the latter half. I suppose that makes it\neasier to graduate the first six or so patches earlier, but they aren't\nreally all that useful without the remaining patches.\n\nAnother split might be:\n\n  - the naive implementation of `for-each-ref --exclude`\n  - jump lists, and using them within `for-each-ref`\n  - teaching upload- and receive-pack to use the jump list\n\nBut juggling this series as three sub-topics feels like it would be\nburdensome to the maintainer ;-). So I think I'd rather leave it as-is,\nunless you feel strongly.\n\nThanks,\nTaylor\n"},{"id":"478552","messageId":"cover.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH v4 00/16] refs: implement jump lists for packed backend","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:20:51Z","receivedAt":"2023-06-20T14:21:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another reroll of my series to implement jump (née skip) lists\nfor the packed refs backend, based on top of the current 'master'.\n\nMost changes are minor, limited to changing variable names, moving\nchanges around between patches and tweaking commit messages for clarity.\nI think that the first 9 or so patches are stable, but we may want some\nmore eyes on the remainder.\n\nAs usual, a range-diff is included below for convenience.\n\nThanks in advance for your review.\n\nJeff King (5):\n  refs.c: rename `ref_filter`\n  ref-filter.h: provide `REF_FILTER_INIT`\n  ref-filter: clear reachable list pointers after freeing\n  ref-filter: add `ref_filter_clear()`\n  ref-filter.c: parameterize match functions over patterns\n\nTaylor Blau (11):\n  builtin/for-each-ref.c: add `--exclude` option\n  refs: plumb `exclude_patterns` argument throughout\n  refs/packed-backend.c: refactor `find_reference_location()`\n  refs/packed-backend.c: implement jump lists to avoid excluded\n    pattern(s)\n  refs/packed-backend.c: add trace2 counters for jump list\n  revision.h: store hidden refs in a `strvec`\n  refs/packed-backend.c: ignore complicated hidden refs rules\n  refs.h: let `for_each_namespaced_ref()` take excluded patterns\n  builtin/receive-pack.c: avoid enumerating hidden references\n  upload-pack.c: avoid enumerating hidden refs where possible\n  ls-refs.c: avoid enumerating hidden refs where possible\n\n Documentation/git-for-each-ref.txt |   6 +\n builtin/branch.c                   |   4 +-\n builtin/for-each-ref.c             |   7 +-\n builtin/receive-pack.c             |   7 +-\n builtin/tag.c                      |   4 +-\n http-backend.c                     |   2 +-\n ls-refs.c                          |   8 +-\n ref-filter.c                       |  63 +++++++--\n ref-filter.h                       |  12 ++\n refs.c                             |  61 ++++----\n refs.h                             |  15 +-\n refs/debug.c                       |   5 +-\n refs/files-backend.c               |   5 +-\n refs/packed-backend.c              | 220 ++++++++++++++++++++++++++---\n refs/refs-internal.h               |   7 +-\n revision.c                         |   4 +-\n revision.h                         |   5 +-\n t/helper/test-reach.c              |   2 +-\n t/helper/test-ref-store.c          |  10 ++\n t/t0041-usage.sh                   |   1 +\n t/t1419-exclude-refs.sh            | 131 +++++++++++++++++\n t/t3402-rebase-merge.sh            |   1 +\n t/t6300-for-each-ref.sh            |  35 +++++\n trace2.h                           |   2 +\n trace2/tr2_ctr.c                   |   5 +\n upload-pack.c                      |  43 ++++--\n 26 files changed, 559 insertions(+), 106 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\nRange-diff against v3:\n 1:  afac948f04 =  1:  c12def5a30 refs.c: rename `ref_filter`\n 2:  b9336e3b77 =  2:  7ce82b6a5a ref-filter.h: provide `REF_FILTER_INIT`\n 3:  fc28b5caaa =  3:  7e6bf7766d ref-filter: clear reachable list pointers after freeing\n 4:  bc5356fe4b =  4:  777e71004d ref-filter: add `ref_filter_clear()`\n 5:  1988ca4c0a =  5:  39e9b0f50d ref-filter.c: parameterize match functions over patterns\n 6:  60d85aa4ad =  6:  c4fd47fd75 builtin/for-each-ref.c: add `--exclude` option\n 7:  c4fe9a1893 !  7:  e6b50c5021 refs: plumb `exclude_patterns` argument throughout\n    @@ ls-refs.c: int ls_refs(struct repository *r, struct packet_reader *request)\n      ## ref-filter.c ##\n     @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n      \n    - \tif (!filter->name_patterns[0]) {\n    - \t\t/* no patterns; we have to look at everything */\n    --\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n    -+\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n    -+\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n    - \t}\n    - \n      \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n      \t\t\t\t\t\t NULL, filter->name_patterns,\n     -\t\t\t\t\t\t cb, cb_data);\n 8:  9cab5e0699 =  8:  a0990b2916 refs/packed-backend.c: refactor `find_reference_location()`\n 9:  8066858bf5 !  9:  386ed468fa refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)\n    @@ Commit message\n             matches.\n     \n           - Clean up the results from the previous step: discard empty regions,\n    -        and combine adjacent regions.\n    +        and combine adjacent regions. The set of regions which remains is\n    +        referred to as the \"jump list\", and never contains any references\n    +        which should be included in the result set.\n     \n         Then when iterating through the `packed-refs` file, if `iter->pos` is\n         ever contained in one of the regions from the previous steps, advance\n    @@ Commit message\n         `jump_pos` is used to track the first next-possible region to jump\n         through.\n     \n    -    Second, note that the exclusion list is best-effort, since we do not\n    -    handle loose references, and because of the meta-character issue above.\n    +    Second, note that the jump list is best-effort, since we do not handle\n    +    loose references, and because of the meta-character issue above. The\n    +    jump list may not skip past all references which won't appear in the\n    +    results, but will never skip over a reference which does appear in the\n    +    result set.\n     \n         In repositories with a large number of hidden references, the speed-up\n         can be significant. Tests here are done with a copy of linux.git with a\n    @@ Commit message\n     \n      ## ref-filter.c ##\n     @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n    + \n      \tif (!filter->name_patterns[0]) {\n      \t\t/* no patterns; we have to look at everything */\n    - \t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n    --\t\t\t\t\t\t \"\", NULL, cb, cb_data);\n    +-\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n    ++\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n     +\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n      \t}\n      \n    @@ refs/packed-backend.c: struct packed_ref_iterator {\n     +\t\tconst char *end;\n     +\t} *jump;\n     +\tsize_t jump_nr, jump_alloc;\n    -+\tsize_t jump_pos;\n    ++\tsize_t jump_cur;\n     +\n      \t/* Scratch space for current values: */\n      \tstruct object_id oid, peeled;\n    @@ refs/packed-backend.c: struct packed_ref_iterator {\n     +\t * Note that each skipped region is considered at most once,\n     +\t * since they are ordered based on their starting position.\n     +\t */\n    -+\twhile (iter->jump_pos < iter->jump_nr) {\n    -+\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_pos];\n    ++\twhile (iter->jump_cur < iter->jump_nr) {\n    ++\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_cur];\n     +\t\tif (iter->pos < curr->start)\n     +\t\t\tbreak; /* not to the next jump yet */\n     +\n    -+\t\titer->jump_pos++;\n    ++\t\titer->jump_cur++;\n     +\t\tif (iter->pos < curr->end) {\n     +\t\t\titer->pos = curr->end;\n     +\t\t\tbreak;\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\treturn 0;\n     +}\n     +\n    -+static const char *ptr_max(const char *x, const char *y)\n    -+{\n    -+\tif (x > y)\n    -+\t\treturn x;\n    -+\treturn y;\n    -+}\n    -+\n     +static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n     +\t\t\t\t\tstruct snapshot *snapshot,\n     +\t\t\t\t\tconst char **excluded_patterns)\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\t\t\tcontinue;\n     +\t\t} else if (ours->start <= last_disjoint->end) {\n     +\t\t\t/* overlapping regions extend the previous one */\n    -+\t\t\tlast_disjoint->end = ptr_max(last_disjoint->end, ours->end);\n    ++\t\t\tlast_disjoint->end = last_disjoint->end > ours->end\n    ++\t\t\t\t? last_disjoint->end : ours->end;\n     +\t\t} else {\n     +\t\t\t/* otherwise, insert a new region */\n     +\t\t\titer->jump[j++] = *ours;\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\t}\n     +\n     +\titer->jump_nr = j;\n    -+\titer->jump_pos = 0;\n    ++\titer->jump_cur = 0;\n     +}\n     +\n      static struct ref_iterator *packed_ref_iterator_begin(\n10:  3c045076a9 ! 10:  49c8f5173a refs/packed-backend.c: add trace2 counters for jump list\n    @@ refs/packed-backend.c\n      enum mmap_strategy {\n      \t/*\n     @@ refs/packed-backend.c: static int next_record(struct packed_ref_iterator *iter)\n    - \t\titer->jump_pos++;\n    + \t\titer->jump_cur++;\n      \t\tif (iter->pos < curr->end) {\n      \t\t\titer->pos = curr->end;\n     +\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);\n    @@ t/t1419-exclude-refs.sh: for_each_ref () {\n     +\tlocal nr=\"$1\"\n     +\tlocal trace=\"$2\"\n     +\n    -+\tgrep -q \"name:jumps_made value:$nr\" $trace\n    ++\tgrep -q \"name:jumps_made value:$nr$\" $trace\n     +}\n     +\n     +assert_no_jumps () {\n    @@ t/t1419-exclude-refs.sh: test_expect_success 'setup' '\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_no_jumps\n    ++\tassert_no_jumps perf\n      '\n      \n      test_expect_success 'meta-characters are discarded' '\n    @@ t/t1419-exclude-refs.sh: test_expect_success 'setup' '\n      \n     -\ttest_cmp expect actual\n     +\ttest_cmp expect actual &&\n    -+\tassert_no_jumps\n    ++\tassert_no_jumps perf\n      '\n      \n      test_done\n11:  0ff542eaef = 11:  dd856a3982 revision.h: store hidden refs in a `strvec`\n12:  ca006b2c3f ! 12:  845904853e refs/packed-backend.c: ignore complicated hidden refs rules\n    @@ Commit message\n         refs/packed-backend.c: ignore complicated hidden refs rules\n     \n         In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\n    -    use the new skip-list feature in the packed-refs iterator by ignoring\n    +    use the new jump list feature in the packed-refs iterator by ignoring\n         references which are mentioned via its respective hideRefs lists.\n     \n    -    However, the packed-ref skip lists cannot handle un-hiding rules (that\n    +    However, the packed-ref jump lists cannot handle un-hiding rules (that\n         begin with '!'), or namespace comparisons (that begin with '^'). Detect\n         and avoid these cases by falling back to the normal enumeration without\n    -    a skip list when such patterns exist.\n    +    a jump list when such patterns exist.\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n    @@ refs/packed-backend.c: static void populate_excluded_jump_list(struct packed_ref\n     \n      ## t/t1419-exclude-refs.sh ##\n     @@ t/t1419-exclude-refs.sh: test_expect_success 'meta-characters are discarded' '\n    - \tassert_no_jumps\n    + \tassert_no_jumps perf\n      '\n      \n     +test_expect_success 'complex hidden ref rules are discarded' '\n13:  cae703a425 ! 13:  8d4d7cc22e refs.h: let `for_each_namespaced_ref()` take excluded patterns\n    @@ Metadata\n      ## Commit message ##\n         refs.h: let `for_each_namespaced_ref()` take excluded patterns\n     \n    -    The following commit will want to call `for_each_namespaced_ref()` with\n    +    A future commit will want to call `for_each_namespaced_ref()` with\n         a list of excluded patterns.\n     \n         We could introduce a variant of that function, say,\n    @@ Commit message\n         introducing a new function.\n     \n         For now, supply NULL for the list of excluded patterns at all callers to\n    -    avoid changing behavior, which we will do in the subsequent commit.\n    +    avoid changing behavior, which we will do in a future change.\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n14:  1db10b76ea = 14:  49c665f9f8 builtin/receive-pack.c: avoid enumerating hidden references\n15:  014243ebe6 = 15:  19bf4a52d6 upload-pack.c: avoid enumerating hidden refs where possible\n16:  e02fe93379 = 16:  ea6cbaf292 ls-refs.c: avoid enumerating hidden refs where possible\n-- \n2.41.0.44.gf2359540d2\n"},{"id":"478553","messageId":"c12def5a30a671590835b70ee3dbd158d1eb5e61.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 01/16] refs.c: rename `ref_filter`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:00Z","receivedAt":"2023-06-20T14:21:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nThe refs machinery has its own implementation of a `ref_filter` (used by\n`for-each-ref`), which is distinct from the `ref-filler.h` API (also\nused by `for-each-ref`, among other things).\n\nRename the one within refs.c to more clearly indicate its purpose.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 881a0da65c..ba63b69090 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -375,8 +375,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,\n \t\t\t\t   oid, flags);\n }\n \n-/* The argument to filter_refs */\n-struct ref_filter {\n+/* The argument to for_each_filter_refs */\n+struct for_each_ref_filter {\n \tconst char *pattern;\n \tconst char *prefix;\n \teach_ref_fn *fn;\n@@ -409,10 +409,11 @@ int ref_exists(const char *refname)\n \treturn refs_ref_exists(get_main_ref_store(the_repository), refname);\n }\n \n-static int filter_refs(const char *refname, const struct object_id *oid,\n-\t\t\t   int flags, void *data)\n+static int for_each_filter_refs(const char *refname,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tint flags, void *data)\n {\n-\tstruct ref_filter *filter = (struct ref_filter *)data;\n+\tstruct for_each_ref_filter *filter = data;\n \n \tif (wildmatch(filter->pattern, refname, 0))\n \t\treturn 0;\n@@ -569,7 +570,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tconst char *prefix, void *cb_data)\n {\n \tstruct strbuf real_pattern = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct for_each_ref_filter filter;\n \tint ret;\n \n \tif (!prefix && !starts_with(pattern, \"refs/\"))\n@@ -589,7 +590,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tfilter.prefix = prefix;\n \tfilter.fn = fn;\n \tfilter.cb_data = cb_data;\n-\tret = for_each_ref(filter_refs, &filter);\n+\tret = for_each_ref(for_each_filter_refs, &filter);\n \n \tstrbuf_release(&real_pattern);\n \treturn ret;\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478554","messageId":"7ce82b6a5a451ff183ba8d35fc0f50acccee8972.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 02/16] ref-filter.h: provide `REF_FILTER_INIT`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:06Z","receivedAt":"2023-06-20T14:21:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nProvide a sane initialization value for `struct ref_filter`, which in a\nsubsequent patch will be used to initialize a new field.\n\nIn the meantime, fix a case in test-reach.c where its `ref_filter` is\nnot even zero-initialized.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c       | 3 +--\n builtin/for-each-ref.c | 3 +--\n builtin/tag.c          | 3 +--\n ref-filter.h           | 3 +++\n t/helper/test-reach.c  | 2 +-\n 5 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex e6c2655af6..7891dec361 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -707,7 +707,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \tint reflog = 0, quiet = 0, icase = 0, force = 0,\n \t    recurse_submodules_explicit = 0;\n \tenum branch_track track;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstatic struct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -765,7 +765,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \n \tsetup_ref_filter_porcelain_msg();\n \n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.kind = FILTER_REFS_BRANCHES;\n \tfilter.abbrev = -1;\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 695fc8f4a5..99ccb73518 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tint maxcount = 0, icase = 0, omit_empty = 0;\n \tstruct ref_array array;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t};\n \n \tmemset(&array, 0, sizeof(array));\n-\tmemset(&filter, 0, sizeof(filter));\n \n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 49b64c7a28..ec778ba860 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -437,7 +437,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tstruct msg_arg msg = { .buf = STRBUF_INIT };\n \tstruct ref_transaction *transaction;\n \tstruct strbuf err = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -496,7 +496,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tgit_config(git_tag_config, &sorting_options);\n \n \tmemset(&opt, 0, sizeof(opt));\n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.lines = -1;\n \topt.sign = -1;\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 430701cfb7..a920f73b29 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,9 @@ struct ref_format {\n \tstruct string_list bases;\n };\n \n+#define REF_FILTER_INIT { \\\n+\t.points_at = OID_ARRAY_INIT, \\\n+}\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n \t.bases = STRING_LIST_INIT_DUP, \\\ndiff --git a/t/helper/test-reach.c b/t/helper/test-reach.c\nindex 5b6f217441..ef58f10c2d 100644\n--- a/t/helper/test-reach.c\n+++ b/t/helper/test-reach.c\n@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)\n \n \t\tprintf(\"%s(X,_,_,0,0):%d\\n\", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));\n \t} else if (!strcmp(av[1], \"commit_contains\")) {\n-\t\tstruct ref_filter filter;\n+\t\tstruct ref_filter filter = REF_FILTER_INIT;\n \t\tstruct contains_cache cache;\n \t\tinit_contains_cache(&cache);\n \n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478555","messageId":"7e6bf7766d020914af53e7d6926f5a6c4c4d0668.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 03/16] ref-filter: clear reachable list pointers after freeing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:11Z","receivedAt":"2023-06-20T14:21:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nIn reach_filter(), we pop all commits from the reachable lists, leaving\nthem empty. But because we're operating on a list pointer that was\npassed by value, the original filter.reachable_from pointer is left\ndangling.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4991cd4f7a..048d277cbf 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2418,13 +2418,13 @@ void ref_array_clear(struct ref_array *array)\n #define EXCLUDE_REACHED 0\n #define INCLUDE_REACHED 1\n static void reach_filter(struct ref_array *array,\n-\t\t\t struct commit_list *check_reachable,\n+\t\t\t struct commit_list **check_reachable,\n \t\t\t int include_reached)\n {\n \tint i, old_nr;\n \tstruct commit **to_clear;\n \n-\tif (!check_reachable)\n+\tif (!*check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n@@ -2434,7 +2434,7 @@ static void reach_filter(struct ref_array *array,\n \t}\n \n \ttips_reachable_from_bases(the_repository,\n-\t\t\t\t  check_reachable,\n+\t\t\t\t  *check_reachable,\n \t\t\t\t  to_clear, array->nr,\n \t\t\t\t  UNINTERESTING);\n \n@@ -2455,8 +2455,8 @@ static void reach_filter(struct ref_array *array,\n \n \tclear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);\n \n-\twhile (check_reachable) {\n-\t\tstruct commit *merge_commit = pop_commit(&check_reachable);\n+\twhile (*check_reachable) {\n+\t\tstruct commit *merge_commit = pop_commit(check_reachable);\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n@@ -2553,8 +2553,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \tclear_contains_cache(&ref_cbdata.no_contains_cache);\n \n \t/*  Filters that need revision walking */\n-\treach_filter(array, filter->reachable_from, INCLUDE_REACHED);\n-\treach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);\n+\treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n+\treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n \n \tsave_commit_buffer = save_commit_buffer_orig;\n \treturn ret;\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478556","messageId":"777e71004d68dc9bb9c4cad2b896acff2c615cbf.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 04/16] ref-filter: add `ref_filter_clear()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:16Z","receivedAt":"2023-06-20T14:21:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nWe did not bother to clean up at all in `git branch` or `git tag`, and\n`git for-each-ref` only cleans up a couple of members.\n\nAdd and call `ref_filter_clear()` when cleaning up a `struct\nref_filter`. Running this patch (without any test changes) indicates a\ncouple of now leak-free tests. This was found by running:\n\n    $ make SANITIZE=leak\n    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate\n\n(Note that the `reachable_from` and `unreachable_from` lists should be\ncleaned as they are used. So this is just covering any case where we\nmight bail before running the reachability check.)\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c        |  1 +\n builtin/for-each-ref.c  |  3 +--\n builtin/tag.c           |  1 +\n ref-filter.c            | 16 ++++++++++++++++\n ref-filter.h            |  3 +++\n t/t0041-usage.sh        |  1 +\n t/t3402-rebase-merge.sh |  1 +\n 7 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 7891dec361..07ee874617 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -858,6 +858,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\tprint_columns(&output, colopts, NULL);\n \t\tstring_list_clear(&output, 0);\n \t\tref_sorting_release(sorting);\n+\t\tref_filter_clear(&filter);\n \t\treturn 0;\n \t} else if (edit_description) {\n \t\tconst char *branch_name;\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 99ccb73518..c01fa6fefe 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&err);\n \tstrbuf_release(&output);\n \tref_array_clear(&array);\n-\tfree_commit_list(filter.with_commit);\n-\tfree_commit_list(filter.no_commit);\n+\tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\n \tstrvec_clear(&vec);\n \treturn 0;\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex ec778ba860..2895ff0e45 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -651,6 +651,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \n cleanup:\n \tref_sorting_release(sorting);\n+\tref_filter_clear(&filter);\n \tstrbuf_release(&buf);\n \tstrbuf_release(&ref);\n \tstrbuf_release(&reflog_msg);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 048d277cbf..d32f426898 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2866,3 +2866,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)\n \n \treturn 0;\n }\n+\n+void ref_filter_init(struct ref_filter *filter)\n+{\n+\tstruct ref_filter blank = REF_FILTER_INIT;\n+\tmemcpy(filter, &blank, sizeof(blank));\n+}\n+\n+void ref_filter_clear(struct ref_filter *filter)\n+{\n+\toid_array_clear(&filter->points_at);\n+\tfree_commit_list(filter->with_commit);\n+\tfree_commit_list(filter->no_commit);\n+\tfree_commit_list(filter->reachable_from);\n+\tfree_commit_list(filter->unreachable_from);\n+\tref_filter_init(filter);\n+}\ndiff --git a/ref-filter.h b/ref-filter.h\nindex a920f73b29..160b807224 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,\n \t\t\t struct ref_format *format,\n \t\t\t struct ref_array *array);\n \n+void ref_filter_init(struct ref_filter *filter);\n+void ref_filter_clear(struct ref_filter *filter);\n+\n #endif /*  REF_FILTER_H  */\ndiff --git a/t/t0041-usage.sh b/t/t0041-usage.sh\nindex c4fc34eb18..9ea974b0c6 100755\n--- a/t/t0041-usage.sh\n+++ b/t/t0041-usage.sh\n@@ -5,6 +5,7 @@ test_description='Test commands behavior when given invalid argument value'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'setup ' '\ndiff --git a/t/t3402-rebase-merge.sh b/t/t3402-rebase-merge.sh\nindex 79b0640c00..e9e03ca4b5 100755\n--- a/t/t3402-rebase-merge.sh\n+++ b/t/t3402-rebase-merge.sh\n@@ -8,6 +8,7 @@ test_description='git rebase --merge test'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n T=\"A quick brown fox\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478557","messageId":"39e9b0f50d07cc75219325e7c7e72a801ca0cf16.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:21Z","receivedAt":"2023-06-20T14:21:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\n`match_pattern()` and `match_name_as_path()` both take a `struct\nref_filter *`, and then store a stack variable `patterns` pointing at\n`filter->patterns`.\n\nThe subsequent patch will add a new array of patterns to match over (the\nexcluded patterns, via a new `git for-each-ref --exclude` option),\ntreating the return value of these functions differently depending on\nwhich patterns are being used to match.\n\nTweak `match_pattern()` and `match_name_as_path()` to take an array of\npatterns to prepare for passing either in.\n\nOnce we start passing either in, `match_pattern()` will have little to\ndo with a particular `struct ref_filter *` instance. To clarify this,\ndrop it from the argument list, and replace it with the only bit of the\n`ref_filter` that we care about (`filter->ignore_case`).\n\nCo-authored-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 16 +++++++++-------\n 1 file changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex d32f426898..6d91c7cb0d 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2104,12 +2104,12 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n  * matches a pattern \"refs/heads/mas\") or a wildcard (e.g. the same ref\n  * matches \"refs/heads/mas*\", too).\n  */\n-static int match_pattern(const struct ref_filter *filter, const char *refname)\n+static int match_pattern(const char **patterns, const char *refname,\n+\t\t\t const int ignore_case)\n {\n-\tconst char **patterns = filter->name_patterns;\n \tunsigned flags = 0;\n \n-\tif (filter->ignore_case)\n+\tif (ignore_case)\n \t\tflags |= WM_CASEFOLD;\n \n \t/*\n@@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n  * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n  * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n  */\n-static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n+static int match_name_as_path(const struct ref_filter *filter,\n+\t\t\t      const char **pattern,\n+\t\t\t      const char *refname)\n {\n-\tconst char **pattern = filter->name_patterns;\n \tint namelen = strlen(refname);\n \tunsigned flags = WM_PATHNAME;\n \n@@ -2165,8 +2166,9 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \tif (!*filter->name_patterns)\n \t\treturn 1; /* No pattern always matches */\n \tif (filter->match_as_path)\n-\t\treturn match_name_as_path(filter, refname);\n-\treturn match_pattern(filter, refname);\n+\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n+\treturn match_pattern(filter->name_patterns, refname,\n+\t\t\t     filter->ignore_case);\n }\n \n /*\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478558","messageId":"c4fd47fd750b0381838aa9fb74a665fb5f89c7f4.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 06/16] builtin/for-each-ref.c: add `--exclude` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:26Z","receivedAt":"2023-06-20T14:21:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When using `for-each-ref`, it is sometimes convenient for the caller to\nbe able to exclude certain parts of the references.\n\nFor example, if there are many `refs/__hidden__/*` references, the\ncaller may want to emit all references *except* the hidden ones.\nCurrently, the only way to do this is to post-process the output, like:\n\n    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'\n\nWhich is do-able, but requires processing a potentially large quantity\nof references.\n\nTeach `git for-each-ref` a new `--exclude=<pattern>` option, which\nexcludes references from the results if they match one or more excluded\npatterns.\n\nThis patch provides a naive implementation where the `ref_filter` still\nsees all references (including ones that it will discard) and is left to\ncheck whether each reference matches any excluded pattern(s) before\nemitting them.\n\nBy culling out references we know the caller doesn't care about, we can\navoid allocating memory for their storage, as well as spending time\nsorting the output (among other things). Even the naive implementation\nprovides a significant speed-up on a modified copy of linux.git (that\nhas a hidden ref pointing at each commit):\n\n    $ hyperfine \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/'\n    Benchmark 1: git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]\n      Range (min … max):   817.7 ms … 823.3 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/\n      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]\n      Range (min … max):   104.7 ms … 109.1 ms    27 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/' ran\n        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"'\n\nSubsequent patches will improve on this by avoiding visiting excluded\nsections of the `packed-refs` file in certain cases.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-for-each-ref.txt |  6 +++++\n builtin/for-each-ref.c             |  1 +\n ref-filter.c                       | 13 +++++++++++\n ref-filter.h                       |  6 +++++\n t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++\n 5 files changed, 61 insertions(+)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 1e215d4e73..5743eb5def 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t\t   [--points-at=<object>]\n \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n+\t\t   [--exclude=<pattern> ...]\n \n DESCRIPTION\n -----------\n@@ -102,6 +103,11 @@ OPTIONS\n \tDo not print a newline after formatted refs where the format expands\n \tto the empty string.\n \n+--exclude=<pattern>::\n+\tIf one or more patterns are given, only refs which do not match\n+\tany excluded pattern(s) are shown. Matching is done using the\n+\tsame rules as `<pattern>` above.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex c01fa6fefe..3384987428 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -47,6 +47,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n+\t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n \t\t\t     N_(\"object\"), N_(\"print only refs which points at the given object\"),\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 6d91c7cb0d..d44418efb7 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2171,6 +2171,15 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \t\t\t     filter->ignore_case);\n }\n \n+static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n+{\n+\tif (!filter->exclude.nr)\n+\t\treturn 0;\n+\tif (filter->match_as_path)\n+\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n+\treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n+}\n+\n /*\n  * This is the same as for_each_fullref_in(), but it tries to iterate\n  * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2338,6 +2347,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \tif (!filter_pattern_match(filter, refname))\n \t\treturn 0;\n \n+\tif (filter_exclude_match(filter, refname))\n+\t\treturn 0;\n+\n \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n \t\treturn 0;\n \n@@ -2877,6 +2889,7 @@ void ref_filter_init(struct ref_filter *filter)\n \n void ref_filter_clear(struct ref_filter *filter)\n {\n+\tstrvec_clear(&filter->exclude);\n \toid_array_clear(&filter->points_at);\n \tfree_commit_list(filter->with_commit);\n \tfree_commit_list(filter->no_commit);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 160b807224..1524bc463a 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -6,6 +6,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -59,6 +60,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\n \tstruct commit_list *no_commit;\n@@ -94,6 +96,7 @@ struct ref_format {\n \n #define REF_FILTER_INIT { \\\n \t.points_at = OID_ARRAY_INIT, \\\n+\t.exclude = STRVEC_INIT, \\\n }\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n@@ -112,6 +115,9 @@ struct ref_format {\n #define OPT_REF_SORT(var) \\\n \tOPT_STRING_LIST(0, \"sort\", (var), \\\n \t\t\tN_(\"key\"), N_(\"field name to sort on\"))\n+#define OPT_REF_FILTER_EXCLUDE(var) \\\n+\tOPT_STRVEC(0, \"exclude\", &(var)->exclude, \\\n+\t\t   N_(\"pattern\"), N_(\"exclude refs which match pattern\"))\n \n /*\n  * API for filtering a set of refs. Based on the type of refs the user\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 5c00607608..7e8d578522 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '\n \ttest_cmp expected actual\n '\n \n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with prefix exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=refs/tags/foo >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/foo/one\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with pattern exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=\"refs/tags/foo/t*\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n cat >expected <<\\EOF\n 'refs/heads/main'\n 'refs/remotes/origin/main'\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478559","messageId":"e6b50c5021983f08291b3f17141e2f68bb809807.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 07/16] refs: plumb `exclude_patterns` argument throughout","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:32Z","receivedAt":"2023-06-20T14:21:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The subsequent patch will want to access an optional `excluded_patterns`\narray within `refs/packed-backend.c` that will cull out certain\nreferences matching any of the given patterns on a best-effort basis.\n\nTo do so, the refs subsystem needs to be updated to pass this value\nacross a number of different locations.\n\nPrepare for a future patch by introducing this plumbing now, passing\nNULLs at top-level APIs in order to make that patch less noisy and more\neasily readable.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.co>\n---\n ls-refs.c             |  2 +-\n ref-filter.c          |  2 +-\n refs.c                | 32 +++++++++++++++++++-------------\n refs.h                |  8 +++++++-\n refs/debug.c          |  5 +++--\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  5 +++--\n refs/refs-internal.h  |  7 ++++---\n revision.c            |  2 +-\n 9 files changed, 42 insertions(+), 26 deletions(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex f385938b64..6f490b2d9c 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  send_ref, &data);\n+\t\t\t\t\t  NULL, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex d44418efb7..5e7ed204dc 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2214,7 +2214,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t cb, cb_data);\n+\t\t\t\t\t\t NULL, cb, cb_data);\n }\n \n /*\ndiff --git a/refs.c b/refs.c\nindex ba63b69090..b4b7165fc0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1526,7 +1526,9 @@ int head_ref(each_ref_fn fn, void *cb_data)\n \n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n+\t\tconst char *prefix,\n+\t\tconst char **exclude_patterns,\n+\t\tint trim,\n \t\tenum do_for_each_ref_flags flags)\n {\n \tstruct ref_iterator *iter;\n@@ -1542,8 +1544,7 @@ struct ref_iterator *refs_ref_iterator_begin(\n \t\t}\n \t}\n \n-\titer = refs->be->iterator_begin(refs, prefix, flags);\n-\n+\titer = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);\n \t/*\n \t * `iterator_begin()` already takes care of prefix, but we\n \t * might need to do some trimming:\n@@ -1577,7 +1578,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);\n \n \treturn do_for_each_repo_ref_iterator(r, iter, fn, cb_data);\n }\n@@ -1599,6 +1600,7 @@ static int do_for_each_ref_helper(struct repository *r,\n }\n \n static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n+\t\t\t   const char **exclude_patterns,\n \t\t\t   each_ref_fn fn, int trim,\n \t\t\t   enum do_for_each_ref_flags flags, void *cb_data)\n {\n@@ -1608,7 +1610,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,\n+\t\t\t\t       flags);\n \n \treturn do_for_each_repo_ref_iterator(the_repository, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n@@ -1616,7 +1619,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \n int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n@@ -1627,7 +1630,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n@@ -1638,13 +1641,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t       prefix, fn, 0, 0, cb_data);\n+\t\t\t       prefix, NULL, fn, 0, 0, cb_data);\n }\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);\n }\n \n int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n@@ -1661,14 +1665,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\n \n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0,\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n \t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n@@ -1738,6 +1742,7 @@ static void find_longest_prefixes(struct string_list *out,\n int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \t\t\t\t      const char *namespace,\n \t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data)\n {\n \tstruct string_list prefixes = STRING_LIST_INIT_DUP;\n@@ -1753,7 +1758,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \n \tfor_each_string_list_item(prefix, &prefixes) {\n \t\tstrbuf_addstr(&buf, prefix->string);\n-\t\tret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);\n+\t\tret = refs_for_each_fullref_in(ref_store, buf.buf,\n+\t\t\t\t\t       exclude_patterns, fn, cb_data);\n \t\tif (ret)\n \t\t\tbreak;\n \t\tstrbuf_setlen(&buf, namespace_len);\n@@ -2408,7 +2414,7 @@ int refs_verify_refname_available(struct ref_store *refs,\n \tstrbuf_addstr(&dirname, refname + dirname.len);\n \tstrbuf_addch(&dirname, '/');\n \n-\titer = refs_ref_iterator_begin(refs, dirname.buf, 0,\n+\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 933fdebe58..3c03b035a3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -344,6 +344,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);\n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data);\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n@@ -351,10 +352,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n  * iterate all refs in \"patterns\" by partitioning patterns into disjoint sets\n  * and iterating the longest-common prefix of each set.\n  *\n+ * references matching any pattern in \"exclude_patterns\" are omitted from the\n+ * result set on a best-effort basis.\n+ *\n  * callers should be prepared to ignore references that they did not ask for.\n  */\n int refs_for_each_fullref_in_prefixes(struct ref_store *refs,\n-\t\t\t\t      const char *namespace, const char **patterns,\n+\t\t\t\t      const char *namespace,\n+\t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data);\n \n /**\ndiff --git a/refs/debug.c b/refs/debug.c\nindex c0fa707a1d..b7ffc4ce67 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -229,11 +229,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \n static struct ref_iterator *\n debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n-\t\t\t unsigned int flags)\n+\t\t\t const char **exclude_patterns, unsigned int flags)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n \tstruct ref_iterator *res =\n-\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix, flags);\n+\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix,\n+\t\t\t\t\t\texclude_patterns, flags);\n \tstruct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));\n \tbase_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);\n \tditer->iter = res;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 9a8333c0d0..0a60037530 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -831,7 +831,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \n static struct ref_iterator *files_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct files_ref_store *refs;\n \tstruct ref_iterator *loose_iter, *packed_iter, *overlay_iter;\n@@ -876,7 +877,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * the packed and loose references.\n \t */\n \tpacked_iter = refs_ref_iterator_begin(\n-\t\t\trefs->packed_ref_store, prefix, 0,\n+\t\t\trefs->packed_ref_store, prefix, exclude_patterns, 0,\n \t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n \n \toverlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 291e53f5cf..6855c9a237 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -924,7 +924,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n@@ -1149,7 +1150,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t * list of refs is exhausted, set iter to NULL. When the list\n \t * of updates is exhausted, leave i set to updates->nr.\n \t */\n-\titer = packed_ref_iterator_begin(&refs->base, \"\",\n+\titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n \tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n \t\titer = NULL;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f72b7be894..9db8aec4da 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n  */\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n-\t\tenum do_for_each_ref_flags flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tint trim, enum do_for_each_ref_flags flags);\n \n /*\n  * A callback function used to instruct merge_ref_iterator how to\n@@ -571,7 +571,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,\n  */\n typedef struct ref_iterator *ref_iterator_begin_fn(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags);\n \n /* reflog functions */\n \ndiff --git a/revision.c b/revision.c\nindex b33cc1d106..89953592f9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2670,7 +2670,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,\n \tstruct strbuf bisect_refs = STRBUF_INIT;\n \tint status;\n \tstrbuf_addf(&bisect_refs, \"refs/bisect/%s\", term);\n-\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);\n+\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);\n \tstrbuf_release(&bisect_refs);\n \treturn status;\n }\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478560","messageId":"a0990b2916c36cd8e52cafe71225478879f13433.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 08/16] refs/packed-backend.c: refactor `find_reference_location()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:37Z","receivedAt":"2023-06-20T14:22:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `find_reference_location()` is used to perform a\nbinary search-like function over the contents of a repository's\n`$GIT_DIR/packed-refs` file.\n\nThe search it implements is unlike a standard binary search in that the\nrecords it searches over are not of a fixed width, so the comparison\nmust locate the end of a record before comparing it.\n\nExtract the core routine of `find_reference_location()` in order to\nimplement a function in the following patch which will find the first\nlocation in the `packed-refs` file that *doesn't* match the given\npattern.\n\nThe behavior of `find_reference_location()` is unchanged.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c | 38 ++++++++++++++++++++++----------------\n 1 file changed, 22 insertions(+), 16 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 6855c9a237..d9b61d9e03 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -527,22 +527,8 @@ static int load_contents(struct snapshot *snapshot)\n \treturn 1;\n }\n \n-/*\n- * Find the place in `snapshot->buf` where the start of the record for\n- * `refname` starts. If `mustexist` is true and the reference doesn't\n- * exist, then return NULL. If `mustexist` is false and the reference\n- * doesn't exist, then return the point where that reference would be\n- * inserted, or `snapshot->eof` (which might be NULL) if it would be\n- * inserted at the end of the file. In the latter mode, `refname`\n- * doesn't have to be a proper reference name; for example, one could\n- * search for \"refs/replace/\" to find the start of any replace\n- * references.\n- *\n- * The record is sought using a binary search, so `snapshot->buf` must\n- * be sorted.\n- */\n-static const char *find_reference_location(struct snapshot *snapshot,\n-\t\t\t\t\t   const char *refname, int mustexist)\n+static const char *find_reference_location_1(struct snapshot *snapshot,\n+\t\t\t\t\t     const char *refname, int mustexist)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -588,6 +574,26 @@ static const char *find_reference_location(struct snapshot *snapshot,\n \t\treturn lo;\n }\n \n+/*\n+ * Find the place in `snapshot->buf` where the start of the record for\n+ * `refname` starts. If `mustexist` is true and the reference doesn't\n+ * exist, then return NULL. If `mustexist` is false and the reference\n+ * doesn't exist, then return the point where that reference would be\n+ * inserted, or `snapshot->eof` (which might be NULL) if it would be\n+ * inserted at the end of the file. In the latter mode, `refname`\n+ * doesn't have to be a proper reference name; for example, one could\n+ * search for \"refs/replace/\" to find the start of any replace\n+ * references.\n+ *\n+ * The record is sought using a binary search, so `snapshot->buf` must\n+ * be sorted.\n+ */\n+static const char *find_reference_location(struct snapshot *snapshot,\n+\t\t\t\t\t   const char *refname, int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist);\n+}\n+\n /*\n  * Create a newly-allocated `snapshot` of the `packed-refs` file in\n  * its current state and return it. The return value will already have\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478561","messageId":"386ed468fa7157e90f2f7324199a68974a7860cc.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:42Z","receivedAt":"2023-06-20T14:22:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When iterating through the `packed-refs` file in order to answer a query\nlike:\n\n    $ git for-each-ref --exclude=refs/__hidden__\n\nit would be useful to avoid walking over all of the entries in\n`refs/__hidden__/*` when possible, since we know that the ref-filter\ncode is going to throw them away anyways.\n\nIn certain circumstances, doing so is possible. The algorithm for doing\nso is as follows:\n\n  - For each excluded pattern, find the first record that matches it,\n    and the first record that *doesn't* match it (i.e. the location\n    you'd next want to consider when excluding that pattern).\n\n  - Sort the set of excluded regions from the previous step in ascending\n    order of the first location within the `packed-refs` file that\n    matches.\n\n  - Clean up the results from the previous step: discard empty regions,\n    and combine adjacent regions. The set of regions which remains is\n    referred to as the \"jump list\", and never contains any references\n    which should be included in the result set.\n\nThen when iterating through the `packed-refs` file, if `iter->pos` is\never contained in one of the regions from the previous steps, advance\n`iter->pos` past the end of that region, and continue enumeration.\n\nNote that we only perform this optimization when none of the excluded\npattern(s) have special meta-characters in them. For a pattern like\n\"refs/foo[ac]\", the excluded regions (\"refs/fooa\", \"refs/fooc\", and\neverything underneath them) are not connected. A future implementation\nthat handles this case may split the character class (pretending as if\ntwo patterns were excluded: \"refs/fooa\", and \"refs/fooc\").\n\nThere are a few other gotchas worth considering. First, note that the\njump list is sorted, so once we jump past a region, we can avoid\nconsidering it (or any regions preceding it) again. The member\n`jump_pos` is used to track the first next-possible region to jump\nthrough.\n\nSecond, note that the jump list is best-effort, since we do not handle\nloose references, and because of the meta-character issue above. The\njump list may not skip past all references which won't appear in the\nresults, but will never skip over a reference which does appear in the\nresult set.\n\nIn repositories with a large number of hidden references, the speed-up\ncan be significant. Tests here are done with a copy of linux.git with a\nreference \"refs/pull/N\" pointing at every commit, as in:\n\n    $ git rev-list HEAD | awk '{ print \"create refs/pull/\" NR \" \" $0 }' |\n        git update-ref --stdin\n    $ git pack-refs --all\n\n, it is significantly faster to have `for-each-ref` jump over the\nexcluded references, as opposed to filtering them out after the fact:\n\n    $ hyperfine \\\n      'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n    Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n      Range (min … max):   800.0 ms … 807.7 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n      Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n      Range (min … max):     4.3 ms …   6.7 ms    422 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n      172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n\nUsing the jump list is fairly straightforward (see the changes to\n`refs/packed-backend.c::next_record()`), but constructing the list is\nnot. To ensure that the construction is correct, add a new suite of\ntests in t1419 covering various corner cases (overlapping regions,\npartially overlapping regions, adjacent regions, etc.).\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c              |   6 +-\n refs/packed-backend.c     | 160 ++++++++++++++++++++++++++++++++++++--\n t/helper/test-ref-store.c |  10 +++\n t/t1419-exclude-refs.sh   | 101 ++++++++++++++++++++++++\n 4 files changed, 269 insertions(+), 8 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5e7ed204dc..ddc7f5204f 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2209,12 +2209,14 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n+\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t NULL, cb, cb_data);\n+\t\t\t\t\t\t filter->exclude.v,\n+\t\t\t\t\t\t cb, cb_data);\n }\n \n /*\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex d9b61d9e03..f624c9921a 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -303,7 +303,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)\n  * Compare a snapshot record at `rec` to the specified NUL-terminated\n  * refname.\n  */\n-static int cmp_record_to_refname(const char *rec, const char *refname)\n+static int cmp_record_to_refname(const char *rec, const char *refname,\n+\t\t\t\t int start)\n {\n \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n \tconst char *r2 = refname;\n@@ -312,7 +313,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)\n \t\tif (*r1 == '\\n')\n \t\t\treturn *r2 ? -1 : 0;\n \t\tif (!*r2)\n-\t\t\treturn 1;\n+\t\t\treturn start ? 1 : -1;\n \t\tif (*r1 != *r2)\n \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n \t\tr1++;\n@@ -528,7 +529,8 @@ static int load_contents(struct snapshot *snapshot)\n }\n \n static const char *find_reference_location_1(struct snapshot *snapshot,\n-\t\t\t\t\t     const char *refname, int mustexist)\n+\t\t\t\t\t     const char *refname, int mustexist,\n+\t\t\t\t\t     int start)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -558,7 +560,7 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n \n \t\tmid = lo + (hi - lo) / 2;\n \t\trec = find_start_of_record(lo, mid);\n-\t\tcmp = cmp_record_to_refname(rec, refname);\n+\t\tcmp = cmp_record_to_refname(rec, refname, start);\n \t\tif (cmp < 0) {\n \t\t\tlo = find_end_of_record(mid, hi);\n \t\t} else if (cmp > 0) {\n@@ -591,7 +593,22 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n static const char *find_reference_location(struct snapshot *snapshot,\n \t\t\t\t\t   const char *refname, int mustexist)\n {\n-\treturn find_reference_location_1(snapshot, refname, mustexist);\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n+}\n+\n+/*\n+ * Find the place in `snapshot->buf` after the end of the record for\n+ * `refname`. In other words, find the location of first thing *after*\n+ * `refname`.\n+ *\n+ * Other semantics are identical to the ones in\n+ * `find_reference_location()`.\n+ */\n+static const char *find_reference_location_end(struct snapshot *snapshot,\n+\t\t\t\t\t       const char *refname,\n+\t\t\t\t\t       int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 0);\n }\n \n /*\n@@ -785,6 +802,13 @@ struct packed_ref_iterator {\n \t/* The end of the part of the buffer that will be iterated over: */\n \tconst char *eof;\n \n+\tstruct jump_list_entry {\n+\t\tconst char *start;\n+\t\tconst char *end;\n+\t} *jump;\n+\tsize_t jump_nr, jump_alloc;\n+\tsize_t jump_cur;\n+\n \t/* Scratch space for current values: */\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n@@ -802,14 +826,34 @@ struct packed_ref_iterator {\n  */\n static int next_record(struct packed_ref_iterator *iter)\n {\n-\tconst char *p = iter->pos, *eol;\n+\tconst char *p, *eol;\n \n \tstrbuf_reset(&iter->refname_buf);\n \n+\t/*\n+\t * If iter->pos is contained within a skipped region, jump past\n+\t * it.\n+\t *\n+\t * Note that each skipped region is considered at most once,\n+\t * since they are ordered based on their starting position.\n+\t */\n+\twhile (iter->jump_cur < iter->jump_nr) {\n+\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_cur];\n+\t\tif (iter->pos < curr->start)\n+\t\t\tbreak; /* not to the next jump yet */\n+\n+\t\titer->jump_cur++;\n+\t\tif (iter->pos < curr->end) {\n+\t\t\titer->pos = curr->end;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n \tif (iter->pos == iter->eof)\n \t\treturn ITER_DONE;\n \n \titer->base.flags = REF_ISPACKED;\n+\tp = iter->pos;\n \n \tif (iter->eof - p < the_hash_algo->hexsz + 2 ||\n \t    parse_oid_hex(p, &iter->oid, &p) ||\n@@ -917,6 +961,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \tint ok = ITER_DONE;\n \n \tstrbuf_release(&iter->refname_buf);\n+\tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n \tbase_ref_iterator_free(ref_iterator);\n \treturn ok;\n@@ -928,6 +973,106 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.abort = packed_ref_iterator_abort\n };\n \n+static int jump_list_entry_cmp(const void *va, const void *vb)\n+{\n+\tconst struct jump_list_entry *a = va;\n+\tconst struct jump_list_entry *b = vb;\n+\n+\tif (a->start < b->start)\n+\t\treturn -1;\n+\tif (a->start > b->start)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int has_glob_special(const char *str)\n+{\n+\tconst char *p;\n+\tfor (p = str; *p; p++) {\n+\t\tif (is_glob_special(*p))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n+\t\t\t\t\tstruct snapshot *snapshot,\n+\t\t\t\t\tconst char **excluded_patterns)\n+{\n+\tsize_t i, j;\n+\tconst char **pattern;\n+\tstruct jump_list_entry *last_disjoint;\n+\n+\tif (!excluded_patterns)\n+\t\treturn;\n+\n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\tstruct jump_list_entry *e;\n+\n+\t\t/*\n+\t\t * We can't feed any excludes with globs in them to the\n+\t\t * refs machinery.  It only understands prefix matching.\n+\t\t * We likewise can't even feed the string leading up to\n+\t\t * the first meta-character, as something like \"foo[a]\"\n+\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n+\t\t * would match that and mark it for exclusion).\n+\t\t */\n+\t\tif (has_glob_special(*pattern))\n+\t\t\tcontinue;\n+\n+\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n+\n+\t\te = &iter->jump[iter->jump_nr++];\n+\t\te->start = find_reference_location(snapshot, *pattern, 0);\n+\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n+\t}\n+\n+\tif (!iter->jump_nr) {\n+\t\t/*\n+\t\t * Every entry in exclude_patterns has a meta-character,\n+\t\t * nothing to do here.\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tQSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);\n+\n+\t/*\n+\t * As an optimization, merge adjacent entries in the jump list\n+\t * to jump forwards as far as possible when entering a skipped\n+\t * region.\n+\t *\n+\t * For example, if we have two skipped regions:\n+\t *\n+\t *\t[[A, B], [B, C]]\n+\t *\n+\t * we want to combine that into a single entry jumping from A to\n+\t * C.\n+\t */\n+\tlast_disjoint = iter->jump;\n+\n+\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n+\t\tstruct jump_list_entry *ours = &iter->jump[i];\n+\n+\t\tif (ours->start == ours->end) {\n+\t\t\t/* ignore empty regions (no matching entries) */\n+\t\t\tcontinue;\n+\t\t} else if (ours->start <= last_disjoint->end) {\n+\t\t\t/* overlapping regions extend the previous one */\n+\t\t\tlast_disjoint->end = last_disjoint->end > ours->end\n+\t\t\t\t? last_disjoint->end : ours->end;\n+\t\t} else {\n+\t\t\t/* otherwise, insert a new region */\n+\t\t\titer->jump[j++] = *ours;\n+\t\t\tlast_disjoint = ours;\n+\n+\t\t}\n+\t}\n+\n+\titer->jump_nr = j;\n+\titer->jump_cur = 0;\n+}\n+\n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n \t\tconst char *prefix, const char **exclude_patterns,\n@@ -963,6 +1108,9 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);\n \n+\tif (exclude_patterns)\n+\t\tpopulate_excluded_jump_list(iter, snapshot, exclude_patterns);\n+\n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n \ndiff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\nindex a6977b5e83..08b6d5a86c 100644\n--- a/t/helper/test-ref-store.c\n+++ b/t/helper/test-ref-store.c\n@@ -184,6 +184,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n \treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n }\n \n+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)\n+{\n+\tconst char *prefix = notnull(*argv++, \"prefix\");\n+\tconst char **exclude_patterns = argv;\n+\n+\treturn refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,\n+\t\t\t\t\tNULL);\n+}\n+\n static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\n {\n \tstruct object_id oid = *null_oid();\n@@ -316,6 +325,7 @@ static struct command commands[] = {\n \t{ \"delete-refs\", cmd_delete_refs },\n \t{ \"rename-ref\", cmd_rename_ref },\n \t{ \"for-each-ref\", cmd_for_each_ref },\n+\t{ \"for-each-ref--exclude\", cmd_for_each_ref__exclude },\n \t{ \"resolve-ref\", cmd_resolve_ref },\n \t{ \"verify-ref\", cmd_verify_ref },\n \t{ \"for-each-reflog\", cmd_for_each_reflog },\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nnew file mode 100755\nindex 0000000000..bc534c8ea1\n--- /dev/null\n+++ b/t/t1419-exclude-refs.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='test exclude_patterns functionality in main ref store'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+for_each_ref__exclude () {\n+\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tcut -d ' ' -f 2 actual.raw\n+}\n+\n+for_each_ref () {\n+\tgit for-each-ref --format='%(refname)' \"$@\"\n+}\n+\n+test_expect_success 'setup' '\n+\ttest_commit --no-tag base &&\n+\tbase=\"$(git rev-parse HEAD)\" &&\n+\n+\tfor name in foo bar baz quux\n+\tdo\n+\t\tfor i in 1 2 3\n+\t\tdo\n+\t\t\techo \"create refs/heads/$name/$i $base\" || return 1\n+\t\tdone || return 1\n+\tdone >in &&\n+\techo \"delete refs/heads/main\" >>in &&\n+\n+\tgit update-ref --stdin <in &&\n+\tgit pack-refs --all\n+'\n+\n+test_expect_success 'excluded region in middle' '\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at beginning' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at end' '\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'disjoint excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'adjacent, non-overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'several overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads \\\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'non-matching excluded section' '\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'meta-characters are discarded' '\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478562","messageId":"49c8f5173aab19112afcd762337012a5c2f0f381.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 10/16] refs/packed-backend.c: add trace2 counters for jump list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:52Z","receivedAt":"2023-06-20T14:22:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The previous commit added low-level tests to ensure that the packed-refs\niterator did not enumerate excluded sections of the refspace.\n\nHowever, there was no guarantee that these sections weren't being\nvisited, only that they were being suppressed from the output. To harden\nthese tests, add a trace2 counter which tracks the number of regions\nskipped by the packed-refs iterator, and assert on its value.\n\nSuggested-by: Derrick Stolee <derrickstolee@github.com>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   |  2 ++\n t/t1419-exclude-refs.sh | 59 ++++++++++++++++++++++++++++-------------\n trace2.h                |  2 ++\n trace2/tr2_ctr.c        |  5 ++++\n 4 files changed, 49 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex f624c9921a..80b877e00c 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -12,6 +12,7 @@\n #include \"../chdir-notify.h\"\n #include \"../wrapper.h\"\n #include \"../write-or-die.h\"\n+#include \"../trace2.h\"\n \n enum mmap_strategy {\n \t/*\n@@ -845,6 +846,7 @@ static int next_record(struct packed_ref_iterator *iter)\n \t\titer->jump_cur++;\n \t\tif (iter->pos < curr->end) {\n \t\t\titer->pos = curr->end;\n+\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);\n \t\t\tbreak;\n \t\t}\n \t}\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex bc534c8ea1..5d8c86b657 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -9,7 +9,8 @@ TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n for_each_ref__exclude () {\n-\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tGIT_TRACE2_PERF=1 test-tool ref-store main \\\n+\t\tfor-each-ref--exclude \"$@\" >actual.raw\n \tcut -d ' ' -f 2 actual.raw\n }\n \n@@ -17,6 +18,17 @@ for_each_ref () {\n \tgit for-each-ref --format='%(refname)' \"$@\"\n }\n \n+assert_jumps () {\n+\tlocal nr=\"$1\"\n+\tlocal trace=\"$2\"\n+\n+\tgrep -q \"name:jumps_made value:$nr$\" $trace\n+}\n+\n+assert_no_jumps () {\n+\t! assert_jumps \".*\" \"$1\"\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit --no-tag base &&\n \tbase=\"$(git rev-parse HEAD)\" &&\n@@ -35,67 +47,76 @@ test_expect_success 'setup' '\n '\n \n test_expect_success 'excluded region in middle' '\n-\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at beginning' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at end' '\n-\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'disjoint excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 2 perf\n '\n \n test_expect_success 'adjacent, non-overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'several overlapping excluded regions' '\n \tfor_each_ref__exclude refs/heads \\\n-\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'non-matching excluded section' '\n-\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps perf\n '\n \n test_expect_success 'meta-characters are discarded' '\n-\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps perf\n '\n \n test_done\ndiff --git a/trace2.h b/trace2.h\nindex 4ced30c0db..9452e291f5 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -551,6 +551,8 @@ enum trace2_counter_id {\n \tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n \tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n \n+\tTRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */\n+\n \t/* Add additional counter definitions before here. */\n \tTRACE2_NUMBER_OF_COUNTERS\n };\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nindex b342d3b1a3..50570d0165 100644\n--- a/trace2/tr2_ctr.c\n+++ b/trace2/tr2_ctr.c\n@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER\n \t\t.name = \"test2\",\n \t\t.want_per_thread_events = 1,\n \t},\n+\t[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {\n+\t\t.category = \"packed-refs\",\n+\t\t.name = \"jumps_made\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n \n \t/* Add additional metadata before here. */\n };\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478563","messageId":"dd856a3982bbc01af7b79399808a5e3baace3899.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 11/16] revision.h: store hidden refs in a `strvec`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:21:57Z","receivedAt":"2023-06-20T14:22:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, it will be convenient to have a 'const char **'\nof hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\netc.), instead of a `string_list`.\n\nConvert spots throughout the tree that store the list of hidden refs\nfrom a `string_list` to a `strvec`.\n\nNote that in `parse_hide_refs_config()` there is an ugly const-cast used\nto avoid an extra copy of each value before trimming any trailing slash\ncharacters. This could instead be written as:\n\n    ref = xstrdup(value);\n    len = strlen(ref);\n    while (len && ref[len - 1] == '/')\n            ref[--len] = '\\0';\n    strvec_push(hide_refs, ref);\n    free(ref);\n\nbut the double-copy (once when calling `xstrdup()`, and another via\n`strvec_push()`) is wasteful.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c |  4 ++--\n ls-refs.c              |  6 +++---\n refs.c                 | 11 ++++++-----\n refs.h                 |  4 ++--\n revision.c             |  2 +-\n revision.h             |  5 +++--\n upload-pack.c          | 10 +++++-----\n 7 files changed, 22 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1a31a58367..1a8472eddc 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -90,7 +90,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+static struct strvec hidden_refs = STRVEC_INIT;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -2619,7 +2619,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n-\tstring_list_clear(&hidden_refs, 0);\n+\tstrvec_clear(&hidden_refs);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 6f490b2d9c..8c3181d051 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -72,7 +72,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -155,7 +155,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n-\tstring_list_init_dup(&data.hidden_refs);\n+\tstrvec_init(&data.hidden_refs);\n \n \tgit_config(ls_refs_config, &data);\n \n@@ -197,7 +197,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-\tstring_list_clear(&data.hidden_refs, 0);\n+\tstrvec_clear(&data.hidden_refs);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex b4b7165fc0..1f01fdf9e8 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1427,7 +1427,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n }\n \n int parse_hide_refs_config(const char *var, const char *value, const char *section,\n-\t\t\t   struct string_list *hide_refs)\n+\t\t\t   struct strvec *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1438,22 +1438,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n-\t\tref = xstrdup(value);\n+\n+\t\t/* drop const to remove trailing '/' characters */\n+\t\tref = (char *)strvec_push(hide_refs, value);\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n int ref_is_hidden(const char *refname, const char *refname_full,\n-\t\t  const struct string_list *hide_refs)\n+\t\t  const struct strvec *hide_refs)\n {\n \tint i;\n \n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n-\t\tconst char *match = hide_refs->items[i].string;\n+\t\tconst char *match = hide_refs->v[i];\n \t\tconst char *subject;\n \t\tint neg = 0;\n \t\tconst char *p;\ndiff --git a/refs.h b/refs.h\nindex 3c03b035a3..f091741bfa 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -816,7 +816,7 @@ int update_ref(const char *msg, const char *refname,\n \t       unsigned int flags, enum action_on_err onerr);\n \n int parse_hide_refs_config(const char *var, const char *value, const char *,\n-\t\t\t   struct string_list *);\n+\t\t\t   struct strvec *);\n \n /*\n  * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -826,7 +826,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,\n  * the ref is outside that namespace, the first parameter is NULL. The second\n  * parameter always points to the full ref name.\n  */\n-int ref_is_hidden(const char *, const char *, const struct string_list *);\n+int ref_is_hidden(const char *, const char *, const struct strvec *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/revision.c b/revision.c\nindex 89953592f9..7c9367a266 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1558,7 +1558,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n-\tstring_list_clear(&exclusions->hidden_refs, 0);\n+\tstrvec_clear(&exclusions->hidden_refs);\n \texclusions->hidden_refs_configured = 0;\n }\n \ndiff --git a/revision.h b/revision.h\nindex 25776af381..7f219cde62 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -10,6 +10,7 @@\n #include \"decorate.h\"\n #include \"ident.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"strvec.h\"\n \n /**\n  * The revision walking API offers functions to build a list of revisions\n@@ -95,7 +96,7 @@ struct ref_exclusions {\n \t * Hidden refs is a list of patterns that is to be hidden via\n \t * `ref_is_hidden()`.\n \t */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \t/*\n \t * Indicates whether hidden refs have been configured. This is to\n@@ -110,7 +111,7 @@ struct ref_exclusions {\n  */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n-\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRVEC_INIT, \\\n }\n \n struct oidset;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d3312006a3..1a213ed775 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -69,7 +69,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -127,7 +127,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n-\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+\tstruct strvec hidden_refs = STRVEC_INIT;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -162,7 +162,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n-\tstring_list_clear(&data->hidden_refs, 0);\n+\tstrvec_clear(&data->hidden_refs);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -1170,7 +1170,7 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n+\t\t\tconst struct object_id *oid, const struct strvec *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n@@ -1465,7 +1465,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t  struct string_list *wanted_refs,\n-\t\t\t  struct string_list *hidden_refs,\n+\t\t\t  struct strvec *hidden_refs,\n \t\t\t  struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478564","messageId":"845904853eeae1741d681a35fdd7816ea16b939a.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 12/16] refs/packed-backend.c: ignore complicated hidden refs rules","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:22:02Z","receivedAt":"2023-06-20T14:22:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\nuse the new jump list feature in the packed-refs iterator by ignoring\nreferences which are mentioned via its respective hideRefs lists.\n\nHowever, the packed-ref jump lists cannot handle un-hiding rules (that\nbegin with '!'), or namespace comparisons (that begin with '^'). Detect\nand avoid these cases by falling back to the normal enumeration without\na jump list when such patterns exist.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   | 19 +++++++++++++++++++\n t/t1419-exclude-refs.sh |  9 +++++++++\n 2 files changed, 28 insertions(+)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 80b877e00c..2aeec5c601 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1008,6 +1008,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n \tif (!excluded_patterns)\n \t\treturn;\n \n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\t/*\n+\t\t * We also can't feed any excludes from hidden refs\n+\t\t * config sections, since later rules may override\n+\t\t * previous ones. For example, with rules \"refs/foo\" and\n+\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n+\t\t * everything underneath it), but the earlier exclusion\n+\t\t * would cause us to skip all of \"refs/foo\". We likewise\n+\t\t * don't implement the namespace stripping required for\n+\t\t * '^' rules.\n+\t\t *\n+\t\t * Both are possible to do, but complicated, so avoid\n+\t\t * populating the jump list at all if we see either of\n+\t\t * these patterns.\n+\t\t */\n+\t\tif (**pattern == '!' || **pattern == '^')\n+\t\t\treturn;\n+\t}\n+\n \tfor (pattern = excluded_patterns; *pattern; pattern++) {\n \t\tstruct jump_list_entry *e;\n \ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex 5d8c86b657..f8abf75ab8 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -119,4 +119,13 @@ test_expect_success 'meta-characters are discarded' '\n \tassert_no_jumps perf\n '\n \n+test_expect_success 'complex hidden ref rules are discarded' '\n+\tfor_each_ref__exclude refs/heads refs/heads/foo \"!refs/heads/foo/1\" \\\n+\t\t>actual 2>perf &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps\n+'\n+\n test_done\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478565","messageId":"8d4d7cc22ee5ae8093ff06c127948356dbe6d501.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 13/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:22:07Z","receivedAt":"2023-06-20T14:22:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A future commit will want to call `for_each_namespaced_ref()` with\na list of excluded patterns.\n\nWe could introduce a variant of that function, say,\n`for_each_namespaced_ref_exclude()` which takes the extra parameter, and\nreimplement the original function in terms of that. But all but one\ncaller (in `http-backend.c`) will supply the new parameter, so add the\nnew parameter to `for_each_namespaced_ref()` itself instead of\nintroducing a new function.\n\nFor now, supply NULL for the list of excluded patterns at all callers to\navoid changing behavior, which we will do in a future change.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n http-backend.c | 2 +-\n refs.c         | 5 +++--\n refs.h         | 3 ++-\n upload-pack.c  | 6 +++---\n 4 files changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex ac146d85c5..ad500683c8 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -559,7 +559,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)\n \n \t} else {\n \t\tselect_getanyfile(hdr);\n-\t\tfor_each_namespaced_ref(show_text_ref, &buf);\n+\t\tfor_each_namespaced_ref(NULL, show_text_ref, &buf);\n \t\tsend_strbuf(hdr, \"text/plain\", &buf);\n \t}\n \tstrbuf_release(&buf);\ndiff --git a/refs.c b/refs.c\nindex 1f01fdf9e8..8613184703 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1660,13 +1660,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat\n \t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, exclude_patterns, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\ndiff --git a/refs.h b/refs.h\nindex f091741bfa..27d341d282 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -378,7 +378,8 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \t\t\t const char *prefix, void *cb_data);\n \n int head_ref_namespaced(each_ref_fn fn, void *cb_data);\n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data);\n \n /* can be used to learn about broken ref and symref */\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 1a213ed775..99d216938c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -855,7 +855,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(check_ref, data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1386,7 +1386,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(send_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n \t\tif (!data.sent_capabilities) {\n \t\t\tconst char *refname = \"capabilities^{}\";\n \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n@@ -1400,7 +1400,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(check_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478566","messageId":"49c665f9f8f889373570ee94f6041e8bd89076cf.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 14/16] builtin/receive-pack.c: avoid enumerating hidden references","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:22:12Z","receivedAt":"2023-06-20T14:22:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that `refs_for_each_fullref_in()` has the ability to avoid\nenumerating references matching certain pattern(s), use that to avoid\nvisiting hidden refs when constructing the ref advertisement via\nreceive-pack.\n\nNote that since this exclusion is best-effort, we still need\n`show_ref_cb()` to check whether or not each reference is hidden or not\nbefore including it in the advertisement.\n\nAs was the case when applying this same optimization to `upload-pack`,\n`receive-pack`'s reference advertisement phase can proceed much quicker\nby avoiding enumerating references that will not be part of the\nadvertisement.\n\n(Below, we're still using linux.git with one hidden refs/pull/N ref per\ncommit):\n\n    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]\n      Range (min … max):    87.7 ms …  95.5 ms    31 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]\n      Range (min … max):     4.1 ms …   5.6 ms    508 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran\n       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 1a8472eddc..bd5bcc375f 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -337,7 +337,8 @@ static void write_head_info(void)\n {\n \tstatic struct oidset seen = OIDSET_INIT;\n \n-\tfor_each_ref(show_ref_cb, &seen);\n+\trefs_for_each_fullref_in(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t hidden_refs.v, show_ref_cb, &seen);\n \tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n \toidset_clear(&seen);\n \tif (!sent_capabilities)\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478567","messageId":"19bf4a52d696b4789e6ab9c62f51d0ba955b7b6a.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 15/16] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:22:17Z","receivedAt":"2023-06-20T14:22:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as a previous commit, teach `upload-pack` to avoid\nenumerating hidden references where possible.\n\nNote, however, that there are certain cases where cannot avoid\nenumerating even hidden references, in particular when either of:\n\n  - `uploadpack.allowTipSHA1InWant`, or\n  - `uploadpack.allowReachableSHA1InWant`\n\nare set, corresponding to `ALLOW_TIP_SHA1` and `ALLOW_REACHABLE_SHA1`,\nrespectively.\n\nWhen either of these bits are set, upload-pack's `is_our_ref()` function\nneeds to consider the `HIDDEN_REF` bit of the referent's object flags.\nSo we must visit all references, including the hidden ones, in order to\nmark their referents with the `HIDDEN_REF` bit.\n\nWhen neither `ALLOW_TIP_SHA1` nor `ALLOW_REACHABLE_SHA1` are set, the\n`is_our_ref()` function considers only the `OUR_REF` bit, and not the\n`HIDDEN_REF` one. `OUR_REF` is applied via `mark_our_ref()`, and only\nto objects at the tips of non-hidden references, so we do not need to\nvisit hidden references in this case.\n\nWhen neither of those bits are set, `upload-pack` can potentially avoid\nenumerating a large number of references. In the same example as a\nprevious commit (linux.git with one hidden reference per commit,\n\"refs/pull/N\"):\n\n    $ printf 0000 >in\n    $ hyperfine --warmup=1 \\\n      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]\n      Range (min … max):   405.7 ms … 409.2 ms    10 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in\n      Time (mean ± σ):     406.5 ms ±   1.3 ms    [User: 356.5 ms, System: 49.9 ms]\n      Range (min … max):   404.6 ms … 408.8 ms    10 runs\n\n    Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):       4.7 ms ±   0.2 ms    [User: 0.7 ms, System: 3.9 ms]\n      Range (min … max):     4.3 ms …   6.1 ms    472 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran\n       86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in'\n       86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'\n\nAs above, we must visit every reference when\nuploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit\nfar fewer references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n upload-pack.c | 33 +++++++++++++++++++++++++++------\n 1 file changed, 27 insertions(+), 6 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 99d216938c..366a101d8d 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -602,11 +602,32 @@ static int get_common_commits(struct upload_pack_data *data,\n \t}\n }\n \n+static int allow_hidden_refs(enum allow_uor allow_uor)\n+{\n+\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n+}\n+\n+static void for_each_namespaced_ref_1(each_ref_fn fn,\n+\t\t\t\t      struct upload_pack_data *data)\n+{\n+\t/*\n+\t * If `data->allow_uor` allows fetching hidden refs, we need to\n+\t * mark all references (including hidden ones), to check in\n+\t * `is_our_ref()` below.\n+\t *\n+\t * Otherwise, we only care about whether each reference's object\n+\t * has the OUR_REF bit set or not, so do not need to visit\n+\t * hidden references.\n+\t */\n+\tif (allow_hidden_refs(data->allow_uor))\n+\t\tfor_each_namespaced_ref(NULL, fn, data);\n+\telse\n+\t\tfor_each_namespaced_ref(data->hidden_refs.v, fn, data);\n+}\n+\n static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n {\n-\tint allow_hidden_ref = (allow_uor &\n-\t\t\t\t(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n-\treturn o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);\n+\treturn o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);\n }\n \n /*\n@@ -855,7 +876,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n+\t\tfor_each_namespaced_ref_1(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1386,7 +1407,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n+\t\tfor_each_namespaced_ref_1(send_ref, &data);\n \t\tif (!data.sent_capabilities) {\n \t\t\tconst char *refname = \"capabilities^{}\";\n \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n@@ -1400,7 +1421,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n+\t\tfor_each_namespaced_ref_1(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.41.0.44.gf2359540d2\n\n"},{"id":"478568","messageId":"ea6cbaf292f59c96bedc304c74a36fe1c53c1c6d.1687270849.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"[PATCH v4 16/16] ls-refs.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-20T14:22:22Z","receivedAt":"2023-06-20T14:22:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach `ls-refs` to avoid\nenumerating hidden references where possible.\n\nAs before, this is linux.git with one hidden reference per commit.\n\n    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'\n    Benchmark 1: git -c protocol.version=2 ls-remote .\n      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]\n      Range (min … max):    88.8 ms …  91.3 ms    32 runs\n\n    Benchmark 2: git.compile -c protocol.version=2 ls-remote .\n      Time (mean ± σ):       6.5 ms ±   0.1 ms    [User: 2.4 ms, System: 4.3 ms]\n      Range (min … max):     6.2 ms …   8.3 ms    397 runs\n\n    Summary\n      'git.compile -c protocol.version=2 ls-remote .' ran\n       13.85 ± 0.33 times faster than 'git -c protocol.version=2 ls-remote .'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ls-refs.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 8c3181d051..c9a723ba89 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -193,7 +193,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  NULL, send_ref, &data);\n+\t\t\t\t\t  data.hidden_refs.v, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-- \n2.41.0.44.gf2359540d2\n"},{"id":"478573","messageId":"xmqqedm6m0ju.fsf@gitster.g","threadId":"59715","inReplyTo":"ZJGV/bYDN9Dn4V8S@nand.local","subject":"Re: [PATCH v3 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-20T18:49:25Z","receivedAt":"2023-06-20T18:49:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n>> > +static const char *ptr_max(const char *x, const char *y)\n>> > +{\n>> > +\tif (x > y)\n>> > +\t\treturn x;\n>> > +\treturn y;\n>> > +}\n>>\n>> Hopefully the compiler would inline the function without being told.\n>>\n>> These pointers point into the same mmapped region of contiguous\n>> memory that holds the contents of the packed-refs file, so\n>> comparison between them is always defined.  Good.\n>>\n>> I wondered if\n>>\n>> \treturn (x > y) ? x : y;\n>>\n>> is easier to read, simply because it treats both cases more equally\n>> (in other words, as written, (x>y) appears more \"special\"), but that\n>> is minor.\n>\n> Yeah, I think that any reasonable compiler would almost certainly inline\n> this, especially at higher optimization levels. But I agree with your\n> suggestion nonetheless, thanks.\n\nHaving seen how this is used (only at a single callsite), I actually\nthink that special casing (x>y) is the right thing to do, especially\nif you inline it in the caller.  That is,\n\n\tif (last_disjoint->end < ours->end)\n\t\tlast_disjoint->end = ours->end;\n\nreads much more naturally than\n\n\tlast_disjoint->end = (last_disjoint->end > ours->end)\n\t\t? last_disjoint->end : ours_end;\n\nas a way to say \"if ours is larger, record it as the largest\nposition we have seen so far\".\n"},{"id":"479106","messageId":"20230703051315.GA3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"c12def5a30a671590835b70ee3dbd158d1eb5e61.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 01/16] refs.c: rename `ref_filter`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T05:13:15Z","receivedAt":"2023-07-03T05:13:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:21:00AM -0400, Taylor Blau wrote:\n\n> From: Jeff King <peff@peff.net>\n> \n> The refs machinery has its own implementation of a `ref_filter` (used by\n> `for-each-ref`), which is distinct from the `ref-filler.h` API (also\n> used by `for-each-ref`, among other things).\n\nSmall typo here: s/filler/filter/\n\n-Peff\n"},{"id":"479107","messageId":"20230703051504.GB3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"7ce82b6a5a451ff183ba8d35fc0f50acccee8972.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 02/16] ref-filter.h: provide `REF_FILTER_INIT`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T05:15:04Z","receivedAt":"2023-07-03T05:15:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:21:06AM -0400, Taylor Blau wrote:\n\n> From: Jeff King <peff@peff.net>\n> \n> Provide a sane initialization value for `struct ref_filter`, which in a\n> subsequent patch will be used to initialize a new field.\n> \n> In the meantime, fix a case in test-reach.c where its `ref_filter` is\n> not even zero-initialized.\n\nThis test-reach case scared me, but it happens to work now because\ncommit_contains() only looks at the one field that we set. So we're not\nfixing a bug, but more like a bug waiting to happen. :)\n\n-Peff\n"},{"id":"479108","messageId":"20230703051625.GC3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"7e6bf7766d020914af53e7d6926f5a6c4c4d0668.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 03/16] ref-filter: clear reachable list pointers after freeing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T05:16:25Z","receivedAt":"2023-07-03T05:16:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:21:11AM -0400, Taylor Blau wrote:\n\n> From: Jeff King <peff@peff.net>\n> \n> In reach_filter(), we pop all commits from the reachable lists, leaving\n> them empty. But because we're operating on a list pointer that was\n> passed by value, the original filter.reachable_from pointer is left\n> dangling.\n\nYep. This isn't a bug (yet) because nobody looks at the now-dangling\npointer. So as with the last patch, we're future-proofing ourselves\nagainst dangerous situations.\n\n-Peff\n"},{"id":"479109","messageId":"20230703051946.GD3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"777e71004d68dc9bb9c4cad2b896acff2c615cbf.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 04/16] ref-filter: add `ref_filter_clear()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T05:19:46Z","receivedAt":"2023-07-03T05:19:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:21:16AM -0400, Taylor Blau wrote:\n\n> From: Jeff King <peff@peff.net>\n> \n> We did not bother to clean up at all in `git branch` or `git tag`, and\n> `git for-each-ref` only cleans up a couple of members.\n> \n> Add and call `ref_filter_clear()` when cleaning up a `struct\n> ref_filter`. Running this patch (without any test changes) indicates a\n> couple of now leak-free tests. This was found by running:\n> \n>     $ make SANITIZE=leak\n>     $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate\n> \n> (Note that the `reachable_from` and `unreachable_from` lists should be\n> cleaned as they are used. So this is just covering any case where we\n> might bail before running the reachability check.)\n\nAnd this is the one that benefits from the earlier future-proofing. :)\n\n(In case anyone is wondering why I am reviewing my own commits, it's\nbecause Taylor and I worked on this topic together off-list, but he\nwrote the commit messages after I dumped a bunch of cleanups on him).\n\n> +void ref_filter_init(struct ref_filter *filter)\n> +{\n> +\tstruct ref_filter blank = REF_FILTER_INIT;\n> +\tmemcpy(filter, &blank, sizeof(blank));\n> +}\n\nI was a little surprised by adding init() here, but we need it at the\nend of clear(). So this is an OK place for it (the other option would be\nin the earlier INIT patch, but it would be unused until now).\n\n-Peff\n"},{"id":"479110","messageId":"20230703052724.GE3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"39e9b0f50d07cc75219325e7c7e72a801ca0cf16.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T05:27:24Z","receivedAt":"2023-07-03T05:27:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:21:21AM -0400, Taylor Blau wrote:\n\n> Once we start passing either in, `match_pattern()` will have little to\n> do with a particular `struct ref_filter *` instance. To clarify this,\n> drop it from the argument list, and replace it with the only bit of the\n> `ref_filter` that we care about (`filter->ignore_case`).\n\nMakes sense, but...\n\n> @@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n>   * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n>   * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n>   */\n> -static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n> +static int match_name_as_path(const struct ref_filter *filter,\n> +\t\t\t      const char **pattern,\n> +\t\t\t      const char *refname)\n\n...wouldn't we then want to do the same for match_name_as_path()?\n\nI.e., this:\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 6aacb99be7..cf10c753e2 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2134,14 +2134,13 @@ static int match_pattern(const char **patterns, const char *refname,\n  * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n  * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n  */\n-static int match_name_as_path(const struct ref_filter *filter,\n-\t\t\t      const char **pattern,\n-\t\t\t      const char *refname)\n+static int match_name_as_path(const char **pattern, const char *refname,\n+\t\t\t      int ignore_case)\n {\n \tint namelen = strlen(refname);\n \tunsigned flags = WM_PATHNAME;\n \n-\tif (filter->ignore_case)\n+\tif (ignore_case)\n \t\tflags |= WM_CASEFOLD;\n \n \tfor (; *pattern; pattern++) {\n@@ -2166,7 +2165,8 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \tif (!*filter->name_patterns)\n \t\treturn 1; /* No pattern always matches */\n \tif (filter->match_as_path)\n-\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n+\t\treturn match_name_as_path(filter->name_patterns, refname,\n+\t\t\t\t\t  filter->ignore_case);\n \treturn match_pattern(filter->name_patterns, refname,\n \t\t\t     filter->ignore_case);\n }\n\nAlso, I noticed that you declared it as \"const int ignore_case\" in\nmatch_pattern(). That's not wrong, but we usually do not bother (it is\npassed by value, so const-ness is irrelevant to the caller, and the\ncompiler can see inside the function that the value is not changed and\noptimize appropriately).\n\n-Peff\n"},{"id":"479111","messageId":"20230703055627.GF3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"386ed468fa7157e90f2f7324199a68974a7860cc.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T05:56:27Z","receivedAt":"2023-07-03T05:56:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:21:42AM -0400, Taylor Blau wrote:\n\n> Second, note that the jump list is best-effort, since we do not handle\n> loose references, and because of the meta-character issue above. The\n> jump list may not skip past all references which won't appear in the\n> results, but will never skip over a reference which does appear in the\n> result set.\n\nI wonder if we should be advertising this in a docstring comment above\nthe relevant function. The problem may be that there are several such\nfunctions. I just think that it's a gotcha that may affect somebody who\nwants to call the function, and they're not going to think to dig up\nthis commit message.\n\n>     $ hyperfine \\\n>       'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n>       'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n>     Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n>       Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n>       Range (min … max):   800.0 ms … 807.7 ms    10 runs\n> \n>     Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n>       Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n>       Range (min … max):     4.3 ms …   6.7 ms    422 runs\n> \n>     Summary\n>       'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n>       172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n\nThis measurement is cheating a little, I think, because the earlier\npatch to implement --exclude sped that up from ~800ms to ~100ms (because\nwe avoid writing and all of the ref-filter malloc slowness for the\nexcluded entries). So the better comparison is between two invocations\nwith \"--exclude\", but before/after this patch. You should still see a\n20x speedup (100ms down to 5).\n\n> @@ -802,14 +826,34 @@ struct packed_ref_iterator {\n>   */\n>  static int next_record(struct packed_ref_iterator *iter)\n>  {\n> -\tconst char *p = iter->pos, *eol;\n> +\tconst char *p, *eol;\n>  \n>  \tstrbuf_reset(&iter->refname_buf);\n>  \n> +\t/*\n> +\t * If iter->pos is contained within a skipped region, jump past\n> +\t * it.\n> +\t *\n> +\t * Note that each skipped region is considered at most once,\n> +\t * since they are ordered based on their starting position.\n> +\t */\n> +\twhile (iter->jump_cur < iter->jump_nr) {\n> +\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_cur];\n> +\t\tif (iter->pos < curr->start)\n> +\t\t\tbreak; /* not to the next jump yet */\n> +\n> +\t\titer->jump_cur++;\n> +\t\tif (iter->pos < curr->end) {\n> +\t\t\titer->pos = curr->end;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n\nIt took me a minute to convince myself that this second \"break\" was\nright. If we get to it, we know that iter->pos (the current record we\nare looking at) is in the current jump region. So it makes sense to\nadvance to curr->end. But might we hit another jump region immediately?\n\nI guess not, because earlier we would have coalesced the jump regions.\nSo either there is a non-excluded entry there _or_ we would have\ncoalesced the later region into a single larger region. So breaking is\nthe right thing to do.\n\n> +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> +\t\tstruct jump_list_entry *e;\n> +\n> +\t\t/*\n> +\t\t * We can't feed any excludes with globs in them to the\n> +\t\t * refs machinery.  It only understands prefix matching.\n> +\t\t * We likewise can't even feed the string leading up to\n> +\t\t * the first meta-character, as something like \"foo[a]\"\n> +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> +\t\t * would match that and mark it for exclusion).\n> +\t\t */\n> +\t\tif (has_glob_special(*pattern))\n> +\t\t\tcontinue;\n\nOK, and here's where we could split \"foo[ac]\" into \"fooa\" and \"foob\" if\nwe wanted. But I think it is a very good idea to leave that out of this\ninitial patch. :)\n\n> +\t/*\n> +\t * As an optimization, merge adjacent entries in the jump list\n> +\t * to jump forwards as far as possible when entering a skipped\n> +\t * region.\n> +\t *\n> +\t * For example, if we have two skipped regions:\n> +\t *\n> +\t *\t[[A, B], [B, C]]\n> +\t *\n> +\t * we want to combine that into a single entry jumping from A to\n> +\t * C.\n> +\t */\n> +\tlast_disjoint = iter->jump;\n> +\n> +\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n> +\t\tstruct jump_list_entry *ours = &iter->jump[i];\n> +\n> +\t\tif (ours->start == ours->end) {\n> +\t\t\t/* ignore empty regions (no matching entries) */\n> +\t\t\tcontinue;\n\nDropping empty regions makes sense, but our iteration starts at \"1\"\n(because the rest of the checks are inherently looking at last_disjoint\nbefore deciding if each region is worth keeping). So we'd fail to throw\naway iter->jump[0] if it is empty, I think.\n\nThat could be fixed here by iterating from 0 and checking for a NULL\nlast_disjoint, but maybe it would be easier to avoid allocating at all\nin the earlier loop, when we find that start == end?\n\n> +\t\t} else if (ours->start <= last_disjoint->end) {\n> +\t\t\t/* overlapping regions extend the previous one */\n> +\t\t\tlast_disjoint->end = last_disjoint->end > ours->end\n> +\t\t\t\t? last_disjoint->end : ours->end;\n\nOK, this covers both ([A,C],[B,D]) via \"<\" and ([A,B],[B,C]) via \"=\".\nGood.\n\n> +\t\t} else {\n> +\t\t\t/* otherwise, insert a new region */\n> +\t\t\titer->jump[j++] = *ours;\n> +\t\t\tlast_disjoint = ours;\n> +\n> +\t\t}\n\nAnd this is the rest. Good. There's an extra blank line here before the\nclosing brace.\n\n-Peff\n"},{"id":"479112","messageId":"20230703055949.GG3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"dd856a3982bbc01af7b79399808a5e3baace3899.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 11/16] revision.h: store hidden refs in a `strvec`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T05:59:49Z","receivedAt":"2023-07-03T06:00:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:21:57AM -0400, Taylor Blau wrote:\n\n> In subsequent commits, it will be convenient to have a 'const char **'\n> of hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\n> etc.), instead of a `string_list`.\n> \n> Convert spots throughout the tree that store the list of hidden refs\n> from a `string_list` to a `strvec`.\n> \n> Note that in `parse_hide_refs_config()` there is an ugly const-cast used\n> to avoid an extra copy of each value before trimming any trailing slash\n> characters. This could instead be written as:\n> \n>     ref = xstrdup(value);\n>     len = strlen(ref);\n>     while (len && ref[len - 1] == '/')\n>             ref[--len] = '\\0';\n>     strvec_push(hide_refs, ref);\n>     free(ref);\n> \n> but the double-copy (once when calling `xstrdup()`, and another via\n> `strvec_push()`) is wasteful.\n\nI saw strvec_push_nodup() suggested here. I'm OK leaving it like this\nfor now, but I do think we'll want that in the long run.\n\n-Peff\n"},{"id":"479113","messageId":"20230703061839.GH3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"845904853eeae1741d681a35fdd7816ea16b939a.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 12/16] refs/packed-backend.c: ignore complicated hidden refs rules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T06:18:39Z","receivedAt":"2023-07-03T06:18:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:22:02AM -0400, Taylor Blau wrote:\n\n> In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\n> use the new jump list feature in the packed-refs iterator by ignoring\n> references which are mentioned via its respective hideRefs lists.\n> \n> However, the packed-ref jump lists cannot handle un-hiding rules (that\n> begin with '!'), or namespace comparisons (that begin with '^'). Detect\n> and avoid these cases by falling back to the normal enumeration without\n> a jump list when such patterns exist.\n\nI'm a fan of punting on such cases to keep things simple and\nincremental. But the location here seems weird to me:\n\n> diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> index 80b877e00c..2aeec5c601 100644\n> --- a/refs/packed-backend.c\n> +++ b/refs/packed-backend.c\n> @@ -1008,6 +1008,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n>  \tif (!excluded_patterns)\n>  \t\treturn;\n>  \n> +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> +\t\t/*\n> +\t\t * We also can't feed any excludes from hidden refs\n> +\t\t * config sections, since later rules may override\n> +\t\t * previous ones. For example, with rules \"refs/foo\" and\n> +\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n> +\t\t * everything underneath it), but the earlier exclusion\n> +\t\t * would cause us to skip all of \"refs/foo\". We likewise\n> +\t\t * don't implement the namespace stripping required for\n> +\t\t * '^' rules.\n> +\t\t *\n> +\t\t * Both are possible to do, but complicated, so avoid\n> +\t\t * populating the jump list at all if we see either of\n> +\t\t * these patterns.\n> +\t\t */\n> +\t\tif (**pattern == '!' || **pattern == '^')\n> +\t\t\treturn;\n> +\t}\n> +\n\nThis is deep in the packed-refs code, but the magic of \"!\" and \"^\" are\nspecific to ref_is_hidden().\n\nSo if I did:\n\n  git for-each-ref --exclude='!refs/heads/foo'\n\nmy understanding is that \"!\" would _not_ have an affect normally, but\nnow it is turning off this optimization.\n\nThe point may be somewhat academic for \"^\", as it is not allowed in a\nrefname anyway. But I don't think \"!\" is forbidden (as stupid as it\nwould be to include it in this way), is it?\n\nIt feels like the hiderefs code should be the one checking for these,\nand then feeding only non-adorned refnames to the \"exclude\" list (though\nthere is no need to un-adorn them; once we see any with either form of\nmagic, we know we cannot use this \"exclude\" feature at all).\n\nSomething along the lines of (you'd want a similar tweak for\nupload-pack):\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 2b2faa5d18..80a6b11c90 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -339,7 +339,8 @@ static void write_head_info(void)\n \tstatic struct oidset seen = OIDSET_INIT;\n \n \trefs_for_each_fullref_in(get_main_ref_store(the_repository), \"\",\n-\t\t\t\t hidden_refs.v, show_ref_cb, &seen);\n+\t\t\t\t hidden_refs_to_excludes(&hidden_refs),\n+\t\t\t\t show_ref_cb, &seen);\n \tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n \toidset_clear(&seen);\n \tif (!sent_capabilities)\ndiff --git a/refs.c b/refs.c\nindex 3065e514fd..213412efd4 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1482,6 +1482,31 @@ int ref_is_hidden(const char *refname, const char *refname_full,\n \treturn 0;\n }\n \n+const char **hidden_refs_to_excludes(const struct strvec *hide_refs)\n+{\n+\tconst char **pattern;\n+\n+\tfor (pattern = hide_refs->v; *pattern; pattern++) {\n+\t\t/*\n+\t\t * We also can't feed any excludes from hidden refs\n+\t\t * config sections, since later rules may override\n+\t\t * previous ones. For example, with rules \"refs/foo\" and\n+\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n+\t\t * everything underneath it), but the earlier exclusion\n+\t\t * would cause us to skip all of \"refs/foo\". We likewise\n+\t\t * don't implement the namespace stripping required for\n+\t\t * '^' rules.\n+\t\t *\n+\t\t * Both are possible to do, but complicated, so avoid\n+\t\t * populating the jump list at all if we see either of\n+\t\t * these patterns.\n+\t\t */\n+\t\tif (**pattern == '!' || **pattern == '^')\n+\t\t\treturn NULL;\n+\t}\n+\treturn hide_refs->v;\n+}\n+\n const char *find_descendant_ref(const char *dirname,\n \t\t\t\tconst struct string_list *extras,\n \t\t\t\tconst struct string_list *skip)\ndiff --git a/refs.h b/refs.h\nindex 27d341d282..50c92d1f55 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -829,6 +829,8 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,\n  */\n int ref_is_hidden(const char *, const char *, const struct strvec *);\n \n+const char **hidden_refs_to_excludes(const struct strvec *hide_refs);\n+\n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\n \ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 9dd1795bf2..59c3fe9d91 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1009,25 +1009,6 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n \tif (!excluded_patterns)\n \t\treturn;\n \n-\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n-\t\t/*\n-\t\t * We also can't feed any excludes from hidden refs\n-\t\t * config sections, since later rules may override\n-\t\t * previous ones. For example, with rules \"refs/foo\" and\n-\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n-\t\t * everything underneath it), but the earlier exclusion\n-\t\t * would cause us to skip all of \"refs/foo\". We likewise\n-\t\t * don't implement the namespace stripping required for\n-\t\t * '^' rules.\n-\t\t *\n-\t\t * Both are possible to do, but complicated, so avoid\n-\t\t * populating the jump list at all if we see either of\n-\t\t * these patterns.\n-\t\t */\n-\t\tif (**pattern == '!' || **pattern == '^')\n-\t\t\treturn;\n-\t}\n-\n \tfor (pattern = excluded_patterns; *pattern; pattern++) {\n \t\tstruct jump_list_entry *e;\n \n"},{"id":"479114","messageId":"20230703062644.GI3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"19bf4a52d696b4789e6ab9c62f51d0ba955b7b6a.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 15/16] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T06:26:44Z","receivedAt":"2023-07-03T06:26:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:22:17AM -0400, Taylor Blau wrote:\n\n> In a similar fashion as a previous commit, teach `upload-pack` to avoid\n> enumerating hidden references where possible.\n> \n> Note, however, that there are certain cases where cannot avoid\n> enumerating even hidden references, in particular when either of:\n> \n>   - `uploadpack.allowTipSHA1InWant`, or\n>   - `uploadpack.allowReachableSHA1InWant`\n> \n> are set, corresponding to `ALLOW_TIP_SHA1` and `ALLOW_REACHABLE_SHA1`,\n> respectively.\n> \n> When either of these bits are set, upload-pack's `is_our_ref()` function\n> needs to consider the `HIDDEN_REF` bit of the referent's object flags.\n> So we must visit all references, including the hidden ones, in order to\n> mark their referents with the `HIDDEN_REF` bit.\n> \n> When neither `ALLOW_TIP_SHA1` nor `ALLOW_REACHABLE_SHA1` are set, the\n> `is_our_ref()` function considers only the `OUR_REF` bit, and not the\n> `HIDDEN_REF` one. `OUR_REF` is applied via `mark_our_ref()`, and only\n> to objects at the tips of non-hidden references, so we do not need to\n> visit hidden references in this case.\n\nBoth of these are noops if uploadpack.allowAnySHA1InWant is set, or if\nwe are using the v2 protocol. Setting both allowAny and allowTip is\nsufficiently stupid that I don't care much that they lose out on an\noptimization. But I could see somebody setting one of those for the\nbenefit of v0 clients, but then also serving v2 clients (which\neffectively ignore those restrictions).\n\nI guess v2 clients don't hit this code at all (they are handled by\nls-refs, which comes in the next patch). So that just leaves the case\nthat allowAny is set. By itself the optimization should kick in (good).\nWith allowTip or allowReachable it would not, but that is the \"this is\nstupid\" case in which it is OK to fall back to the existing behavior\n(even though we _could_ enable the optimization). OTOH, it would be easy\nto check it, as it's just another bit in allow_uor.\n\nI'm OK going either way.\n\n-Peff\n"},{"id":"479115","messageId":"20230703062748.GJ3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"ea6cbaf292f59c96bedc304c74a36fe1c53c1c6d.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 16/16] ls-refs.c: avoid enumerating hidden refs where possible","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T06:27:48Z","receivedAt":"2023-07-03T06:27:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:22:22AM -0400, Taylor Blau wrote:\n\n> In a similar fashion as in previous commits, teach `ls-refs` to avoid\n> enumerating hidden references where possible.\n> \n> As before, this is linux.git with one hidden reference per commit.\n> \n>     $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'\n>     Benchmark 1: git -c protocol.version=2 ls-remote .\n>       Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]\n>       Range (min … max):    88.8 ms …  91.3 ms    32 runs\n> \n>     Benchmark 2: git.compile -c protocol.version=2 ls-remote .\n>       Time (mean ± σ):       6.5 ms ±   0.1 ms    [User: 2.4 ms, System: 4.3 ms]\n>       Range (min … max):     6.2 ms …   8.3 ms    397 runs\n\nVery nice. I think this may have big real-world consequences for certain\nrepositories on forges (where they may accrue a large number of hidden\nmetadata like refs/pull).\n\n-Peff\n"},{"id":"479116","messageId":"20230703062925.GK3502534@coredump.intra.peff.net","threadId":"59715","inReplyTo":"cover.1687270849.git.me@ttaylorr.com","subject":"Re: [PATCH v4 00/16] refs: implement jump lists for packed backend","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T06:29:25Z","receivedAt":"2023-07-03T06:29:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2023 at 10:20:51AM -0400, Taylor Blau wrote:\n\n> Here is another reroll of my series to implement jump (née skip) lists\n> for the packed refs backend, based on top of the current 'master'.\n> \n> Most changes are minor, limited to changing variable names, moving\n> changes around between patches and tweaking commit messages for clarity.\n> I think that the first 9 or so patches are stable, but we may want some\n> more eyes on the remainder.\n\nI had been avoiding reading this series too carefully, as I was involved\nin many of the early patches. But now it's been long enough that I\nmostly forgot everything, so I could look at it with fresh eyes. ;) Plus\nthe hidden-refs bits at the end were totally new to me.\n\nIt looks good to me overall. I left a few small comments, some of which\nI think probably justify a re-roll.\n\n-Peff\n"},{"id":"479140","messageId":"ZKMAaI3I7NDDatXk@nand.local","threadId":"59715","inReplyTo":"20230703051504.GB3502534@coredump.intra.peff.net","subject":"Re: [PATCH v4 02/16] ref-filter.h: provide `REF_FILTER_INIT`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-03T17:07:52Z","receivedAt":"2023-07-03T17:08:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 03, 2023 at 01:15:04AM -0400, Jeff King wrote:\n> > In the meantime, fix a case in test-reach.c where its `ref_filter` is\n> > not even zero-initialized.\n>\n> This test-reach case scared me, but it happens to work now because\n> commit_contains() only looks at the one field that we set. So we're not\n> fixing a bug, but more like a bug waiting to happen. :)\n\nGood point, I updated the commit message to more clearly signal that\nthis isn't fixing a bug, but rather preventing one.\n\nThanks,\nTaylor\n"},{"id":"479141","messageId":"ZKMBxnQo1MyB7s9e@nand.local","threadId":"59715","inReplyTo":"20230703051946.GD3502534@coredump.intra.peff.net","subject":"Re: [PATCH v4 04/16] ref-filter: add `ref_filter_clear()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-03T17:13:42Z","receivedAt":"2023-07-03T17:13:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 03, 2023 at 01:19:46AM -0400, Jeff King wrote:\n> > +void ref_filter_init(struct ref_filter *filter)\n> > +{\n> > +\tstruct ref_filter blank = REF_FILTER_INIT;\n> > +\tmemcpy(filter, &blank, sizeof(blank));\n> > +}\n>\n> I was a little surprised by adding init() here, but we need it at the\n> end of clear(). So this is an OK place for it (the other option would be\n> in the earlier INIT patch, but it would be unused until now).\n\nI used to write more patches in this style where I would add as much of\na new API as possible as early as possible in the series. But I think\nreviewers seem to have an easier time reviewing API additions in the\nsame patch that adds their caller.\n\nSo I tend to agree that this patch is probably a good spot to introduce\n`ref_filter_init()`. But if you feel strongly, I'm happy to drag it\naround.\n\nThanks,\nTaylor\n"},{"id":"479142","messageId":"ZKMCzow9EH2BJIfp@nand.local","threadId":"59715","inReplyTo":"20230703052724.GE3502534@coredump.intra.peff.net","subject":"Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-03T17:18:06Z","receivedAt":"2023-07-03T17:18:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 03, 2023 at 01:27:24AM -0400, Jeff King wrote:\n> On Tue, Jun 20, 2023 at 10:21:21AM -0400, Taylor Blau wrote:\n>\n> > Once we start passing either in, `match_pattern()` will have little to\n> > do with a particular `struct ref_filter *` instance. To clarify this,\n> > drop it from the argument list, and replace it with the only bit of the\n> > `ref_filter` that we care about (`filter->ignore_case`).\n>\n> Makes sense, but...\n>\n> > @@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n> >   * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n> >   * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n> >   */\n> > -static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n> > +static int match_name_as_path(const struct ref_filter *filter,\n> > +\t\t\t      const char **pattern,\n> > +\t\t\t      const char *refname)\n>\n> ...wouldn't we then want to do the same for match_name_as_path()?\n\nYes, definitely :-). I'm not sure how I missed this, since the patch\nmessage even says that `match_name_as_path()` gets the same treatment as\nthe other function.\n\nBut in any case, I obviously agree (and the diff below makes sense to\nme). Applied.\n\n> Also, I noticed that you declared it as \"const int ignore_case\" in\n> match_pattern(). That's not wrong, but we usually do not bother (it is\n> passed by value, so const-ness is irrelevant to the caller, and the\n> compiler can see inside the function that the value is not changed and\n> optimize appropriately).\n\nIndeed :-).\n\nThanks,\nTaylor\n"},{"id":"479143","messageId":"ZKMD1L6rSph7vb7u@nand.local","threadId":"59715","inReplyTo":"ZKMCzow9EH2BJIfp@nand.local","subject":"Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-03T17:22:28Z","receivedAt":"2023-07-03T17:22:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 03, 2023 at 01:18:06PM -0400, Taylor Blau wrote:\n> On Mon, Jul 03, 2023 at 01:27:24AM -0400, Jeff King wrote:\n> > On Tue, Jun 20, 2023 at 10:21:21AM -0400, Taylor Blau wrote:\n> >\n> > > Once we start passing either in, `match_pattern()` will have little to\n> > > do with a particular `struct ref_filter *` instance. To clarify this,\n> > > drop it from the argument list, and replace it with the only bit of the\n> > > `ref_filter` that we care about (`filter->ignore_case`).\n> >\n> > Makes sense, but...\n> >\n> > > @@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n> > >   * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n> > >   * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n> > >   */\n> > > -static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n> > > +static int match_name_as_path(const struct ref_filter *filter,\n> > > +\t\t\t      const char **pattern,\n> > > +\t\t\t      const char *refname)\n> >\n> > ...wouldn't we then want to do the same for match_name_as_path()?\n>\n> Yes, definitely :-). I'm not sure how I missed this, since the patch\n> message even says that `match_name_as_path()` gets the same treatment as\n> the other function.\n>\n> But in any case, I obviously agree (and the diff below makes sense to\n> me). Applied.\n\nAh, this is missing one more spot (that we wouldn't complain about\nduring a non-DEVELOPER build). This needs to go on top, but I otherwise\nagree:\n\n--- 8< ---\ndiff --git a/ref-filter.c b/ref-filter.c\nindex de85904b8d..845173a904 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2176,7 +2176,8 @@ static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n \tif (!filter->exclude.nr)\n \t\treturn 0;\n \tif (filter->match_as_path)\n-\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n+\t\treturn match_name_as_path(filter->exclude.v, refname,\n+\t\t\t\t\t  filter->ignore_case);\n \treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n }\n--- >8 ---\n\nThanks,\nTaylor\n"},{"id":"479144","messageId":"20230703173241.GA3544413@coredump.intra.peff.net","threadId":"59715","inReplyTo":"ZKMBxnQo1MyB7s9e@nand.local","subject":"Re: [PATCH v4 04/16] ref-filter: add `ref_filter_clear()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T17:32:41Z","receivedAt":"2023-07-03T17:32:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 03, 2023 at 01:13:42PM -0400, Taylor Blau wrote:\n\n> On Mon, Jul 03, 2023 at 01:19:46AM -0400, Jeff King wrote:\n> > > +void ref_filter_init(struct ref_filter *filter)\n> > > +{\n> > > +\tstruct ref_filter blank = REF_FILTER_INIT;\n> > > +\tmemcpy(filter, &blank, sizeof(blank));\n> > > +}\n> >\n> > I was a little surprised by adding init() here, but we need it at the\n> > end of clear(). So this is an OK place for it (the other option would be\n> > in the earlier INIT patch, but it would be unused until now).\n> \n> I used to write more patches in this style where I would add as much of\n> a new API as possible as early as possible in the series. But I think\n> reviewers seem to have an easier time reviewing API additions in the\n> same patch that adds their caller.\n> \n> So I tend to agree that this patch is probably a good spot to introduce\n> `ref_filter_init()`. But if you feel strongly, I'm happy to drag it\n> around.\n\nNope, I don't feel strongly at all. Let's leave it as you have it.\n\n-Peff\n"},{"id":"479145","messageId":"20230703173345.GB3544413@coredump.intra.peff.net","threadId":"59715","inReplyTo":"ZKMD1L6rSph7vb7u@nand.local","subject":"Re: [PATCH v4 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-07-03T17:33:45Z","receivedAt":"2023-07-03T17:33:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 03, 2023 at 01:22:28PM -0400, Taylor Blau wrote:\n\n> On Mon, Jul 03, 2023 at 01:18:06PM -0400, Taylor Blau wrote:\n> > On Mon, Jul 03, 2023 at 01:27:24AM -0400, Jeff King wrote:\n> > > On Tue, Jun 20, 2023 at 10:21:21AM -0400, Taylor Blau wrote:\n> > >\n> > > > Once we start passing either in, `match_pattern()` will have little to\n> > > > do with a particular `struct ref_filter *` instance. To clarify this,\n> > > > drop it from the argument list, and replace it with the only bit of the\n> > > > `ref_filter` that we care about (`filter->ignore_case`).\n> > >\n> > > Makes sense, but...\n> > >\n> > > > @@ -2134,9 +2134,10 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n> > > >   * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n> > > >   * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n> > > >   */\n> > > > -static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n> > > > +static int match_name_as_path(const struct ref_filter *filter,\n> > > > +\t\t\t      const char **pattern,\n> > > > +\t\t\t      const char *refname)\n> > >\n> > > ...wouldn't we then want to do the same for match_name_as_path()?\n> >\n> > Yes, definitely :-). I'm not sure how I missed this, since the patch\n> > message even says that `match_name_as_path()` gets the same treatment as\n> > the other function.\n> >\n> > But in any case, I obviously agree (and the diff below makes sense to\n> > me). Applied.\n> \n> Ah, this is missing one more spot (that we wouldn't complain about\n> during a non-DEVELOPER build). This needs to go on top, but I otherwise\n> agree:\n\nHeh, yes. I was applying them in order, and had to make the same fixup\non top of the next patch. I think that was the only other fallout as I\napplied the rest, though.\n\n-Peff\n"},{"id":"479146","messageId":"ZKMHhUVQ3EckJFIE@nand.local","threadId":"59715","inReplyTo":"20230703055627.GF3502534@coredump.intra.peff.net","subject":"Re: [PATCH v4 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-03T17:38:13Z","receivedAt":"2023-07-03T17:38:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 03, 2023 at 01:56:27AM -0400, Jeff King wrote:\n> On Tue, Jun 20, 2023 at 10:21:42AM -0400, Taylor Blau wrote:\n>\n> > Second, note that the jump list is best-effort, since we do not handle\n> > loose references, and because of the meta-character issue above. The\n> > jump list may not skip past all references which won't appear in the\n> > results, but will never skip over a reference which does appear in the\n> > result set.\n>\n> I wonder if we should be advertising this in a docstring comment above\n> the relevant function. The problem may be that there are several such\n> functions. I just think that it's a gotcha that may affect somebody who\n> wants to call the function, and they're not going to think to dig up\n> this commit message.\n\nGood idea, thanks.\n\n> >     $ hyperfine \\\n> >       'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n> >       'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n> >     Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n> >       Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n> >       Range (min … max):   800.0 ms … 807.7 ms    10 runs\n> >\n> >     Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n> >       Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n> >       Range (min … max):     4.3 ms …   6.7 ms    422 runs\n> >\n> >     Summary\n> >       'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n> >       172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n>\n> This measurement is cheating a little, I think, because the earlier\n> patch to implement --exclude sped that up from ~800ms to ~100ms (because\n> we avoid writing and all of the ref-filter malloc slowness for the\n> excluded entries). So the better comparison is between two invocations\n> with \"--exclude\", but before/after this patch. You should still see a\n> 20x speedup (100ms down to 5).\n\nI agree. I included a build from the previous commit in this benchmark.\nAs expected, it's around ~100ms, but at least it gives readers a clearer\npicture of how performance changes as a result of this patch.\n(\n\n> > @@ -802,14 +826,34 @@ struct packed_ref_iterator {\n> >   */\n> >  static int next_record(struct packed_ref_iterator *iter)\n> >  {\n> > -\tconst char *p = iter->pos, *eol;\n> > +\tconst char *p, *eol;\n> >\n> >  \tstrbuf_reset(&iter->refname_buf);\n> >\n> > +\t/*\n> > +\t * If iter->pos is contained within a skipped region, jump past\n> > +\t * it.\n> > +\t *\n> > +\t * Note that each skipped region is considered at most once,\n> > +\t * since they are ordered based on their starting position.\n> > +\t */\n> > +\twhile (iter->jump_cur < iter->jump_nr) {\n> > +\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_cur];\n> > +\t\tif (iter->pos < curr->start)\n> > +\t\t\tbreak; /* not to the next jump yet */\n> > +\n> > +\t\titer->jump_cur++;\n> > +\t\tif (iter->pos < curr->end) {\n> > +\t\t\titer->pos = curr->end;\n> > +\t\t\tbreak;\n> > +\t\t}\n> > +\t}\n>\n> It took me a minute to convince myself that this second \"break\" was\n> right. If we get to it, we know that iter->pos (the current record we\n> are looking at) is in the current jump region. So it makes sense to\n> advance to curr->end. But might we hit another jump region immediately?\n>\n> I guess not, because earlier we would have coalesced the jump regions.\n> So either there is a non-excluded entry there _or_ we would have\n> coalesced the later region into a single larger region. So breaking is\n> the right thing to do.\n\nExactly. I added a short comment to this effect to hopefully avoid any\nconfusion here.\n\n> > +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> > +\t\tstruct jump_list_entry *e;\n> > +\n> > +\t\t/*\n> > +\t\t * We can't feed any excludes with globs in them to the\n> > +\t\t * refs machinery.  It only understands prefix matching.\n> > +\t\t * We likewise can't even feed the string leading up to\n> > +\t\t * the first meta-character, as something like \"foo[a]\"\n> > +\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n> > +\t\t * would match that and mark it for exclusion).\n> > +\t\t */\n> > +\t\tif (has_glob_special(*pattern))\n> > +\t\t\tcontinue;\n>\n> OK, and here's where we could split \"foo[ac]\" into \"fooa\" and \"foob\" if\n> we wanted. But I think it is a very good idea to leave that out of this\n> initial patch. :)\n\nOh, definitely ;-).\n\n> > +\t/*\n> > +\t * As an optimization, merge adjacent entries in the jump list\n> > +\t * to jump forwards as far as possible when entering a skipped\n> > +\t * region.\n> > +\t *\n> > +\t * For example, if we have two skipped regions:\n> > +\t *\n> > +\t *\t[[A, B], [B, C]]\n> > +\t *\n> > +\t * we want to combine that into a single entry jumping from A to\n> > +\t * C.\n> > +\t */\n> > +\tlast_disjoint = iter->jump;\n> > +\n> > +\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n> > +\t\tstruct jump_list_entry *ours = &iter->jump[i];\n> > +\n> > +\t\tif (ours->start == ours->end) {\n> > +\t\t\t/* ignore empty regions (no matching entries) */\n> > +\t\t\tcontinue;\n>\n> Dropping empty regions makes sense, but our iteration starts at \"1\"\n> (because the rest of the checks are inherently looking at last_disjoint\n> before deciding if each region is worth keeping). So we'd fail to throw\n> away iter->jump[0] if it is empty, I think.\n>\n> That could be fixed here by iterating from 0 and checking for a NULL\n> last_disjoint, but maybe it would be easier to avoid allocating at all\n> in the earlier loop, when we find that start == end?\n\nYeah, I agree with this. I think Patrick made a similar suggestion in an\nearlier response, and I decided not to take it since it makes the patch\nmore verbose.\n\nBut I think that avoiding the empty region special case is worth it.\nThanks, both :-).\n\nThanks,\nTaylor\n"},{"id":"479168","messageId":"ZKRjUkN/ggLBeIUu@nand.local","threadId":"59715","inReplyTo":"20230703061839.GH3502534@coredump.intra.peff.net","subject":"Re: [PATCH v4 12/16] refs/packed-backend.c: ignore complicated hidden refs rules","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-04T18:22:10Z","receivedAt":"2023-07-04T18:22:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 03, 2023 at 02:18:39AM -0400, Jeff King wrote:\n> On Tue, Jun 20, 2023 at 10:22:02AM -0400, Taylor Blau wrote:\n>\n> > In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\n> > use the new jump list feature in the packed-refs iterator by ignoring\n> > references which are mentioned via its respective hideRefs lists.\n> >\n> > However, the packed-ref jump lists cannot handle un-hiding rules (that\n> > begin with '!'), or namespace comparisons (that begin with '^'). Detect\n> > and avoid these cases by falling back to the normal enumeration without\n> > a jump list when such patterns exist.\n>\n> I'm a fan of punting on such cases to keep things simple and\n> incremental. But the location here seems weird to me:\n>\n> > diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> > index 80b877e00c..2aeec5c601 100644\n> > --- a/refs/packed-backend.c\n> > +++ b/refs/packed-backend.c\n> > @@ -1008,6 +1008,25 @@ static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n> >  \tif (!excluded_patterns)\n> >  \t\treturn;\n> >\n> > +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n> > +\t\t/*\n> > +\t\t * We also can't feed any excludes from hidden refs\n> > +\t\t * config sections, since later rules may override\n> > +\t\t * previous ones. For example, with rules \"refs/foo\" and\n> > +\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n> > +\t\t * everything underneath it), but the earlier exclusion\n> > +\t\t * would cause us to skip all of \"refs/foo\". We likewise\n> > +\t\t * don't implement the namespace stripping required for\n> > +\t\t * '^' rules.\n> > +\t\t *\n> > +\t\t * Both are possible to do, but complicated, so avoid\n> > +\t\t * populating the jump list at all if we see either of\n> > +\t\t * these patterns.\n> > +\t\t */\n> > +\t\tif (**pattern == '!' || **pattern == '^')\n> > +\t\t\treturn;\n> > +\t}\n> > +\n>\n> This is deep in the packed-refs code, but the magic of \"!\" and \"^\" are\n> specific to ref_is_hidden().\n>\n> So if I did:\n>\n>   git for-each-ref --exclude='!refs/heads/foo'\n>\n> my understanding is that \"!\" would _not_ have an affect normally, but\n> now it is turning off this optimization.\n\nYeah, that makes sense -- I agree that it is silly to have a reference\nwith \"!\" at the beginning, but since it's allowed we should absolutely\nsupport it.\n\n> Something along the lines of (you'd want a similar tweak for\n> upload-pack):\n\nYep. Here's the extra tweak for upload-pack:\n\n--- 8< ---\ncommit 5a8902731b91a8fc6900586968a79ebc6272e502\nAuthor: Taylor Blau <me@ttaylorr.com>\nDate:   Tue Jul 4 14:21:22 2023 -0400\n\n    fixup! upload-pack.c: avoid enumerating hidden refs where possible\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 3a176a7209..ef2ca36feb 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -610,6 +610,7 @@ static int allow_hidden_refs(enum allow_uor allow_uor)\n static void for_each_namespaced_ref_1(each_ref_fn fn,\n \t\t\t\t      struct upload_pack_data *data)\n {\n+\tconst char **excludes = NULL;\n \t/*\n \t * If `data->allow_uor` allows fetching hidden refs, we need to\n \t * mark all references (including hidden ones), to check in\n@@ -619,12 +620,13 @@ static void for_each_namespaced_ref_1(each_ref_fn fn,\n \t * has the OUR_REF bit set or not, so do not need to visit\n \t * hidden references.\n \t */\n-\tif (allow_hidden_refs(data->allow_uor))\n-\t\tfor_each_namespaced_ref(NULL, fn, data);\n-\telse\n-\t\tfor_each_namespaced_ref(data->hidden_refs.v, fn, data);\n+\tif (!allow_hidden_refs(data->allow_uor))\n+\t\texcludes = hidden_refs_to_excludes(&data->hidden_refs);\n+\n+\tfor_each_namespaced_ref(excludes, fn, data);\n }\n\n+\n static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n {\n \treturn o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);\n--- >8 ---\n\nThanks,\nTaylor\n"},{"id":"479169","messageId":"ZKRoW0epxO4rfn1U@nand.local","threadId":"59715","inReplyTo":"20230703062644.GI3502534@coredump.intra.peff.net","subject":"Re: [PATCH v4 15/16] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-04T18:43:39Z","receivedAt":"2023-07-04T18:43:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 03, 2023 at 02:26:44AM -0400, Jeff King wrote:\n> I guess v2 clients don't hit this code at all (they are handled by\n> ls-refs, which comes in the next patch). So that just leaves the case\n> that allowAny is set. By itself the optimization should kick in (good).\n> With allowTip or allowReachable it would not, but that is the \"this is\n> stupid\" case in which it is OK to fall back to the existing behavior\n> (even though we _could_ enable the optimization). OTOH, it would be easy\n> to check it, as it's just another bit in allow_uor.\n>\n> I'm OK going either way.\n\nAs am I, though I agree that checking for it is not hard. The return\nvalue of `allow_hidden_refs()` as currently written is confusing, so\nhere's an inversion of that function with this suggestion on top:\n\n--- >8 ---\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ef2ca36feb..5a0fa028c6 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -604,7 +604,9 @@ static int get_common_commits(struct upload_pack_data *data,\n\n static int allow_hidden_refs(enum allow_uor allow_uor)\n {\n-\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n+\tif ((allow_uor & ALLOW_ANY_SHA1) == ALLOW_ANY_SHA1)\n+\t\treturn 1;\n+\treturn !(allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n }\n\n static void for_each_namespaced_ref_1(each_ref_fn fn,\n@@ -620,7 +622,7 @@ static void for_each_namespaced_ref_1(each_ref_fn fn,\n \t * has the OUR_REF bit set or not, so do not need to visit\n \t * hidden references.\n \t */\n-\tif (!allow_hidden_refs(data->allow_uor))\n+\tif (allow_hidden_refs(data->allow_uor))\n \t\texcludes = hidden_refs_to_excludes(&data->hidden_refs);\n\n \tfor_each_namespaced_ref(excludes, fn, data);\n@@ -629,7 +631,7 @@ static void for_each_namespaced_ref_1(each_ref_fn fn,\n\n static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n {\n-\treturn o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);\n+\treturn o->flags & ((allow_hidden_refs(allow_uor) ? 0 : HIDDEN_REF) | OUR_REF);\n }\n\n /*\n--- 8< ---\n\nThanks,\nTaylor\n"},{"id":"479351","messageId":"cover.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1683581621.git.me@ttaylorr.com","subject":"[PATCH v5 00/16] refs: implement jump lists for packed backend","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:01Z","receivedAt":"2023-07-10T21:12:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another reroll of my series to implement jump (née skip) lists\nfor the packed refs backend, based on top of the current 'master'.\n\nThis responds to a review of the entire series that Peff gave a couple\nof weeks ago[1], which should satisfy all of the comments that he\nraised, most of which were relatively minor.\n\nThe notable changes from last time are:\n\n  - clarifying some commit messages, notably to explicitly state that\n    certain clean-ups do not indicate existing bugs, but are defensive\n    against potential future ones\n  - polishing up some of the API changes, to avoid having\n    `match_name_as_path()` take a ref_filter pointer.\n  - more timing data and documentation in the first substantive patch\n  - clean-up in the upload-pack.c to make the changes clearer\n\nAs usual, a range-diff is included below for convenience.\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/20230703062925.GK3502534@coredump.intra.peff.net/\n\nJeff King (5):\n  refs.c: rename `ref_filter`\n  ref-filter.h: provide `REF_FILTER_INIT`\n  ref-filter: clear reachable list pointers after freeing\n  ref-filter: add `ref_filter_clear()`\n  ref-filter.c: parameterize match functions over patterns\n\nTaylor Blau (11):\n  builtin/for-each-ref.c: add `--exclude` option\n  refs: plumb `exclude_patterns` argument throughout\n  refs/packed-backend.c: refactor `find_reference_location()`\n  refs/packed-backend.c: implement jump lists to avoid excluded\n    pattern(s)\n  refs/packed-backend.c: add trace2 counters for jump list\n  revision.h: store hidden refs in a `strvec`\n  refs.h: let `for_each_namespaced_ref()` take excluded patterns\n  refs.h: implement `hidden_refs_to_excludes()`\n  builtin/receive-pack.c: avoid enumerating hidden references\n  upload-pack.c: avoid enumerating hidden refs where possible\n  ls-refs.c: avoid enumerating hidden refs where possible\n\n Documentation/git-for-each-ref.txt |   6 +\n builtin/branch.c                   |   4 +-\n builtin/for-each-ref.c             |   7 +-\n builtin/receive-pack.c             |   8 +-\n builtin/tag.c                      |   4 +-\n http-backend.c                     |   2 +-\n ls-refs.c                          |   7 +-\n ref-filter.c                       |  66 +++++++---\n ref-filter.h                       |  12 ++\n refs.c                             |  85 ++++++++----\n refs.h                             |  29 +++-\n refs/debug.c                       |   5 +-\n refs/files-backend.c               |   5 +-\n refs/packed-backend.c              | 204 +++++++++++++++++++++++++----\n refs/refs-internal.h               |   7 +-\n revision.c                         |   4 +-\n revision.h                         |   5 +-\n t/helper/test-reach.c              |   2 +-\n t/helper/test-ref-store.c          |  10 ++\n t/t0041-usage.sh                   |   1 +\n t/t1419-exclude-refs.sh            | 122 +++++++++++++++++\n t/t3402-rebase-merge.sh            |   1 +\n t/t6300-for-each-ref.sh            |  35 +++++\n trace2.h                           |   2 +\n trace2/tr2_ctr.c                   |   5 +\n upload-pack.c                      |  47 +++++--\n 26 files changed, 579 insertions(+), 106 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\nRange-diff against v4:\n 1:  c12def5a30a !  1:  5b5ccc40d6b refs.c: rename `ref_filter`\n    @@ Commit message\n         refs.c: rename `ref_filter`\n     \n         The refs machinery has its own implementation of a `ref_filter` (used by\n    -    `for-each-ref`), which is distinct from the `ref-filler.h` API (also\n    +    `for-each-ref`), which is distinct from the `ref-filter.h` API (also\n         used by `for-each-ref`, among other things).\n     \n         Rename the one within refs.c to more clearly indicate its purpose.\n 2:  7ce82b6a5a4 !  2:  0c4e995f1d3 ref-filter.h: provide `REF_FILTER_INIT`\n    @@ Commit message\n         Provide a sane initialization value for `struct ref_filter`, which in a\n         subsequent patch will be used to initialize a new field.\n     \n    -    In the meantime, fix a case in test-reach.c where its `ref_filter` is\n    -    not even zero-initialized.\n    +    In the meantime, ensure that the `ref_filter` struct used in the\n    +    test-helper's `cmd__reach()` is zero-initialized. The lack of\n    +    initialization is OK, since `commit_contains()` only looks at the single\n    +    `with_commit_tag_algo` field that *is* initialized directly above.\n    +\n    +    So this does not fix a bug, but rather prevents one from biting us in\n    +    the future.\n     \n         Signed-off-by: Jeff King <peff@peff.net>\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n 3:  7e6bf7766d0 !  3:  cea1e88d358 ref-filter: clear reachable list pointers after freeing\n    @@ Metadata\n      ## Commit message ##\n         ref-filter: clear reachable list pointers after freeing\n     \n    -    In reach_filter(), we pop all commits from the reachable lists, leaving\n    -    them empty. But because we're operating on a list pointer that was\n    -    passed by value, the original filter.reachable_from pointer is left\n    -    dangling.\n    +    In `reach_filter()`, we pop all commits from the reachable lists,\n    +    leaving them empty. But because we're operating on a list pointer that\n    +    was passed by value, the original `filter.reachable_from` and\n    +    `filter.unreachable_from` pointers are left dangling.\n    +\n    +    As is the case with the previous commit, nobody touches either of these\n    +    fields after calling `reach_filter()`, so leaving them dangling is OK.\n    +    But this future proofs against dangerous situations.\n     \n         Signed-off-by: Jeff King <peff@peff.net>\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n 4:  777e71004d6 =  4:  00a4532d54d ref-filter: add `ref_filter_clear()`\n 5:  39e9b0f50d0 !  5:  3ab03ac20df ref-filter.c: parameterize match functions over patterns\n    @@ ref-filter.c: static int get_ref_atom_value(struct ref_array_item *ref, int atom\n       */\n     -static int match_pattern(const struct ref_filter *filter, const char *refname)\n     +static int match_pattern(const char **patterns, const char *refname,\n    -+\t\t\t const int ignore_case)\n    ++\t\t\t int ignore_case)\n      {\n     -\tconst char **patterns = filter->name_patterns;\n      \tunsigned flags = 0;\n    @@ ref-filter.c: static int match_pattern(const struct ref_filter *filter, const ch\n       * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n       */\n     -static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n    -+static int match_name_as_path(const struct ref_filter *filter,\n    -+\t\t\t      const char **pattern,\n    -+\t\t\t      const char *refname)\n    ++static int match_name_as_path(const char **pattern, const char *refname,\n    ++\t\t\t      int ignore_case)\n      {\n     -\tconst char **pattern = filter->name_patterns;\n      \tint namelen = strlen(refname);\n      \tunsigned flags = WM_PATHNAME;\n      \n    +-\tif (filter->ignore_case)\n    ++\tif (ignore_case)\n    + \t\tflags |= WM_CASEFOLD;\n    + \n    + \tfor (; *pattern; pattern++) {\n     @@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n      \tif (!*filter->name_patterns)\n      \t\treturn 1; /* No pattern always matches */\n      \tif (filter->match_as_path)\n     -\t\treturn match_name_as_path(filter, refname);\n     -\treturn match_pattern(filter, refname);\n    -+\t\treturn match_name_as_path(filter, filter->name_patterns, refname);\n    ++\t\treturn match_name_as_path(filter->name_patterns, refname,\n    ++\t\t\t\t\t  filter->ignore_case);\n     +\treturn match_pattern(filter->name_patterns, refname,\n     +\t\t\t     filter->ignore_case);\n      }\n 6:  c4fd47fd750 !  6:  aa881ca06fa builtin/for-each-ref.c: add `--exclude` option\n    @@ ref-filter.c: static int filter_pattern_match(struct ref_filter *filter, const c\n     +\tif (!filter->exclude.nr)\n     +\t\treturn 0;\n     +\tif (filter->match_as_path)\n    -+\t\treturn match_name_as_path(filter, filter->exclude.v, refname);\n    ++\t\treturn match_name_as_path(filter->exclude.v, refname,\n    ++\t\t\t\t\t  filter->ignore_case);\n     +\treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n     +}\n     +\n 7:  e6b50c50219 =  7:  81e223de0c8 refs: plumb `exclude_patterns` argument throughout\n 8:  a0990b2916c =  8:  25c099a528c refs/packed-backend.c: refactor `find_reference_location()`\n 9:  386ed468fa7 !  9:  af0ce43cc90 refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)\n    @@ Commit message\n     \n             $ hyperfine \\\n               'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n    +          'git.prev for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' \\\n               'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n             Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n    -          Time (mean ± σ):     802.7 ms ±   2.1 ms    [User: 691.6 ms, System: 147.0 ms]\n    -          Range (min … max):   800.0 ms … 807.7 ms    10 runs\n    +          Time (mean ± σ):     798.1 ms ±   3.3 ms    [User: 687.6 ms, System: 146.4 ms]\n    +          Range (min … max):   794.5 ms … 805.5 ms    10 runs\n     \n    -        Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n    -          Time (mean ± σ):       4.7 ms ±   0.3 ms    [User: 0.7 ms, System: 4.0 ms]\n    -          Range (min … max):     4.3 ms …   6.7 ms    422 runs\n    +        Benchmark 2: git.prev for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n    +          Time (mean ± σ):      98.9 ms ±   1.4 ms    [User: 93.1 ms, System: 5.7 ms]\n    +          Range (min … max):    97.0 ms … 104.0 ms    29 runs\n    +\n    +        Benchmark 3: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n    +          Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.7 ms, System: 3.8 ms]\n    +          Range (min … max):     4.1 ms …   5.8 ms    524 runs\n     \n             Summary\n               'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n    -          172.03 ± 9.60 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n    +           21.87 ± 1.05 times faster than 'git.prev for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n    +          176.52 ± 8.19 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n    +\n    +    (Comparing stock git and this patch isn't quite fair, since an earlier\n    +    commit in this series adds a naive implementation of the `--exclude`\n    +    option. `git.prev` is built from the previous commit and includes this\n    +    naive implementation).\n     \n         Using the jump list is fairly straightforward (see the changes to\n         `refs/packed-backend.c::next_record()`), but constructing the list is\n    @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n      \n      /*\n     \n    + ## refs.h ##\n    +@@ refs.h: int for_each_ref(each_ref_fn fn, void *cb_data);\n    +  */\n    + int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n    + \n    ++/*\n    ++ * references matching any pattern in \"exclude_patterns\" are omitted from the\n    ++ * result set on a best-effort basis.\n    ++ */\n    + int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n    + \t\t\t     const char **exclude_patterns,\n    + \t\t\t     each_ref_fn fn, void *cb_data);\n    +\n      ## refs/packed-backend.c ##\n     @@ refs/packed-backend.c: static int cmp_packed_ref_records(const void *v1, const void *v2)\n       * Compare a snapshot record at `rec` to the specified NUL-terminated\n    @@ refs/packed-backend.c: struct packed_ref_iterator {\n     +\t\titer->jump_cur++;\n     +\t\tif (iter->pos < curr->end) {\n     +\t\t\titer->pos = curr->end;\n    ++\t\t\t/* jumps are coalesced, so only one jump is necessary */\n     +\t\t\tbreak;\n     +\t\t}\n     +\t}\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\n     +\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n     +\t\tstruct jump_list_entry *e;\n    ++\t\tconst char *start, *end;\n     +\n     +\t\t/*\n     +\t\t * We can't feed any excludes with globs in them to the\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\t\tif (has_glob_special(*pattern))\n     +\t\t\tcontinue;\n     +\n    ++\t\tstart = find_reference_location(snapshot, *pattern, 0);\n    ++\t\tend = find_reference_location_end(snapshot, *pattern, 0);\n    ++\n    ++\t\tif (start == end)\n    ++\t\t\tcontinue; /* nothing to jump over */\n    ++\n     +\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n     +\n     +\t\te = &iter->jump[iter->jump_nr++];\n    -+\t\te->start = find_reference_location(snapshot, *pattern, 0);\n    -+\t\te->end = find_reference_location_end(snapshot, *pattern, 0);\n    ++\t\te->start = start;\n    ++\t\te->end = end;\n     +\t}\n     +\n     +\tif (!iter->jump_nr) {\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\n     +\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n     +\t\tstruct jump_list_entry *ours = &iter->jump[i];\n    -+\n    -+\t\tif (ours->start == ours->end) {\n    -+\t\t\t/* ignore empty regions (no matching entries) */\n    -+\t\t\tcontinue;\n    -+\t\t} else if (ours->start <= last_disjoint->end) {\n    ++\t\tif (ours->start <= last_disjoint->end) {\n     +\t\t\t/* overlapping regions extend the previous one */\n     +\t\t\tlast_disjoint->end = last_disjoint->end > ours->end\n     +\t\t\t\t? last_disjoint->end : ours->end;\n    @@ refs/packed-backend.c: static struct ref_iterator_vtable packed_ref_iterator_vta\n     +\t\t\t/* otherwise, insert a new region */\n     +\t\t\titer->jump[j++] = *ours;\n     +\t\t\tlast_disjoint = ours;\n    -+\n     +\t\t}\n     +\t}\n     +\n10:  49c8f5173aa ! 10:  e150941c1d1 refs/packed-backend.c: add trace2 counters for jump list\n    @@ Commit message\n     \n      ## refs/packed-backend.c ##\n     @@\n    - #include \"../chdir-notify.h\"\n    + #include \"../statinfo.h\"\n      #include \"../wrapper.h\"\n      #include \"../write-or-die.h\"\n     +#include \"../trace2.h\"\n    @@ refs/packed-backend.c: static int next_record(struct packed_ref_iterator *iter)\n      \t\tif (iter->pos < curr->end) {\n      \t\t\titer->pos = curr->end;\n     +\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);\n    + \t\t\t/* jumps are coalesced, so only one jump is necessary */\n      \t\t\tbreak;\n      \t\t}\n    - \t}\n     \n      ## t/t1419-exclude-refs.sh ##\n     @@ t/t1419-exclude-refs.sh: TEST_PASSES_SANITIZE_LEAK=true\n11:  dd856a3982b = 11:  a5093d52008 revision.h: store hidden refs in a `strvec`\n12:  845904853ee <  -:  ----------- refs/packed-backend.c: ignore complicated hidden refs rules\n13:  8d4d7cc22ee ! 12:  7e72c23c8a4 refs.h: let `for_each_namespaced_ref()` take excluded patterns\n    @@ refs.h: int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n      \n      int head_ref_namespaced(each_ref_fn fn, void *cb_data);\n     -int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);\n    ++/*\n    ++ * references matching any pattern in \"exclude_patterns\" are omitted from the\n    ++ * result set on a best-effort basis.\n    ++ */\n     +int for_each_namespaced_ref(const char **exclude_patterns,\n     +\t\t\t    each_ref_fn fn, void *cb_data);\n      \n -:  ----------- > 13:  f99d89d53b9 refs.h: implement `hidden_refs_to_excludes()`\n14:  49c665f9f8f ! 14:  96aada36f9b builtin/receive-pack.c: avoid enumerating hidden references\n    @@ builtin/receive-pack.c: static void write_head_info(void)\n      \n     -\tfor_each_ref(show_ref_cb, &seen);\n     +\trefs_for_each_fullref_in(get_main_ref_store(the_repository), \"\",\n    -+\t\t\t\t hidden_refs.v, show_ref_cb, &seen);\n    ++\t\t\t\t hidden_refs_to_excludes(&hidden_refs),\n    ++\t\t\t\t show_ref_cb, &seen);\n      \tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n      \toidset_clear(&seen);\n      \tif (!sent_capabilities)\n15:  19bf4a52d69 ! 15:  8544a647798 upload-pack.c: avoid enumerating hidden refs where possible\n    @@ upload-pack.c: static int get_common_commits(struct upload_pack_data *data,\n      \n     +static int allow_hidden_refs(enum allow_uor allow_uor)\n     +{\n    -+\treturn allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1);\n    ++\tif ((allow_uor & ALLOW_ANY_SHA1) == ALLOW_ANY_SHA1)\n    ++\t\treturn 1;\n    ++\treturn !(allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n     +}\n     +\n     +static void for_each_namespaced_ref_1(each_ref_fn fn,\n     +\t\t\t\t      struct upload_pack_data *data)\n     +{\n    ++\tconst char **excludes = NULL;\n     +\t/*\n     +\t * If `data->allow_uor` allows fetching hidden refs, we need to\n     +\t * mark all references (including hidden ones), to check in\n    @@ upload-pack.c: static int get_common_commits(struct upload_pack_data *data,\n     +\t * hidden references.\n     +\t */\n     +\tif (allow_hidden_refs(data->allow_uor))\n    -+\t\tfor_each_namespaced_ref(NULL, fn, data);\n    -+\telse\n    -+\t\tfor_each_namespaced_ref(data->hidden_refs.v, fn, data);\n    ++\t\texcludes = hidden_refs_to_excludes(&data->hidden_refs);\n    ++\n    ++\tfor_each_namespaced_ref(excludes, fn, data);\n     +}\n    ++\n     +\n      static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n      {\n     -\tint allow_hidden_ref = (allow_uor &\n     -\t\t\t\t(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n     -\treturn o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);\n    -+\treturn o->flags & ((allow_hidden_refs(allow_uor) ? HIDDEN_REF : 0) | OUR_REF);\n    ++\treturn o->flags & ((allow_hidden_refs(allow_uor) ? 0 : HIDDEN_REF) | OUR_REF);\n      }\n      \n      /*\n16:  ea6cbaf292f ! 16:  dff068c469f ls-refs.c: avoid enumerating hidden refs where possible\n    @@ ls-refs.c: int ls_refs(struct repository *r, struct packet_reader *request)\n      \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n      \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n     -\t\t\t\t\t  NULL, send_ref, &data);\n    -+\t\t\t\t\t  data.hidden_refs.v, send_ref, &data);\n    ++\t\t\t\t\t  hidden_refs_to_excludes(&data.hidden_refs),\n    ++\t\t\t\t\t  send_ref, &data);\n      \tpacket_fflush(stdout);\n      \tstrvec_clear(&data.prefixes);\n      \tstrbuf_release(&data.buf);\n-- \n2.41.0.343.gdff068c469f\n"},{"id":"479352","messageId":"5b5ccc40d6b87f0011fc1a88b99a7dda03b36886.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 01/16] refs.c: rename `ref_filter`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:05Z","receivedAt":"2023-07-10T21:12:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nThe refs machinery has its own implementation of a `ref_filter` (used by\n`for-each-ref`), which is distinct from the `ref-filter.h` API (also\nused by `for-each-ref`, among other things).\n\nRename the one within refs.c to more clearly indicate its purpose.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs.c | 15 ++++++++-------\n 1 file changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex c029f64982f..876df4931f6 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -377,8 +377,8 @@ char *resolve_refdup(const char *refname, int resolve_flags,\n \t\t\t\t   oid, flags);\n }\n \n-/* The argument to filter_refs */\n-struct ref_filter {\n+/* The argument to for_each_filter_refs */\n+struct for_each_ref_filter {\n \tconst char *pattern;\n \tconst char *prefix;\n \teach_ref_fn *fn;\n@@ -411,10 +411,11 @@ int ref_exists(const char *refname)\n \treturn refs_ref_exists(get_main_ref_store(the_repository), refname);\n }\n \n-static int filter_refs(const char *refname, const struct object_id *oid,\n-\t\t\t   int flags, void *data)\n+static int for_each_filter_refs(const char *refname,\n+\t\t\t\tconst struct object_id *oid,\n+\t\t\t\tint flags, void *data)\n {\n-\tstruct ref_filter *filter = (struct ref_filter *)data;\n+\tstruct for_each_ref_filter *filter = data;\n \n \tif (wildmatch(filter->pattern, refname, 0))\n \t\treturn 0;\n@@ -571,7 +572,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tconst char *prefix, void *cb_data)\n {\n \tstruct strbuf real_pattern = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct for_each_ref_filter filter;\n \tint ret;\n \n \tif (!prefix && !starts_with(pattern, \"refs/\"))\n@@ -591,7 +592,7 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \tfilter.prefix = prefix;\n \tfilter.fn = fn;\n \tfilter.cb_data = cb_data;\n-\tret = for_each_ref(filter_refs, &filter);\n+\tret = for_each_ref(for_each_filter_refs, &filter);\n \n \tstrbuf_release(&real_pattern);\n \treturn ret;\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479353","messageId":"0c4e995f1d3bb918ea0d4c0e6aed34aa5e9fe9da.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 02/16] ref-filter.h: provide `REF_FILTER_INIT`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:07Z","receivedAt":"2023-07-10T21:12:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nProvide a sane initialization value for `struct ref_filter`, which in a\nsubsequent patch will be used to initialize a new field.\n\nIn the meantime, ensure that the `ref_filter` struct used in the\ntest-helper's `cmd__reach()` is zero-initialized. The lack of\ninitialization is OK, since `commit_contains()` only looks at the single\n`with_commit_tag_algo` field that *is* initialized directly above.\n\nSo this does not fix a bug, but rather prevents one from biting us in\nthe future.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c       | 3 +--\n builtin/for-each-ref.c | 3 +--\n builtin/tag.c          | 3 +--\n ref-filter.h           | 3 +++\n t/helper/test-reach.c  | 2 +-\n 5 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex e8ff3ecc072..322646f38f5 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -702,7 +702,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \tint reflog = 0, quiet = 0, icase = 0, force = 0,\n \t    recurse_submodules_explicit = 0;\n \tenum branch_track track;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstatic struct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -760,7 +760,6 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \n \tsetup_ref_filter_porcelain_msg();\n \n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.kind = FILTER_REFS_BRANCHES;\n \tfilter.abbrev = -1;\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 15409337f83..6b5e313123f 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -24,7 +24,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tint maxcount = 0, icase = 0, omit_empty = 0;\n \tstruct ref_array array;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n@@ -61,7 +61,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t};\n \n \tmemset(&array, 0, sizeof(array));\n-\tmemset(&filter, 0, sizeof(filter));\n \n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 7d34af416c7..a624185d105 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -445,7 +445,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tstruct msg_arg msg = { .buf = STRBUF_INIT };\n \tstruct ref_transaction *transaction;\n \tstruct strbuf err = STRBUF_INIT;\n-\tstruct ref_filter filter;\n+\tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -504,7 +504,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \tgit_config(git_tag_config, &sorting_options);\n \n \tmemset(&opt, 0, sizeof(opt));\n-\tmemset(&filter, 0, sizeof(filter));\n \tfilter.lines = -1;\n \topt.sign = -1;\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 430701cfb76..a920f73b297 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,9 @@ struct ref_format {\n \tstruct string_list bases;\n };\n \n+#define REF_FILTER_INIT { \\\n+\t.points_at = OID_ARRAY_INIT, \\\n+}\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n \t.bases = STRING_LIST_INIT_DUP, \\\ndiff --git a/t/helper/test-reach.c b/t/helper/test-reach.c\nindex 5b6f2174418..ef58f10c2d6 100644\n--- a/t/helper/test-reach.c\n+++ b/t/helper/test-reach.c\n@@ -139,7 +139,7 @@ int cmd__reach(int ac, const char **av)\n \n \t\tprintf(\"%s(X,_,_,0,0):%d\\n\", av[1], can_all_from_reach_with_flag(&X_obj, 2, 4, 0, 0));\n \t} else if (!strcmp(av[1], \"commit_contains\")) {\n-\t\tstruct ref_filter filter;\n+\t\tstruct ref_filter filter = REF_FILTER_INIT;\n \t\tstruct contains_cache cache;\n \t\tinit_contains_cache(&cache);\n \n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479354","messageId":"cea1e88d35871f84d504a7e5eaa6186fcc6f46f0.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 03/16] ref-filter: clear reachable list pointers after freeing","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:10Z","receivedAt":"2023-07-10T21:12:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nIn `reach_filter()`, we pop all commits from the reachable lists,\nleaving them empty. But because we're operating on a list pointer that\nwas passed by value, the original `filter.reachable_from` and\n`filter.unreachable_from` pointers are left dangling.\n\nAs is the case with the previous commit, nobody touches either of these\nfields after calling `reach_filter()`, so leaving them dangling is OK.\nBut this future proofs against dangerous situations.\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex e0d03a9f8e9..4dd6575400c 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2418,13 +2418,13 @@ void ref_array_clear(struct ref_array *array)\n #define EXCLUDE_REACHED 0\n #define INCLUDE_REACHED 1\n static void reach_filter(struct ref_array *array,\n-\t\t\t struct commit_list *check_reachable,\n+\t\t\t struct commit_list **check_reachable,\n \t\t\t int include_reached)\n {\n \tint i, old_nr;\n \tstruct commit **to_clear;\n \n-\tif (!check_reachable)\n+\tif (!*check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n@@ -2434,7 +2434,7 @@ static void reach_filter(struct ref_array *array,\n \t}\n \n \ttips_reachable_from_bases(the_repository,\n-\t\t\t\t  check_reachable,\n+\t\t\t\t  *check_reachable,\n \t\t\t\t  to_clear, array->nr,\n \t\t\t\t  UNINTERESTING);\n \n@@ -2455,8 +2455,8 @@ static void reach_filter(struct ref_array *array,\n \n \tclear_commit_marks_many(old_nr, to_clear, ALL_REV_FLAGS);\n \n-\twhile (check_reachable) {\n-\t\tstruct commit *merge_commit = pop_commit(&check_reachable);\n+\twhile (*check_reachable) {\n+\t\tstruct commit *merge_commit = pop_commit(check_reachable);\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n@@ -2553,8 +2553,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \tclear_contains_cache(&ref_cbdata.no_contains_cache);\n \n \t/*  Filters that need revision walking */\n-\treach_filter(array, filter->reachable_from, INCLUDE_REACHED);\n-\treach_filter(array, filter->unreachable_from, EXCLUDE_REACHED);\n+\treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n+\treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n \n \tsave_commit_buffer = save_commit_buffer_orig;\n \treturn ret;\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479355","messageId":"00a4532d54d81953c931bdd948e708f90c0df15a.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 04/16] ref-filter: add `ref_filter_clear()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:13Z","receivedAt":"2023-07-10T21:12:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nWe did not bother to clean up at all in `git branch` or `git tag`, and\n`git for-each-ref` only cleans up a couple of members.\n\nAdd and call `ref_filter_clear()` when cleaning up a `struct\nref_filter`. Running this patch (without any test changes) indicates a\ncouple of now leak-free tests. This was found by running:\n\n    $ make SANITIZE=leak\n    $ make -C t GIT_TEST_PASSING_SANITIZE_LEAK=check GIT_TEST_OPTS=--immediate\n\n(Note that the `reachable_from` and `unreachable_from` lists should be\ncleaned as they are used. So this is just covering any case where we\nmight bail before running the reachability check.)\n\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/branch.c        |  1 +\n builtin/for-each-ref.c  |  3 +--\n builtin/tag.c           |  1 +\n ref-filter.c            | 16 ++++++++++++++++\n ref-filter.h            |  3 +++\n t/t0041-usage.sh        |  1 +\n t/t3402-rebase-merge.sh |  1 +\n 7 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 322646f38f5..f06df4be7a5 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -855,6 +855,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\tprint_columns(&output, colopts, NULL);\n \t\tstring_list_clear(&output, 0);\n \t\tref_sorting_release(sorting);\n+\t\tref_filter_clear(&filter);\n \t\treturn 0;\n \t} else if (edit_description) {\n \t\tconst char *branch_name;\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 6b5e313123f..ccceba54aa1 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -120,8 +120,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&err);\n \tstrbuf_release(&output);\n \tref_array_clear(&array);\n-\tfree_commit_list(filter.with_commit);\n-\tfree_commit_list(filter.no_commit);\n+\tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\n \tstrvec_clear(&vec);\n \treturn 0;\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex a624185d105..f33600c0506 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -659,6 +659,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \n cleanup:\n \tref_sorting_release(sorting);\n+\tref_filter_clear(&filter);\n \tstrbuf_release(&buf);\n \tstrbuf_release(&ref);\n \tstrbuf_release(&reflog_msg);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4dd6575400c..84cd3921307 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2866,3 +2866,19 @@ int parse_opt_merge_filter(const struct option *opt, const char *arg, int unset)\n \n \treturn 0;\n }\n+\n+void ref_filter_init(struct ref_filter *filter)\n+{\n+\tstruct ref_filter blank = REF_FILTER_INIT;\n+\tmemcpy(filter, &blank, sizeof(blank));\n+}\n+\n+void ref_filter_clear(struct ref_filter *filter)\n+{\n+\toid_array_clear(&filter->points_at);\n+\tfree_commit_list(filter->with_commit);\n+\tfree_commit_list(filter->no_commit);\n+\tfree_commit_list(filter->reachable_from);\n+\tfree_commit_list(filter->unreachable_from);\n+\tref_filter_init(filter);\n+}\ndiff --git a/ref-filter.h b/ref-filter.h\nindex a920f73b297..160b8072245 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -170,4 +170,7 @@ void filter_ahead_behind(struct repository *r,\n \t\t\t struct ref_format *format,\n \t\t\t struct ref_array *array);\n \n+void ref_filter_init(struct ref_filter *filter);\n+void ref_filter_clear(struct ref_filter *filter);\n+\n #endif /*  REF_FILTER_H  */\ndiff --git a/t/t0041-usage.sh b/t/t0041-usage.sh\nindex c4fc34eb18e..9ea974b0c6c 100755\n--- a/t/t0041-usage.sh\n+++ b/t/t0041-usage.sh\n@@ -5,6 +5,7 @@ test_description='Test commands behavior when given invalid argument value'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n test_expect_success 'setup ' '\ndiff --git a/t/t3402-rebase-merge.sh b/t/t3402-rebase-merge.sh\nindex 79b0640c004..e9e03ca4b5e 100755\n--- a/t/t3402-rebase-merge.sh\n+++ b/t/t3402-rebase-merge.sh\n@@ -8,6 +8,7 @@ test_description='git rebase --merge test'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n T=\"A quick brown fox\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479356","messageId":"3ab03ac20dfd83acd5e00e027cf0da362529b7e1.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 05/16] ref-filter.c: parameterize match functions over patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:16Z","receivedAt":"2023-07-10T21:12:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King <peff@peff.net>\n\n`match_pattern()` and `match_name_as_path()` both take a `struct\nref_filter *`, and then store a stack variable `patterns` pointing at\n`filter->patterns`.\n\nThe subsequent patch will add a new array of patterns to match over (the\nexcluded patterns, via a new `git for-each-ref --exclude` option),\ntreating the return value of these functions differently depending on\nwhich patterns are being used to match.\n\nTweak `match_pattern()` and `match_name_as_path()` to take an array of\npatterns to prepare for passing either in.\n\nOnce we start passing either in, `match_pattern()` will have little to\ndo with a particular `struct ref_filter *` instance. To clarify this,\ndrop it from the argument list, and replace it with the only bit of the\n`ref_filter` that we care about (`filter->ignore_case`).\n\nCo-authored-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c | 18 ++++++++++--------\n 1 file changed, 10 insertions(+), 8 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 84cd3921307..c70a430adf3 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2104,12 +2104,12 @@ static int get_ref_atom_value(struct ref_array_item *ref, int atom,\n  * matches a pattern \"refs/heads/mas\") or a wildcard (e.g. the same ref\n  * matches \"refs/heads/mas*\", too).\n  */\n-static int match_pattern(const struct ref_filter *filter, const char *refname)\n+static int match_pattern(const char **patterns, const char *refname,\n+\t\t\t int ignore_case)\n {\n-\tconst char **patterns = filter->name_patterns;\n \tunsigned flags = 0;\n \n-\tif (filter->ignore_case)\n+\tif (ignore_case)\n \t\tflags |= WM_CASEFOLD;\n \n \t/*\n@@ -2134,13 +2134,13 @@ static int match_pattern(const struct ref_filter *filter, const char *refname)\n  * matches a pattern \"refs/heads/\" but not \"refs/heads/m\") or a\n  * wildcard (e.g. the same ref matches \"refs/heads/m*\", too).\n  */\n-static int match_name_as_path(const struct ref_filter *filter, const char *refname)\n+static int match_name_as_path(const char **pattern, const char *refname,\n+\t\t\t      int ignore_case)\n {\n-\tconst char **pattern = filter->name_patterns;\n \tint namelen = strlen(refname);\n \tunsigned flags = WM_PATHNAME;\n \n-\tif (filter->ignore_case)\n+\tif (ignore_case)\n \t\tflags |= WM_CASEFOLD;\n \n \tfor (; *pattern; pattern++) {\n@@ -2165,8 +2165,10 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \tif (!*filter->name_patterns)\n \t\treturn 1; /* No pattern always matches */\n \tif (filter->match_as_path)\n-\t\treturn match_name_as_path(filter, refname);\n-\treturn match_pattern(filter, refname);\n+\t\treturn match_name_as_path(filter->name_patterns, refname,\n+\t\t\t\t\t  filter->ignore_case);\n+\treturn match_pattern(filter->name_patterns, refname,\n+\t\t\t     filter->ignore_case);\n }\n \n /*\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479357","messageId":"aa881ca06faf1a13004b53c467704c9d338c3819.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 06/16] builtin/for-each-ref.c: add `--exclude` option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:19Z","receivedAt":"2023-07-10T21:12:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When using `for-each-ref`, it is sometimes convenient for the caller to\nbe able to exclude certain parts of the references.\n\nFor example, if there are many `refs/__hidden__/*` references, the\ncaller may want to emit all references *except* the hidden ones.\nCurrently, the only way to do this is to post-process the output, like:\n\n    $ git for-each-ref --format='%(refname)' | grep -v '^refs/hidden/'\n\nWhich is do-able, but requires processing a potentially large quantity\nof references.\n\nTeach `git for-each-ref` a new `--exclude=<pattern>` option, which\nexcludes references from the results if they match one or more excluded\npatterns.\n\nThis patch provides a naive implementation where the `ref_filter` still\nsees all references (including ones that it will discard) and is left to\ncheck whether each reference matches any excluded pattern(s) before\nemitting them.\n\nBy culling out references we know the caller doesn't care about, we can\navoid allocating memory for their storage, as well as spending time\nsorting the output (among other things). Even the naive implementation\nprovides a significant speed-up on a modified copy of linux.git (that\nhas a hidden ref pointing at each commit):\n\n    $ hyperfine \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/'\n    Benchmark 1: git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     820.1 ms ±   2.0 ms    [User: 703.7 ms, System: 152.0 ms]\n      Range (min … max):   817.7 ms … 823.3 ms    10 runs\n\n    Benchmark 2: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/\n      Time (mean ± σ):     106.6 ms ±   1.1 ms    [User: 99.4 ms, System: 7.1 ms]\n      Range (min … max):   104.7 ms … 109.1 ms    27 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude refs/pull/' ran\n        7.69 ± 0.08 times faster than 'git.compile for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"[0-9a-f]{40} refs/pull/\"'\n\nSubsequent patches will improve on this by avoiding visiting excluded\nsections of the `packed-refs` file in certain cases.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-for-each-ref.txt |  6 +++++\n builtin/for-each-ref.c             |  1 +\n ref-filter.c                       | 14 ++++++++++++\n ref-filter.h                       |  6 +++++\n t/t6300-for-each-ref.sh            | 35 ++++++++++++++++++++++++++++++\n 5 files changed, 62 insertions(+)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 1e215d4e734..5743eb5def4 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -14,6 +14,7 @@ SYNOPSIS\n \t\t   [--points-at=<object>]\n \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n+\t\t   [--exclude=<pattern> ...]\n \n DESCRIPTION\n -----------\n@@ -102,6 +103,11 @@ OPTIONS\n \tDo not print a newline after formatted refs where the format expands\n \tto the empty string.\n \n+--exclude=<pattern>::\n+\tIf one or more patterns are given, only refs which do not match\n+\tany excluded pattern(s) are shown. Matching is done using the\n+\tsame rules as `<pattern>` above.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex ccceba54aa1..350bfa6e811 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -47,6 +47,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n+\t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n \t\t\t     N_(\"object\"), N_(\"print only refs which points at the given object\"),\ndiff --git a/ref-filter.c b/ref-filter.c\nindex c70a430adf3..845173a904a 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2171,6 +2171,16 @@ static int filter_pattern_match(struct ref_filter *filter, const char *refname)\n \t\t\t     filter->ignore_case);\n }\n \n+static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n+{\n+\tif (!filter->exclude.nr)\n+\t\treturn 0;\n+\tif (filter->match_as_path)\n+\t\treturn match_name_as_path(filter->exclude.v, refname,\n+\t\t\t\t\t  filter->ignore_case);\n+\treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n+}\n+\n /*\n  * This is the same as for_each_fullref_in(), but it tries to iterate\n  * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2338,6 +2348,9 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \tif (!filter_pattern_match(filter, refname))\n \t\treturn 0;\n \n+\tif (filter_exclude_match(filter, refname))\n+\t\treturn 0;\n+\n \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n \t\treturn 0;\n \n@@ -2877,6 +2890,7 @@ void ref_filter_init(struct ref_filter *filter)\n \n void ref_filter_clear(struct ref_filter *filter)\n {\n+\tstrvec_clear(&filter->exclude);\n \toid_array_clear(&filter->points_at);\n \tfree_commit_list(filter->with_commit);\n \tfree_commit_list(filter->no_commit);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 160b8072245..1524bc463a5 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -6,6 +6,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -59,6 +60,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\n \tstruct commit_list *no_commit;\n@@ -94,6 +96,7 @@ struct ref_format {\n \n #define REF_FILTER_INIT { \\\n \t.points_at = OID_ARRAY_INIT, \\\n+\t.exclude = STRVEC_INIT, \\\n }\n #define REF_FORMAT_INIT {             \\\n \t.use_color = -1,              \\\n@@ -112,6 +115,9 @@ struct ref_format {\n #define OPT_REF_SORT(var) \\\n \tOPT_STRING_LIST(0, \"sort\", (var), \\\n \t\t\tN_(\"key\"), N_(\"field name to sort on\"))\n+#define OPT_REF_FILTER_EXCLUDE(var) \\\n+\tOPT_STRVEC(0, \"exclude\", &(var)->exclude, \\\n+\t\t   N_(\"pattern\"), N_(\"exclude refs which match pattern\"))\n \n /*\n  * API for filtering a set of refs. Based on the type of refs the user\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 5c00607608a..7e8d5785223 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -447,6 +447,41 @@ test_expect_success 'exercise glob patterns with prefixes' '\n \ttest_cmp expected actual\n '\n \n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with prefix exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=refs/tags/foo >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+cat >expected <<\\EOF\n+refs/tags/bar\n+refs/tags/baz\n+refs/tags/foo/one\n+refs/tags/testtag\n+EOF\n+\n+test_expect_success 'exercise patterns with pattern exclusions' '\n+\tfor tag in foo/one foo/two foo/three bar baz\n+\tdo\n+\t\tgit tag \"$tag\" || return 1\n+\tdone &&\n+\ttest_when_finished \"git tag -d foo/one foo/two foo/three bar baz\" &&\n+\tgit for-each-ref --format=\"%(refname)\" \\\n+\t\trefs/tags/ --exclude=\"refs/tags/foo/t*\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n cat >expected <<\\EOF\n 'refs/heads/main'\n 'refs/remotes/origin/main'\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479358","messageId":"81e223de0c82dfc213ea7d7041e66c9505c1599d.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 07/16] refs: plumb `exclude_patterns` argument throughout","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:22Z","receivedAt":"2023-07-10T21:12:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The subsequent patch will want to access an optional `excluded_patterns`\narray within `refs/packed-backend.c` that will cull out certain\nreferences matching any of the given patterns on a best-effort basis.\n\nTo do so, the refs subsystem needs to be updated to pass this value\nacross a number of different locations.\n\nPrepare for a future patch by introducing this plumbing now, passing\nNULLs at top-level APIs in order to make that patch less noisy and more\neasily readable.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.co>\n---\n ls-refs.c             |  2 +-\n ref-filter.c          |  2 +-\n refs.c                | 32 +++++++++++++++++++-------------\n refs.h                |  8 +++++++-\n refs/debug.c          |  5 +++--\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  5 +++--\n refs/refs-internal.h  |  7 ++++---\n revision.c            |  2 +-\n 9 files changed, 42 insertions(+), 26 deletions(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex a29c2364a59..65d1fa9029e 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -194,7 +194,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  send_ref, &data);\n+\t\t\t\t\t  NULL, send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 845173a904a..925d77b7b79 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2215,7 +2215,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t cb, cb_data);\n+\t\t\t\t\t\t NULL, cb, cb_data);\n }\n \n /*\ndiff --git a/refs.c b/refs.c\nindex 876df4931f6..7e5cd84bf28 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1528,7 +1528,9 @@ int head_ref(each_ref_fn fn, void *cb_data)\n \n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n+\t\tconst char *prefix,\n+\t\tconst char **exclude_patterns,\n+\t\tint trim,\n \t\tenum do_for_each_ref_flags flags)\n {\n \tstruct ref_iterator *iter;\n@@ -1544,8 +1546,7 @@ struct ref_iterator *refs_ref_iterator_begin(\n \t\t}\n \t}\n \n-\titer = refs->be->iterator_begin(refs, prefix, flags);\n-\n+\titer = refs->be->iterator_begin(refs, prefix, exclude_patterns, flags);\n \t/*\n \t * `iterator_begin()` already takes care of prefix, but we\n \t * might need to do some trimming:\n@@ -1579,7 +1580,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, NULL, trim, flags);\n \n \treturn do_for_each_repo_ref_iterator(r, iter, fn, cb_data);\n }\n@@ -1601,6 +1602,7 @@ static int do_for_each_ref_helper(struct repository *r,\n }\n \n static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n+\t\t\t   const char **exclude_patterns,\n \t\t\t   each_ref_fn fn, int trim,\n \t\t\t   enum do_for_each_ref_flags flags, void *cb_data)\n {\n@@ -1610,7 +1612,8 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, exclude_patterns, trim,\n+\t\t\t\t       flags);\n \n \treturn do_for_each_repo_ref_iterator(the_repository, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n@@ -1618,7 +1621,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \n int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n@@ -1629,7 +1632,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, NULL, fn, strlen(prefix), 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n@@ -1640,13 +1643,14 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t       prefix, fn, 0, 0, cb_data);\n+\t\t\t       prefix, NULL, fn, 0, 0, cb_data);\n }\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, exclude_patterns, fn, 0, 0, cb_data);\n }\n \n int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n@@ -1663,14 +1667,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\n \n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0,\n+\treturn do_for_each_ref(refs, \"\", NULL, fn, 0,\n \t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n@@ -1740,6 +1744,7 @@ static void find_longest_prefixes(struct string_list *out,\n int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \t\t\t\t      const char *namespace,\n \t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data)\n {\n \tstruct string_list prefixes = STRING_LIST_INIT_DUP;\n@@ -1755,7 +1760,8 @@ int refs_for_each_fullref_in_prefixes(struct ref_store *ref_store,\n \n \tfor_each_string_list_item(prefix, &prefixes) {\n \t\tstrbuf_addstr(&buf, prefix->string);\n-\t\tret = refs_for_each_fullref_in(ref_store, buf.buf, fn, cb_data);\n+\t\tret = refs_for_each_fullref_in(ref_store, buf.buf,\n+\t\t\t\t\t       exclude_patterns, fn, cb_data);\n \t\tif (ret)\n \t\t\tbreak;\n \t\tstrbuf_setlen(&buf, namespace_len);\n@@ -2410,7 +2416,7 @@ int refs_verify_refname_available(struct ref_store *refs,\n \tstrbuf_addstr(&dirname, refname + dirname.len);\n \tstrbuf_addch(&dirname, '/');\n \n-\titer = refs_ref_iterator_begin(refs, dirname.buf, 0,\n+\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 933fdebe584..3c03b035a3f 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -344,6 +344,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data);\n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n+\t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data);\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n@@ -351,10 +352,15 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n  * iterate all refs in \"patterns\" by partitioning patterns into disjoint sets\n  * and iterating the longest-common prefix of each set.\n  *\n+ * references matching any pattern in \"exclude_patterns\" are omitted from the\n+ * result set on a best-effort basis.\n+ *\n  * callers should be prepared to ignore references that they did not ask for.\n  */\n int refs_for_each_fullref_in_prefixes(struct ref_store *refs,\n-\t\t\t\t      const char *namespace, const char **patterns,\n+\t\t\t\t      const char *namespace,\n+\t\t\t\t      const char **patterns,\n+\t\t\t\t      const char **exclude_patterns,\n \t\t\t\t      each_ref_fn fn, void *cb_data);\n \n /**\ndiff --git a/refs/debug.c b/refs/debug.c\nindex c0fa707a1da..b7ffc4ce67e 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -229,11 +229,12 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \n static struct ref_iterator *\n debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n-\t\t\t unsigned int flags)\n+\t\t\t const char **exclude_patterns, unsigned int flags)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n \tstruct ref_iterator *res =\n-\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix, flags);\n+\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix,\n+\t\t\t\t\t\texclude_patterns, flags);\n \tstruct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));\n \tbase_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);\n \tditer->iter = res;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 09b4954f21c..341354182bb 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -832,7 +832,8 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \n static struct ref_iterator *files_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct files_ref_store *refs;\n \tstruct ref_iterator *loose_iter, *packed_iter, *overlay_iter;\n@@ -877,7 +878,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * the packed and loose references.\n \t */\n \tpacked_iter = refs_ref_iterator_begin(\n-\t\t\trefs->packed_ref_store, prefix, 0,\n+\t\t\trefs->packed_ref_store, prefix, exclude_patterns, 0,\n \t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n \n \toverlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 27bd6339ff3..dec15dee31e 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -925,7 +925,8 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags)\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n@@ -1150,7 +1151,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t * list of refs is exhausted, set iter to NULL. When the list\n \t * of updates is exhausted, leave i set to updates->nr.\n \t */\n-\titer = packed_ref_iterator_begin(&refs->base, \"\",\n+\titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n \tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n \t\titer = NULL;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f72b7be8941..9db8aec4da8 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -367,8 +367,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n  */\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim,\n-\t\tenum do_for_each_ref_flags flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tint trim, enum do_for_each_ref_flags flags);\n \n /*\n  * A callback function used to instruct merge_ref_iterator how to\n@@ -571,7 +571,8 @@ typedef int copy_ref_fn(struct ref_store *ref_store,\n  */\n typedef struct ref_iterator *ref_iterator_begin_fn(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags);\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tunsigned int flags);\n \n /* reflog functions */\n \ndiff --git a/revision.c b/revision.c\nindex d66857ecc09..af11d599d35 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2642,7 +2642,7 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,\n \tstruct strbuf bisect_refs = STRBUF_INIT;\n \tint status;\n \tstrbuf_addf(&bisect_refs, \"refs/bisect/%s\", term);\n-\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, fn, cb_data);\n+\tstatus = refs_for_each_fullref_in(refs, bisect_refs.buf, NULL, fn, cb_data);\n \tstrbuf_release(&bisect_refs);\n \treturn status;\n }\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479359","messageId":"25c099a528c87651aeea3dfbd667fb51770dd24a.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 08/16] refs/packed-backend.c: refactor `find_reference_location()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:25Z","receivedAt":"2023-07-10T21:12:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `find_reference_location()` is used to perform a\nbinary search-like function over the contents of a repository's\n`$GIT_DIR/packed-refs` file.\n\nThe search it implements is unlike a standard binary search in that the\nrecords it searches over are not of a fixed width, so the comparison\nmust locate the end of a record before comparing it.\n\nExtract the core routine of `find_reference_location()` in order to\nimplement a function in the following patch which will find the first\nlocation in the `packed-refs` file that *doesn't* match the given\npattern.\n\nThe behavior of `find_reference_location()` is unchanged.\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c | 38 ++++++++++++++++++++++----------------\n 1 file changed, 22 insertions(+), 16 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex dec15dee31e..e36334e3320 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -528,22 +528,8 @@ static int load_contents(struct snapshot *snapshot)\n \treturn 1;\n }\n \n-/*\n- * Find the place in `snapshot->buf` where the start of the record for\n- * `refname` starts. If `mustexist` is true and the reference doesn't\n- * exist, then return NULL. If `mustexist` is false and the reference\n- * doesn't exist, then return the point where that reference would be\n- * inserted, or `snapshot->eof` (which might be NULL) if it would be\n- * inserted at the end of the file. In the latter mode, `refname`\n- * doesn't have to be a proper reference name; for example, one could\n- * search for \"refs/replace/\" to find the start of any replace\n- * references.\n- *\n- * The record is sought using a binary search, so `snapshot->buf` must\n- * be sorted.\n- */\n-static const char *find_reference_location(struct snapshot *snapshot,\n-\t\t\t\t\t   const char *refname, int mustexist)\n+static const char *find_reference_location_1(struct snapshot *snapshot,\n+\t\t\t\t\t     const char *refname, int mustexist)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -589,6 +575,26 @@ static const char *find_reference_location(struct snapshot *snapshot,\n \t\treturn lo;\n }\n \n+/*\n+ * Find the place in `snapshot->buf` where the start of the record for\n+ * `refname` starts. If `mustexist` is true and the reference doesn't\n+ * exist, then return NULL. If `mustexist` is false and the reference\n+ * doesn't exist, then return the point where that reference would be\n+ * inserted, or `snapshot->eof` (which might be NULL) if it would be\n+ * inserted at the end of the file. In the latter mode, `refname`\n+ * doesn't have to be a proper reference name; for example, one could\n+ * search for \"refs/replace/\" to find the start of any replace\n+ * references.\n+ *\n+ * The record is sought using a binary search, so `snapshot->buf` must\n+ * be sorted.\n+ */\n+static const char *find_reference_location(struct snapshot *snapshot,\n+\t\t\t\t\t   const char *refname, int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist);\n+}\n+\n /*\n  * Create a newly-allocated `snapshot` of the `packed-refs` file in\n  * its current state and return it. The return value will already have\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479360","messageId":"af0ce43cc90d3b83e377ba19ab1add9fa6881ab1.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 09/16] refs/packed-backend.c: implement jump lists to avoid excluded pattern(s)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:28Z","receivedAt":"2023-07-10T21:13:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When iterating through the `packed-refs` file in order to answer a query\nlike:\n\n    $ git for-each-ref --exclude=refs/__hidden__\n\nit would be useful to avoid walking over all of the entries in\n`refs/__hidden__/*` when possible, since we know that the ref-filter\ncode is going to throw them away anyways.\n\nIn certain circumstances, doing so is possible. The algorithm for doing\nso is as follows:\n\n  - For each excluded pattern, find the first record that matches it,\n    and the first record that *doesn't* match it (i.e. the location\n    you'd next want to consider when excluding that pattern).\n\n  - Sort the set of excluded regions from the previous step in ascending\n    order of the first location within the `packed-refs` file that\n    matches.\n\n  - Clean up the results from the previous step: discard empty regions,\n    and combine adjacent regions. The set of regions which remains is\n    referred to as the \"jump list\", and never contains any references\n    which should be included in the result set.\n\nThen when iterating through the `packed-refs` file, if `iter->pos` is\never contained in one of the regions from the previous steps, advance\n`iter->pos` past the end of that region, and continue enumeration.\n\nNote that we only perform this optimization when none of the excluded\npattern(s) have special meta-characters in them. For a pattern like\n\"refs/foo[ac]\", the excluded regions (\"refs/fooa\", \"refs/fooc\", and\neverything underneath them) are not connected. A future implementation\nthat handles this case may split the character class (pretending as if\ntwo patterns were excluded: \"refs/fooa\", and \"refs/fooc\").\n\nThere are a few other gotchas worth considering. First, note that the\njump list is sorted, so once we jump past a region, we can avoid\nconsidering it (or any regions preceding it) again. The member\n`jump_pos` is used to track the first next-possible region to jump\nthrough.\n\nSecond, note that the jump list is best-effort, since we do not handle\nloose references, and because of the meta-character issue above. The\njump list may not skip past all references which won't appear in the\nresults, but will never skip over a reference which does appear in the\nresult set.\n\nIn repositories with a large number of hidden references, the speed-up\ncan be significant. Tests here are done with a copy of linux.git with a\nreference \"refs/pull/N\" pointing at every commit, as in:\n\n    $ git rev-list HEAD | awk '{ print \"create refs/pull/\" NR \" \" $0 }' |\n        git update-ref --stdin\n    $ git pack-refs --all\n\n, it is significantly faster to have `for-each-ref` jump over the\nexcluded references, as opposed to filtering them out after the fact:\n\n    $ hyperfine \\\n      'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"' \\\n      'git.prev for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' \\\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n    Benchmark 1: git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"\n      Time (mean ± σ):     798.1 ms ±   3.3 ms    [User: 687.6 ms, System: 146.4 ms]\n      Range (min … max):   794.5 ms … 805.5 ms    10 runs\n\n    Benchmark 2: git.prev for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n      Time (mean ± σ):      98.9 ms ±   1.4 ms    [User: 93.1 ms, System: 5.7 ms]\n      Range (min … max):    97.0 ms … 104.0 ms    29 runs\n\n    Benchmark 3: git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"\n      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.7 ms, System: 3.8 ms]\n      Range (min … max):     4.1 ms …   5.8 ms    524 runs\n\n    Summary\n      'git.compile for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"' ran\n       21.87 ± 1.05 times faster than 'git.prev for-each-ref --format=\"%(objectname) %(refname)\" --exclude=\"refs/pull\"'\n      176.52 ± 8.19 times faster than 'git for-each-ref --format=\"%(objectname) %(refname)\" | grep -vE \"^[0-9a-f]{40} refs/pull/\"'\n\n(Comparing stock git and this patch isn't quite fair, since an earlier\ncommit in this series adds a naive implementation of the `--exclude`\noption. `git.prev` is built from the previous commit and includes this\nnaive implementation).\n\nUsing the jump list is fairly straightforward (see the changes to\n`refs/packed-backend.c::next_record()`), but constructing the list is\nnot. To ensure that the construction is correct, add a new suite of\ntests in t1419 covering various corner cases (overlapping regions,\npartially overlapping regions, adjacent regions, etc.).\n\nCo-authored-by: Jeff King <peff@peff.net>\nSigned-off-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ref-filter.c              |   6 +-\n refs.h                    |   4 +\n refs/packed-backend.c     | 163 ++++++++++++++++++++++++++++++++++++--\n t/helper/test-ref-store.c |  10 +++\n t/t1419-exclude-refs.sh   | 101 +++++++++++++++++++++++\n 5 files changed, 276 insertions(+), 8 deletions(-)\n create mode 100755 t/t1419-exclude-refs.sh\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 925d77b7b79..07b65affa9a 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2210,12 +2210,14 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn for_each_fullref_in(\"\", cb, cb_data);\n+\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n-\t\t\t\t\t\t NULL, cb, cb_data);\n+\t\t\t\t\t\t filter->exclude.v,\n+\t\t\t\t\t\t cb, cb_data);\n }\n \n /*\ndiff --git a/refs.h b/refs.h\nindex 3c03b035a3f..1ca1cdb9da5 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -343,6 +343,10 @@ int for_each_ref(each_ref_fn fn, void *cb_data);\n  */\n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data);\n \n+/*\n+ * references matching any pattern in \"exclude_patterns\" are omitted from the\n+ * result set on a best-effort basis.\n+ */\n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n \t\t\t     const char **exclude_patterns,\n \t\t\t     each_ref_fn fn, void *cb_data);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex e36334e3320..d9e008819b5 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -304,7 +304,8 @@ static int cmp_packed_ref_records(const void *v1, const void *v2)\n  * Compare a snapshot record at `rec` to the specified NUL-terminated\n  * refname.\n  */\n-static int cmp_record_to_refname(const char *rec, const char *refname)\n+static int cmp_record_to_refname(const char *rec, const char *refname,\n+\t\t\t\t int start)\n {\n \tconst char *r1 = rec + the_hash_algo->hexsz + 1;\n \tconst char *r2 = refname;\n@@ -313,7 +314,7 @@ static int cmp_record_to_refname(const char *rec, const char *refname)\n \t\tif (*r1 == '\\n')\n \t\t\treturn *r2 ? -1 : 0;\n \t\tif (!*r2)\n-\t\t\treturn 1;\n+\t\t\treturn start ? 1 : -1;\n \t\tif (*r1 != *r2)\n \t\t\treturn (unsigned char)*r1 < (unsigned char)*r2 ? -1 : +1;\n \t\tr1++;\n@@ -529,7 +530,8 @@ static int load_contents(struct snapshot *snapshot)\n }\n \n static const char *find_reference_location_1(struct snapshot *snapshot,\n-\t\t\t\t\t     const char *refname, int mustexist)\n+\t\t\t\t\t     const char *refname, int mustexist,\n+\t\t\t\t\t     int start)\n {\n \t/*\n \t * This is not *quite* a garden-variety binary search, because\n@@ -559,7 +561,7 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n \n \t\tmid = lo + (hi - lo) / 2;\n \t\trec = find_start_of_record(lo, mid);\n-\t\tcmp = cmp_record_to_refname(rec, refname);\n+\t\tcmp = cmp_record_to_refname(rec, refname, start);\n \t\tif (cmp < 0) {\n \t\t\tlo = find_end_of_record(mid, hi);\n \t\t} else if (cmp > 0) {\n@@ -592,7 +594,22 @@ static const char *find_reference_location_1(struct snapshot *snapshot,\n static const char *find_reference_location(struct snapshot *snapshot,\n \t\t\t\t\t   const char *refname, int mustexist)\n {\n-\treturn find_reference_location_1(snapshot, refname, mustexist);\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 1);\n+}\n+\n+/*\n+ * Find the place in `snapshot->buf` after the end of the record for\n+ * `refname`. In other words, find the location of first thing *after*\n+ * `refname`.\n+ *\n+ * Other semantics are identical to the ones in\n+ * `find_reference_location()`.\n+ */\n+static const char *find_reference_location_end(struct snapshot *snapshot,\n+\t\t\t\t\t       const char *refname,\n+\t\t\t\t\t       int mustexist)\n+{\n+\treturn find_reference_location_1(snapshot, refname, mustexist, 0);\n }\n \n /*\n@@ -786,6 +803,13 @@ struct packed_ref_iterator {\n \t/* The end of the part of the buffer that will be iterated over: */\n \tconst char *eof;\n \n+\tstruct jump_list_entry {\n+\t\tconst char *start;\n+\t\tconst char *end;\n+\t} *jump;\n+\tsize_t jump_nr, jump_alloc;\n+\tsize_t jump_cur;\n+\n \t/* Scratch space for current values: */\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n@@ -803,14 +827,35 @@ struct packed_ref_iterator {\n  */\n static int next_record(struct packed_ref_iterator *iter)\n {\n-\tconst char *p = iter->pos, *eol;\n+\tconst char *p, *eol;\n \n \tstrbuf_reset(&iter->refname_buf);\n \n+\t/*\n+\t * If iter->pos is contained within a skipped region, jump past\n+\t * it.\n+\t *\n+\t * Note that each skipped region is considered at most once,\n+\t * since they are ordered based on their starting position.\n+\t */\n+\twhile (iter->jump_cur < iter->jump_nr) {\n+\t\tstruct jump_list_entry *curr = &iter->jump[iter->jump_cur];\n+\t\tif (iter->pos < curr->start)\n+\t\t\tbreak; /* not to the next jump yet */\n+\n+\t\titer->jump_cur++;\n+\t\tif (iter->pos < curr->end) {\n+\t\t\titer->pos = curr->end;\n+\t\t\t/* jumps are coalesced, so only one jump is necessary */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n \tif (iter->pos == iter->eof)\n \t\treturn ITER_DONE;\n \n \titer->base.flags = REF_ISPACKED;\n+\tp = iter->pos;\n \n \tif (iter->eof - p < the_hash_algo->hexsz + 2 ||\n \t    parse_oid_hex(p, &iter->oid, &p) ||\n@@ -918,6 +963,7 @@ static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \tint ok = ITER_DONE;\n \n \tstrbuf_release(&iter->refname_buf);\n+\tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n \tbase_ref_iterator_free(ref_iterator);\n \treturn ok;\n@@ -929,6 +975,108 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.abort = packed_ref_iterator_abort\n };\n \n+static int jump_list_entry_cmp(const void *va, const void *vb)\n+{\n+\tconst struct jump_list_entry *a = va;\n+\tconst struct jump_list_entry *b = vb;\n+\n+\tif (a->start < b->start)\n+\t\treturn -1;\n+\tif (a->start > b->start)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int has_glob_special(const char *str)\n+{\n+\tconst char *p;\n+\tfor (p = str; *p; p++) {\n+\t\tif (is_glob_special(*p))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static void populate_excluded_jump_list(struct packed_ref_iterator *iter,\n+\t\t\t\t\tstruct snapshot *snapshot,\n+\t\t\t\t\tconst char **excluded_patterns)\n+{\n+\tsize_t i, j;\n+\tconst char **pattern;\n+\tstruct jump_list_entry *last_disjoint;\n+\n+\tif (!excluded_patterns)\n+\t\treturn;\n+\n+\tfor (pattern = excluded_patterns; *pattern; pattern++) {\n+\t\tstruct jump_list_entry *e;\n+\t\tconst char *start, *end;\n+\n+\t\t/*\n+\t\t * We can't feed any excludes with globs in them to the\n+\t\t * refs machinery.  It only understands prefix matching.\n+\t\t * We likewise can't even feed the string leading up to\n+\t\t * the first meta-character, as something like \"foo[a]\"\n+\t\t * should not exclude \"foobar\" (but the prefix \"foo\"\n+\t\t * would match that and mark it for exclusion).\n+\t\t */\n+\t\tif (has_glob_special(*pattern))\n+\t\t\tcontinue;\n+\n+\t\tstart = find_reference_location(snapshot, *pattern, 0);\n+\t\tend = find_reference_location_end(snapshot, *pattern, 0);\n+\n+\t\tif (start == end)\n+\t\t\tcontinue; /* nothing to jump over */\n+\n+\t\tALLOC_GROW(iter->jump, iter->jump_nr + 1, iter->jump_alloc);\n+\n+\t\te = &iter->jump[iter->jump_nr++];\n+\t\te->start = start;\n+\t\te->end = end;\n+\t}\n+\n+\tif (!iter->jump_nr) {\n+\t\t/*\n+\t\t * Every entry in exclude_patterns has a meta-character,\n+\t\t * nothing to do here.\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\tQSORT(iter->jump, iter->jump_nr, jump_list_entry_cmp);\n+\n+\t/*\n+\t * As an optimization, merge adjacent entries in the jump list\n+\t * to jump forwards as far as possible when entering a skipped\n+\t * region.\n+\t *\n+\t * For example, if we have two skipped regions:\n+\t *\n+\t *\t[[A, B], [B, C]]\n+\t *\n+\t * we want to combine that into a single entry jumping from A to\n+\t * C.\n+\t */\n+\tlast_disjoint = iter->jump;\n+\n+\tfor (i = 1, j = 1; i < iter->jump_nr; i++) {\n+\t\tstruct jump_list_entry *ours = &iter->jump[i];\n+\t\tif (ours->start <= last_disjoint->end) {\n+\t\t\t/* overlapping regions extend the previous one */\n+\t\t\tlast_disjoint->end = last_disjoint->end > ours->end\n+\t\t\t\t? last_disjoint->end : ours->end;\n+\t\t} else {\n+\t\t\t/* otherwise, insert a new region */\n+\t\t\titer->jump[j++] = *ours;\n+\t\t\tlast_disjoint = ours;\n+\t\t}\n+\t}\n+\n+\titer->jump_nr = j;\n+\titer->jump_cur = 0;\n+}\n+\n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n \t\tconst char *prefix, const char **exclude_patterns,\n@@ -964,6 +1112,9 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable, 1);\n \n+\tif (exclude_patterns)\n+\t\tpopulate_excluded_jump_list(iter, snapshot, exclude_patterns);\n+\n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n \ndiff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\nindex fb18831ec2f..13865425dd1 100644\n--- a/t/helper/test-ref-store.c\n+++ b/t/helper/test-ref-store.c\n@@ -186,6 +186,15 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n \treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n }\n \n+static int cmd_for_each_ref__exclude(struct ref_store *refs, const char **argv)\n+{\n+\tconst char *prefix = notnull(*argv++, \"prefix\");\n+\tconst char **exclude_patterns = argv;\n+\n+\treturn refs_for_each_fullref_in(refs, prefix, exclude_patterns, each_ref,\n+\t\t\t\t\tNULL);\n+}\n+\n static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\n {\n \tstruct object_id oid = *null_oid();\n@@ -318,6 +327,7 @@ static struct command commands[] = {\n \t{ \"delete-refs\", cmd_delete_refs },\n \t{ \"rename-ref\", cmd_rename_ref },\n \t{ \"for-each-ref\", cmd_for_each_ref },\n+\t{ \"for-each-ref--exclude\", cmd_for_each_ref__exclude },\n \t{ \"resolve-ref\", cmd_resolve_ref },\n \t{ \"verify-ref\", cmd_verify_ref },\n \t{ \"for-each-reflog\", cmd_for_each_reflog },\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nnew file mode 100755\nindex 00000000000..bc534c8ea18\n--- /dev/null\n+++ b/t/t1419-exclude-refs.sh\n@@ -0,0 +1,101 @@\n+#!/bin/sh\n+\n+test_description='test exclude_patterns functionality in main ref store'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+TEST_PASSES_SANITIZE_LEAK=true\n+. ./test-lib.sh\n+\n+for_each_ref__exclude () {\n+\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tcut -d ' ' -f 2 actual.raw\n+}\n+\n+for_each_ref () {\n+\tgit for-each-ref --format='%(refname)' \"$@\"\n+}\n+\n+test_expect_success 'setup' '\n+\ttest_commit --no-tag base &&\n+\tbase=\"$(git rev-parse HEAD)\" &&\n+\n+\tfor name in foo bar baz quux\n+\tdo\n+\t\tfor i in 1 2 3\n+\t\tdo\n+\t\t\techo \"create refs/heads/$name/$i $base\" || return 1\n+\t\tdone || return 1\n+\tdone >in &&\n+\techo \"delete refs/heads/main\" >>in &&\n+\n+\tgit update-ref --stdin <in &&\n+\tgit pack-refs --all\n+'\n+\n+test_expect_success 'excluded region in middle' '\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at beginning' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'excluded region at end' '\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'disjoint excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'adjacent, non-overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'several overlapping excluded regions' '\n+\tfor_each_ref__exclude refs/heads \\\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\tfor_each_ref refs/heads/quux >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'non-matching excluded section' '\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'meta-characters are discarded' '\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref >expect &&\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479361","messageId":"e150941c1d1b98ee10094f9a7b7ce0346a2cd9f4.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 10/16] refs/packed-backend.c: add trace2 counters for jump list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:31Z","receivedAt":"2023-07-10T21:13:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The previous commit added low-level tests to ensure that the packed-refs\niterator did not enumerate excluded sections of the refspace.\n\nHowever, there was no guarantee that these sections weren't being\nvisited, only that they were being suppressed from the output. To harden\nthese tests, add a trace2 counter which tracks the number of regions\nskipped by the packed-refs iterator, and assert on its value.\n\nSuggested-by: Derrick Stolee <derrickstolee@github.com>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs/packed-backend.c   |  2 ++\n t/t1419-exclude-refs.sh | 59 ++++++++++++++++++++++++++++-------------\n trace2.h                |  2 ++\n trace2/tr2_ctr.c        |  5 ++++\n 4 files changed, 49 insertions(+), 19 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex d9e008819b5..59c78d7941f 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -13,6 +13,7 @@\n #include \"../statinfo.h\"\n #include \"../wrapper.h\"\n #include \"../write-or-die.h\"\n+#include \"../trace2.h\"\n \n enum mmap_strategy {\n \t/*\n@@ -846,6 +847,7 @@ static int next_record(struct packed_ref_iterator *iter)\n \t\titer->jump_cur++;\n \t\tif (iter->pos < curr->end) {\n \t\t\titer->pos = curr->end;\n+\t\t\ttrace2_counter_add(TRACE2_COUNTER_ID_PACKED_REFS_JUMPS, 1);\n \t\t\t/* jumps are coalesced, so only one jump is necessary */\n \t\t\tbreak;\n \t\t}\ndiff --git a/t/t1419-exclude-refs.sh b/t/t1419-exclude-refs.sh\nindex bc534c8ea18..5d8c86b6573 100755\n--- a/t/t1419-exclude-refs.sh\n+++ b/t/t1419-exclude-refs.sh\n@@ -9,7 +9,8 @@ TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n for_each_ref__exclude () {\n-\ttest-tool ref-store main for-each-ref--exclude \"$@\" >actual.raw\n+\tGIT_TRACE2_PERF=1 test-tool ref-store main \\\n+\t\tfor-each-ref--exclude \"$@\" >actual.raw\n \tcut -d ' ' -f 2 actual.raw\n }\n \n@@ -17,6 +18,17 @@ for_each_ref () {\n \tgit for-each-ref --format='%(refname)' \"$@\"\n }\n \n+assert_jumps () {\n+\tlocal nr=\"$1\"\n+\tlocal trace=\"$2\"\n+\n+\tgrep -q \"name:jumps_made value:$nr$\" $trace\n+}\n+\n+assert_no_jumps () {\n+\t! assert_jumps \".*\" \"$1\"\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit --no-tag base &&\n \tbase=\"$(git rev-parse HEAD)\" &&\n@@ -35,67 +47,76 @@ test_expect_success 'setup' '\n '\n \n test_expect_success 'excluded region in middle' '\n-\tfor_each_ref__exclude refs/heads refs/heads/foo >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/bar refs/heads/baz refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at beginning' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'excluded region at end' '\n-\tfor_each_ref__exclude refs/heads refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/bar refs/heads/baz >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'disjoint excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/quux >actual 2>perf &&\n \tfor_each_ref refs/heads/baz refs/heads/foo >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 2 perf\n '\n \n test_expect_success 'adjacent, non-overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/bar refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'overlapping excluded regions' '\n-\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/ba refs/heads/baz >actual 2>perf &&\n \tfor_each_ref refs/heads/foo refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'several overlapping excluded regions' '\n \tfor_each_ref__exclude refs/heads \\\n-\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual &&\n+\t\trefs/heads/bar refs/heads/baz refs/heads/foo >actual 2>perf &&\n \tfor_each_ref refs/heads/quux >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_jumps 1 perf\n '\n \n test_expect_success 'non-matching excluded section' '\n-\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual &&\n+\tfor_each_ref__exclude refs/heads refs/heads/does/not/exist >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps perf\n '\n \n test_expect_success 'meta-characters are discarded' '\n-\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual &&\n+\tfor_each_ref__exclude refs/heads \"refs/heads/ba*\" >actual 2>perf &&\n \tfor_each_ref >expect &&\n \n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\tassert_no_jumps perf\n '\n \n test_done\ndiff --git a/trace2.h b/trace2.h\nindex f5c5a9e6bac..850056128fc 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -552,6 +552,8 @@ enum trace2_counter_id {\n \tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n \tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n \n+\tTRACE2_COUNTER_ID_PACKED_REFS_JUMPS, /* counts number of jumps */\n+\n \t/* Add additional counter definitions before here. */\n \tTRACE2_NUMBER_OF_COUNTERS\n };\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nindex b342d3b1a3c..50570d01653 100644\n--- a/trace2/tr2_ctr.c\n+++ b/trace2/tr2_ctr.c\n@@ -27,6 +27,11 @@ static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTER\n \t\t.name = \"test2\",\n \t\t.want_per_thread_events = 1,\n \t},\n+\t[TRACE2_COUNTER_ID_PACKED_REFS_JUMPS] = {\n+\t\t.category = \"packed-refs\",\n+\t\t.name = \"jumps_made\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n \n \t/* Add additional metadata before here. */\n };\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479362","messageId":"a5093d52008898353623613f0b42752c1a58f9aa.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 11/16] revision.h: store hidden refs in a `strvec`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:33Z","receivedAt":"2023-07-10T21:13:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, it will be convenient to have a 'const char **'\nof hidden refs (matching `transfer.hiderefs`, `uploadpack.hideRefs`,\netc.), instead of a `string_list`.\n\nConvert spots throughout the tree that store the list of hidden refs\nfrom a `string_list` to a `strvec`.\n\nNote that in `parse_hide_refs_config()` there is an ugly const-cast used\nto avoid an extra copy of each value before trimming any trailing slash\ncharacters. This could instead be written as:\n\n    ref = xstrdup(value);\n    len = strlen(ref);\n    while (len && ref[len - 1] == '/')\n            ref[--len] = '\\0';\n    strvec_push(hide_refs, ref);\n    free(ref);\n\nbut the double-copy (once when calling `xstrdup()`, and another via\n`strvec_push()`) is wasteful.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c |  4 ++--\n ls-refs.c              |  6 +++---\n refs.c                 | 11 ++++++-----\n refs.h                 |  4 ++--\n revision.c             |  2 +-\n revision.h             |  5 +++--\n upload-pack.c          | 10 +++++-----\n 7 files changed, 22 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex faa8f84c5a1..56e8b4e6af4 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -91,7 +91,7 @@ static struct object_id push_cert_oid;\n static struct signature_check sigcheck;\n static const char *push_cert_nonce;\n static const char *cert_nonce_seed;\n-static struct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+static struct strvec hidden_refs = STRVEC_INIT;\n \n static const char *NONCE_UNSOLICITED = \"UNSOLICITED\";\n static const char *NONCE_BAD = \"BAD\";\n@@ -2621,7 +2621,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tpacket_flush(1);\n \toid_array_clear(&shallow);\n \toid_array_clear(&ref);\n-\tstring_list_clear(&hidden_refs, 0);\n+\tstrvec_clear(&hidden_refs);\n \tfree((void *)push_cert_nonce);\n \treturn 0;\n }\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 65d1fa9029e..9291ddfdf8c 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -72,7 +72,7 @@ struct ls_refs_data {\n \tunsigned symrefs;\n \tstruct strvec prefixes;\n \tstruct strbuf buf;\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \tunsigned unborn : 1;\n };\n \n@@ -156,7 +156,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tmemset(&data, 0, sizeof(data));\n \tstrvec_init(&data.prefixes);\n \tstrbuf_init(&data.buf, 0);\n-\tstring_list_init_dup(&data.hidden_refs);\n+\tstrvec_init(&data.hidden_refs);\n \n \tgit_config(ls_refs_config, &data);\n \n@@ -198,7 +198,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-\tstring_list_clear(&data.hidden_refs, 0);\n+\tstrvec_clear(&data.hidden_refs);\n \treturn 0;\n }\n \ndiff --git a/refs.c b/refs.c\nindex 7e5cd84bf28..64dabc40cae 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1429,7 +1429,7 @@ char *shorten_unambiguous_ref(const char *refname, int strict)\n }\n \n int parse_hide_refs_config(const char *var, const char *value, const char *section,\n-\t\t\t   struct string_list *hide_refs)\n+\t\t\t   struct strvec *hide_refs)\n {\n \tconst char *key;\n \tif (!strcmp(\"transfer.hiderefs\", var) ||\n@@ -1440,22 +1440,23 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti\n \n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n-\t\tref = xstrdup(value);\n+\n+\t\t/* drop const to remove trailing '/' characters */\n+\t\tref = (char *)strvec_push(hide_refs, value);\n \t\tlen = strlen(ref);\n \t\twhile (len && ref[len - 1] == '/')\n \t\t\tref[--len] = '\\0';\n-\t\tstring_list_append_nodup(hide_refs, ref);\n \t}\n \treturn 0;\n }\n \n int ref_is_hidden(const char *refname, const char *refname_full,\n-\t\t  const struct string_list *hide_refs)\n+\t\t  const struct strvec *hide_refs)\n {\n \tint i;\n \n \tfor (i = hide_refs->nr - 1; i >= 0; i--) {\n-\t\tconst char *match = hide_refs->items[i].string;\n+\t\tconst char *match = hide_refs->v[i];\n \t\tconst char *subject;\n \t\tint neg = 0;\n \t\tconst char *p;\ndiff --git a/refs.h b/refs.h\nindex 1ca1cdb9da5..fa3ce4b0bc1 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -820,7 +820,7 @@ int update_ref(const char *msg, const char *refname,\n \t       unsigned int flags, enum action_on_err onerr);\n \n int parse_hide_refs_config(const char *var, const char *value, const char *,\n-\t\t\t   struct string_list *);\n+\t\t\t   struct strvec *);\n \n /*\n  * Check whether a ref is hidden. If no namespace is set, both the first and\n@@ -830,7 +830,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,\n  * the ref is outside that namespace, the first parameter is NULL. The second\n  * parameter always points to the full ref name.\n  */\n-int ref_is_hidden(const char *, const char *, const struct string_list *);\n+int ref_is_hidden(const char *, const char *, const struct strvec *);\n \n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\ndiff --git a/revision.c b/revision.c\nindex af11d599d35..a15b18fc8e6 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1561,7 +1561,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)\n void clear_ref_exclusions(struct ref_exclusions *exclusions)\n {\n \tstring_list_clear(&exclusions->excluded_refs, 0);\n-\tstring_list_clear(&exclusions->hidden_refs, 0);\n+\tstrvec_clear(&exclusions->hidden_refs);\n \texclusions->hidden_refs_configured = 0;\n }\n \ndiff --git a/revision.h b/revision.h\nindex 25776af3815..7f219cde62f 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -10,6 +10,7 @@\n #include \"decorate.h\"\n #include \"ident.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"strvec.h\"\n \n /**\n  * The revision walking API offers functions to build a list of revisions\n@@ -95,7 +96,7 @@ struct ref_exclusions {\n \t * Hidden refs is a list of patterns that is to be hidden via\n \t * `ref_is_hidden()`.\n \t */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \t/*\n \t * Indicates whether hidden refs have been configured. This is to\n@@ -110,7 +111,7 @@ struct ref_exclusions {\n  */\n #define REF_EXCLUSIONS_INIT { \\\n \t.excluded_refs = STRING_LIST_INIT_DUP, \\\n-\t.hidden_refs = STRING_LIST_INIT_DUP, \\\n+\t.hidden_refs = STRVEC_INIT, \\\n }\n \n struct oidset;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 946074920a8..cfb61ccbb5c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -69,7 +69,7 @@ struct upload_pack_data {\n \tstruct object_array have_obj;\n \tstruct oid_array haves;\t\t\t\t\t/* v2 only */\n \tstruct string_list wanted_refs;\t\t\t\t/* v2 only */\n-\tstruct string_list hidden_refs;\n+\tstruct strvec hidden_refs;\n \n \tstruct object_array shallows;\n \tstruct string_list deepen_not;\n@@ -127,7 +127,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n \tstruct string_list wanted_refs = STRING_LIST_INIT_DUP;\n-\tstruct string_list hidden_refs = STRING_LIST_INIT_DUP;\n+\tstruct strvec hidden_refs = STRVEC_INIT;\n \tstruct object_array want_obj = OBJECT_ARRAY_INIT;\n \tstruct object_array have_obj = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n@@ -162,7 +162,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tstring_list_clear(&data->symref, 1);\n \tstring_list_clear(&data->wanted_refs, 1);\n-\tstring_list_clear(&data->hidden_refs, 0);\n+\tstrvec_clear(&data->hidden_refs);\n \tobject_array_clear(&data->want_obj);\n \tobject_array_clear(&data->have_obj);\n \toid_array_clear(&data->haves);\n@@ -1170,7 +1170,7 @@ static void receive_needs(struct upload_pack_data *data,\n \n /* return non-zero if the ref is hidden, otherwise 0 */\n static int mark_our_ref(const char *refname, const char *refname_full,\n-\t\t\tconst struct object_id *oid, const struct string_list *hidden_refs)\n+\t\t\tconst struct object_id *oid, const struct strvec *hidden_refs)\n {\n \tstruct object *o = lookup_unknown_object(the_repository, oid);\n \n@@ -1471,7 +1471,7 @@ static int parse_want(struct packet_writer *writer, const char *line,\n \n static int parse_want_ref(struct packet_writer *writer, const char *line,\n \t\t\t  struct string_list *wanted_refs,\n-\t\t\t  struct string_list *hidden_refs,\n+\t\t\t  struct strvec *hidden_refs,\n \t\t\t  struct object_array *want_obj)\n {\n \tconst char *refname_nons;\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479363","messageId":"7e72c23c8a4e579d324446910e8b680cca560d8a.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 12/16] refs.h: let `for_each_namespaced_ref()` take excluded patterns","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:36Z","receivedAt":"2023-07-10T21:13:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A future commit will want to call `for_each_namespaced_ref()` with\na list of excluded patterns.\n\nWe could introduce a variant of that function, say,\n`for_each_namespaced_ref_exclude()` which takes the extra parameter, and\nreimplement the original function in terms of that. But all but one\ncaller (in `http-backend.c`) will supply the new parameter, so add the\nnew parameter to `for_each_namespaced_ref()` itself instead of\nintroducing a new function.\n\nFor now, supply NULL for the list of excluded patterns at all callers to\navoid changing behavior, which we will do in a future change.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n http-backend.c | 2 +-\n refs.c         | 5 +++--\n refs.h         | 7 ++++++-\n upload-pack.c  | 6 +++---\n 4 files changed, 13 insertions(+), 7 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex e1969c05dc6..895fac35f79 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -560,7 +560,7 @@ static void get_info_refs(struct strbuf *hdr, char *arg UNUSED)\n \n \t} else {\n \t\tselect_getanyfile(hdr);\n-\t\tfor_each_namespaced_ref(show_text_ref, &buf);\n+\t\tfor_each_namespaced_ref(NULL, show_text_ref, &buf);\n \t\tsend_strbuf(hdr, \"text/plain\", &buf);\n \t}\n \tstrbuf_release(&buf);\ndiff --git a/refs.c b/refs.c\nindex 64dabc40cae..3065e514fd0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1662,13 +1662,14 @@ int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_dat\n \t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, NULL, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, exclude_patterns, fn, 0, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\ndiff --git a/refs.h b/refs.h\nindex fa3ce4b0bc1..7871faed632 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -382,7 +382,12 @@ int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,\n \t\t\t const char *prefix, void *cb_data);\n \n int head_ref_namespaced(each_ref_fn fn, void *cb_data);\n-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);\n+/*\n+ * references matching any pattern in \"exclude_patterns\" are omitted from the\n+ * result set on a best-effort basis.\n+ */\n+int for_each_namespaced_ref(const char **exclude_patterns,\n+\t\t\t    each_ref_fn fn, void *cb_data);\n \n /* can be used to learn about broken ref and symref */\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex cfb61ccbb5c..da4f17f64ac 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -855,7 +855,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(check_ref, data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1392,7 +1392,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(send_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n \t\tif (!data.sent_capabilities) {\n \t\t\tconst char *refname = \"capabilities^{}\";\n \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n@@ -1406,7 +1406,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(check_ref, &data);\n+\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479364","messageId":"96aada36f9b7322ea408d1a49f94e051265596f0.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 14/16] builtin/receive-pack.c: avoid enumerating hidden references","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:42Z","receivedAt":"2023-07-10T21:13:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that `refs_for_each_fullref_in()` has the ability to avoid\nenumerating references matching certain pattern(s), use that to avoid\nvisiting hidden refs when constructing the ref advertisement via\nreceive-pack.\n\nNote that since this exclusion is best-effort, we still need\n`show_ref_cb()` to check whether or not each reference is hidden or not\nbefore including it in the advertisement.\n\nAs was the case when applying this same optimization to `upload-pack`,\n`receive-pack`'s reference advertisement phase can proceed much quicker\nby avoiding enumerating references that will not be part of the\nadvertisement.\n\n(Below, we're still using linux.git with one hidden refs/pull/N ref per\ncommit):\n\n    $ hyperfine -L v ,.compile 'git{v} -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):      89.1 ms ±   1.7 ms    [User: 82.0 ms, System: 7.0 ms]\n      Range (min … max):    87.7 ms …  95.5 ms    31 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git\n      Time (mean ± σ):       4.5 ms ±   0.2 ms    [User: 0.5 ms, System: 3.9 ms]\n      Range (min … max):     4.1 ms …   5.6 ms    508 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git' ran\n       20.00 ± 1.05 times faster than 'git -c transfer.hideRefs=refs/pull receive-pack --advertise-refs .git'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/receive-pack.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 56e8b4e6af4..90aedb97cc8 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -339,7 +339,9 @@ static void write_head_info(void)\n {\n \tstatic struct oidset seen = OIDSET_INIT;\n \n-\tfor_each_ref(show_ref_cb, &seen);\n+\trefs_for_each_fullref_in(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t hidden_refs_to_excludes(&hidden_refs),\n+\t\t\t\t show_ref_cb, &seen);\n \tfor_each_alternate_ref(show_one_alternate_ref, &seen);\n \toidset_clear(&seen);\n \tif (!sent_capabilities)\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479365","messageId":"f99d89d53b9640db656109de06dd5e084754cc0e.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 13/16] refs.h: implement `hidden_refs_to_excludes()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:39Z","receivedAt":"2023-07-10T21:13:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, we'll teach `receive-pack` and `upload-pack` to\nuse the new jump list feature in the packed-refs iterator by ignoring\nreferences which are mentioned via its respective hideRefs lists.\n\nHowever, the packed-ref jump lists cannot handle un-hiding rules (that\nbegin with '!'), or namespace comparisons (that begin with '^'). Add a\nconvenience function to the refs.h API to detect when either of these\nconditions are met, and returns an appropriate value to pass as excluded\npatterns.\n\nSuggested-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n refs.c | 24 ++++++++++++++++++++++++\n refs.h |  6 ++++++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 3065e514fd0..0916a6ba582 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1482,6 +1482,30 @@ int ref_is_hidden(const char *refname, const char *refname_full,\n \treturn 0;\n }\n \n+const char **hidden_refs_to_excludes(const struct strvec *hide_refs)\n+{\n+\tconst char **pattern;\n+\tfor (pattern = hide_refs->v; *pattern; pattern++) {\n+\t\t/*\n+\t\t * We can't feed any excludes from hidden refs config\n+\t\t * sections, since later rules may override previous\n+\t\t * ones. For example, with rules \"refs/foo\" and\n+\t\t * \"!refs/foo/bar\", we should show \"refs/foo/bar\" (and\n+\t\t * everything underneath it), but the earlier exclusion\n+\t\t * would cause us to skip all of \"refs/foo\".  We\n+\t\t * likewise don't implement the namespace stripping\n+\t\t * required for '^' rules.\n+\t\t *\n+\t\t * Both are possible to do, but complicated, so avoid\n+\t\t * populating the jump list at all if we see either of\n+\t\t * these patterns.\n+\t\t */\n+\t\tif (**pattern == '!' || **pattern == '^')\n+\t\t\treturn NULL;\n+\t}\n+\treturn hide_refs->v;\n+}\n+\n const char *find_descendant_ref(const char *dirname,\n \t\t\t\tconst struct string_list *extras,\n \t\t\t\tconst struct string_list *skip)\ndiff --git a/refs.h b/refs.h\nindex 7871faed632..23211a5ea1c 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -837,6 +837,12 @@ int parse_hide_refs_config(const char *var, const char *value, const char *,\n  */\n int ref_is_hidden(const char *, const char *, const struct strvec *);\n \n+/*\n+ * Returns an array of patterns to use as excluded_patterns, if none of the\n+ * hidden references use the token '!' or '^'.\n+ */\n+const char **hidden_refs_to_excludes(const struct strvec *hide_refs);\n+\n /* Is this a per-worktree ref living in the refs/ namespace? */\n int is_per_worktree_ref(const char *refname);\n \n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479366","messageId":"8544a647798de68eb46f9f5c5b269d03ed3500a5.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 15/16] upload-pack.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:45Z","receivedAt":"2023-07-10T21:13:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as a previous commit, teach `upload-pack` to avoid\nenumerating hidden references where possible.\n\nNote, however, that there are certain cases where cannot avoid\nenumerating even hidden references, in particular when either of:\n\n  - `uploadpack.allowTipSHA1InWant`, or\n  - `uploadpack.allowReachableSHA1InWant`\n\nare set, corresponding to `ALLOW_TIP_SHA1` and `ALLOW_REACHABLE_SHA1`,\nrespectively.\n\nWhen either of these bits are set, upload-pack's `is_our_ref()` function\nneeds to consider the `HIDDEN_REF` bit of the referent's object flags.\nSo we must visit all references, including the hidden ones, in order to\nmark their referents with the `HIDDEN_REF` bit.\n\nWhen neither `ALLOW_TIP_SHA1` nor `ALLOW_REACHABLE_SHA1` are set, the\n`is_our_ref()` function considers only the `OUR_REF` bit, and not the\n`HIDDEN_REF` one. `OUR_REF` is applied via `mark_our_ref()`, and only\nto objects at the tips of non-hidden references, so we do not need to\nvisit hidden references in this case.\n\nWhen neither of those bits are set, `upload-pack` can potentially avoid\nenumerating a large number of references. In the same example as a\nprevious commit (linux.git with one hidden reference per commit,\n\"refs/pull/N\"):\n\n    $ printf 0000 >in\n    $ hyperfine --warmup=1 \\\n      'git -c transfer.hideRefs=refs/pull upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in' \\\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in'\n    Benchmark 1: git -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):     406.9 ms ±   1.1 ms    [User: 357.3 ms, System: 49.5 ms]\n      Range (min … max):   405.7 ms … 409.2 ms    10 runs\n\n    Benchmark 2: git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in\n      Time (mean ± σ):     406.5 ms ±   1.3 ms    [User: 356.5 ms, System: 49.9 ms]\n      Range (min … max):   404.6 ms … 408.8 ms    10 runs\n\n    Benchmark 3: git.compile -c transfer.hideRefs=refs/pull upload-pack . <in\n      Time (mean ± σ):       4.7 ms ±   0.2 ms    [User: 0.7 ms, System: 3.9 ms]\n      Range (min … max):     4.3 ms …   6.1 ms    472 runs\n\n    Summary\n      'git.compile -c transfer.hideRefs=refs/pull upload-pack . <in' ran\n       86.62 ± 4.33 times faster than 'git.compile -c transfer.hideRefs=refs/pull -c uploadpack.allowTipSHA1InWant upload-pack . <in'\n       86.70 ± 4.33 times faster than 'git -c transfer.hideRefs=refs/pull upload-pack . <in'\n\nAs above, we must visit every reference when\nuploadPack.allowTipSHA1InWant is set. But when it is unset, we can visit\nfar fewer references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n upload-pack.c | 37 +++++++++++++++++++++++++++++++------\n 1 file changed, 31 insertions(+), 6 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex da4f17f64ac..db4709adb77 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -602,11 +602,36 @@ static int get_common_commits(struct upload_pack_data *data,\n \t}\n }\n \n+static int allow_hidden_refs(enum allow_uor allow_uor)\n+{\n+\tif ((allow_uor & ALLOW_ANY_SHA1) == ALLOW_ANY_SHA1)\n+\t\treturn 1;\n+\treturn !(allow_uor & (ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n+}\n+\n+static void for_each_namespaced_ref_1(each_ref_fn fn,\n+\t\t\t\t      struct upload_pack_data *data)\n+{\n+\tconst char **excludes = NULL;\n+\t/*\n+\t * If `data->allow_uor` allows fetching hidden refs, we need to\n+\t * mark all references (including hidden ones), to check in\n+\t * `is_our_ref()` below.\n+\t *\n+\t * Otherwise, we only care about whether each reference's object\n+\t * has the OUR_REF bit set or not, so do not need to visit\n+\t * hidden references.\n+\t */\n+\tif (allow_hidden_refs(data->allow_uor))\n+\t\texcludes = hidden_refs_to_excludes(&data->hidden_refs);\n+\n+\tfor_each_namespaced_ref(excludes, fn, data);\n+}\n+\n+\n static int is_our_ref(struct object *o, enum allow_uor allow_uor)\n {\n-\tint allow_hidden_ref = (allow_uor &\n-\t\t\t\t(ALLOW_TIP_SHA1 | ALLOW_REACHABLE_SHA1));\n-\treturn o->flags & ((allow_hidden_ref ? HIDDEN_REF : 0) | OUR_REF);\n+\treturn o->flags & ((allow_hidden_refs(allow_uor) ? 0 : HIDDEN_REF) | OUR_REF);\n }\n \n /*\n@@ -855,7 +880,7 @@ static void deepen(struct upload_pack_data *data, int depth)\n \t\t * marked with OUR_REF.\n \t\t */\n \t\thead_ref_namespaced(check_ref, data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, data);\n+\t\tfor_each_namespaced_ref_1(check_ref, data);\n \n \t\tget_reachable_list(data, &reachable_shallows);\n \t\tresult = get_shallow_commits(&reachable_shallows,\n@@ -1392,7 +1417,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tif (advertise_refs)\n \t\t\tdata.no_done = 1;\n \t\thead_ref_namespaced(send_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, send_ref, &data);\n+\t\tfor_each_namespaced_ref_1(send_ref, &data);\n \t\tif (!data.sent_capabilities) {\n \t\t\tconst char *refname = \"capabilities^{}\";\n \t\t\twrite_v0_ref(&data, refname, refname, null_oid());\n@@ -1406,7 +1431,7 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(check_ref, &data);\n-\t\tfor_each_namespaced_ref(NULL, check_ref, &data);\n+\t\tfor_each_namespaced_ref_1(check_ref, &data);\n \t}\n \n \tif (!advertise_refs) {\n-- \n2.41.0.343.gdff068c469f\n\n"},{"id":"479367","messageId":"dff068c469f9c59c94775209e34edfb678b627d9.1689023520.git.me@ttaylorr.com","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"[PATCH v5 16/16] ls-refs.c: avoid enumerating hidden refs where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-10T21:12:48Z","receivedAt":"2023-07-10T21:13:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as in previous commits, teach `ls-refs` to avoid\nenumerating hidden references where possible.\n\nAs before, this is linux.git with one hidden reference per commit.\n\n    $ hyperfine -L v ,.compile 'git{v} -c protocol.version=2 ls-remote .'\n    Benchmark 1: git -c protocol.version=2 ls-remote .\n      Time (mean ± σ):      89.8 ms ±   0.6 ms    [User: 84.3 ms, System: 5.7 ms]\n      Range (min … max):    88.8 ms …  91.3 ms    32 runs\n\n    Benchmark 2: git.compile -c protocol.version=2 ls-remote .\n      Time (mean ± σ):       6.5 ms ±   0.1 ms    [User: 2.4 ms, System: 4.3 ms]\n      Range (min … max):     6.2 ms …   8.3 ms    397 runs\n\n    Summary\n      'git.compile -c protocol.version=2 ls-remote .' ran\n       13.85 ± 0.33 times faster than 'git -c protocol.version=2 ls-remote .'\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n ls-refs.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/ls-refs.c b/ls-refs.c\nindex 9291ddfdf8c..0e49b932c30 100644\n--- a/ls-refs.c\n+++ b/ls-refs.c\n@@ -194,7 +194,8 @@ int ls_refs(struct repository *r, struct packet_reader *request)\n \t\tstrvec_push(&data.prefixes, \"\");\n \trefs_for_each_fullref_in_prefixes(get_main_ref_store(r),\n \t\t\t\t\t  get_git_namespace(), data.prefixes.v,\n-\t\t\t\t\t  NULL, send_ref, &data);\n+\t\t\t\t\t  hidden_refs_to_excludes(&data.hidden_refs),\n+\t\t\t\t\t  send_ref, &data);\n \tpacket_fflush(stdout);\n \tstrvec_clear(&data.prefixes);\n \tstrbuf_release(&data.buf);\n-- \n2.41.0.343.gdff068c469f\n"},{"id":"479369","messageId":"xmqq5y6r4con.fsf@gitster.g","threadId":"59715","inReplyTo":"cover.1689023520.git.me@ttaylorr.com","subject":"Re: [PATCH v5 00/16] refs: implement jump lists for packed backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-10T22:35:52Z","receivedAt":"2023-07-10T22:39:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Here is another reroll of my series to implement jump (née skip) lists\n> for the packed refs backend, based on top of the current 'master'.\n\nI've skimmed the whole set again and nothing jumped at me as an\nunexpected change.  Let's wait for a few days to see if we have\nothers comment on the patches and then merge it down to 'next'\nunless there is something spotted in there.\n\nThanks.\n"},{"id":"479373","messageId":"eddehgbfqnmkhvhkacbvnqiiripbn6jvjawpy76ysfnpohsygt@a43fbutqg64z","threadId":"59715","inReplyTo":"xmqq5y6r4con.fsf@gitster.g","subject":"Re: [PATCH v5 00/16] refs: implement jump lists for packed backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-07-11T09:37:56Z","receivedAt":"2023-07-11T09:38:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jul 10, 2023 at 03:35:52PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n> \n> > Here is another reroll of my series to implement jump (née skip) lists\n> > for the packed refs backend, based on top of the current 'master'.\n> \n> I've skimmed the whole set again and nothing jumped at me as an\n> unexpected change.  Let's wait for a few days to see if we have\n> others comment on the patches and then merge it down to 'next'\n> unless there is something spotted in there.\n> \n> Thanks.\n\nThe patch series looks good to me, too, so please feel free to add my\nReviewed-by. Thanks for these exciting changes!\n\nPatrick\n"},{"id":"479380","messageId":"xmqqilaq30hi.fsf@gitster.g","threadId":"59715","inReplyTo":"eddehgbfqnmkhvhkacbvnqiiripbn6jvjawpy76ysfnpohsygt@a43fbutqg64z","subject":"Re: [PATCH v5 00/16] refs: implement jump lists for packed backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-11T15:56:57Z","receivedAt":"2023-07-11T15:57:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> On Mon, Jul 10, 2023 at 03:35:52PM -0700, Junio C Hamano wrote:\n>> Taylor Blau <me@ttaylorr.com> writes:\n>> \n>> > Here is another reroll of my series to implement jump (née skip) lists\n>> > for the packed refs backend, based on top of the current 'master'.\n>> \n>> I've skimmed the whole set again and nothing jumped at me as an\n>> unexpected change.  Let's wait for a few days to see if we have\n>> others comment on the patches and then merge it down to 'next'\n>> unless there is something spotted in there.\n>> \n>> Thanks.\n>\n> The patch series looks good to me, too, so please feel free to add my\n> Reviewed-by. Thanks for these exciting changes!\n\nThanks.\n"},{"id":"479383","messageId":"ZK2PDJbGQoTcR/si@nand.local","threadId":"59715","inReplyTo":"xmqqilaq30hi.fsf@gitster.g","subject":"Re: [PATCH v5 00/16] refs: implement jump lists for packed backend","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-11T17:19:08Z","receivedAt":"2023-07-11T17:20:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jul 11, 2023 at 08:56:57AM -0700, Junio C Hamano wrote:\n> >> I've skimmed the whole set again and nothing jumped at me as an\n> >> unexpected change.  Let's wait for a few days to see if we have\n> >> others comment on the patches and then merge it down to 'next'\n> >> unless there is something spotted in there.\n> >>\n> >> Thanks.\n> >\n> > The patch series looks good to me, too, so please feel free to add my\n> > Reviewed-by. Thanks for these exciting changes!\n>\n> Thanks.\n\nThanks, both :-).\n\nThanks,\nTaylor\n"}]}