{"thread":{"id":"60483","subject":"[PATCH 2/9] for-each-ref: clarify interaction of --omit-empty & --count","startedAt":"2023-11-07T01:26:08Z","lastAt":"2023-11-16T12:06:46Z","messageCount":49,"participants":["Victoria Dye via GitGitGadget","Junio C Hamano","Victoria Dye","Patrick Steinhardt","Øystein Walle","Kristoffer Haugsbakk"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"484486","messageId":"pull.1609.git.1699320361.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":null,"subject":"[PATCH 0/9] for-each-ref optimizations & usability improvements","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:52Z","receivedAt":"2023-11-07T01:26:08Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series is a bit of an informal follow-up to [1], adding some more\nsubstantial optimizations and usability fixes around ref\nfiltering/formatting. Some of the changes here affect user-facing behavior,\nsome are internal-only, but they're all interdependent enough to warrant\nputting them together in one series.\n\n[1]\nhttps://lore.kernel.org/git/pull.1594.v2.git.1696888736.gitgitgadget@gmail.com/\n\nPatch 1 changes the behavior of the '--no-sort' option in 'for-each-ref',\n'tag', and 'branch'. Currently, it just removes previous sort keys and, if\nno further keys are specified, falls back on ascending refname sort (which,\nIMO, makes the name '--no-sort' somewhat misleading).\n\nPatch 2 updates the 'for-each-ref' docs to clearly state what happens if you\nuse '--omit-empty' and '--count' together. I based the explanation on what\nthe current behavior is (i.e., refs omitted with '--omit-empty' do count\ntowards the total limited by '--count').\n\nPatches 3-7 incrementally refactor various parts of the ref\nfiltering/formatting workflows in order to create a\n'filter_and_format_refs()' function. If certain conditions are met (sorting\ndisabled, no reachability filtering or ahead-behind formatting), ref\nfiltering & formatting is done within a single 'for_each_fullref_in'\ncallback. Especially in large repositories, this makes a huge difference in\nmemory usage & runtime for certain usages of 'for-each-ref', since it's no\nlonger writing everything to a 'struct ref_array' then repeatedly whittling\ndown/updating its contents.\n\nPatch 8 introduces a new option to 'for-each-ref' called '--full-deref'.\nWhen provided, any format fields for the dereferenced value of a tag (e.g.\n\"%(*objectname)\") will be populated with the fully peeled target of the tag;\nright now, those fields are populated with the immediate target of a tag\n(which can be another tag). This avoids the need to pipe 'for-each-ref'\nresults to 'cat-file --batch-check' to get fully-peeled tag information. It\nalso benefits from the 'filter_and_format_refs()' single-iteration\noptimization, since 'peel_iterated_oid()' may be able to read the\npre-computed peeled OID from a packed ref. A couple notes on this one:\n\n * I went with a command line option for '--full-deref' rather than another\n   format specifier (like ** instead of *) because it seems unlikely that a\n   user is going to want to perform a shallow dereference and a full\n   dereference in the same 'for-each-ref'. There's also a NEEDSWORK going\n   all the way back to the introduction of 'for-each-ref' in 9f613ddd21c\n   (Add git-for-each-ref: helper for language bindings, 2006-09-15) that (to\n   me) implies different dereferencing behavior corresponds to different use\n   cases/user needs.\n * I'm not attached to '--full-deref' as a name - if someone has an idea for\n   a more descriptive name, please suggest it!\n\nFinally, patch 9 adds performance tests for 'for-each-ref', showing the\neffects of optimizations made throughout the series. Here are some sample\nresults from my Ubuntu VM (test names shortened for space):\n\nTest                                                         this branch    \n----------------------------------------------------------------------------\n6300.2: (loose)                                              4.78(0.89+3.82)\n6300.3: (loose, no sort)                                     4.51(0.86+3.58)\n6300.4: (loose, --count=1)                                   4.70(0.90+3.73)\n6300.5: (loose, --count=1, no sort)                          4.35(0.58+3.73)\n6300.6: (loose, tags)                                        2.45(0.44+1.95)\n6300.7: (loose, tags, no sort)                               2.38(0.44+1.90)\n6300.8: (loose, tags, shallow deref)                         3.33(1.27+1.99)\n6300.9: (loose, tags, shallow deref, no sort)                3.29(1.29+1.93)\n6300.10: (loose, tags, full deref)                           3.76(1.69+1.99)\n6300.11: (loose, tags, full deref, no sort)                  3.73(1.71+1.94)\n6300.12: for-each-ref + cat-file (loose, tags)               4.25(2.16+2.17)\n6300.14: (packed)                                            0.61(0.50+0.09)\n6300.15: (packed, no sort)                                   0.46(0.40+0.04)\n6300.16: (packed, --count=1)                                 0.59(0.44+0.13)\n6300.17: (packed, --count=1, no sort)                        0.02(0.01+0.01)\n6300.18: (packed, tags)                                      0.28(0.18+0.09)\n6300.19: (packed, tags, no sort)                             0.29(0.24+0.03)\n6300.20: (packed, tags, shallow deref)                       1.20(1.03+0.13)\n6300.21: (packed, tags, shallow deref, no sort)              1.13(0.99+0.08)\n6300.22: (packed, tags, full deref)                          1.57(1.45+0.11)\n6300.23: (packed, tags, full deref, no sort)                 1.07(1.01+0.05)\n6300.24: for-each-ref + cat-file (packed, tags)              2.01(1.81+0.33)\n\n\n * Victoria\n\nVictoria Dye (9):\n  ref-filter.c: really don't sort when using --no-sort\n  for-each-ref: clarify interaction of --omit-empty & --count\n  ref-filter.h: add max_count and omit_empty to ref_format\n  ref-filter.h: move contains caches into filter\n  ref-filter.h: add functions for filter/format & format-only\n  ref-filter.c: refactor to create common helper functions\n  ref-filter.c: filter & format refs in the same callback\n  for-each-ref: add option to fully dereference tags\n  t/perf: add perf tests for for-each-ref\n\n Documentation/git-for-each-ref.txt |  12 +-\n builtin/branch.c                   |  42 +++--\n builtin/for-each-ref.c             |  41 ++---\n builtin/ls-remote.c                |  10 +-\n builtin/tag.c                      |  32 +---\n ref-filter.c                       | 277 ++++++++++++++++++++---------\n ref-filter.h                       |  26 +++\n t/perf/p6300-for-each-ref.sh       |  87 +++++++++\n t/t3200-branch.sh                  |  68 ++++++-\n t/t6300-for-each-ref.sh            |  55 ++++++\n t/t7004-tag.sh                     |  45 +++++\n 11 files changed, 532 insertions(+), 163 deletions(-)\n create mode 100755 t/perf/p6300-for-each-ref.sh\n\n\nbase-commit: bc5204569f7db44d22477485afd52ea410d83743\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1609%2Fvdye%2Fvdye%2Ffor-each-ref-optimizations-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1609/vdye/vdye/for-each-ref-optimizations-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1609\n-- \ngitgitgadget\n"},{"id":"484485","messageId":"88eba4146cd250fcabfb9ffa9b410ce912a82ce7.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 2/9] for-each-ref: clarify interaction of --omit-empty & --count","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:54Z","receivedAt":"2023-11-07T01:26:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nUpdate the 'for-each-ref' builtin documentation to clarify that refs\n\"omitted\" by --omit-empty are still counted toward the limit specified by\n--count. The use of the term \"omit\" would otherwise be somewhat ambiguous\nand could incorrectly be construed as excluding empty refs entirely (i.e.\nnot counting them towards the total ref count).\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n Documentation/git-for-each-ref.txt | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex e86d5700ddf..407f624fbaa 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -101,7 +101,8 @@ OPTIONS\n \n --omit-empty::\n \tDo not print a newline after formatted refs where the format expands\n-\tto the empty string.\n+\tto the empty string. Although omitted refs are not shown in the output,\n+\tthey still count toward the total limited by `--count`.\n \n --exclude=<pattern>::\n \tIf one or more patterns are given, only refs which do not match\n-- \ngitgitgadget\n\n"},{"id":"484487","messageId":"dea8d7d1e866d9784320051b372ff729fca855d7.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 1/9] ref-filter.c: really don't sort when using --no-sort","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:53Z","receivedAt":"2023-11-07T01:26:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nUpdate 'ref_sorting_options()' to return a NULL 'struct ref_sorting *' if\nthe string list provided to it is empty, rather than returning the default\nrefname sort structure. Also update 'ref_array_sort()' to explicitly skip\nsorting if its 'struct ref_sorting *' arg is NULL. Other functions using\n'struct ref_sorting *' do not need any changes because they already properly\nignore NULL values.\n\nThe goal of this change is to have the '--no-sort' option truly disable\nsorting in commands like 'for-each-ref, 'tag', and 'branch'. Right now,\n'--no-sort' will still trigger refname sorting by default in 'for-each-ref',\n'tag', and 'branch'.\n\nTo match existing behavior as closely as possible, explicitly add \"refname\"\nto the list of sort keys in 'for-each-ref', 'tag', and 'branch' before\nparsing options (if no config-based sort keys are set). This ensures that\nsorting will only be fully disabled if '--no-sort' is provided as an option;\notherwise, \"refname\" sorting will remain the default. Note: this also means\nthat even when sort keys are provided on the command line, \"refname\" will be\nthe final sort key in the sorting structure. This doesn't actually change\nany behavior, since 'compare_refs()' already falls back on comparing\nrefnames if two refs are equal w.r.t all other sort keys.\n\nFinally, remove the condition around sorting in 'ls-remote', since it's no\nlonger necessary. Unlike 'for-each-ref' et. al., it does *not* set any sort\nkeys by default. The default empty list of sort keys will produce a NULL\n'struct ref_sorting *', which causes the sorting to be skipped in\n'ref_array_sort()'.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n builtin/branch.c        |  6 ++++\n builtin/for-each-ref.c  |  3 ++\n builtin/ls-remote.c     | 10 ++----\n builtin/tag.c           |  6 ++++\n ref-filter.c            | 19 ++----------\n t/t3200-branch.sh       | 68 +++++++++++++++++++++++++++++++++++++++--\n t/t6300-for-each-ref.sh | 21 +++++++++++++\n t/t7004-tag.sh          | 45 +++++++++++++++++++++++++++\n 8 files changed, 152 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex e7ee9bd0f15..d67738bbcaa 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -767,7 +767,13 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_branch_usage, options);\n \n+\t/*\n+\t * Try to set sort keys from config. If config does not set any,\n+\t * fall back on default (refname) sorting.\n+\t */\n \tgit_config(git_branch_config, &sorting_options);\n+\tif (!sorting_options.nr)\n+\t\tstring_list_append(&sorting_options, \"refname\");\n \n \ttrack = git_branch_track;\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 350bfa6e811..93b370f550b 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -67,6 +67,9 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\t/* Set default (refname) sorting */\n+\tstring_list_append(&sorting_options, \"refname\");\n+\n \tparse_options(argc, argv, prefix, opts, for_each_ref_usage, 0);\n \tif (maxcount < 0) {\n \t\terror(\"invalid --count argument: `%d'\", maxcount);\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex fc765754305..436249b720c 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -58,6 +58,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tstruct transport *transport;\n \tconst struct ref *ref;\n \tstruct ref_array ref_array;\n+\tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \n \tstruct option options[] = {\n@@ -141,13 +142,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \t\titem->symref = xstrdup_or_null(ref->symref);\n \t}\n \n-\tif (sorting_options.nr) {\n-\t\tstruct ref_sorting *sorting;\n-\n-\t\tsorting = ref_sorting_options(&sorting_options);\n-\t\tref_array_sort(sorting, &ref_array);\n-\t\tref_sorting_release(sorting);\n-\t}\n+\tsorting = ref_sorting_options(&sorting_options);\n+\tref_array_sort(sorting, &ref_array);\n \n \tfor (i = 0; i < ref_array.nr; i++) {\n \t\tconst struct ref_array_item *ref = ref_array.items[i];\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 3918eacbb57..64f3196cd4c 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -501,7 +501,13 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \n \tsetup_ref_filter_porcelain_msg();\n \n+\t/*\n+\t * Try to set sort keys from config. If config does not set any,\n+\t * fall back on default (refname) sorting.\n+\t */\n \tgit_config(git_tag_config, &sorting_options);\n+\tif (!sorting_options.nr)\n+\t\tstring_list_append(&sorting_options, \"refname\");\n \n \tmemset(&opt, 0, sizeof(opt));\n \tfilter.lines = -1;\ndiff --git a/ref-filter.c b/ref-filter.c\nindex e4d3510e28e..7250089b7c6 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -3142,7 +3142,8 @@ void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting,\n \n void ref_array_sort(struct ref_sorting *sorting, struct ref_array *array)\n {\n-\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n+\tif (sorting)\n+\t\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n }\n \n static void append_literal(const char *cp, const char *ep, struct ref_formatting_state *state)\n@@ -3248,18 +3249,6 @@ static int parse_sorting_atom(const char *atom)\n \treturn res;\n }\n \n-/*  If no sorting option is given, use refname to sort as default */\n-static struct ref_sorting *ref_default_sorting(void)\n-{\n-\tstatic const char cstr_name[] = \"refname\";\n-\n-\tstruct ref_sorting *sorting = xcalloc(1, sizeof(*sorting));\n-\n-\tsorting->next = NULL;\n-\tsorting->atom = parse_sorting_atom(cstr_name);\n-\treturn sorting;\n-}\n-\n static void parse_ref_sorting(struct ref_sorting **sorting_tail, const char *arg)\n {\n \tstruct ref_sorting *s;\n@@ -3283,9 +3272,7 @@ struct ref_sorting *ref_sorting_options(struct string_list *options)\n \tstruct string_list_item *item;\n \tstruct ref_sorting *sorting = NULL, **tail = &sorting;\n \n-\tif (!options->nr) {\n-\t\tsorting = ref_default_sorting();\n-\t} else {\n+\tif (options->nr) {\n \t\tfor_each_string_list_item(item, options)\n \t\t\tparse_ref_sorting(tail, item->string);\n \t}\ndiff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\nindex 3182abde27f..9918ba05dec 100755\n--- a/t/t3200-branch.sh\n+++ b/t/t3200-branch.sh\n@@ -1570,9 +1570,10 @@ test_expect_success 'tracking with unexpected .fetch refspec' '\n \n test_expect_success 'configured committerdate sort' '\n \tgit init -b main sort &&\n+\ttest_config -C sort branch.sort \"committerdate\" &&\n+\n \t(\n \t\tcd sort &&\n-\t\tgit config branch.sort committerdate &&\n \t\ttest_commit initial &&\n \t\tgit checkout -b a &&\n \t\ttest_commit a &&\n@@ -1592,9 +1593,10 @@ test_expect_success 'configured committerdate sort' '\n '\n \n test_expect_success 'option override configured sort' '\n+\ttest_config -C sort branch.sort \"committerdate\" &&\n+\n \t(\n \t\tcd sort &&\n-\t\tgit config branch.sort committerdate &&\n \t\tgit branch --sort=refname >actual &&\n \t\tcat >expect <<-\\EOF &&\n \t\t  a\n@@ -1606,10 +1608,70 @@ test_expect_success 'option override configured sort' '\n \t)\n '\n \n+test_expect_success '--no-sort cancels config sort keys' '\n+\ttest_config -C sort branch.sort \"-refname\" &&\n+\n+\t(\n+\t\tcd sort &&\n+\n+\t\t# objecttype is identical for all of them, so sort falls back on\n+\t\t# default (ascending refname)\n+\t\tgit branch \\\n+\t\t\t--no-sort \\\n+\t\t\t--sort=\"objecttype\" >actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\t  a\n+\t\t* b\n+\t\t  c\n+\t\t  main\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+\n+'\n+\n+test_expect_success '--no-sort cancels command line sort keys' '\n+\t(\n+\t\tcd sort &&\n+\n+\t\t# objecttype is identical for all of them, so sort falls back on\n+\t\t# default (ascending refname)\n+\t\tgit branch \\\n+\t\t\t--sort=\"-refname\" \\\n+\t\t\t--no-sort \\\n+\t\t\t--sort=\"objecttype\" >actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\t  a\n+\t\t* b\n+\t\t  c\n+\t\t  main\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success '--no-sort without subsequent --sort prints expected branches' '\n+\t(\n+\t\tcd sort &&\n+\n+\t\t# Sort the results with `sort` for a consistent comparison\n+\t\t# against expected\n+\t\tgit branch --no-sort | sort >actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\t  a\n+\t\t  c\n+\t\t  main\n+\t\t* b\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'invalid sort parameter in configuration' '\n+\ttest_config -C sort branch.sort \"v:notvalid\" &&\n+\n \t(\n \t\tcd sort &&\n-\t\tgit config branch.sort \"v:notvalid\" &&\n \n \t\t# this works in the \"listing\" mode, so bad sort key\n \t\t# is a dying offence.\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 00a060df0b5..0613e5e3623 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1335,6 +1335,27 @@ test_expect_success '--no-sort cancels the previous sort keys' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success '--no-sort without subsequent --sort prints expected refs' '\n+\tcat >expected <<-\\EOF &&\n+\trefs/tags/multi-ref1-100000-user1\n+\trefs/tags/multi-ref1-100000-user2\n+\trefs/tags/multi-ref1-200000-user1\n+\trefs/tags/multi-ref1-200000-user2\n+\trefs/tags/multi-ref2-100000-user1\n+\trefs/tags/multi-ref2-100000-user2\n+\trefs/tags/multi-ref2-200000-user1\n+\trefs/tags/multi-ref2-200000-user2\n+\tEOF\n+\n+\t# Sort the results with `sort` for a consistent comparison against\n+\t# expected\n+\tgit for-each-ref \\\n+\t\t--format=\"%(refname)\" \\\n+\t\t--no-sort \\\n+\t\t\"refs/tags/multi-*\" | sort >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'do not dereference NULL upon %(HEAD) on unborn branch' '\n \ttest_when_finished \"git checkout main\" &&\n \tgit for-each-ref --format=\"%(HEAD) %(refname:short)\" refs/heads/ >actual &&\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex e689db42929..b41a47eb943 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1862,6 +1862,51 @@ test_expect_success 'option override configured sort' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success '--no-sort cancels config sort keys' '\n+\ttest_config tag.sort \"-refname\" &&\n+\n+\t# objecttype is identical for all of them, so sort falls back on\n+\t# default (ascending refname)\n+\tgit tag -l \\\n+\t\t--no-sort \\\n+\t\t--sort=\"objecttype\" \\\n+\t\t\"foo*\" >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tfoo1.10\n+\tfoo1.3\n+\tfoo1.6\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--no-sort cancels command line sort keys' '\n+\t# objecttype is identical for all of them, so sort falls back on\n+\t# default (ascending refname)\n+\tgit tag -l \\\n+\t\t--sort=\"-refname\" \\\n+\t\t--no-sort \\\n+\t\t--sort=\"objecttype\" \\\n+\t\t\"foo*\" >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tfoo1.10\n+\tfoo1.3\n+\tfoo1.6\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--no-sort without subsequent --sort prints expected tags' '\n+\t# Sort the results with `sort` for a consistent comparison against\n+\t# expected\n+\tgit tag -l --no-sort \"foo*\" | sort >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tfoo1.10\n+\tfoo1.3\n+\tfoo1.6\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'invalid sort parameter on command line' '\n \ttest_must_fail git tag -l --sort=notvalid \"foo*\" >actual\n '\n-- \ngitgitgadget\n\n"},{"id":"484488","messageId":"2e2f9738205f732c2e640fd9a7183d16d0bf7f00.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 3/9] ref-filter.h: add max_count and omit_empty to ref_format","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:55Z","receivedAt":"2023-11-07T01:26:11Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nAdd an internal 'array_opts' struct to 'struct ref_format' containing\nformatting options that pertain to the formatting of an entire ref array:\n'max_count' and 'omit_empty'. These values are specified by the '--count'\nand '--omit-empty' options, respectively, to 'for-each-ref'/'tag'/'branch'.\nStoring these values in the 'ref_format' will simplify the consolidation of\nref array formatting logic across builtins in later patches.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n builtin/branch.c       |  5 ++---\n builtin/for-each-ref.c | 21 +++++++++++----------\n builtin/tag.c          |  5 ++---\n ref-filter.h           |  5 +++++\n 4 files changed, 20 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex d67738bbcaa..5a1ec1cd04f 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -45,7 +45,6 @@ static const char *head;\n static struct object_id head_oid;\n static int recurse_submodules = 0;\n static int submodule_propagate_branches = 0;\n-static int omit_empty = 0;\n \n static int branch_use_color = -1;\n static char branch_colors[][COLOR_MAXLEN] = {\n@@ -480,7 +479,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \t\t\tstring_list_append(output, out.buf);\n \t\t} else {\n \t\t\tfwrite(out.buf, 1, out.len, stdout);\n-\t\t\tif (out.len || !omit_empty)\n+\t\t\tif (out.len || !format->array_opts.omit_empty)\n \t\t\t\tputchar('\\n');\n \t\t}\n \t}\n@@ -737,7 +736,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\tOPT_BIT('D', NULL, &delete, N_(\"delete branch (even if not merged)\"), 2),\n \t\tOPT_BIT('m', \"move\", &rename, N_(\"move/rename a branch and its reflog\"), 1),\n \t\tOPT_BIT('M', NULL, &rename, N_(\"move/rename a branch, even if target exists\"), 2),\n-\t\tOPT_BOOL(0, \"omit-empty\",  &omit_empty,\n+\t\tOPT_BOOL(0, \"omit-empty\",  &format.array_opts.omit_empty,\n \t\t\tN_(\"do not output a newline after empty formatted refs\")),\n \t\tOPT_BIT('c', \"copy\", &copy, N_(\"copy a branch and its reflog\"), 1),\n \t\tOPT_BIT('C', NULL, &copy, N_(\"copy a branch, even if target exists\"), 2),\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 93b370f550b..881c3ee055f 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -19,10 +19,10 @@ static char const * const for_each_ref_usage[] = {\n \n int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n-\tint i;\n+\tint i, total;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n-\tint maxcount = 0, icase = 0, omit_empty = 0;\n+\tint icase = 0;\n \tstruct ref_array array;\n \tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -40,11 +40,11 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"quote placeholders suitably for python\"), QUOTE_PYTHON),\n \t\tOPT_BIT(0 , \"tcl\",  &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for Tcl\"), QUOTE_TCL),\n-\t\tOPT_BOOL(0, \"omit-empty\",  &omit_empty,\n+\t\tOPT_BOOL(0, \"omit-empty\",  &format.array_opts.omit_empty,\n \t\t\tN_(\"do not output a newline after empty formatted refs\")),\n \n \t\tOPT_GROUP(\"\"),\n-\t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n+\t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n@@ -71,8 +71,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstring_list_append(&sorting_options, \"refname\");\n \n \tparse_options(argc, argv, prefix, opts, for_each_ref_usage, 0);\n-\tif (maxcount < 0) {\n-\t\terror(\"invalid --count argument: `%d'\", maxcount);\n+\tif (format.array_opts.max_count < 0) {\n+\t\terror(\"invalid --count argument: `%d'\", format.array_opts.max_count);\n \t\tusage_with_options(for_each_ref_usage, opts);\n \t}\n \tif (HAS_MULTI_BITS(format.quote_style)) {\n@@ -109,15 +109,16 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tref_array_sort(sorting, &array);\n \n-\tif (!maxcount || array.nr < maxcount)\n-\t\tmaxcount = array.nr;\n-\tfor (i = 0; i < maxcount; i++) {\n+\ttotal = format.array_opts.max_count;\n+\tif (!total || array.nr < total)\n+\t\ttotal = array.nr;\n+\tfor (i = 0; i < total; i++) {\n \t\tstrbuf_reset(&err);\n \t\tstrbuf_reset(&output);\n \t\tif (format_ref_array_item(array.items[i], &format, &output, &err))\n \t\t\tdie(\"%s\", err.buf);\n \t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !omit_empty)\n+\t\tif (output.len || !format.array_opts.omit_empty)\n \t\t\tputchar('\\n');\n \t}\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 64f3196cd4c..2d599245d48 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -44,7 +44,6 @@ static const char * const git_tag_usage[] = {\n static unsigned int colopts;\n static int force_sign_annotate;\n static int config_sign_tag = -1; /* unspecified */\n-static int omit_empty = 0;\n \n static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\t     struct ref_format *format)\n@@ -83,7 +82,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\tif (format_ref_array_item(array.items[i], format, &output, &err))\n \t\t\tdie(\"%s\", err.buf);\n \t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !omit_empty)\n+\t\tif (output.len || !format->array_opts.omit_empty)\n \t\t\tputchar('\\n');\n \t}\n \n@@ -481,7 +480,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \t\tOPT_WITHOUT(&filter.no_commit, N_(\"print only tags that don't contain the commit\")),\n \t\tOPT_MERGED(&filter, N_(\"print only tags that are merged\")),\n \t\tOPT_NO_MERGED(&filter, N_(\"print only tags that are not merged\")),\n-\t\tOPT_BOOL(0, \"omit-empty\",  &omit_empty,\n+\t\tOPT_BOOL(0, \"omit-empty\",  &format.array_opts.omit_empty,\n \t\t\tN_(\"do not output a newline after empty formatted refs\")),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\t{\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 1524bc463a5..d87d61238b7 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,11 @@ struct ref_format {\n \n \t/* List of bases for ahead-behind counts. */\n \tstruct string_list bases;\n+\n+\tstruct {\n+\t\tint max_count;\n+\t\tint omit_empty;\n+\t} array_opts;\n };\n \n #define REF_FILTER_INIT { \\\n-- \ngitgitgadget\n\n"},{"id":"484489","messageId":"6c66445ee31dd4117e1384d8da7be81f401317b3.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 4/9] ref-filter.h: move contains caches into filter","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:56Z","receivedAt":"2023-11-07T01:26:12Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nMove the 'contains_cache' and 'no_contains_cache' used in filter_refs into\nan 'internal' struct of the 'struct ref_filter'. In later patches, the\n'struct ref_filter *' will be a common data structure across multiple\nfiltering functions. These caches are part of the common functionality the\nfilter struct will support, so they are updated to be internally accessible\nwherever the filter is used.\n\nThe design used here is mirrors what was introduced in 576de3d956\n(unpack_trees: start splitting internal fields from public API, 2023-02-27)\nfor 'unpack_trees_options'.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n ref-filter.c | 14 ++++++--------\n ref-filter.h |  6 ++++++\n 2 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7250089b7c6..5129b6986c9 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2764,8 +2764,6 @@ static int filter_ref_kind(struct ref_filter *filter, const char *refname)\n struct ref_filter_cbdata {\n \tstruct ref_array *array;\n \tstruct ref_filter *filter;\n-\tstruct contains_cache contains_cache;\n-\tstruct contains_cache no_contains_cache;\n };\n \n /*\n@@ -2816,11 +2814,11 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \t\t\treturn 0;\n \t\t/* We perform the filtering for the '--contains' option... */\n \t\tif (filter->with_commit &&\n-\t\t    !commit_contains(filter, commit, filter->with_commit, &ref_cbdata->contains_cache))\n+\t\t    !commit_contains(filter, commit, filter->with_commit, &filter->internal.contains_cache))\n \t\t\treturn 0;\n \t\t/* ...or for the `--no-contains' option */\n \t\tif (filter->no_commit &&\n-\t\t    commit_contains(filter, commit, filter->no_commit, &ref_cbdata->no_contains_cache))\n+\t\t    commit_contains(filter, commit, filter->no_commit, &filter->internal.no_contains_cache))\n \t\t\treturn 0;\n \t}\n \n@@ -2989,8 +2987,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \tsave_commit_buffer_orig = save_commit_buffer;\n \tsave_commit_buffer = 0;\n \n-\tinit_contains_cache(&ref_cbdata.contains_cache);\n-\tinit_contains_cache(&ref_cbdata.no_contains_cache);\n+\tinit_contains_cache(&filter->internal.contains_cache);\n+\tinit_contains_cache(&filter->internal.no_contains_cache);\n \n \t/*  Simple per-ref filtering */\n \tif (!filter->kind)\n@@ -3014,8 +3012,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \t\t\thead_ref(ref_filter_handler, &ref_cbdata);\n \t}\n \n-\tclear_contains_cache(&ref_cbdata.contains_cache);\n-\tclear_contains_cache(&ref_cbdata.no_contains_cache);\n+\tclear_contains_cache(&filter->internal.contains_cache);\n+\tclear_contains_cache(&filter->internal.no_contains_cache);\n \n \t/*  Filters that need revision walking */\n \treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex d87d61238b7..0db3ff52889 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -7,6 +7,7 @@\n #include \"commit.h\"\n #include \"string-list.h\"\n #include \"strvec.h\"\n+#include \"commit-reach.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -75,6 +76,11 @@ struct ref_filter {\n \t\tlines;\n \tint abbrev,\n \t\tverbose;\n+\n+\tstruct {\n+\t\tstruct contains_cache contains_cache;\n+\t\tstruct contains_cache no_contains_cache;\n+\t} internal;\n };\n \n struct ref_format {\n-- \ngitgitgadget\n\n"},{"id":"484490","messageId":"f5be57eea7d5b73d4614d2e1816f5cc5b1118a13.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 5/9] ref-filter.h: add functions for filter/format & format-only","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:57Z","receivedAt":"2023-11-07T01:26:13Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nAdd two new public methods to 'ref-filter.h':\n\n* 'print_formatted_ref_array()' which, given a format specification & array\n  of ref items, formats and prints the items to stdout.\n* 'filter_and_format_refs()' which combines 'filter_refs()',\n  'ref_array_sort()', and 'print_formatted_ref_array()' into a single\n  function.\n\nThis consolidates much of the code used to filter and format refs in\n'builtin/for-each-ref.c', 'builtin/tag.c', and 'builtin/branch.c', reducing\nduplication and simplifying the future changes needed to optimize the filter\n& format process.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n builtin/branch.c       | 33 +++++++++++++++++----------------\n builtin/for-each-ref.c | 27 +--------------------------\n builtin/tag.c          | 23 +----------------------\n ref-filter.c           | 35 +++++++++++++++++++++++++++++++++++\n ref-filter.h           | 14 ++++++++++++++\n 5 files changed, 68 insertions(+), 64 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 5a1ec1cd04f..2ed59f16f1c 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -437,8 +437,6 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n {\n \tint i;\n \tstruct ref_array array;\n-\tstruct strbuf out = STRBUF_INIT;\n-\tstruct strbuf err = STRBUF_INIT;\n \tint maxwidth = 0;\n \tconst char *remote_prefix = \"\";\n \tchar *to_free = NULL;\n@@ -468,24 +466,27 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n-\tfor (i = 0; i < array.nr; i++) {\n-\t\tstrbuf_reset(&err);\n-\t\tstrbuf_reset(&out);\n-\t\tif (format_ref_array_item(array.items[i], format, &out, &err))\n-\t\t\tdie(\"%s\", err.buf);\n-\t\tif (column_active(colopts)) {\n-\t\t\tassert(!filter->verbose && \"--column and --verbose are incompatible\");\n-\t\t\t /* format to a string_list to let print_columns() do its job */\n+\tif (column_active(colopts)) {\n+\t\tstruct strbuf out = STRBUF_INIT, err = STRBUF_INIT;\n+\n+\t\tassert(!filter->verbose && \"--column and --verbose are incompatible\");\n+\n+\t\tfor (i = 0; i < array.nr; i++) {\n+\t\t\tstrbuf_reset(&err);\n+\t\t\tstrbuf_reset(&out);\n+\t\t\tif (format_ref_array_item(array.items[i], format, &out, &err))\n+\t\t\t\tdie(\"%s\", err.buf);\n+\n+\t\t\t/* format to a string_list to let print_columns() do its job */\n \t\t\tstring_list_append(output, out.buf);\n-\t\t} else {\n-\t\t\tfwrite(out.buf, 1, out.len, stdout);\n-\t\t\tif (out.len || !format->array_opts.omit_empty)\n-\t\t\t\tputchar('\\n');\n \t\t}\n+\n+\t\tstrbuf_release(&err);\n+\t\tstrbuf_release(&out);\n+\t} else {\n+\t\tprint_formatted_ref_array(&array, format);\n \t}\n \n-\tstrbuf_release(&err);\n-\tstrbuf_release(&out);\n \tref_array_clear(&array);\n \tfree(to_free);\n }\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 881c3ee055f..1c19cd5bd34 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -19,15 +19,11 @@ static char const * const for_each_ref_usage[] = {\n \n int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n-\tint i, total;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tint icase = 0;\n-\tstruct ref_array array;\n \tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n-\tstruct strbuf output = STRBUF_INIT;\n-\tstruct strbuf err = STRBUF_INIT;\n \tint from_stdin = 0;\n \tstruct strvec vec = STRVEC_INIT;\n \n@@ -61,8 +57,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n-\tmemset(&array, 0, sizeof(array));\n-\n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \n \tgit_config(git_default_config, NULL);\n@@ -104,27 +98,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t}\n \n \tfilter.match_as_path = 1;\n-\tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n-\tfilter_ahead_behind(the_repository, &format, &array);\n-\n-\tref_array_sort(sorting, &array);\n-\n-\ttotal = format.array_opts.max_count;\n-\tif (!total || array.nr < total)\n-\t\ttotal = array.nr;\n-\tfor (i = 0; i < total; i++) {\n-\t\tstrbuf_reset(&err);\n-\t\tstrbuf_reset(&output);\n-\t\tif (format_ref_array_item(array.items[i], &format, &output, &err))\n-\t\t\tdie(\"%s\", err.buf);\n-\t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !format.array_opts.omit_empty)\n-\t\t\tputchar('\\n');\n-\t}\n+\tfilter_and_format_refs(&filter, FILTER_REFS_ALL, sorting, &format);\n \n-\tstrbuf_release(&err);\n-\tstrbuf_release(&output);\n-\tref_array_clear(&array);\n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\n \tstrvec_clear(&vec);\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 2d599245d48..2528d499dd8 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -48,13 +48,7 @@ static int config_sign_tag = -1; /* unspecified */\n static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\t     struct ref_format *format)\n {\n-\tstruct ref_array array;\n-\tstruct strbuf output = STRBUF_INIT;\n-\tstruct strbuf err = STRBUF_INIT;\n \tchar *to_free = NULL;\n-\tint i;\n-\n-\tmemset(&array, 0, sizeof(array));\n \n \tif (filter->lines == -1)\n \t\tfilter->lines = 0;\n@@ -72,23 +66,8 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n-\tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n-\tfilter_ahead_behind(the_repository, format, &array);\n-\tref_array_sort(sorting, &array);\n-\n-\tfor (i = 0; i < array.nr; i++) {\n-\t\tstrbuf_reset(&output);\n-\t\tstrbuf_reset(&err);\n-\t\tif (format_ref_array_item(array.items[i], format, &output, &err))\n-\t\t\tdie(\"%s\", err.buf);\n-\t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !format->array_opts.omit_empty)\n-\t\t\tputchar('\\n');\n-\t}\n+\tfilter_and_format_refs(filter, FILTER_REFS_TAGS, sorting, format);\n \n-\tstrbuf_release(&err);\n-\tstrbuf_release(&output);\n-\tref_array_clear(&array);\n \tfree(to_free);\n \n \treturn 0;\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5129b6986c9..8992fbf45b1 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -3023,6 +3023,18 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \treturn ret;\n }\n \n+void filter_and_format_refs(struct ref_filter *filter, unsigned int type,\n+\t\t\t    struct ref_sorting *sorting,\n+\t\t\t    struct ref_format *format)\n+{\n+\tstruct ref_array array = { 0 };\n+\tfilter_refs(&array, filter, type);\n+\tfilter_ahead_behind(the_repository, format, &array);\n+\tref_array_sort(sorting, &array);\n+\tprint_formatted_ref_array(&array, format);\n+\tref_array_clear(&array);\n+}\n+\n static int compare_detached_head(struct ref_array_item *a, struct ref_array_item *b)\n {\n \tif (!(a->kind ^ b->kind))\n@@ -3212,6 +3224,29 @@ int format_ref_array_item(struct ref_array_item *info,\n \treturn 0;\n }\n \n+void print_formatted_ref_array(struct ref_array *array, struct ref_format *format)\n+{\n+\tint total;\n+\tstruct strbuf output = STRBUF_INIT, err = STRBUF_INIT;\n+\n+\ttotal = format->array_opts.max_count;\n+\tif (!total || array->nr < total)\n+\t\ttotal = array->nr;\n+\tfor (int i = 0; i < total; i++) {\n+\t\tstrbuf_reset(&err);\n+\t\tstrbuf_reset(&output);\n+\t\tif (format_ref_array_item(array->items[i], format, &output, &err))\n+\t\t\tdie(\"%s\", err.buf);\n+\t\tif (output.len || !format->array_opts.omit_empty) {\n+\t\t\tfwrite(output.buf, 1, output.len, stdout);\n+\t\t\tputchar('\\n');\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&err);\n+\tstrbuf_release(&output);\n+}\n+\n void pretty_print_ref(const char *name, const struct object_id *oid,\n \t\t      struct ref_format *format)\n {\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 0db3ff52889..0ce5af58ab3 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -137,6 +137,14 @@ struct ref_format {\n  * filtered refs in the ref_array structure.\n  */\n int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type);\n+/*\n+ * Filter refs using the given ref_filter and type, sort the contents\n+ * according to the given ref_sorting, format the filtered refs with the\n+ * given ref_format, and print them to stdout.\n+ */\n+void filter_and_format_refs(struct ref_filter *filter, unsigned int type,\n+\t\t\t    struct ref_sorting *sorting,\n+\t\t\t    struct ref_format *format);\n /*  Clear all memory allocated to ref_array */\n void ref_array_clear(struct ref_array *array);\n /*  Used to verify if the given format is correct and to parse out the used atoms */\n@@ -161,6 +169,12 @@ char *get_head_description(void);\n /*  Set up translated strings in the output. */\n void setup_ref_filter_porcelain_msg(void);\n \n+/*\n+ * Print up to maxcount ref_array elements to stdout using the given\n+ * ref_format.\n+ */\n+void print_formatted_ref_array(struct ref_array *array, struct ref_format *format);\n+\n /*\n  * Print a single ref, outside of any ref-filter. Note that the\n  * name must be a fully qualified refname.\n-- \ngitgitgadget\n\n"},{"id":"484491","messageId":"8c77452e5dd8d5cafd95c68480bf5675d51b4736.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 6/9] ref-filter.c: refactor to create common helper functions","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:58Z","receivedAt":"2023-11-07T01:26:13Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nFactor out parts of 'ref_array_push()', 'ref_filter_handler()', and\n'filter_refs()' into new helper functions ('ref_array_append()',\n'apply_ref_filter()', and 'do_filter_refs()' respectively), as well as\nrename 'ref_filter_handler()' to 'filter_one()'. In this and later\npatches, these helpers will be used by new ref-filter API functions. This\npatch does not result in any user-facing behavior changes or changes to\ncallers outside of 'ref-filter.c'.\n\nThe changes are as follows:\n\n* The logic to grow a 'struct ref_array' and append a given 'struct\n  ref_array_item *' to it is extracted from 'ref_array_push()' into\n  'ref_array_append()'.\n* 'ref_filter_handler()' is renamed to 'filter_one()' to more clearly\n  distinguish it from other ref filtering callbacks that will be added in\n  later patches. The \"*_one()\" naming convention is common throughout the\n  codebase for iteration callbacks.\n* The code to filter a given ref by refname & object ID then create a new\n  'struct ref_array_item' is moved out of 'filter_one()' and into\n  'apply_ref_filter()'. 'apply_ref_filter()' returns either NULL (if the ref\n  does not match the given filter) or a 'struct ref_array_item *' created\n  with 'new_ref_array_item()'; 'filter_one()' appends that item to\n  its ref array with 'ref_array_append()'.\n* The filter pre-processing, contains cache creation, and ref iteration of\n  'filter_refs()' is extracted into 'do_filter_refs()'. 'do_filter_refs()'\n  takes its ref iterator function & callback data as an input from the\n  caller, setting it up to be used with additional filtering callbacks in\n  later patches.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n ref-filter.c | 115 ++++++++++++++++++++++++++++++---------------------\n 1 file changed, 69 insertions(+), 46 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 8992fbf45b1..ff00ab4b8d8 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2716,15 +2716,18 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \treturn ref;\n }\n \n+static void ref_array_append(struct ref_array *array, struct ref_array_item *ref)\n+{\n+\tALLOC_GROW(array->items, array->nr + 1, array->alloc);\n+\tarray->items[array->nr++] = ref;\n+}\n+\n struct ref_array_item *ref_array_push(struct ref_array *array,\n \t\t\t\t      const char *refname,\n \t\t\t\t      const struct object_id *oid)\n {\n \tstruct ref_array_item *ref = new_ref_array_item(refname, oid);\n-\n-\tALLOC_GROW(array->items, array->nr + 1, array->alloc);\n-\tarray->items[array->nr++] = ref;\n-\n+\tref_array_append(array, ref);\n \treturn ref;\n }\n \n@@ -2761,46 +2764,36 @@ static int filter_ref_kind(struct ref_filter *filter, const char *refname)\n \treturn ref_kind_from_refname(refname);\n }\n \n-struct ref_filter_cbdata {\n-\tstruct ref_array *array;\n-\tstruct ref_filter *filter;\n-};\n-\n-/*\n- * A call-back given to for_each_ref().  Filter refs and keep them for\n- * later object processing.\n- */\n-static int ref_filter_handler(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n+static struct ref_array_item *apply_ref_filter(const char *refname, const struct object_id *oid,\n+\t\t\t    int flag, struct ref_filter *filter)\n {\n-\tstruct ref_filter_cbdata *ref_cbdata = cb_data;\n-\tstruct ref_filter *filter = ref_cbdata->filter;\n \tstruct ref_array_item *ref;\n \tstruct commit *commit = NULL;\n \tunsigned int kind;\n \n \tif (flag & REF_BAD_NAME) {\n \t\twarning(_(\"ignoring ref with broken name %s\"), refname);\n-\t\treturn 0;\n+\t\treturn NULL;\n \t}\n \n \tif (flag & REF_ISBROKEN) {\n \t\twarning(_(\"ignoring broken ref %s\"), refname);\n-\t\treturn 0;\n+\t\treturn NULL;\n \t}\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n \tif (!(kind & filter->kind))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tif (!filter_pattern_match(filter, refname))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tif (filter_exclude_match(filter, refname))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \t/*\n \t * A merge filter is applied on refs pointing to commits. Hence\n@@ -2811,15 +2804,15 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \t    filter->with_commit || filter->no_commit || filter->verbose) {\n \t\tcommit = lookup_commit_reference_gently(the_repository, oid, 1);\n \t\tif (!commit)\n-\t\t\treturn 0;\n+\t\t\treturn NULL;\n \t\t/* We perform the filtering for the '--contains' option... */\n \t\tif (filter->with_commit &&\n \t\t    !commit_contains(filter, commit, filter->with_commit, &filter->internal.contains_cache))\n-\t\t\treturn 0;\n+\t\t\treturn NULL;\n \t\t/* ...or for the `--no-contains' option */\n \t\tif (filter->no_commit &&\n \t\t    commit_contains(filter, commit, filter->no_commit, &filter->internal.no_contains_cache))\n-\t\t\treturn 0;\n+\t\t\treturn NULL;\n \t}\n \n \t/*\n@@ -2827,11 +2820,32 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \t * to do its job and the resulting list may yet to be pruned\n \t * by maxcount logic.\n \t */\n-\tref = ref_array_push(ref_cbdata->array, refname, oid);\n+\tref = new_ref_array_item(refname, oid);\n \tref->commit = commit;\n \tref->flag = flag;\n \tref->kind = kind;\n \n+\treturn ref;\n+}\n+\n+struct ref_filter_cbdata {\n+\tstruct ref_array *array;\n+\tstruct ref_filter *filter;\n+};\n+\n+/*\n+ * A call-back given to for_each_ref().  Filter refs and keep them for\n+ * later object processing.\n+ */\n+static int filter_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n+{\n+\tstruct ref_filter_cbdata *ref_cbdata = cb_data;\n+\tstruct ref_array_item *ref;\n+\n+\tref = apply_ref_filter(refname, oid, flag, ref_cbdata->filter);\n+\tif (ref)\n+\t\tref_array_append(ref_cbdata->array, ref);\n+\n \treturn 0;\n }\n \n@@ -2967,26 +2981,12 @@ void filter_ahead_behind(struct repository *r,\n \tfree(commits);\n }\n \n-/*\n- * API for filtering a set of refs. Based on the type of refs the user\n- * has requested, we iterate through those refs and apply filters\n- * as per the given ref_filter structure and finally store the\n- * filtered refs in the ref_array structure.\n- */\n-int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type)\n+static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)\n {\n-\tstruct ref_filter_cbdata ref_cbdata;\n-\tint save_commit_buffer_orig;\n \tint ret = 0;\n \n-\tref_cbdata.array = array;\n-\tref_cbdata.filter = filter;\n-\n \tfilter->kind = type & FILTER_REFS_KIND_MASK;\n \n-\tsave_commit_buffer_orig = save_commit_buffer;\n-\tsave_commit_buffer = 0;\n-\n \tinit_contains_cache(&filter->internal.contains_cache);\n \tinit_contains_cache(&filter->internal.no_contains_cache);\n \n@@ -3001,20 +3001,43 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \t\t * of filter_ref_kind().\n \t\t */\n \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = for_each_fullref_in(\"refs/heads/\", ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/heads/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = for_each_fullref_in(\"refs/remotes/\", ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = for_each_fullref_in(\"refs/tags/\", ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n \t\telse if (filter->kind & FILTER_REFS_ALL)\n-\t\t\tret = for_each_fullref_in_pattern(filter, ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n-\t\t\thead_ref(ref_filter_handler, &ref_cbdata);\n+\t\t\thead_ref(fn, cb_data);\n \t}\n \n \tclear_contains_cache(&filter->internal.contains_cache);\n \tclear_contains_cache(&filter->internal.no_contains_cache);\n \n+\treturn ret;\n+}\n+\n+/*\n+ * API for filtering a set of refs. Based on the type of refs the user\n+ * has requested, we iterate through those refs and apply filters\n+ * as per the given ref_filter structure and finally store the\n+ * filtered refs in the ref_array structure.\n+ */\n+int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type)\n+{\n+\tstruct ref_filter_cbdata ref_cbdata;\n+\tint save_commit_buffer_orig;\n+\tint ret = 0;\n+\n+\tref_cbdata.array = array;\n+\tref_cbdata.filter = filter;\n+\n+\tsave_commit_buffer_orig = save_commit_buffer;\n+\tsave_commit_buffer = 0;\n+\n+\tret = do_filter_refs(filter, type, filter_one, &ref_cbdata);\n+\n \t/*  Filters that need revision walking */\n \treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n \treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n-- \ngitgitgadget\n\n"},{"id":"484492","messageId":"84db440896c162bcbeeaaf00d528839056aefaa5.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 7/9] ref-filter.c: filter & format refs in the same callback","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:25:59Z","receivedAt":"2023-11-07T01:26:14Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nUpdate 'filter_and_format_refs()' to try to perform ref filtering &\nformatting in a single ref iteration, without an intermediate 'struct\nref_array'. This can only be done if no operations need to be performed on a\npre-filtered array; specifically, if the refs are\n\n- filtered on reachability,\n- sorted, or\n- formatted with ahead-behind information\n\nthey cannot be filtered & formatted in the same iteration. In that case,\nfall back on the current filter-then-sort-then-format flow.\n\nThis optimization substantially improves memory usage due to no longer\nstoring a ref array in memory. In some cases, it also dramatically reduces\nruntime (e.g. 'git for-each-ref --no-sort --count=1', which no longer loads\nall refs into a 'struct ref_array' to printing only the first ref).\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n ref-filter.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 74 insertions(+), 6 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex ff00ab4b8d8..384cf1595ff 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2863,6 +2863,44 @@ static void free_array_item(struct ref_array_item *item)\n \tfree(item);\n }\n \n+struct ref_filter_and_format_cbdata {\n+\tstruct ref_filter *filter;\n+\tstruct ref_format *format;\n+\n+\tstruct ref_filter_and_format_internal {\n+\t\tint count;\n+\t} internal;\n+};\n+\n+static int filter_and_format_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n+{\n+\tstruct ref_filter_and_format_cbdata *ref_cbdata = cb_data;\n+\tstruct ref_array_item *ref;\n+\tstruct strbuf output = STRBUF_INIT, err = STRBUF_INIT;\n+\n+\tref = apply_ref_filter(refname, oid, flag, ref_cbdata->filter);\n+\tif (!ref)\n+\t\treturn 0;\n+\n+\tif (format_ref_array_item(ref, ref_cbdata->format, &output, &err))\n+\t\tdie(\"%s\", err.buf);\n+\n+\tif (output.len || !ref_cbdata->format->array_opts.omit_empty) {\n+\t\tfwrite(output.buf, 1, output.len, stdout);\n+\t\tputchar('\\n');\n+\t}\n+\n+\tstrbuf_release(&output);\n+\tstrbuf_release(&err);\n+\tfree_array_item(ref);\n+\n+\tif (ref_cbdata->format->array_opts.max_count &&\n+\t    ++ref_cbdata->internal.count >= ref_cbdata->format->array_opts.max_count)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n /* Free all memory allocated for ref_array */\n void ref_array_clear(struct ref_array *array)\n {\n@@ -3046,16 +3084,46 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \treturn ret;\n }\n \n+static inline int can_do_iterative_format(struct ref_filter *filter,\n+\t\t\t\t\t  struct ref_sorting *sorting,\n+\t\t\t\t\t  struct ref_format *format)\n+{\n+\t/*\n+\t * Refs can be filtered and formatted in the same iteration as long\n+\t * as we aren't filtering on reachability, sorting the results, or\n+\t * including ahead-behind information in the formatted output.\n+\t */\n+\treturn !(filter->reachable_from ||\n+\t\t filter->unreachable_from ||\n+\t\t sorting ||\n+\t\t format->bases.nr);\n+}\n+\n void filter_and_format_refs(struct ref_filter *filter, unsigned int type,\n \t\t\t    struct ref_sorting *sorting,\n \t\t\t    struct ref_format *format)\n {\n-\tstruct ref_array array = { 0 };\n-\tfilter_refs(&array, filter, type);\n-\tfilter_ahead_behind(the_repository, format, &array);\n-\tref_array_sort(sorting, &array);\n-\tprint_formatted_ref_array(&array, format);\n-\tref_array_clear(&array);\n+\tif (can_do_iterative_format(filter, sorting, format)) {\n+\t\tint save_commit_buffer_orig;\n+\t\tstruct ref_filter_and_format_cbdata ref_cbdata = {\n+\t\t\t.filter = filter,\n+\t\t\t.format = format,\n+\t\t};\n+\n+\t\tsave_commit_buffer_orig = save_commit_buffer;\n+\t\tsave_commit_buffer = 0;\n+\n+\t\tdo_filter_refs(filter, type, filter_and_format_one, &ref_cbdata);\n+\n+\t\tsave_commit_buffer = save_commit_buffer_orig;\n+\t} else {\n+\t\tstruct ref_array array = { 0 };\n+\t\tfilter_refs(&array, filter, type);\n+\t\tfilter_ahead_behind(the_repository, format, &array);\n+\t\tref_array_sort(sorting, &array);\n+\t\tprint_formatted_ref_array(&array, format);\n+\t\tref_array_clear(&array);\n+\t}\n }\n \n static int compare_detached_head(struct ref_array_item *a, struct ref_array_item *b)\n-- \ngitgitgadget\n\n"},{"id":"484493","messageId":"352b5c42ac39d5d2646a1b6d47d6d707637db539.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:26:00Z","receivedAt":"2023-11-07T01:26:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nAdd a boolean flag '--full-deref' that, when enabled, fills '%(*fieldname)'\nformat fields using the fully peeled target of tag objects, rather than the\nimmediate target.\n\nIn other builtins ('rev-parse', 'show-ref'), \"dereferencing\" tags typically\nmeans peeling them down to their non-tag target. Unlike these commands,\n'for-each-ref' dereferences only one \"level\" of tags in '*' format fields\n(like \"%(*objectname)\"). For most annotated tags, one level of dereferencing\nis enough, since most tags point to commits or trees. However, nested tags\n(annotated tags whose target is another annotated tag) dereferenced once\nwill point to their target tag, different a full peel to e.g. a commit.\n\nCurrently, if a user wants to filter & format refs and include information\nabout the fully dereferenced tag, they can do so with something like\n'cat-file --batch-check':\n\n    git for-each-ref --format=\"%(objectname)^{} %(refname)\" <pattern> |\n        git cat-file --batch-check=\"%(objectname) %(rest)\"\n\nBut the combination of commands is inefficient. So, to improve the\nefficiency of this use case, add a '--full-deref' option that causes\n'for-each-ref' to fully dereference tags when formatting with '*' fields.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n Documentation/git-for-each-ref.txt |  9 ++++++++\n builtin/for-each-ref.c             |  2 ++\n ref-filter.c                       | 26 ++++++++++++++---------\n ref-filter.h                       |  1 +\n t/t6300-for-each-ref.sh            | 34 ++++++++++++++++++++++++++++++\n 5 files changed, 62 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 407f624fbaa..2714a87088e 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -11,6 +11,7 @@ SYNOPSIS\n 'git for-each-ref' [--count=<count>] [--shell|--perl|--python|--tcl]\n \t\t   [(--sort=<key>)...] [--format=<format>]\n \t\t   [ --stdin | <pattern>... ]\n+\t\t   [--full-deref]\n \t\t   [--points-at=<object>]\n \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n@@ -77,6 +78,14 @@ OPTIONS\n \tthe specified host language.  This is meant to produce\n \ta scriptlet that can directly be `eval`ed.\n \n+--full-deref::\n+\tPopulate dereferenced format fields (indicated with an asterisk (`*`)\n+\tprefix before the fieldname) with information about the fully-peeled\n+\ttarget object of a tag ref, rather than its immediate target object.\n+\tThis only affects the output for nested annotated tags, where the tag's\n+\timmediate target is another tag but its fully-peeled target is another\n+\tobject type (e.g. a commit).\n+\n --points-at=<object>::\n \tOnly list refs which points at the given object.\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 1c19cd5bd34..7a2127a3bc4 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -43,6 +43,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n+\t\tOPT_BOOL(0, \"full-deref\", &format.full_deref,\n+\t\t\t N_(\"fully dereference tags to populate '*' format fields\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 384cf1595ff..a66ac7921b1 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -237,7 +237,14 @@ static struct used_atom {\n \t\tchar *head;\n \t} u;\n } *used_atom;\n-static int used_atom_cnt, need_tagged, need_symref;\n+static int used_atom_cnt, need_symref;\n+\n+enum tag_dereference_mode {\n+\tNO_DEREF = 0,\n+\tDEREF_ONE,\n+\tDEREF_ALL\n+};\n+static enum tag_dereference_mode need_tagged;\n \n /*\n  * Expand string, append it to strbuf *sb, then return error code ret.\n@@ -1066,8 +1073,8 @@ static int parse_ref_filter_atom(struct ref_format *format,\n \tmemset(&used_atom[at].u, 0, sizeof(used_atom[at].u));\n \tif (valid_atom[i].parser && valid_atom[i].parser(format, &used_atom[at], arg, err))\n \t\treturn -1;\n-\tif (*atom == '*')\n-\t\tneed_tagged = 1;\n+\tif (*atom == '*' && !need_tagged)\n+\t\tneed_tagged = format->full_deref ? DEREF_ALL : DEREF_ONE;\n \tif (i == ATOM_SYMREF)\n \t\tneed_symref = 1;\n \treturn at;\n@@ -2511,14 +2518,13 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t * If it is a tag object, see if we use a value that derefs\n \t * the object, and if we do grab the object it refers to.\n \t */\n-\toi_deref.oid = *get_tagged_oid((struct tag *)obj);\n+\tif (need_tagged == DEREF_ALL) {\n+\t\tif (peel_iterated_oid(&obj->oid, &oi_deref.oid))\n+\t\t\tdie(\"bad tag\");\n+\t} else {\n+\t\toi_deref.oid = *get_tagged_oid((struct tag *)obj);\n+\t}\n \n-\t/*\n-\t * NEEDSWORK: This derefs tag only once, which\n-\t * is good to deal with chains of trust, but\n-\t * is not consistent with what deref_tag() does\n-\t * which peels the onion to the core.\n-\t */\n \treturn get_object(ref, 1, &obj, &oi_deref, err);\n }\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 0ce5af58ab3..0caa39ecee5 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,7 @@ struct ref_format {\n \tconst char *rest;\n \tint quote_style;\n \tint use_color;\n+\tint full_deref;\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 0613e5e3623..3c2af785cdb 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1839,6 +1839,40 @@ test_expect_success 'git for-each-ref with non-existing refs' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'git for-each-ref with nested tags' '\n+\tgit tag -am \"Normal tag\" nested/base HEAD &&\n+\tgit tag -am \"Nested tag\" nested/nest1 refs/tags/nested/base &&\n+\tgit tag -am \"Double nested tag\" nested/nest2 refs/tags/nested/nest1 &&\n+\n+\thead_oid=\"$(git rev-parse HEAD)\" &&\n+\tbase_tag_oid=\"$(git rev-parse refs/tags/nested/base)\" &&\n+\tnest1_tag_oid=\"$(git rev-parse refs/tags/nested/nest1)\" &&\n+\tnest2_tag_oid=\"$(git rev-parse refs/tags/nested/nest2)\" &&\n+\n+\t# Without full dereference\n+\tcat >expect <<-EOF &&\n+\trefs/tags/nested/base $base_tag_oid tag $head_oid commit\n+\trefs/tags/nested/nest1 $nest1_tag_oid tag $base_tag_oid tag\n+\trefs/tags/nested/nest2 $nest2_tag_oid tag $nest1_tag_oid tag\n+\tEOF\n+\n+\tgit for-each-ref --format=\"%(refname) %(objectname) %(objecttype) %(*objectname) %(*objecttype)\" \\\n+\t\trefs/tags/nested/ >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# With full dereference\n+\tcat >expect <<-EOF &&\n+\trefs/tags/nested/base $base_tag_oid tag $head_oid commit\n+\trefs/tags/nested/nest1 $nest1_tag_oid tag $head_oid commit\n+\trefs/tags/nested/nest2 $nest2_tag_oid tag $head_oid commit\n+\tEOF\n+\n+\tgit for-each-ref --full-deref \\\n+\t\t--format=\"%(refname) %(objectname) %(objecttype) %(*objectname) %(*objecttype)\" \\\n+\t\trefs/tags/nested/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n GRADE_FORMAT=\"%(signature:grade)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n TRUSTLEVEL_FORMAT=\"%(signature:trustlevel)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n \n-- \ngitgitgadget\n\n"},{"id":"484494","messageId":"a409d77305766b7e4d391837f393cc22f9adaeca.1699320362.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH 9/9] t/perf: add perf tests for for-each-ref","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-07T01:26:01Z","receivedAt":"2023-11-07T01:26:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nAdd performance tests for 'for-each-ref'. The tests exercise different\ncombinations of filters/formats/options, as well as the overall performance\nof 'git for-each-ref | git cat-file --batch-check' to demonstrate the\nperformance difference vs. 'git for-each-ref --full-deref'.\n\nAll tests are run against a repository with 40k loose refs - 10k commits,\neach having a unique:\n\n- branch\n- custom ref (refs/custom/special_*)\n- annotated tag pointing at the commit\n- annotated tag pointing at the other annotated tag (i.e., a nested tag)\n\nAfter those tests are finished, the refs are packed with 'pack-refs --all'\nand the same tests are rerun.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n t/perf/p6300-for-each-ref.sh | 87 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 87 insertions(+)\n create mode 100755 t/perf/p6300-for-each-ref.sh\n\ndiff --git a/t/perf/p6300-for-each-ref.sh b/t/perf/p6300-for-each-ref.sh\nnew file mode 100755\nindex 00000000000..172fd68a4e9\n--- /dev/null\n+++ b/t/perf/p6300-for-each-ref.sh\n@@ -0,0 +1,87 @@\n+#!/bin/sh\n+\n+test_description='performance of for-each-ref'\n+. ./perf-lib.sh\n+\n+test_perf_fresh_repo\n+\n+ref_count_per_type=10000\n+test_iteration_count=10\n+\n+test_expect_success \"setup\" '\n+\ttest_commit_bulk $(( 1 + $ref_count_per_type )) &&\n+\n+\t# Create refs\n+\ttest_seq $ref_count_per_type |\n+\t\tsed \"s,.*,update refs/heads/branch_& HEAD~&\\nupdate refs/custom/special_& HEAD~&,\" |\n+\t\tgit update-ref --stdin &&\n+\n+\t# Create annotated tags\n+\tfor i in $(test_seq $ref_count_per_type)\n+\tdo\n+\t\t# Base tags\n+\t\techo \"tag tag_$i\" &&\n+\t\techo \"mark :$i\" &&\n+\t\techo \"from HEAD~$i\" &&\n+\t\tprintf \"tagger %s <%s> %s\\n\" \\\n+\t\t\t\"$GIT_COMMITTER_NAME\" \\\n+\t\t\t\"$GIT_COMMITTER_EMAIL\" \\\n+\t\t\t\"$GIT_COMMITTER_DATE\" &&\n+\t\techo \"data <<EOF\" &&\n+\t\techo \"tag $i\" &&\n+\t\techo \"EOF\" &&\n+\n+\t\t# Nested tags\n+\t\techo \"tag nested_$i\" &&\n+\t\techo \"from :$i\" &&\n+\t\tprintf \"tagger %s <%s> %s\\n\" \\\n+\t\t\t\"$GIT_COMMITTER_NAME\" \\\n+\t\t\t\"$GIT_COMMITTER_EMAIL\" \\\n+\t\t\t\"$GIT_COMMITTER_DATE\" &&\n+\t\techo \"data <<EOF\" &&\n+\t\techo \"nested tag $i\" &&\n+\t\techo \"EOF\" || return 1\n+\tdone | git fast-import\n+'\n+\n+test_for_each_ref () {\n+\ttitle=\"for-each-ref\"\n+\tif test $# -gt 0; then\n+\t\ttitle=\"$title ($1)\"\n+\t\tshift\n+\tfi\n+\targs=\"$@\"\n+\n+\ttest_perf \"$title\" \"\n+\t\tfor i in \\$(test_seq $test_iteration_count); do\n+\t\t\tgit for-each-ref $args >/dev/null\n+\t\tdone\n+\t\"\n+}\n+\n+run_tests () {\n+\ttest_for_each_ref \"$1\"\n+\ttest_for_each_ref \"$1, no sort\" --no-sort\n+\ttest_for_each_ref \"$1, tags\" refs/tags/\n+\ttest_for_each_ref \"$1, tags, no sort\" --no-sort refs/tags/\n+\ttest_for_each_ref \"$1, tags, shallow deref\" '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n+\ttest_for_each_ref \"$1, tags, shallow deref, no sort\" --no-sort '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n+\ttest_for_each_ref \"$1, tags, full deref\" --full-deref '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n+\ttest_for_each_ref \"$1, tags, full deref, no sort\" --no-sort --full-deref '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n+\n+\ttest_perf \"for-each-ref ($1, tags) + cat-file --batch-check (full deref)\" \"\n+\t\tfor i in \\$(test_seq $test_iteration_count); do\n+\t\t\tgit for-each-ref --format='%(objectname)^{} %(refname) %(objectname)' refs/tags/ | \\\n+\t\t\t\tgit cat-file --batch-check='%(objectname) %(rest)' >/dev/null\n+\t\tdone\n+\t\"\n+}\n+\n+run_tests \"loose\"\n+\n+test_expect_success 'pack refs' '\n+\tgit pack-refs --all\n+'\n+run_tests \"packed\"\n+\n+test_done\n-- \ngitgitgadget\n"},{"id":"484500","messageId":"xmqqo7g69tmf.fsf@gitster.g","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/9] for-each-ref optimizations & usability improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-07T02:36:56Z","receivedAt":"2023-11-07T02:37:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> This series is a bit of an informal follow-up to [1], adding some more\n> substantial optimizations and usability fixes around ref\n> filtering/formatting. Some of the changes here affect user-facing behavior,\n> some are internal-only, but they're all interdependent enough to warrant\n> putting them together in one series.\n>\n> [1]\n> https://lore.kernel.org/git/pull.1594.v2.git.1696888736.gitgitgadget@gmail.com/\n>\n> Patch 1 changes the behavior of the '--no-sort' option in 'for-each-ref',\n> 'tag', and 'branch'. Currently, it just removes previous sort keys and, if\n> no further keys are specified, falls back on ascending refname sort (which,\n> IMO, makes the name '--no-sort' somewhat misleading).\n\nWe can read it changes the behaviour and what the current behaviour\nis, but I presume that the untold new behaviour with --no-sort is to\nshow the output in an unspecified order of implementation's\nconvenience?  I think it makes quite a lot of sense if that is what\nis done.\n\n> Patch 2 updates the 'for-each-ref' docs to clearly state what happens if you\n> use '--omit-empty' and '--count' together. I based the explanation on what\n> the current behavior is (i.e., refs omitted with '--omit-empty' do count\n> towards the total limited by '--count').\n\nOK.\n\n> Patches 3-7 incrementally refactor various parts of the ref\n> filtering/formatting workflows in order to create a\n> 'filter_and_format_refs()' function. If certain conditions are met (sorting\n> disabled, no reachability filtering or ahead-behind formatting), ref\n> filtering & formatting is done within a single 'for_each_fullref_in'\n> callback. Especially in large repositories, this makes a huge difference in\n> memory usage & runtime for certain usages of 'for-each-ref', since it's no\n> longer writing everything to a 'struct ref_array' then repeatedly whittling\n> down/updating its contents.\n\nOK.  I was wondering if you are going threaded implementation, until\nI read into 6th line ;-)\n\n> Patch 8 introduces a new option to 'for-each-ref' called '--full-deref'.\n> When provided, any format fields for the dereferenced value of a tag (e.g.\n> \"%(*objectname)\") will be populated with the fully peeled target of the tag;\n> right now, those fields are populated with the immediate target of a tag\n> (which can be another tag). This avoids the need to pipe 'for-each-ref'\n> results to 'cat-file --batch-check' to get fully-peeled tag information. It\n> also benefits from the 'filter_and_format_refs()' single-iteration\n> optimization, since 'peel_iterated_oid()' may be able to read the\n> pre-computed peeled OID from a packed ref. A couple notes on this one:\n>\n>  * I went with a command line option for '--full-deref' rather than another\n>    format specifier (like ** instead of *) because it seems unlikely that a\n>    user is going to want to perform a shallow dereference and a full\n>    dereference in the same 'for-each-ref'. There's also a NEEDSWORK going\n>    all the way back to the introduction of 'for-each-ref' in 9f613ddd21c\n>    (Add git-for-each-ref: helper for language bindings, 2006-09-15) that (to\n>    me) implies different dereferencing behavior corresponds to different use\n>    cases/user needs.\n\nMakes quite a lot of sense.\n\n>  * I'm not attached to '--full-deref' as a name - if someone has an idea for\n>    a more descriptive name, please suggest it!\n\nAnother candidate verb may be \"to peel\", and I have no strong\nopinion between it and \"to dereference\".  But I have a mild aversion\nto an abbreviation that is not strongly established.\n\n> Finally, patch 9 adds performance tests for 'for-each-ref', showing the\n> effects of optimizations made throughout the series. Here are some sample\n> results from my Ubuntu VM (test names shortened for space):\n\nNice.\n"},{"id":"484503","messageId":"dbcbcf0e-aeee-4bb9-9e39-e2e85194d083@github.com","threadId":"60483","inReplyTo":"xmqqo7g69tmf.fsf@gitster.g","subject":"Re: [PATCH 0/9] for-each-ref optimizations & usability improvements","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-07T02:48:29Z","receivedAt":"2023-11-07T02:48:34Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>> This series is a bit of an informal follow-up to [1], adding some more\n>> substantial optimizations and usability fixes around ref\n>> filtering/formatting. Some of the changes here affect user-facing behavior,\n>> some are internal-only, but they're all interdependent enough to warrant\n>> putting them together in one series.\n>>\n>> [1]\n>> https://lore.kernel.org/git/pull.1594.v2.git.1696888736.gitgitgadget@gmail.com/\n>>\n>> Patch 1 changes the behavior of the '--no-sort' option in 'for-each-ref',\n>> 'tag', and 'branch'. Currently, it just removes previous sort keys and, if\n>> no further keys are specified, falls back on ascending refname sort (which,\n>> IMO, makes the name '--no-sort' somewhat misleading).\n> \n> We can read it changes the behaviour and what the current behaviour\n> is, but I presume that the untold new behaviour with --no-sort is to\n> show the output in an unspecified order of implementation's\n> convenience?  I think it makes quite a lot of sense if that is what\n> is done.\n\nAh sorry, I over-edited my cover letter and accidentally removed the\nexplanation of what this patch does! Yes - the new behavior is that\n'--no-sort' (assuming there are no subsequent --sort=<something> options)\nwill completely skip sorting the filtered refs. \n\n>>  * I'm not attached to '--full-deref' as a name - if someone has an idea for\n>>    a more descriptive name, please suggest it!\n> \n> Another candidate verb may be \"to peel\", and I have no strong\n> opinion between it and \"to dereference\".  But I have a mild aversion\n> to an abbreviation that is not strongly established.\n> \n\nMakes sense. I got the \"deref\" abbreviation for 'update-ref --no-deref', but\n'show-ref' has a \"--dereference\" option and protocol v2's \"ls-refs\" includes\na \"peel\" arg. \"Dereference\" is the term already used in the 'for-each-ref'\ndocumentation, though, so if no one comes in with an especially strong\nopinion on this I'll change the option to '--full-dereference'. Thanks!\n"},{"id":"484504","messageId":"xmqqedh29sc6.fsf@gitster.g","threadId":"60483","inReplyTo":"dbcbcf0e-aeee-4bb9-9e39-e2e85194d083@github.com","subject":"Re: [PATCH 0/9] for-each-ref optimizations & usability improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-07T03:04:41Z","receivedAt":"2023-11-07T03:04:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye <vdye@github.com> writes:\n\n> Ah sorry, I over-edited my cover letter and accidentally removed the\n> explanation of what this patch does! Yes - the new behavior is that\n> '--no-sort' (assuming there are no subsequent --sort=<something> options)\n> will completely skip sorting the filtered refs. \n\nMakes sense.\n\nAnd the way to countermand \"--no-sort\" that appears earlier on the\ncommand line to revert to the default sort order is \"--sort\" that\nuses \"refname\" as the sort key, which is also nice.\n\n"},{"id":"484518","messageId":"ZUoWPpFHEi-PZjoD@tanuki","threadId":"60483","inReplyTo":"dbcbcf0e-aeee-4bb9-9e39-e2e85194d083@github.com","subject":"Re: [PATCH 0/9] for-each-ref optimizations & usability improvements","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-07T10:49:34Z","receivedAt":"2023-11-07T10:49:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 06, 2023 at 06:48:29PM -0800, Victoria Dye wrote:\n> Junio C Hamano wrote:\n> > \"Victoria Dye via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n[snip]\n> >>  * I'm not attached to '--full-deref' as a name - if someone has an idea for\n> >>    a more descriptive name, please suggest it!\n> > \n> > Another candidate verb may be \"to peel\", and I have no strong\n> > opinion between it and \"to dereference\".  But I have a mild aversion\n> > to an abbreviation that is not strongly established.\n> > \n> \n> Makes sense. I got the \"deref\" abbreviation for 'update-ref --no-deref', but\n> 'show-ref' has a \"--dereference\" option and protocol v2's \"ls-refs\" includes\n> a \"peel\" arg. \"Dereference\" is the term already used in the 'for-each-ref'\n> documentation, though, so if no one comes in with an especially strong\n> opinion on this I'll change the option to '--full-dereference'. Thanks!\n\nBut doesn't dereferencing in the context of git-update-ref(1) refer to\nsomething different? It's not about tags, but it is about symbolic\nreferences and whether we want to update the symref or the pointee. But\ntrue enough, in git-show-ref(1) \"dereference\" actually means that we\nshould peel the tag.\n\nTo me it feels like preexisting commands are confused already. In my\nmind model:\n\n    - \"peel\" means that an object gets resolved to one of its pointees.\n      This also includes the case here, where a tag gets peeled to its\n      pointee.\n\n    - \"dereference\" means that a symbolic reference gets resolved to its\n      pointee. This matches what we do in `git update-ref --no-deref`.\n\nBut after reading through the code I don't think we distinguish those\nterms cleanly throughout our codebase. Still, \"peeling\" feels like a\nbetter match in my opinion.\n\nPatrick\n"},{"id":"484519","messageId":"ZUoWRZcD0xyfgVnc@tanuki","threadId":"60483","inReplyTo":"dea8d7d1e866d9784320051b372ff729fca855d7.1699320362.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/9] ref-filter.c: really don't sort when using --no-sort","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-07T10:49:41Z","receivedAt":"2023-11-07T10:49:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 07, 2023 at 01:25:53AM +0000, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye <vdye@github.com>\n> \n> Update 'ref_sorting_options()' to return a NULL 'struct ref_sorting *' if\n> the string list provided to it is empty, rather than returning the default\n> refname sort structure. Also update 'ref_array_sort()' to explicitly skip\n> sorting if its 'struct ref_sorting *' arg is NULL. Other functions using\n> 'struct ref_sorting *' do not need any changes because they already properly\n> ignore NULL values.\n> \n> The goal of this change is to have the '--no-sort' option truly disable\n> sorting in commands like 'for-each-ref, 'tag', and 'branch'. Right now,\n> '--no-sort' will still trigger refname sorting by default in 'for-each-ref',\n> 'tag', and 'branch'.\n> \n> To match existing behavior as closely as possible, explicitly add \"refname\"\n> to the list of sort keys in 'for-each-ref', 'tag', and 'branch' before\n> parsing options (if no config-based sort keys are set). This ensures that\n> sorting will only be fully disabled if '--no-sort' is provided as an option;\n> otherwise, \"refname\" sorting will remain the default. Note: this also means\n> that even when sort keys are provided on the command line, \"refname\" will be\n> the final sort key in the sorting structure. This doesn't actually change\n> any behavior, since 'compare_refs()' already falls back on comparing\n> refnames if two refs are equal w.r.t all other sort keys.\n> \n> Finally, remove the condition around sorting in 'ls-remote', since it's no\n> longer necessary. Unlike 'for-each-ref' et. al., it does *not* set any sort\n> keys by default. The default empty list of sort keys will produce a NULL\n> 'struct ref_sorting *', which causes the sorting to be skipped in\n> 'ref_array_sort()'.\n\nI found the order in this commit message a bit funny because you first\nexplain what you're doing, then explain the goal, and then jump into the\nchanges again. The message might be a bit easier to read if the goal was\nstated up front.\n\nI was also briefly wondering whether it would make sense to split up\nthis commit, as you're doing two different things:\n\n    - Refactor how git-for-each-ref(1), git-tag(1) and git-branch(1) set\n      up their default sorting.\n\n    - Change `ref_array_sort()` to not sort when its sorting option is\n      `NULL`.\n\nIf this was split up into two commits, then the result might be a bit\neasier to reason about. But I don't feel strongly about this.\n\n> Signed-off-by: Victoria Dye <vdye@github.com>\n> ---\n>  builtin/branch.c        |  6 ++++\n>  builtin/for-each-ref.c  |  3 ++\n>  builtin/ls-remote.c     | 10 ++----\n>  builtin/tag.c           |  6 ++++\n>  ref-filter.c            | 19 ++----------\n>  t/t3200-branch.sh       | 68 +++++++++++++++++++++++++++++++++++++++--\n>  t/t6300-for-each-ref.sh | 21 +++++++++++++\n>  t/t7004-tag.sh          | 45 +++++++++++++++++++++++++++\n>  8 files changed, 152 insertions(+), 26 deletions(-)\n> \n> diff --git a/builtin/branch.c b/builtin/branch.c\n> index e7ee9bd0f15..d67738bbcaa 100644\n> --- a/builtin/branch.c\n> +++ b/builtin/branch.c\n> @@ -767,7 +767,13 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n>  \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n>  \t\tusage_with_options(builtin_branch_usage, options);\n>  \n> +\t/*\n> +\t * Try to set sort keys from config. If config does not set any,\n> +\t * fall back on default (refname) sorting.\n> +\t */\n>  \tgit_config(git_branch_config, &sorting_options);\n> +\tif (!sorting_options.nr)\n> +\t\tstring_list_append(&sorting_options, \"refname\");\n>  \n>  \ttrack = git_branch_track;\n>  \n> diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\n> index 350bfa6e811..93b370f550b 100644\n> --- a/builtin/for-each-ref.c\n> +++ b/builtin/for-each-ref.c\n> @@ -67,6 +67,9 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n>  \n>  \tgit_config(git_default_config, NULL);\n>  \n> +\t/* Set default (refname) sorting */\n> +\tstring_list_append(&sorting_options, \"refname\");\n> +\n>  \tparse_options(argc, argv, prefix, opts, for_each_ref_usage, 0);\n>  \tif (maxcount < 0) {\n>  \t\terror(\"invalid --count argument: `%d'\", maxcount);\n> diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\n> index fc765754305..436249b720c 100644\n> --- a/builtin/ls-remote.c\n> +++ b/builtin/ls-remote.c\n> @@ -58,6 +58,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n>  \tstruct transport *transport;\n>  \tconst struct ref *ref;\n>  \tstruct ref_array ref_array;\n> +\tstruct ref_sorting *sorting;\n>  \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n>  \n>  \tstruct option options[] = {\n> @@ -141,13 +142,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n>  \t\titem->symref = xstrdup_or_null(ref->symref);\n>  \t}\n>  \n> -\tif (sorting_options.nr) {\n> -\t\tstruct ref_sorting *sorting;\n> -\n> -\t\tsorting = ref_sorting_options(&sorting_options);\n> -\t\tref_array_sort(sorting, &ref_array);\n> -\t\tref_sorting_release(sorting);\n> -\t}\n> +\tsorting = ref_sorting_options(&sorting_options);\n> +\tref_array_sort(sorting, &ref_array);\n\nWe stopped calling `ref_sorting_release()`. Doesn't that cause us to\nleak memory?\n\n>  \tfor (i = 0; i < ref_array.nr; i++) {\n>  \t\tconst struct ref_array_item *ref = ref_array.items[i];\n> diff --git a/builtin/tag.c b/builtin/tag.c\n> index 3918eacbb57..64f3196cd4c 100644\n> --- a/builtin/tag.c\n> +++ b/builtin/tag.c\n> @@ -501,7 +501,13 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n>  \n>  \tsetup_ref_filter_porcelain_msg();\n>  \n> +\t/*\n> +\t * Try to set sort keys from config. If config does not set any,\n> +\t * fall back on default (refname) sorting.\n> +\t */\n>  \tgit_config(git_tag_config, &sorting_options);\n> +\tif (!sorting_options.nr)\n> +\t\tstring_list_append(&sorting_options, \"refname\");\n>  \n>  \tmemset(&opt, 0, sizeof(opt));\n>  \tfilter.lines = -1;\n> diff --git a/ref-filter.c b/ref-filter.c\n> index e4d3510e28e..7250089b7c6 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -3142,7 +3142,8 @@ void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting,\n>  \n>  void ref_array_sort(struct ref_sorting *sorting, struct ref_array *array)\n>  {\n> -\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n> +\tif (sorting)\n> +\t\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n>  }\n>  \n>  static void append_literal(const char *cp, const char *ep, struct ref_formatting_state *state)\n> @@ -3248,18 +3249,6 @@ static int parse_sorting_atom(const char *atom)\n>  \treturn res;\n>  }\n>  \n> -/*  If no sorting option is given, use refname to sort as default */\n> -static struct ref_sorting *ref_default_sorting(void)\n> -{\n> -\tstatic const char cstr_name[] = \"refname\";\n> -\n> -\tstruct ref_sorting *sorting = xcalloc(1, sizeof(*sorting));\n> -\n> -\tsorting->next = NULL;\n> -\tsorting->atom = parse_sorting_atom(cstr_name);\n> -\treturn sorting;\n> -}\n> -\n>  static void parse_ref_sorting(struct ref_sorting **sorting_tail, const char *arg)\n>  {\n>  \tstruct ref_sorting *s;\n> @@ -3283,9 +3272,7 @@ struct ref_sorting *ref_sorting_options(struct string_list *options)\n>  \tstruct string_list_item *item;\n>  \tstruct ref_sorting *sorting = NULL, **tail = &sorting;\n>  \n> -\tif (!options->nr) {\n> -\t\tsorting = ref_default_sorting();\n> -\t} else {\n> +\tif (options->nr) {\n>  \t\tfor_each_string_list_item(item, options)\n>  \t\t\tparse_ref_sorting(tail, item->string);\n>  \t}\n> diff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\n> index 3182abde27f..9918ba05dec 100755\n> --- a/t/t3200-branch.sh\n> +++ b/t/t3200-branch.sh\n> @@ -1570,9 +1570,10 @@ test_expect_success 'tracking with unexpected .fetch refspec' '\n>  \n>  test_expect_success 'configured committerdate sort' '\n>  \tgit init -b main sort &&\n> +\ttest_config -C sort branch.sort \"committerdate\" &&\n> +\n>  \t(\n>  \t\tcd sort &&\n> -\t\tgit config branch.sort committerdate &&\n>  \t\ttest_commit initial &&\n>  \t\tgit checkout -b a &&\n>  \t\ttest_commit a &&\n> @@ -1592,9 +1593,10 @@ test_expect_success 'configured committerdate sort' '\n>  '\n>  \n>  test_expect_success 'option override configured sort' '\n> +\ttest_config -C sort branch.sort \"committerdate\" &&\n> +\n>  \t(\n>  \t\tcd sort &&\n> -\t\tgit config branch.sort committerdate &&\n>  \t\tgit branch --sort=refname >actual &&\n>  \t\tcat >expect <<-\\EOF &&\n>  \t\t  a\n> @@ -1606,10 +1608,70 @@ test_expect_success 'option override configured sort' '\n>  \t)\n>  '\n>  \n> +test_expect_success '--no-sort cancels config sort keys' '\n> +\ttest_config -C sort branch.sort \"-refname\" &&\n> +\n> +\t(\n> +\t\tcd sort &&\n> +\n> +\t\t# objecttype is identical for all of them, so sort falls back on\n> +\t\t# default (ascending refname)\n> +\t\tgit branch \\\n> +\t\t\t--no-sort \\\n> +\t\t\t--sort=\"objecttype\" >actual &&\n\nThis test is a bit confusing to me. Shouldn't we in fact ignore the\nconfigured sorting order as soon as we pass `--sort=` anyway? In other\nwords, I would expect the `--no-sort` option to not make a difference\nhere. What should make a difference is if you _only_ passed `--no-sort`.\n\n> +\t\tcat >expect <<-\\EOF &&\n> +\t\t  a\n> +\t\t* b\n> +\t\t  c\n> +\t\t  main\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t)\n> +\n> +'\n> +\n> +test_expect_success '--no-sort cancels command line sort keys' '\n> +\t(\n> +\t\tcd sort &&\n> +\n> +\t\t# objecttype is identical for all of them, so sort falls back on\n> +\t\t# default (ascending refname)\n> +\t\tgit branch \\\n> +\t\t\t--sort=\"-refname\" \\\n> +\t\t\t--no-sort \\\n> +\t\t\t--sort=\"objecttype\" >actual &&\n> +\t\tcat >expect <<-\\EOF &&\n> +\t\t  a\n> +\t\t* b\n> +\t\t  c\n> +\t\t  main\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success '--no-sort without subsequent --sort prints expected branches' '\n> +\t(\n> +\t\tcd sort &&\n> +\n> +\t\t# Sort the results with `sort` for a consistent comparison\n> +\t\t# against expected\n> +\t\tgit branch --no-sort | sort >actual &&\n> +\t\tcat >expect <<-\\EOF &&\n> +\t\t  a\n> +\t\t  c\n> +\t\t  main\n> +\t\t* b\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n>  test_expect_success 'invalid sort parameter in configuration' '\n> +\ttest_config -C sort branch.sort \"v:notvalid\" &&\n> +\n>  \t(\n>  \t\tcd sort &&\n> -\t\tgit config branch.sort \"v:notvalid\" &&\n>  \n>  \t\t# this works in the \"listing\" mode, so bad sort key\n>  \t\t# is a dying offence.\n> diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\n> index 00a060df0b5..0613e5e3623 100755\n> --- a/t/t6300-for-each-ref.sh\n> +++ b/t/t6300-for-each-ref.sh\n> @@ -1335,6 +1335,27 @@ test_expect_success '--no-sort cancels the previous sort keys' '\n>  \ttest_cmp expected actual\n>  '\n>  \n> +test_expect_success '--no-sort without subsequent --sort prints expected refs' '\n> +\tcat >expected <<-\\EOF &&\n> +\trefs/tags/multi-ref1-100000-user1\n> +\trefs/tags/multi-ref1-100000-user2\n> +\trefs/tags/multi-ref1-200000-user1\n> +\trefs/tags/multi-ref1-200000-user2\n> +\trefs/tags/multi-ref2-100000-user1\n> +\trefs/tags/multi-ref2-100000-user2\n> +\trefs/tags/multi-ref2-200000-user1\n> +\trefs/tags/multi-ref2-200000-user2\n> +\tEOF\n> +\n> +\t# Sort the results with `sort` for a consistent comparison against\n> +\t# expected\n> +\tgit for-each-ref \\\n> +\t\t--format=\"%(refname)\" \\\n> +\t\t--no-sort \\\n> +\t\t\"refs/tags/multi-*\" | sort >actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n>  test_expect_success 'do not dereference NULL upon %(HEAD) on unborn branch' '\n>  \ttest_when_finished \"git checkout main\" &&\n>  \tgit for-each-ref --format=\"%(HEAD) %(refname:short)\" refs/heads/ >actual &&\n> diff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\n> index e689db42929..b41a47eb943 100755\n> --- a/t/t7004-tag.sh\n> +++ b/t/t7004-tag.sh\n> @@ -1862,6 +1862,51 @@ test_expect_success 'option override configured sort' '\n>  \ttest_cmp expect actual\n>  '\n>  \n> +test_expect_success '--no-sort cancels config sort keys' '\n> +\ttest_config tag.sort \"-refname\" &&\n> +\n> +\t# objecttype is identical for all of them, so sort falls back on\n> +\t# default (ascending refname)\n> +\tgit tag -l \\\n> +\t\t--no-sort \\\n> +\t\t--sort=\"objecttype\" \\\n> +\t\t\"foo*\" >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\tfoo1.10\n> +\tfoo1.3\n> +\tfoo1.6\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n\nSame question here.\n\nPatrick\n\n> +test_expect_success '--no-sort cancels command line sort keys' '\n> +\t# objecttype is identical for all of them, so sort falls back on\n> +\t# default (ascending refname)\n> +\tgit tag -l \\\n> +\t\t--sort=\"-refname\" \\\n> +\t\t--no-sort \\\n> +\t\t--sort=\"objecttype\" \\\n> +\t\t\"foo*\" >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\tfoo1.10\n> +\tfoo1.3\n> +\tfoo1.6\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success '--no-sort without subsequent --sort prints expected tags' '\n> +\t# Sort the results with `sort` for a consistent comparison against\n> +\t# expected\n> +\tgit tag -l --no-sort \"foo*\" | sort >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\tfoo1.10\n> +\tfoo1.3\n> +\tfoo1.6\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n>  test_expect_success 'invalid sort parameter on command line' '\n>  \ttest_must_fail git tag -l --sort=notvalid \"foo*\" >actual\n>  '\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"484520","messageId":"ZUoWSocLddxm_7WK@tanuki","threadId":"60483","inReplyTo":"6c66445ee31dd4117e1384d8da7be81f401317b3.1699320362.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/9] ref-filter.h: move contains caches into filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-07T10:49:46Z","receivedAt":"2023-11-07T10:49:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 07, 2023 at 01:25:56AM +0000, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye <vdye@github.com>\n> \n> Move the 'contains_cache' and 'no_contains_cache' used in filter_refs into\n> an 'internal' struct of the 'struct ref_filter'. In later patches, the\n> 'struct ref_filter *' will be a common data structure across multiple\n> filtering functions. These caches are part of the common functionality the\n> filter struct will support, so they are updated to be internally accessible\n> wherever the filter is used.\n> \n> The design used here is mirrors what was introduced in 576de3d956\n\nNit: s/is //\n\nPatrick\n\n> (unpack_trees: start splitting internal fields from public API, 2023-02-27)\n> for 'unpack_trees_options'.\n> \n> Signed-off-by: Victoria Dye <vdye@github.com>\n> ---\n>  ref-filter.c | 14 ++++++--------\n>  ref-filter.h |  6 ++++++\n>  2 files changed, 12 insertions(+), 8 deletions(-)\n> \n> diff --git a/ref-filter.c b/ref-filter.c\n> index 7250089b7c6..5129b6986c9 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2764,8 +2764,6 @@ static int filter_ref_kind(struct ref_filter *filter, const char *refname)\n>  struct ref_filter_cbdata {\n>  \tstruct ref_array *array;\n>  \tstruct ref_filter *filter;\n> -\tstruct contains_cache contains_cache;\n> -\tstruct contains_cache no_contains_cache;\n>  };\n>  \n>  /*\n> @@ -2816,11 +2814,11 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n>  \t\t\treturn 0;\n>  \t\t/* We perform the filtering for the '--contains' option... */\n>  \t\tif (filter->with_commit &&\n> -\t\t    !commit_contains(filter, commit, filter->with_commit, &ref_cbdata->contains_cache))\n> +\t\t    !commit_contains(filter, commit, filter->with_commit, &filter->internal.contains_cache))\n>  \t\t\treturn 0;\n>  \t\t/* ...or for the `--no-contains' option */\n>  \t\tif (filter->no_commit &&\n> -\t\t    commit_contains(filter, commit, filter->no_commit, &ref_cbdata->no_contains_cache))\n> +\t\t    commit_contains(filter, commit, filter->no_commit, &filter->internal.no_contains_cache))\n>  \t\t\treturn 0;\n>  \t}\n>  \n> @@ -2989,8 +2987,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n>  \tsave_commit_buffer_orig = save_commit_buffer;\n>  \tsave_commit_buffer = 0;\n>  \n> -\tinit_contains_cache(&ref_cbdata.contains_cache);\n> -\tinit_contains_cache(&ref_cbdata.no_contains_cache);\n> +\tinit_contains_cache(&filter->internal.contains_cache);\n> +\tinit_contains_cache(&filter->internal.no_contains_cache);\n>  \n>  \t/*  Simple per-ref filtering */\n>  \tif (!filter->kind)\n> @@ -3014,8 +3012,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n>  \t\t\thead_ref(ref_filter_handler, &ref_cbdata);\n>  \t}\n>  \n> -\tclear_contains_cache(&ref_cbdata.contains_cache);\n> -\tclear_contains_cache(&ref_cbdata.no_contains_cache);\n> +\tclear_contains_cache(&filter->internal.contains_cache);\n> +\tclear_contains_cache(&filter->internal.no_contains_cache);\n>  \n>  \t/*  Filters that need revision walking */\n>  \treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n> diff --git a/ref-filter.h b/ref-filter.h\n> index d87d61238b7..0db3ff52889 100644\n> --- a/ref-filter.h\n> +++ b/ref-filter.h\n> @@ -7,6 +7,7 @@\n>  #include \"commit.h\"\n>  #include \"string-list.h\"\n>  #include \"strvec.h\"\n> +#include \"commit-reach.h\"\n>  \n>  /* Quoting styles */\n>  #define QUOTE_NONE 0\n> @@ -75,6 +76,11 @@ struct ref_filter {\n>  \t\tlines;\n>  \tint abbrev,\n>  \t\tverbose;\n> +\n> +\tstruct {\n> +\t\tstruct contains_cache contains_cache;\n> +\t\tstruct contains_cache no_contains_cache;\n> +\t} internal;\n>  };\n>  \n>  struct ref_format {\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"484521","messageId":"ZUoWT8GyrZlvH_Go@tanuki","threadId":"60483","inReplyTo":"8c77452e5dd8d5cafd95c68480bf5675d51b4736.1699320362.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/9] ref-filter.c: refactor to create common helper functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-07T10:49:51Z","receivedAt":"2023-11-07T10:49:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 07, 2023 at 01:25:58AM +0000, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye <vdye@github.com>\n> \n> Factor out parts of 'ref_array_push()', 'ref_filter_handler()', and\n> 'filter_refs()' into new helper functions ('ref_array_append()',\n> 'apply_ref_filter()', and 'do_filter_refs()' respectively), as well as\n> rename 'ref_filter_handler()' to 'filter_one()'. In this and later\n> patches, these helpers will be used by new ref-filter API functions. This\n> patch does not result in any user-facing behavior changes or changes to\n> callers outside of 'ref-filter.c'.\n> \n> The changes are as follows:\n> \n> * The logic to grow a 'struct ref_array' and append a given 'struct\n>   ref_array_item *' to it is extracted from 'ref_array_push()' into\n>   'ref_array_append()'.\n> * 'ref_filter_handler()' is renamed to 'filter_one()' to more clearly\n>   distinguish it from other ref filtering callbacks that will be added in\n>   later patches. The \"*_one()\" naming convention is common throughout the\n>   codebase for iteration callbacks.\n> * The code to filter a given ref by refname & object ID then create a new\n>   'struct ref_array_item' is moved out of 'filter_one()' and into\n>   'apply_ref_filter()'. 'apply_ref_filter()' returns either NULL (if the ref\n>   does not match the given filter) or a 'struct ref_array_item *' created\n>   with 'new_ref_array_item()'; 'filter_one()' appends that item to\n>   its ref array with 'ref_array_append()'.\n> * The filter pre-processing, contains cache creation, and ref iteration of\n>   'filter_refs()' is extracted into 'do_filter_refs()'. 'do_filter_refs()'\n>   takes its ref iterator function & callback data as an input from the\n>   caller, setting it up to be used with additional filtering callbacks in\n>   later patches.\n\nTo me, a bulleted list spelling out the different changes I'm doing\noften indicates that I might want to split up the commit into one for\neach of the items. I don't feel strongly about this, but think that it\nmight help the reviewer in this case.\n\nPatrick\n\n> Signed-off-by: Victoria Dye <vdye@github.com>\n> ---\n>  ref-filter.c | 115 ++++++++++++++++++++++++++++++---------------------\n>  1 file changed, 69 insertions(+), 46 deletions(-)\n> \n> diff --git a/ref-filter.c b/ref-filter.c\n> index 8992fbf45b1..ff00ab4b8d8 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2716,15 +2716,18 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n>  \treturn ref;\n>  }\n>  \n> +static void ref_array_append(struct ref_array *array, struct ref_array_item *ref)\n> +{\n> +\tALLOC_GROW(array->items, array->nr + 1, array->alloc);\n> +\tarray->items[array->nr++] = ref;\n> +}\n> +\n>  struct ref_array_item *ref_array_push(struct ref_array *array,\n>  \t\t\t\t      const char *refname,\n>  \t\t\t\t      const struct object_id *oid)\n>  {\n>  \tstruct ref_array_item *ref = new_ref_array_item(refname, oid);\n> -\n> -\tALLOC_GROW(array->items, array->nr + 1, array->alloc);\n> -\tarray->items[array->nr++] = ref;\n> -\n> +\tref_array_append(array, ref);\n>  \treturn ref;\n>  }\n>  \n> @@ -2761,46 +2764,36 @@ static int filter_ref_kind(struct ref_filter *filter, const char *refname)\n>  \treturn ref_kind_from_refname(refname);\n>  }\n>  \n> -struct ref_filter_cbdata {\n> -\tstruct ref_array *array;\n> -\tstruct ref_filter *filter;\n> -};\n> -\n> -/*\n> - * A call-back given to for_each_ref().  Filter refs and keep them for\n> - * later object processing.\n> - */\n> -static int ref_filter_handler(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n> +static struct ref_array_item *apply_ref_filter(const char *refname, const struct object_id *oid,\n> +\t\t\t    int flag, struct ref_filter *filter)\n>  {\n> -\tstruct ref_filter_cbdata *ref_cbdata = cb_data;\n> -\tstruct ref_filter *filter = ref_cbdata->filter;\n>  \tstruct ref_array_item *ref;\n>  \tstruct commit *commit = NULL;\n>  \tunsigned int kind;\n>  \n>  \tif (flag & REF_BAD_NAME) {\n>  \t\twarning(_(\"ignoring ref with broken name %s\"), refname);\n> -\t\treturn 0;\n> +\t\treturn NULL;\n>  \t}\n>  \n>  \tif (flag & REF_ISBROKEN) {\n>  \t\twarning(_(\"ignoring broken ref %s\"), refname);\n> -\t\treturn 0;\n> +\t\treturn NULL;\n>  \t}\n>  \n>  \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n>  \tkind = filter_ref_kind(filter, refname);\n>  \tif (!(kind & filter->kind))\n> -\t\treturn 0;\n> +\t\treturn NULL;\n>  \n>  \tif (!filter_pattern_match(filter, refname))\n> -\t\treturn 0;\n> +\t\treturn NULL;\n>  \n>  \tif (filter_exclude_match(filter, refname))\n> -\t\treturn 0;\n> +\t\treturn NULL;\n>  \n>  \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n> -\t\treturn 0;\n> +\t\treturn NULL;\n>  \n>  \t/*\n>  \t * A merge filter is applied on refs pointing to commits. Hence\n> @@ -2811,15 +2804,15 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n>  \t    filter->with_commit || filter->no_commit || filter->verbose) {\n>  \t\tcommit = lookup_commit_reference_gently(the_repository, oid, 1);\n>  \t\tif (!commit)\n> -\t\t\treturn 0;\n> +\t\t\treturn NULL;\n>  \t\t/* We perform the filtering for the '--contains' option... */\n>  \t\tif (filter->with_commit &&\n>  \t\t    !commit_contains(filter, commit, filter->with_commit, &filter->internal.contains_cache))\n> -\t\t\treturn 0;\n> +\t\t\treturn NULL;\n>  \t\t/* ...or for the `--no-contains' option */\n>  \t\tif (filter->no_commit &&\n>  \t\t    commit_contains(filter, commit, filter->no_commit, &filter->internal.no_contains_cache))\n> -\t\t\treturn 0;\n> +\t\t\treturn NULL;\n>  \t}\n>  \n>  \t/*\n> @@ -2827,11 +2820,32 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n>  \t * to do its job and the resulting list may yet to be pruned\n>  \t * by maxcount logic.\n>  \t */\n> -\tref = ref_array_push(ref_cbdata->array, refname, oid);\n> +\tref = new_ref_array_item(refname, oid);\n>  \tref->commit = commit;\n>  \tref->flag = flag;\n>  \tref->kind = kind;\n>  \n> +\treturn ref;\n> +}\n> +\n> +struct ref_filter_cbdata {\n> +\tstruct ref_array *array;\n> +\tstruct ref_filter *filter;\n> +};\n> +\n> +/*\n> + * A call-back given to for_each_ref().  Filter refs and keep them for\n> + * later object processing.\n> + */\n> +static int filter_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n> +{\n> +\tstruct ref_filter_cbdata *ref_cbdata = cb_data;\n> +\tstruct ref_array_item *ref;\n> +\n> +\tref = apply_ref_filter(refname, oid, flag, ref_cbdata->filter);\n> +\tif (ref)\n> +\t\tref_array_append(ref_cbdata->array, ref);\n> +\n>  \treturn 0;\n>  }\n>  \n> @@ -2967,26 +2981,12 @@ void filter_ahead_behind(struct repository *r,\n>  \tfree(commits);\n>  }\n>  \n> -/*\n> - * API for filtering a set of refs. Based on the type of refs the user\n> - * has requested, we iterate through those refs and apply filters\n> - * as per the given ref_filter structure and finally store the\n> - * filtered refs in the ref_array structure.\n> - */\n> -int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type)\n> +static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)\n>  {\n> -\tstruct ref_filter_cbdata ref_cbdata;\n> -\tint save_commit_buffer_orig;\n>  \tint ret = 0;\n>  \n> -\tref_cbdata.array = array;\n> -\tref_cbdata.filter = filter;\n> -\n>  \tfilter->kind = type & FILTER_REFS_KIND_MASK;\n>  \n> -\tsave_commit_buffer_orig = save_commit_buffer;\n> -\tsave_commit_buffer = 0;\n> -\n>  \tinit_contains_cache(&filter->internal.contains_cache);\n>  \tinit_contains_cache(&filter->internal.no_contains_cache);\n>  \n> @@ -3001,20 +3001,43 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n>  \t\t * of filter_ref_kind().\n>  \t\t */\n>  \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n> -\t\t\tret = for_each_fullref_in(\"refs/heads/\", ref_filter_handler, &ref_cbdata);\n> +\t\t\tret = for_each_fullref_in(\"refs/heads/\", fn, cb_data);\n>  \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n> -\t\t\tret = for_each_fullref_in(\"refs/remotes/\", ref_filter_handler, &ref_cbdata);\n> +\t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n>  \t\telse if (filter->kind == FILTER_REFS_TAGS)\n> -\t\t\tret = for_each_fullref_in(\"refs/tags/\", ref_filter_handler, &ref_cbdata);\n> +\t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n>  \t\telse if (filter->kind & FILTER_REFS_ALL)\n> -\t\t\tret = for_each_fullref_in_pattern(filter, ref_filter_handler, &ref_cbdata);\n> +\t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n>  \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n> -\t\t\thead_ref(ref_filter_handler, &ref_cbdata);\n> +\t\t\thead_ref(fn, cb_data);\n>  \t}\n>  \n>  \tclear_contains_cache(&filter->internal.contains_cache);\n>  \tclear_contains_cache(&filter->internal.no_contains_cache);\n>  \n> +\treturn ret;\n> +}\n> +\n> +/*\n> + * API for filtering a set of refs. Based on the type of refs the user\n> + * has requested, we iterate through those refs and apply filters\n> + * as per the given ref_filter structure and finally store the\n> + * filtered refs in the ref_array structure.\n> + */\n> +int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type)\n> +{\n> +\tstruct ref_filter_cbdata ref_cbdata;\n> +\tint save_commit_buffer_orig;\n> +\tint ret = 0;\n> +\n> +\tref_cbdata.array = array;\n> +\tref_cbdata.filter = filter;\n> +\n> +\tsave_commit_buffer_orig = save_commit_buffer;\n> +\tsave_commit_buffer = 0;\n> +\n> +\tret = do_filter_refs(filter, type, filter_one, &ref_cbdata);\n> +\n>  \t/*  Filters that need revision walking */\n>  \treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n>  \treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"484522","messageId":"ZUoWVPSE1GcJdHFE@tanuki","threadId":"60483","inReplyTo":"84db440896c162bcbeeaaf00d528839056aefaa5.1699320362.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/9] ref-filter.c: filter & format refs in the same callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-07T10:49:56Z","receivedAt":"2023-11-07T10:50:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 07, 2023 at 01:25:59AM +0000, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye <vdye@github.com>\n> \n> Update 'filter_and_format_refs()' to try to perform ref filtering &\n> formatting in a single ref iteration, without an intermediate 'struct\n> ref_array'. This can only be done if no operations need to be performed on a\n> pre-filtered array; specifically, if the refs are\n> \n> - filtered on reachability,\n> - sorted, or\n> - formatted with ahead-behind information\n> \n> they cannot be filtered & formatted in the same iteration. In that case,\n> fall back on the current filter-then-sort-then-format flow.\n> \n> This optimization substantially improves memory usage due to no longer\n> storing a ref array in memory. In some cases, it also dramatically reduces\n> runtime (e.g. 'git for-each-ref --no-sort --count=1', which no longer loads\n> all refs into a 'struct ref_array' to printing only the first ref).\n> \n> Signed-off-by: Victoria Dye <vdye@github.com>\n> ---\n>  ref-filter.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++----\n>  1 file changed, 74 insertions(+), 6 deletions(-)\n> \n> diff --git a/ref-filter.c b/ref-filter.c\n> index ff00ab4b8d8..384cf1595ff 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2863,6 +2863,44 @@ static void free_array_item(struct ref_array_item *item)\n>  \tfree(item);\n>  }\n>  \n> +struct ref_filter_and_format_cbdata {\n> +\tstruct ref_filter *filter;\n> +\tstruct ref_format *format;\n> +\n> +\tstruct ref_filter_and_format_internal {\n> +\t\tint count;\n> +\t} internal;\n> +};\n> +\n> +static int filter_and_format_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n> +{\n> +\tstruct ref_filter_and_format_cbdata *ref_cbdata = cb_data;\n> +\tstruct ref_array_item *ref;\n> +\tstruct strbuf output = STRBUF_INIT, err = STRBUF_INIT;\n> +\n> +\tref = apply_ref_filter(refname, oid, flag, ref_cbdata->filter);\n> +\tif (!ref)\n> +\t\treturn 0;\n> +\n> +\tif (format_ref_array_item(ref, ref_cbdata->format, &output, &err))\n> +\t\tdie(\"%s\", err.buf);\n> +\n> +\tif (output.len || !ref_cbdata->format->array_opts.omit_empty) {\n> +\t\tfwrite(output.buf, 1, output.len, stdout);\n> +\t\tputchar('\\n');\n> +\t}\n> +\n> +\tstrbuf_release(&output);\n> +\tstrbuf_release(&err);\n> +\tfree_array_item(ref);\n> +\n> +\tif (ref_cbdata->format->array_opts.max_count &&\n> +\t    ++ref_cbdata->internal.count >= ref_cbdata->format->array_opts.max_count)\n> +\t\treturn -1;\n\nIt feels a bit weird to return a negative value here, which usually\nindicates that an error has happened whereas we only use it here to\nabort the iteration. But we ignore the return value of\n`do_iterate_refs()` anyway, so it doesn't make much of a difference.\n\n> +\treturn 0;\n> +}\n> +\n>  /* Free all memory allocated for ref_array */\n>  void ref_array_clear(struct ref_array *array)\n>  {\n> @@ -3046,16 +3084,46 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n>  \treturn ret;\n>  }\n>  \n> +static inline int can_do_iterative_format(struct ref_filter *filter,\n> +\t\t\t\t\t  struct ref_sorting *sorting,\n> +\t\t\t\t\t  struct ref_format *format)\n> +{\n> +\t/*\n> +\t * Refs can be filtered and formatted in the same iteration as long\n> +\t * as we aren't filtering on reachability, sorting the results, or\n> +\t * including ahead-behind information in the formatted output.\n> +\t */\n\nDo we want to format this as a bulleted list so that it's more readily\nextensible if we ever need to pay attention to new options here? Also, I\nnoted that this commit doesn't add any new tests -- do we already\nexercise all of these conditions?\n\nMore generally, I worry a bit about maintainability of this code snippet\nas we need to remember to always update this condition whenever we add a\nnew option, and this can be quite easy to miss. The performance benefit\nmight be worth the effort though.\n\nPatrick\n\n> +\treturn !(filter->reachable_from ||\n> +\t\t filter->unreachable_from ||\n> +\t\t sorting ||\n> +\t\t format->bases.nr);\n> +}\n> +\n>  void filter_and_format_refs(struct ref_filter *filter, unsigned int type,\n>  \t\t\t    struct ref_sorting *sorting,\n>  \t\t\t    struct ref_format *format)\n>  {\n> -\tstruct ref_array array = { 0 };\n> -\tfilter_refs(&array, filter, type);\n> -\tfilter_ahead_behind(the_repository, format, &array);\n> -\tref_array_sort(sorting, &array);\n> -\tprint_formatted_ref_array(&array, format);\n> -\tref_array_clear(&array);\n> +\tif (can_do_iterative_format(filter, sorting, format)) {\n> +\t\tint save_commit_buffer_orig;\n> +\t\tstruct ref_filter_and_format_cbdata ref_cbdata = {\n> +\t\t\t.filter = filter,\n> +\t\t\t.format = format,\n> +\t\t};\n> +\n> +\t\tsave_commit_buffer_orig = save_commit_buffer;\n> +\t\tsave_commit_buffer = 0;\n> +\n> +\t\tdo_filter_refs(filter, type, filter_and_format_one, &ref_cbdata);\n> +\n> +\t\tsave_commit_buffer = save_commit_buffer_orig;\n> +\t} else {\n> +\t\tstruct ref_array array = { 0 };\n> +\t\tfilter_refs(&array, filter, type);\n> +\t\tfilter_ahead_behind(the_repository, format, &array);\n> +\t\tref_array_sort(sorting, &array);\n> +\t\tprint_formatted_ref_array(&array, format);\n> +\t\tref_array_clear(&array);\n> +\t}\n>  }\n>  \n>  static int compare_detached_head(struct ref_array_item *a, struct ref_array_item *b)\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"484523","messageId":"ZUoWWo7IEKsiSx-C@tanuki","threadId":"60483","inReplyTo":"352b5c42ac39d5d2646a1b6d47d6d707637db539.1699320362.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-07T10:50:02Z","receivedAt":"2023-11-07T10:50:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 07, 2023 at 01:26:00AM +0000, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye <vdye@github.com>\n> \n> Add a boolean flag '--full-deref' that, when enabled, fills '%(*fieldname)'\n> format fields using the fully peeled target of tag objects, rather than the\n> immediate target.\n> \n> In other builtins ('rev-parse', 'show-ref'), \"dereferencing\" tags typically\n> means peeling them down to their non-tag target. Unlike these commands,\n> 'for-each-ref' dereferences only one \"level\" of tags in '*' format fields\n> (like \"%(*objectname)\"). For most annotated tags, one level of dereferencing\n> is enough, since most tags point to commits or trees. However, nested tags\n> (annotated tags whose target is another annotated tag) dereferenced once\n> will point to their target tag, different a full peel to e.g. a commit.\n> \n> Currently, if a user wants to filter & format refs and include information\n> about the fully dereferenced tag, they can do so with something like\n> 'cat-file --batch-check':\n> \n>     git for-each-ref --format=\"%(objectname)^{} %(refname)\" <pattern> |\n>         git cat-file --batch-check=\"%(objectname) %(rest)\"\n> \n> But the combination of commands is inefficient. So, to improve the\n> efficiency of this use case, add a '--full-deref' option that causes\n> 'for-each-ref' to fully dereference tags when formatting with '*' fields.\n\nI do wonder whether it would make sense to introduce this feature in the\nform of a separate field prefix, as you also mentioned in your cover\nletter. It would buy the user more flexibility, but the question is\nwhether such flexibility would really ever be needed.\n\nThe only thing I could really think of where it might make sense is to\ndistinguish tags that peel to a commit immediately from ones that don't.\nThat feels rather esoteric to me and doesn't seem to be of much use. But\nregardless of whether or not we can see the usefulness now, if this\nwouldn't be significantly more complex I wonder whether it would make\nmore sense to use a new field prefix instead anyway.\n\nIn any case, I think it would be helpful if this was discussed in the\ncommit message.\n\nPatrick\n\n> Signed-off-by: Victoria Dye <vdye@github.com>\n> ---\n>  Documentation/git-for-each-ref.txt |  9 ++++++++\n>  builtin/for-each-ref.c             |  2 ++\n>  ref-filter.c                       | 26 ++++++++++++++---------\n>  ref-filter.h                       |  1 +\n>  t/t6300-for-each-ref.sh            | 34 ++++++++++++++++++++++++++++++\n>  5 files changed, 62 insertions(+), 10 deletions(-)\n> \n> diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\n> index 407f624fbaa..2714a87088e 100644\n> --- a/Documentation/git-for-each-ref.txt\n> +++ b/Documentation/git-for-each-ref.txt\n> @@ -11,6 +11,7 @@ SYNOPSIS\n>  'git for-each-ref' [--count=<count>] [--shell|--perl|--python|--tcl]\n>  \t\t   [(--sort=<key>)...] [--format=<format>]\n>  \t\t   [ --stdin | <pattern>... ]\n> +\t\t   [--full-deref]\n>  \t\t   [--points-at=<object>]\n>  \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n>  \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n> @@ -77,6 +78,14 @@ OPTIONS\n>  \tthe specified host language.  This is meant to produce\n>  \ta scriptlet that can directly be `eval`ed.\n>  \n> +--full-deref::\n> +\tPopulate dereferenced format fields (indicated with an asterisk (`*`)\n> +\tprefix before the fieldname) with information about the fully-peeled\n> +\ttarget object of a tag ref, rather than its immediate target object.\n> +\tThis only affects the output for nested annotated tags, where the tag's\n> +\timmediate target is another tag but its fully-peeled target is another\n> +\tobject type (e.g. a commit).\n> +\n>  --points-at=<object>::\n>  \tOnly list refs which points at the given object.\n>  \n> diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\n> index 1c19cd5bd34..7a2127a3bc4 100644\n> --- a/builtin/for-each-ref.c\n> +++ b/builtin/for-each-ref.c\n> @@ -43,6 +43,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n>  \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only <n> matched refs\")),\n>  \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n>  \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n> +\t\tOPT_BOOL(0, \"full-deref\", &format.full_deref,\n> +\t\t\t N_(\"fully dereference tags to populate '*' format fields\")),\n>  \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n>  \t\tOPT_REF_SORT(&sorting_options),\n>  \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 384cf1595ff..a66ac7921b1 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -237,7 +237,14 @@ static struct used_atom {\n>  \t\tchar *head;\n>  \t} u;\n>  } *used_atom;\n> -static int used_atom_cnt, need_tagged, need_symref;\n> +static int used_atom_cnt, need_symref;\n> +\n> +enum tag_dereference_mode {\n> +\tNO_DEREF = 0,\n> +\tDEREF_ONE,\n> +\tDEREF_ALL\n> +};\n> +static enum tag_dereference_mode need_tagged;\n>  \n>  /*\n>   * Expand string, append it to strbuf *sb, then return error code ret.\n> @@ -1066,8 +1073,8 @@ static int parse_ref_filter_atom(struct ref_format *format,\n>  \tmemset(&used_atom[at].u, 0, sizeof(used_atom[at].u));\n>  \tif (valid_atom[i].parser && valid_atom[i].parser(format, &used_atom[at], arg, err))\n>  \t\treturn -1;\n> -\tif (*atom == '*')\n> -\t\tneed_tagged = 1;\n> +\tif (*atom == '*' && !need_tagged)\n> +\t\tneed_tagged = format->full_deref ? DEREF_ALL : DEREF_ONE;\n>  \tif (i == ATOM_SYMREF)\n>  \t\tneed_symref = 1;\n>  \treturn at;\n> @@ -2511,14 +2518,13 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n>  \t * If it is a tag object, see if we use a value that derefs\n>  \t * the object, and if we do grab the object it refers to.\n>  \t */\n> -\toi_deref.oid = *get_tagged_oid((struct tag *)obj);\n> +\tif (need_tagged == DEREF_ALL) {\n> +\t\tif (peel_iterated_oid(&obj->oid, &oi_deref.oid))\n> +\t\t\tdie(\"bad tag\");\n> +\t} else {\n> +\t\toi_deref.oid = *get_tagged_oid((struct tag *)obj);\n> +\t}\n>  \n> -\t/*\n> -\t * NEEDSWORK: This derefs tag only once, which\n> -\t * is good to deal with chains of trust, but\n> -\t * is not consistent with what deref_tag() does\n> -\t * which peels the onion to the core.\n> -\t */\n>  \treturn get_object(ref, 1, &obj, &oi_deref, err);\n>  }\n>  \n> diff --git a/ref-filter.h b/ref-filter.h\n> index 0ce5af58ab3..0caa39ecee5 100644\n> --- a/ref-filter.h\n> +++ b/ref-filter.h\n> @@ -92,6 +92,7 @@ struct ref_format {\n>  \tconst char *rest;\n>  \tint quote_style;\n>  \tint use_color;\n> +\tint full_deref;\n>  \n>  \t/* Internal state to ref-filter */\n>  \tint need_color_reset_at_eol;\n> diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\n> index 0613e5e3623..3c2af785cdb 100755\n> --- a/t/t6300-for-each-ref.sh\n> +++ b/t/t6300-for-each-ref.sh\n> @@ -1839,6 +1839,40 @@ test_expect_success 'git for-each-ref with non-existing refs' '\n>  \ttest_must_be_empty actual\n>  '\n>  \n> +test_expect_success 'git for-each-ref with nested tags' '\n> +\tgit tag -am \"Normal tag\" nested/base HEAD &&\n> +\tgit tag -am \"Nested tag\" nested/nest1 refs/tags/nested/base &&\n> +\tgit tag -am \"Double nested tag\" nested/nest2 refs/tags/nested/nest1 &&\n> +\n> +\thead_oid=\"$(git rev-parse HEAD)\" &&\n> +\tbase_tag_oid=\"$(git rev-parse refs/tags/nested/base)\" &&\n> +\tnest1_tag_oid=\"$(git rev-parse refs/tags/nested/nest1)\" &&\n> +\tnest2_tag_oid=\"$(git rev-parse refs/tags/nested/nest2)\" &&\n> +\n> +\t# Without full dereference\n> +\tcat >expect <<-EOF &&\n> +\trefs/tags/nested/base $base_tag_oid tag $head_oid commit\n> +\trefs/tags/nested/nest1 $nest1_tag_oid tag $base_tag_oid tag\n> +\trefs/tags/nested/nest2 $nest2_tag_oid tag $nest1_tag_oid tag\n> +\tEOF\n> +\n> +\tgit for-each-ref --format=\"%(refname) %(objectname) %(objecttype) %(*objectname) %(*objecttype)\" \\\n> +\t\trefs/tags/nested/ >actual &&\n> +\ttest_cmp expect actual &&\n> +\n> +\t# With full dereference\n> +\tcat >expect <<-EOF &&\n> +\trefs/tags/nested/base $base_tag_oid tag $head_oid commit\n> +\trefs/tags/nested/nest1 $nest1_tag_oid tag $head_oid commit\n> +\trefs/tags/nested/nest2 $nest2_tag_oid tag $head_oid commit\n> +\tEOF\n> +\n> +\tgit for-each-ref --full-deref \\\n> +\t\t--format=\"%(refname) %(objectname) %(objecttype) %(*objectname) %(*objecttype)\" \\\n> +\t\trefs/tags/nested/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n>  GRADE_FORMAT=\"%(signature:grade)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n>  TRUSTLEVEL_FORMAT=\"%(signature:trustlevel)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n>  \n> -- \n> gitgitgadget\n> \n> \n"},{"id":"484536","messageId":"a833b5a7-0201-4c2e-8821-f2a1930cb403@github.com","threadId":"60483","inReplyTo":"ZUoWRZcD0xyfgVnc@tanuki","subject":"Re: [PATCH 1/9] ref-filter.c: really don't sort when using --no-sort","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-07T18:13:17Z","receivedAt":"2023-11-07T18:13:20Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:\n> On Tue, Nov 07, 2023 at 01:25:53AM +0000, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye <vdye@github.com>\n>>\n>> Update 'ref_sorting_options()' to return a NULL 'struct ref_sorting *' if\n>> the string list provided to it is empty, rather than returning the default\n>> refname sort structure. Also update 'ref_array_sort()' to explicitly skip\n>> sorting if its 'struct ref_sorting *' arg is NULL. Other functions using\n>> 'struct ref_sorting *' do not need any changes because they already properly\n>> ignore NULL values.\n>>\n>> The goal of this change is to have the '--no-sort' option truly disable\n>> sorting in commands like 'for-each-ref, 'tag', and 'branch'. Right now,\n>> '--no-sort' will still trigger refname sorting by default in 'for-each-ref',\n>> 'tag', and 'branch'.\n>>\n>> To match existing behavior as closely as possible, explicitly add \"refname\"\n>> to the list of sort keys in 'for-each-ref', 'tag', and 'branch' before\n>> parsing options (if no config-based sort keys are set). This ensures that\n>> sorting will only be fully disabled if '--no-sort' is provided as an option;\n>> otherwise, \"refname\" sorting will remain the default. Note: this also means\n>> that even when sort keys are provided on the command line, \"refname\" will be\n>> the final sort key in the sorting structure. This doesn't actually change\n>> any behavior, since 'compare_refs()' already falls back on comparing\n>> refnames if two refs are equal w.r.t all other sort keys.\n>>\n>> Finally, remove the condition around sorting in 'ls-remote', since it's no\n>> longer necessary. Unlike 'for-each-ref' et. al., it does *not* set any sort\n>> keys by default. The default empty list of sort keys will produce a NULL\n>> 'struct ref_sorting *', which causes the sorting to be skipped in\n>> 'ref_array_sort()'.\n> \n> I found the order in this commit message a bit funny because you first\n> explain what you're doing, then explain the goal, and then jump into the\n> changes again. The message might be a bit easier to read if the goal was\n> stated up front.\n\nI'll try to restructure it.\n\n> \n> I was also briefly wondering whether it would make sense to split up\n> this commit, as you're doing two different things:\n> \n>     - Refactor how git-for-each-ref(1), git-tag(1) and git-branch(1) set\n>       up their default sorting.\n> \n>     - Change `ref_array_sort()` to not sort when its sorting option is\n>       `NULL`.\n> \n> If this was split up into two commits, then the result might be a bit\n> easier to reason about. But I don't feel strongly about this.\n\nThe addition of \"refname\" to the sorting defaults really only makes sense in\nthe context of needing it to update 'ref_array_sort()', though. While you\ncan convey some of that in a commit message, when reading through commits\n(mine and others') I find it much easier to contextualize small refactors\nwith their associated behavior change if they're done in a single patch.\nThere's a limit to that, of course; even within this series I have a lot of\n\"this will make sense later\" commit messages (more than I'd like really)\nbecause the refactors are large & varied enough that they'd be overwhelming\nif squashed into a single patch.\n\nSo, while I definitely see where you're coming from, I think this patch is\nbetter off not being split.\n\n>> diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\n>> index fc765754305..436249b720c 100644\n>> --- a/builtin/ls-remote.c\n>> +++ b/builtin/ls-remote.c\n>> @@ -58,6 +58,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n>>  \tstruct transport *transport;\n>>  \tconst struct ref *ref;\n>>  \tstruct ref_array ref_array;\n>> +\tstruct ref_sorting *sorting;\n>>  \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n>>  \n>>  \tstruct option options[] = {\n>> @@ -141,13 +142,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n>>  \t\titem->symref = xstrdup_or_null(ref->symref);\n>>  \t}\n>>  \n>> -\tif (sorting_options.nr) {\n>> -\t\tstruct ref_sorting *sorting;\n>> -\n>> -\t\tsorting = ref_sorting_options(&sorting_options);\n>> -\t\tref_array_sort(sorting, &ref_array);\n>> -\t\tref_sorting_release(sorting);\n>> -\t}\n>> +\tsorting = ref_sorting_options(&sorting_options);\n>> +\tref_array_sort(sorting, &ref_array);\n> \n> We stopped calling `ref_sorting_release()`. Doesn't that cause us to\n> leak memory?\n\nNice catch, thanks! It should have been moved to the end of this function\n(right before the 'ref_array_clear()').\n\n>> diff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\n>> index 3182abde27f..9918ba05dec 100755\n>> --- a/t/t3200-branch.sh\n>> +++ b/t/t3200-branch.sh\n>> @@ -1606,10 +1608,70 @@ test_expect_success 'option override configured sort' '\n>>  \t)\n>>  '\n>>  \n>> +test_expect_success '--no-sort cancels config sort keys' '\n>> +\ttest_config -C sort branch.sort \"-refname\" &&\n>> +\n>> +\t(\n>> +\t\tcd sort &&\n>> +\n>> +\t\t# objecttype is identical for all of them, so sort falls back on\n>> +\t\t# default (ascending refname)\n>> +\t\tgit branch \\\n>> +\t\t\t--no-sort \\\n>> +\t\t\t--sort=\"objecttype\" >actual &&\n> \n> This test is a bit confusing to me. Shouldn't we in fact ignore the\n> configured sorting order as soon as we pass `--sort=` anyway? In other\n> words, I would expect the `--no-sort` option to not make a difference\n> here. What should make a difference is if you _only_ passed `--no-sort`.\n\nThe existing behavior (as demonstrated by this test) is that the command\nline sort keys append to, rather than replace, the config-based sort keys. I\ndon't see any evidence in the commit history to indicate that this was an\nintentional design decision, but it's not necessarily incorrect either.\n\nFor one, it's not universal in string list options that the command line\nreplaces the config. There are examples of both approaches to string list\noptions in other commands:\n\n- in 'git push', specifying '--push-option' on the command line even once\n  will remove any values set by 'push.pushoption'\n- in 'git blame', any values specified with '--ignore-revs-file' are\n  appended to those set by 'blame.ignorerevsfile'\n\nIn the case of 'git (tag|branch)', I can see why users might not want\ncommand line sort keys to completely remove config-based ones. The only time\nthe config-based keys will come into play is when two entries are identical\nw.r.t _all_ of the command line sort keys. In that scenario, I'd expect a\nuser would want to use their configured defaults to \"break the tie\" instead\nof the hardcoded ascending refname sort. If they do actually want to remove\nthe config keys, they can set '--no-sort' before their other sort keys.\n\n"},{"id":"484542","messageId":"ecd7b723-0ff7-412b-a332-8c011ff12f86@github.com","threadId":"60483","inReplyTo":"ZUoWT8GyrZlvH_Go@tanuki","subject":"Re: [PATCH 6/9] ref-filter.c: refactor to create common helper functions","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-07T18:41:30Z","receivedAt":"2023-11-07T18:41:32Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:\n> On Tue, Nov 07, 2023 at 01:25:58AM +0000, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye <vdye@github.com>\n>>\n>> Factor out parts of 'ref_array_push()', 'ref_filter_handler()', and\n>> 'filter_refs()' into new helper functions ('ref_array_append()',\n>> 'apply_ref_filter()', and 'do_filter_refs()' respectively), as well as\n>> rename 'ref_filter_handler()' to 'filter_one()'. In this and later\n>> patches, these helpers will be used by new ref-filter API functions. This\n>> patch does not result in any user-facing behavior changes or changes to\n>> callers outside of 'ref-filter.c'.\n>>\n>> The changes are as follows:\n>>\n>> * The logic to grow a 'struct ref_array' and append a given 'struct\n>>   ref_array_item *' to it is extracted from 'ref_array_push()' into\n>>   'ref_array_append()'.\n>> * 'ref_filter_handler()' is renamed to 'filter_one()' to more clearly\n>>   distinguish it from other ref filtering callbacks that will be added in\n>>   later patches. The \"*_one()\" naming convention is common throughout the\n>>   codebase for iteration callbacks.\n>> * The code to filter a given ref by refname & object ID then create a new\n>>   'struct ref_array_item' is moved out of 'filter_one()' and into\n>>   'apply_ref_filter()'. 'apply_ref_filter()' returns either NULL (if the ref\n>>   does not match the given filter) or a 'struct ref_array_item *' created\n>>   with 'new_ref_array_item()'; 'filter_one()' appends that item to\n>>   its ref array with 'ref_array_append()'.\n>> * The filter pre-processing, contains cache creation, and ref iteration of\n>>   'filter_refs()' is extracted into 'do_filter_refs()'. 'do_filter_refs()'\n>>   takes its ref iterator function & callback data as an input from the\n>>   caller, setting it up to be used with additional filtering callbacks in\n>>   later patches.\n> \n> To me, a bulleted list spelling out the different changes I'm doing\n> often indicates that I might want to split up the commit into one for\n> each of the items. I don't feel strongly about this, but think that it\n> might help the reviewer in this case.\n\nWhile that's a good guideline to keep in mind, it's not universally\napplicable. In this case, (almost) all of the changes are done the same way,\nfocused on the same goal: extract bits of 'filter_refs()' into generic,\ninternal helpers so we can use those bits elsewhere in later patches.\nSplitting those extractions into multiple patches would essentially lead to\na handful of very small patches that more-or-less have the same commit\nmessage. As I mentioned in [1], I think there's value to having the\nimmediate context of related changes in a single patch (as long as that\nsingle patch doesn't become unwieldy), so I'm not inclined to split this up.\n\nThat said, I did say \"(almost) all\" of the changes are conceptually similar.\nLooking at this now, the rename of 'ref_filter_handler()' => 'filter_one()'\ndoesn't really fit the \"extract into helper functions\" theme of the rest of\nthe patch, I'll pull that out into its own.\n\n[1] https://lore.kernel.org/git/a833b5a7-0201-4c2e-8821-f2a1930cb403@github.com/\n\n"},{"id":"484543","messageId":"20231107192326.48296-1-oystwa@gmail.com","threadId":"60483","inReplyTo":"88eba4146cd250fcabfb9ffa9b410ce912a82ce7.1699320362.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/9] for-each-ref: clarify interaction of --omit-empty & --count","fromName":"Øystein Walle","fromEmail":"oystwa@gmail.com","sentAt":"2023-11-07T19:23:26Z","receivedAt":"2023-11-07T19:23:56Z","isPatch":true,"sender":{"key":"oystwa@gmail.com","avatar":"https://avatars.githubusercontent.com/u/794585?v=4"},"body":"Hi Victoria,\n\nVictoria Dye <vdye@github.com> writes:\n\n> Update the 'for-each-ref' builtin documentation to clarify that refs\n> \"omitted\" by --omit-empty are still counted toward the limit specified\n> by --count. The use of the term \"omit\" would otherwise be somewhat\n> ambiguous and could incorrectly be construed as excluding empty refs\n> entirely (i.e. not counting them towards the total ref count).\n\nI implemented --omit-empty and I completely overlooked --count!\n\n(If I were to do it all over again I probably would have implemented it\nso that so-called omitted refs did not count towards the total. It makes\nsense to me since e.g.  `git log -3 -- git.c` prints the three most\nrecent commits that touch git.c regardless of how many commits were\nwalked in the process.)\n\nThis is a good and welcome clarification. \n\nAcked-by: Øystein Walle <oystwa@gmail.com>\n"},{"id":"484544","messageId":"e166abeb-3566-4acf-a252-bc493ee37f41@github.com","threadId":"60483","inReplyTo":"20231107192326.48296-1-oystwa@gmail.com","subject":"Re: [PATCH 2/9] for-each-ref: clarify interaction of --omit-empty & --count","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-07T19:30:22Z","receivedAt":"2023-11-07T19:30:24Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Øystein Walle wrote:\n> Hi Victoria,\n> \n> Victoria Dye <vdye@github.com> writes:\n> \n>> Update the 'for-each-ref' builtin documentation to clarify that refs\n>> \"omitted\" by --omit-empty are still counted toward the limit specified\n>> by --count. The use of the term \"omit\" would otherwise be somewhat\n>> ambiguous and could incorrectly be construed as excluding empty refs\n>> entirely (i.e. not counting them towards the total ref count).\n> \n> I implemented --omit-empty and I completely overlooked --count!\n> \n> (If I were to do it all over again I probably would have implemented it\n> so that so-called omitted refs did not count towards the total. It makes\n> sense to me since e.g.  `git log -3 -- git.c` prints the three most\n> recent commits that touch git.c regardless of how many commits were\n> walked in the process.)\n\nSince the interaction isn't clearly defined at the moment, we could probably\nstill update it to work like you're describing here. I'm happy to drop this\npatch and implement your recommendation in a follow-up series. Let me know\nwhat you think!\n\n> \n> This is a good and welcome clarification. \n> \n> Acked-by: Øystein Walle <oystwa@gmail.com>\n\n"},{"id":"484545","messageId":"df782f9a-2590-4c9e-a1cd-6cb6758f9248@github.com","threadId":"60483","inReplyTo":"ZUoWVPSE1GcJdHFE@tanuki","subject":"Re: [PATCH 7/9] ref-filter.c: filter & format refs in the same callback","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-07T19:45:22Z","receivedAt":"2023-11-07T19:45:24Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:\n>> diff --git a/ref-filter.c b/ref-filter.c\n>> index ff00ab4b8d8..384cf1595ff 100644\n>> --- a/ref-filter.c\n>> +++ b/ref-filter.c\n>> @@ -2863,6 +2863,44 @@ static void free_array_item(struct ref_array_item *item)\n>>  \tfree(item);\n>>  }\n>>  \n>> +struct ref_filter_and_format_cbdata {\n>> +\tstruct ref_filter *filter;\n>> +\tstruct ref_format *format;\n>> +\n>> +\tstruct ref_filter_and_format_internal {\n>> +\t\tint count;\n>> +\t} internal;\n>> +};\n>> +\n>> +static int filter_and_format_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n>> +{\n>> +\tstruct ref_filter_and_format_cbdata *ref_cbdata = cb_data;\n>> +\tstruct ref_array_item *ref;\n>> +\tstruct strbuf output = STRBUF_INIT, err = STRBUF_INIT;\n>> +\n>> +\tref = apply_ref_filter(refname, oid, flag, ref_cbdata->filter);\n>> +\tif (!ref)\n>> +\t\treturn 0;\n>> +\n>> +\tif (format_ref_array_item(ref, ref_cbdata->format, &output, &err))\n>> +\t\tdie(\"%s\", err.buf);\n>> +\n>> +\tif (output.len || !ref_cbdata->format->array_opts.omit_empty) {\n>> +\t\tfwrite(output.buf, 1, output.len, stdout);\n>> +\t\tputchar('\\n');\n>> +\t}\n>> +\n>> +\tstrbuf_release(&output);\n>> +\tstrbuf_release(&err);\n>> +\tfree_array_item(ref);\n>> +\n>> +\tif (ref_cbdata->format->array_opts.max_count &&\n>> +\t    ++ref_cbdata->internal.count >= ref_cbdata->format->array_opts.max_count)\n>> +\t\treturn -1;\n> \n> It feels a bit weird to return a negative value here, which usually\n> indicates that an error has happened whereas we only use it here to\n> abort the iteration. But we ignore the return value of\n> `do_iterate_refs()` anyway, so it doesn't make much of a difference.\n\nI'll update it to 1, and also add a comment that the non-zero return value\nstops iteration since it's not immediately clear from other 'each_ref_fn's\nwhat that means. For reference, there appears to only be one other\n'each_ref_fn' that even has the potential to return a nonzero return value\n('ref_present()' in 'refs/files-backend.c).\n\n> \n>> +\treturn 0;\n>> +}\n>> +\n>>  /* Free all memory allocated for ref_array */\n>>  void ref_array_clear(struct ref_array *array)\n>>  {\n>> @@ -3046,16 +3084,46 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n>>  \treturn ret;\n>>  }\n>>  \n>> +static inline int can_do_iterative_format(struct ref_filter *filter,\n>> +\t\t\t\t\t  struct ref_sorting *sorting,\n>> +\t\t\t\t\t  struct ref_format *format)\n>> +{\n>> +\t/*\n>> +\t * Refs can be filtered and formatted in the same iteration as long\n>> +\t * as we aren't filtering on reachability, sorting the results, or\n>> +\t * including ahead-behind information in the formatted output.\n>> +\t */\n> \n> Do we want to format this as a bulleted list so that it's more readily\n> extensible if we ever need to pay attention to new options here? Also, I\n> noted that this commit doesn't add any new tests -- do we already\n> exercise all of these conditions?\n\nSure, I'll convert it to a bulleted list. I don't really expect it to change\nmuch, though; to have any effect on this condition, the new filter/format\nwould need to act on the pre-filtered ref_array, which isn't particularly\ncommon.\n\nAnd yes, the existing tests cover scenarios where this function returns true\n(e.g. 'git for-each-ref --no-sort') & where it returns false (essentially\nanything else).\n\n> \n> More generally, I worry a bit about maintainability of this code snippet\n> as we need to remember to always update this condition whenever we add a\n> new option, and this can be quite easy to miss. The performance benefit\n> might be worth the effort though.\n\nI'll add more detailed comments to clarify what's going on here.\n\nIn practice, though, I don't think this would be all that easy to miss. As I\nnoted above, the only filters/formats that affect this are ones that need to\nloop over an entire filtered ref_array after the initial\n'for_each_fullref_in()'. To have it actually apply to commands that use\n'filter_and_format_refs()', they'll need to add that behavior here (like\n'filter_ahead_behind()'), where it should be apparent that\n'can_do_iterative_format()' is relevant to their change. \n\n> \n> Patrick\n"},{"id":"484548","messageId":"cf691b7c-288f-4cc9-a2ac-1a43972ae446@github.com","threadId":"60483","inReplyTo":"ZUoWWo7IEKsiSx-C@tanuki","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-08T01:13:52Z","receivedAt":"2023-11-08T01:13:55Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:\n> On Tue, Nov 07, 2023 at 01:26:00AM +0000, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye <vdye@github.com>\n>>\n>> Add a boolean flag '--full-deref' that, when enabled, fills '%(*fieldname)'\n>> format fields using the fully peeled target of tag objects, rather than the\n>> immediate target.\n>>\n>> In other builtins ('rev-parse', 'show-ref'), \"dereferencing\" tags typically\n>> means peeling them down to their non-tag target. Unlike these commands,\n>> 'for-each-ref' dereferences only one \"level\" of tags in '*' format fields\n>> (like \"%(*objectname)\"). For most annotated tags, one level of dereferencing\n>> is enough, since most tags point to commits or trees. However, nested tags\n>> (annotated tags whose target is another annotated tag) dereferenced once\n>> will point to their target tag, different a full peel to e.g. a commit.\n>>\n>> Currently, if a user wants to filter & format refs and include information\n>> about the fully dereferenced tag, they can do so with something like\n>> 'cat-file --batch-check':\n>>\n>>     git for-each-ref --format=\"%(objectname)^{} %(refname)\" <pattern> |\n>>         git cat-file --batch-check=\"%(objectname) %(rest)\"\n>>\n>> But the combination of commands is inefficient. So, to improve the\n>> efficiency of this use case, add a '--full-deref' option that causes\n>> 'for-each-ref' to fully dereference tags when formatting with '*' fields.\n> \n> I do wonder whether it would make sense to introduce this feature in the\n> form of a separate field prefix, as you also mentioned in your cover\n> letter. It would buy the user more flexibility, but the question is\n> whether such flexibility would really ever be needed.\n> \n> The only thing I could really think of where it might make sense is to\n> distinguish tags that peel to a commit immediately from ones that don't.\n> That feels rather esoteric to me and doesn't seem to be of much use. But\n> regardless of whether or not we can see the usefulness now, if this\n> wouldn't be significantly more complex I wonder whether it would make\n> more sense to use a new field prefix instead anyway.\n> \n> In any case, I think it would be helpful if this was discussed in the\n> commit message.\nI've been going back and forth on this, but I think a field specifier might\nbe the way to go after all. Using a field specifier would inherently be more\ncomplex than the command line option (since the formatting code is a bit\ncomplicated), but that's not an insurmountable problem. The thing I kept\ngetting caught up on was which symbol (or symbols?) to use to indicate a full\nobject peel. I mentioned `**fieldname` in the cover letter, but that looks\nmore like a double dereference than a recursive one.\n\nI think `^{}fieldname` would be a good candidate, but it's *extremely*\nimportant (for the sake of avoiding user confusion/frustration) that it\nproduces the same object & associated info as the standard revision parsing\nmachinery [1]. One notable difference (it might be the only one) from\n`*fieldname` would be, if a ref points to a non-tag object, then that\nobject's information would printed (rather than an empty string). But maybe\nthat difference is what we'd want anyway, since it's a better one-for-one\nreplacement of 'git for-each-ref | git cat-file --batch-check'.\n\nI'll try implementing that for V2. If it doesn't work for some reason,\nthough, I'll explain why in the commit message.\n\n[1] https://git-scm.com/docs/git-rev-parse#Documentation/git-rev-parse.txt-emltrevgtemegemv0998em\n\n> \n> Patrick\n> \n"},{"id":"484550","messageId":"21dfe606-39f5-4154-aaa4-695e5f6f784d@github.com","threadId":"60483","inReplyTo":"ZUoWPpFHEi-PZjoD@tanuki","subject":"Re: [PATCH 0/9] for-each-ref optimizations & usability improvements","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-08T01:31:37Z","receivedAt":"2023-11-08T01:31:40Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:\n> On Mon, Nov 06, 2023 at 06:48:29PM -0800, Victoria Dye wrote:\n>> Junio C Hamano wrote:\n>>> \"Victoria Dye via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> [snip]\n>>>>  * I'm not attached to '--full-deref' as a name - if someone has an idea for\n>>>>    a more descriptive name, please suggest it!\n>>>\n>>> Another candidate verb may be \"to peel\", and I have no strong\n>>> opinion between it and \"to dereference\".  But I have a mild aversion\n>>> to an abbreviation that is not strongly established.\n>>>\n>>\n>> Makes sense. I got the \"deref\" abbreviation for 'update-ref --no-deref', but\n>> 'show-ref' has a \"--dereference\" option and protocol v2's \"ls-refs\" includes\n>> a \"peel\" arg. \"Dereference\" is the term already used in the 'for-each-ref'\n>> documentation, though, so if no one comes in with an especially strong\n>> opinion on this I'll change the option to '--full-dereference'. Thanks!\n> \n> But doesn't dereferencing in the context of git-update-ref(1) refer to\n> something different? It's not about tags, but it is about symbolic\n> references and whether we want to update the symref or the pointee. But\n> true enough, in git-show-ref(1) \"dereference\" actually means that we\n> should peel the tag.\n\nSince both annotated tags and symbolic refs are essentially pointers, it's\nnot surprising that they both use the term \"dereference.\" Even though\n\"deref\" refers to symbolic refs in 'update-ref', its existence as an\nabbreviation for \"dereference\" is relevant when coming up with a way to\nabbreviate \"dereference\" when referring to tags.\n\n> \n> To me it feels like preexisting commands are confused already. In my\n> mind model:\n> \n>     - \"peel\" means that an object gets resolved to one of its pointees.\n>       This also includes the case here, where a tag gets peeled to its\n>       pointee.\n> \n>     - \"dereference\" means that a symbolic reference gets resolved to its\n>       pointee. This matches what we do in `git update-ref --no-deref`.\n> \n> But after reading through the code I don't think we distinguish those\n> terms cleanly throughout our codebase. Still, \"peeling\" feels like a\n> better match in my opinion.\n\nHmm. I think I mostly agree on your definition of \"peel\". In the docs, it's\nused to refer to:\n\n- recursively resolving an OID to an object of a specified type [1]\n- recursively resolving a tag OID to a non-tag object [2]\n\nNotably, there seems to be a strong association of \"peeling\" to \"recursive\nresolution\". Which means it doesn't necessarily describe what \"*\" currently\ndoes.\n\n\"Dereference\" generally seems like a looser term than what you've suggested.\nIt does refer to symbolic ref resolution as you describe [3], but \"recursive\ndereference\" is definitely also a synonym for \"peel\" [4]. That, combined\nwith the fact that \"*\" is the \"dereference operator\", leads me to believe\nthat \"%(*fieldname)\" would accurately be described as a \"tag dereference\"\nfield in the context of 'for-each-ref'.\n\nAs I mentioned in [5], I'm going to try adding this functionality with a\nfield specifier rather than a command line option, so the name of the option\nmight be moot. But, since dereferencing/peeling will still be relevant to\nthe changes, I'll make sure the terminology I use in the documentation is as\nprecise as possible (i.e., use \"peel\" where I previously used \"fully\ndereference\").\n\nSeparately, this has inspired me to revisit something I've been putting off,\nwhich is to add a definition for \"peel\" (and now probably \"dereference\" as\nwell) in 'gitglossary'. I'll try to send that out in the next couple days.\n\nThanks!\n\n[1] https://git-scm.com/docs/git-rev-parse#Documentation/git-rev-parse.txt---verify\n[2] https://git-scm.com/docs/gitprotocol-v2#_ls_refs\n[3] https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddefsymrefasymref\n[4] https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddefcommit-ishacommit-ishalsocommittish\n[5] https://lore.kernel.org/git/cf691b7c-288f-4cc9-a2ac-1a43972ae446@github.com/\n\n> \n> Patrick\n\n"},{"id":"484555","messageId":"xmqq4jhx7x8l.fsf@gitster.g","threadId":"60483","inReplyTo":"cf691b7c-288f-4cc9-a2ac-1a43972ae446@github.com","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-08T03:14:02Z","receivedAt":"2023-11-08T03:14:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye <vdye@github.com> writes:\n\n> I think `^{}fieldname` would be a good candidate, but it's *extremely*\n\nGaah.  Why?  fieldname^{} I may understand, but in the prefix form?\n\nIn any case, has anybody considered that we may be better off to\ndeclare that \"*field\" peeling a tag only once is a longstanding bug?\n\nIOW, can we not add \"fully peel\" command line option or a new syntax\nand instead just \"fix\" the bug to fully peel when \"*field\" is asked\nfor?\n\nAn application that cares about handling a chain of annotatetd tags\nwould want to be able to say \"this is the outermost tag's\ninformation; one level down, the tag was signed by this person;\nanother level down, the tag was signed by this person, etc.\"  which\nwould mean either\n\n * we have a syntax that shows the information from all levels\n   (e.g., \"**taggername\" may say \"Victoria\\nPatrick\\nGitster\")\n\n * we have a syntax that allows to specify how many levels to peel,\n   (e.g., \"*0*taggername\" may be the same as \"taggername\",\n   \"*1*taggername\" may be the same as \"*taggername\") plus some\n   programming construct like variables and loops.\n\nbut the repertoire being proposed that consists only of \"peel only\nonce\" and \"peel all levels\" is way too insufficient.\n\nNote that I do not advocate for allowing inspection of each levels\nseparately.  Quite the contrary.  I would say that --format=<>\nplaceholder should not be a programming language to satisify such a\nniche need.  And my conclusion from that stance is \"peel once\" plus\n\"peel all\" are already one level too many, and \"peel once\" was a\nvery flawed implementation from day one, when 9f613ddd (Add\ngit-for-each-ref: helper for language bindings, 2006-09-15)\nintroduced it.\n\n\n"},{"id":"484561","messageId":"ZUs2kxOEr4vqCJi0@tanuki","threadId":"60483","inReplyTo":"xmqq4jhx7x8l.fsf@gitster.g","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-08T07:19:47Z","receivedAt":"2023-11-08T07:19:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Nov 08, 2023 at 12:14:02PM +0900, Junio C Hamano wrote:\n> Victoria Dye <vdye@github.com> writes:\n> \n> > I think `^{}fieldname` would be a good candidate, but it's *extremely*\n> \n> Gaah.  Why?  fieldname^{} I may understand, but in the prefix form?\n> \n> In any case, has anybody considered that we may be better off to\n> declare that \"*field\" peeling a tag only once is a longstanding bug?\n> \n> IOW, can we not add \"fully peel\" command line option or a new syntax\n> and instead just \"fix\" the bug to fully peel when \"*field\" is asked\n> for?\n\nI see where you're coming from, but I wonder whether this wouldn't break\nscripts. To me, the documentation seems to explicitly state that this\nwill only deref tags once:\n\n    If fieldname is prefixed with an asterisk (*) and the ref points at\n    a tag object, use the value for the field in the object which the\n    tag object refers to (instead of the field in the tag object).\n\nSo changing that now would break both the documented and the actual\nbehaviour. Now whether anybody actually cares about such a breaking\nchange is of course a different question, and you're probably correct\nthat in practice nobody does.\n\nPatrick\n\n> An application that cares about handling a chain of annotatetd tags\n> would want to be able to say \"this is the outermost tag's\n> information; one level down, the tag was signed by this person;\n> another level down, the tag was signed by this person, etc.\"  which\n> would mean either\n> \n>  * we have a syntax that shows the information from all levels\n>    (e.g., \"**taggername\" may say \"Victoria\\nPatrick\\nGitster\")\n> \n>  * we have a syntax that allows to specify how many levels to peel,\n>    (e.g., \"*0*taggername\" may be the same as \"taggername\",\n>    \"*1*taggername\" may be the same as \"*taggername\") plus some\n>    programming construct like variables and loops.\n> \n> but the repertoire being proposed that consists only of \"peel only\n> once\" and \"peel all levels\" is way too insufficient.\n> \n> Note that I do not advocate for allowing inspection of each levels\n> separately.  Quite the contrary.  I would say that --format=<>\n> placeholder should not be a programming language to satisify such a\n> niche need.  And my conclusion from that stance is \"peel once\" plus\n> \"peel all\" are already one level too many, and \"peel once\" was a\n> very flawed implementation from day one, when 9f613ddd (Add\n> git-for-each-ref: helper for language bindings, 2006-09-15)\n> introduced it.\n> \n> \n"},{"id":"484567","messageId":"CAFaJEquUBYUO=scjHw2qrUyP-4wZJWtdmWAtRrW0mH9x9PbZZw@mail.gmail.com","threadId":"60483","inReplyTo":"e166abeb-3566-4acf-a252-bc493ee37f41@github.com","subject":"Re: [PATCH 2/9] for-each-ref: clarify interaction of --omit-empty & --count","fromName":"Øystein Walle","fromEmail":"oystwa@gmail.com","sentAt":"2023-11-08T07:53:57Z","receivedAt":"2023-11-08T07:54:36Z","isPatch":true,"sender":{"key":"oystwa@gmail.com","avatar":"https://avatars.githubusercontent.com/u/794585?v=4"},"body":"On Tue, 7 Nov 2023 at 20:30, Victoria Dye <vdye@github.com> wrote:\n\n> Since the interaction isn't clearly defined at the moment, we could probably\n> still update it to work like you're describing here. I'm happy to drop this\n> patch and implement your recommendation in a follow-up series. Let me know\n> what you think!\n\nRegardless of whether the logic is changed in a follow-up series or not\nI think the current behavior is worth documenting even if it doesn't\nexist for much longer in the tree. So I am favor of having this patch as\npart of this series.\n\nI am also in favor of changing the behavior but that warrants a separate\nseries and discussion.\n\nØsse\n"},{"id":"484570","messageId":"674aa6c8-ccc7-4120-a864-57cce657f6a9@app.fastmail.com","threadId":"60483","inReplyTo":"CAFaJEquUBYUO=scjHw2qrUyP-4wZJWtdmWAtRrW0mH9x9PbZZw@mail.gmail.com","subject":"Re: [PATCH 2/9] for-each-ref: clarify interaction of --omit-empty & --count","fromName":"Kristoffer Haugsbakk","fromEmail":"code@khaugsbakk.name","sentAt":"2023-11-08T10:00:59Z","receivedAt":"2023-11-08T10:01:29Z","isPatch":true,"sender":{"key":"code@khaugsbakk.name","avatar":"https://avatars.githubusercontent.com/u/2229597?v=4"},"body":"On Wed, Nov 8, 2023, at 08:53, Øystein Walle wrote:\n> On Tue, 7 Nov 2023 at 20:30, Victoria Dye <vdye@github.com> wrote:\n>\n>> Since the interaction isn't clearly defined at the moment, we could probably\n>> still update it to work like you're describing here. I'm happy to drop this\n>> patch and implement your recommendation in a follow-up series. Let me know\n>> what you think!\n>\n> Regardless of whether the logic is changed in a follow-up series or not\n> I think the current behavior is worth documenting even if it doesn't\n> exist for much longer in the tree. So I am favor of having this patch as\n> part of this series.\n\nThe funny thing though is that once it’s documented then you also kind of\ncommit yourself to it, right? That it’s how it’s supposed to behave.[1] If\nyou instead change the behavior (to the correct one) and document it in\nthe same series then there is no in-between time when people can claim to\nrely on it via the documentation.\n\n[1] Modulo “subject to change” hedging, but it seems that even\n    experimental commands who are documented as that are now resistant to\n    change in practice.\n"},{"id":"484590","messageId":"898d3850-b0ca-485e-9489-320eee3121e4@github.com","threadId":"60483","inReplyTo":"xmqq4jhx7x8l.fsf@gitster.g","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-11-08T18:02:55Z","receivedAt":"2023-11-08T18:02:58Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> Victoria Dye <vdye@github.com> writes:\n> \n>> I think `^{}fieldname` would be a good candidate, but it's *extremely*\n> \n> Gaah.  Why?  fieldname^{} I may understand, but in the prefix form?\n\n'fieldname^{}' seemed like more of a misuse of \"^{}\" than the prefixed form,\nsince we're not peeling \"fieldname\" but instead getting the value of\n\"fieldname\" from the peeled tag. But then we're not dereferencing\n\"fieldname\" in '*fieldname' either, so 'fieldname^{}' is no worse than what\nalready exists.\n\n> \n> In any case, has anybody considered that we may be better off to\n> declare that \"*field\" peeling a tag only once is a longstanding bug?\n> \n> IOW, can we not add \"fully peel\" command line option or a new syntax\n> and instead just \"fix\" the bug to fully peel when \"*field\" is asked\n> for?\n\nI'd certainly prefer that from a technical standpoint; it simplifies this\npatch if I can just replace 'get_tagged_oid' with 'peel_iterated_oid'. The\ntwo things that make me hesitate are:\n\n1. There isn't a straightforward 1:1 substitute available for getting info\n   on the immediate target of a list of tags. \n2. The performance of a recursive peel can be worse than that of a single\n   tag dereference, since (unless the formatting is done in a ref_iterator\n   iteration *and* the tag is a packed ref) the dereferenced object needs to\n   be resolved to determine whether it's another tag or not.\n\n#1 may not be an issue in practice, but I don't have enough information on\nhow applications use that formatting atom to say for sure. #2 is a bigger\nissue, IMO, since one of the goals of this series was to improve performance\nfor some cases of 'for-each-ref' without hurting it in others.\n\n> An application that cares about handling a chain of annotatetd tags\n> would want to be able to say \"this is the outermost tag's\n> information; one level down, the tag was signed by this person;\n> another level down, the tag was signed by this person, etc.\"  which\n> would mean either\n> \n>  * we have a syntax that shows the information from all levels\n>    (e.g., \"**taggername\" may say \"Victoria\\nPatrick\\nGitster\")\n> \n>  * we have a syntax that allows to specify how many levels to peel,\n>    (e.g., \"*0*taggername\" may be the same as \"taggername\",\n>    \"*1*taggername\" may be the same as \"*taggername\") plus some\n>    programming construct like variables and loops.\n> \n> but the repertoire being proposed that consists only of \"peel only\n> once\" and \"peel all levels\" is way too insufficient.\n> \n> Note that I do not advocate for allowing inspection of each levels\n> separately.  Quite the contrary.  I would say that --format=<>\n> placeholder should not be a programming language to satisify such a\n> niche need.  And my conclusion from that stance is \"peel once\" plus\n> \"peel all\" are already one level too many, and \"peel once\" was a\n> very flawed implementation from day one, when 9f613ddd (Add\n> git-for-each-ref: helper for language bindings, 2006-09-15)\n> introduced it.\n\nI can (and would like to) deprecate the \"peel once\" behavior and replace it\nwith \"peel all\", but with how long it's been around and the potential\nperformance impact, such a change should probably be clearly communicated.\nHow that happens depends on how aggressively we want to cut over. We could:\n\n1. Change the behavior of '*' from single dereference to recursive\n   dereference, make a note of it in the documentation.\n2. Same as #1, but also add an option like '--no-recursive-dereference' or\n   something to use the old behavior. Remove the option after 1-2 release\n   cycles?\n3. Add a new format specifier '^{}', note that '*' is deprecated in the\n   docs.\n4. Same as #3, but also show a warning/advice if '*' is used.\n5. Same as #3, but die() if '*' is used.\n\nI'm open to other options, those were just the first few I could think of. \n\n"},{"id":"484598","messageId":"xmqqleb73el4.fsf@gitster.g","threadId":"60483","inReplyTo":"898d3850-b0ca-485e-9489-320eee3121e4@github.com","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-09T01:22:47Z","receivedAt":"2023-11-09T01:22:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye <vdye@github.com> writes:\n\n> I can (and would like to) deprecate the \"peel once\" behavior and replace it\n> with \"peel all\", but with how long it's been around and the potential\n> performance impact, such a change should probably be clearly communicated.\n\nI've written a fairly detailed response on this about the reason why\nI think that \"leave a mention in the backward compatibility notes\nsection of the release notes\" (your #1) is sufficient, but it seems\nto have been lost in the ether.  I'll wait a bit and if the previous\nresponse does not materialize, I may type it again.\n\nBut in addition to what I wrote there, there is this thread [*] from\n2019 that indicates that our position is to mildly discourage\ntag-to-tag in the first place.\n\n\n[Reference]\n\n* https://lore.kernel.org/git/20190404020226.GG4409@sigill.intra.peff.net/\n"},{"id":"484599","messageId":"xmqqfs1f3eji.fsf@gitster.g","threadId":"60483","inReplyTo":"898d3850-b0ca-485e-9489-320eee3121e4@github.com","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-09T01:23:45Z","receivedAt":"2023-11-09T01:23:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye <vdye@github.com> writes:\n\n> I'd certainly prefer that from a technical standpoint; it simplifies this\n> patch if I can just replace 'get_tagged_oid' with 'peel_iterated_oid'. The\n> two things that make me hesitate are:\n>\n> 1. There isn't a straightforward 1:1 substitute available for getting info\n>    on the immediate target of a list of tags. \n> 2. The performance of a recursive peel can be worse than that of a single\n>    tag dereference, since (unless the formatting is done in a ref_iterator\n>    iteration *and* the tag is a packed ref) the dereferenced object needs to\n>    be resolved to determine whether it's another tag or not.\n>\n> #1 may not be an issue in practice, but I don't have enough information on\n> how applications use that formatting atom to say for sure. #2 is a bigger\n> issue, IMO, since one of the goals of this series was to improve performance\n> for some cases of 'for-each-ref' without hurting it in others.\n\nIn a repository without any tag-to-tag at tips of refs, would #2\nabove still be an issue?  My assumption when I raised \"isn't this\nsimply a bug?\" question was that the use of tag-to-tag is a mere\nintellectual curiosity, there is no serious use case, and they are\nnot heavily used.  Hence I was envisioning that #1 below (i.e., a\nmention in the Release Notes' backward compatibility notes section)\nwould be sufficient.\n\nIf it weren't the case, then I do not think any \"transition\" would\nwork, either.\n\nAnd stepping back a bit, even though \"peel only once\" is how\nfor-each-ref works, I do not think anybody who really cares about\ntag-to-tag and inspecting each level of peeled tag is helped by it\nall that much.  Yes, you can get the result of single level peeling\nvia \"git format-patch --format=%(*objectname)\", but then what would\nyou do to dig further from that point?  You cannot ask rev-parse to\npeel the result with \"^{}\", as that will peel all the way down.\n\nYou have to feed it to \"git cat-file tag\" and parse the contents of\nthe tag obbject yourself to manually peel further levels of onion.\nAnybody who do care must already have such a machinery, and such a\nmachinery does not depend on \"git for-each-ref --format='%(*field)'\"\npeeling just once, I would say.  They would most likely learn the\n\"%(objectname) %(objecttype) %(refname)\" from the command, and for\nthose that are tags, they would manually peel the object with such a\nmachinery, because they have to do that for second and further\nlevels anyway.\n\nAnd that is why I am not so worried about \"breaking\" existing users\nin this particular case.  Our existing support with tag-to-tag is so\npoor that those who truly need it would have invented necessary\nsupport without relying on for-each-ref's peeling (if such people\ndid exist, that is).\n\nBut perhaps I am so overly optimistic against Hyrum's law.\n\n> I can (and would like to) deprecate the \"peel once\" behavior and replace it\n> with \"peel all\", but with how long it's been around and the potential\n> performance impact, such a change should probably be clearly communicated.\n> How that happens depends on how aggressively we want to cut over. We could:\n>\n> 1. Change the behavior of '*' from single dereference to recursive\n>    dereference, make a note of it in the documentation.\n> 2. Same as #1, but also add an option like '--no-recursive-dereference' or\n>    something to use the old behavior. Remove the option after 1-2 release\n>    cycles?\n> 3. Add a new format specifier '^{}', note that '*' is deprecated in the\n>    docs.\n> 4. Same as #3, but also show a warning/advice if '*' is used.\n> 5. Same as #3, but die() if '*' is used.\n>\n> I'm open to other options, those were just the first few I could think of. \n"},{"id":"484600","messageId":"xmqq5y2b3e5p.fsf@gitster.g","threadId":"60483","inReplyTo":"xmqqfs1f3eji.fsf@gitster.g","subject":"Re: [PATCH 8/9] for-each-ref: add option to fully dereference tags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-09T01:32:02Z","receivedAt":"2023-11-09T01:32:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> You have to feed it to \"git cat-file tag\" and parse the contents of\n> the tag obbject yourself to manually peel further levels of onion.\n\nAlternatively, you can drive \"git show -s\" with \"--format\" and you\nprobably can produce a machine parseable output.\n\nBut it does not change the argument fundamentally.  The point is\nthat \"for-each-ref --format=%(*field)\" that peels only the first\nlayer would not have helped all that much, if somebody really cares\nabout each levels of nested tags.  They would have been relying on\na solution to deal with the second and further layers anyway, and\nthat solution would have been working with the first layer, too.\n\n"},{"id":"484872","messageId":"074da1ff3e85927324c42a3fa65e4239f051cd70.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 01/10] ref-filter.c: really don't sort when using --no-sort","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:49Z","receivedAt":"2023-11-14T19:54:06Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nWhen '--no-sort' is passed to 'for-each-ref', 'tag', and 'branch', the\nprinted refs are still sorted by ascending refname. Change the handling of\nsort options in these commands so that '--no-sort' to truly disables\nsorting.\n\n'--no-sort' does not disable sorting in these commands is because their\noption parsing does not distinguish between \"the absence of '--sort'\"\n(and/or values for tag.sort & branch.sort) and '--no-sort'. Both result in\nan empty 'sorting_options' string list, which is parsed by\n'ref_sorting_options()' to create the 'struct ref_sorting *' for the\ncommand. If the string list is empty, 'ref_sorting_options()' interprets\nthat as \"the absence of '--sort'\" and returns the default ref sorting\nstructure (equivalent to \"refname\" sort).\n\nTo handle '--no-sort' properly while preserving the \"refname\" sort in the\n\"absence of --sort'\" case, first explicitly add \"refname\" to the string list\n*before* parsing options. This alone doesn't actually change any behavior,\nsince 'compare_refs()' already falls back on comparing refnames if two refs\nare equal w.r.t all other sort keys.\n\nNow that the string list is populated by default, '--no-sort' is the only\nway to empty the 'sorting_options' string list. Update\n'ref_sorting_options()' to return a NULL 'struct ref_sorting *' if the\nstring list is empty, and add a condition to 'ref_array_sort()' to skip the\nsort altogether if the sort structure is NULL. Note that other functions\nusing 'struct ref_sorting *' do not need any changes because they already\nignore NULL values.\n\nFinally, remove the condition around sorting in 'ls-remote', since it's no\nlonger necessary. Unlike 'for-each-ref' et. al., it does *not* do any\nsorting by default. This default is preserved by simply leaving its sort key\nstring list empty before parsing options; if no additional sort keys are\nset, 'struct ref_sorting *' is NULL and sorting is skipped.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n builtin/branch.c        |  6 ++++\n builtin/for-each-ref.c  |  3 ++\n builtin/ls-remote.c     | 11 +++----\n builtin/tag.c           |  6 ++++\n ref-filter.c            | 19 ++----------\n t/t3200-branch.sh       | 68 +++++++++++++++++++++++++++++++++++++++--\n t/t6300-for-each-ref.sh | 21 +++++++++++++\n t/t7004-tag.sh          | 45 +++++++++++++++++++++++++++\n 8 files changed, 153 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex e7ee9bd0f15..d67738bbcaa 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -767,7 +767,13 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_branch_usage, options);\n \n+\t/*\n+\t * Try to set sort keys from config. If config does not set any,\n+\t * fall back on default (refname) sorting.\n+\t */\n \tgit_config(git_branch_config, &sorting_options);\n+\tif (!sorting_options.nr)\n+\t\tstring_list_append(&sorting_options, \"refname\");\n \n \ttrack = git_branch_track;\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 350bfa6e811..93b370f550b 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -67,6 +67,9 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\t/* Set default (refname) sorting */\n+\tstring_list_append(&sorting_options, \"refname\");\n+\n \tparse_options(argc, argv, prefix, opts, for_each_ref_usage, 0);\n \tif (maxcount < 0) {\n \t\terror(\"invalid --count argument: `%d'\", maxcount);\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex fc765754305..b416602b4d3 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -58,6 +58,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tstruct transport *transport;\n \tconst struct ref *ref;\n \tstruct ref_array ref_array;\n+\tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \n \tstruct option options[] = {\n@@ -141,13 +142,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \t\titem->symref = xstrdup_or_null(ref->symref);\n \t}\n \n-\tif (sorting_options.nr) {\n-\t\tstruct ref_sorting *sorting;\n-\n-\t\tsorting = ref_sorting_options(&sorting_options);\n-\t\tref_array_sort(sorting, &ref_array);\n-\t\tref_sorting_release(sorting);\n-\t}\n+\tsorting = ref_sorting_options(&sorting_options);\n+\tref_array_sort(sorting, &ref_array);\n \n \tfor (i = 0; i < ref_array.nr; i++) {\n \t\tconst struct ref_array_item *ref = ref_array.items[i];\n@@ -157,6 +153,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \t\tstatus = 0; /* we found something */\n \t}\n \n+\tref_sorting_release(sorting);\n \tref_array_clear(&ref_array);\n \tif (transport_disconnect(transport))\n \t\tstatus = 1;\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 3918eacbb57..64f3196cd4c 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -501,7 +501,13 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \n \tsetup_ref_filter_porcelain_msg();\n \n+\t/*\n+\t * Try to set sort keys from config. If config does not set any,\n+\t * fall back on default (refname) sorting.\n+\t */\n \tgit_config(git_tag_config, &sorting_options);\n+\tif (!sorting_options.nr)\n+\t\tstring_list_append(&sorting_options, \"refname\");\n \n \tmemset(&opt, 0, sizeof(opt));\n \tfilter.lines = -1;\ndiff --git a/ref-filter.c b/ref-filter.c\nindex e4d3510e28e..7250089b7c6 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -3142,7 +3142,8 @@ void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting,\n \n void ref_array_sort(struct ref_sorting *sorting, struct ref_array *array)\n {\n-\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n+\tif (sorting)\n+\t\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n }\n \n static void append_literal(const char *cp, const char *ep, struct ref_formatting_state *state)\n@@ -3248,18 +3249,6 @@ static int parse_sorting_atom(const char *atom)\n \treturn res;\n }\n \n-/*  If no sorting option is given, use refname to sort as default */\n-static struct ref_sorting *ref_default_sorting(void)\n-{\n-\tstatic const char cstr_name[] = \"refname\";\n-\n-\tstruct ref_sorting *sorting = xcalloc(1, sizeof(*sorting));\n-\n-\tsorting->next = NULL;\n-\tsorting->atom = parse_sorting_atom(cstr_name);\n-\treturn sorting;\n-}\n-\n static void parse_ref_sorting(struct ref_sorting **sorting_tail, const char *arg)\n {\n \tstruct ref_sorting *s;\n@@ -3283,9 +3272,7 @@ struct ref_sorting *ref_sorting_options(struct string_list *options)\n \tstruct string_list_item *item;\n \tstruct ref_sorting *sorting = NULL, **tail = &sorting;\n \n-\tif (!options->nr) {\n-\t\tsorting = ref_default_sorting();\n-\t} else {\n+\tif (options->nr) {\n \t\tfor_each_string_list_item(item, options)\n \t\t\tparse_ref_sorting(tail, item->string);\n \t}\ndiff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\nindex 3182abde27f..9918ba05dec 100755\n--- a/t/t3200-branch.sh\n+++ b/t/t3200-branch.sh\n@@ -1570,9 +1570,10 @@ test_expect_success 'tracking with unexpected .fetch refspec' '\n \n test_expect_success 'configured committerdate sort' '\n \tgit init -b main sort &&\n+\ttest_config -C sort branch.sort \"committerdate\" &&\n+\n \t(\n \t\tcd sort &&\n-\t\tgit config branch.sort committerdate &&\n \t\ttest_commit initial &&\n \t\tgit checkout -b a &&\n \t\ttest_commit a &&\n@@ -1592,9 +1593,10 @@ test_expect_success 'configured committerdate sort' '\n '\n \n test_expect_success 'option override configured sort' '\n+\ttest_config -C sort branch.sort \"committerdate\" &&\n+\n \t(\n \t\tcd sort &&\n-\t\tgit config branch.sort committerdate &&\n \t\tgit branch --sort=refname >actual &&\n \t\tcat >expect <<-\\EOF &&\n \t\t  a\n@@ -1606,10 +1608,70 @@ test_expect_success 'option override configured sort' '\n \t)\n '\n \n+test_expect_success '--no-sort cancels config sort keys' '\n+\ttest_config -C sort branch.sort \"-refname\" &&\n+\n+\t(\n+\t\tcd sort &&\n+\n+\t\t# objecttype is identical for all of them, so sort falls back on\n+\t\t# default (ascending refname)\n+\t\tgit branch \\\n+\t\t\t--no-sort \\\n+\t\t\t--sort=\"objecttype\" >actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\t  a\n+\t\t* b\n+\t\t  c\n+\t\t  main\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+\n+'\n+\n+test_expect_success '--no-sort cancels command line sort keys' '\n+\t(\n+\t\tcd sort &&\n+\n+\t\t# objecttype is identical for all of them, so sort falls back on\n+\t\t# default (ascending refname)\n+\t\tgit branch \\\n+\t\t\t--sort=\"-refname\" \\\n+\t\t\t--no-sort \\\n+\t\t\t--sort=\"objecttype\" >actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\t  a\n+\t\t* b\n+\t\t  c\n+\t\t  main\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success '--no-sort without subsequent --sort prints expected branches' '\n+\t(\n+\t\tcd sort &&\n+\n+\t\t# Sort the results with `sort` for a consistent comparison\n+\t\t# against expected\n+\t\tgit branch --no-sort | sort >actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\t  a\n+\t\t  c\n+\t\t  main\n+\t\t* b\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'invalid sort parameter in configuration' '\n+\ttest_config -C sort branch.sort \"v:notvalid\" &&\n+\n \t(\n \t\tcd sort &&\n-\t\tgit config branch.sort \"v:notvalid\" &&\n \n \t\t# this works in the \"listing\" mode, so bad sort key\n \t\t# is a dying offence.\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 00a060df0b5..0613e5e3623 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1335,6 +1335,27 @@ test_expect_success '--no-sort cancels the previous sort keys' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success '--no-sort without subsequent --sort prints expected refs' '\n+\tcat >expected <<-\\EOF &&\n+\trefs/tags/multi-ref1-100000-user1\n+\trefs/tags/multi-ref1-100000-user2\n+\trefs/tags/multi-ref1-200000-user1\n+\trefs/tags/multi-ref1-200000-user2\n+\trefs/tags/multi-ref2-100000-user1\n+\trefs/tags/multi-ref2-100000-user2\n+\trefs/tags/multi-ref2-200000-user1\n+\trefs/tags/multi-ref2-200000-user2\n+\tEOF\n+\n+\t# Sort the results with `sort` for a consistent comparison against\n+\t# expected\n+\tgit for-each-ref \\\n+\t\t--format=\"%(refname)\" \\\n+\t\t--no-sort \\\n+\t\t\"refs/tags/multi-*\" | sort >actual &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'do not dereference NULL upon %(HEAD) on unborn branch' '\n \ttest_when_finished \"git checkout main\" &&\n \tgit for-each-ref --format=\"%(HEAD) %(refname:short)\" refs/heads/ >actual &&\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex e689db42929..b41a47eb943 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -1862,6 +1862,51 @@ test_expect_success 'option override configured sort' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success '--no-sort cancels config sort keys' '\n+\ttest_config tag.sort \"-refname\" &&\n+\n+\t# objecttype is identical for all of them, so sort falls back on\n+\t# default (ascending refname)\n+\tgit tag -l \\\n+\t\t--no-sort \\\n+\t\t--sort=\"objecttype\" \\\n+\t\t\"foo*\" >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tfoo1.10\n+\tfoo1.3\n+\tfoo1.6\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--no-sort cancels command line sort keys' '\n+\t# objecttype is identical for all of them, so sort falls back on\n+\t# default (ascending refname)\n+\tgit tag -l \\\n+\t\t--sort=\"-refname\" \\\n+\t\t--no-sort \\\n+\t\t--sort=\"objecttype\" \\\n+\t\t\"foo*\" >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tfoo1.10\n+\tfoo1.3\n+\tfoo1.6\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--no-sort without subsequent --sort prints expected tags' '\n+\t# Sort the results with `sort` for a consistent comparison against\n+\t# expected\n+\tgit tag -l --no-sort \"foo*\" | sort >actual &&\n+\tcat >expect <<-\\EOF &&\n+\tfoo1.10\n+\tfoo1.3\n+\tfoo1.6\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'invalid sort parameter on command line' '\n \ttest_must_fail git tag -l --sort=notvalid \"foo*\" >actual\n '\n-- \ngitgitgadget\n\n"},{"id":"484873","messageId":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.git.1699320361.gitgitgadget@gmail.com","subject":"[PATCH v2 00/10] for-each-ref optimizations & usability improvements","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:48Z","receivedAt":"2023-11-14T19:54:07Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series is a bit of an informal follow-up to [1], adding some more\nsubstantial optimizations and usability fixes around ref\nfiltering/formatting. Some of the changes here affect user-facing behavior,\nsome are internal-only, but they're all interdependent enough to warrant\nputting them together in one series.\n\n[1]\nhttps://lore.kernel.org/git/pull.1594.v2.git.1696888736.gitgitgadget@gmail.com/\n\nPatch 1 changes the behavior of the '--no-sort' option in 'for-each-ref',\n'tag', and 'branch'. Currently, it just removes previous sort keys and, if\nno further keys are specified, falls back on ascending refname sort (which,\nIMO, makes the name '--no-sort' somewhat misleading). Now, '--no-sort'\ncompletely disables sorting (unless subsequent '--sort' options are\nprovided).\n\nPatches 2-7 incrementally refactor various parts of the ref\nfiltering/formatting workflows in order to create a\n'filter_and_format_refs()' function. If certain conditions are met (sorting\ndisabled, no reachability filtering or ahead-behind formatting), ref\nfiltering & formatting is done within a single 'for_each_fullref_in'\ncallback. Especially in large repositories, this makes a huge difference in\nmemory usage & runtime for certain usages of 'for-each-ref', since it's no\nlonger writing everything to a 'struct ref_array' then repeatedly whittling\ndown/updating its contents.\n\nPatch 8 updates the 'for-each-ref' documentation, making the '--format'\ndescription a bit less jumbled and more clearly explaining the '*' prefix\n(to be updated in the next patch)\n\nPatch 9 changes the dereferencing done by the '*' format prefix from a\nsingle dereference to a recursive peel. See [1] + replies for the discussion\nthat led to this approach (as opposed to a command line option or new format\nspecifier).\n\n[1] https://lore.kernel.org/git/ZUoWWo7IEKsiSx-C@tanuki/\n\nFinally, patch 10 adds performance tests for 'for-each-ref', showing the\neffects of optimizations made throughout the series. Here are some sample\nresults from my Ubuntu VM (test names shortened for space):\n\nTest                                                         HEAD\n----------------------------------------------------------------------------\n6300.2: (loose)                                              4.68(0.98+3.64)\n6300.3: (loose, no sort)                                     4.65(0.91+3.67)\n6300.4: (loose, --count=1)                                   4.50(0.84+3.60)\n6300.5: (loose, --count=1, no sort)                          4.24(0.46+3.71)\n6300.6: (loose, tags)                                        2.41(0.45+1.93)\n6300.7: (loose, tags, no sort)                               2.33(0.48+1.83)\n6300.8: (loose, tags, dereferenced)                          3.65(1.66+1.95)\n6300.9: (loose, tags, dereferenced, no sort)                 3.48(1.59+1.87)\n6300.10: for-each-ref + cat-file (loose, tags)               4.48(2.27+2.22)\n6300.12: (packed)                                            0.90(0.68+0.18)\n6300.13: (packed, no sort)                                   0.71(0.55+0.06)\n6300.14: (packed, --count=1)                                 0.77(0.52+0.16)\n6300.15: (packed, --count=1, no sort)                        0.03(0.01+0.02)\n6300.16: (packed, tags)                                      0.45(0.33+0.10)\n6300.17: (packed, tags, no sort)                             0.39(0.33+0.03)\n6300.18: (packed, tags, dereferenced)                        1.83(1.67+0.10)\n6300.19: (packed, tags, dereferenced, no sort)               1.42(1.28+0.08)\n6300.20: for-each-ref + cat-file (packed, tags)              2.36(2.11+0.29)\n\n\n * Victoria\n\n\nChanges since V1\n================\n\n * Restructured commit message of patch 1 for better readability\n * Re-added 'ref_sorting_release(sorting)' to 'ls-remote'\n * Dropped patch 2 so we don't commit to behavior we don't want in\n   'for-each-ref --omit-empty --count'\n * Split patch 6 into one that renames 'ref_filter_handler()' to\n   'filter_one()' and another that creates helper functions from existing\n   code\n * Added/updated code comments in patch 7, changed ref iteration \"break\"\n   return value from -1 to 1\n * Added a patch to reword 'for-each-ref' documentation in anticipation of\n   updating the description of what '*' does in the format\n * Removed command-line option '--full-deref' for peeling tags in '*' format\n   fields in favor of simply cutting over from the current single\n   dereference to recursive dereference in all cases. Updated tests to match\n   new behavior.\n * Added the '--count=1' tests back to p6300 (I must have unintentionally\n   removed them before submitting V1)\n\nVictoria Dye (10):\n  ref-filter.c: really don't sort when using --no-sort\n  ref-filter.h: add max_count and omit_empty to ref_format\n  ref-filter.h: move contains caches into filter\n  ref-filter.h: add functions for filter/format & format-only\n  ref-filter.c: rename 'ref_filter_handler()' to 'filter_one()'\n  ref-filter.c: refactor to create common helper functions\n  ref-filter.c: filter & format refs in the same callback\n  for-each-ref: clean up documentation of --format\n  ref-filter.c: use peeled tag for '*' format fields\n  t/perf: add perf tests for for-each-ref\n\n Documentation/git-for-each-ref.txt |  23 +--\n builtin/branch.c                   |  42 +++--\n builtin/for-each-ref.c             |  39 +----\n builtin/ls-remote.c                |  11 +-\n builtin/tag.c                      |  32 +---\n ref-filter.c                       | 272 ++++++++++++++++++++---------\n ref-filter.h                       |  25 +++\n t/perf/p6300-for-each-ref.sh       |  87 +++++++++\n t/t3200-branch.sh                  |  68 +++++++-\n t/t6300-for-each-ref.sh            |  43 +++++\n t/t6302-for-each-ref-filter.sh     |   4 +-\n t/t7004-tag.sh                     |  45 +++++\n 12 files changed, 517 insertions(+), 174 deletions(-)\n create mode 100755 t/perf/p6300-for-each-ref.sh\n\n\nbase-commit: bc5204569f7db44d22477485afd52ea410d83743\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1609%2Fvdye%2Fvdye%2Ffor-each-ref-optimizations-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1609/vdye/vdye/for-each-ref-optimizations-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1609\n\nRange-diff vs v1:\n\n  1:  dea8d7d1e86 !  1:  074da1ff3e8 ref-filter.c: really don't sort when using --no-sort\n     @@ Metadata\n       ## Commit message ##\n          ref-filter.c: really don't sort when using --no-sort\n      \n     -    Update 'ref_sorting_options()' to return a NULL 'struct ref_sorting *' if\n     -    the string list provided to it is empty, rather than returning the default\n     -    refname sort structure. Also update 'ref_array_sort()' to explicitly skip\n     -    sorting if its 'struct ref_sorting *' arg is NULL. Other functions using\n     -    'struct ref_sorting *' do not need any changes because they already properly\n     -    ignore NULL values.\n     +    When '--no-sort' is passed to 'for-each-ref', 'tag', and 'branch', the\n     +    printed refs are still sorted by ascending refname. Change the handling of\n     +    sort options in these commands so that '--no-sort' to truly disables\n     +    sorting.\n     +\n     +    '--no-sort' does not disable sorting in these commands is because their\n     +    option parsing does not distinguish between \"the absence of '--sort'\"\n     +    (and/or values for tag.sort & branch.sort) and '--no-sort'. Both result in\n     +    an empty 'sorting_options' string list, which is parsed by\n     +    'ref_sorting_options()' to create the 'struct ref_sorting *' for the\n     +    command. If the string list is empty, 'ref_sorting_options()' interprets\n     +    that as \"the absence of '--sort'\" and returns the default ref sorting\n     +    structure (equivalent to \"refname\" sort).\n      \n     -    The goal of this change is to have the '--no-sort' option truly disable\n     -    sorting in commands like 'for-each-ref, 'tag', and 'branch'. Right now,\n     -    '--no-sort' will still trigger refname sorting by default in 'for-each-ref',\n     -    'tag', and 'branch'.\n     +    To handle '--no-sort' properly while preserving the \"refname\" sort in the\n     +    \"absence of --sort'\" case, first explicitly add \"refname\" to the string list\n     +    *before* parsing options. This alone doesn't actually change any behavior,\n     +    since 'compare_refs()' already falls back on comparing refnames if two refs\n     +    are equal w.r.t all other sort keys.\n      \n     -    To match existing behavior as closely as possible, explicitly add \"refname\"\n     -    to the list of sort keys in 'for-each-ref', 'tag', and 'branch' before\n     -    parsing options (if no config-based sort keys are set). This ensures that\n     -    sorting will only be fully disabled if '--no-sort' is provided as an option;\n     -    otherwise, \"refname\" sorting will remain the default. Note: this also means\n     -    that even when sort keys are provided on the command line, \"refname\" will be\n     -    the final sort key in the sorting structure. This doesn't actually change\n     -    any behavior, since 'compare_refs()' already falls back on comparing\n     -    refnames if two refs are equal w.r.t all other sort keys.\n     +    Now that the string list is populated by default, '--no-sort' is the only\n     +    way to empty the 'sorting_options' string list. Update\n     +    'ref_sorting_options()' to return a NULL 'struct ref_sorting *' if the\n     +    string list is empty, and add a condition to 'ref_array_sort()' to skip the\n     +    sort altogether if the sort structure is NULL. Note that other functions\n     +    using 'struct ref_sorting *' do not need any changes because they already\n     +    ignore NULL values.\n      \n          Finally, remove the condition around sorting in 'ls-remote', since it's no\n     -    longer necessary. Unlike 'for-each-ref' et. al., it does *not* set any sort\n     -    keys by default. The default empty list of sort keys will produce a NULL\n     -    'struct ref_sorting *', which causes the sorting to be skipped in\n     -    'ref_array_sort()'.\n     +    longer necessary. Unlike 'for-each-ref' et. al., it does *not* do any\n     +    sorting by default. This default is preserved by simply leaving its sort key\n     +    string list empty before parsing options; if no additional sort keys are\n     +    set, 'struct ref_sorting *' is NULL and sorting is skipped.\n      \n          Signed-off-by: Victoria Dye <vdye@github.com>\n      \n     @@ builtin/ls-remote.c: int cmd_ls_remote(int argc, const char **argv, const char *\n       \n       \tfor (i = 0; i < ref_array.nr; i++) {\n       \t\tconst struct ref_array_item *ref = ref_array.items[i];\n     +@@ builtin/ls-remote.c: int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n     + \t\tstatus = 0; /* we found something */\n     + \t}\n     + \n     ++\tref_sorting_release(sorting);\n     + \tref_array_clear(&ref_array);\n     + \tif (transport_disconnect(transport))\n     + \t\tstatus = 1;\n      \n       ## builtin/tag.c ##\n      @@ builtin/tag.c: int cmd_tag(int argc, const char **argv, const char *prefix)\n  2:  88eba4146cd <  -:  ----------- for-each-ref: clarify interaction of --omit-empty & --count\n  3:  2e2f9738205 =  2:  adac101bc60 ref-filter.h: add max_count and omit_empty to ref_format\n  4:  6c66445ee31 !  3:  f44c4b42c93 ref-filter.h: move contains caches into filter\n     @@ Commit message\n          filter struct will support, so they are updated to be internally accessible\n          wherever the filter is used.\n      \n     -    The design used here is mirrors what was introduced in 576de3d956\n     +    The design used here mirrors what was introduced in 576de3d956\n          (unpack_trees: start splitting internal fields from public API, 2023-02-27)\n          for 'unpack_trees_options'.\n      \n  5:  f5be57eea7d =  4:  187b1d6610f ref-filter.h: add functions for filter/format & format-only\n  -:  ----------- >  5:  040d291ca45 ref-filter.c: rename 'ref_filter_handler()' to 'filter_one()'\n  6:  8c77452e5dd !  6:  633c0c74c2e ref-filter.c: refactor to create common helper functions\n     @@ Commit message\n          ref-filter.c: refactor to create common helper functions\n      \n          Factor out parts of 'ref_array_push()', 'ref_filter_handler()', and\n     -    'filter_refs()' into new helper functions ('ref_array_append()',\n     -    'apply_ref_filter()', and 'do_filter_refs()' respectively), as well as\n     -    rename 'ref_filter_handler()' to 'filter_one()'. In this and later\n     -    patches, these helpers will be used by new ref-filter API functions. This\n     -    patch does not result in any user-facing behavior changes or changes to\n     -    callers outside of 'ref-filter.c'.\n     +    'filter_refs()' into new helper functions:\n      \n     -    The changes are as follows:\n     +    * Extract the code to grow a 'struct ref_array' and append a given 'struct\n     +      ref_array_item *' to it from 'ref_array_push()' into 'ref_array_append()'.\n     +    * Extract the code to filter a given ref by refname & object ID then create\n     +      a new 'struct ref_array_item *' from 'filter_one()' into\n     +      'apply_ref_filter()'.\n     +    * Extract the code for filter pre-processing, contains cache creation, and\n     +      ref iteration from 'filter_refs()' into 'do_filter_refs()'.\n      \n     -    * The logic to grow a 'struct ref_array' and append a given 'struct\n     -      ref_array_item *' to it is extracted from 'ref_array_push()' into\n     -      'ref_array_append()'.\n     -    * 'ref_filter_handler()' is renamed to 'filter_one()' to more clearly\n     -      distinguish it from other ref filtering callbacks that will be added in\n     -      later patches. The \"*_one()\" naming convention is common throughout the\n     -      codebase for iteration callbacks.\n     -    * The code to filter a given ref by refname & object ID then create a new\n     -      'struct ref_array_item' is moved out of 'filter_one()' and into\n     -      'apply_ref_filter()'. 'apply_ref_filter()' returns either NULL (if the ref\n     -      does not match the given filter) or a 'struct ref_array_item *' created\n     -      with 'new_ref_array_item()'; 'filter_one()' appends that item to\n     -      its ref array with 'ref_array_append()'.\n     -    * The filter pre-processing, contains cache creation, and ref iteration of\n     -      'filter_refs()' is extracted into 'do_filter_refs()'. 'do_filter_refs()'\n     -      takes its ref iterator function & callback data as an input from the\n     -      caller, setting it up to be used with additional filtering callbacks in\n     -      later patches.\n     +    In later patches, these helpers will be used by new ref-filter API\n     +    functions. This patch does not result in any user-facing behavior changes or\n     +    changes to callers outside of 'ref-filter.c'.\n      \n          Signed-off-by: Victoria Dye <vdye@github.com>\n      \n     @@ ref-filter.c: static int filter_ref_kind(struct ref_filter *filter, const char *\n      - * A call-back given to for_each_ref().  Filter refs and keep them for\n      - * later object processing.\n      - */\n     --static int ref_filter_handler(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n     +-static int filter_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n      +static struct ref_array_item *apply_ref_filter(const char *refname, const struct object_id *oid,\n      +\t\t\t    int flag, struct ref_filter *filter)\n       {\n     @@ ref-filter.c: static int filter_ref_kind(struct ref_filter *filter, const char *\n       \n       \t/*\n       \t * A merge filter is applied on refs pointing to commits. Hence\n     -@@ ref-filter.c: static int ref_filter_handler(const char *refname, const struct object_id *oid,\n     +@@ ref-filter.c: static int filter_one(const char *refname, const struct object_id *oid, int flag\n       \t    filter->with_commit || filter->no_commit || filter->verbose) {\n       \t\tcommit = lookup_commit_reference_gently(the_repository, oid, 1);\n       \t\tif (!commit)\n     @@ ref-filter.c: static int ref_filter_handler(const char *refname, const struct ob\n       \t}\n       \n       \t/*\n     -@@ ref-filter.c: static int ref_filter_handler(const char *refname, const struct object_id *oid,\n     +@@ ref-filter.c: static int filter_one(const char *refname, const struct object_id *oid, int flag\n       \t * to do its job and the resulting list may yet to be pruned\n       \t * by maxcount logic.\n       \t */\n     @@ ref-filter.c: int filter_refs(struct ref_array *array, struct ref_filter *filter\n       \t\t * of filter_ref_kind().\n       \t\t */\n       \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n     --\t\t\tret = for_each_fullref_in(\"refs/heads/\", ref_filter_handler, &ref_cbdata);\n     +-\t\t\tret = for_each_fullref_in(\"refs/heads/\", filter_one, &ref_cbdata);\n      +\t\t\tret = for_each_fullref_in(\"refs/heads/\", fn, cb_data);\n       \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n     --\t\t\tret = for_each_fullref_in(\"refs/remotes/\", ref_filter_handler, &ref_cbdata);\n     +-\t\t\tret = for_each_fullref_in(\"refs/remotes/\", filter_one, &ref_cbdata);\n      +\t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n       \t\telse if (filter->kind == FILTER_REFS_TAGS)\n     --\t\t\tret = for_each_fullref_in(\"refs/tags/\", ref_filter_handler, &ref_cbdata);\n     +-\t\t\tret = for_each_fullref_in(\"refs/tags/\", filter_one, &ref_cbdata);\n      +\t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n       \t\telse if (filter->kind & FILTER_REFS_ALL)\n     --\t\t\tret = for_each_fullref_in_pattern(filter, ref_filter_handler, &ref_cbdata);\n     +-\t\t\tret = for_each_fullref_in_pattern(filter, filter_one, &ref_cbdata);\n      +\t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n       \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n     --\t\t\thead_ref(ref_filter_handler, &ref_cbdata);\n     +-\t\t\thead_ref(filter_one, &ref_cbdata);\n      +\t\t\thead_ref(fn, cb_data);\n       \t}\n       \n  7:  84db440896c !  7:  91a77c1a834 ref-filter.c: filter & format refs in the same callback\n     @@ ref-filter.c: static void free_array_item(struct ref_array_item *item)\n      +\tstrbuf_release(&err);\n      +\tfree_array_item(ref);\n      +\n     ++\t/*\n     ++\t * Increment the running count of refs that match the filter. If\n     ++\t * max_count is set and we've reached the max, stop the ref\n     ++\t * iteration by returning a nonzero value.\n     ++\t */\n      +\tif (ref_cbdata->format->array_opts.max_count &&\n      +\t    ++ref_cbdata->internal.count >= ref_cbdata->format->array_opts.max_count)\n     -+\t\treturn -1;\n     ++\t\treturn 1;\n      +\n      +\treturn 0;\n      +}\n     @@ ref-filter.c: int filter_refs(struct ref_array *array, struct ref_filter *filter\n      +\t\t\t\t\t  struct ref_format *format)\n      +{\n      +\t/*\n     -+\t * Refs can be filtered and formatted in the same iteration as long\n     -+\t * as we aren't filtering on reachability, sorting the results, or\n     -+\t * including ahead-behind information in the formatted output.\n     ++\t * Filtering & formatting results within a single ref iteration\n     ++\t * callback is not compatible with options that require\n     ++\t * post-processing a filtered ref_array. These include:\n     ++\t * - filtering on reachability\n     ++\t * - sorting the filtered results\n     ++\t * - including ahead-behind information in the formatted output\n      +\t */\n      +\treturn !(filter->reachable_from ||\n      +\t\t filter->unreachable_from ||\n  -:  ----------- >  8:  8eb2fc2950c for-each-ref: clean up documentation of --format\n  8:  352b5c42ac3 !  9:  48254d8e161 for-each-ref: add option to fully dereference tags\n     @@ Metadata\n      Author: Victoria Dye <vdye@github.com>\n      \n       ## Commit message ##\n     -    for-each-ref: add option to fully dereference tags\n     +    ref-filter.c: use peeled tag for '*' format fields\n      \n     -    Add a boolean flag '--full-deref' that, when enabled, fills '%(*fieldname)'\n     -    format fields using the fully peeled target of tag objects, rather than the\n     -    immediate target.\n     -\n     -    In other builtins ('rev-parse', 'show-ref'), \"dereferencing\" tags typically\n     -    means peeling them down to their non-tag target. Unlike these commands,\n     -    'for-each-ref' dereferences only one \"level\" of tags in '*' format fields\n     -    (like \"%(*objectname)\"). For most annotated tags, one level of dereferencing\n     -    is enough, since most tags point to commits or trees. However, nested tags\n     -    (annotated tags whose target is another annotated tag) dereferenced once\n     -    will point to their target tag, different a full peel to e.g. a commit.\n     +    In most builtins ('rev-parse <revision>^{}', 'show-ref --dereference'),\n     +    \"dereferencing\" a tag refers to a recursive peel of the tag object. Unlike\n     +    these cases, the dereferencing prefix ('*') in 'for-each-ref' format\n     +    specifiers triggers only a single, non-recursive dereference of a given tag\n     +    object. For most annotated tags, a single dereference is all that is needed\n     +    to access the tag's associated commit or tree; \"recursive\" and\n     +    \"non-recursive\" dereferencing are functionally equivalent in these cases.\n     +    However, nested tags (annotated tags whose target is another annotated tag)\n     +    dereferenced once return another tag, where a recursive dereference would\n     +    return the commit or tree.\n      \n          Currently, if a user wants to filter & format refs and include information\n     -    about the fully dereferenced tag, they can do so with something like\n     +    about a recursively-dereferenced tag, they can do so with something like\n          'cat-file --batch-check':\n      \n              git for-each-ref --format=\"%(objectname)^{} %(refname)\" <pattern> |\n                  git cat-file --batch-check=\"%(objectname) %(rest)\"\n      \n          But the combination of commands is inefficient. So, to improve the\n     -    efficiency of this use case, add a '--full-deref' option that causes\n     -    'for-each-ref' to fully dereference tags when formatting with '*' fields.\n     +    performance of this use case and align the defererencing behavior of\n     +    'for-each-ref' with that of other commands, update the ref formatting code\n     +    to use the peeled tag (from 'peel_iterated_oid()') to populate '*' fields\n     +    rather than the tag's immediate target object (from 'get_tagged_oid()').\n     +\n     +    Additionally, add a test to 't6300-for-each-ref' to verify new nested tag\n     +    behavior and update 't6302-for-each-ref-filter.sh' to print the correct\n     +    value for nested dereferenced fields.\n      \n          Signed-off-by: Victoria Dye <vdye@github.com>\n      \n       ## Documentation/git-for-each-ref.txt ##\n     -@@ Documentation/git-for-each-ref.txt: SYNOPSIS\n     - 'git for-each-ref' [--count=<count>] [--shell|--perl|--python|--tcl]\n     - \t\t   [(--sort=<key>)...] [--format=<format>]\n     - \t\t   [ --stdin | <pattern>... ]\n     -+\t\t   [--full-deref]\n     - \t\t   [--points-at=<object>]\n     - \t\t   [--merged[=<object>]] [--no-merged[=<object>]]\n     - \t\t   [--contains[=<object>]] [--no-contains[=<object>]]\n     -@@ Documentation/git-for-each-ref.txt: OPTIONS\n     - \tthe specified host language.  This is meant to produce\n     - \ta scriptlet that can directly be `eval`ed.\n     +@@ Documentation/git-for-each-ref.txt: from the `committer` or `tagger` fields depending on the object type.\n     + These are intended for working on a mix of annotated and lightweight tags.\n       \n     -+--full-deref::\n     -+\tPopulate dereferenced format fields (indicated with an asterisk (`*`)\n     -+\tprefix before the fieldname) with information about the fully-peeled\n     -+\ttarget object of a tag ref, rather than its immediate target object.\n     -+\tThis only affects the output for nested annotated tags, where the tag's\n     -+\timmediate target is another tag but its fully-peeled target is another\n     -+\tobject type (e.g. a commit).\n     -+\n     - --points-at=<object>::\n     - \tOnly list refs which points at the given object.\n     + For tag objects, a `fieldname` prefixed with an asterisk (`*`) expands to\n     +-the `fieldname` value of object the tag points at, rather than that of the\n     +-tag object itself.\n     ++the `fieldname` value of the peeled object, rather than that of the tag\n     ++object itself.\n       \n     -\n     - ## builtin/for-each-ref.c ##\n     -@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n     - \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only <n> matched refs\")),\n     - \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n     - \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n     -+\t\tOPT_BOOL(0, \"full-deref\", &format.full_deref,\n     -+\t\t\t N_(\"fully dereference tags to populate '*' format fields\")),\n     - \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n     - \t\tOPT_REF_SORT(&sorting_options),\n     - \t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n     + Fields that have name-email-date tuple as its value (`author`,\n     + `committer`, and `tagger`) can be suffixed with `name`, `email`,\n      \n       ## ref-filter.c ##\n     -@@ ref-filter.c: static struct used_atom {\n     - \t\tchar *head;\n     - \t} u;\n     - } *used_atom;\n     --static int used_atom_cnt, need_tagged, need_symref;\n     -+static int used_atom_cnt, need_symref;\n     -+\n     -+enum tag_dereference_mode {\n     -+\tNO_DEREF = 0,\n     -+\tDEREF_ONE,\n     -+\tDEREF_ALL\n     -+};\n     -+static enum tag_dereference_mode need_tagged;\n     - \n     - /*\n     -  * Expand string, append it to strbuf *sb, then return error code ret.\n     -@@ ref-filter.c: static int parse_ref_filter_atom(struct ref_format *format,\n     - \tmemset(&used_atom[at].u, 0, sizeof(used_atom[at].u));\n     - \tif (valid_atom[i].parser && valid_atom[i].parser(format, &used_atom[at], arg, err))\n     - \t\treturn -1;\n     --\tif (*atom == '*')\n     --\t\tneed_tagged = 1;\n     -+\tif (*atom == '*' && !need_tagged)\n     -+\t\tneed_tagged = format->full_deref ? DEREF_ALL : DEREF_ONE;\n     - \tif (i == ATOM_SYMREF)\n     - \t\tneed_symref = 1;\n     - \treturn at;\n      @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n     - \t * If it is a tag object, see if we use a value that derefs\n     - \t * the object, and if we do grab the object it refers to.\n     + \t\treturn 0;\n     + \n     + \t/*\n     +-\t * If it is a tag object, see if we use a value that derefs\n     +-\t * the object, and if we do grab the object it refers to.\n     ++\t * If it is a tag object, see if we use the peeled value. If we do,\n     ++\t * grab the peeled OID.\n       \t */\n      -\toi_deref.oid = *get_tagged_oid((struct tag *)obj);\n     -+\tif (need_tagged == DEREF_ALL) {\n     -+\t\tif (peel_iterated_oid(&obj->oid, &oi_deref.oid))\n     -+\t\t\tdie(\"bad tag\");\n     -+\t} else {\n     -+\t\toi_deref.oid = *get_tagged_oid((struct tag *)obj);\n     -+\t}\n     ++\tif (need_tagged && peel_iterated_oid(&obj->oid, &oi_deref.oid))\n     ++\t\tdie(\"bad tag\");\n       \n      -\t/*\n      -\t * NEEDSWORK: This derefs tag only once, which\n     @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbu\n       }\n       \n      \n     - ## ref-filter.h ##\n     -@@ ref-filter.h: struct ref_format {\n     - \tconst char *rest;\n     - \tint quote_style;\n     - \tint use_color;\n     -+\tint full_deref;\n     - \n     - \t/* Internal state to ref-filter */\n     - \tint need_color_reset_at_eol;\n     -\n       ## t/t6300-for-each-ref.sh ##\n      @@ t/t6300-for-each-ref.sh: test_expect_success 'git for-each-ref with non-existing refs' '\n       \ttest_must_be_empty actual\n     @@ t/t6300-for-each-ref.sh: test_expect_success 'git for-each-ref with non-existing\n      +\tnest1_tag_oid=\"$(git rev-parse refs/tags/nested/nest1)\" &&\n      +\tnest2_tag_oid=\"$(git rev-parse refs/tags/nested/nest2)\" &&\n      +\n     -+\t# Without full dereference\n     -+\tcat >expect <<-EOF &&\n     -+\trefs/tags/nested/base $base_tag_oid tag $head_oid commit\n     -+\trefs/tags/nested/nest1 $nest1_tag_oid tag $base_tag_oid tag\n     -+\trefs/tags/nested/nest2 $nest2_tag_oid tag $nest1_tag_oid tag\n     -+\tEOF\n     -+\n     -+\tgit for-each-ref --format=\"%(refname) %(objectname) %(objecttype) %(*objectname) %(*objecttype)\" \\\n     -+\t\trefs/tags/nested/ >actual &&\n     -+\ttest_cmp expect actual &&\n     -+\n     -+\t# With full dereference\n      +\tcat >expect <<-EOF &&\n      +\trefs/tags/nested/base $base_tag_oid tag $head_oid commit\n      +\trefs/tags/nested/nest1 $nest1_tag_oid tag $head_oid commit\n      +\trefs/tags/nested/nest2 $nest2_tag_oid tag $head_oid commit\n      +\tEOF\n      +\n     -+\tgit for-each-ref --full-deref \\\n     ++\tgit for-each-ref \\\n      +\t\t--format=\"%(refname) %(objectname) %(objecttype) %(*objectname) %(*objecttype)\" \\\n      +\t\trefs/tags/nested/ >actual &&\n      +\ttest_cmp expect actual\n     @@ t/t6300-for-each-ref.sh: test_expect_success 'git for-each-ref with non-existing\n       GRADE_FORMAT=\"%(signature:grade)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n       TRUSTLEVEL_FORMAT=\"%(signature:trustlevel)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n       \n     +\n     + ## t/t6302-for-each-ref-filter.sh ##\n     +@@ t/t6302-for-each-ref-filter.sh: test_expect_success 'check signed tags with --points-at' '\n     + \tsed -e \"s/Z$//\" >expect <<-\\EOF &&\n     + \trefs/heads/side Z\n     + \trefs/tags/annotated-tag four\n     +-\trefs/tags/doubly-annotated-tag An annotated tag\n     +-\trefs/tags/doubly-signed-tag A signed tag\n     ++\trefs/tags/doubly-annotated-tag four\n     ++\trefs/tags/doubly-signed-tag four\n     + \trefs/tags/four Z\n     + \trefs/tags/signed-tag four\n     + \tEOF\n  9:  a409d773057 ! 10:  d51d073aa4a t/perf: add perf tests for for-each-ref\n     @@ Commit message\n          Add performance tests for 'for-each-ref'. The tests exercise different\n          combinations of filters/formats/options, as well as the overall performance\n          of 'git for-each-ref | git cat-file --batch-check' to demonstrate the\n     -    performance difference vs. 'git for-each-ref --full-deref'.\n     +    performance difference vs. 'git for-each-ref' with \"%(*fieldname)\" format\n     +    specifiers.\n      \n          All tests are run against a repository with 40k loose refs - 10k commits,\n          each having a unique:\n     @@ t/perf/p6300-for-each-ref.sh (new)\n      +run_tests () {\n      +\ttest_for_each_ref \"$1\"\n      +\ttest_for_each_ref \"$1, no sort\" --no-sort\n     ++\ttest_for_each_ref \"$1, --count=1\" --count=1\n     ++\ttest_for_each_ref \"$1, --count=1, no sort\" --no-sort --count=1\n      +\ttest_for_each_ref \"$1, tags\" refs/tags/\n      +\ttest_for_each_ref \"$1, tags, no sort\" --no-sort refs/tags/\n     -+\ttest_for_each_ref \"$1, tags, shallow deref\" '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n     -+\ttest_for_each_ref \"$1, tags, shallow deref, no sort\" --no-sort '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n     -+\ttest_for_each_ref \"$1, tags, full deref\" --full-deref '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n     -+\ttest_for_each_ref \"$1, tags, full deref, no sort\" --no-sort --full-deref '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n     ++\ttest_for_each_ref \"$1, tags, dereferenced\" '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n     ++\ttest_for_each_ref \"$1, tags, dereferenced, no sort\" --no-sort '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n      +\n     -+\ttest_perf \"for-each-ref ($1, tags) + cat-file --batch-check (full deref)\" \"\n     ++\ttest_perf \"for-each-ref ($1, tags) + cat-file --batch-check (dereferenced)\" \"\n      +\t\tfor i in \\$(test_seq $test_iteration_count); do\n      +\t\t\tgit for-each-ref --format='%(objectname)^{} %(refname) %(objectname)' refs/tags/ | \\\n      +\t\t\t\tgit cat-file --batch-check='%(objectname) %(rest)' >/dev/null\n\n-- \ngitgitgadget\n"},{"id":"484874","messageId":"adac101bc6022d5477371d6a94225f38da7fffee.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 02/10] ref-filter.h: add max_count and omit_empty to ref_format","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:50Z","receivedAt":"2023-11-14T19:54:08Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nAdd an internal 'array_opts' struct to 'struct ref_format' containing\nformatting options that pertain to the formatting of an entire ref array:\n'max_count' and 'omit_empty'. These values are specified by the '--count'\nand '--omit-empty' options, respectively, to 'for-each-ref'/'tag'/'branch'.\nStoring these values in the 'ref_format' will simplify the consolidation of\nref array formatting logic across builtins in later patches.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n builtin/branch.c       |  5 ++---\n builtin/for-each-ref.c | 21 +++++++++++----------\n builtin/tag.c          |  5 ++---\n ref-filter.h           |  5 +++++\n 4 files changed, 20 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex d67738bbcaa..5a1ec1cd04f 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -45,7 +45,6 @@ static const char *head;\n static struct object_id head_oid;\n static int recurse_submodules = 0;\n static int submodule_propagate_branches = 0;\n-static int omit_empty = 0;\n \n static int branch_use_color = -1;\n static char branch_colors[][COLOR_MAXLEN] = {\n@@ -480,7 +479,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \t\t\tstring_list_append(output, out.buf);\n \t\t} else {\n \t\t\tfwrite(out.buf, 1, out.len, stdout);\n-\t\t\tif (out.len || !omit_empty)\n+\t\t\tif (out.len || !format->array_opts.omit_empty)\n \t\t\t\tputchar('\\n');\n \t\t}\n \t}\n@@ -737,7 +736,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)\n \t\tOPT_BIT('D', NULL, &delete, N_(\"delete branch (even if not merged)\"), 2),\n \t\tOPT_BIT('m', \"move\", &rename, N_(\"move/rename a branch and its reflog\"), 1),\n \t\tOPT_BIT('M', NULL, &rename, N_(\"move/rename a branch, even if target exists\"), 2),\n-\t\tOPT_BOOL(0, \"omit-empty\",  &omit_empty,\n+\t\tOPT_BOOL(0, \"omit-empty\",  &format.array_opts.omit_empty,\n \t\t\tN_(\"do not output a newline after empty formatted refs\")),\n \t\tOPT_BIT('c', \"copy\", &copy, N_(\"copy a branch and its reflog\"), 1),\n \t\tOPT_BIT('C', NULL, &copy, N_(\"copy a branch, even if target exists\"), 2),\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 93b370f550b..881c3ee055f 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -19,10 +19,10 @@ static char const * const for_each_ref_usage[] = {\n \n int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n-\tint i;\n+\tint i, total;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n-\tint maxcount = 0, icase = 0, omit_empty = 0;\n+\tint icase = 0;\n \tstruct ref_array array;\n \tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n@@ -40,11 +40,11 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"quote placeholders suitably for python\"), QUOTE_PYTHON),\n \t\tOPT_BIT(0 , \"tcl\",  &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for Tcl\"), QUOTE_TCL),\n-\t\tOPT_BOOL(0, \"omit-empty\",  &omit_empty,\n+\t\tOPT_BOOL(0, \"omit-empty\",  &format.array_opts.omit_empty,\n \t\t\tN_(\"do not output a newline after empty formatted refs\")),\n \n \t\tOPT_GROUP(\"\"),\n-\t\tOPT_INTEGER( 0 , \"count\", &maxcount, N_(\"show only <n> matched refs\")),\n+\t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only <n> matched refs\")),\n \t\tOPT_STRING(  0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n@@ -71,8 +71,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstring_list_append(&sorting_options, \"refname\");\n \n \tparse_options(argc, argv, prefix, opts, for_each_ref_usage, 0);\n-\tif (maxcount < 0) {\n-\t\terror(\"invalid --count argument: `%d'\", maxcount);\n+\tif (format.array_opts.max_count < 0) {\n+\t\terror(\"invalid --count argument: `%d'\", format.array_opts.max_count);\n \t\tusage_with_options(for_each_ref_usage, opts);\n \t}\n \tif (HAS_MULTI_BITS(format.quote_style)) {\n@@ -109,15 +109,16 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tref_array_sort(sorting, &array);\n \n-\tif (!maxcount || array.nr < maxcount)\n-\t\tmaxcount = array.nr;\n-\tfor (i = 0; i < maxcount; i++) {\n+\ttotal = format.array_opts.max_count;\n+\tif (!total || array.nr < total)\n+\t\ttotal = array.nr;\n+\tfor (i = 0; i < total; i++) {\n \t\tstrbuf_reset(&err);\n \t\tstrbuf_reset(&output);\n \t\tif (format_ref_array_item(array.items[i], &format, &output, &err))\n \t\t\tdie(\"%s\", err.buf);\n \t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !omit_empty)\n+\t\tif (output.len || !format.array_opts.omit_empty)\n \t\t\tputchar('\\n');\n \t}\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 64f3196cd4c..2d599245d48 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -44,7 +44,6 @@ static const char * const git_tag_usage[] = {\n static unsigned int colopts;\n static int force_sign_annotate;\n static int config_sign_tag = -1; /* unspecified */\n-static int omit_empty = 0;\n \n static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\t     struct ref_format *format)\n@@ -83,7 +82,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\tif (format_ref_array_item(array.items[i], format, &output, &err))\n \t\t\tdie(\"%s\", err.buf);\n \t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !omit_empty)\n+\t\tif (output.len || !format->array_opts.omit_empty)\n \t\t\tputchar('\\n');\n \t}\n \n@@ -481,7 +480,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)\n \t\tOPT_WITHOUT(&filter.no_commit, N_(\"print only tags that don't contain the commit\")),\n \t\tOPT_MERGED(&filter, N_(\"print only tags that are merged\")),\n \t\tOPT_NO_MERGED(&filter, N_(\"print only tags that are not merged\")),\n-\t\tOPT_BOOL(0, \"omit-empty\",  &omit_empty,\n+\t\tOPT_BOOL(0, \"omit-empty\",  &format.array_opts.omit_empty,\n \t\t\tN_(\"do not output a newline after empty formatted refs\")),\n \t\tOPT_REF_SORT(&sorting_options),\n \t\t{\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 1524bc463a5..d87d61238b7 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -92,6 +92,11 @@ struct ref_format {\n \n \t/* List of bases for ahead-behind counts. */\n \tstruct string_list bases;\n+\n+\tstruct {\n+\t\tint max_count;\n+\t\tint omit_empty;\n+\t} array_opts;\n };\n \n #define REF_FILTER_INIT { \\\n-- \ngitgitgadget\n\n"},{"id":"484875","messageId":"f44c4b42c93983a8755dcc6da5146205c11ef422.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 03/10] ref-filter.h: move contains caches into filter","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:51Z","receivedAt":"2023-11-14T19:54:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nMove the 'contains_cache' and 'no_contains_cache' used in filter_refs into\nan 'internal' struct of the 'struct ref_filter'. In later patches, the\n'struct ref_filter *' will be a common data structure across multiple\nfiltering functions. These caches are part of the common functionality the\nfilter struct will support, so they are updated to be internally accessible\nwherever the filter is used.\n\nThe design used here mirrors what was introduced in 576de3d956\n(unpack_trees: start splitting internal fields from public API, 2023-02-27)\nfor 'unpack_trees_options'.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n ref-filter.c | 14 ++++++--------\n ref-filter.h |  6 ++++++\n 2 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7250089b7c6..5129b6986c9 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2764,8 +2764,6 @@ static int filter_ref_kind(struct ref_filter *filter, const char *refname)\n struct ref_filter_cbdata {\n \tstruct ref_array *array;\n \tstruct ref_filter *filter;\n-\tstruct contains_cache contains_cache;\n-\tstruct contains_cache no_contains_cache;\n };\n \n /*\n@@ -2816,11 +2814,11 @@ static int ref_filter_handler(const char *refname, const struct object_id *oid,\n \t\t\treturn 0;\n \t\t/* We perform the filtering for the '--contains' option... */\n \t\tif (filter->with_commit &&\n-\t\t    !commit_contains(filter, commit, filter->with_commit, &ref_cbdata->contains_cache))\n+\t\t    !commit_contains(filter, commit, filter->with_commit, &filter->internal.contains_cache))\n \t\t\treturn 0;\n \t\t/* ...or for the `--no-contains' option */\n \t\tif (filter->no_commit &&\n-\t\t    commit_contains(filter, commit, filter->no_commit, &ref_cbdata->no_contains_cache))\n+\t\t    commit_contains(filter, commit, filter->no_commit, &filter->internal.no_contains_cache))\n \t\t\treturn 0;\n \t}\n \n@@ -2989,8 +2987,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \tsave_commit_buffer_orig = save_commit_buffer;\n \tsave_commit_buffer = 0;\n \n-\tinit_contains_cache(&ref_cbdata.contains_cache);\n-\tinit_contains_cache(&ref_cbdata.no_contains_cache);\n+\tinit_contains_cache(&filter->internal.contains_cache);\n+\tinit_contains_cache(&filter->internal.no_contains_cache);\n \n \t/*  Simple per-ref filtering */\n \tif (!filter->kind)\n@@ -3014,8 +3012,8 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \t\t\thead_ref(ref_filter_handler, &ref_cbdata);\n \t}\n \n-\tclear_contains_cache(&ref_cbdata.contains_cache);\n-\tclear_contains_cache(&ref_cbdata.no_contains_cache);\n+\tclear_contains_cache(&filter->internal.contains_cache);\n+\tclear_contains_cache(&filter->internal.no_contains_cache);\n \n \t/*  Filters that need revision walking */\n \treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\ndiff --git a/ref-filter.h b/ref-filter.h\nindex d87d61238b7..0db3ff52889 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -7,6 +7,7 @@\n #include \"commit.h\"\n #include \"string-list.h\"\n #include \"strvec.h\"\n+#include \"commit-reach.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -75,6 +76,11 @@ struct ref_filter {\n \t\tlines;\n \tint abbrev,\n \t\tverbose;\n+\n+\tstruct {\n+\t\tstruct contains_cache contains_cache;\n+\t\tstruct contains_cache no_contains_cache;\n+\t} internal;\n };\n \n struct ref_format {\n-- \ngitgitgadget\n\n"},{"id":"484876","messageId":"187b1d6610f96ba16bb7e1ff80d1c994a67b8753.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 04/10] ref-filter.h: add functions for filter/format & format-only","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:52Z","receivedAt":"2023-11-14T19:54:11Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nAdd two new public methods to 'ref-filter.h':\n\n* 'print_formatted_ref_array()' which, given a format specification & array\n  of ref items, formats and prints the items to stdout.\n* 'filter_and_format_refs()' which combines 'filter_refs()',\n  'ref_array_sort()', and 'print_formatted_ref_array()' into a single\n  function.\n\nThis consolidates much of the code used to filter and format refs in\n'builtin/for-each-ref.c', 'builtin/tag.c', and 'builtin/branch.c', reducing\nduplication and simplifying the future changes needed to optimize the filter\n& format process.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n builtin/branch.c       | 33 +++++++++++++++++----------------\n builtin/for-each-ref.c | 27 +--------------------------\n builtin/tag.c          | 23 +----------------------\n ref-filter.c           | 35 +++++++++++++++++++++++++++++++++++\n ref-filter.h           | 14 ++++++++++++++\n 5 files changed, 68 insertions(+), 64 deletions(-)\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 5a1ec1cd04f..2ed59f16f1c 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -437,8 +437,6 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n {\n \tint i;\n \tstruct ref_array array;\n-\tstruct strbuf out = STRBUF_INIT;\n-\tstruct strbuf err = STRBUF_INIT;\n \tint maxwidth = 0;\n \tconst char *remote_prefix = \"\";\n \tchar *to_free = NULL;\n@@ -468,24 +466,27 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n-\tfor (i = 0; i < array.nr; i++) {\n-\t\tstrbuf_reset(&err);\n-\t\tstrbuf_reset(&out);\n-\t\tif (format_ref_array_item(array.items[i], format, &out, &err))\n-\t\t\tdie(\"%s\", err.buf);\n-\t\tif (column_active(colopts)) {\n-\t\t\tassert(!filter->verbose && \"--column and --verbose are incompatible\");\n-\t\t\t /* format to a string_list to let print_columns() do its job */\n+\tif (column_active(colopts)) {\n+\t\tstruct strbuf out = STRBUF_INIT, err = STRBUF_INIT;\n+\n+\t\tassert(!filter->verbose && \"--column and --verbose are incompatible\");\n+\n+\t\tfor (i = 0; i < array.nr; i++) {\n+\t\t\tstrbuf_reset(&err);\n+\t\t\tstrbuf_reset(&out);\n+\t\t\tif (format_ref_array_item(array.items[i], format, &out, &err))\n+\t\t\t\tdie(\"%s\", err.buf);\n+\n+\t\t\t/* format to a string_list to let print_columns() do its job */\n \t\t\tstring_list_append(output, out.buf);\n-\t\t} else {\n-\t\t\tfwrite(out.buf, 1, out.len, stdout);\n-\t\t\tif (out.len || !format->array_opts.omit_empty)\n-\t\t\t\tputchar('\\n');\n \t\t}\n+\n+\t\tstrbuf_release(&err);\n+\t\tstrbuf_release(&out);\n+\t} else {\n+\t\tprint_formatted_ref_array(&array, format);\n \t}\n \n-\tstrbuf_release(&err);\n-\tstrbuf_release(&out);\n \tref_array_clear(&array);\n \tfree(to_free);\n }\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 881c3ee055f..1c19cd5bd34 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -19,15 +19,11 @@ static char const * const for_each_ref_usage[] = {\n \n int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n-\tint i, total;\n \tstruct ref_sorting *sorting;\n \tstruct string_list sorting_options = STRING_LIST_INIT_DUP;\n \tint icase = 0;\n-\tstruct ref_array array;\n \tstruct ref_filter filter = REF_FILTER_INIT;\n \tstruct ref_format format = REF_FORMAT_INIT;\n-\tstruct strbuf output = STRBUF_INIT;\n-\tstruct strbuf err = STRBUF_INIT;\n \tint from_stdin = 0;\n \tstruct strvec vec = STRVEC_INIT;\n \n@@ -61,8 +57,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n-\tmemset(&array, 0, sizeof(array));\n-\n \tformat.format = \"%(objectname) %(objecttype)\\t%(refname)\";\n \n \tgit_config(git_default_config, NULL);\n@@ -104,27 +98,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t}\n \n \tfilter.match_as_path = 1;\n-\tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n-\tfilter_ahead_behind(the_repository, &format, &array);\n-\n-\tref_array_sort(sorting, &array);\n-\n-\ttotal = format.array_opts.max_count;\n-\tif (!total || array.nr < total)\n-\t\ttotal = array.nr;\n-\tfor (i = 0; i < total; i++) {\n-\t\tstrbuf_reset(&err);\n-\t\tstrbuf_reset(&output);\n-\t\tif (format_ref_array_item(array.items[i], &format, &output, &err))\n-\t\t\tdie(\"%s\", err.buf);\n-\t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !format.array_opts.omit_empty)\n-\t\t\tputchar('\\n');\n-\t}\n+\tfilter_and_format_refs(&filter, FILTER_REFS_ALL, sorting, &format);\n \n-\tstrbuf_release(&err);\n-\tstrbuf_release(&output);\n-\tref_array_clear(&array);\n \tref_filter_clear(&filter);\n \tref_sorting_release(sorting);\n \tstrvec_clear(&vec);\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 2d599245d48..2528d499dd8 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -48,13 +48,7 @@ static int config_sign_tag = -1; /* unspecified */\n static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\t     struct ref_format *format)\n {\n-\tstruct ref_array array;\n-\tstruct strbuf output = STRBUF_INIT;\n-\tstruct strbuf err = STRBUF_INIT;\n \tchar *to_free = NULL;\n-\tint i;\n-\n-\tmemset(&array, 0, sizeof(array));\n \n \tif (filter->lines == -1)\n \t\tfilter->lines = 0;\n@@ -72,23 +66,8 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n-\tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n-\tfilter_ahead_behind(the_repository, format, &array);\n-\tref_array_sort(sorting, &array);\n-\n-\tfor (i = 0; i < array.nr; i++) {\n-\t\tstrbuf_reset(&output);\n-\t\tstrbuf_reset(&err);\n-\t\tif (format_ref_array_item(array.items[i], format, &output, &err))\n-\t\t\tdie(\"%s\", err.buf);\n-\t\tfwrite(output.buf, 1, output.len, stdout);\n-\t\tif (output.len || !format->array_opts.omit_empty)\n-\t\t\tputchar('\\n');\n-\t}\n+\tfilter_and_format_refs(filter, FILTER_REFS_TAGS, sorting, format);\n \n-\tstrbuf_release(&err);\n-\tstrbuf_release(&output);\n-\tref_array_clear(&array);\n \tfree(to_free);\n \n \treturn 0;\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5129b6986c9..8992fbf45b1 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -3023,6 +3023,18 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \treturn ret;\n }\n \n+void filter_and_format_refs(struct ref_filter *filter, unsigned int type,\n+\t\t\t    struct ref_sorting *sorting,\n+\t\t\t    struct ref_format *format)\n+{\n+\tstruct ref_array array = { 0 };\n+\tfilter_refs(&array, filter, type);\n+\tfilter_ahead_behind(the_repository, format, &array);\n+\tref_array_sort(sorting, &array);\n+\tprint_formatted_ref_array(&array, format);\n+\tref_array_clear(&array);\n+}\n+\n static int compare_detached_head(struct ref_array_item *a, struct ref_array_item *b)\n {\n \tif (!(a->kind ^ b->kind))\n@@ -3212,6 +3224,29 @@ int format_ref_array_item(struct ref_array_item *info,\n \treturn 0;\n }\n \n+void print_formatted_ref_array(struct ref_array *array, struct ref_format *format)\n+{\n+\tint total;\n+\tstruct strbuf output = STRBUF_INIT, err = STRBUF_INIT;\n+\n+\ttotal = format->array_opts.max_count;\n+\tif (!total || array->nr < total)\n+\t\ttotal = array->nr;\n+\tfor (int i = 0; i < total; i++) {\n+\t\tstrbuf_reset(&err);\n+\t\tstrbuf_reset(&output);\n+\t\tif (format_ref_array_item(array->items[i], format, &output, &err))\n+\t\t\tdie(\"%s\", err.buf);\n+\t\tif (output.len || !format->array_opts.omit_empty) {\n+\t\t\tfwrite(output.buf, 1, output.len, stdout);\n+\t\t\tputchar('\\n');\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&err);\n+\tstrbuf_release(&output);\n+}\n+\n void pretty_print_ref(const char *name, const struct object_id *oid,\n \t\t      struct ref_format *format)\n {\ndiff --git a/ref-filter.h b/ref-filter.h\nindex 0db3ff52889..0ce5af58ab3 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -137,6 +137,14 @@ struct ref_format {\n  * filtered refs in the ref_array structure.\n  */\n int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type);\n+/*\n+ * Filter refs using the given ref_filter and type, sort the contents\n+ * according to the given ref_sorting, format the filtered refs with the\n+ * given ref_format, and print them to stdout.\n+ */\n+void filter_and_format_refs(struct ref_filter *filter, unsigned int type,\n+\t\t\t    struct ref_sorting *sorting,\n+\t\t\t    struct ref_format *format);\n /*  Clear all memory allocated to ref_array */\n void ref_array_clear(struct ref_array *array);\n /*  Used to verify if the given format is correct and to parse out the used atoms */\n@@ -161,6 +169,12 @@ char *get_head_description(void);\n /*  Set up translated strings in the output. */\n void setup_ref_filter_porcelain_msg(void);\n \n+/*\n+ * Print up to maxcount ref_array elements to stdout using the given\n+ * ref_format.\n+ */\n+void print_formatted_ref_array(struct ref_array *array, struct ref_format *format);\n+\n /*\n  * Print a single ref, outside of any ref-filter. Note that the\n  * name must be a fully qualified refname.\n-- \ngitgitgadget\n\n"},{"id":"484877","messageId":"040d291ca458ef0068e3fd543e7f441b2e5e71ad.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 05/10] ref-filter.c: rename 'ref_filter_handler()' to 'filter_one()'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:53Z","receivedAt":"2023-11-14T19:54:13Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nRename 'ref_filter_handler()' to 'filter_one()' to more clearly distinguish\nit from other ref filtering callbacks that will be added in later patches.\nThe \"*_one()\" naming convention is common throughout the codebase for\niteration callbacks.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n ref-filter.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 8992fbf45b1..5186ee2687b 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2770,7 +2770,7 @@ struct ref_filter_cbdata {\n  * A call-back given to for_each_ref().  Filter refs and keep them for\n  * later object processing.\n  */\n-static int ref_filter_handler(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n+static int filter_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n {\n \tstruct ref_filter_cbdata *ref_cbdata = cb_data;\n \tstruct ref_filter *filter = ref_cbdata->filter;\n@@ -3001,15 +3001,15 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \t\t * of filter_ref_kind().\n \t\t */\n \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = for_each_fullref_in(\"refs/heads/\", ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/heads/\", filter_one, &ref_cbdata);\n \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = for_each_fullref_in(\"refs/remotes/\", ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/remotes/\", filter_one, &ref_cbdata);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = for_each_fullref_in(\"refs/tags/\", ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/tags/\", filter_one, &ref_cbdata);\n \t\telse if (filter->kind & FILTER_REFS_ALL)\n-\t\t\tret = for_each_fullref_in_pattern(filter, ref_filter_handler, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in_pattern(filter, filter_one, &ref_cbdata);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n-\t\t\thead_ref(ref_filter_handler, &ref_cbdata);\n+\t\t\thead_ref(filter_one, &ref_cbdata);\n \t}\n \n \tclear_contains_cache(&filter->internal.contains_cache);\n-- \ngitgitgadget\n\n"},{"id":"484878","messageId":"633c0c74c2efb42e1491d53809a9825599690301.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 06/10] ref-filter.c: refactor to create common helper functions","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:54Z","receivedAt":"2023-11-14T19:54:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nFactor out parts of 'ref_array_push()', 'ref_filter_handler()', and\n'filter_refs()' into new helper functions:\n\n* Extract the code to grow a 'struct ref_array' and append a given 'struct\n  ref_array_item *' to it from 'ref_array_push()' into 'ref_array_append()'.\n* Extract the code to filter a given ref by refname & object ID then create\n  a new 'struct ref_array_item *' from 'filter_one()' into\n  'apply_ref_filter()'.\n* Extract the code for filter pre-processing, contains cache creation, and\n  ref iteration from 'filter_refs()' into 'do_filter_refs()'.\n\nIn later patches, these helpers will be used by new ref-filter API\nfunctions. This patch does not result in any user-facing behavior changes or\nchanges to callers outside of 'ref-filter.c'.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n ref-filter.c | 115 ++++++++++++++++++++++++++++++---------------------\n 1 file changed, 69 insertions(+), 46 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5186ee2687b..ff00ab4b8d8 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2716,15 +2716,18 @@ static struct ref_array_item *new_ref_array_item(const char *refname,\n \treturn ref;\n }\n \n+static void ref_array_append(struct ref_array *array, struct ref_array_item *ref)\n+{\n+\tALLOC_GROW(array->items, array->nr + 1, array->alloc);\n+\tarray->items[array->nr++] = ref;\n+}\n+\n struct ref_array_item *ref_array_push(struct ref_array *array,\n \t\t\t\t      const char *refname,\n \t\t\t\t      const struct object_id *oid)\n {\n \tstruct ref_array_item *ref = new_ref_array_item(refname, oid);\n-\n-\tALLOC_GROW(array->items, array->nr + 1, array->alloc);\n-\tarray->items[array->nr++] = ref;\n-\n+\tref_array_append(array, ref);\n \treturn ref;\n }\n \n@@ -2761,46 +2764,36 @@ static int filter_ref_kind(struct ref_filter *filter, const char *refname)\n \treturn ref_kind_from_refname(refname);\n }\n \n-struct ref_filter_cbdata {\n-\tstruct ref_array *array;\n-\tstruct ref_filter *filter;\n-};\n-\n-/*\n- * A call-back given to for_each_ref().  Filter refs and keep them for\n- * later object processing.\n- */\n-static int filter_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n+static struct ref_array_item *apply_ref_filter(const char *refname, const struct object_id *oid,\n+\t\t\t    int flag, struct ref_filter *filter)\n {\n-\tstruct ref_filter_cbdata *ref_cbdata = cb_data;\n-\tstruct ref_filter *filter = ref_cbdata->filter;\n \tstruct ref_array_item *ref;\n \tstruct commit *commit = NULL;\n \tunsigned int kind;\n \n \tif (flag & REF_BAD_NAME) {\n \t\twarning(_(\"ignoring ref with broken name %s\"), refname);\n-\t\treturn 0;\n+\t\treturn NULL;\n \t}\n \n \tif (flag & REF_ISBROKEN) {\n \t\twarning(_(\"ignoring broken ref %s\"), refname);\n-\t\treturn 0;\n+\t\treturn NULL;\n \t}\n \n \t/* Obtain the current ref kind from filter_ref_kind() and ignore unwanted refs. */\n \tkind = filter_ref_kind(filter, refname);\n \tif (!(kind & filter->kind))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tif (!filter_pattern_match(filter, refname))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tif (filter_exclude_match(filter, refname))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tif (filter->points_at.nr && !match_points_at(&filter->points_at, oid, refname))\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \t/*\n \t * A merge filter is applied on refs pointing to commits. Hence\n@@ -2811,15 +2804,15 @@ static int filter_one(const char *refname, const struct object_id *oid, int flag\n \t    filter->with_commit || filter->no_commit || filter->verbose) {\n \t\tcommit = lookup_commit_reference_gently(the_repository, oid, 1);\n \t\tif (!commit)\n-\t\t\treturn 0;\n+\t\t\treturn NULL;\n \t\t/* We perform the filtering for the '--contains' option... */\n \t\tif (filter->with_commit &&\n \t\t    !commit_contains(filter, commit, filter->with_commit, &filter->internal.contains_cache))\n-\t\t\treturn 0;\n+\t\t\treturn NULL;\n \t\t/* ...or for the `--no-contains' option */\n \t\tif (filter->no_commit &&\n \t\t    commit_contains(filter, commit, filter->no_commit, &filter->internal.no_contains_cache))\n-\t\t\treturn 0;\n+\t\t\treturn NULL;\n \t}\n \n \t/*\n@@ -2827,11 +2820,32 @@ static int filter_one(const char *refname, const struct object_id *oid, int flag\n \t * to do its job and the resulting list may yet to be pruned\n \t * by maxcount logic.\n \t */\n-\tref = ref_array_push(ref_cbdata->array, refname, oid);\n+\tref = new_ref_array_item(refname, oid);\n \tref->commit = commit;\n \tref->flag = flag;\n \tref->kind = kind;\n \n+\treturn ref;\n+}\n+\n+struct ref_filter_cbdata {\n+\tstruct ref_array *array;\n+\tstruct ref_filter *filter;\n+};\n+\n+/*\n+ * A call-back given to for_each_ref().  Filter refs and keep them for\n+ * later object processing.\n+ */\n+static int filter_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n+{\n+\tstruct ref_filter_cbdata *ref_cbdata = cb_data;\n+\tstruct ref_array_item *ref;\n+\n+\tref = apply_ref_filter(refname, oid, flag, ref_cbdata->filter);\n+\tif (ref)\n+\t\tref_array_append(ref_cbdata->array, ref);\n+\n \treturn 0;\n }\n \n@@ -2967,26 +2981,12 @@ void filter_ahead_behind(struct repository *r,\n \tfree(commits);\n }\n \n-/*\n- * API for filtering a set of refs. Based on the type of refs the user\n- * has requested, we iterate through those refs and apply filters\n- * as per the given ref_filter structure and finally store the\n- * filtered refs in the ref_array structure.\n- */\n-int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type)\n+static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)\n {\n-\tstruct ref_filter_cbdata ref_cbdata;\n-\tint save_commit_buffer_orig;\n \tint ret = 0;\n \n-\tref_cbdata.array = array;\n-\tref_cbdata.filter = filter;\n-\n \tfilter->kind = type & FILTER_REFS_KIND_MASK;\n \n-\tsave_commit_buffer_orig = save_commit_buffer;\n-\tsave_commit_buffer = 0;\n-\n \tinit_contains_cache(&filter->internal.contains_cache);\n \tinit_contains_cache(&filter->internal.no_contains_cache);\n \n@@ -3001,20 +3001,43 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \t\t * of filter_ref_kind().\n \t\t */\n \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = for_each_fullref_in(\"refs/heads/\", filter_one, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/heads/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = for_each_fullref_in(\"refs/remotes/\", filter_one, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/remotes/\", fn, cb_data);\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = for_each_fullref_in(\"refs/tags/\", filter_one, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in(\"refs/tags/\", fn, cb_data);\n \t\telse if (filter->kind & FILTER_REFS_ALL)\n-\t\t\tret = for_each_fullref_in_pattern(filter, filter_one, &ref_cbdata);\n+\t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \t\tif (!ret && (filter->kind & FILTER_REFS_DETACHED_HEAD))\n-\t\t\thead_ref(filter_one, &ref_cbdata);\n+\t\t\thead_ref(fn, cb_data);\n \t}\n \n \tclear_contains_cache(&filter->internal.contains_cache);\n \tclear_contains_cache(&filter->internal.no_contains_cache);\n \n+\treturn ret;\n+}\n+\n+/*\n+ * API for filtering a set of refs. Based on the type of refs the user\n+ * has requested, we iterate through those refs and apply filters\n+ * as per the given ref_filter structure and finally store the\n+ * filtered refs in the ref_array structure.\n+ */\n+int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int type)\n+{\n+\tstruct ref_filter_cbdata ref_cbdata;\n+\tint save_commit_buffer_orig;\n+\tint ret = 0;\n+\n+\tref_cbdata.array = array;\n+\tref_cbdata.filter = filter;\n+\n+\tsave_commit_buffer_orig = save_commit_buffer;\n+\tsave_commit_buffer = 0;\n+\n+\tret = do_filter_refs(filter, type, filter_one, &ref_cbdata);\n+\n \t/*  Filters that need revision walking */\n \treach_filter(array, &filter->reachable_from, INCLUDE_REACHED);\n \treach_filter(array, &filter->unreachable_from, EXCLUDE_REACHED);\n-- \ngitgitgadget\n\n"},{"id":"484879","messageId":"91a77c1a834cfc2fbe0676222135c2beb6ed3e01.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 07/10] ref-filter.c: filter & format refs in the same callback","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:55Z","receivedAt":"2023-11-14T19:54:17Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nUpdate 'filter_and_format_refs()' to try to perform ref filtering &\nformatting in a single ref iteration, without an intermediate 'struct\nref_array'. This can only be done if no operations need to be performed on a\npre-filtered array; specifically, if the refs are\n\n- filtered on reachability,\n- sorted, or\n- formatted with ahead-behind information\n\nthey cannot be filtered & formatted in the same iteration. In that case,\nfall back on the current filter-then-sort-then-format flow.\n\nThis optimization substantially improves memory usage due to no longer\nstoring a ref array in memory. In some cases, it also dramatically reduces\nruntime (e.g. 'git for-each-ref --no-sort --count=1', which no longer loads\nall refs into a 'struct ref_array' to printing only the first ref).\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n ref-filter.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 82 insertions(+), 6 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex ff00ab4b8d8..48453db24f7 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2863,6 +2863,49 @@ static void free_array_item(struct ref_array_item *item)\n \tfree(item);\n }\n \n+struct ref_filter_and_format_cbdata {\n+\tstruct ref_filter *filter;\n+\tstruct ref_format *format;\n+\n+\tstruct ref_filter_and_format_internal {\n+\t\tint count;\n+\t} internal;\n+};\n+\n+static int filter_and_format_one(const char *refname, const struct object_id *oid, int flag, void *cb_data)\n+{\n+\tstruct ref_filter_and_format_cbdata *ref_cbdata = cb_data;\n+\tstruct ref_array_item *ref;\n+\tstruct strbuf output = STRBUF_INIT, err = STRBUF_INIT;\n+\n+\tref = apply_ref_filter(refname, oid, flag, ref_cbdata->filter);\n+\tif (!ref)\n+\t\treturn 0;\n+\n+\tif (format_ref_array_item(ref, ref_cbdata->format, &output, &err))\n+\t\tdie(\"%s\", err.buf);\n+\n+\tif (output.len || !ref_cbdata->format->array_opts.omit_empty) {\n+\t\tfwrite(output.buf, 1, output.len, stdout);\n+\t\tputchar('\\n');\n+\t}\n+\n+\tstrbuf_release(&output);\n+\tstrbuf_release(&err);\n+\tfree_array_item(ref);\n+\n+\t/*\n+\t * Increment the running count of refs that match the filter. If\n+\t * max_count is set and we've reached the max, stop the ref\n+\t * iteration by returning a nonzero value.\n+\t */\n+\tif (ref_cbdata->format->array_opts.max_count &&\n+\t    ++ref_cbdata->internal.count >= ref_cbdata->format->array_opts.max_count)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n /* Free all memory allocated for ref_array */\n void ref_array_clear(struct ref_array *array)\n {\n@@ -3046,16 +3089,49 @@ int filter_refs(struct ref_array *array, struct ref_filter *filter, unsigned int\n \treturn ret;\n }\n \n+static inline int can_do_iterative_format(struct ref_filter *filter,\n+\t\t\t\t\t  struct ref_sorting *sorting,\n+\t\t\t\t\t  struct ref_format *format)\n+{\n+\t/*\n+\t * Filtering & formatting results within a single ref iteration\n+\t * callback is not compatible with options that require\n+\t * post-processing a filtered ref_array. These include:\n+\t * - filtering on reachability\n+\t * - sorting the filtered results\n+\t * - including ahead-behind information in the formatted output\n+\t */\n+\treturn !(filter->reachable_from ||\n+\t\t filter->unreachable_from ||\n+\t\t sorting ||\n+\t\t format->bases.nr);\n+}\n+\n void filter_and_format_refs(struct ref_filter *filter, unsigned int type,\n \t\t\t    struct ref_sorting *sorting,\n \t\t\t    struct ref_format *format)\n {\n-\tstruct ref_array array = { 0 };\n-\tfilter_refs(&array, filter, type);\n-\tfilter_ahead_behind(the_repository, format, &array);\n-\tref_array_sort(sorting, &array);\n-\tprint_formatted_ref_array(&array, format);\n-\tref_array_clear(&array);\n+\tif (can_do_iterative_format(filter, sorting, format)) {\n+\t\tint save_commit_buffer_orig;\n+\t\tstruct ref_filter_and_format_cbdata ref_cbdata = {\n+\t\t\t.filter = filter,\n+\t\t\t.format = format,\n+\t\t};\n+\n+\t\tsave_commit_buffer_orig = save_commit_buffer;\n+\t\tsave_commit_buffer = 0;\n+\n+\t\tdo_filter_refs(filter, type, filter_and_format_one, &ref_cbdata);\n+\n+\t\tsave_commit_buffer = save_commit_buffer_orig;\n+\t} else {\n+\t\tstruct ref_array array = { 0 };\n+\t\tfilter_refs(&array, filter, type);\n+\t\tfilter_ahead_behind(the_repository, format, &array);\n+\t\tref_array_sort(sorting, &array);\n+\t\tprint_formatted_ref_array(&array, format);\n+\t\tref_array_clear(&array);\n+\t}\n }\n \n static int compare_detached_head(struct ref_array_item *a, struct ref_array_item *b)\n-- \ngitgitgadget\n\n"},{"id":"484880","messageId":"8eb2fc2950c04bd6fe91112e7056dedeec774e6a.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 08/10] for-each-ref: clean up documentation of --format","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:56Z","receivedAt":"2023-11-14T19:54:18Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nMove the description of the `*` prefix from the --format option\ndocumentation to the part of the command documentation that deals with other\nobject type-specific modifiers. Also reorganize and reword the remaining\n--format documentation so that the explanation of the default format doesn't\ninterrupt the details on format string interpolation.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n Documentation/git-for-each-ref.txt | 23 ++++++++++++-----------\n 1 file changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex e86d5700ddf..b136c9fa908 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -51,17 +51,14 @@ OPTIONS\n \tkey.\n \n --format=<format>::\n-\tA string that interpolates `%(fieldname)` from a ref being shown\n-\tand the object it points at.  If `fieldname`\n-\tis prefixed with an asterisk (`*`) and the ref points\n-\tat a tag object, use the value for the field in the object\n-\twhich the tag object refers to (instead of the field in the tag object).\n-\tWhen unspecified, `<format>` defaults to\n-\t`%(objectname) SPC %(objecttype) TAB %(refname)`.\n-\tIt also interpolates `%%` to `%`, and `%xx` where `xx`\n-\tare hex digits interpolates to character with hex code\n-\t`xx`; for example `%00` interpolates to `\\0` (NUL),\n-\t`%09` to `\\t` (TAB) and `%0a` to `\\n` (LF).\n+\tA string that interpolates `%(fieldname)` from a ref being shown and\n+\tthe object it points at. In addition, the string literal `%%`\n+\trenders as `%` and `%xx` - where `xx` are hex digits - renders as\n+\tthe character with hex code `xx`. For example, `%00` interpolates to\n+\t`\\0` (NUL), `%09` to `\\t` (TAB), and `%0a` to `\\n` (LF).\n++\n+When unspecified, `<format>` defaults to `%(objectname) SPC %(objecttype)\n+TAB %(refname)`.\n \n --color[=<when>]::\n \tRespect any colors specified in the `--format` option. The\n@@ -298,6 +295,10 @@ fields will correspond to the appropriate date or name-email-date tuple\n from the `committer` or `tagger` fields depending on the object type.\n These are intended for working on a mix of annotated and lightweight tags.\n \n+For tag objects, a `fieldname` prefixed with an asterisk (`*`) expands to\n+the `fieldname` value of object the tag points at, rather than that of the\n+tag object itself.\n+\n Fields that have name-email-date tuple as its value (`author`,\n `committer`, and `tagger`) can be suffixed with `name`, `email`,\n and `date` to extract the named component.  For email fields (`authoremail`,\n-- \ngitgitgadget\n\n"},{"id":"484881","messageId":"48254d8e161de7f0e165510c06801195f9b0a8fd.1699991638.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 09/10] ref-filter.c: use peeled tag for '*' format fields","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:57Z","receivedAt":"2023-11-14T19:54:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nIn most builtins ('rev-parse <revision>^{}', 'show-ref --dereference'),\n\"dereferencing\" a tag refers to a recursive peel of the tag object. Unlike\nthese cases, the dereferencing prefix ('*') in 'for-each-ref' format\nspecifiers triggers only a single, non-recursive dereference of a given tag\nobject. For most annotated tags, a single dereference is all that is needed\nto access the tag's associated commit or tree; \"recursive\" and\n\"non-recursive\" dereferencing are functionally equivalent in these cases.\nHowever, nested tags (annotated tags whose target is another annotated tag)\ndereferenced once return another tag, where a recursive dereference would\nreturn the commit or tree.\n\nCurrently, if a user wants to filter & format refs and include information\nabout a recursively-dereferenced tag, they can do so with something like\n'cat-file --batch-check':\n\n    git for-each-ref --format=\"%(objectname)^{} %(refname)\" <pattern> |\n        git cat-file --batch-check=\"%(objectname) %(rest)\"\n\nBut the combination of commands is inefficient. So, to improve the\nperformance of this use case and align the defererencing behavior of\n'for-each-ref' with that of other commands, update the ref formatting code\nto use the peeled tag (from 'peel_iterated_oid()') to populate '*' fields\nrather than the tag's immediate target object (from 'get_tagged_oid()').\n\nAdditionally, add a test to 't6300-for-each-ref' to verify new nested tag\nbehavior and update 't6302-for-each-ref-filter.sh' to print the correct\nvalue for nested dereferenced fields.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n Documentation/git-for-each-ref.txt |  4 ++--\n ref-filter.c                       | 13 ++++---------\n t/t6300-for-each-ref.sh            | 22 ++++++++++++++++++++++\n t/t6302-for-each-ref-filter.sh     |  4 ++--\n 4 files changed, 30 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex b136c9fa908..be9543f6840 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -296,8 +296,8 @@ from the `committer` or `tagger` fields depending on the object type.\n These are intended for working on a mix of annotated and lightweight tags.\n \n For tag objects, a `fieldname` prefixed with an asterisk (`*`) expands to\n-the `fieldname` value of object the tag points at, rather than that of the\n-tag object itself.\n+the `fieldname` value of the peeled object, rather than that of the tag\n+object itself.\n \n Fields that have name-email-date tuple as its value (`author`,\n `committer`, and `tagger`) can be suffixed with `name`, `email`,\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 48453db24f7..fdaabb5bb45 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2508,17 +2508,12 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\treturn 0;\n \n \t/*\n-\t * If it is a tag object, see if we use a value that derefs\n-\t * the object, and if we do grab the object it refers to.\n+\t * If it is a tag object, see if we use the peeled value. If we do,\n+\t * grab the peeled OID.\n \t */\n-\toi_deref.oid = *get_tagged_oid((struct tag *)obj);\n+\tif (need_tagged && peel_iterated_oid(&obj->oid, &oi_deref.oid))\n+\t\tdie(\"bad tag\");\n \n-\t/*\n-\t * NEEDSWORK: This derefs tag only once, which\n-\t * is good to deal with chains of trust, but\n-\t * is not consistent with what deref_tag() does\n-\t * which peels the onion to the core.\n-\t */\n \treturn get_object(ref, 1, &obj, &oi_deref, err);\n }\n \ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex 0613e5e3623..54e22812598 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1839,6 +1839,28 @@ test_expect_success 'git for-each-ref with non-existing refs' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'git for-each-ref with nested tags' '\n+\tgit tag -am \"Normal tag\" nested/base HEAD &&\n+\tgit tag -am \"Nested tag\" nested/nest1 refs/tags/nested/base &&\n+\tgit tag -am \"Double nested tag\" nested/nest2 refs/tags/nested/nest1 &&\n+\n+\thead_oid=\"$(git rev-parse HEAD)\" &&\n+\tbase_tag_oid=\"$(git rev-parse refs/tags/nested/base)\" &&\n+\tnest1_tag_oid=\"$(git rev-parse refs/tags/nested/nest1)\" &&\n+\tnest2_tag_oid=\"$(git rev-parse refs/tags/nested/nest2)\" &&\n+\n+\tcat >expect <<-EOF &&\n+\trefs/tags/nested/base $base_tag_oid tag $head_oid commit\n+\trefs/tags/nested/nest1 $nest1_tag_oid tag $head_oid commit\n+\trefs/tags/nested/nest2 $nest2_tag_oid tag $head_oid commit\n+\tEOF\n+\n+\tgit for-each-ref \\\n+\t\t--format=\"%(refname) %(objectname) %(objecttype) %(*objectname) %(*objecttype)\" \\\n+\t\trefs/tags/nested/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n GRADE_FORMAT=\"%(signature:grade)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n TRUSTLEVEL_FORMAT=\"%(signature:trustlevel)%0a%(signature:key)%0a%(signature:signer)%0a%(signature:fingerprint)%0a%(signature:primarykeyfingerprint)\"\n \ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex af223e44d67..82f3d1ea0f2 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -45,8 +45,8 @@ test_expect_success 'check signed tags with --points-at' '\n \tsed -e \"s/Z$//\" >expect <<-\\EOF &&\n \trefs/heads/side Z\n \trefs/tags/annotated-tag four\n-\trefs/tags/doubly-annotated-tag An annotated tag\n-\trefs/tags/doubly-signed-tag A signed tag\n+\trefs/tags/doubly-annotated-tag four\n+\trefs/tags/doubly-signed-tag four\n \trefs/tags/four Z\n \trefs/tags/signed-tag four\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"484882","messageId":"d51d073aa4aad22750edcc0214caf0dedd46df79.1699991639.git.gitgitgadget@gmail.com","threadId":"60483","inReplyTo":"pull.1609.v2.git.1699991638.gitgitgadget@gmail.com","subject":"[PATCH v2 10/10] t/perf: add perf tests for for-each-ref","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-11-14T19:53:58Z","receivedAt":"2023-11-14T19:54:22Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye <vdye@github.com>\n\nAdd performance tests for 'for-each-ref'. The tests exercise different\ncombinations of filters/formats/options, as well as the overall performance\nof 'git for-each-ref | git cat-file --batch-check' to demonstrate the\nperformance difference vs. 'git for-each-ref' with \"%(*fieldname)\" format\nspecifiers.\n\nAll tests are run against a repository with 40k loose refs - 10k commits,\neach having a unique:\n\n- branch\n- custom ref (refs/custom/special_*)\n- annotated tag pointing at the commit\n- annotated tag pointing at the other annotated tag (i.e., a nested tag)\n\nAfter those tests are finished, the refs are packed with 'pack-refs --all'\nand the same tests are rerun.\n\nSigned-off-by: Victoria Dye <vdye@github.com>\n---\n t/perf/p6300-for-each-ref.sh | 87 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 87 insertions(+)\n create mode 100755 t/perf/p6300-for-each-ref.sh\n\ndiff --git a/t/perf/p6300-for-each-ref.sh b/t/perf/p6300-for-each-ref.sh\nnew file mode 100755\nindex 00000000000..fa7289c7522\n--- /dev/null\n+++ b/t/perf/p6300-for-each-ref.sh\n@@ -0,0 +1,87 @@\n+#!/bin/sh\n+\n+test_description='performance of for-each-ref'\n+. ./perf-lib.sh\n+\n+test_perf_fresh_repo\n+\n+ref_count_per_type=10000\n+test_iteration_count=10\n+\n+test_expect_success \"setup\" '\n+\ttest_commit_bulk $(( 1 + $ref_count_per_type )) &&\n+\n+\t# Create refs\n+\ttest_seq $ref_count_per_type |\n+\t\tsed \"s,.*,update refs/heads/branch_& HEAD~&\\nupdate refs/custom/special_& HEAD~&,\" |\n+\t\tgit update-ref --stdin &&\n+\n+\t# Create annotated tags\n+\tfor i in $(test_seq $ref_count_per_type)\n+\tdo\n+\t\t# Base tags\n+\t\techo \"tag tag_$i\" &&\n+\t\techo \"mark :$i\" &&\n+\t\techo \"from HEAD~$i\" &&\n+\t\tprintf \"tagger %s <%s> %s\\n\" \\\n+\t\t\t\"$GIT_COMMITTER_NAME\" \\\n+\t\t\t\"$GIT_COMMITTER_EMAIL\" \\\n+\t\t\t\"$GIT_COMMITTER_DATE\" &&\n+\t\techo \"data <<EOF\" &&\n+\t\techo \"tag $i\" &&\n+\t\techo \"EOF\" &&\n+\n+\t\t# Nested tags\n+\t\techo \"tag nested_$i\" &&\n+\t\techo \"from :$i\" &&\n+\t\tprintf \"tagger %s <%s> %s\\n\" \\\n+\t\t\t\"$GIT_COMMITTER_NAME\" \\\n+\t\t\t\"$GIT_COMMITTER_EMAIL\" \\\n+\t\t\t\"$GIT_COMMITTER_DATE\" &&\n+\t\techo \"data <<EOF\" &&\n+\t\techo \"nested tag $i\" &&\n+\t\techo \"EOF\" || return 1\n+\tdone | git fast-import\n+'\n+\n+test_for_each_ref () {\n+\ttitle=\"for-each-ref\"\n+\tif test $# -gt 0; then\n+\t\ttitle=\"$title ($1)\"\n+\t\tshift\n+\tfi\n+\targs=\"$@\"\n+\n+\ttest_perf \"$title\" \"\n+\t\tfor i in \\$(test_seq $test_iteration_count); do\n+\t\t\tgit for-each-ref $args >/dev/null\n+\t\tdone\n+\t\"\n+}\n+\n+run_tests () {\n+\ttest_for_each_ref \"$1\"\n+\ttest_for_each_ref \"$1, no sort\" --no-sort\n+\ttest_for_each_ref \"$1, --count=1\" --count=1\n+\ttest_for_each_ref \"$1, --count=1, no sort\" --no-sort --count=1\n+\ttest_for_each_ref \"$1, tags\" refs/tags/\n+\ttest_for_each_ref \"$1, tags, no sort\" --no-sort refs/tags/\n+\ttest_for_each_ref \"$1, tags, dereferenced\" '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n+\ttest_for_each_ref \"$1, tags, dereferenced, no sort\" --no-sort '--format=\"%(refname) %(objectname) %(*objectname)\"' refs/tags/\n+\n+\ttest_perf \"for-each-ref ($1, tags) + cat-file --batch-check (dereferenced)\" \"\n+\t\tfor i in \\$(test_seq $test_iteration_count); do\n+\t\t\tgit for-each-ref --format='%(objectname)^{} %(refname) %(objectname)' refs/tags/ | \\\n+\t\t\t\tgit cat-file --batch-check='%(objectname) %(rest)' >/dev/null\n+\t\tdone\n+\t\"\n+}\n+\n+run_tests \"loose\"\n+\n+test_expect_success 'pack refs' '\n+\tgit pack-refs --all\n+'\n+run_tests \"packed\"\n+\n+test_done\n-- \ngitgitgadget\n"},{"id":"484963","messageId":"xmqqcywas1ty.fsf@gitster.g","threadId":"60483","inReplyTo":"074da1ff3e85927324c42a3fa65e4239f051cd70.1699991638.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/10] ref-filter.c: really don't sort when using --no-sort","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-16T05:29:29Z","receivedAt":"2023-11-16T05:29:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Victoria Dye <vdye@github.com>\n>\n> When '--no-sort' is passed to 'for-each-ref', 'tag', and 'branch', the\n> printed refs are still sorted by ascending refname. Change the handling of\n> sort options in these commands so that '--no-sort' to truly disables\n> sorting.\n\n\"to truly disables\" -> \"truly disables\" I think?\n\n> '--no-sort' does not disable sorting in these commands is because their\n\n\"'--no-sort' does not\" -> \"The reason why '--no-sort' does not\", or\n\"is because\" -> \"because\".\n\n> option parsing does not distinguish between \"the absence of '--sort'\"\n> (and/or values for tag.sort & branch.sort) and '--no-sort'. Both result in\n> an empty 'sorting_options' string list, which is parsed by\n> 'ref_sorting_options()' to create the 'struct ref_sorting *' for the\n> command. If the string list is empty, 'ref_sorting_options()' interprets\n> that as \"the absence of '--sort'\" and returns the default ref sorting\n> structure (equivalent to \"refname\" sort).\n>\n> To handle '--no-sort' properly while preserving the \"refname\" sort in the\n> \"absence of --sort'\" case, first explicitly add \"refname\" to the string list\n> *before* parsing options. This alone doesn't actually change any behavior,\n> since 'compare_refs()' already falls back on comparing refnames if two refs\n> are equal w.r.t all other sort keys.\n>\n> Now that the string list is populated by default, '--no-sort' is the only\n> way to empty the 'sorting_options' string list. Update\n> 'ref_sorting_options()' to return a NULL 'struct ref_sorting *' if the\n> string list is empty, and add a condition to 'ref_array_sort()' to skip the\n> sort altogether if the sort structure is NULL. Note that other functions\n> using 'struct ref_sorting *' do not need any changes because they already\n> ignore NULL values.\n\nNice.\n\n> Finally, remove the condition around sorting in 'ls-remote', since it's no\n> longer necessary. Unlike 'for-each-ref' et. al., it does *not* do any\n> sorting by default. This default is preserved by simply leaving its sort key\n> string list empty before parsing options; if no additional sort keys are\n> set, 'struct ref_sorting *' is NULL and sorting is skipped.\n\nDoubly nice.\n\n> diff --git a/ref-filter.c b/ref-filter.c\n> index e4d3510e28e..7250089b7c6 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -3142,7 +3142,8 @@ void ref_sorting_set_sort_flags_all(struct ref_sorting *sorting,\n>  \n>  void ref_array_sort(struct ref_sorting *sorting, struct ref_array *array)\n>  {\n> -\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n> +\tif (sorting)\n> +\t\tQSORT_S(array->items, array->nr, compare_refs, sorting);\n>  }\n\nNice.  We do allow passing NULL to ref_sorting_release(), and we can\nreturn NULL from ref_sorting_options(), and allowing NULL to be\npassed to this function makes it easier for the callers to deal with\nthe case where no sorting is specified.\n\n> diff --git a/t/t3200-branch.sh b/t/t3200-branch.sh\n> index 3182abde27f..9918ba05dec 100755\n> --- a/t/t3200-branch.sh\n> +++ b/t/t3200-branch.sh\n> @@ -1570,9 +1570,10 @@ test_expect_success 'tracking with unexpected .fetch refspec' '\n>  \n>  test_expect_success 'configured committerdate sort' '\n>  \tgit init -b main sort &&\n> +\ttest_config -C sort branch.sort \"committerdate\" &&\n> +\n>  \t(\n>  \t\tcd sort &&\n> -\t\tgit config branch.sort committerdate &&\n>  \t\ttest_commit initial &&\n>  \t\tgit checkout -b a &&\n>  \t\ttest_commit a &&\n> @@ -1592,9 +1593,10 @@ test_expect_success 'configured committerdate sort' '\n>  '\n>  \n>  test_expect_success 'option override configured sort' '\n> +\ttest_config -C sort branch.sort \"committerdate\" &&\n> +\n>  \t(\n>  \t\tcd sort &&\n> -\t\tgit config branch.sort committerdate &&\n>  \t\tgit branch --sort=refname >actual &&\n>  \t\tcat >expect <<-\\EOF &&\n>  \t\t  a\n\nThe above two are not strictly necessary for the purpose of this\npatch, in that the tests that come after these tests do not care if\nthe branch.sort configuration variable is set in the \"sort\"\nrepository, as they set their own value before doing their test.\n\nBut of course, cleaning up after yourself with test_config and\nfriends is a good idea regardless, and a handful of new tests added\nafter this point follow the same pattern.  Good.\n\n> @@ -1606,10 +1608,70 @@ test_expect_success 'option override configured sort' '\n>  \t)\n>  '\n>  \n> +test_expect_success '--no-sort cancels config sort keys' '\n> +\ttest_config -C sort branch.sort \"-refname\" &&\n> +\n> +\t(\n> +\t\tcd sort &&\n> +\n> +\t\t# objecttype is identical for all of them, so sort falls back on\n> +\t\t# default (ascending refname)\n\nInteresting.\n\n> +\t\tgit branch \\\n> +\t\t\t--no-sort \\\n> +\t\t\t--sort=\"objecttype\" >actual &&\n> +\t\tcat >expect <<-\\EOF &&\n> +\t\t  a\n> +\t\t* b\n> +\t\t  c\n> +\t\t  main\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t)\n> +\n> +'\n> +\n> +test_expect_success '--no-sort cancels command line sort keys' '\n> +\t(\n> +\t\tcd sort &&\n> +\n> +\t\t# objecttype is identical for all of them, so sort falls back on\n> +\t\t# default (ascending refname)\n> +\t\tgit branch \\\n> +\t\t\t--sort=\"-refname\" \\\n> +\t\t\t--no-sort \\\n> +\t\t\t--sort=\"objecttype\" >actual &&\n\nOK, this exercises the same \"--no-sort cleans the slate\" as before,\nand for this one it is essential that we lack branch.sort after the\nprevious step is done, which is ensured thanks to the use of\ntest_config in the previous one.  Nice.\n\n> +\t\tcat >expect <<-\\EOF &&\n> +\t\t  a\n> +\t\t* b\n> +\t\t  c\n> +\t\t  main\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success '--no-sort without subsequent --sort prints expected branches' '\n> +\t(\n> +\t\tcd sort &&\n> +\n> +\t\t# Sort the results with `sort` for a consistent comparison\n> +\t\t# against expected\n> +\t\tgit branch --no-sort | sort >actual &&\n> +\t\tcat >expect <<-\\EOF &&\n> +\t\t  a\n> +\t\t  c\n> +\t\t  main\n> +\t\t* b\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n>  test_expect_success 'invalid sort parameter in configuration' '\n> +\ttest_config -C sort branch.sort \"v:notvalid\" &&\n> +\n>  \t(\n>  \t\tcd sort &&\n> -\t\tgit config branch.sort \"v:notvalid\" &&\n>  \n>  \t\t# this works in the \"listing\" mode, so bad sort key\n>  \t\t# is a dying offence.\n\nWith such an invalid configuration value set, running the command\nwith \"--no-sort\" would stop the command from failing?  Is that worth\nprotecting with a new test, I wonder.\n\nOverall very nicely done.\n\nThanks.\n"},{"id":"484964","messageId":"xmqq8r6ys1dw.fsf@gitster.g","threadId":"60483","inReplyTo":"187b1d6610f96ba16bb7e1ff80d1c994a67b8753.1699991638.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/10] ref-filter.h: add functions for filter/format & format-only","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-16T05:39:07Z","receivedAt":"2023-11-16T05:39:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> This consolidates much of the code used to filter and format refs in\n> 'builtin/for-each-ref.c', 'builtin/tag.c', and 'builtin/branch.c', reducing\n> duplication and simplifying the future changes needed to optimize the filter\n> & format process.\n>\n> Signed-off-by: Victoria Dye <vdye@github.com>\n> ---\n>  builtin/branch.c       | 33 +++++++++++++++++----------------\n>  builtin/for-each-ref.c | 27 +--------------------------\n>  builtin/tag.c          | 23 +----------------------\n>  ref-filter.c           | 35 +++++++++++++++++++++++++++++++++++\n>  ref-filter.h           | 14 ++++++++++++++\n>  5 files changed, 68 insertions(+), 64 deletions(-)\n\nThe amount of existing duplication of code is rather surprising, and\nthis patch nicely refactors to improve.  Good.\n\nThanks.\n"},{"id":"484966","messageId":"xmqq4jhms0xq.fsf@gitster.g","threadId":"60483","inReplyTo":"48254d8e161de7f0e165510c06801195f9b0a8fd.1699991638.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/10] ref-filter.c: use peeled tag for '*' format fields","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-16T05:48:49Z","receivedAt":"2023-11-16T05:48:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Victoria Dye <vdye@github.com>\n>\n> In most builtins ('rev-parse <revision>^{}', 'show-ref --dereference'),\n> \"dereferencing\" a tag refers to a recursive peel of the tag object. Unlike\n> these cases, the dereferencing prefix ('*') in 'for-each-ref' format\n> specifiers triggers only a single, non-recursive dereference of a given tag\n> object. For most annotated tags, a single dereference is all that is needed\n> to access the tag's associated commit or tree; \"recursive\" and\n> \"non-recursive\" dereferencing are functionally equivalent in these cases.\n> However, nested tags (annotated tags whose target is another annotated tag)\n> dereferenced once return another tag, where a recursive dereference would\n> return the commit or tree.\n\nThis may be the only potentially controversial step in the series.\n\n> -\t/*\n> -\t * NEEDSWORK: This derefs tag only once, which\n> -\t * is good to deal with chains of trust, but\n> -\t * is not consistent with what deref_tag() does\n> -\t * which peels the onion to the core.\n> -\t */\n>  \treturn get_object(ref, 1, &obj, &oi_deref, err);\n>  }\n\nVery nice to see an ancient comment I added at 9f613ddd (Add\ngit-for-each-ref: helper for language bindings, 2006-09-15) finally\ngo.\n\nThanks.\n"},{"id":"484976","messageId":"20231116120627.3029-1-oystwa@gmail.com","threadId":"60483","inReplyTo":"adac101bc6022d5477371d6a94225f38da7fffee.1699991638.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/10] ref-filter.h: add max_count and omit_empty to ref_format","fromName":"Øystein Walle","fromEmail":"oystwa@gmail.com","sentAt":"2023-11-16T12:06:27Z","receivedAt":"2023-11-16T12:06:46Z","isPatch":true,"sender":{"key":"oystwa@gmail.com","avatar":"https://avatars.githubusercontent.com/u/794585?v=4"},"body":"Victoria Dye <vdye@github.com> writes:\n\n> diff --git a/ref-filter.h b/ref-filter.h\n> index 1524bc463a5..d87d61238b7 100644\n> --- a/ref-filter.h\n> +++ b/ref-filter.h\n> @@ -92,6 +92,11 @@ struct ref_format {\n>  \n>  \t/* List of bases for ahead-behind counts. */\n>  \tstruct string_list bases;\n> +\n> +\tstruct {\n> +\t\tint max_count;\n> +\t\tint omit_empty;\n> +\t} array_opts;\n>  };\n\nWhat the benefit of having them in a nested struct is compared to just\ntwo distinct members?\n\nRegardless this is the kind of deduplication I wanted to achieve when I\nadded --omit-empty, but never did. Either way, I meant to ack this in\nthe last round never got around to it. Nice work.\n\nAcked-by: Øystein Walle <oystwa@gmail.com>\n\nØsse\n"}]}