{"thread":{"id":"59347","subject":"[PATCH 1/8] ahead-behind: create empty builtin","startedAt":"2023-03-06T14:07:28Z","lastAt":"2023-03-20T20:41:14Z","messageCount":90,"participants":["Derrick Stolee via GitGitGadget","Taylor Blau via GitGitGadget","Junio C Hamano","Derrick Stolee","Taylor Blau","Jeff King","Phillip Wood","Ævar Arnfjörð Bjarmason","Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"473049","messageId":"0fd18b6d740f1e8a6f62a25947bc3ad49c2674a6.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 1/8] ahead-behind: create empty builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:31Z","receivedAt":"2023-03-06T14:07:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git ahead-behind' builtin _will_ allow users to specify multiple\ntip revisions relative to a common base and _will_ report the number of\ncommits on each side of the symmetric difference between each tip and\nthe base. However, that algorithm is not implemented yet and instead\nthis change introduces the builtin and the basic boilerplate for a new\nbuiltin.\n\nThis builtin could be replaced with multiple invocations of 'git\nrev-list --count ..' (for ahead values) and 'git rev-list\n--count ..' (for behind values). However, it is important to\nbe able to batch these calls into a single process.\n\nFor example, we will be able to track all local branches relative to an\nupstream branch using an invocation such as\n\n git for-each-ref --format=%(refname) refs/heads/* |\n git ahead-behind --base=origin/main --stdin\n\nThis would report each local branch and how far ahead or behind it is\nrelative to the remote branch 'origin/main'. This could be used to\nsignal some branches are very old and need to be updated via 'git\nrebase' or deleted. We will see in future changes how such commit\ncounting can be done efficiently within a single process (and a single\ncommit walk) instead of multiple processes.\n\nFor now, only 'git ahead-behind -h' works, and the builtin reports\nfailure and shows the usage if the '--base' option is skipped. The\ndocumentation is light. These will be updated in the coming changes.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-ahead-behind.txt | 62 ++++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/ahead-behind.c | 30 +++++++++++++++\n git.c | 1 +\n t/t4218-ahead-behind.sh | 17 ++++++++\n 7 files changed, 113 insertions(+)\n create mode 100644 Documentation/git-ahead-behind.txt\n create mode 100644 builtin/ahead-behind.c\n create mode 100755 t/t4218-ahead-behind.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex e875c590545..cc064a4817a 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -14,6 +14,7 @@\n /bin-wrappers/\n /git\n /git-add\n+/git-ahead-behind\n /git-am\n /git-annotate\n /git-apply\ndiff --git a/Documentation/git-ahead-behind.txt b/Documentation/git-ahead-behind.txt\nnew file mode 100644\nindex 00000000000..0e2f989a1a0\n--- /dev/null\n+++ b/Documentation/git-ahead-behind.txt\n@@ -0,0 +1,62 @@\n+git-ahead-behind(1)\n+===================\n+\n+NAME\n+----\n+git-ahead-behind - Count the commits on each side of a revision range\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git ahead-behind' --base= [ --stdin | ]\n+\n+DESCRIPTION\n+-----------\n+\n+Given a list of commit ranges, report the number of commits reachable from\n+each of the sides of the range, but not the other. Consider a commit range\n+specified as `...`, allowing for the following definitions:\n+\n+* The `` is *ahead* of `` by the number of commits reachable\n+ from `` but not reachable from ``. This is the same as the\n+ number of the commits in the range `..`.\n+\n+* The `` is *behind* `` by the number of commits reachable from\n+ `` but not reachble from ``. This is the same as the number\n+ of commits in the range `..`.\n+\n+The sum of the ahead and behind counts equals the number of commits in the\n+symmetric difference, the range `...`.\n+\n+Multiple revisions may be specified, and they are all compared against a\n+common base revision, as specified by the `--base` option. The values are\n+reported to stdout one line at a time as follows:\n+\n+---\n+ \n+---\n+\n+There will be exactly one line per input revision, but the lines may be\n+in an arbitrary order.\n+\n+\n+OPTIONS\n+-------\n+--base=::\n+\tSpecify that `` should be used as a common base for all\n+\tprovided revisions that are not specified in the form of a range.\n+\n+--stdin::\n+\tRead revision tips and ranges from stdin instead of from the\n+\tcommand-line.\n+\n+\n+SEE ALSO\n+--------\n+linkgit:git-branch[1]\n+linkgit:git-rev-list[1]\n+linkgit:git-tag[1]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 50ee51fde32..691f84e8d4e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1199,6 +1199,7 @@ LIB_OBJS += xdiff-interface.o\n LIB_OBJS += zlib.o\n \n BUILTIN_OBJS += builtin/add.o\n+BUILTIN_OBJS += builtin/ahead-behind.o\n BUILTIN_OBJS += builtin/am.o\n BUILTIN_OBJS += builtin/annotate.o\n BUILTIN_OBJS += builtin/apply.o\ndiff --git a/builtin.h b/builtin.h\nindex 46cc7897898..1ae168fa3e3 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -108,6 +108,7 @@ void setup_auto_pager(const char *cmd, int def);\n int is_builtin(const char *s);\n \n int cmd_add(int argc, const char **argv, const char *prefix);\n+int cmd_ahead_behind(int argc, const char **argv, const char *prefix);\n int cmd_am(int argc, const char **argv, const char *prefix);\n int cmd_annotate(int argc, const char **argv, const char *prefix);\n int cmd_apply(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/ahead-behind.c b/builtin/ahead-behind.c\nnew file mode 100644\nindex 00000000000..a56cc565def\n--- /dev/null\n+++ b/builtin/ahead-behind.c\n@@ -0,0 +1,30 @@\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"config.h\"\n+\n+static const char * const ahead_behind_usage[] = {\n+\tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n+\tNULL\n+};\n+\n+int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *base_ref = NULL;\n+\tint from_stdin = 0;\n+\n+\tstruct option ahead_behind_opts[] = {\n+\t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n+\t\tOPT_BOOL(0 , \"stdin\", &from_stdin, N_(\"read rev names from stdin\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, ahead_behind_opts,\n+\t\t\t ahead_behind_usage, PARSE_OPT_KEEP_UNKNOWN_OPT);\n+\n+\tif (!base_ref)\n+\t\tusage_with_options(ahead_behind_usage, ahead_behind_opts);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 6171fd6769d..64e3d493561 100644\n--- a/git.c\n+++ b/git.c\n@@ -467,6 +467,7 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \n static struct cmd_struct commands[] = {\n \t{ \"add\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"ahead-behind\", cmd_ahead_behind, RUN_SETUP },\n \t{ \"am\", cmd_am, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"annotate\", cmd_annotate, RUN_SETUP },\n \t{ \"apply\", cmd_apply, RUN_SETUP_GENTLY },\ndiff --git a/t/t4218-ahead-behind.sh b/t/t4218-ahead-behind.sh\nnew file mode 100755\nindex 00000000000..bc08f1207a0\n--- /dev/null\n+++ b/t/t4218-ahead-behind.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+\n+test_description='git ahead-behind command-line options'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'git ahead-behind -h' '\n+\ttest_must_fail git ahead-behind -h >out &&\n+\tgrep \"usage:\" out\n+'\n+\n+test_expect_success 'git ahead-behind without --base' '\n+\ttest_must_fail git ahead-behind HEAD 2>err &&\n+\tgrep \"usage:\" err\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"473050","messageId":"08fc0710017d4b6178a8c5772e0f15fc69c84ad6.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 2/8] ahead-behind: parse tip references","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:32Z","receivedAt":"2023-03-06T14:07:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore implementing the logic to compute the ahead/behind counts, parse\nthe unknown options as commits and place them in a string_list.\n\nBe sure to error out when the reference is not found.\n\nCo-authored-by: Taylor Blau \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n builtin/ahead-behind.c | 39 +++++++++++++++++++++++++++++++++++++++\n t/t4218-ahead-behind.sh | 10 ++++++++++\n 2 files changed, 49 insertions(+)\n\ndiff --git a/builtin/ahead-behind.c b/builtin/ahead-behind.c\nindex a56cc565def..c1212cc8d46 100644\n--- a/builtin/ahead-behind.c\n+++ b/builtin/ahead-behind.c\n@@ -1,16 +1,31 @@\n #include \"builtin.h\"\n #include \"parse-options.h\"\n #include \"config.h\"\n+#include \"commit.h\"\n \n static const char * const ahead_behind_usage[] = {\n \tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n \tNULL\n };\n \n+static int handle_arg(struct string_list *tips, const char *arg)\n+{\n+\tstruct string_list_item *item;\n+\tstruct commit *c = lookup_commit_reference_by_name(arg);\n+\n+\tif (!c)\n+\t\treturn error(_(\"could not resolve '%s'\"), arg);\n+\n+\titem = string_list_append(tips, arg);\n+\titem->util = c;\n+\treturn 0;\n+}\n+\n int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n {\n \tconst char *base_ref = NULL;\n \tint from_stdin = 0;\n+\tstruct string_list tips = STRING_LIST_INIT_DUP;\n \n \tstruct option ahead_behind_opts[] = {\n \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n@@ -26,5 +41,29 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\tif (from_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n+\t\t\tif (!line.len)\n+\t\t\t\tbreak;\n+\n+\t\t\tif (handle_arg(&tips, line.buf))\n+\t\t\t\treturn 1;\n+\t\t}\n+\n+\t\tstrbuf_release(&line);\n+\t} else {\n+\t\tint i;\n+\t\tfor (i = 0; i < argc; ++i) {\n+\t\t\tif (handle_arg(&tips, argv[i]))\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\t/* Early return for no tips. */\n+\tif (!tips.nr)\n+\t\treturn 0;\n+\n \treturn 0;\n }\ndiff --git a/t/t4218-ahead-behind.sh b/t/t4218-ahead-behind.sh\nindex bc08f1207a0..3b8b9dc9887 100755\n--- a/t/t4218-ahead-behind.sh\n+++ b/t/t4218-ahead-behind.sh\n@@ -14,4 +14,14 @@ test_expect_success 'git ahead-behind without --base' '\n \tgrep \"usage:\" err\n '\n \n+test_expect_success 'git ahead-behind with broken tip' '\n+\ttest_must_fail git ahead-behind --base=HEAD bogus 2>err &&\n+\tgrep \"could not resolve '\\''bogus'\\''\" err\n+'\n+\n+test_expect_success 'git ahead-behind without tips' '\n+\tgit ahead-behind --base=HEAD 2>err &&\n+\ttest_must_be_empty err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473051","messageId":"pull.1489.git.1678111598.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":null,"subject":"[PATCH 0/8] ahead-behind: new builtin for counting multiple commit ranges","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:30Z","receivedAt":"2023-03-06T14:07:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series introduces the 'git ahead-behind' builtin, which has been used\nat $DAYJOB for many years, but took many forms before landing on the current\nversion.\n\nThe main goal of the builtin is to compare multiple references against a\ncommon base reference. The comparison is number of commits that are in each\nside of the symmtric difference of their reachable sets. A commit C is\n\"ahead\" of a commit B by the number of commits in B..C (reachable from C but\nnot reachable from B). Similarly, the commit C is \"behind\" the commit B by\nthe number of commits in C..B (reachable from B but not reachable from C).\n\nThese numbers can be computed by 'git rev-list --count B..C' and 'git\nrev-list --count C..B', but there are common needs that benefit from having\nthe checks being done in the same process:\n\n 1. Our \"branches\" page lists ahead/behind counts for each listed branch as\n compared to the repo's default branch. This can be done with a single\n 'git ahead-behind' process.\n 2. When a branch is updated, a background job checks if any pull requests\n that target that branch should be closed because their branches were\n merged implicitly by that update. These queries can e batched into 'git\n ahead-behind' calls.\n\nIn that second example, we don't need the full ahead/behind counts (although\nit is sufficient to look for branches that are \"zero commits ahead\", meaning\nthey are reachable from the base), so this builtin has an extra '--contains'\nmode that only checks reachability from the base to each of the tips. 'git\nahead-behind --contains' is sort of the reverse of 'git branch --contains'.\n\nThe series starts with some basic boilerplate and argument parsing, along\nwith error conditions for missing objects. To avoid TOCTOU races, an\n'--ignore-missing' option allows being flexible when a tip reference does\nnot exist. This is all covered in patches 1-3.\n\nPatches 4-6 introduce a new method: ensure_generations_valid(). Patch 4 does\nsome refactoring of the existing generation number computations to make it\nmore generic, and patch 5 updates the definition of\ncommit_graph_generation() slightly, making way for patch 6 to implement the\nmethod. With an existing commit-graph file, the commits that are not present\nin the file are considered as having generation number \"infinity\". This is\nuseful for most of our reachability queries to this point, since those\ncommits are \"above\" the ones tracked by the commit-graph. When these commits\nare low in number, then there is very little performance cost and zero\ncorrectness cost.\n\nHowever, we will see that the ahead/behind computation requires accurate\ngeneration numbers to avoid overcounting. Thus, ensure_generations_valid()\nis a way to specify a list of commits that need generation numbers computed\nbefore continuing. It's a no-op if all of those commits are in the\ncommit-graph file. It's expensive if the commit-graph doesn't exist.\nHowever, 'git ahead-behind' computations are likely to be slow no matter\nwhat without a commit-graph, so assuming an existing commit-graph file is\nreasonable. If we find sufficient desire to have an implementation that does\nnot have this requirement, we could create a second implementation and\ntoggle to it when generation_numbers_enabled() returns false.\n\nPatch 7 implements the ahead-behind algorithm, as well as integrating the\nbuiltin with that implementation. It's a long commit message, so hopefully\nit explains the algorithm sufficiently.\n\nPatch 8 implements the --contains option, which is another algorithm, but\nmore similar to other depth-first searches that already exist in\ncommit-reach.c.\n\nThanks, -Stolee\n\nDerrick Stolee (7):\n ahead-behind: create empty builtin\n ahead-behind: parse tip references\n ahead-behind: implement --ignore-missing option\n commit-graph: combine generation computations\n commit-graph: return generation from memory\n ahead-behind: implement ahead_behind() logic\n ahead-behind: add --contains mode\n\nTaylor Blau (1):\n commit-graph: introduce `ensure_generations_valid()`\n\n .gitignore | 1 +\n Documentation/git-ahead-behind.txt | 78 +++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/ahead-behind.c | 121 +++++++++++++++++\n commit-graph.c | 209 +++++++++++++++++++----------\n commit-graph.h | 7 +\n commit-reach.c | 205 ++++++++++++++++++++++++++++\n commit-reach.h | 37 +++++\n git.c | 1 +\n t/perf/p1500-graph-walks.sh | 29 ++++\n t/t4218-ahead-behind.sh | 162 ++++++++++++++++++++++\n t/t5318-commit-graph.sh | 2 +-\n t/t6600-test-reach.sh | 120 +++++++++++++++++\n 14 files changed, 904 insertions(+), 70 deletions(-)\n create mode 100644 Documentation/git-ahead-behind.txt\n create mode 100644 builtin/ahead-behind.c\n create mode 100755 t/perf/p1500-graph-walks.sh\n create mode 100755 t/t4218-ahead-behind.sh\n\n\nbase-commit: d15644fe0226af7ffc874572d968598564a230dd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1489%2Fderrickstolee%2Fstolee%2Fupstream-ahead-behind-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1489/derrickstolee/stolee/upstream-ahead-behind-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1489\n-- \ngitgitgadget\n"},{"id":"473052","messageId":"b1d022c7cac5aed6e2d64b45f20dba5b3297536c.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 3/8] ahead-behind: implement --ignore-missing option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:33Z","receivedAt":"2023-03-06T14:07:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen parsing the tip revisions from the ahead-behind inputs, it is\nimportant to check that those tips exist before adding them to the list\nfor computation. The previous change caused the builtin to return with\nerrors if the revisions could not be resolved.\n\nHowever, when running 'git ahead-behind' in an environment with\nconcurrent edits, such as a Git server, then the references could be\ndeleted from underneath the caller between reading the reference list\nand starting the 'git ahead-behind' process. Avoid this race by allowing\nthe caller to specify '--ignore-missing' and continue using the\ninformation that is still available.\n\nCo-authored-by: Taylor Blau \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-ahead-behind.txt | 6 ++++++\n builtin/ahead-behind.c | 8 +++++++-\n t/t4218-ahead-behind.sh | 6 ++++++\n 3 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-ahead-behind.txt b/Documentation/git-ahead-behind.txt\nindex 0e2f989a1a0..2dd5147f6b2 100644\n--- a/Documentation/git-ahead-behind.txt\n+++ b/Documentation/git-ahead-behind.txt\n@@ -50,6 +50,12 @@ OPTIONS\n \tRead revision tips and ranges from stdin instead of from the\n \tcommand-line.\n \n+--ignore-missing::\n+\tWhen parsing tip references, ignore any references that are not\n+\tfound. This is useful when operating in an environment where a\n+\treference could be deleted between reading the reference and\n+\tstarting the `git ahead-behind` process.\n+\n \n SEE ALSO\n --------\ndiff --git a/builtin/ahead-behind.c b/builtin/ahead-behind.c\nindex c1212cc8d46..e4f65fc0548 100644\n--- a/builtin/ahead-behind.c\n+++ b/builtin/ahead-behind.c\n@@ -8,13 +8,18 @@ static const char * const ahead_behind_usage[] = {\n \tNULL\n };\n \n+static int ignore_missing;\n+\n static int handle_arg(struct string_list *tips, const char *arg)\n {\n \tstruct string_list_item *item;\n \tstruct commit *c = lookup_commit_reference_by_name(arg);\n \n-\tif (!c)\n+\tif (!c) {\n+\t\tif (ignore_missing)\n+\t\t\treturn 0;\n \t\treturn error(_(\"could not resolve '%s'\"), arg);\n+\t}\n \n \titem = string_list_append(tips, arg);\n \titem->util = c;\n@@ -30,6 +35,7 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n \tstruct option ahead_behind_opts[] = {\n \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n \t\tOPT_BOOL(0 , \"stdin\", &from_stdin, N_(\"read rev names from stdin\")),\n+\t\tOPT_BOOL(0 , \"ignore-missing\", &ignore_missing, N_(\"ignore missing tip references\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/t/t4218-ahead-behind.sh b/t/t4218-ahead-behind.sh\nindex 3b8b9dc9887..56f16515896 100755\n--- a/t/t4218-ahead-behind.sh\n+++ b/t/t4218-ahead-behind.sh\n@@ -19,6 +19,12 @@ test_expect_success 'git ahead-behind with broken tip' '\n \tgrep \"could not resolve '\\''bogus'\\''\" err\n '\n \n+test_expect_success 'git ahead-behind with broken tip and --ignore-missing' '\n+\tgit ahead-behind --base=HEAD --ignore-missing bogus 2>err >out &&\n+\ttest_must_be_empty err &&\n+\ttest_must_be_empty out\n+'\n+\n test_expect_success 'git ahead-behind without tips' '\n \tgit ahead-behind --base=HEAD 2>err &&\n \ttest_must_be_empty err\n-- \ngitgitgadget\n\n"},{"id":"473053","messageId":"853891c0b14b2a97c980a436a755f9005415ea7a.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 4/8] commit-graph: combine generation computations","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:34Z","receivedAt":"2023-03-06T14:07:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis patch extracts the common code used to compute topological levels\nand corrected committer dates into a common routine,\ncompute_reachable_generation_numbers_1().\n\nThis new routine dispatches to call the necessary functions to get and\nset the generation number for a given commit through a vtable (the\ncompute_generation_info struct).\n\nComputing the generation number itself is done in\ncompute_generation_from_max(), which dispatches its implementation based\non the generation version requested, or issuing a BUG() for unrecognized\ngeneration versions.\n\nThis patch cleans up the two places that currently compute topological\nlevels and corrected commit dates by reducing the amount of duplicated\ncode. It also makes it possible to introduce a function which\ndynamically computes those values for commits that aren't stored in a\ncommit-graph, which will be required for the forthcoming ahead-behind\nrewrite.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 171 +++++++++++++++++++++++++++++++------------------\n 1 file changed, 107 insertions(+), 64 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b3..deccf984a0d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1446,24 +1446,53 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n-static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+struct compute_generation_info {\n+\tstruct repository *r;\n+\tstruct packed_commit_list *commits;\n+\tstruct progress *progress;\n+\tint progress_cnt;\n+\n+\ttimestamp_t (*get_generation)(struct commit *c, void *data);\n+\tvoid (*set_generation)(struct commit *c, timestamp_t gen, void *data);\n+\tvoid *data;\n+};\n+\n+static timestamp_t compute_generation_from_max(struct commit *c,\n+\t\t\t\t\t timestamp_t max_gen,\n+\t\t\t\t\t int generation_version)\n+{\n+\tswitch (generation_version) {\n+\tcase 1: /* topological levels */\n+\t\tif (max_gen > GENERATION_NUMBER_V1_MAX - 1)\n+\t\t\tmax_gen = GENERATION_NUMBER_V1_MAX - 1;\n+\t\treturn max_gen + 1;\n+\n+\tcase 2: /* corrected commit date */\n+\t\tif (c->date && c->date > max_gen)\n+\t\t\tmax_gen = c->date - 1;\n+\t\treturn max_gen + 1;\n+\n+\tdefault:\n+\t\tBUG(\"attempting unimplemented version\");\n+\t}\n+}\n+\n+static void compute_reachable_generation_numbers_1(\n+\t\t\tstruct compute_generation_info *info,\n+\t\t\tint generation_version)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n \n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n-\t\t\t\t\tctx->commits.nr);\n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\tuint32_t level;\n+\tfor (i = 0; i < info->commits->nr; i++) {\n+\t\tstruct commit *c = info->commits->list[i];\n+\t\ttimestamp_t gen;\n+\t\trepo_parse_commit(info->r, c);\n+\t\tgen = info->get_generation(c, info->data);\n \n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n+\t\tdisplay_progress(info->progress, info->progress_cnt + 1);\n \n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (level != GENERATION_NUMBER_ZERO)\n+\t\tif (gen != GENERATION_NUMBER_ZERO && gen != GENERATION_NUMBER_INFINITY)\n \t\t\tcontinue;\n \n \t\tcommit_list_insert(c, &list);\n@@ -1471,38 +1500,91 @@ static void compute_topological_levels(struct write_commit_graph_context *ctx)\n \t\t\tstruct commit *current = list->item;\n \t\t\tstruct commit_list *parent;\n \t\t\tint all_parents_computed = 1;\n-\t\t\tuint32_t max_level = 0;\n+\t\t\tuint32_t max_gen = 0;\n \n \t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tlevel = *topo_level_slab_at(ctx->topo_levels, parent->item);\n+\t\t\t\trepo_parse_commit(info->r, parent->item);\n+\t\t\t\tgen = info->get_generation(parent->item, info->data);\n \n-\t\t\t\tif (level == GENERATION_NUMBER_ZERO) {\n+\t\t\t\tif (gen == GENERATION_NUMBER_ZERO) {\n \t\t\t\t\tall_parents_computed = 0;\n \t\t\t\t\tcommit_list_insert(parent->item, &list);\n \t\t\t\t\tbreak;\n \t\t\t\t}\n \n-\t\t\t\tif (level > max_level)\n-\t\t\t\t\tmax_level = level;\n+\t\t\t\tif (gen > max_gen)\n+\t\t\t\t\tmax_gen = gen;\n \t\t\t}\n \n \t\t\tif (all_parents_computed) {\n \t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (max_level > GENERATION_NUMBER_V1_MAX - 1)\n-\t\t\t\t\tmax_level = GENERATION_NUMBER_V1_MAX - 1;\n-\t\t\t\t*topo_level_slab_at(ctx->topo_levels, current) = max_level + 1;\n+\t\t\t\tgen = compute_generation_from_max(\n+\t\t\t\t\t\tcurrent, max_gen,\n+\t\t\t\t\t\tgeneration_version);\n+\t\t\t\tinfo->set_generation(current, gen, info->data);\n \t\t\t}\n \t\t}\n \t}\n+}\n+\n+static timestamp_t get_topo_level(struct commit *c, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\treturn *topo_level_slab_at(ctx->topo_levels, c);\n+}\n+\n+static void set_topo_level(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\t*topo_level_slab_at(ctx->topo_levels, c) = (uint32_t)t;\n+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n+}\n+\n+static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+{\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.progress = ctx->progress,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_topo_level,\n+\t\t.set_generation = set_topo_level,\n+\t\t.data = ctx,\n+\t};\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tcompute_reachable_generation_numbers_1(&info, 1);\n+\n \tstop_progress(&ctx->progress);\n }\n \n+static timestamp_t get_generation_from_graph_data(struct commit *c, void *data)\n+{\n+\treturn commit_graph_data_at(c)->generation;\n+}\n+\n+static void set_generation_v2(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\tstruct commit_graph_data *g = commit_graph_data_at(c);\n+\tg->generation = (uint32_t)t;\n+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n+}\n+\n static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n-\tstruct commit_list *list = NULL;\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.progress = ctx->progress,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_v2,\n+\t\t.data = ctx,\n+\t};\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1517,47 +1599,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\t}\n \t}\n \n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\ttimestamp_t corrected_commit_date;\n-\n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tcorrected_commit_date = commit_graph_data_at(c)->generation;\n-\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (corrected_commit_date != GENERATION_NUMBER_ZERO)\n-\t\t\tcontinue;\n-\n-\t\tcommit_list_insert(c, &list);\n-\t\twhile (list) {\n-\t\t\tstruct commit *current = list->item;\n-\t\t\tstruct commit_list *parent;\n-\t\t\tint all_parents_computed = 1;\n-\t\t\ttimestamp_t max_corrected_commit_date = 0;\n-\n-\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tcorrected_commit_date = commit_graph_data_at(parent->item)->generation;\n-\n-\t\t\t\tif (corrected_commit_date == GENERATION_NUMBER_ZERO) {\n-\t\t\t\t\tall_parents_computed = 0;\n-\t\t\t\t\tcommit_list_insert(parent->item, &list);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\n-\t\t\t\tif (corrected_commit_date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = corrected_commit_date;\n-\t\t\t}\n-\n-\t\t\tif (all_parents_computed) {\n-\t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n-\t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n-\t\t\t}\n-\t\t}\n-\t}\n+\tcompute_reachable_generation_numbers_1(&info, 2);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n@@ -1565,6 +1607,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n \t\t\tctx->num_generation_data_overflows++;\n \t}\n+\n \tstop_progress(&ctx->progress);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"473054","messageId":"c6e6581e0ea8717520f6116c7136d6d084589e94.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 5/8] commit-graph: return generation from memory","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:35Z","receivedAt":"2023-03-06T14:07:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit_graph_generation() method used to report a value of\nGENERATION_NUMBER_INFINITY if the commit_graph_data_slab had an instance\nfor the given commit but the graph_pos indicated the commit was not in\nthe commit-graph file.\n\nInstead, trust the 'generation' member if the commit has a value in the\nslab _and_ the 'generation' member is non-zero. Otherwise, treat it as\nGENERATION_NUMBER_INFINITY.\n\nThis only makes a difference for a very old case for the commit-graph:\nthe very first Git release to write commit-graph files wrote zeroes in\nthe topological level positions. If we are parsing a commit-graph with\nall zeroes, those commits will now appear to have\nGENERATION_NUMBER_INFINITY (as if they were not parsed from the\ncommit-graph).\n\nI attempted several variations to work around the need for providing an\nuninitialized 'generation' member, but this was the best one I found. It\ndoes require a change to a verification test in t5318 because it reports\na different error than the one about non-zero generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 ++++-----\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex deccf984a0d..f04b02be1bb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -111,17 +111,16 @@ uint32_t commit_graph_position(const struct commit *c)\n \treturn data ? data->graph_pos : COMMIT_NOT_FROM_GRAPH;\n }\n \n+\n timestamp_t commit_graph_generation(const struct commit *c)\n {\n \tstruct commit_graph_data *data =\n \t\tcommit_graph_data_slab_peek(&commit_graph_data_slab, c);\n \n-\tif (!data)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n-\telse if (data->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n+\tif (data && data->generation)\n+\t\treturn data->generation;\n \n-\treturn data->generation;\n+\treturn GENERATION_NUMBER_INFINITY;\n }\n \n static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 049c5fc8ead..b6e12115786 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -630,7 +630,7 @@ test_expect_success 'detect incorrect generation number' '\n \n test_expect_success 'detect incorrect generation number' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n-\t\t\"non-zero generation number\"\n+\t\t\"commit-graph generation for commit\"\n '\n \n test_expect_success 'detect incorrect commit date' '\n-- \ngitgitgadget\n\n"},{"id":"473055","messageId":"f31994ac78aa7aff11d923a77be6b0652f3af70d.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 6/8] commit-graph: introduce `ensure_generations_valid()`","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:36Z","receivedAt":"2023-03-06T14:07:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nUse the just-introduced compute_reachable_generation_numbers_1() to\nimplement a function which dynamically computes topological levels (or\ncorrected commit dates) for out-of-graph commits.\n\nThis will be useful for the ahead-behind algorithm we are about to\nintroduce, which needs accurate topological levels on _all_ commits\nreachable from the tips in order to avoid over-counting.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 +++++++++++++++++++++++++++++\n commit-graph.h | 7 +++++++\n 2 files changed, 36 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f04b02be1bb..a573d1b89ff 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1610,6 +1610,35 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void set_generation_in_graph_data(struct commit *c, timestamp_t t,\n+\t\t\t\t\t void *data)\n+{\n+\tcommit_graph_data_at(c)->generation = t;\n+}\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct commit **commits, size_t nr)\n+{\n+\tstruct repository *r = the_repository;\n+\tint generation_version = get_configured_generation_version(r);\n+\tstruct packed_commit_list list = {\n+\t\t.list = commits,\n+\t\t.alloc = nr,\n+\t\t.nr = nr,\n+\t};\n+\tstruct compute_generation_info info = {\n+\t\t.r = r,\n+\t\t.commits = &list,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_in_graph_data,\n+\t};\n+\n+\tcompute_reachable_generation_numbers_1(&info, generation_version);\n+}\n+\n static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n {\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-computed\",\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 37faee6b66d..a529c62b518 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -190,4 +190,11 @@ struct commit_graph_data {\n */\n timestamp_t commit_graph_generation(const struct commit *);\n uint32_t commit_graph_position(const struct commit *);\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct commit **commits, size_t nr);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"473056","messageId":"b8c55ecf88d6229f13e05e8369adaf9e70ae1de0.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 7/8] ahead-behind: implement ahead_behind() logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:37Z","receivedAt":"2023-03-06T14:08:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nFully implement the commit-counting logic behind the ahead-behind\nbuiltin as a new ahead_behind() method in commit-reach.h. Add tests\nfor the functionality in both t4218-ahead-behind.sh and\nt6600-test-reach.sh. The tests in t4218 are rather simple, but cover a\nsimple diamond commit history completely while the tests in t6600 make\nuse of the more complicated commit history and the test setup to check\nthree repository states: no commit-graph, a complete commit-graph, and a\nhalf-filled commit-graph. These extra states are particularly helpful to\ncheck due to the implementation of ahead_behind() and how it relies upon\nensure_generations_valid().\n\nThe interface for ahead_behind() uses two arrays. The first array of\ncommits contains the list of all starting points for the walk. This\nincludes all tip commits _and_ base commits. The second array, using the\nnew ahead_behind_count struct, indicates which commits from that initial\narray form the base/tip pair for the ahead/behind count it will store.\n\nWhile the ahead-behind builtin currently only supports one base, this\nimplementation of ahead_behind() allows multiple bases, if desired. Even\nwith multiple bases, there is only one commit walk used for counting the\nahead/behind values, saving time when the base/tip ranges overlap\nsignificantly.\n\nThis interface for ahead_behind() also makes it very easy to call\nensure_generations_valid() on the entire array of bases and tips. This\ncall is necessary because it is critical that the walk that counts\nahead/behind values never walks a commit more than once. Without\ngeneration numbers on every commit, there is a possibility that a\ncommit date skew could cause the walk to revisit a commit and then\ndouble-count it. For this reason, it is strongly recommended that 'git\nahead-behind' is only run in a repository with a commit-graph file that\ncovers most of the reachable commits, storing precomputed generation\nnumbers. If no commit-graph exists, this walk will be much slower as it\nmust walk all reachable commits in ensure_generations_valid() before\nperforming the counting logic.\n\nIt is possible to detect if generation numbers are available at run time\nand redirect the implementation to another algorithm that does not\nrequire this property. However, that implementation requires a commit\nwalk per base/tip pair _and_ can be slower due to the commit date\nheuristics required. Such an implementation could be considered in the\nfuture if there is a reason to include it, but most Git hosts should\nalready be generating a commit-graph file as part of repository\nmaintenance. Most Git clients should also be generating commit-graph\nfiles as part of background maintenance or automatic GCs.\n\nNow, let's discuss the ahead/behind counting algorithm.\n\nEach commit in the input commit list is associated with a bit position\nindicating \"the ith commit can reach this commit\". Each of these commits\nis associated with a bitmap with its position flipped on and then\nplaced in a queue for walking commit history. We walk commits by popping\nthe commit with maximum generation number out of the queue, guaranteeing\nthat we will never walk a child of that commit in any future steps.\n\nAs we walk, we load the bitmap for the current commit and perform two\nmain steps. The _second_ step examines each parent of the current commit\nand adds the current commit's bitmap bits to each parent's bitmap. (We\ncreate a new bitmap for the parent if this is our first time seeing that\nparent.) After adding the bits to the parent's bitmap, the parent is\nadded to the walk queue. Due to this passing of bits to parents, the\ncurrent commit has a guarantee that the ith bit is enabled on its bitmap\nif and only if the ith commit can reach the current commit.\n\nThe first step of the walk is to examine the bitmask on the current\ncommit and decide which ranges the commit is in or not. Due to the \"bit\npushing\" in the second step, we have a guarantee that the ith bit of the\ncurrent commit's bitmap is on if and only if the ith starting commit can\nreach it. For each ahead_behind_count struct, check the base_index and\ntip_index to see if those bits are enabled on the current bitmap. If\nexactly one bit is enabled, then increment the corresponding 'ahead' or\n'behind' count. This increment is the reason we _absolutely need_ to\nwalk commits at most once.\n\nThe only subtle thing to do with this walk is to check to see if a\nparent has all bits on in its bitmap, in which case it becomes \"stale\"\nand is marked with the STALE bit. This allows queue_has_nonstale() to be\nthe terminating condition of the walk, which greatly reduces the number\nof commits walked if all of the commits are nearby in history. It avoids\nwalking a large number of common commits when there is a deep history.\nWe also use the helper method insert_no_dup() to add commits to the\npriority queue without adding them multiple times. This uses the PARENT2\nflag. Thus, we must clear both the STALE and PARENT2 bits of all\ncommits, in case ahead_behind() is called multiple times in the same\nprocess.\n\nThere is no previous implementation of ahead-behind to compare against.\nA previous implementation in another fork of Git used a single process\nto essentially do the same walk as 'git rev-list --count ..'\nfor every base/tip pair given as input. The single-walk implementation\nin this change was a significant improvement over that implementation.\nAnother version from that fork used reachability bitmaps for the\ncomparison, but that implementation was slower than the current commit\nwalk implementation in almost all cases.\n\nTo best present _some_ amount of evidence for this performance gain,\ncreate a new performance test, p1500-graph-walks.sh. This script could\nbe used for other walks than just ahead-behind in the future, but let's\nlimit to ahead-behind now.\n\nTo gain some amount of a baseline, create one test that checks 'git\nahead-behind' against up to 50 tips and another that uses 'git rev-list\n--count' in a loop. Be sure to write a commit-graph before running the\nperformance tests.\n\nUsing the Git source code as the repository, we see a pronounced\nimprovement:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git ahead-behind 0.08(0.07+0.01)\n1500.3: ahead-behind counts: git rev-list 1.11(0.92+0.18)\n\nBut the de-facto performance benchmark is the Linux kernel repository,\nwhich presents these values for my copy:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git ahead-behind 0.27(0.25+0.02)\n1500.3: ahead-behind counts: git rev-list 4.53(3.92+0.60)\n\nSigned-off-by: Derrick Stolee \n---\n builtin/ahead-behind.c | 23 +++++++++\n commit-reach.c | 95 +++++++++++++++++++++++++++++++++++++\n commit-reach.h | 30 ++++++++++++\n t/perf/p1500-graph-walks.sh | 25 ++++++++++\n t/t4218-ahead-behind.sh | 67 ++++++++++++++++++++++++++\n t/t6600-test-reach.sh | 62 ++++++++++++++++++++++++\n 6 files changed, 302 insertions(+)\n create mode 100755 t/perf/p1500-graph-walks.sh\n\ndiff --git a/builtin/ahead-behind.c b/builtin/ahead-behind.c\nindex e4f65fc0548..c06b95b5f37 100644\n--- a/builtin/ahead-behind.c\n+++ b/builtin/ahead-behind.c\n@@ -2,6 +2,7 @@\n #include \"parse-options.h\"\n #include \"config.h\"\n #include \"commit.h\"\n+#include \"commit-reach.h\"\n \n static const char * const ahead_behind_usage[] = {\n \tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n@@ -29,8 +30,12 @@ static int handle_arg(struct string_list *tips, const char *arg)\n int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n {\n \tconst char *base_ref = NULL;\n+\tstruct commit *base;\n \tint from_stdin = 0;\n \tstruct string_list tips = STRING_LIST_INIT_DUP;\n+\tstruct commit **commits;\n+\tstruct ahead_behind_count *counts;\n+\tsize_t i;\n \n \tstruct option ahead_behind_opts[] = {\n \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n@@ -71,5 +76,23 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n \tif (!tips.nr)\n \t\treturn 0;\n \n+\tALLOC_ARRAY(commits, tips.nr + 1);\n+\tALLOC_ARRAY(counts, tips.nr);\n+\n+\tfor (i = 0; i < tips.nr; i++) {\n+\t\tcommits[i] = tips.items[i].util;\n+\t\tcounts[i].tip_index = i;\n+\t\tcounts[i].base_index = tips.nr;\n+\t}\n+\tcommits[tips.nr] = base;\n+\n+\tahead_behind(commits, tips.nr + 1, counts, tips.nr);\n+\n+\tfor (i = 0; i < tips.nr; i++)\n+\t\tprintf(\"%s %d %d\\n\", tips.items[i].string,\n+\t\t counts[i].ahead, counts[i].behind);\n+\n+\tfree(counts);\n+\tfree(commits);\n \treturn 0;\n }\ndiff --git a/commit-reach.c b/commit-reach.c\nindex 2e33c599a82..87ccc2cd4f5 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -8,6 +8,7 @@\n #include \"revision.h\"\n #include \"tag.h\"\n #include \"commit-reach.h\"\n+#include \"ewah/ewok.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define PARENT1\t\t(1u<<16)\n@@ -941,3 +942,97 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \n \treturn found_commits;\n }\n+\n+define_commit_slab(bit_arrays, struct bitmap *);\n+static struct bit_arrays bit_arrays;\n+\n+static void insert_no_dup(struct prio_queue *queue, struct commit *c)\n+{\n+\tif (c->object.flags & PARENT2)\n+\t\treturn;\n+\tprio_queue_put(queue, c);\n+\tc->object.flags |= PARENT2;\n+}\n+\n+static struct bitmap *init_bit_array(struct commit *c, int width)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\t*bitmap = bitmap_word_alloc(width);\n+\treturn *bitmap;\n+}\n+\n+static void free_bit_array(struct commit *c)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\treturn;\n+\tbitmap_free(*bitmap);\n+\t*bitmap = NULL;\n+}\n+\n+void ahead_behind(struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr)\n+{\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n+\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n+\tsize_t i;\n+\n+\tif (!commits_nr || !counts_nr)\n+\t\treturn;\n+\n+\tfor (i = 0; i < counts_nr; i++) {\n+\t\tcounts[i].ahead = 0;\n+\t\tcounts[i].behind = 0;\n+\t}\n+\n+\tensure_generations_valid(commits, commits_nr);\n+\n+\tinit_bit_arrays(&bit_arrays);\n+\n+\tfor (i = 0; i < commits_nr; i++) {\n+\t\tstruct commit *c = commits[i];\n+\t\tstruct bitmap *bitmap = init_bit_array(c, width);\n+\n+\t\tbitmap_set(bitmap, i);\n+\t\tinsert_no_dup(&queue, c);\n+\t}\n+\n+\twhile (queue_has_nonstale(&queue)) {\n+\t\tstruct commit *c = prio_queue_get(&queue);\n+\t\tstruct commit_list *p;\n+\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n+\n+\t\tfor (i = 0; i < counts_nr; i++) {\n+\t\t\tint reach_from_tip = bitmap_get(bitmap_c, counts[i].tip_index);\n+\t\t\tint reach_from_base = bitmap_get(bitmap_c, counts[i].base_index);\n+\n+\t\t\tif (reach_from_tip ^ reach_from_base) {\n+\t\t\t\tif (reach_from_base)\n+\t\t\t\t\tcounts[i].behind++;\n+\t\t\t\telse\n+\t\t\t\t\tcounts[i].ahead++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tstruct bitmap *bitmap_p;\n+\n+\t\t\tparse_commit(p->item);\n+\n+\t\t\tbitmap_p = init_bit_array(p->item, width);\n+\t\t\tbitmap_or(bitmap_p, bitmap_c);\n+\n+\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n+\t\t\t\tp->item->object.flags |= STALE;\n+\n+\t\t\tinsert_no_dup(&queue, p->item);\n+\t\t}\n+\n+\t\tfree_bit_array(c);\n+\t}\n+\n+\trepo_clear_commit_marks(the_repository, PARENT2 | STALE);\n+\tclear_bit_arrays(&bit_arrays);\n+\tclear_prio_queue(&queue);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex 148b56fea50..1780f9317bf 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -104,4 +104,34 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \t\t\t\t\t struct commit **to, int nr_to,\n \t\t\t\t\t unsigned int reachable_flag);\n \n+struct ahead_behind_count {\n+\t/**\n+\t * As input, the *_index members indicate which positions in\n+\t * the 'tips' array correspond to the tip and base of this\n+\t * comparison.\n+\t */\n+\tsize_t tip_index;\n+\tsize_t base_index;\n+\n+\t/**\n+\t * These values store the computed counts for each side of the\n+\t * symmetric difference:\n+\t *\n+\t * 'ahead' stores the number of commits reachable from the tip\n+\t * and not reachable from the base.\n+\t *\n+\t * 'behind' stores the number of commits reachable from the base\n+\t * and not reachable from the tip.\n+\t */\n+\tint ahead;\n+\tint behind;\n+};\n+\n+/**\n+ * Given an array of commits and an array of ahead_behind_count pairs,\n+ * compute the ahead/behind counts for each pair.\n+ */\n+void ahead_behind(struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr);\n+\n #endif\ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nnew file mode 100755\nindex 00000000000..c9ac4b7e6e2\n--- /dev/null\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -0,0 +1,25 @@\n+#!/bin/sh\n+\n+test_description='Commit walk performance tests'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit for-each-ref --format=\"%(refname)\" \"refs/heads/*\" \"refs/tags/*\" >allrefs &&\n+\tsort -r allrefs | head -n 50 >refs &&\n+\tgit commit-graph write --reachable\n+'\n+\n+test_perf 'ahead-behind counts: git ahead-behind' '\n+\tgit ahead-behind --base=HEAD --stdin out &&\n \tgrep \"usage:\" out\n@@ -14,6 +24,11 @@ test_expect_success 'git ahead-behind without --base' '\n \tgrep \"usage:\" err\n '\n \n+test_expect_success 'git ahead-behind with broken --base' '\n+\ttest_must_fail git ahead-behind --base=bogus HEAD 2>err &&\n+\tgrep \"could not resolve '\\''bogus'\\''\" err\n+'\n+\n test_expect_success 'git ahead-behind with broken tip' '\n \ttest_must_fail git ahead-behind --base=HEAD bogus 2>err &&\n \tgrep \"could not resolve '\\''bogus'\\''\" err\n@@ -30,4 +45,56 @@ test_expect_success 'git ahead-behind without tips' '\n \ttest_must_be_empty err\n '\n \n+test_expect_success 'git ahead-behind --base=base' '\n+\tgit ahead-behind --base=base base left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase 0 0\n+\tleft 1 0\n+\tright 1 0\n+\tmerge 3 0\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ahead-behind --base=left' '\n+\tgit ahead-behind --base=left base left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase 0 1\n+\tleft 0 0\n+\tright 1 1\n+\tmerge 2 0\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ahead-behind --base=right' '\n+\tgit ahead-behind --base=right base left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase 0 1\n+\tleft 1 1\n+\tright 0 0\n+\tmerge 2 0\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ahead-behind --base=merge' '\n+\tgit ahead-behind --base=merge base left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase 0 3\n+\tleft 0 2\n+\tright 0 2\n+\tmerge 0 0\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n test_done\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 338a9c46a24..951e07100f6 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -443,4 +443,66 @@ test_expect_success 'get_reachable_subset:none' '\n \ttest_all_modes get_reachable_subset\n '\n \n+test_expect_success 'ahead-behind:linear' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-1-1\n+\tcommit-1-3\n+\tcommit-1-5\n+\tcommit-1-8\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\tcommit-1-1 0 8\n+\tcommit-1-3 0 6\n+\tcommit-1-5 0 4\n+\tcommit-1-8 0 1\n+\tEOF\n+\trun_all_modes git ahead-behind --base=commit-1-9 --stdin\n+'\n+\n+test_expect_success 'ahead-behind:all' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-1-1\n+\tcommit-2-4\n+\tcommit-4-2\n+\tcommit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\tcommit-1-1 0 24\n+\tcommit-2-4 0 17\n+\tcommit-4-2 0 17\n+\tcommit-4-4 0 9\n+\tEOF\n+\trun_all_modes git ahead-behind --base=commit-5-5 --stdin\n+'\n+\n+test_expect_success 'ahead-behind:some' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-1-1\n+\tcommit-5-3\n+\tcommit-4-8\n+\tcommit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\tcommit-1-1 0 53\n+\tcommit-5-3 0 39\n+\tcommit-4-8 8 30\n+\tcommit-9-9 27 0\n+\tEOF\n+\trun_all_modes git ahead-behind --base=commit-9-6 --stdin\n+'\n+\n+test_expect_success 'ahead-behind:none' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-7-5\n+\tcommit-4-8\n+\tcommit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\tcommit-7-5 7 4\n+\tcommit-4-8 16 16\n+\tcommit-9-9 49 0\n+\tEOF\n+\trun_all_modes git ahead-behind --base=commit-8-4 --stdin\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473057","messageId":"07eb2cbb699c875e5fdca86d675c97d4d434f511.1678111599.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH 8/8] ahead-behind: add --contains mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-06T14:06:38Z","receivedAt":"2023-03-06T14:08:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git ahead-behind' builtin can answer a list of questions that do\nnot require the full ahead/behind counts. For example, instead of using\n'git branch --contains=' to get the full list of branches that\ncontain the commit at , a list of tips could be passed to 'git\nahead-behind --base=' and the rows that report a \"behind\" value of\nzero show which tips can reach .\n\nBy contract, the rows that report an \"ahead\" value of zero show which\ntips are reachable from the given base. This type of query does not have\nan existing equivalent for batching this request. While extracting the\ninformation from 'git ahead-behind' is not terribly difficult, it does\nmore work than required to answer this query: it _counts_.\n\nAdd a new '--contains' mode to 'git ahead-behind' that removes the\ncounting behavior and focuses instead on the reachability concern. The\noutput of the builtin changes in this mode: instead of reporting \"\n \" for every input tip, it will instead report \"\"\nfor the input tips that are reachable from the specified base.\n\nThe algorithm is implemented in commit-reach.c in the method\ntips_reachable_from_base(). This method takes a string_list of tips and\nassigns the 'util' for each item with the value 1 if the base commit can\nreach those tips.\n\nLike other reachability queries in commit-reach.c, the fastest way to\nsearch for \"can A reach B?\" is to do a depth-first search up to the\ngeneration number of B, preferring to explore first parents before later\nparents. While we must walk all reachable commits up to that generation\nnumber when the answer is \"no\", the depth-first search can answer \"yes\"\nmuch faster than other approaches in most cases.\n\nThis search becomes trickier when there are multiple targets for the\ndepth-first search. The commits with lower generation number are more\nlikely to be within the history of the start commit, but we don't want\nto waste time searching commits of low generation number if the commit\ntarget with lowest generation number has already been found.\n\nThe trick here is to take the input commits and sort them by generation\nnumber in ascending order. Track the index within this order as\nmin_generation_index. When we find a commit, if its index in the list is\nequal to min_generation_index, then we can increase the generation\nnumber boundary of our search to the next-lowest value in the list.\n\nWith this mechanism, the number of commits to search is minimized with\nrespect to the depth-first search heuristic. We will walk all commits up\nto the minimum generation number of a commit that is _not_ reachable\nfrom the start, but we will walk only the necessary portion of the\ndepth-first search for the reachable commits of lower generation.\n\nWe can test this completely in the simple repo example in t4218 and more\nsubstantially in the larger repository example in t6600. We can also add\na performance test to demonstrate the speedup relative to the 'git\nahead-behind' builtin without the '--contains' option.\n\nFor the Git source code repository, I was able to measure a speedup,\neven though both are quite fast.\n\nTest this tree\n--------------------------------------------------------------------------\n1500.2: ahead-behind counts: git ahead-behind 0.06(0.06+0.00)\n1500.3: ahead-behind counts: git rev-list 1.08(0.90+0.18)\n1500.4: ahead-behind counts: git ahead-behind --contains 0.02(0.02+0.00)\n\nIn the Linux kernel repository, the impact is more pronounced:\n\nTest this tree\n--------------------------------------------------------------------------\n1500.2: ahead-behind counts: git ahead-behind 0.26(0.25+0.01)\n1500.3: ahead-behind counts: git rev-list 4.58(3.92+0.66)\n1500.4: ahead-behind counts: git ahead-behind --contains 0.02(0.00+0.02)\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-ahead-behind.txt | 12 +++-\n builtin/ahead-behind.c | 27 ++++++-\n commit-reach.c | 110 +++++++++++++++++++++++++++++\n commit-reach.h | 7 ++\n t/perf/p1500-graph-walks.sh | 4 ++\n t/t4218-ahead-behind.sh | 62 ++++++++++++++++\n t/t6600-test-reach.sh | 58 +++++++++++++++\n 7 files changed, 277 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-ahead-behind.txt b/Documentation/git-ahead-behind.txt\nindex 2dd5147f6b2..1652a51d719 100644\n--- a/Documentation/git-ahead-behind.txt\n+++ b/Documentation/git-ahead-behind.txt\n@@ -8,7 +8,7 @@ git-ahead-behind - Count the commits on each side of a revision range\n SYNOPSIS\n --------\n [verse]\n-'git ahead-behind' --base= [ --stdin | ]\n+'git ahead-behind' --base= [ --contains ] [ --stdin | ]\n \n DESCRIPTION\n -----------\n@@ -39,6 +39,9 @@ reported to stdout one line at a time as follows:\n There will be exactly one line per input revision, but the lines may be\n in an arbitrary order.\n \n+If the `--contains` option is provided, then the output will list the\n+`` refs are reachable from the provided ``, one per line.\n+\n \n OPTIONS\n -------\n@@ -50,6 +53,13 @@ OPTIONS\n \tRead revision tips and ranges from stdin instead of from the\n \tcommand-line.\n \n+--contains::\n+\tSpecify that instead of counting the ahead/behind values, only\n+\tindicate whether each tip reference is reachable from the base. In\n+\tthis mode, the output format changes to include only the name of\n+\teach tip by name, one per line, and only the tips reachable from\n+\tthe base are included in the output.\n+\n --ignore-missing::\n \tWhen parsing tip references, ignore any references that are not\n \tfound. This is useful when operating in an environment where a\ndiff --git a/builtin/ahead-behind.c b/builtin/ahead-behind.c\nindex c06b95b5f37..4efd324d5d9 100644\n--- a/builtin/ahead-behind.c\n+++ b/builtin/ahead-behind.c\n@@ -5,7 +5,7 @@\n #include \"commit-reach.h\"\n \n static const char * const ahead_behind_usage[] = {\n-\tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n+\tN_(\"git ahead-behind --base= [ --contains ] [ --stdin | ]\"),\n \tNULL\n };\n \n@@ -31,7 +31,7 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n {\n \tconst char *base_ref = NULL;\n \tstruct commit *base;\n-\tint from_stdin = 0;\n+\tint from_stdin = 0, contains = 0;\n \tstruct string_list tips = STRING_LIST_INIT_DUP;\n \tstruct commit **commits;\n \tstruct ahead_behind_count *counts;\n@@ -41,6 +41,7 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n \t\tOPT_BOOL(0 , \"stdin\", &from_stdin, N_(\"read rev names from stdin\")),\n \t\tOPT_BOOL(0 , \"ignore-missing\", &ignore_missing, N_(\"ignore missing tip references\")),\n+\t\tOPT_BOOL(0 , \"contains\", &contains, N_(\"only check that tips are reachable from the base\")),\n \t\tOPT_END()\n \t};\n \n@@ -52,6 +53,10 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\tbase = lookup_commit_reference_by_name(base_ref);\n+\tif (!base)\n+\t\tdie(_(\"could not resolve '%s'\"), base_ref);\n+\n \tif (from_stdin) {\n \t\tstruct strbuf line = STRBUF_INIT;\n \n@@ -76,6 +81,24 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n \tif (!tips.nr)\n \t\treturn 0;\n \n+\tif (contains) {\n+\t\tstruct string_list_item *item;\n+\n+\t\t/* clear out */\n+\t\tfor_each_string_list_item(item, &tips)\n+\t\t\titem->util = NULL;\n+\n+\t\ttips_reachable_from_base(base, &tips);\n+\n+\t\tfor_each_string_list_item(item, &tips) {\n+\t\t\tif (item->util)\n+\t\t\t\tprintf(\"%s\\n\", item->string);\n+\t\t}\n+\n+\t\treturn 0;\n+\t}\n+\t/* else: not --contains, but normal ahead-behind counting. */\n+\n \tALLOC_ARRAY(commits, tips.nr + 1);\n \tALLOC_ARRAY(counts, tips.nr);\n \ndiff --git a/commit-reach.c b/commit-reach.c\nindex 87ccc2cd4f5..a7a2c045551 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -1036,3 +1036,113 @@ void ahead_behind(struct commit **commits, size_t commits_nr,\n \tclear_bit_arrays(&bit_arrays);\n \tclear_prio_queue(&queue);\n }\n+\n+struct commit_and_index {\n+\tstruct commit *commit;\n+\tunsigned int index;\n+\ttimestamp_t generation;\n+};\n+\n+static int compare_commit_and_index_by_generation(const void *va, const void *vb)\n+{\n+\tconst struct commit_and_index *a = (const struct commit_and_index *)va;\n+\tconst struct commit_and_index *b = (const struct commit_and_index *)vb;\n+\n+\tif (a->generation > b->generation)\n+\t\treturn 1;\n+\tif (a->generation < b->generation)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n+void tips_reachable_from_base(struct commit *base,\n+\t\t\t struct string_list *tips)\n+{\n+\tunsigned int i;\n+\tstruct commit_and_index *commits;\n+\tunsigned int min_generation_index = 0;\n+\ttimestamp_t min_generation;\n+\tstruct commit_list *stack = NULL;\n+\n+\tif (!base || !tips || !tips->nr)\n+\t\treturn;\n+\n+\t/*\n+\t * Do a depth-first search starting at 'base' to search for the\n+\t * tips. Stop at the lowest (un-found) generation number. When\n+\t * finding the lowest commit, increase the minimum generation\n+\t * number to the next lowest (un-found) generation number.\n+\t */\n+\n+\tCALLOC_ARRAY(commits, tips->nr);\n+\n+\tfor (i = 0; i < tips->nr; i++) {\n+\t\tcommits[i].commit = lookup_commit_reference_by_name(tips->items[i].string);\n+\t\tcommits[i].index = i;\n+\t\tcommits[i].generation = commit_graph_generation(commits[i].commit);\n+\t}\n+\n+\t/* Sort with generation number ascending. */\n+\tQSORT(commits, tips->nr, compare_commit_and_index_by_generation);\n+\tmin_generation = commits[0].generation;\n+\n+\tparse_commit(base);\n+\tcommit_list_insert(base, &stack);\n+\n+\twhile (stack) {\n+\t\tunsigned int j;\n+\t\tint explored_all_parents = 1;\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = stack->item;\n+\t\ttimestamp_t c_gen = commit_graph_generation(c);\n+\n+\t\t/* Does it match any of our bases? */\n+\t\tfor (j = min_generation_index; j < tips->nr; j++) {\n+\t\t\tif (c_gen < commits[j].generation)\n+\t\t\t\tbreak;\n+\n+\t\t\tif (commits[j].commit == c) {\n+\t\t\t\ttips->items[commits[j].index].util = (void *)(uintptr_t)1;\n+\n+\t\t\t\tif (j == min_generation_index) {\n+\t\t\t\t\tunsigned int k = j + 1;\n+\t\t\t\t\twhile (k < tips->nr &&\n+\t\t\t\t\t tips->items[commits[k].index].util)\n+\t\t\t\t\t\tk++;\n+\n+\t\t\t\t\t/* Terminate early if all found. */\n+\t\t\t\t\tif (k >= tips->nr)\n+\t\t\t\t\t\tgoto done;\n+\n+\t\t\t\t\tmin_generation_index = k;\n+\t\t\t\t\tmin_generation = commits[k].generation;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tparse_commit(p->item);\n+\n+\t\t\t/* Have we already explored this parent? */\n+\t\t\tif (p->item->object.flags & SEEN)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Is it below the current minimum generation? */\n+\t\t\tif (commit_graph_generation(p->item) < min_generation)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ok, we will explore from here on. */\n+\t\t\tp->item->object.flags |= SEEN;\n+\t\t\texplored_all_parents = 0;\n+\t\t\tcommit_list_insert(p->item, &stack);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (explored_all_parents)\n+\t\t\tpop_commit(&stack);\n+\t}\n+\n+done:\n+\tfree(commits);\n+\trepo_clear_commit_marks(the_repository, SEEN);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex 1780f9317bf..fa8994f5696 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -134,4 +134,11 @@ struct ahead_behind_count {\n void ahead_behind(struct commit **commits, size_t commits_nr,\n \t\t struct ahead_behind_count *counts, size_t counts_nr);\n \n+/*\n+ * Populate the \"util\" of each string_list item with the boolean value\n+ * corresponding to \"can 'base' reach this tip?\"\n+ */\n+void tips_reachable_from_base(struct commit *base,\n+\t\t\t struct string_list *tips);\n+\n #endif\ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nindex c9ac4b7e6e2..56de6c5b13d 100755\n--- a/t/perf/p1500-graph-walks.sh\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -22,4 +22,8 @@ test_perf 'ahead-behind counts: git rev-list' '\n \tdone\n '\n \n+test_perf 'ahead-behind counts: git ahead-behind --contains' '\n+\tgit ahead-behind --contains --base=HEAD --stdin err &&\n+\tgrep \"could not resolve '\\''bogus'\\''\" err\n+'\n+\n+test_expect_success 'git ahead-behind --contains with broken tip and --ignore-missing' '\n+\tgit ahead-behind --base=HEAD --contains \\\n+\t\t--ignore-missing bogus 2>err >out &&\n+\ttest_must_be_empty err &&\n+\ttest_must_be_empty out\n+'\n+\n test_expect_success 'git ahead-behind without tips' '\n \tgit ahead-behind --base=HEAD 2>err &&\n \ttest_must_be_empty err\n@@ -97,4 +110,53 @@ test_expect_success 'git ahead-behind --base=merge' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git ahead-behind --contains --base=base' '\n+\tgit ahead-behind --contains --base=base \\\n+\t\tbase left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ahead-behind --contains --base=left' '\n+\tgit ahead-behind --contains --base=left \\\n+\t\tbase left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase\n+\tleft\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ahead-behind --contains --base=right' '\n+\tgit ahead-behind --contains --base=right \\\n+\t\tbase left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase\n+\tright\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ahead-behind --contains --base=merge' '\n+\tgit ahead-behind --contains --base=merge \\\n+\t\tbase left right merge >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tbase\n+\tleft\n+\tright\n+\tmerge\n+\tEOF\n+\n+\ttest_cmp expect actual\n+'\n+\n test_done\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 951e07100f6..2fdad7b3619 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -505,4 +505,62 @@ test_expect_success 'ahead-behind:none' '\n \trun_all_modes git ahead-behind --base=commit-8-4 --stdin\n '\n \n+test_expect_success 'ahead-behind--contains:all' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-1-1\n+\tcommit-2-4\n+\tcommit-4-2\n+\tcommit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\tcommit-1-1\n+\tcommit-2-4\n+\tcommit-4-2\n+\tcommit-4-4\n+\tEOF\n+\trun_all_modes git ahead-behind --contains --base=commit-5-5 \\\n+\t\t--stdin --use-bitmap-index\n+'\n+\n+test_expect_success 'ahead-behind--contains:some' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-1-1\n+\tcommit-5-3\n+\tcommit-4-8\n+\tcommit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\tcommit-1-1\n+\tcommit-5-3\n+\tEOF\n+\trun_all_modes git ahead-behind --contains --base=commit-9-6 \\\n+\t\t--stdin --use-bitmap-index\n+'\n+\n+test_expect_success 'ahead-behind--contains:some, reordered' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-4-8\n+\tcommit-5-3\n+\tcommit-9-9\n+\tcommit-1-1\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\tcommit-5-3\n+\tcommit-1-1\n+\tEOF\n+\trun_all_modes git ahead-behind --contains --base=commit-9-6 \\\n+\t\t--stdin --use-bitmap-index\n+'\n+\n+test_expect_success 'ahead-behind--contains:none' '\n+\tcat >input <<-\\EOF &&\n+\tcommit-7-5\n+\tcommit-4-8\n+\tcommit-9-9\n+\tEOF\n+\t>expect &&\n+\trun_all_modes git ahead-behind --contains --base=commit-8-4 \\\n+\t\t--stdin --use-bitmap-index\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"473071","messageId":"xmqqedq1ag8d.fsf@gitster.g","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/8] ahead-behind: new builtin for counting multiple commit ranges","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-06T18:26:26Z","receivedAt":"2023-03-06T18:26:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> These numbers can be computed by 'git rev-list --count B..C' and 'git\n> rev-list --count C..B', but there are common needs that benefit from having\n> the checks being done in the same process:\n\nThis makes readers wonder if \"git rev-list --count B...C\" should be\nthe end-user facing UI for this new feature, perhaps?\n\nOf course if you are checking how C0, C1, C2,... relate to a single\nB, the existing rev-list syntax would not work, and makes a totally\nnew subcommand a possibilty.\n\n> 2. When a branch is updated, a background job checks if any pull requests\n> that target that branch should be closed because their branches were\n> merged implicitly by that update. These queries can e batched into 'git\n> ahead-behind' calls.\n>\n> In that second example, we don't need the full ahead/behind counts (although\n> it is sufficient to look for branches that are \"zero commits ahead\", meaning\n> they are reachable from the base), so this builtin has an extra '--contains'\n> mode that only checks reachability from the base to each of the tips. 'git\n> ahead-behind --contains' is sort of the reverse of 'git branch --contains'.\n\nI thought that the reverse of \"git branch --contains\" was \"git\nbranch --merged\". \"git branch --merged maint ??/\\*\" is how I cull\ntopic branches that have already served their purpose. \n\nIsn't closing pull requests because they have been already merged\nthe same idea? \"git for-each-ref --merged main refs/pull/\\*\" or\nsomething, perhaps?\n\nAll of the above are from only reading the cover letter. I'm sure\nI'll have more thoughts or even change my mind after reading the\npatches.\n\nThanks.\n"},{"id":"473074","messageId":"xmqqzg8p90mq.fsf@gitster.g","threadId":"59347","inReplyTo":"0fd18b6d740f1e8a6f62a25947bc3ad49c2674a6.1678111599.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/8] ahead-behind: create empty builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-06T18:48:45Z","receivedAt":"2023-03-06T18:48:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> For example, we will be able to track all local branches relative to an\n> upstream branch using an invocation such as\n>\n> git for-each-ref --format=%(refname) refs/heads/* |\n> git ahead-behind --base=origin/main --stdin\n\nStepping back a bit, this motivating example makes me wonder if\n\n $ git for-each-ref --format='%(refname) %(aheadbehind)' refs/heads/\\*\n\nthat computes the ahead-behind number for each ref (that matches the\npattern) based on their own \"upstream\" (presumably each branch is\nconfigured to track the same, or different, upstreams), or\noverrriding @{upstream}, a specified base, i.e.\n\n $ git for-each-ref --format='%(refname) %(aheadbehind:origin/main)' refs/heads/\\*\n\nwould be a more intuitive interface to the end-users.\n\nIt would probably work well in conjunction with\n\n git for-each-ref --format='%(refname)' --merged origin/main refs/heads/\\*\n\nwhich is a way to list local branches that are already merged into\nthe upstream, to have the feature appear in the same command,\nperhaps?\n"},{"id":"473075","messageId":"xmqqttyx90gr.fsf@gitster.g","threadId":"59347","inReplyTo":"f31994ac78aa7aff11d923a77be6b0652f3af70d.1678111599.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/8] commit-graph: introduce `ensure_generations_valid()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-06T18:52:20Z","receivedAt":"2023-03-06T18:52:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Taylor Blau via GitGitGadget\" writes:\n\n> From: Taylor Blau \n>\n> Use the just-introduced compute_reachable_generation_numbers_1() to\n> implement a function which dynamically computes topological levels (or\n> corrected commit dates) for out-of-graph commits.\n>\n> This will be useful for the ahead-behind algorithm we are about to\n> introduce, which needs accurate topological levels on _all_ commits\n> reachable from the tips in order to avoid over-counting.\n\nInteresting and nice to see it done with so small a change thanks to\nthe previous refactoring.\n\n>\n> Co-authored-by: Derrick Stolee \n> Signed-off-by: Taylor Blau \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 29 +++++++++++++++++++++++++++++\n> commit-graph.h | 7 +++++++\n> 2 files changed, 36 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index f04b02be1bb..a573d1b89ff 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1610,6 +1610,35 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n> \tstop_progress(&ctx->progress);\n> }\n> \n> +static void set_generation_in_graph_data(struct commit *c, timestamp_t t,\n> +\t\t\t\t\t void *data)\n> +{\n> +\tcommit_graph_data_at(c)->generation = t;\n> +}\n> +\n> +/*\n> + * After this method, all commits reachable from those in the given\n> + * list will have non-zero, non-infinite generation numbers.\n> + */\n> +void ensure_generations_valid(struct commit **commits, size_t nr)\n> +{\n> +\tstruct repository *r = the_repository;\n> +\tint generation_version = get_configured_generation_version(r);\n> +\tstruct packed_commit_list list = {\n> +\t\t.list = commits,\n> +\t\t.alloc = nr,\n> +\t\t.nr = nr,\n> +\t};\n> +\tstruct compute_generation_info info = {\n> +\t\t.r = r,\n> +\t\t.commits = &list,\n> +\t\t.get_generation = get_generation_from_graph_data,\n> +\t\t.set_generation = set_generation_in_graph_data,\n> +\t};\n> +\n> +\tcompute_reachable_generation_numbers_1(&info, generation_version);\n> +}\n> +\n> static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n> {\n> \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-computed\",\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 37faee6b66d..a529c62b518 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -190,4 +190,11 @@ struct commit_graph_data {\n> */\n> timestamp_t commit_graph_generation(const struct commit *);\n> uint32_t commit_graph_position(const struct commit *);\n> +\n> +/*\n> + * After this method, all commits reachable from those in the given\n> + * list will have non-zero, non-infinite generation numbers.\n> + */\n> +void ensure_generations_valid(struct commit **commits, size_t nr);\n> +\n> #endif\n"},{"id":"473082","messageId":"8be01cb6-8937-faed-0850-4501a2f1ef55@github.com","threadId":"59347","inReplyTo":"xmqqedq1ag8d.fsf@gitster.g","subject":"Re: [PATCH 0/8] ahead-behind: new builtin for counting multiple commit ranges","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-06T20:18:11Z","receivedAt":"2023-03-06T20:18:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/6/2023 1:26 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> These numbers can be computed by 'git rev-list --count B..C' and 'git\n>> rev-list --count C..B', but there are common needs that benefit from having\n>> the checks being done in the same process:\n> \n> This makes readers wonder if \"git rev-list --count B...C\" should be\n> the end-user facing UI for this new feature, perhaps?\n> \n> Of course if you are checking how C0, C1, C2,... relate to a single\n> B, the existing rev-list syntax would not work, and makes a totally\n> new subcommand a possibilty.\n> \n>> 2. When a branch is updated, a background job checks if any pull requests\n>> that target that branch should be closed because their branches were\n>> merged implicitly by that update. These queries can e batched into 'git\n>> ahead-behind' calls.\n>>\n>> In that second example, we don't need the full ahead/behind counts (although\n>> it is sufficient to look for branches that are \"zero commits ahead\", meaning\n>> they are reachable from the base), so this builtin has an extra '--contains'\n>> mode that only checks reachability from the base to each of the tips. 'git\n>> ahead-behind --contains' is sort of the reverse of 'git branch --contains'.\n> \n> I thought that the reverse of \"git branch --contains\" was \"git\n> branch --merged\". \"git branch --merged maint ??/\\*\" is how I cull\n> topic branches that have already served their purpose. \n> \n> Isn't closing pull requests because they have been already merged\n> the same idea? \"git for-each-ref --merged main refs/pull/\\*\" or\n> something, perhaps?\n\nYou are definitely on to something, and I was not aware of --merged as\nan option to either of these.\n\n'git branch --merged' has some limitations that tags cannot be used.\n\n'git for-each-ref --merged' is probably sufficient. The only difference\nbeing that it would be nice to specify the matching refs over stdin\nwith --stdin to avoid long argument lists.\n\nWith this in mind, I can update the performance test to look like this\n(after updating the setup step to add branches for each line in 'refs')\n\n\ntest_perf 'batch reachability: git ahead-behind --contains' '\n\tgit ahead-behind --contains --base=HEAD --stdin writes:\n\n> 'git for-each-ref --merged' is probably sufficient. The only difference\n> being that it would be nice to specify the matching refs over stdin\n> with --stdin to avoid long argument lists.\n\nYeah, if you have a list of concrete refs maintained externally\n(as opposed to the example I responded to, where you generate the\nrefs by telling for-each-ref what pattern they should match), having\n--stdin would be a good thing.\n\n> So, there is benefit in using this tips_reachable_from_base() method in\n> the two existing 'git (branch|for-each-ref) --merged' computations. The\n> API boundary selected in this series might not be the most appropriate\n> for those builtins, so let's kick out patch 8 from this series for now\n> and I'll revisit it separately.\n\nYup, if the reachability API refactoring in these patches can also\nhelp the \"for-each-ref\" listing (which \"git branch\" and \"git tag\"\nbases their listing behaviour), it would be very good.\n\nThanks.\n"},{"id":"473098","messageId":"ZAaGYA8grLSt6Ukt@nand.local","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/8] ahead-behind: new builtin for counting multiple commit ranges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-07T00:33:36Z","receivedAt":"2023-03-07T00:33:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 06, 2023 at 02:06:30PM +0000, Derrick Stolee via GitGitGadget wrote:\n> This series introduces the 'git ahead-behind' builtin, which has been used\n> at $DAYJOB for many years, but took many forms before landing on the current\n> version.\n\nThanks for a helpful summary of all of the details here. I am of course\nfamiliar with your use of this builtin at our common $DAYJOB, but it was\nnice to see a from-scratch explanation of what you're trying to do here.\n\nNow let's take a look at how the patches came together... ;-).\n\nThanks,\nTaylor\n"},{"id":"473099","messageId":"ZAaHB7XTz8HX75LD@nand.local","threadId":"59347","inReplyTo":"xmqqedq1ag8d.fsf@gitster.g","subject":"Re: [PATCH 0/8] ahead-behind: new builtin for counting multiple commit ranges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-07T00:36:23Z","receivedAt":"2023-03-07T00:36:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 06, 2023 at 10:26:26AM -0800, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n> > These numbers can be computed by 'git rev-list --count B..C' and 'git\n> > rev-list --count C..B', but there are common needs that benefit from having\n> > the checks being done in the same process:\n>\n> This makes readers wonder if \"git rev-list --count B...C\" should be\n> the end-user facing UI for this new feature, perhaps?\n>\n> Of course if you are checking how C0, C1, C2,... relate to a single\n> B, the existing rev-list syntax would not work, and makes a totally\n> new subcommand a possibilty.\n\nYeah. You could imagine that `rev-list --count` might do something\nfancy like coalescing\n\n git rev-list --count B...C1 B...C2 B...C3\n\ninto a single walk. But I am not sure that just because `rev-list\n--count` provides similar functionality that we should fold in the\nproposed `ahead-behind` interface into that flag.\n\nOn the other hand, I could see a compelling argument for a slightly\ndifferent syntax (maybe `--count-ahead-behind` or\n`--count=ahead-behind`) that would fold this functionality into\n`rev-list`.\n\nAnd that is the sort of thing that we would want to settle on sooner\nrather than later, since it's fairly baked in once we decide one way or\nanother and then merge this up.\n\nMy personal feeling is that we ought to avoid (further) overloading\n`rev-list` absent of a compelling reason to do so. But I am definitely\nopen to other thoughts here.\n\nThanks,\nTaylor\n"},{"id":"473100","messageId":"ZAaH/iCsqdewYrUj@nand.local","threadId":"59347","inReplyTo":"xmqqzg8p90mq.fsf@gitster.g","subject":"Re: [PATCH 1/8] ahead-behind: create empty builtin","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-07T00:40:30Z","receivedAt":"2023-03-07T00:40:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 06, 2023 at 10:48:45AM -0800, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n> > For example, we will be able to track all local branches relative to an\n> > upstream branch using an invocation such as\n> >\n> > git for-each-ref --format=%(refname) refs/heads/* |\n> > git ahead-behind --base=origin/main --stdin\n>\n> Stepping back a bit, this motivating example makes me wonder if\n>\n> $ git for-each-ref --format='%(refname) %(aheadbehind)' refs/heads/\\*\n\nOne disadvantage to using for-each-ref here is that we are bound to use\nall of the ref-sorting code, so callers can't see intermediate results\nuntil the entire walk is complete.\n\nI can't remember enough of the details about the custom traversal we use\nhere to know if that would even matter or not (i.e., do we need to\ntraverse through the whole set of objects entirely before outputting a\nsingle result anyway?). But something to think about nonetheless.\n\nAt the very least, it is quite a cute idea (especially something like\n'%(aheadbehind:origin/main)') ;-).\n\n> that computes the ahead-behind number for each ref (that matches the\n> pattern) based on their own \"upstream\" (presumably each branch is\n> configured to track the same, or different, upstreams), or\n> overrriding @{upstream}, a specified base, i.e.\n>\n> $ git for-each-ref --format='%(refname) %(aheadbehind:origin/main)' refs/heads/\\*\n>\n> would be a more intuitive interface to the end-users.\n>\n> It would probably work well in conjunction with\n>\n> git for-each-ref --format='%(refname)' --merged origin/main refs/heads/\\*\n>\n> which is a way to list local branches that are already merged into\n> the upstream, to have the feature appear in the same command,\n> perhaps?\n\nOne thing that we had talked about internally[^1] was the idea of\nspecifying multiple bases. IOW, having some way to invoke the\nahead-behind builtin that gives some set of tips with a common base B1,\nand another set of tips (which could--but doesn't have to--intersect\nwith the first) and a common base to compare *them* to, say, B2.\n\nThere are some technical reasons that we might want to consider such a\nthing at least motivated by GitHub's proposed future use of it. But they\nare kind of technical and not that interesting to this discussion, so I\nwouldn't be sad if we didn't have a way to specify multiple bases.\n\nOTOH, it would be nice to avoid painting ourselves into a corner from a\nUI-perspective if we can avoid it.\n\nThanks,\nTaylor\n\n[^1]: ...and couldn't decide if it was going to be a nice future\naddition or simply another case of YAGNI ;-).\n"},{"id":"473101","messageId":"ZAaIojLkjw3hHGm+@nand.local","threadId":"59347","inReplyTo":"08fc0710017d4b6178a8c5772e0f15fc69c84ad6.1678111599.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/8] ahead-behind: parse tip references","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-07T00:43:14Z","receivedAt":"2023-03-07T00:43:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 06, 2023 at 02:06:32PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n>\n> Before implementing the logic to compute the ahead/behind counts, parse\n> the unknown options as commits and place them in a string_list.\n>\n> Be sure to error out when the reference is not found.\n>\n> Co-authored-by: Taylor Blau \n> Signed-off-by: Taylor Blau \n\nThis all looks reasonable (and forging my S-o-b here and elsewhere\nthroughout this series is all fine). I have seen most of this code\nbefore at least in its final state, but the intermediate bits are new to\nme.\n\nAnd they all look fine and familiar, except...\n\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/ahead-behind.c | 39 +++++++++++++++++++++++++++++++++++++++\n> t/t4218-ahead-behind.sh | 10 ++++++++++\n> 2 files changed, 49 insertions(+)\n>\n> diff --git a/builtin/ahead-behind.c b/builtin/ahead-behind.c\n> index a56cc565def..c1212cc8d46 100644\n> --- a/builtin/ahead-behind.c\n> +++ b/builtin/ahead-behind.c\n> @@ -1,16 +1,31 @@\n> #include \"builtin.h\"\n> #include \"parse-options.h\"\n> #include \"config.h\"\n> +#include \"commit.h\"\n>\n> static const char * const ahead_behind_usage[] = {\n> \tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n> \tNULL\n> };\n>\n> +static int handle_arg(struct string_list *tips, const char *arg)\n> +{\n> +\tstruct string_list_item *item;\n> +\tstruct commit *c = lookup_commit_reference_by_name(arg);\n> +\n> +\tif (!c)\n> +\t\treturn error(_(\"could not resolve '%s'\"), arg);\n> +\n> +\titem = string_list_append(tips, arg);\n> +\titem->util = c;\n> +\treturn 0;\n> +}\n> +\n> int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char *base_ref = NULL;\n> \tint from_stdin = 0;\n> +\tstruct string_list tips = STRING_LIST_INIT_DUP;\n>\n> \tstruct option ahead_behind_opts[] = {\n> \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n> @@ -26,5 +41,29 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n>\n> \tgit_config(git_default_config, NULL);\n>\n> +\tif (from_stdin) {\n> +\t\tstruct strbuf line = STRBUF_INIT;\n> +\n> +\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n> +\t\t\tif (!line.len)\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tif (handle_arg(&tips, line.buf))\n> +\t\t\t\treturn 1;\n> +\t\t}\n> +\n> +\t\tstrbuf_release(&line);\n> +\t} else {\n> +\t\tint i;\n> +\t\tfor (i = 0; i < argc; ++i) {\n> +\t\t\tif (handle_arg(&tips, argv[i]))\n> +\t\t\t\treturn 1;\n> +\t\t}\n> +\t}\n> +\n> +\t/* Early return for no tips. */\n> +\tif (!tips.nr)\n> +\t\treturn 0;\n> +\n\n...are we missing a call to `string_list_clear()` here on `&tips`?\n\nThanks,\nTaylor\n"},{"id":"473102","messageId":"ZAaJckdkz6zaF6lw@nand.local","threadId":"59347","inReplyTo":"b1d022c7cac5aed6e2d64b45f20dba5b3297536c.1678111599.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/8] ahead-behind: implement --ignore-missing option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-07T00:46:42Z","receivedAt":"2023-03-07T00:46:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 06, 2023 at 02:06:33PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n>\n> When parsing the tip revisions from the ahead-behind inputs, it is\n> important to check that those tips exist before adding them to the list\n> for computation. The previous change caused the builtin to return with\n> errors if the revisions could not be resolved.\n>\n> However, when running 'git ahead-behind' in an environment with\n> concurrent edits, such as a Git server, then the references could be\n> deleted from underneath the caller between reading the reference list\n> and starting the 'git ahead-behind' process. Avoid this race by allowing\n> the caller to specify '--ignore-missing' and continue using the\n> information that is still available.\n\nWell explained, thanks for writing this all out :-).\n\n> diff --git a/builtin/ahead-behind.c b/builtin/ahead-behind.c\n> index c1212cc8d46..e4f65fc0548 100644\n> --- a/builtin/ahead-behind.c\n> +++ b/builtin/ahead-behind.c\n> @@ -8,13 +8,18 @@ static const char * const ahead_behind_usage[] = {\n> \tNULL\n> };\n>\n> +static int ignore_missing;\n> +\n> static int handle_arg(struct string_list *tips, const char *arg)\n> {\n> \tstruct string_list_item *item;\n> \tstruct commit *c = lookup_commit_reference_by_name(arg);\n>\n> -\tif (!c)\n> +\tif (!c) {\n> +\t\tif (ignore_missing)\n> +\t\t\treturn 0;\n> \t\treturn error(_(\"could not resolve '%s'\"), arg);\n> +\t}\n\nAnd the diff makes complete sense here, too.\n\n> \titem = string_list_append(tips, arg);\n> \titem->util = c;\n> @@ -30,6 +35,7 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n> \tstruct option ahead_behind_opts[] = {\n> \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n> \t\tOPT_BOOL(0 , \"stdin\", &from_stdin, N_(\"read rev names from stdin\")),\n> +\t\tOPT_BOOL(0 , \"ignore-missing\", &ignore_missing, N_(\"ignore missing tip references\")),\n\nThe spacing between \"0\" and the comma and \"ignore-missing\" (as well with\n\"stdin\" above, though I didn't notice it when reading the previous\npatch) is a little funky.\n\nI suspect that it is carried over from some historical typo from many\nyears ago, but probably worth fixing while we're thinking about it.\n\nThanks,\nTaylor\n"},{"id":"473103","messageId":"ZAaKX8eUeAHBhA7B@nand.local","threadId":"59347","inReplyTo":"xmqqttyx90gr.fsf@gitster.g","subject":"Re: [PATCH 6/8] commit-graph: introduce `ensure_generations_valid()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-07T00:50:39Z","receivedAt":"2023-03-07T00:50:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 06, 2023 at 10:52:20AM -0800, Junio C Hamano wrote:\n> \"Taylor Blau via GitGitGadget\" writes:\n>\n> > From: Taylor Blau \n> >\n> > Use the just-introduced compute_reachable_generation_numbers_1() to\n> > implement a function which dynamically computes topological levels (or\n> > corrected commit dates) for out-of-graph commits.\n> >\n> > This will be useful for the ahead-behind algorithm we are about to\n> > introduce, which needs accurate topological levels on _all_ commits\n> > reachable from the tips in order to avoid over-counting.\n>\n> Interesting and nice to see it done with so small a change thanks to\n> the previous refactoring.\n\nStolee did all of the hard work ;-). But chiming in to say that I\nremember all of these changes well and they look faithful to my original\nread of them (as well as what I wrote in this patch).\n\nSo, LGTM so far.\n\nThanks,\nTaylor\n"},{"id":"473105","messageId":"ZAaN4w4kltPIeYlt@nand.local","threadId":"59347","inReplyTo":"b8c55ecf88d6229f13e05e8369adaf9e70ae1de0.1678111599.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/8] ahead-behind: implement ahead_behind() logic","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-07T01:05:39Z","receivedAt":"2023-03-07T01:05:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 06, 2023 at 02:06:37PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Signed-off-by: Derrick Stolee \n\nHaving read and worked with this code before, I don't have a ton of\nsubstance to add here. But it was interesting to reread, and I left a\nfew sprinklings here and there of some thing that we may want to\nconsider for v2.\n\nBefore that, though, IIRC we wrote most of this together, so I would be\nhappy to have my:\n\n Co-authored-by: Taylor Blau \n Signed-off-by: Taylor Blau \n\nabove your S-o-b here. But you've done so much work since we originally\nwrote this together that I don't mind being dropped here. Up to you :-).\n\n> @@ -71,5 +76,23 @@ int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n> \tif (!tips.nr)\n> \t\treturn 0;\n>\n> +\tALLOC_ARRAY(commits, tips.nr + 1);\n> +\tALLOC_ARRAY(counts, tips.nr);\n> +\n> +\tfor (i = 0; i < tips.nr; i++) {\n> +\t\tcommits[i] = tips.items[i].util;\n> +\t\tcounts[i].tip_index = i;\n> +\t\tcounts[i].base_index = tips.nr;\n> +\t}\n> +\tcommits[tips.nr] = base;\n> +\n> +\tahead_behind(commits, tips.nr + 1, counts, tips.nr);\n> +\n> +\tfor (i = 0; i < tips.nr; i++)\n> +\t\tprintf(\"%s %d %d\\n\", tips.items[i].string,\n> +\t\t counts[i].ahead, counts[i].behind);\n> +\n> +\tfree(counts);\n> +\tfree(commits);\n> \treturn 0;\n> }\n\nI have to say, the interface looks particularly well designed when you\nsee the patches come together in this fashion. The builtin is doing\nbasically no work except collating the user's input, passing it off to\nahead_behind(), and then spitting out the results.\n\nVery nice ;-).\n\n> diff --git a/commit-reach.c b/commit-reach.c\n> index 2e33c599a82..87ccc2cd4f5 100644\n> --- a/commit-reach.c\n> +++ b/commit-reach.c\n> @@ -8,6 +8,7 @@\n> #include \"revision.h\"\n> #include \"tag.h\"\n> #include \"commit-reach.h\"\n> +#include \"ewah/ewok.h\"\n>\n> /* Remember to update object flag allocation in object.h */\n\nThere is a new use of PARENT2 (which we hardcode here as bit 17) below,\nbut it is already covered as part of the object flag allocation table in\nobject.h. So this comment has done its job over the years ;-).\n\n> #define PARENT1\t\t(1u<<16)\n> @@ -941,3 +942,97 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n>\n> \treturn found_commits;\n> }\n> +\n> +define_commit_slab(bit_arrays, struct bitmap *);\n> +static struct bit_arrays bit_arrays;\n> +\n> +static void insert_no_dup(struct prio_queue *queue, struct commit *c)\n> +{\n> +\tif (c->object.flags & PARENT2)\n> +\t\treturn;\n> +\tprio_queue_put(queue, c);\n> +\tc->object.flags |= PARENT2;\n> +}\n\nYou mentioned this in the patch message, but:\n\nIt may be worth noting here (or in the call to repo_clear_commit_marks()\nbelow) that the PARENT2 flag is used to detect and avoid duplicates in\nthis list.\n\n> +static struct bitmap *init_bit_array(struct commit *c, int width)\n> +{\n> +\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n> +\tif (!*bitmap)\n> +\t\t*bitmap = bitmap_word_alloc(width);\n> +\treturn *bitmap;\n> +}\n> +\n> +static void free_bit_array(struct commit *c)\n> +{\n> +\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n> +\tif (!*bitmap)\n> +\t\treturn;\n> +\tbitmap_free(*bitmap);\n> +\t*bitmap = NULL;\n> +}\n> +\n> +void ahead_behind(struct commit **commits, size_t commits_nr,\n> +\t\t struct ahead_behind_count *counts, size_t counts_nr)\n> +{\n> +\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n> +\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n> +\tsize_t i;\n> +\n> +\tif (!commits_nr || !counts_nr)\n> +\t\treturn;\n> +\n> +\tfor (i = 0; i < counts_nr; i++) {\n> +\t\tcounts[i].ahead = 0;\n> +\t\tcounts[i].behind = 0;\n> +\t}\n> +\n> +\tensure_generations_valid(commits, commits_nr);\n> +\n> +\tinit_bit_arrays(&bit_arrays);\n> +\n> +\tfor (i = 0; i < commits_nr; i++) {\n> +\t\tstruct commit *c = commits[i];\n> +\t\tstruct bitmap *bitmap = init_bit_array(c, width);\n> +\n> +\t\tbitmap_set(bitmap, i);\n> +\t\tinsert_no_dup(&queue, c);\n> +\t}\n> +\n> +\twhile (queue_has_nonstale(&queue)) {\n> +\t\tstruct commit *c = prio_queue_get(&queue);\n> +\t\tstruct commit_list *p;\n> +\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n> +\n> +\t\tfor (i = 0; i < counts_nr; i++) {\n> +\t\t\tint reach_from_tip = bitmap_get(bitmap_c, counts[i].tip_index);\n> +\t\t\tint reach_from_base = bitmap_get(bitmap_c, counts[i].base_index);\n\nSince we're XORing these, I'd hate to get bit by bitmap_get returning\nsomething other than 0 or 1. It doesn't, since the return value (for any\n\"pos\" for which it holds that `EWAH_BLOCKI(pos) < self->word_alloc`) is:\n\n (self->words[EWAH_BLOCK(pos)] & EWAH_MASK(pos)) != 0\n\nso we'll always be guaranteed to zero or one. But if we retuned instead:\n\n self->words[EWAH_BLOCK(pos)] & EWAH_MASK(pos)\n\n...this code would break in a very annoying and hard-to-debug way ;-).\n\nI wonder if we might do a little of belt-and-suspenders here by calling\nthese like:\n\n int reach_from_tip = !!(bitmap_get(bitmap_c, counts[i].tip_index));\n int reach_from_base = !!(bitmap_get(bitmap_c, counts[i].base_index));\n\nwhere the \"!!(...)\" is new.\n\n> +\t\t\tif (reach_from_tip ^ reach_from_base) {\n> +\t\t\t\tif (reach_from_base)\n> +\t\t\t\t\tcounts[i].behind++;\n> +\t\t\t\telse\n> +\t\t\t\t\tcounts[i].ahead++;\n> +\t\t\t}\n> +\t\t}\n\nI have gone back and forth so many times on this code :-). I think the\nXORs are fine, though.\n\n> +\t\tfor (p = c->parents; p; p = p->next) {\n> +\t\t\tstruct bitmap *bitmap_p;\n> +\n> +\t\t\tparse_commit(p->item);\n> +\n> +\t\t\tbitmap_p = init_bit_array(p->item, width);\n> +\t\t\tbitmap_or(bitmap_p, bitmap_c);\n> +\n> +\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n> +\t\t\t\tp->item->object.flags |= STALE;\n> +\n> +\t\t\tinsert_no_dup(&queue, p->item);\n\nDo we care about inserting p->item when the above condition is met? IOW,\nwould it be OK to instead write:\n\n if (bitmap_popcount(bitmap_p) == commits_nr)\n p->item->object.flags |= STALE;\n else\n insert_no_dup(&queue, p->item);\n\n> diff --git a/commit-reach.h b/commit-reach.h\n> index 148b56fea50..1780f9317bf 100644\n> --- a/commit-reach.h\n> +++ b/commit-reach.h\n> @@ -104,4 +104,34 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n> \t\t\t\t\t struct commit **to, int nr_to,\n> \t\t\t\t\t unsigned int reachable_flag);\n>\n> +struct ahead_behind_count {\n> +\t/**\n> +\t * As input, the *_index members indicate which positions in\n> +\t * the 'tips' array correspond to the tip and base of this\n> +\t * comparison.\n> +\t */\n> +\tsize_t tip_index;\n> +\tsize_t base_index;\n> +\n> +\t/**\n> +\t * These values store the computed counts for each side of the\n> +\t * symmetric difference:\n> +\t *\n> +\t * 'ahead' stores the number of commits reachable from the tip\n> +\t * and not reachable from the base.\n> +\t *\n> +\t * 'behind' stores the number of commits reachable from the base\n> +\t * and not reachable from the tip.\n> +\t */\n> +\tint ahead;\n> +\tint behind;\n> +};\n\nShould these be unsigned values? I don't think we have a sensible\ninterpretation for what a negative \"ahead\" or \"behind\" could would mean.\nI guess behind \"behind\" by \"N\" means you're \"ahead\" by \"-N\", but I don't\nthink it's practical ;-).\n\n> +\n> +/**\n\nHere and elsewhere, these kind of doc-comments are a little\nnon-standard, and IIRC the opening should instead be \"/*\" (with one\nasterisk instead of two).\n\nThanks,\nTaylor\n"},{"id":"473227","messageId":"7328e095-83c6-dd33-1d36-9220612e99c0@github.com","threadId":"59347","inReplyTo":"ZAaH/iCsqdewYrUj@nand.local","subject":"Re: [PATCH 1/8] ahead-behind: create empty builtin","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-08T22:14:37Z","receivedAt":"2023-03-08T22:14:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/6/2023 7:40 PM, Taylor Blau wrote:\n> On Mon, Mar 06, 2023 at 10:48:45AM -0800, Junio C Hamano wrote:\n>> \"Derrick Stolee via GitGitGadget\" writes:\n>>\n>>> For example, we will be able to track all local branches relative to an\n>>> upstream branch using an invocation such as\n>>>\n>>> git for-each-ref --format=%(refname) refs/heads/* |\n>>> git ahead-behind --base=origin/main --stdin\n>>\n>> Stepping back a bit, this motivating example makes me wonder if\n>>\n>> $ git for-each-ref --format='%(refname) %(aheadbehind)' refs/heads/\\*\n> \n> One disadvantage to using for-each-ref here is that we are bound to use\n> all of the ref-sorting code, so callers can't see intermediate results\n> until the entire walk is complete.\n> \n> I can't remember enough of the details about the custom traversal we use\n> here to know if that would even matter or not (i.e., do we need to\n> traverse through the whole set of objects entirely before outputting a\n> single result anyway?). But something to think about nonetheless.\n> \n> At the very least, it is quite a cute idea (especially something like\n> '%(aheadbehind:origin/main)') ;-).\n> \n>> that computes the ahead-behind number for each ref (that matches the\n>> pattern) based on their own \"upstream\" (presumably each branch is\n>> configured to track the same, or different, upstreams), or\n>> overrriding @{upstream}, a specified base, i.e.\n>>\n>> $ git for-each-ref --format='%(refname) %(aheadbehind:origin/main)' refs/heads/\\*\n>>\n>> would be a more intuitive interface to the end-users.\n>>\n>> It would probably work well in conjunction with\n>>\n>> git for-each-ref --format='%(refname)' --merged origin/main refs/heads/\\*\n>>\n>> which is a way to list local branches that are already merged into\n>> the upstream, to have the feature appear in the same command,\n>> perhaps?\n> \n> One thing that we had talked about internally[^1] was the idea of\n> specifying multiple bases. IOW, having some way to invoke the\n> ahead-behind builtin that gives some set of tips with a common base B1,\n> and another set of tips (which could--but doesn't have to--intersect\n> with the first) and a common base to compare *them* to, say, B2.\n> \n> There are some technical reasons that we might want to consider such a\n> thing at least motivated by GitHub's proposed future use of it. But they\n> are kind of technical and not that interesting to this discussion, so I\n> wouldn't be sad if we didn't have a way to specify multiple bases.\n> \n> OTOH, it would be nice to avoid painting ourselves into a corner from a\n> UI-perspective if we can avoid it.\n> \n> Thanks,\n> Taylor\n> \n> [^1]: ...and couldn't decide if it was going to be a nice future\n> addition or simply another case of YAGNI ;-).\n\nThis use of 'git for-each-ref --format=\"\"' actually fixes some of the\nissues I had with how to specify multiple bases. I'm not sure there is\na huge need for it, except that if we allow a \"%(ahead-behind:)\"\nformat token, then we would need to support multiple bases.\n\nThankfully, the implementation in this series is already prepared for\nthat, so the following diff implements this format token:\n\n--- >8 ---\n\n builtin/for-each-ref.c | 50 ++++++++++++++++++++++++++++++++++++++++++\n ref-filter.c | 23 +++++++++++++++++++\n ref-filter.h | 15 ++++++++++++-\n 3 files changed, 87 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 6f62f40d126..c8dd21d7e13 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n+#include \"commit-reach.h\"\n \n static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n@@ -14,6 +15,51 @@ static char const * const for_each_ref_usage[] = {\n \tNULL\n };\n \n+static void compute_ahead_behind(struct ref_format *format,\n+\t\t\t\t struct ref_array *array)\n+{\n+\tstruct commit **commits;\n+\tsize_t commits_nr = format->bases.nr + array->nr;\n+\n+\tif (!format->bases.nr || !array->nr)\n+\t\treturn;\n+\n+\tALLOC_ARRAY(commits, commits_nr);\n+\tfor (size_t i = 0; i < format->bases.nr; i++) {\n+\t\tconst char *name = format->bases.items[i].string;\n+\t\tcommits[i] = lookup_commit_reference_by_name(name);\n+\t\tif (!commits[i])\n+\t\t\tdie(\"failed to find '%s'\", name);\n+\t}\n+\n+\tALLOC_ARRAY(array->counts, st_mult(format->bases.nr, array->nr));\n+\n+\tcommits_nr = format->bases.nr;\n+\tarray->counts_nr = 0;\n+\tfor (size_t i = 0; i < array->nr; i++) {\n+\t\tconst char *name = array->items[i]->refname;\n+\t\tcommits[commits_nr] = lookup_commit_reference_by_name(name);\n+\n+\t\tif (!commits[commits_nr]) {\n+\t\t\twarning(_(\"could not find '%s'\"), name);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tCALLOC_ARRAY(array->items[i]->counts, format->bases.nr);\n+\t\tfor (size_t j = 0; j < format->bases.nr; j++) {\n+\t\t\tstruct ahead_behind_count *count;\n+\t\t\tcount = &array->counts[array->counts_nr++];\n+\t\t\tcount->tip_index = format->bases.nr + i;\n+\t\t\tcount->base_index = j;\n+\n+\t\t\tarray->items[i]->counts[j] = count;\n+\t\t}\n+\t\tcommits_nr++;\n+\t}\n+\n+\tahead_behind(commits, commits_nr, array->counts, array->counts_nr);\n+}\n+\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -78,6 +124,10 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tfilter.name_patterns = argv;\n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n+\n+\t/* Do ahead-behind things, if necessary. */\n+\tcompute_ahead_behind(&format, &array);\n+\n \tref_array_sort(sorting, &array);\n \n \tif (!maxcount || array.nr < maxcount)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f8203c6b052..1706b9dd0d5 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -158,6 +158,7 @@ enum atom_type {\n \tATOM_THEN,\n \tATOM_ELSE,\n \tATOM_REST,\n+\tATOM_AHEADBEHIND,\n };\n \n /*\n@@ -586,6 +587,16 @@ static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int ahead_behind_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"expected format: %%(ahead-behind:)\"));\n+\n+\tstring_list_append(&format->bases, arg);\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -645,6 +656,7 @@ static struct {\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n \t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n+\t[ATOM_AHEADBEHIND] = { \"ahead-behind\", SOURCE_OTHER, FIELD_STR, ahead_behind_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -1848,6 +1860,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \tstruct object *obj;\n \tint i;\n \tstruct object_info empty = OBJECT_INFO_INIT;\n+\tint ahead_behind_atoms = 0;\n \n \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n \n@@ -1978,6 +1991,16 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\telse\n \t\t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_AHEADBEHIND) {\n+\t\t\tif (ref->counts) {\n+\t\t\t\tconst struct ahead_behind_count *count;\n+\t\t\t\tcount = ref->counts[ahead_behind_atoms++];\n+\t\t\t\tv->s = xstrfmt(\"%d %d\", count->ahead, count->behind);\n+\t\t\t} else {\n+\t\t\t\t/* Not a commit. */\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\t}\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex aa0eea4ecf5..937a857ddee 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -5,6 +5,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"parse-options.h\"\n+#include \"string-list.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -24,6 +25,7 @@\n \n struct atom_value;\n struct ref_sorting;\n+struct ahead_behind_count;\n \n enum ref_sorting_order {\n \tREF_SORTING_REVERSE = 1<<0,\n@@ -40,6 +42,8 @@ struct ref_array_item {\n \tconst char *symref;\n \tstruct commit *commit;\n \tstruct atom_value *value;\n+\tstruct ahead_behind_count **counts;\n+\n \tchar refname[FLEX_ARRAY];\n };\n \n@@ -47,6 +51,9 @@ struct ref_array {\n \tint nr, alloc;\n \tstruct ref_array_item **items;\n \tstruct rev_info *revs;\n+\n+\tstruct ahead_behind_count *counts;\n+\tsize_t counts_nr;\n };\n \n struct ref_filter {\n@@ -80,9 +87,15 @@ struct ref_format {\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n+\n+\t/* List of bases for ahead-behind counts. */\n+\tstruct string_list bases;\n };\n \n-#define REF_FORMAT_INIT { .use_color = -1 }\n+#define REF_FORMAT_INIT { \\\n+\t.use_color = -1, \\\n+\t.bases = STRING_LIST_INIT_DUP, \\\n+}\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\n-- \n2.40.0.vfs.0.0.3.g5872ac9aaa4\n\n--- >8 ---\n\nI can already see some things I want to change about this quick\nand dirty implementation, but it gets the point across. This\n\"test\" can be added to the end of t6302 for some demonstration:\n\ntest_expect_success 'ahead-behind' '\n\tgit for-each-ref --format=\"%(refname) %(ahead-behind:HEAD)\" &&\n\tgit for-each-ref --format=\"%(refname) %(ahead-behind:HEAD) %(ahead-behind:refs/heads/side)\"\n'\n\nWhat I have yet to determine is that 'git for-each-ref' does\nnot have significant overhead due to how it's implementation is\nbuilt around listing \"all refs that match\" versus an explicit\ninput list of refs. There's also the concept of '--stdin' that\nwould be interesting to interact with.\n\nI'll continue to investigate this path and report back when I\nhave more of this information. This is as far I as I could get\ntoday.\n\nThanks,\n-Stolee\n"},{"id":"473234","messageId":"xmqq4jqusvhq.fsf@gitster.g","threadId":"59347","inReplyTo":"7328e095-83c6-dd33-1d36-9220612e99c0@github.com","subject":"Re: [PATCH 1/8] ahead-behind: create empty builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-08T22:56:17Z","receivedAt":"2023-03-08T22:56:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> What I have yet to determine is that 'git for-each-ref' does\n> not have significant overhead due to how it's implementation is\n> built around listing \"all refs that match\" versus an explicit\n> input list of refs. There's also the concept of '--stdin' that\n> would be interesting to interact with.\n\nYeah, we could add --no-sort to allow streaming better and --stdin\nto feed list of refs to work on, if the end-user facing interface\nbased on --format is what people find reasonable.\n\n> I'll continue to investigate this path and report back when I\n> have more of this information. This is as far I as I could get\n> today.\n\nThanks. It is a very interesting experiment.\n\n"},{"id":"473268","messageId":"ZAmk9V54c/jIbJ65@coredump.intra.peff.net","threadId":"59347","inReplyTo":"ZAaHB7XTz8HX75LD@nand.local","subject":"Re: [PATCH 0/8] ahead-behind: new builtin for counting multiple commit ranges","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-09T09:20:53Z","receivedAt":"2023-03-09T09:21:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 06, 2023 at 07:36:23PM -0500, Taylor Blau wrote:\n\n> > This makes readers wonder if \"git rev-list --count B...C\" should be\n> > the end-user facing UI for this new feature, perhaps?\n> >\n> > Of course if you are checking how C0, C1, C2,... relate to a single\n> > B, the existing rev-list syntax would not work, and makes a totally\n> > new subcommand a possibilty.\n> \n> Yeah. You could imagine that `rev-list --count` might do something\n> fancy like coalescing\n> \n> git rev-list --count B...C1 B...C2 B...C3\n> \n> into a single walk. But I am not sure that just because `rev-list\n> --count` provides similar functionality that we should fold in the\n> proposed `ahead-behind` interface into that flag.\n\nIt does coalesce all of that into a single walk. The problem is somewhat\nthe opposite: it only has a notion of two \"sides\" for a symmetric\ntraversal: left and right. But in your example there are many sides, and\nwe have to remember which is which.\n\nI think getting the answer from one walk would require an arbitrary\nnumber of bits to paint down each path. Certainly the ahead-behind that\nVicent and I wrote long ago didn't do that (IIRC it mostly relied on\ndoing multiple traversals in the same process, which amortized the cost\nof commit parsing; that's not really an issue these days with commit\ngraphs).\n\nPeeking at patch 7 of Stolee's series...yep. That's exactly what it\ndoes. :)\n\nI wondered how much it would matter on top of a naive loop of\nsingle-traversals, now that we have commit graphs. It looks like there's\nstill quite a nice speedup from the numbers in patch 7 (though the\ntotally naive \"loop of rev-list\" is incurring extra startup overhead,\ntoo).\n\n> My personal feeling is that we ought to avoid (further) overloading\n> `rev-list` absent of a compelling reason to do so. But I am definitely\n> open to other thoughts here.\n\nSo I think this actually is what \"git rev-list --left-right --count\nold...new\" does now. But extending it to multiple sets in one traversal\nmeans you need:\n\n - being able to ask for individual left-right markers for each pair,\n not treating all lefts and all rights together\n\n - don't stop traversing when you hit an UNINTERESTING commit if there\n are still bits to paint. In a single-pair traversal, those two are\n the same thing (we stop at the merge base), but with multiple pairs\n you may have to keep walking past a commit that is excluded from one\n pair, but not another. This _might_ be doable if you assume all of\n the left-hand bases are the same, but I didn't think hard enough to\n feel confident in that. But even so, that only solves cases like\n \"how do these branches compare to HEAD\" (which I think is what\n GitHub does). But it doesn't allow \"how do these branches compare to\n to their respective @{upstream} refs\".\n\nSo I don't think it would be impossible to make this a mode of rev-list.\nAnd that mode might even provide flexibility for other similar\noperations, like a mass \"git rev-list --cherry-mark\"[1]. But it is a\npretty big departure from the current rev-list traversal (to my mind,\nespecially the \"keep walking past UNINTERESTING part). I don't mind it\nas its own command.\n\n-Peff\n\n[1] The reason you might want a mass cherry-mark is basically doing\n something like the \"branches\" page, but in a workflow where upstream\n applies patches, like git.git. There you may want to ask about\n \"origin/next...$branch\" for all of your branches to see which ones\n have been merged where.\n"},{"id":"473301","messageId":"068757a3-9dcd-4158-9dc3-d939ff4ed484@github.com","threadId":"59347","inReplyTo":"ZAaN4w4kltPIeYlt@nand.local","subject":"Re: [PATCH 7/8] ahead-behind: implement ahead_behind() logic","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-09T17:32:08Z","receivedAt":"2023-03-09T17:32:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/6/2023 8:05 PM, Taylor Blau wrote:\n> On Mon, Mar 06, 2023 at 02:06:37PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> Signed-off-by: Derrick Stolee \n> \n> Having read and worked with this code before, I don't have a ton of\n> substance to add here. But it was interesting to reread, and I left a\n> few sprinklings here and there of some thing that we may want to\n> consider for v2.\n> \n> Before that, though, IIRC we wrote most of this together, so I would be\n> happy to have my:\n> \n> Co-authored-by: Taylor Blau \n> Signed-off-by: Taylor Blau \n> \n> above your S-o-b here. But you've done so much work since we originally\n> wrote this together that I don't mind being dropped here. Up to you :-).\n\nSounds good. Sorry for forgetting that collaboration.\n\n>> +static void insert_no_dup(struct prio_queue *queue, struct commit *c)\n>> +{\n>> +\tif (c->object.flags & PARENT2)\n>> +\t\treturn;\n>> +\tprio_queue_put(queue, c);\n>> +\tc->object.flags |= PARENT2;\n>> +}\n> \n> You mentioned this in the patch message, but:\n> \n> It may be worth noting here (or in the call to repo_clear_commit_marks()\n> below) that the PARENT2 flag is used to detect and avoid duplicates in\n> this list.\n\nI'll add a comment before we clear the bits, to be clear about\nwhy we need each bit.\n \n>> +\twhile (queue_has_nonstale(&queue)) {\n>> +\t\tstruct commit *c = prio_queue_get(&queue);\n>> +\t\tstruct commit_list *p;\n>> +\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n>> +\n>> +\t\tfor (i = 0; i < counts_nr; i++) {\n>> +\t\t\tint reach_from_tip = bitmap_get(bitmap_c, counts[i].tip_index);\n>> +\t\t\tint reach_from_base = bitmap_get(bitmap_c, counts[i].base_index);\n> \n> Since we're XORing these, I'd hate to get bit by bitmap_get returning\n> something other than 0 or 1. It doesn't, since the return value (for any\n> \"pos\" for which it holds that `EWAH_BLOCKI(pos) < self->word_alloc`) is:\n\n> I wonder if we might do a little of belt-and-suspenders here by calling\n> these like:\n> \n> int reach_from_tip = !!(bitmap_get(bitmap_c, counts[i].tip_index));\n> int reach_from_base = !!(bitmap_get(bitmap_c, counts[i].base_index));\n> \n> where the \"!!(...)\" is new.\n\nCan't hurt.\n \n>> +\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n>> +\t\t\t\tp->item->object.flags |= STALE;\n>> +\n>> +\t\t\tinsert_no_dup(&queue, p->item);\n> \n> Do we care about inserting p->item when the above condition is met? IOW,\n> would it be OK to instead write:\n> \n> if (bitmap_popcount(bitmap_p) == commits_nr)\n> p->item->object.flags |= STALE;\n> else\n> insert_no_dup(&queue, p->item);\n\nWe need to push p->item to the queue, even if stale, because it\nmay need to be walked in order to pass the bitmaps (and the STALE\nbit) to other commits that were reached by only a subset of the\ntips.\n\nHere's an example:\n\n A B\n |/ \\\n C D\n | /\n E /\n |/\n F\n\nIf A and B are the starting commits, then C is stale when we\nwalk it, so its parent E would be stale. Your proposed change\nwould not add it to the queue, and thus F would never see that\nit is stale and would be counted as reachable from B but not A.\n\n>> diff --git a/commit-reach.h b/commit-reach.h\n>> index 148b56fea50..1780f9317bf 100644\n>> --- a/commit-reach.h\n>> +++ b/commit-reach.h\n>> @@ -104,4 +104,34 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n>> \t\t\t\t\t struct commit **to, int nr_to,\n>> \t\t\t\t\t unsigned int reachable_flag);\n>>\n>> +struct ahead_behind_count {\n>> +\t/**\n>> +\t * As input, the *_index members indicate which positions in\n>> +\t * the 'tips' array correspond to the tip and base of this\n>> +\t * comparison.\n>> +\t */\n>> +\tsize_t tip_index;\n>> +\tsize_t base_index;\n>> +\n>> +\t/**\n>> +\t * These values store the computed counts for each side of the\n>> +\t * symmetric difference:\n>> +\t *\n>> +\t * 'ahead' stores the number of commits reachable from the tip\n>> +\t * and not reachable from the base.\n>> +\t *\n>> +\t * 'behind' stores the number of commits reachable from the base\n>> +\t * and not reachable from the tip.\n>> +\t */\n>> +\tint ahead;\n>> +\tint behind;\n>> +};\n> \n> Should these be unsigned values? I don't think we have a sensible\n> interpretation for what a negative \"ahead\" or \"behind\" could would mean.\n> I guess behind \"behind\" by \"N\" means you're \"ahead\" by \"-N\", but I don't\n> think it's practical ;-).\nUnsigned sounds good.\n \n>> +\n>> +/**\n> \n> Here and elsewhere, these kind of doc-comments are a little\n> non-standard, and IIRC the opening should instead be \"/*\" (with one\n> asterisk instead of two).\n\nI think double-asterisk is the preferred choice for new things, but\ncommit-reach.h only uses a single asterisk so I'll change this to\nbe consistent.\n\nThanks,\n-Stolee\n"},{"id":"473318","messageId":"xmqqv8j98uga.fsf@gitster.g","threadId":"59347","inReplyTo":"ZAmk9V54c/jIbJ65@coredump.intra.peff.net","subject":"Re: [PATCH 0/8] ahead-behind: new builtin for counting multiple commit ranges","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-09T21:51:17Z","receivedAt":"2023-03-09T21:51:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Yeah. You could imagine that `rev-list --count` might do something\n>> fancy like coalescing\n>> \n>> git rev-list --count B...C1 B...C2 B...C3\n>> \n>> into a single walk. But I am not sure that just because `rev-list\n>> --count` provides similar functionality that we should fold in the\n>> proposed `ahead-behind` interface into that flag.\n>\n> It does coalesce all of that into a single walk. The problem is somewhat\n> the opposite: it only has a notion of two \"sides\" for a symmetric\n> traversal: left and right. But in your example there are many sides, and\n> we have to remember which is which.\n\nYeah, this reminds me of what I had to do in \"show-branch\", where\neach tip gets assigned a bit in the object->flags (which means it\ncan only traverse from a very small limited number of tips, like 30\nor so), which I once planned to extend to arbitrary number of tips\nby storing these bits in commit slab, but it never materialized.\n\n> So I don't think it would be impossible to make this a mode of rev-list.\n> And that mode might even provide flexibility for other similar\n> operations, like a mass \"git rev-list --cherry-mark\"[1]. But it is a\n> pretty big departure from the current rev-list traversal (to my mind,\n> especially the \"keep walking past UNINTERESTING part). I don't mind it\n> as its own command.\n\nI agree this is not a good fit for the mental model of rev-list or\nthe revision.c::get_revision() traversal.\n\nThanks.\n"},{"id":"473361","messageId":"a1d9e0f6ff6660c9264673be18bc24956f74eb9c.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:20:56Z","receivedAt":"2023-03-10T17:21:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a user wishes to input a large list of patterns to 'git\nfor-each-ref' (likely a long list of exact refs) there are frequently\nsystem limits on the number of command-line arguments.\n\nAdd a new --stdin option to instead read the patterns from standard\ninput. Add tests that check that any unrecognized arguments are\nconsidered an error when --stdin is provided. Also, an empty pattern\nlist is interpreted as the complete ref set.\n\nWhen reading from stdin, we populate the filter.name_patterns array\ndynamically as opposed to pointing to the 'argv' array directly. This\nrequires a careful cast while freeing the individual strings,\nconditioned on the --stdin option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-for-each-ref.txt | 7 +++++-\n builtin/for-each-ref.c | 29 ++++++++++++++++++++++-\n t/t6300-for-each-ref.sh | 37 ++++++++++++++++++++++++++++++\n 3 files changed, 71 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 6da899c6296..ccdc2911bb9 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -9,7 +9,8 @@ SYNOPSIS\n --------\n [verse]\n 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n-\t\t [(--sort=)...] [--format=] [...]\n+\t\t [(--sort=)...] [--format=]\n+\t\t [ --stdin | ... ]\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n@@ -32,6 +33,10 @@ OPTIONS\n \tliterally, in the latter case matching completely or from the\n \tbeginning up to a slash.\n \n+--stdin::\n+\tIf `--stdin` is supplied, then the list of patterns is read from\n+\tstandard input instead of from the argument list.\n+\n --count=::\n \tBy default the command shows all refs that match\n \t``. This option makes it stop after showing\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 6f62f40d126..e005a7ef3ce 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -25,6 +25,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n+\tint from_stdin = 0;\n \n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n@@ -49,6 +50,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_CONTAINS(&filter.with_commit, N_(\"print only refs which contain the commit\")),\n \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n+\t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n \t\tOPT_END(),\n \t};\n \n@@ -75,7 +77,27 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n \tfilter.ignore_case = icase;\n \n-\tfilter.name_patterns = argv;\n+\tif (from_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\t\tsize_t nr = 0, alloc = 16;\n+\n+\t\tif (argv[0])\n+\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n+\n+\t\tCALLOC_ARRAY(filter.name_patterns, alloc);\n+\n+\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n+\t\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n+\t\t\tfilter.name_patterns[nr++] = strbuf_detach(&line, NULL);\n+\t\t}\n+\n+\t\t/* Add a terminating NULL string. */\n+\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n+\t\tfilter.name_patterns[nr + 1] = NULL;\n+\t} else {\n+\t\tfilter.name_patterns = argv;\n+\t}\n+\n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n \tref_array_sort(sorting, &array);\n@@ -97,5 +119,10 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tfree_commit_list(filter.with_commit);\n \tfree_commit_list(filter.no_commit);\n \tref_sorting_release(sorting);\n+\tif (from_stdin) {\n+\t\tfor (size_t i = 0; filter.name_patterns[i]; i++)\n+\t\t\tfree((char *)filter.name_patterns[i]);\n+\t\tfree(filter.name_patterns);\n+\t}\n \treturn 0;\n }\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex c466fd989f1..a58053a54c5 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1464,4 +1464,41 @@ sig_crlf=\"$(printf \"%s\" \"$sig\" | append_cr; echo dummy)\"\n sig_crlf=${sig_crlf%dummy}\n test_atom refs/tags/fake-sig-crlf contents:signature \"$sig_crlf\"\n \n+test_expect_success 'git for-each-ref --stdin: empty' '\n+\t>in &&\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\tgit for-each-ref --format=\"%(refname)\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git for-each-ref --stdin: fails if extra args' '\n+\t>in &&\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" \\\n+\t\t--stdin refs/heads/extra err &&\n+\tgrep \"unknown arguments supplied with --stdin\" err\n+'\n+\n+test_expect_success 'git for-each-ref --stdin: matches' '\n+\tcat >in <<-EOF &&\n+\trefs/tags/multi*\n+\trefs/heads/amb*\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/tags/multi-ref1-100000-user1\n+\trefs/tags/multi-ref1-100000-user2\n+\trefs/tags/multi-ref1-200000-user1\n+\trefs/tags/multi-ref1-200000-user2\n+\trefs/tags/multi-ref2-100000-user1\n+\trefs/tags/multi-ref2-100000-user2\n+\trefs/tags/multi-ref2-200000-user1\n+\trefs/tags/multi-ref2-200000-user2\n+\trefs/tags/multiline\n+\tEOF\n+\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473362","messageId":"2f162a2f39f5d413fbe196e56d15ecf3080a2a8b.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 2/8] for-each-ref: explicitly test no matches","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:20:57Z","receivedAt":"2023-03-10T17:21:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe for-each-ref builtin can take a list of ref patterns, but if none\nmatch, it still succeeds (but with no output). Add an explicit test that\ndemonstrates that behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t6300-for-each-ref.sh | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex a58053a54c5..6614469d2d6 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1501,4 +1501,17 @@ test_expect_success 'git for-each-ref --stdin: matches' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git for-each-ref with non-existing refs' '\n+\tcat >in <<-EOF &&\n+\trefs/heads/this-ref-does-not-exist\n+\trefs/tags/bogus\n+\tEOF\n+\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\ttest_must_be_empty actual &&\n+\n+\txargs git for-each-ref --format=\"%(refname)\" actual &&\n+\ttest_must_be_empty actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473363","messageId":"db28e82d2a60ebd8d2dc31f11fed0e408c2799e8.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 3/8] commit-graph: combine generation computations","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:20:58Z","receivedAt":"2023-03-10T17:21:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis patch extracts the common code used to compute topological levels\nand corrected committer dates into a common routine,\ncompute_reachable_generation_numbers_1().\n\nThis new routine dispatches to call the necessary functions to get and\nset the generation number for a given commit through a vtable (the\ncompute_generation_info struct).\n\nComputing the generation number itself is done in\ncompute_generation_from_max(), which dispatches its implementation based\non the generation version requested, or issuing a BUG() for unrecognized\ngeneration versions.\n\nThis patch cleans up the two places that currently compute topological\nlevels and corrected commit dates by reducing the amount of duplicated\ncode. It also makes it possible to introduce a function which\ndynamically computes those values for commits that aren't stored in a\ncommit-graph, which will be required for the forthcoming ahead-behind\nrewrite.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 171 +++++++++++++++++++++++++++++++------------------\n 1 file changed, 107 insertions(+), 64 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b3..deccf984a0d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1446,24 +1446,53 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n-static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+struct compute_generation_info {\n+\tstruct repository *r;\n+\tstruct packed_commit_list *commits;\n+\tstruct progress *progress;\n+\tint progress_cnt;\n+\n+\ttimestamp_t (*get_generation)(struct commit *c, void *data);\n+\tvoid (*set_generation)(struct commit *c, timestamp_t gen, void *data);\n+\tvoid *data;\n+};\n+\n+static timestamp_t compute_generation_from_max(struct commit *c,\n+\t\t\t\t\t timestamp_t max_gen,\n+\t\t\t\t\t int generation_version)\n+{\n+\tswitch (generation_version) {\n+\tcase 1: /* topological levels */\n+\t\tif (max_gen > GENERATION_NUMBER_V1_MAX - 1)\n+\t\t\tmax_gen = GENERATION_NUMBER_V1_MAX - 1;\n+\t\treturn max_gen + 1;\n+\n+\tcase 2: /* corrected commit date */\n+\t\tif (c->date && c->date > max_gen)\n+\t\t\tmax_gen = c->date - 1;\n+\t\treturn max_gen + 1;\n+\n+\tdefault:\n+\t\tBUG(\"attempting unimplemented version\");\n+\t}\n+}\n+\n+static void compute_reachable_generation_numbers_1(\n+\t\t\tstruct compute_generation_info *info,\n+\t\t\tint generation_version)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n \n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n-\t\t\t\t\tctx->commits.nr);\n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\tuint32_t level;\n+\tfor (i = 0; i < info->commits->nr; i++) {\n+\t\tstruct commit *c = info->commits->list[i];\n+\t\ttimestamp_t gen;\n+\t\trepo_parse_commit(info->r, c);\n+\t\tgen = info->get_generation(c, info->data);\n \n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n+\t\tdisplay_progress(info->progress, info->progress_cnt + 1);\n \n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (level != GENERATION_NUMBER_ZERO)\n+\t\tif (gen != GENERATION_NUMBER_ZERO && gen != GENERATION_NUMBER_INFINITY)\n \t\t\tcontinue;\n \n \t\tcommit_list_insert(c, &list);\n@@ -1471,38 +1500,91 @@ static void compute_topological_levels(struct write_commit_graph_context *ctx)\n \t\t\tstruct commit *current = list->item;\n \t\t\tstruct commit_list *parent;\n \t\t\tint all_parents_computed = 1;\n-\t\t\tuint32_t max_level = 0;\n+\t\t\tuint32_t max_gen = 0;\n \n \t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tlevel = *topo_level_slab_at(ctx->topo_levels, parent->item);\n+\t\t\t\trepo_parse_commit(info->r, parent->item);\n+\t\t\t\tgen = info->get_generation(parent->item, info->data);\n \n-\t\t\t\tif (level == GENERATION_NUMBER_ZERO) {\n+\t\t\t\tif (gen == GENERATION_NUMBER_ZERO) {\n \t\t\t\t\tall_parents_computed = 0;\n \t\t\t\t\tcommit_list_insert(parent->item, &list);\n \t\t\t\t\tbreak;\n \t\t\t\t}\n \n-\t\t\t\tif (level > max_level)\n-\t\t\t\t\tmax_level = level;\n+\t\t\t\tif (gen > max_gen)\n+\t\t\t\t\tmax_gen = gen;\n \t\t\t}\n \n \t\t\tif (all_parents_computed) {\n \t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (max_level > GENERATION_NUMBER_V1_MAX - 1)\n-\t\t\t\t\tmax_level = GENERATION_NUMBER_V1_MAX - 1;\n-\t\t\t\t*topo_level_slab_at(ctx->topo_levels, current) = max_level + 1;\n+\t\t\t\tgen = compute_generation_from_max(\n+\t\t\t\t\t\tcurrent, max_gen,\n+\t\t\t\t\t\tgeneration_version);\n+\t\t\t\tinfo->set_generation(current, gen, info->data);\n \t\t\t}\n \t\t}\n \t}\n+}\n+\n+static timestamp_t get_topo_level(struct commit *c, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\treturn *topo_level_slab_at(ctx->topo_levels, c);\n+}\n+\n+static void set_topo_level(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\t*topo_level_slab_at(ctx->topo_levels, c) = (uint32_t)t;\n+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n+}\n+\n+static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+{\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.progress = ctx->progress,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_topo_level,\n+\t\t.set_generation = set_topo_level,\n+\t\t.data = ctx,\n+\t};\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tcompute_reachable_generation_numbers_1(&info, 1);\n+\n \tstop_progress(&ctx->progress);\n }\n \n+static timestamp_t get_generation_from_graph_data(struct commit *c, void *data)\n+{\n+\treturn commit_graph_data_at(c)->generation;\n+}\n+\n+static void set_generation_v2(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\tstruct commit_graph_data *g = commit_graph_data_at(c);\n+\tg->generation = (uint32_t)t;\n+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n+}\n+\n static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n-\tstruct commit_list *list = NULL;\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.progress = ctx->progress,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_v2,\n+\t\t.data = ctx,\n+\t};\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1517,47 +1599,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\t}\n \t}\n \n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\ttimestamp_t corrected_commit_date;\n-\n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tcorrected_commit_date = commit_graph_data_at(c)->generation;\n-\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (corrected_commit_date != GENERATION_NUMBER_ZERO)\n-\t\t\tcontinue;\n-\n-\t\tcommit_list_insert(c, &list);\n-\t\twhile (list) {\n-\t\t\tstruct commit *current = list->item;\n-\t\t\tstruct commit_list *parent;\n-\t\t\tint all_parents_computed = 1;\n-\t\t\ttimestamp_t max_corrected_commit_date = 0;\n-\n-\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tcorrected_commit_date = commit_graph_data_at(parent->item)->generation;\n-\n-\t\t\t\tif (corrected_commit_date == GENERATION_NUMBER_ZERO) {\n-\t\t\t\t\tall_parents_computed = 0;\n-\t\t\t\t\tcommit_list_insert(parent->item, &list);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\n-\t\t\t\tif (corrected_commit_date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = corrected_commit_date;\n-\t\t\t}\n-\n-\t\t\tif (all_parents_computed) {\n-\t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n-\t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n-\t\t\t}\n-\t\t}\n-\t}\n+\tcompute_reachable_generation_numbers_1(&info, 2);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n@@ -1565,6 +1607,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n \t\t\tctx->num_generation_data_overflows++;\n \t}\n+\n \tstop_progress(&ctx->progress);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"473364","messageId":"3cf3380144347630137fece29dbc991d31ff3af2.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 4/8] commit-graph: return generation from memory","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:20:59Z","receivedAt":"2023-03-10T17:21:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit_graph_generation() method used to report a value of\nGENERATION_NUMBER_INFINITY if the commit_graph_data_slab had an instance\nfor the given commit but the graph_pos indicated the commit was not in\nthe commit-graph file.\n\nInstead, trust the 'generation' member if the commit has a value in the\nslab _and_ the 'generation' member is non-zero. Otherwise, treat it as\nGENERATION_NUMBER_INFINITY.\n\nThis only makes a difference for a very old case for the commit-graph:\nthe very first Git release to write commit-graph files wrote zeroes in\nthe topological level positions. If we are parsing a commit-graph with\nall zeroes, those commits will now appear to have\nGENERATION_NUMBER_INFINITY (as if they were not parsed from the\ncommit-graph).\n\nI attempted several variations to work around the need for providing an\nuninitialized 'generation' member, but this was the best one I found. It\ndoes require a change to a verification test in t5318 because it reports\na different error than the one about non-zero generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 8 +++-----\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 4 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex deccf984a0d..b4da4e05067 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -116,12 +116,10 @@ timestamp_t commit_graph_generation(const struct commit *c)\n \tstruct commit_graph_data *data =\n \t\tcommit_graph_data_slab_peek(&commit_graph_data_slab, c);\n \n-\tif (!data)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n-\telse if (data->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n+\tif (data && data->generation)\n+\t\treturn data->generation;\n \n-\treturn data->generation;\n+\treturn GENERATION_NUMBER_INFINITY;\n }\n \n static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 049c5fc8ead..b6e12115786 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -630,7 +630,7 @@ test_expect_success 'detect incorrect generation number' '\n \n test_expect_success 'detect incorrect generation number' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n-\t\t\"non-zero generation number\"\n+\t\t\"commit-graph generation for commit\"\n '\n \n test_expect_success 'detect incorrect commit date' '\n-- \ngitgitgadget\n\n"},{"id":"473365","messageId":"34dffd836b1475967593e68e5f8697b260c03d16.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 5/8] commit-graph: introduce `ensure_generations_valid()`","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:21:00Z","receivedAt":"2023-03-10T17:21:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nUse the just-introduced compute_reachable_generation_numbers_1() to\nimplement a function which dynamically computes topological levels (or\ncorrected commit dates) for out-of-graph commits.\n\nThis will be useful for the ahead-behind algorithm we are about to\nintroduce, which needs accurate topological levels on _all_ commits\nreachable from the tips in order to avoid over-counting.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 +++++++++++++++++++++++++++++\n commit-graph.h | 7 +++++++\n 2 files changed, 36 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b4da4e05067..7311d62a110 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1609,6 +1609,35 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void set_generation_in_graph_data(struct commit *c, timestamp_t t,\n+\t\t\t\t\t void *data)\n+{\n+\tcommit_graph_data_at(c)->generation = t;\n+}\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct commit **commits, size_t nr)\n+{\n+\tstruct repository *r = the_repository;\n+\tint generation_version = get_configured_generation_version(r);\n+\tstruct packed_commit_list list = {\n+\t\t.list = commits,\n+\t\t.alloc = nr,\n+\t\t.nr = nr,\n+\t};\n+\tstruct compute_generation_info info = {\n+\t\t.r = r,\n+\t\t.commits = &list,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_in_graph_data,\n+\t};\n+\n+\tcompute_reachable_generation_numbers_1(&info, generation_version);\n+}\n+\n static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n {\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-computed\",\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 37faee6b66d..a529c62b518 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -190,4 +190,11 @@ struct commit_graph_data {\n */\n timestamp_t commit_graph_generation(const struct commit *);\n uint32_t commit_graph_position(const struct commit *);\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct commit **commits, size_t nr);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"473366","messageId":"9831c23eadbcd990ca09628e5846056e4879ee3d.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 6/8] commit-reach: implement ahead_behind() logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:21:01Z","receivedAt":"2023-03-10T17:21:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nFully implement the commit-counting logic required to determine\nahead/behind counts for a batch of commit pairs. This is a new library\nmethod within commit-reach.h. This method will be linked to the\nfor-each-ref builtin in the next change.\n\nThe interface for ahead_behind() uses two arrays. The first array of\ncommits contains the list of all starting points for the walk. This\nincludes all tip commits _and_ base commits. The second array, using the\nnew ahead_behind_count struct, indicates which commits from that initial\narray form the base/tip pair for the ahead/behind count it will store.\n\nThis implementation of ahead_behind() allows multiple bases, if desired.\nEven with multiple bases, there is only one commit walk used for\ncounting the ahead/behind values, saving time when the base/tip ranges\noverlap significantly.\n\nThis interface for ahead_behind() also makes it very easy to call\nensure_generations_valid() on the entire array of bases and tips. This\ncall is necessary because it is critical that the walk that counts\nahead/behind values never walks a commit more than once. Without\ngeneration numbers on every commit, there is a possibility that a\ncommit date skew could cause the walk to revisit a commit and then\ndouble-count it. For this reason, it is strongly recommended that 'git\nahead-behind' is only run in a repository with a commit-graph file that\ncovers most of the reachable commits, storing precomputed generation\nnumbers. If no commit-graph exists, this walk will be much slower as it\nmust walk all reachable commits in ensure_generations_valid() before\nperforming the counting logic.\n\nIt is possible to detect if generation numbers are available at run time\nand redirect the implementation to another algorithm that does not\nrequire this property. However, that implementation requires a commit\nwalk per base/tip pair _and_ can be slower due to the commit date\nheuristics required. Such an implementation could be considered in the\nfuture if there is a reason to include it, but most Git hosts should\nalready be generating a commit-graph file as part of repository\nmaintenance. Most Git clients should also be generating commit-graph\nfiles as part of background maintenance or automatic GCs.\n\nNow, let's discuss the ahead/behind counting algorithm.\n\nEach commit in the input commit list is associated with a bit position\nindicating \"the ith commit can reach this commit\". Each of these commits\nis associated with a bitmap with its position flipped on and then\nplaced in a queue for walking commit history. We walk commits by popping\nthe commit with maximum generation number out of the queue, guaranteeing\nthat we will never walk a child of that commit in any future steps.\n\nAs we walk, we load the bitmap for the current commit and perform two\nmain steps. The _second_ step examines each parent of the current commit\nand adds the current commit's bitmap bits to each parent's bitmap. (We\ncreate a new bitmap for the parent if this is our first time seeing that\nparent.) After adding the bits to the parent's bitmap, the parent is\nadded to the walk queue. Due to this passing of bits to parents, the\ncurrent commit has a guarantee that the ith bit is enabled on its bitmap\nif and only if the ith commit can reach the current commit.\n\nThe first step of the walk is to examine the bitmask on the current\ncommit and decide which ranges the commit is in or not. Due to the \"bit\npushing\" in the second step, we have a guarantee that the ith bit of the\ncurrent commit's bitmap is on if and only if the ith starting commit can\nreach it. For each ahead_behind_count struct, check the base_index and\ntip_index to see if those bits are enabled on the current bitmap. If\nexactly one bit is enabled, then increment the corresponding 'ahead' or\n'behind' count. This increment is the reason we _absolutely need_ to\nwalk commits at most once.\n\nThe only subtle thing to do with this walk is to check to see if a\nparent has all bits on in its bitmap, in which case it becomes \"stale\"\nand is marked with the STALE bit. This allows queue_has_nonstale() to be\nthe terminating condition of the walk, which greatly reduces the number\nof commits walked if all of the commits are nearby in history. It avoids\nwalking a large number of common commits when there is a deep history.\nWe also use the helper method insert_no_dup() to add commits to the\npriority queue without adding them multiple times. This uses the PARENT2\nflag. Thus, we must clear both the STALE and PARENT2 bits of all\ncommits, in case ahead_behind() is called multiple times in the same\nprocess.\n\nCo-authored-by: Taylor Blau \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-reach.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++\n commit-reach.h | 30 ++++++++++++++++\n 2 files changed, 126 insertions(+)\n\ndiff --git a/commit-reach.c b/commit-reach.c\nindex 2e33c599a82..338ca8084b2 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -8,6 +8,7 @@\n #include \"revision.h\"\n #include \"tag.h\"\n #include \"commit-reach.h\"\n+#include \"ewah/ewok.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define PARENT1\t\t(1u<<16)\n@@ -941,3 +942,98 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \n \treturn found_commits;\n }\n+\n+define_commit_slab(bit_arrays, struct bitmap *);\n+static struct bit_arrays bit_arrays;\n+\n+static void insert_no_dup(struct prio_queue *queue, struct commit *c)\n+{\n+\tif (c->object.flags & PARENT2)\n+\t\treturn;\n+\tprio_queue_put(queue, c);\n+\tc->object.flags |= PARENT2;\n+}\n+\n+static struct bitmap *init_bit_array(struct commit *c, int width)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\t*bitmap = bitmap_word_alloc(width);\n+\treturn *bitmap;\n+}\n+\n+static void free_bit_array(struct commit *c)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\treturn;\n+\tbitmap_free(*bitmap);\n+\t*bitmap = NULL;\n+}\n+\n+void ahead_behind(struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr)\n+{\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n+\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n+\tsize_t i;\n+\n+\tif (!commits_nr || !counts_nr)\n+\t\treturn;\n+\n+\tfor (i = 0; i < counts_nr; i++) {\n+\t\tcounts[i].ahead = 0;\n+\t\tcounts[i].behind = 0;\n+\t}\n+\n+\tensure_generations_valid(commits, commits_nr);\n+\n+\tinit_bit_arrays(&bit_arrays);\n+\n+\tfor (i = 0; i < commits_nr; i++) {\n+\t\tstruct commit *c = commits[i];\n+\t\tstruct bitmap *bitmap = init_bit_array(c, width);\n+\n+\t\tbitmap_set(bitmap, i);\n+\t\tinsert_no_dup(&queue, c);\n+\t}\n+\n+\twhile (queue_has_nonstale(&queue)) {\n+\t\tstruct commit *c = prio_queue_get(&queue);\n+\t\tstruct commit_list *p;\n+\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n+\n+\t\tfor (i = 0; i < counts_nr; i++) {\n+\t\t\tint reach_from_tip = !!bitmap_get(bitmap_c, counts[i].tip_index);\n+\t\t\tint reach_from_base = !!bitmap_get(bitmap_c, counts[i].base_index);\n+\n+\t\t\tif (reach_from_tip ^ reach_from_base) {\n+\t\t\t\tif (reach_from_base)\n+\t\t\t\t\tcounts[i].behind++;\n+\t\t\t\telse\n+\t\t\t\t\tcounts[i].ahead++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tstruct bitmap *bitmap_p;\n+\n+\t\t\tparse_commit(p->item);\n+\n+\t\t\tbitmap_p = init_bit_array(p->item, width);\n+\t\t\tbitmap_or(bitmap_p, bitmap_c);\n+\n+\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n+\t\t\t\tp->item->object.flags |= STALE;\n+\n+\t\t\tinsert_no_dup(&queue, p->item);\n+\t\t}\n+\n+\t\tfree_bit_array(c);\n+\t}\n+\n+\t/* STALE is used here, PARENT2 is used by insert_no_dup(). */\n+\trepo_clear_commit_marks(the_repository, PARENT2 | STALE);\n+\tclear_bit_arrays(&bit_arrays);\n+\tclear_prio_queue(&queue);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex 148b56fea50..f871b5dcce9 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -104,4 +104,34 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \t\t\t\t\t struct commit **to, int nr_to,\n \t\t\t\t\t unsigned int reachable_flag);\n \n+struct ahead_behind_count {\n+\t/**\n+\t * As input, the *_index members indicate which positions in\n+\t * the 'tips' array correspond to the tip and base of this\n+\t * comparison.\n+\t */\n+\tsize_t tip_index;\n+\tsize_t base_index;\n+\n+\t/**\n+\t * These values store the computed counts for each side of the\n+\t * symmetric difference:\n+\t *\n+\t * 'ahead' stores the number of commits reachable from the tip\n+\t * and not reachable from the base.\n+\t *\n+\t * 'behind' stores the number of commits reachable from the base\n+\t * and not reachable from the tip.\n+\t */\n+\tunsigned int ahead;\n+\tunsigned int behind;\n+};\n+\n+/*\n+ * Given an array of commits and an array of ahead_behind_count pairs,\n+ * compute the ahead/behind counts for each pair.\n+ */\n+void ahead_behind(struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"473367","messageId":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.git.1678111598.gitgitgadget@gmail.com","subject":"[PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:20:55Z","receivedAt":"2023-03-10T17:21:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"At $DAYJOB, we have used a custom 'ahead-behind' builtin in our fork of Git\nfor lots of reasons. The main goal of the builtin is to compare multiple\nreferences against a common base reference. The comparison is number of\ncommits that are in each side of the symmtric difference of their reachable\nsets. A commit C is \"ahead\" of a commit B by the number of commits in B..C\n(reachable from C but not reachable from B). Similarly, the commit C is\n\"behind\" the commit B by the number of commits in C..B (reachable from B but\nnot reachable from C).\n\nThese numbers can be computed by 'git rev-list --count B..C' and 'git\nrev-list --count C..B', but there are common needs that benefit from having\nthe checks being done in the same process:\n\n 1. Our \"branches\" page lists ahead/behind counts for each listed branch as\n compared to the repo's default branch. This can be done with a single\n 'git ahead-behind' process.\n 2. When a branch is updated, a background job checks if any pull requests\n that target that branch should be closed because their branches were\n merged implicitly by that update. These queries can e batched into 'git\n ahead-behind' calls.\n\nIn that second example, we don't need the full ahead/behind counts (although\nit is sufficient to look for branches that are \"zero commits ahead\", meaning\nthey are reachable from the base), and instead reachability is the critical\npiece.\n\nThis series contributes the custom algorithms we used for our 'git\nahead-behind' builtin, but as extensions to 'git for-each-ref':\n\n * Add a new \"%(ahead-behind:)\" format token to for-each-ref which allows\n outputting the ahead/behind values in the format string for a matching\n ref.\n * Add a new algorithm that speeds up the 'git for-each-ref --merged='\n option. This also applies to the 'git branch --merged=' option.\n\nThe idea to use 'git for-each-ref' instead of creating a new builtin is from\nJunio, and simplifies this series significantly compared to v1. I was\ninitially concerned about the overhead of 'git for-each-ref' and its\ngenerality and sorting, but I was not able to measure any important\ndifference between this implementation and our internal 'git ahead-behind'\nimplementation. In particular, when a pattern is given to 'git for-each-ref'\nthat looks like an exact ref, it navigates directly to the ref instead of\nscanning all references for matches.\n\nHowever, for our specific uses, we like to batch a list of exact references\nthat could be very long. We introduce a new --stdin option here.\n\nTo keep things close to the v1 outline, I replaced the existing patches with\nclosely-related ones, when possible.\n\nPatch 1 adds the --stdin option to 'git for-each-ref'. (This is similar to\nthe boilerplate patch from v1.)\n\nPatch 2 adds a test to explicitly check that 'git for-each-ref' will still\nsucceed when all input refs are missing. (This is similar to the\n--ignore-missing patch from v1.)\n\nPatches 3-5 introduce a new method: ensure_generations_valid(). Patch 3 does\nsome refactoring of the existing generation number computations to make it\nmore generic, and patch 4 updates the definition of\ncommit_graph_generation() slightly, making way for patch 5 to implement the\nmethod. With an existing commit-graph file, the commits that are not present\nin the file are considered as having generation number \"infinity\". This is\nuseful for most of our reachability queries to this point, since those\ncommits are \"above\" the ones tracked by the commit-graph. When these commits\nare low in number, then there is very little performance cost and zero\ncorrectness cost. (These patches match v1 exactly.)\n\nHowever, we will see that the ahead/behind computation requires accurate\ngeneration numbers to avoid overcounting. Thus, ensure_generations_valid()\nis a way to specify a list of commits that need generation numbers computed\nbefore continuing. It's a no-op if all of those commits are in the\ncommit-graph file. It's expensive if the commit-graph doesn't exist.\nHowever, '%(ahead-behind:)' computations are likely to be slow no matter\nwhat without a commit-graph, so assuming an existing commit-graph file is\nreasonable. If we find sufficient desire to have an implementation that does\nnot have this requirement, we could create a second implementation and\ntoggle to it when generation_numbers_enabled() returns false.\n\nPatch 6 implements the ahead-behind algorithm, but it is not connected to a\nbuiltin. It's a long commit message, so hopefully it explains the algorithm\nsufficiently. (The difference from v1 is that it no longer integrates with a\nbuiltin and there are no new tests. It also uses 'unsigned int' and is\ncorrectly co-authored by Taylor.)\n\nPatch 7 integrates the ahead-behind algorithm with the ref-filter code,\nincluding parsing the \"ahead-behind\" token. This finally adds tests that\ncheck both ahead_behind() and ensure_generations_valid() via\nt6600-test-reach.sh. (This patch is essentially completely new in v2.)\n\nPatch 8 implements the tips_reachable_from_base() method, and uses it within\nthe ref-filter code to speed up 'git for-each-ref --merged' and 'git branch\n--merged'. (The interface is slightly different than v1, due to the needs of\nthe new caller.)\n\nThanks, -Stolee\n\nDerrick Stolee (7):\n for-each-ref: add --stdin option\n for-each-ref: explicitly test no matches\n commit-graph: combine generation computations\n commit-graph: return generation from memory\n commit-reach: implement ahead_behind() logic\n for-each-ref: add ahead-behind format atom\n commit-reach: add tips_reachable_from_bases()\n\nTaylor Blau (1):\n commit-graph: introduce `ensure_generations_valid()`\n\n Documentation/git-for-each-ref.txt | 12 +-\n builtin/branch.c | 1 +\n builtin/for-each-ref.c | 32 ++++-\n builtin/tag.c | 1 +\n commit-graph.c | 208 ++++++++++++++++++----------\n commit-graph.h | 7 +\n commit-reach.c | 210 +++++++++++++++++++++++++++++\n commit-reach.h | 38 ++++++\n ref-filter.c | 89 +++++++++---\n ref-filter.h | 25 +++-\n t/perf/p1500-graph-walks.sh | 50 +++++++\n t/t3203-branch-output.sh | 14 ++\n t/t5318-commit-graph.sh | 2 +-\n t/t6300-for-each-ref.sh | 50 +++++++\n t/t6301-for-each-ref-errors.sh | 12 ++\n t/t6600-test-reach.sh | 169 +++++++++++++++++++++++\n t/t7004-tag.sh | 28 ++++\n 17 files changed, 859 insertions(+), 89 deletions(-)\n create mode 100755 t/perf/p1500-graph-walks.sh\n\n\nbase-commit: 725f57037d81e24eacfda6e59a19c60c0b4c8062\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1489%2Fderrickstolee%2Fstolee%2Fupstream-ahead-behind-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1489/derrickstolee/stolee/upstream-ahead-behind-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1489\n\nRange-diff vs v1:\n\n 1: 0fd18b6d740 < -: ----------- ahead-behind: create empty builtin\n 2: 08fc0710017 ! 1: a1d9e0f6ff6 ahead-behind: parse tip references\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - ahead-behind: parse tip references\n + for-each-ref: add --stdin option\n \n - Before implementing the logic to compute the ahead/behind counts, parse\n - the unknown options as commits and place them in a string_list.\n + When a user wishes to input a large list of patterns to 'git\n + for-each-ref' (likely a long list of exact refs) there are frequently\n + system limits on the number of command-line arguments.\n \n - Be sure to error out when the reference is not found.\n + Add a new --stdin option to instead read the patterns from standard\n + input. Add tests that check that any unrecognized arguments are\n + considered an error when --stdin is provided. Also, an empty pattern\n + list is interpreted as the complete ref set.\n +\n + When reading from stdin, we populate the filter.name_patterns array\n + dynamically as opposed to pointing to the 'argv' array directly. This\n + requires a careful cast while freeing the individual strings,\n + conditioned on the --stdin option.\n \n - Co-authored-by: Taylor Blau \n - Signed-off-by: Taylor Blau \n Signed-off-by: Derrick Stolee \n \n - ## builtin/ahead-behind.c ##\n -@@\n - #include \"builtin.h\"\n - #include \"parse-options.h\"\n - #include \"config.h\"\n -+#include \"commit.h\"\n - \n - static const char * const ahead_behind_usage[] = {\n - \tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n - \tNULL\n - };\n + ## Documentation/git-for-each-ref.txt ##\n +@@ Documentation/git-for-each-ref.txt: SYNOPSIS\n + --------\n + [verse]\n + 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n +-\t\t [(--sort=)...] [--format=] [...]\n ++\t\t [(--sort=)...] [--format=]\n ++\t\t [ --stdin | ... ]\n + \t\t [--points-at=]\n + \t\t [--merged[=]] [--no-merged[=]]\n + \t\t [--contains[=]] [--no-contains[=]]\n +@@ Documentation/git-for-each-ref.txt: OPTIONS\n + \tliterally, in the latter case matching completely or from the\n + \tbeginning up to a slash.\n \n -+static int handle_arg(struct string_list *tips, const char *arg)\n -+{\n -+\tstruct string_list_item *item;\n -+\tstruct commit *c = lookup_commit_reference_by_name(arg);\n -+\n -+\tif (!c)\n -+\t\treturn error(_(\"could not resolve '%s'\"), arg);\n -+\n -+\titem = string_list_append(tips, arg);\n -+\titem->util = c;\n -+\treturn 0;\n -+}\n ++--stdin::\n ++\tIf `--stdin` is supplied, then the list of patterns is read from\n ++\tstandard input instead of from the argument list.\n +\n - int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n - {\n - \tconst char *base_ref = NULL;\n - \tint from_stdin = 0;\n -+\tstruct string_list tips = STRING_LIST_INIT_DUP;\n + --count=::\n + \tBy default the command shows all refs that match\n + \t``. This option makes it stop after showing\n +\n + ## builtin/for-each-ref.c ##\n +@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n + \tstruct ref_format format = REF_FORMAT_INIT;\n + \tstruct strbuf output = STRBUF_INIT;\n + \tstruct strbuf err = STRBUF_INIT;\n ++\tint from_stdin = 0;\n \n - \tstruct option ahead_behind_opts[] = {\n - \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n -@@ builtin/ahead-behind.c: int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n + \tstruct option opts[] = {\n + \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n +@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n + \t\tOPT_CONTAINS(&filter.with_commit, N_(\"print only refs which contain the commit\")),\n + \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n + \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n ++\t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n + \t\tOPT_END(),\n + \t};\n \n - \tgit_config(git_default_config, NULL);\n +@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n + \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n + \tfilter.ignore_case = icase;\n \n +-\tfilter.name_patterns = argv;\n +\tif (from_stdin) {\n +\t\tstruct strbuf line = STRBUF_INIT;\n ++\t\tsize_t nr = 0, alloc = 16;\n +\n -+\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n -+\t\t\tif (!line.len)\n -+\t\t\t\tbreak;\n ++\t\tif (argv[0])\n ++\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n +\n -+\t\t\tif (handle_arg(&tips, line.buf))\n -+\t\t\t\treturn 1;\n ++\t\tCALLOC_ARRAY(filter.name_patterns, alloc);\n ++\n ++\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n ++\t\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n ++\t\t\tfilter.name_patterns[nr++] = strbuf_detach(&line, NULL);\n +\t\t}\n +\n -+\t\tstrbuf_release(&line);\n ++\t\t/* Add a terminating NULL string. */\n ++\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n ++\t\tfilter.name_patterns[nr + 1] = NULL;\n +\t} else {\n -+\t\tint i;\n -+\t\tfor (i = 0; i < argc; ++i) {\n -+\t\t\tif (handle_arg(&tips, argv[i]))\n -+\t\t\t\treturn 1;\n -+\t\t}\n ++\t\tfilter.name_patterns = argv;\n +\t}\n +\n -+\t/* Early return for no tips. */\n -+\tif (!tips.nr)\n -+\t\treturn 0;\n -+\n + \tfilter.match_as_path = 1;\n + \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n + \tref_array_sort(sorting, &array);\n +@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n + \tfree_commit_list(filter.with_commit);\n + \tfree_commit_list(filter.no_commit);\n + \tref_sorting_release(sorting);\n ++\tif (from_stdin) {\n ++\t\tfor (size_t i = 0; filter.name_patterns[i]; i++)\n ++\t\t\tfree((char *)filter.name_patterns[i]);\n ++\t\tfree(filter.name_patterns);\n ++\t}\n \treturn 0;\n }\n \n - ## t/t4218-ahead-behind.sh ##\n -@@ t/t4218-ahead-behind.sh: test_expect_success 'git ahead-behind without --base' '\n - \tgrep \"usage:\" err\n - '\n + ## t/t6300-for-each-ref.sh ##\n +@@ t/t6300-for-each-ref.sh: sig_crlf=\"$(printf \"%s\" \"$sig\" | append_cr; echo dummy)\"\n + sig_crlf=${sig_crlf%dummy}\n + test_atom refs/tags/fake-sig-crlf contents:signature \"$sig_crlf\"\n \n -+test_expect_success 'git ahead-behind with broken tip' '\n -+\ttest_must_fail git ahead-behind --base=HEAD bogus 2>err &&\n -+\tgrep \"could not resolve '\\''bogus'\\''\" err\n ++test_expect_success 'git for-each-ref --stdin: empty' '\n ++\t>in &&\n ++\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" >expect &&\n ++\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'git ahead-behind without tips' '\n -+\tgit ahead-behind --base=HEAD 2>err &&\n -+\ttest_must_be_empty err\n ++test_expect_success 'git for-each-ref --stdin: fails if extra args' '\n ++\t>in &&\n ++\ttest_must_fail git for-each-ref --format=\"%(refname)\" \\\n ++\t\t--stdin refs/heads/extra err &&\n ++\tgrep \"unknown arguments supplied with --stdin\" err\n ++'\n ++\n ++test_expect_success 'git for-each-ref --stdin: matches' '\n ++\tcat >in <<-EOF &&\n ++\trefs/tags/multi*\n ++\trefs/heads/amb*\n ++\tEOF\n ++\n ++\tcat >expect <<-EOF &&\n ++\trefs/heads/ambiguous\n ++\trefs/tags/multi-ref1-100000-user1\n ++\trefs/tags/multi-ref1-100000-user2\n ++\trefs/tags/multi-ref1-200000-user1\n ++\trefs/tags/multi-ref1-200000-user2\n ++\trefs/tags/multi-ref2-100000-user1\n ++\trefs/tags/multi-ref2-100000-user2\n ++\trefs/tags/multi-ref2-200000-user1\n ++\trefs/tags/multi-ref2-200000-user2\n ++\trefs/tags/multiline\n ++\tEOF\n ++\n ++\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n ++\ttest_cmp expect actual\n +'\n +\n test_done\n 3: b1d022c7cac < -: ----------- ahead-behind: implement --ignore-missing option\n -: ----------- > 2: 2f162a2f39f for-each-ref: explicitly test no matches\n 4: 853891c0b14 = 3: db28e82d2a6 commit-graph: combine generation computations\n 5: c6e6581e0ea ! 4: 3cf33801443 commit-graph: return generation from memory\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## commit-graph.c ##\n -@@ commit-graph.c: uint32_t commit_graph_position(const struct commit *c)\n - \treturn data ? data->graph_pos : COMMIT_NOT_FROM_GRAPH;\n - }\n - \n -+\n - timestamp_t commit_graph_generation(const struct commit *c)\n - {\n +@@ commit-graph.c: timestamp_t commit_graph_generation(const struct commit *c)\n \tstruct commit_graph_data *data =\n \t\tcommit_graph_data_slab_peek(&commit_graph_data_slab, c);\n \n 6: f31994ac78a = 5: 34dffd836b1 commit-graph: introduce `ensure_generations_valid()`\n 7: b8c55ecf88d ! 6: 9831c23eadb ahead-behind: implement ahead_behind() logic\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - ahead-behind: implement ahead_behind() logic\n + commit-reach: implement ahead_behind() logic\n \n - Fully implement the commit-counting logic behind the ahead-behind\n - builtin as a new ahead_behind() method in commit-reach.h. Add tests\n - for the functionality in both t4218-ahead-behind.sh and\n - t6600-test-reach.sh. The tests in t4218 are rather simple, but cover a\n - simple diamond commit history completely while the tests in t6600 make\n - use of the more complicated commit history and the test setup to check\n - three repository states: no commit-graph, a complete commit-graph, and a\n - half-filled commit-graph. These extra states are particularly helpful to\n - check due to the implementation of ahead_behind() and how it relies upon\n - ensure_generations_valid().\n + Fully implement the commit-counting logic required to determine\n + ahead/behind counts for a batch of commit pairs. This is a new library\n + method within commit-reach.h. This method will be linked to the\n + for-each-ref builtin in the next change.\n \n The interface for ahead_behind() uses two arrays. The first array of\n commits contains the list of all starting points for the walk. This\n @@ Commit message\n new ahead_behind_count struct, indicates which commits from that initial\n array form the base/tip pair for the ahead/behind count it will store.\n \n - While the ahead-behind builtin currently only supports one base, this\n - implementation of ahead_behind() allows multiple bases, if desired. Even\n - with multiple bases, there is only one commit walk used for counting the\n - ahead/behind values, saving time when the base/tip ranges overlap\n - significantly.\n + This implementation of ahead_behind() allows multiple bases, if desired.\n + Even with multiple bases, there is only one commit walk used for\n + counting the ahead/behind values, saving time when the base/tip ranges\n + overlap significantly.\n \n This interface for ahead_behind() also makes it very easy to call\n ensure_generations_valid() on the entire array of bases and tips. This\n @@ Commit message\n commits, in case ahead_behind() is called multiple times in the same\n process.\n \n - There is no previous implementation of ahead-behind to compare against.\n - A previous implementation in another fork of Git used a single process\n - to essentially do the same walk as 'git rev-list --count ..'\n - for every base/tip pair given as input. The single-walk implementation\n - in this change was a significant improvement over that implementation.\n - Another version from that fork used reachability bitmaps for the\n - comparison, but that implementation was slower than the current commit\n - walk implementation in almost all cases.\n -\n - To best present _some_ amount of evidence for this performance gain,\n - create a new performance test, p1500-graph-walks.sh. This script could\n - be used for other walks than just ahead-behind in the future, but let's\n - limit to ahead-behind now.\n -\n - To gain some amount of a baseline, create one test that checks 'git\n - ahead-behind' against up to 50 tips and another that uses 'git rev-list\n - --count' in a loop. Be sure to write a commit-graph before running the\n - performance tests.\n -\n - Using the Git source code as the repository, we see a pronounced\n - improvement:\n -\n - Test this tree\n - ---------------------------------------------------------------\n - 1500.2: ahead-behind counts: git ahead-behind 0.08(0.07+0.01)\n - 1500.3: ahead-behind counts: git rev-list 1.11(0.92+0.18)\n -\n - But the de-facto performance benchmark is the Linux kernel repository,\n - which presents these values for my copy:\n -\n - Test this tree\n - ---------------------------------------------------------------\n - 1500.2: ahead-behind counts: git ahead-behind 0.27(0.25+0.02)\n - 1500.3: ahead-behind counts: git rev-list 4.53(3.92+0.60)\n -\n + Co-authored-by: Taylor Blau \n + Signed-off-by: Taylor Blau \n Signed-off-by: Derrick Stolee \n \n - ## builtin/ahead-behind.c ##\n -@@\n - #include \"parse-options.h\"\n - #include \"config.h\"\n - #include \"commit.h\"\n -+#include \"commit-reach.h\"\n - \n - static const char * const ahead_behind_usage[] = {\n - \tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n -@@ builtin/ahead-behind.c: static int handle_arg(struct string_list *tips, const char *arg)\n - int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n - {\n - \tconst char *base_ref = NULL;\n -+\tstruct commit *base;\n - \tint from_stdin = 0;\n - \tstruct string_list tips = STRING_LIST_INIT_DUP;\n -+\tstruct commit **commits;\n -+\tstruct ahead_behind_count *counts;\n -+\tsize_t i;\n - \n - \tstruct option ahead_behind_opts[] = {\n - \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n -@@ builtin/ahead-behind.c: int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n - \tif (!tips.nr)\n - \t\treturn 0;\n - \n -+\tALLOC_ARRAY(commits, tips.nr + 1);\n -+\tALLOC_ARRAY(counts, tips.nr);\n -+\n -+\tfor (i = 0; i < tips.nr; i++) {\n -+\t\tcommits[i] = tips.items[i].util;\n -+\t\tcounts[i].tip_index = i;\n -+\t\tcounts[i].base_index = tips.nr;\n -+\t}\n -+\tcommits[tips.nr] = base;\n -+\n -+\tahead_behind(commits, tips.nr + 1, counts, tips.nr);\n -+\n -+\tfor (i = 0; i < tips.nr; i++)\n -+\t\tprintf(\"%s %d %d\\n\", tips.items[i].string,\n -+\t\t counts[i].ahead, counts[i].behind);\n -+\n -+\tfree(counts);\n -+\tfree(commits);\n - \treturn 0;\n - }\n -\n ## commit-reach.c ##\n @@\n #include \"revision.h\"\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n +\n +\t\tfor (i = 0; i < counts_nr; i++) {\n -+\t\t\tint reach_from_tip = bitmap_get(bitmap_c, counts[i].tip_index);\n -+\t\t\tint reach_from_base = bitmap_get(bitmap_c, counts[i].base_index);\n ++\t\t\tint reach_from_tip = !!bitmap_get(bitmap_c, counts[i].tip_index);\n ++\t\t\tint reach_from_base = !!bitmap_get(bitmap_c, counts[i].base_index);\n +\n +\t\t\tif (reach_from_tip ^ reach_from_base) {\n +\t\t\t\tif (reach_from_base)\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t\tfree_bit_array(c);\n +\t}\n +\n ++\t/* STALE is used here, PARENT2 is used by insert_no_dup(). */\n +\trepo_clear_commit_marks(the_repository, PARENT2 | STALE);\n +\tclear_bit_arrays(&bit_arrays);\n +\tclear_prio_queue(&queue);\n @@ commit-reach.h: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t * 'behind' stores the number of commits reachable from the base\n +\t * and not reachable from the tip.\n +\t */\n -+\tint ahead;\n -+\tint behind;\n ++\tunsigned int ahead;\n ++\tunsigned int behind;\n +};\n +\n -+/**\n ++/*\n + * Given an array of commits and an array of ahead_behind_count pairs,\n + * compute the ahead/behind counts for each pair.\n + */\n @@ commit-reach.h: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t\t struct ahead_behind_count *counts, size_t counts_nr);\n +\n #endif\n -\n - ## t/perf/p1500-graph-walks.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description='Commit walk performance tests'\n -+. ./perf-lib.sh\n -+\n -+test_perf_large_repo\n -+\n -+test_expect_success 'setup' '\n -+\tgit for-each-ref --format=\"%(refname)\" \"refs/heads/*\" \"refs/tags/*\" >allrefs &&\n -+\tsort -r allrefs | head -n 50 >refs &&\n -+\tgit commit-graph write --reachable\n -+'\n -+\n -+test_perf 'ahead-behind counts: git ahead-behind' '\n -+\tgit ahead-behind --base=HEAD --stdin out &&\n - \tgrep \"usage:\" out\n -@@ t/t4218-ahead-behind.sh: test_expect_success 'git ahead-behind without --base' '\n - \tgrep \"usage:\" err\n - '\n - \n -+test_expect_success 'git ahead-behind with broken --base' '\n -+\ttest_must_fail git ahead-behind --base=bogus HEAD 2>err &&\n -+\tgrep \"could not resolve '\\''bogus'\\''\" err\n -+'\n -+\n - test_expect_success 'git ahead-behind with broken tip' '\n - \ttest_must_fail git ahead-behind --base=HEAD bogus 2>err &&\n - \tgrep \"could not resolve '\\''bogus'\\''\" err\n -@@ t/t4218-ahead-behind.sh: test_expect_success 'git ahead-behind without tips' '\n - \ttest_must_be_empty err\n - '\n - \n -+test_expect_success 'git ahead-behind --base=base' '\n -+\tgit ahead-behind --base=base base left right merge >actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase 0 0\n -+\tleft 1 0\n -+\tright 1 0\n -+\tmerge 3 0\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n -+test_expect_success 'git ahead-behind --base=left' '\n -+\tgit ahead-behind --base=left base left right merge >actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase 0 1\n -+\tleft 0 0\n -+\tright 1 1\n -+\tmerge 2 0\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n -+test_expect_success 'git ahead-behind --base=right' '\n -+\tgit ahead-behind --base=right base left right merge >actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase 0 1\n -+\tleft 1 1\n -+\tright 0 0\n -+\tmerge 2 0\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n -+test_expect_success 'git ahead-behind --base=merge' '\n -+\tgit ahead-behind --base=merge base left right merge >actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase 0 3\n -+\tleft 0 2\n -+\tright 0 2\n -+\tmerge 0 0\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n - test_done\n -\n - ## t/t6600-test-reach.sh ##\n -@@ t/t6600-test-reach.sh: test_expect_success 'get_reachable_subset:none' '\n - \ttest_all_modes get_reachable_subset\n - '\n - \n -+test_expect_success 'ahead-behind:linear' '\n -+\tcat >input <<-\\EOF &&\n -+\tcommit-1-1\n -+\tcommit-1-3\n -+\tcommit-1-5\n -+\tcommit-1-8\n -+\tEOF\n -+\tcat >expect <<-\\EOF &&\n -+\tcommit-1-1 0 8\n -+\tcommit-1-3 0 6\n -+\tcommit-1-5 0 4\n -+\tcommit-1-8 0 1\n -+\tEOF\n -+\trun_all_modes git ahead-behind --base=commit-1-9 --stdin\n -+'\n -+\n -+test_expect_success 'ahead-behind:all' '\n -+\tcat >input <<-\\EOF &&\n -+\tcommit-1-1\n -+\tcommit-2-4\n -+\tcommit-4-2\n -+\tcommit-4-4\n -+\tEOF\n -+\tcat >expect <<-\\EOF &&\n -+\tcommit-1-1 0 24\n -+\tcommit-2-4 0 17\n -+\tcommit-4-2 0 17\n -+\tcommit-4-4 0 9\n -+\tEOF\n -+\trun_all_modes git ahead-behind --base=commit-5-5 --stdin\n -+'\n -+\n -+test_expect_success 'ahead-behind:some' '\n -+\tcat >input <<-\\EOF &&\n -+\tcommit-1-1\n -+\tcommit-5-3\n -+\tcommit-4-8\n -+\tcommit-9-9\n -+\tEOF\n -+\tcat >expect <<-\\EOF &&\n -+\tcommit-1-1 0 53\n -+\tcommit-5-3 0 39\n -+\tcommit-4-8 8 30\n -+\tcommit-9-9 27 0\n -+\tEOF\n -+\trun_all_modes git ahead-behind --base=commit-9-6 --stdin\n -+'\n -+\n -+test_expect_success 'ahead-behind:none' '\n -+\tcat >input <<-\\EOF &&\n -+\tcommit-7-5\n -+\tcommit-4-8\n -+\tcommit-9-9\n -+\tEOF\n -+\tcat >expect <<-\\EOF &&\n -+\tcommit-7-5 7 4\n -+\tcommit-4-8 16 16\n -+\tcommit-9-9 49 0\n -+\tEOF\n -+\trun_all_modes git ahead-behind --base=commit-8-4 --stdin\n -+'\n -+\n - test_done\n -: ----------- > 7: 82dd6f44a33 for-each-ref: add ahead-behind format atom\n 8: 07eb2cbb699 ! 8: f3fb6833bd7 ahead-behind: add --contains mode\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - ahead-behind: add --contains mode\n + commit-reach: add tips_reachable_from_bases()\n \n - The 'git ahead-behind' builtin can answer a list of questions that do\n - not require the full ahead/behind counts. For example, instead of using\n - 'git branch --contains=' to get the full list of branches that\n - contain the commit at , a list of tips could be passed to 'git\n - ahead-behind --base=' and the rows that report a \"behind\" value of\n - zero show which tips can reach .\n + Both 'git for-each-ref --merged=' and 'git branch --merged=' use\n + the ref-filter machinery to select references or branches (respectively)\n + that are reachable from a set of commits presented by one or more\n + --merged arguments. This happens within reach_filter(), which uses the\n + revision-walk machinery to walk history in a standard way.\n \n - By contract, the rows that report an \"ahead\" value of zero show which\n - tips are reachable from the given base. This type of query does not have\n - an existing equivalent for batching this request. While extracting the\n - information from 'git ahead-behind' is not terribly difficult, it does\n - more work than required to answer this query: it _counts_.\n + However, the commit-reach.c file is full of custom searches that are\n + more efficient, especially for reachability queries that can terminate\n + early when reachability is discovered. Add a new\n + tips_reachable_from_bases() method to commit-reach.c and call it from\n + within reach_filter() in ref-filter.c. This affects both 'git branch'\n + and 'git for-each-ref' as tested in p1500-graph-walks.sh.\n \n - Add a new '--contains' mode to 'git ahead-behind' that removes the\n - counting behavior and focuses instead on the reachability concern. The\n - output of the builtin changes in this mode: instead of reporting \"\n - \" for every input tip, it will instead report \"\"\n - for the input tips that are reachable from the specified base.\n + For the Linux kernel repository, we take an already-fast algorithm and\n + make it even faster:\n +\n + Test HEAD~1 HEAD\n + -------------------------------------------------------------------\n + 1500.5: contains: git for-each-ref --merged 0.13 0.02 -84.6%\n + 1500.6: contains: git branch --merged 0.14 0.02 -85.7%\n + 1500.7: contains: git tag --merged 0.15 0.03 -80.0%\n +\n + (Note that we remove the iterative 'git rev-list' test from p1500\n + because it no longer makes sense as a comparison to 'git for-each-ref'\n + and would just waste time running it for these comparisons.)\n \n The algorithm is implemented in commit-reach.c in the method\n tips_reachable_from_base(). This method takes a string_list of tips and\n @@ Commit message\n from the start, but we will walk only the necessary portion of the\n depth-first search for the reachable commits of lower generation.\n \n - We can test this completely in the simple repo example in t4218 and more\n - substantially in the larger repository example in t6600. We can also add\n - a performance test to demonstrate the speedup relative to the 'git\n - ahead-behind' builtin without the '--contains' option.\n -\n - For the Git source code repository, I was able to measure a speedup,\n - even though both are quite fast.\n -\n - Test this tree\n - --------------------------------------------------------------------------\n - 1500.2: ahead-behind counts: git ahead-behind 0.06(0.06+0.00)\n - 1500.3: ahead-behind counts: git rev-list 1.08(0.90+0.18)\n - 1500.4: ahead-behind counts: git ahead-behind --contains 0.02(0.02+0.00)\n -\n - In the Linux kernel repository, the impact is more pronounced:\n -\n - Test this tree\n - --------------------------------------------------------------------------\n - 1500.2: ahead-behind counts: git ahead-behind 0.26(0.25+0.01)\n - 1500.3: ahead-behind counts: git rev-list 4.58(3.92+0.66)\n - 1500.4: ahead-behind counts: git ahead-behind --contains 0.02(0.00+0.02)\n + Add extra tests for this behavior in t6600-test-reach.sh as the\n + interesting data shape of that repository can sometimes demonstrate\n + corner case bugs.\n \n Signed-off-by: Derrick Stolee \n \n - ## Documentation/git-ahead-behind.txt ##\n -@@ Documentation/git-ahead-behind.txt: git-ahead-behind - Count the commits on each side of a revision range\n - SYNOPSIS\n - --------\n - [verse]\n --'git ahead-behind' --base= [ --stdin | ]\n -+'git ahead-behind' --base= [ --contains ] [ --stdin | ]\n - \n - DESCRIPTION\n - -----------\n -@@ Documentation/git-ahead-behind.txt: reported to stdout one line at a time as follows:\n - There will be exactly one line per input revision, but the lines may be\n - in an arbitrary order.\n - \n -+If the `--contains` option is provided, then the output will list the\n -+`` refs are reachable from the provided ``, one per line.\n -+\n - \n - OPTIONS\n - -------\n -@@ Documentation/git-ahead-behind.txt: OPTIONS\n - \tRead revision tips and ranges from stdin instead of from the\n - \tcommand-line.\n - \n -+--contains::\n -+\tSpecify that instead of counting the ahead/behind values, only\n -+\tindicate whether each tip reference is reachable from the base. In\n -+\tthis mode, the output format changes to include only the name of\n -+\teach tip by name, one per line, and only the tips reachable from\n -+\tthe base are included in the output.\n -+\n - --ignore-missing::\n - \tWhen parsing tip references, ignore any references that are not\n - \tfound. This is useful when operating in an environment where a\n -\n - ## builtin/ahead-behind.c ##\n -@@\n - #include \"commit-reach.h\"\n - \n - static const char * const ahead_behind_usage[] = {\n --\tN_(\"git ahead-behind --base= [ --stdin | ]\"),\n -+\tN_(\"git ahead-behind --base= [ --contains ] [ --stdin | ]\"),\n - \tNULL\n - };\n - \n -@@ builtin/ahead-behind.c: int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n - {\n - \tconst char *base_ref = NULL;\n - \tstruct commit *base;\n --\tint from_stdin = 0;\n -+\tint from_stdin = 0, contains = 0;\n - \tstruct string_list tips = STRING_LIST_INIT_DUP;\n - \tstruct commit **commits;\n - \tstruct ahead_behind_count *counts;\n -@@ builtin/ahead-behind.c: int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n - \t\tOPT_STRING('b', \"base\", &base_ref, N_(\"base\"), N_(\"base reference to process\")),\n - \t\tOPT_BOOL(0 , \"stdin\", &from_stdin, N_(\"read rev names from stdin\")),\n - \t\tOPT_BOOL(0 , \"ignore-missing\", &ignore_missing, N_(\"ignore missing tip references\")),\n -+\t\tOPT_BOOL(0 , \"contains\", &contains, N_(\"only check that tips are reachable from the base\")),\n - \t\tOPT_END()\n - \t};\n - \n -@@ builtin/ahead-behind.c: int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n - \n - \tgit_config(git_default_config, NULL);\n - \n -+\tbase = lookup_commit_reference_by_name(base_ref);\n -+\tif (!base)\n -+\t\tdie(_(\"could not resolve '%s'\"), base_ref);\n -+\n - \tif (from_stdin) {\n - \t\tstruct strbuf line = STRBUF_INIT;\n - \n -@@ builtin/ahead-behind.c: int cmd_ahead_behind(int argc, const char **argv, const char *prefix)\n - \tif (!tips.nr)\n - \t\treturn 0;\n - \n -+\tif (contains) {\n -+\t\tstruct string_list_item *item;\n -+\n -+\t\t/* clear out */\n -+\t\tfor_each_string_list_item(item, &tips)\n -+\t\t\titem->util = NULL;\n -+\n -+\t\ttips_reachable_from_base(base, &tips);\n -+\n -+\t\tfor_each_string_list_item(item, &tips) {\n -+\t\t\tif (item->util)\n -+\t\t\t\tprintf(\"%s\\n\", item->string);\n -+\t\t}\n -+\n -+\t\treturn 0;\n -+\t}\n -+\t/* else: not --contains, but normal ahead-behind counting. */\n -+\n - \tALLOC_ARRAY(commits, tips.nr + 1);\n - \tALLOC_ARRAY(counts, tips.nr);\n - \n -\n ## commit-reach.c ##\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n \tclear_bit_arrays(&bit_arrays);\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +\treturn 0;\n +}\n +\n -+void tips_reachable_from_base(struct commit *base,\n -+\t\t\t struct string_list *tips)\n ++void tips_reachable_from_bases(struct commit_list *bases,\n ++\t\t\t struct commit **tips, size_t tips_nr,\n ++\t\t\t int mark)\n +{\n -+\tunsigned int i;\n ++\tsize_t i;\n +\tstruct commit_and_index *commits;\n +\tunsigned int min_generation_index = 0;\n +\ttimestamp_t min_generation;\n +\tstruct commit_list *stack = NULL;\n +\n -+\tif (!base || !tips || !tips->nr)\n ++\tif (!bases || !tips || !tips_nr)\n +\t\treturn;\n +\n +\t/*\n -+\t * Do a depth-first search starting at 'base' to search for the\n ++\t * Do a depth-first search starting at 'bases' to search for the\n +\t * tips. Stop at the lowest (un-found) generation number. When\n +\t * finding the lowest commit, increase the minimum generation\n +\t * number to the next lowest (un-found) generation number.\n +\t */\n +\n -+\tCALLOC_ARRAY(commits, tips->nr);\n ++\tCALLOC_ARRAY(commits, tips_nr);\n +\n -+\tfor (i = 0; i < tips->nr; i++) {\n -+\t\tcommits[i].commit = lookup_commit_reference_by_name(tips->items[i].string);\n ++\tfor (i = 0; i < tips_nr; i++) {\n ++\t\tcommits[i].commit = tips[i];\n +\t\tcommits[i].index = i;\n -+\t\tcommits[i].generation = commit_graph_generation(commits[i].commit);\n ++\t\tcommits[i].generation = commit_graph_generation(tips[i]);\n +\t}\n +\n +\t/* Sort with generation number ascending. */\n -+\tQSORT(commits, tips->nr, compare_commit_and_index_by_generation);\n ++\tQSORT(commits, tips_nr, compare_commit_and_index_by_generation);\n +\tmin_generation = commits[0].generation;\n +\n -+\tparse_commit(base);\n -+\tcommit_list_insert(base, &stack);\n ++\twhile (bases) {\n ++\t\tparse_commit(bases->item);\n ++\t\tcommit_list_insert(bases->item, &stack);\n ++\t\tbases = bases->next;\n ++\t}\n +\n +\twhile (stack) {\n +\t\tunsigned int j;\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +\t\tstruct commit *c = stack->item;\n +\t\ttimestamp_t c_gen = commit_graph_generation(c);\n +\n -+\t\t/* Does it match any of our bases? */\n -+\t\tfor (j = min_generation_index; j < tips->nr; j++) {\n ++\t\t/* Does it match any of our tips? */\n ++\t\tfor (j = min_generation_index; j < tips_nr; j++) {\n +\t\t\tif (c_gen < commits[j].generation)\n +\t\t\t\tbreak;\n +\n +\t\t\tif (commits[j].commit == c) {\n -+\t\t\t\ttips->items[commits[j].index].util = (void *)(uintptr_t)1;\n ++\t\t\t\ttips[commits[j].index]->object.flags |= mark;\n +\n +\t\t\t\tif (j == min_generation_index) {\n +\t\t\t\t\tunsigned int k = j + 1;\n -+\t\t\t\t\twhile (k < tips->nr &&\n -+\t\t\t\t\t tips->items[commits[k].index].util)\n ++\t\t\t\t\twhile (k < tips_nr &&\n ++\t\t\t\t\t (tips[commits[k].index]->object.flags & mark))\n +\t\t\t\t\t\tk++;\n +\n +\t\t\t\t\t/* Terminate early if all found. */\n -+\t\t\t\t\tif (k >= tips->nr)\n ++\t\t\t\t\tif (k >= tips_nr)\n +\t\t\t\t\t\tgoto done;\n +\n +\t\t\t\t\tmin_generation_index = k;\n @@ commit-reach.h: struct ahead_behind_count {\n \t\t struct ahead_behind_count *counts, size_t counts_nr);\n \n +/*\n -+ * Populate the \"util\" of each string_list item with the boolean value\n -+ * corresponding to \"can 'base' reach this tip?\"\n ++ * For all tip commits, add 'mark' to their flags if and only if they\n ++ * are reachable from one of the commits in 'bases'.\n + */\n -+void tips_reachable_from_base(struct commit *base,\n -+\t\t\t struct string_list *tips);\n ++void tips_reachable_from_bases(struct commit_list *bases,\n ++\t\t\t struct commit **tips, size_t tips_nr,\n ++\t\t\t int mark);\n +\n #endif\n \n - ## t/perf/p1500-graph-walks.sh ##\n -@@ t/perf/p1500-graph-walks.sh: test_perf 'ahead-behind counts: git rev-list' '\n - \tdone\n - '\n + ## ref-filter.c ##\n +@@ ref-filter.c: static void reach_filter(struct ref_array *array,\n + \t\t\t struct commit_list *check_reachable,\n + \t\t\t int include_reached)\n + {\n +-\tstruct rev_info revs;\n + \tint i, old_nr;\n + \tstruct commit **to_clear;\n +-\tstruct commit_list *cr;\n + \n + \tif (!check_reachable)\n + \t\treturn;\n + \n + \tCALLOC_ARRAY(to_clear, array->nr);\n +-\n +-\trepo_init_revisions(the_repository, &revs, NULL);\n +-\n + \tfor (i = 0; i < array->nr; i++) {\n + \t\tstruct ref_array_item *item = array->items[i];\n +-\t\tadd_pending_object(&revs, &item->commit->object, item->refname);\n + \t\tto_clear[i] = item->commit;\n + \t}\n + \n +-\tfor (cr = check_reachable; cr; cr = cr->next) {\n +-\t\tstruct commit *merge_commit = cr->item;\n +-\t\tmerge_commit->object.flags |= UNINTERESTING;\n +-\t\tadd_pending_object(&revs, &merge_commit->object, \"\");\n +-\t}\n +-\n +-\trevs.limited = 1;\n +-\tif (prepare_revision_walk(&revs))\n +-\t\tdie(_(\"revision walk setup failed\"));\n ++\ttips_reachable_from_bases(check_reachable,\n ++\t\t\t\t to_clear, array->nr,\n ++\t\t\t\t UNINTERESTING);\n + \n + \told_nr = array->nr;\n + \tarray->nr = 0;\n +@@ ref-filter.c: static void reach_filter(struct ref_array *array,\n + \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n + \t}\n + \n +-\trelease_revisions(&revs);\n + \tfree(to_clear);\n + }\n \n -+test_perf 'ahead-behind counts: git ahead-behind --contains' '\n -+\tgit ahead-behind --contains --base=HEAD --stdin err &&\n -+\tgrep \"could not resolve '\\''bogus'\\''\" err\n +-test_perf 'ahead-behind counts: git rev-list' '\n +-\tfor r in $(cat refs)\n +-\tdo\n +-\t\tgit rev-list --count \"HEAD..$r\" || return 1\n +-\tdone\n ++test_perf 'contains: git for-each-ref --merged' '\n ++\tgit for-each-ref --merged=HEAD --stdin err >out &&\n -+\ttest_must_be_empty err &&\n -+\ttest_must_be_empty out\n ++test_perf 'contains: git branch --merged' '\n ++\txargs git branch --merged=HEAD err &&\n - \ttest_must_be_empty err\n -@@ t/t4218-ahead-behind.sh: test_expect_success 'git ahead-behind --base=merge' '\n - \ttest_cmp expect actual\n ++test_perf 'contains: git tag --merged' '\n ++\txargs git tag --merged=HEAD actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n -+test_expect_success 'git ahead-behind --contains --base=left' '\n -+\tgit ahead-behind --contains --base=left \\\n -+\t\tbase left right merge >actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase\n -+\tleft\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n -+test_expect_success 'git ahead-behind --contains --base=right' '\n -+\tgit ahead-behind --contains --base=right \\\n -+\t\tbase left right merge >actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase\n -+\tright\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n -+test_expect_success 'git ahead-behind --contains --base=merge' '\n -+\tgit ahead-behind --contains --base=merge \\\n -+\t\tbase left right merge >actual &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\tbase\n -+\tleft\n -+\tright\n -+\tmerge\n -+\tEOF\n -+\n -+\ttest_cmp expect actual\n -+'\n -+\n test_done\n \n ## t/t6600-test-reach.sh ##\n -@@ t/t6600-test-reach.sh: test_expect_success 'ahead-behind:none' '\n - \trun_all_modes git ahead-behind --base=commit-8-4 --stdin\n +@@ t/t6600-test-reach.sh: test_expect_success 'for-each-ref ahead-behind:none' '\n + \t\t--format=\"%(refname) %(ahead-behind:commit-8-4)\" --stdin\n '\n \n -+test_expect_success 'ahead-behind--contains:all' '\n ++test_expect_success 'for-each-ref merged:linear' '\n ++\tcat >input <<-\\EOF &&\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-1-3\n ++\trefs/heads/commit-1-5\n ++\trefs/heads/commit-1-8\n ++\trefs/heads/commit-2-1\n ++\trefs/heads/commit-5-1\n ++\trefs/heads/commit-9-1\n ++\tEOF\n ++\tcat >expect <<-\\EOF &&\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-1-3\n ++\trefs/heads/commit-1-5\n ++\trefs/heads/commit-1-8\n ++\tEOF\n ++\trun_all_modes git for-each-ref --merged=commit-1-9 \\\n ++\t\t--format=\"%(refname)\" --stdin\n ++'\n ++\n ++test_expect_success 'for-each-ref merged:all' '\n +\tcat >input <<-\\EOF &&\n -+\tcommit-1-1\n -+\tcommit-2-4\n -+\tcommit-4-2\n -+\tcommit-4-4\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-2-4\n ++\trefs/heads/commit-4-2\n ++\trefs/heads/commit-4-4\n +\tEOF\n +\tcat >expect <<-\\EOF &&\n -+\tcommit-1-1\n -+\tcommit-2-4\n -+\tcommit-4-2\n -+\tcommit-4-4\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-2-4\n ++\trefs/heads/commit-4-2\n ++\trefs/heads/commit-4-4\n +\tEOF\n -+\trun_all_modes git ahead-behind --contains --base=commit-5-5 \\\n -+\t\t--stdin --use-bitmap-index\n ++\trun_all_modes git for-each-ref --merged=commit-5-5 \\\n ++\t\t--format=\"%(refname)\" --stdin\n +'\n +\n -+test_expect_success 'ahead-behind--contains:some' '\n ++test_expect_success 'for-each-ref ahead-behind:some' '\n +\tcat >input <<-\\EOF &&\n -+\tcommit-1-1\n -+\tcommit-5-3\n -+\tcommit-4-8\n -+\tcommit-9-9\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-5-3\n ++\trefs/heads/commit-4-8\n ++\trefs/heads/commit-9-9\n +\tEOF\n +\tcat >expect <<-\\EOF &&\n -+\tcommit-1-1\n -+\tcommit-5-3\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-5-3\n +\tEOF\n -+\trun_all_modes git ahead-behind --contains --base=commit-9-6 \\\n -+\t\t--stdin --use-bitmap-index\n ++\trun_all_modes git for-each-ref --merged=commit-9-6 \\\n ++\t\t--format=\"%(refname)\" --stdin\n +'\n +\n -+test_expect_success 'ahead-behind--contains:some, reordered' '\n ++test_expect_success 'for-each-ref merged:some, multibase' '\n +\tcat >input <<-\\EOF &&\n -+\tcommit-4-8\n -+\tcommit-5-3\n -+\tcommit-9-9\n -+\tcommit-1-1\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-5-3\n ++\trefs/heads/commit-7-8\n ++\trefs/heads/commit-4-8\n ++\trefs/heads/commit-9-9\n +\tEOF\n +\tcat >expect <<-\\EOF &&\n -+\tcommit-5-3\n -+\tcommit-1-1\n ++\trefs/heads/commit-1-1\n ++\trefs/heads/commit-4-8\n ++\trefs/heads/commit-5-3\n +\tEOF\n -+\trun_all_modes git ahead-behind --contains --base=commit-9-6 \\\n -+\t\t--stdin --use-bitmap-index\n ++\trun_all_modes git for-each-ref \\\n ++\t\t--merged=commit-5-8 \\\n ++\t\t--merged=commit-8-5 \\\n ++\t\t--format=\"%(refname)\" \\\n ++\t\t--stdin\n +'\n +\n -+test_expect_success 'ahead-behind--contains:none' '\n ++test_expect_success 'for-each-ref merged:none' '\n +\tcat >input <<-\\EOF &&\n -+\tcommit-7-5\n -+\tcommit-4-8\n -+\tcommit-9-9\n ++\trefs/heads/commit-7-5\n ++\trefs/heads/commit-4-8\n ++\trefs/heads/commit-9-9\n +\tEOF\n +\t>expect &&\n -+\trun_all_modes git ahead-behind --contains --base=commit-8-4 \\\n -+\t\t--stdin --use-bitmap-index\n ++\trun_all_modes git for-each-ref --merged=commit-8-4 \\\n ++\t\t--format=\"%(refname)\" --stdin\n +'\n +\n test_done\n\n-- \ngitgitgadget\n"},{"id":"473368","messageId":"f3fb6833bd71d559a3076d9617a235614ad9a5f8.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 8/8] commit-reach: add tips_reachable_from_bases()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:21:03Z","receivedAt":"2023-03-10T17:21:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBoth 'git for-each-ref --merged=' and 'git branch --merged=' use\nthe ref-filter machinery to select references or branches (respectively)\nthat are reachable from a set of commits presented by one or more\n--merged arguments. This happens within reach_filter(), which uses the\nrevision-walk machinery to walk history in a standard way.\n\nHowever, the commit-reach.c file is full of custom searches that are\nmore efficient, especially for reachability queries that can terminate\nearly when reachability is discovered. Add a new\ntips_reachable_from_bases() method to commit-reach.c and call it from\nwithin reach_filter() in ref-filter.c. This affects both 'git branch'\nand 'git for-each-ref' as tested in p1500-graph-walks.sh.\n\nFor the Linux kernel repository, we take an already-fast algorithm and\nmake it even faster:\n\nTest HEAD~1 HEAD\n-------------------------------------------------------------------\n1500.5: contains: git for-each-ref --merged 0.13 0.02 -84.6%\n1500.6: contains: git branch --merged 0.14 0.02 -85.7%\n1500.7: contains: git tag --merged 0.15 0.03 -80.0%\n\n(Note that we remove the iterative 'git rev-list' test from p1500\nbecause it no longer makes sense as a comparison to 'git for-each-ref'\nand would just waste time running it for these comparisons.)\n\nThe algorithm is implemented in commit-reach.c in the method\ntips_reachable_from_base(). This method takes a string_list of tips and\nassigns the 'util' for each item with the value 1 if the base commit can\nreach those tips.\n\nLike other reachability queries in commit-reach.c, the fastest way to\nsearch for \"can A reach B?\" is to do a depth-first search up to the\ngeneration number of B, preferring to explore first parents before later\nparents. While we must walk all reachable commits up to that generation\nnumber when the answer is \"no\", the depth-first search can answer \"yes\"\nmuch faster than other approaches in most cases.\n\nThis search becomes trickier when there are multiple targets for the\ndepth-first search. The commits with lower generation number are more\nlikely to be within the history of the start commit, but we don't want\nto waste time searching commits of low generation number if the commit\ntarget with lowest generation number has already been found.\n\nThe trick here is to take the input commits and sort them by generation\nnumber in ascending order. Track the index within this order as\nmin_generation_index. When we find a commit, if its index in the list is\nequal to min_generation_index, then we can increase the generation\nnumber boundary of our search to the next-lowest value in the list.\n\nWith this mechanism, the number of commits to search is minimized with\nrespect to the depth-first search heuristic. We will walk all commits up\nto the minimum generation number of a commit that is _not_ reachable\nfrom the start, but we will walk only the necessary portion of the\ndepth-first search for the reachable commits of lower generation.\n\nAdd extra tests for this behavior in t6600-test-reach.sh as the\ninteresting data shape of that repository can sometimes demonstrate\ncorner case bugs.\n\nSigned-off-by: Derrick Stolee \n---\n commit-reach.c | 114 ++++++++++++++++++++++++++++++++++++\n commit-reach.h | 8 +++\n ref-filter.c | 19 +-----\n t/perf/p1500-graph-walks.sh | 15 +++--\n t/t6600-test-reach.sh | 83 ++++++++++++++++++++++++++\n 5 files changed, 218 insertions(+), 21 deletions(-)\n\ndiff --git a/commit-reach.c b/commit-reach.c\nindex 338ca8084b2..f6c4a3c93c7 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -1037,3 +1037,117 @@ void ahead_behind(struct commit **commits, size_t commits_nr,\n \tclear_bit_arrays(&bit_arrays);\n \tclear_prio_queue(&queue);\n }\n+\n+struct commit_and_index {\n+\tstruct commit *commit;\n+\tunsigned int index;\n+\ttimestamp_t generation;\n+};\n+\n+static int compare_commit_and_index_by_generation(const void *va, const void *vb)\n+{\n+\tconst struct commit_and_index *a = (const struct commit_and_index *)va;\n+\tconst struct commit_and_index *b = (const struct commit_and_index *)vb;\n+\n+\tif (a->generation > b->generation)\n+\t\treturn 1;\n+\tif (a->generation < b->generation)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n+void tips_reachable_from_bases(struct commit_list *bases,\n+\t\t\t struct commit **tips, size_t tips_nr,\n+\t\t\t int mark)\n+{\n+\tsize_t i;\n+\tstruct commit_and_index *commits;\n+\tunsigned int min_generation_index = 0;\n+\ttimestamp_t min_generation;\n+\tstruct commit_list *stack = NULL;\n+\n+\tif (!bases || !tips || !tips_nr)\n+\t\treturn;\n+\n+\t/*\n+\t * Do a depth-first search starting at 'bases' to search for the\n+\t * tips. Stop at the lowest (un-found) generation number. When\n+\t * finding the lowest commit, increase the minimum generation\n+\t * number to the next lowest (un-found) generation number.\n+\t */\n+\n+\tCALLOC_ARRAY(commits, tips_nr);\n+\n+\tfor (i = 0; i < tips_nr; i++) {\n+\t\tcommits[i].commit = tips[i];\n+\t\tcommits[i].index = i;\n+\t\tcommits[i].generation = commit_graph_generation(tips[i]);\n+\t}\n+\n+\t/* Sort with generation number ascending. */\n+\tQSORT(commits, tips_nr, compare_commit_and_index_by_generation);\n+\tmin_generation = commits[0].generation;\n+\n+\twhile (bases) {\n+\t\tparse_commit(bases->item);\n+\t\tcommit_list_insert(bases->item, &stack);\n+\t\tbases = bases->next;\n+\t}\n+\n+\twhile (stack) {\n+\t\tunsigned int j;\n+\t\tint explored_all_parents = 1;\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = stack->item;\n+\t\ttimestamp_t c_gen = commit_graph_generation(c);\n+\n+\t\t/* Does it match any of our tips? */\n+\t\tfor (j = min_generation_index; j < tips_nr; j++) {\n+\t\t\tif (c_gen < commits[j].generation)\n+\t\t\t\tbreak;\n+\n+\t\t\tif (commits[j].commit == c) {\n+\t\t\t\ttips[commits[j].index]->object.flags |= mark;\n+\n+\t\t\t\tif (j == min_generation_index) {\n+\t\t\t\t\tunsigned int k = j + 1;\n+\t\t\t\t\twhile (k < tips_nr &&\n+\t\t\t\t\t (tips[commits[k].index]->object.flags & mark))\n+\t\t\t\t\t\tk++;\n+\n+\t\t\t\t\t/* Terminate early if all found. */\n+\t\t\t\t\tif (k >= tips_nr)\n+\t\t\t\t\t\tgoto done;\n+\n+\t\t\t\t\tmin_generation_index = k;\n+\t\t\t\t\tmin_generation = commits[k].generation;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tparse_commit(p->item);\n+\n+\t\t\t/* Have we already explored this parent? */\n+\t\t\tif (p->item->object.flags & SEEN)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Is it below the current minimum generation? */\n+\t\t\tif (commit_graph_generation(p->item) < min_generation)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ok, we will explore from here on. */\n+\t\t\tp->item->object.flags |= SEEN;\n+\t\t\texplored_all_parents = 0;\n+\t\t\tcommit_list_insert(p->item, &stack);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (explored_all_parents)\n+\t\t\tpop_commit(&stack);\n+\t}\n+\n+done:\n+\tfree(commits);\n+\trepo_clear_commit_marks(the_repository, SEEN);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex f871b5dcce9..14043ed8562 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -134,4 +134,12 @@ struct ahead_behind_count {\n void ahead_behind(struct commit **commits, size_t commits_nr,\n \t\t struct ahead_behind_count *counts, size_t counts_nr);\n \n+/*\n+ * For all tip commits, add 'mark' to their flags if and only if they\n+ * are reachable from one of the commits in 'bases'.\n+ */\n+void tips_reachable_from_bases(struct commit_list *bases,\n+\t\t\t struct commit **tips, size_t tips_nr,\n+\t\t\t int mark);\n+\n #endif\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 896bf703f59..ece77d7e8ba 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2390,33 +2390,21 @@ static void reach_filter(struct ref_array *array,\n \t\t\t struct commit_list *check_reachable,\n \t\t\t int include_reached)\n {\n-\tstruct rev_info revs;\n \tint i, old_nr;\n \tstruct commit **to_clear;\n-\tstruct commit_list *cr;\n \n \tif (!check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\n \tfor (i = 0; i < array->nr; i++) {\n \t\tstruct ref_array_item *item = array->items[i];\n-\t\tadd_pending_object(&revs, &item->commit->object, item->refname);\n \t\tto_clear[i] = item->commit;\n \t}\n \n-\tfor (cr = check_reachable; cr; cr = cr->next) {\n-\t\tstruct commit *merge_commit = cr->item;\n-\t\tmerge_commit->object.flags |= UNINTERESTING;\n-\t\tadd_pending_object(&revs, &merge_commit->object, \"\");\n-\t}\n-\n-\trevs.limited = 1;\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n+\ttips_reachable_from_bases(check_reachable,\n+\t\t\t\t to_clear, array->nr,\n+\t\t\t\t UNINTERESTING);\n \n \told_nr = array->nr;\n \tarray->nr = 0;\n@@ -2440,7 +2428,6 @@ static void reach_filter(struct ref_array *array,\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n-\trelease_revisions(&revs);\n \tfree(to_clear);\n }\n \ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nindex 439a448c2e6..e14e7620cce 100755\n--- a/t/perf/p1500-graph-walks.sh\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -35,11 +35,16 @@ test_perf 'ahead-behind counts: git tag' '\n \txargs git tag -l --format=\"%(ahead-behind:HEAD)\" input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\trefs/heads/commit-2-1\n+\trefs/heads/commit-5-1\n+\trefs/heads/commit-9-1\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-1-9 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:all' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-5-5 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-9-6 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:some, multibase' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-7-8\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-5-3\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--merged=commit-5-8 \\\n+\t\t--merged=commit-8-5 \\\n+\t\t--format=\"%(refname)\" \\\n+\t\t--stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:none' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-7-5\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\t>expect &&\n+\trun_all_modes git for-each-ref --merged=commit-8-4 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"473369","messageId":"82dd6f44a33279551bb638357df4bc82253283e5.1678468864.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v2 7/8] for-each-ref: add ahead-behind format atom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-10T17:21:02Z","receivedAt":"2023-03-10T17:21:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change implemented the ahead_behind() method, including an\nalgorithm to compute the ahead/behind values for a number of commit tips\nrelative to a number of commit bases. Now, integrate that algorithm as\npart of 'git for-each-ref' hidden behind a new format atom,\nahead-behind. This naturally extends to 'git branch' and 'git tag'\nbuiltins, as well.\n\nThis format allows specifying multiple bases, if so desired, and all\nmatching references are compared against all of those bases. For this\nreason, failing to read a reference provided from these atoms results in\nan error.\n\nIn order to translate the ahead_behind() method information to the\nformat output code in ref-filter.c, we must populate arrays of\nahead_behind_count structs. In struct ref_array, we store the full array\nthat will be passed to ahead_behind(). In struct ref_array_item, we\nstore an array of pointers that point to the relvant items within the\nfull array. In this way, we can pull all relevant ahead/behind values\ndirectly when formatting output for a specific item. It also ensures the\nlifetime of the ahead_behind_count structs matches the time that the\narray is being used.\n\nAdd specific tests of the ahead/behind counts in t6600-test-reach.sh, as\nit has an interesting repository shape. In particular, its merging\nstrategy and its use of different commit-graphs would demonstrate over-\ncounting if the ahead_behind() method did not already account for that\npossibility.\n\nAlso add tests for the specific for-each-ref, branch, and tag builtins.\nIn the case of 'git tag', there are intersting cases that happen when\nsome of the selected tips are not commits. This requires careful logic\naround commits_nr in the second loop of filter_ahead_behind(). Also, the\ntest in t7004 is carefully located to avoid being dependent on the GPG\nprereq. It also avoids using the test_commit helper, as that will add\nticks to the time and disrupt the expected timestampes in later tag\ntests.\n\nAlso add performance tests in a new p1300-graph-walks.sh script. This\nwill be useful for more uses in the future, but for now compare the\nahead-behind counting algorithm in 'git for-each-ref' to the naive\nimplementation by running 'git rev-list --count' processes for each\ninput.\n\nFor the Git source code repository, the improvement is already obvious:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git for-each-ref 0.07(0.07+0.00)\n1500.3: ahead-behind counts: git branch 0.07(0.06+0.00)\n1500.4: ahead-behind counts: git tag 0.07(0.06+0.00)\n1500.5: ahead-behind counts: git rev-list 1.32(1.04+0.27)\n\nBut the standard performance benchmark is the Linux kernel repository,\nwhich demosntrates a significant improvement:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git for-each-ref 0.27(0.24+0.02)\n1500.3: ahead-behind counts: git branch 0.27(0.24+0.03)\n1500.4: ahead-behind counts: git tag 0.28(0.27+0.01)\n1500.5: ahead-behind counts: git rev-list 4.57(4.03+0.54)\n\nThe 'git rev-list' test exists in this change as a demonstration, but it\nwill be removed in the next change to avoid wasting time on this\ncomparison.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-for-each-ref.txt | 5 ++\n builtin/branch.c | 1 +\n builtin/for-each-ref.c | 3 ++\n builtin/tag.c | 1 +\n ref-filter.c | 70 ++++++++++++++++++++++++\n ref-filter.h | 25 ++++++++-\n t/perf/p1500-graph-walks.sh | 45 ++++++++++++++++\n t/t3203-branch-output.sh | 14 +++++\n t/t6301-for-each-ref-errors.sh | 12 +++++\n t/t6600-test-reach.sh | 86 ++++++++++++++++++++++++++++++\n t/t7004-tag.sh | 28 ++++++++++\n 11 files changed, 289 insertions(+), 1 deletion(-)\n create mode 100755 t/perf/p1500-graph-walks.sh\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex ccdc2911bb9..d5c3cda4228 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -222,6 +222,11 @@ worktreepath::\n \tout, if it is checked out in any linked worktree. Empty string\n \totherwise.\n \n+ahead-behind:::\n+\tTwo integers, separated by a space, demonstrating the number of\n+\tcommits ahead and behind, respectively, when comparing the output\n+\tref to the `` specified in the format.\n+\n In addition to the above, for commit and tag objects, the header\n field names (`tree`, `parent`, `object`, `type`, and `tag`) can\n be used to specify the value in the header field.\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex f63fd45edb9..d46ca6147e3 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -448,6 +448,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \n+\tfilter_ahead_behind(format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex e005a7ef3ce..08945ad6802 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n+#include \"commit-reach.h\"\n \n static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n@@ -100,6 +101,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n+\tfilter_ahead_behind(&format, &array);\n+\n \tref_array_sort(sorting, &array);\n \n \tif (!maxcount || array.nr < maxcount)\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex d428c45dc8d..4f203a4ad21 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -66,6 +66,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n \tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n+\tfilter_ahead_behind(format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f8203c6b052..896bf703f59 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -158,6 +158,7 @@ enum atom_type {\n \tATOM_THEN,\n \tATOM_ELSE,\n \tATOM_REST,\n+\tATOM_AHEADBEHIND,\n };\n \n /*\n@@ -586,6 +587,16 @@ static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int ahead_behind_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"expected format: %%(ahead-behind:)\"));\n+\n+\tstring_list_append(&format->bases, arg);\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -645,6 +656,7 @@ static struct {\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n \t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n+\t[ATOM_AHEADBEHIND] = { \"ahead-behind\", SOURCE_OTHER, FIELD_STR, ahead_behind_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -1848,6 +1860,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \tstruct object *obj;\n \tint i;\n \tstruct object_info empty = OBJECT_INFO_INIT;\n+\tint ahead_behind_atoms = 0;\n \n \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n \n@@ -1978,6 +1991,16 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\telse\n \t\t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_AHEADBEHIND) {\n+\t\t\tif (ref->counts) {\n+\t\t\t\tconst struct ahead_behind_count *count;\n+\t\t\t\tcount = ref->counts[ahead_behind_atoms++];\n+\t\t\t\tv->s = xstrfmt(\"%d %d\", count->ahead, count->behind);\n+\t\t\t} else {\n+\t\t\t\t/* Not a commit. */\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\t}\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2328,6 +2351,7 @@ static void free_array_item(struct ref_array_item *item)\n \t\t\tfree((char *)item->value[i].s);\n \t\tfree(item->value);\n \t}\n+\tfree(item->counts);\n \tfree(item);\n }\n \n@@ -2356,6 +2380,8 @@ void ref_array_clear(struct ref_array *array)\n \t\tfree_worktrees(ref_to_worktree_map.worktrees);\n \t\tref_to_worktree_map.worktrees = NULL;\n \t}\n+\n+\tFREE_AND_NULL(array->counts);\n }\n \n #define EXCLUDE_REACHED 0\n@@ -2418,6 +2444,50 @@ static void reach_filter(struct ref_array *array,\n \tfree(to_clear);\n }\n \n+void filter_ahead_behind(struct ref_format *format,\n+\t\t\t struct ref_array *array)\n+{\n+\tstruct commit **commits;\n+\tsize_t commits_nr = format->bases.nr + array->nr;\n+\n+\tif (!format->bases.nr || !array->nr)\n+\t\treturn;\n+\n+\tALLOC_ARRAY(commits, commits_nr);\n+\tfor (size_t i = 0; i < format->bases.nr; i++) {\n+\t\tconst char *name = format->bases.items[i].string;\n+\t\tcommits[i] = lookup_commit_reference_by_name(name);\n+\t\tif (!commits[i])\n+\t\t\tdie(\"failed to find '%s'\", name);\n+\t}\n+\n+\tALLOC_ARRAY(array->counts, st_mult(format->bases.nr, array->nr));\n+\n+\tcommits_nr = format->bases.nr;\n+\tarray->counts_nr = 0;\n+\tfor (size_t i = 0; i < array->nr; i++) {\n+\t\tconst char *name = array->items[i]->refname;\n+\t\tcommits[commits_nr] = lookup_commit_reference_by_name(name);\n+\n+\t\tif (!commits[commits_nr])\n+\t\t\tcontinue;\n+\n+\t\tCALLOC_ARRAY(array->items[i]->counts, format->bases.nr);\n+\t\tfor (size_t j = 0; j < format->bases.nr; j++) {\n+\t\t\tstruct ahead_behind_count *count;\n+\t\t\tcount = &array->counts[array->counts_nr++];\n+\t\t\tcount->tip_index = commits_nr;\n+\t\t\tcount->base_index = j;\n+\n+\t\t\tarray->items[i]->counts[j] = count;\n+\t\t}\n+\t\tcommits_nr++;\n+\t}\n+\n+\tahead_behind(commits, commits_nr, array->counts, array->counts_nr);\n+\tfree(commits);\n+}\n+\n /*\n * API for filtering a set of refs. Based on the type of refs the user\n * has requested, we iterate through those refs and apply filters\ndiff --git a/ref-filter.h b/ref-filter.h\nindex aa0eea4ecf5..7e8bff3864e 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -5,6 +5,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"parse-options.h\"\n+#include \"string-list.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -24,6 +25,7 @@\n \n struct atom_value;\n struct ref_sorting;\n+struct ahead_behind_count;\n \n enum ref_sorting_order {\n \tREF_SORTING_REVERSE = 1<<0,\n@@ -40,6 +42,8 @@ struct ref_array_item {\n \tconst char *symref;\n \tstruct commit *commit;\n \tstruct atom_value *value;\n+\tstruct ahead_behind_count **counts;\n+\n \tchar refname[FLEX_ARRAY];\n };\n \n@@ -47,6 +51,9 @@ struct ref_array {\n \tint nr, alloc;\n \tstruct ref_array_item **items;\n \tstruct rev_info *revs;\n+\n+\tstruct ahead_behind_count *counts;\n+\tsize_t counts_nr;\n };\n \n struct ref_filter {\n@@ -80,9 +87,15 @@ struct ref_format {\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n+\n+\t/* List of bases for ahead-behind counts. */\n+\tstruct string_list bases;\n };\n \n-#define REF_FORMAT_INIT { .use_color = -1 }\n+#define REF_FORMAT_INIT { \\\n+\t.use_color = -1, \\\n+\t.bases = STRING_LIST_INIT_DUP, \\\n+}\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\n@@ -143,4 +156,14 @@ struct ref_array_item *ref_array_push(struct ref_array *array,\n \t\t\t\t const char *refname,\n \t\t\t\t const struct object_id *oid);\n \n+/*\n+ * If the provided format includes ahead-behind atoms, then compute the\n+ * ahead-behind values for the array of filtered references. Must be\n+ * called after filter_refs() but before outputting the formatted refs.\n+ *\n+ * If this is not called, then any ahead-behind atoms will be blank.\n+ */\n+void filter_ahead_behind(struct ref_format *format,\n+\t\t\t struct ref_array *array);\n+\n #endif /* REF_FILTER_H */\ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nnew file mode 100755\nindex 00000000000..439a448c2e6\n--- /dev/null\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='Commit walk performance tests'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit for-each-ref --format=\"%(refname)\" \"refs/heads/*\" \"refs/tags/*\" >allrefs &&\n+\tsort -r allrefs | head -n 50 >refs &&\n+\tfor ref in $(cat refs)\n+\tdo\n+\t\tgit branch -f ref-$ref $ref &&\n+\t\techo ref-$ref ||\n+\t\treturn 1\n+\tdone >branches &&\n+\tfor ref in $(cat refs)\n+\tdo\n+\t\tgit tag -f tag-$ref $ref &&\n+\t\techo tag-$ref ||\n+\t\treturn 1\n+\tdone >tags &&\n+\tgit commit-graph write --reachable\n+'\n+\n+test_perf 'ahead-behind counts: git for-each-ref' '\n+\tgit for-each-ref --format=\"%(ahead-behind:HEAD)\" --stdin expect <<-\\EOF &&\n+\t(HEAD detached from fromtag) 0 0\n+\trefs/heads/ambiguous 0 0\n+\trefs/heads/branch-one 1 0\n+\trefs/heads/branch-two 0 0\n+\trefs/heads/main 1 0\n+\trefs/heads/ref-to-branch 1 0\n+\trefs/heads/ref-to-remote 1 0\n+\tEOF\n+\tgit branch --format=\"%(refname) %(ahead-behind:HEAD)\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'git branch with --format=%(rest) must fail' '\n \ttest_must_fail git branch --format=\"%(rest)\" >actual\n '\ndiff --git a/t/t6301-for-each-ref-errors.sh b/t/t6301-for-each-ref-errors.sh\nindex bfda1f46ad2..7db1fc4d7b3 100755\n--- a/t/t6301-for-each-ref-errors.sh\n+++ b/t/t6301-for-each-ref-errors.sh\n@@ -54,4 +54,16 @@ test_expect_success 'Missing objects are reported correctly' '\n \ttest_must_be_empty brief-err\n '\n \n+test_expect_success 'ahead-behind requires an argument' '\n+\ttest_must_fail git for-each-ref \\\n+\t\t--format=\"%(ahead-behind)\" 2>err &&\n+\tgrep \"expected format: %(ahead-behind:)\" err\n+'\n+\n+test_expect_success 'missing ahead-behind base' '\n+\ttest_must_fail git for-each-ref \\\n+\t\t--format=\"%(ahead-behind:refs/heads/missing)\" 2>err &&\n+\tgrep \"failed to find '\\''refs/heads/missing'\\''\" err\n+'\n+\n test_done\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 338a9c46a24..0cb50797ef7 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -443,4 +443,90 @@ test_expect_success 'get_reachable_subset:none' '\n \ttest_all_modes get_reachable_subset\n '\n \n+test_expect_success 'for-each-ref ahead-behind:linear' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 8\n+\trefs/heads/commit-1-3 0 6\n+\trefs/heads/commit-1-5 0 4\n+\trefs/heads/commit-1-8 0 1\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-1-9)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:all' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 24\n+\trefs/heads/commit-2-4 0 17\n+\trefs/heads/commit-4-2 0 17\n+\trefs/heads/commit-4-4 0 9\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-5-5)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 53\n+\trefs/heads/commit-4-8 8 30\n+\trefs/heads/commit-5-3 0 39\n+\trefs/heads/commit-9-9 27 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-9-6)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some, multibase' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-7-8\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 53 0 53\n+\trefs/heads/commit-4-8 8 30 0 22\n+\trefs/heads/commit-5-3 0 39 0 39\n+\trefs/heads/commit-7-8 14 12 8 6\n+\trefs/heads/commit-9-9 27 0 27 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-9-6) %(ahead-behind:commit-6-9)\" \\\n+\t\t--stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:none' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-7-5\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-4-8 16 16\n+\trefs/heads/commit-7-5 7 4\n+\trefs/heads/commit-9-9 49 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-8-4)\" --stdin\n+'\n+\n test_done\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 9aa1660651b..d099e707efd 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -792,6 +792,34 @@ test_expect_success 'annotations for blobs are empty' '\n \ttest_cmp expect actual\n '\n \n+# Run this before doing any signing, so the test has the same results\n+# regardless of the GPG prereq.\n+test_expect_success 'git tag --format with ahead-behind' '\n+\ttest_when_finished git reset --hard tag-one-line &&\n+\tgit commit --allow-empty -m \"left\" &&\n+\tgit tag -a -m left tag-left &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit commit --allow-empty -m \"right\" &&\n+\tgit tag -a -m left tag-right &&\n+\n+\t# Use \" !\" at the end to demonstrate whitepsace\n+\t# around empty ahead-behind token for tag-blob.\n+\tcat >expect <<-EOF &&\n+\trefs/tags/tag-blob !\n+\trefs/tags/tag-left 1 1 !\n+\trefs/tags/tag-lines 0 1 !\n+\trefs/tags/tag-one-line 0 1 !\n+\trefs/tags/tag-right 0 0 !\n+\trefs/tags/tag-zero-lines 0 1 !\n+\tEOF\n+\tgit tag -l --format=\"%(refname) %(ahead-behind:HEAD) !\" >actual 2>err &&\n+\tgrep \"refs/tags/tag\" actual >actual.focus &&\n+\ttest_cmp expect actual.focus &&\n+\n+\t# Error reported for tags that point to non-commits.\n+\tgrep \"error: object [0-9a-f]* is a blob, not a commit\" err\n+'\n+\n # trying to verify annotated non-signed tags:\n \n test_expect_success GPG \\\n-- \ngitgitgadget\n\n"},{"id":"473372","messageId":"xmqqsfec5viq.fsf@gitster.g","threadId":"59347","inReplyTo":"a1d9e0f6ff6660c9264673be18bc24956f74eb9c.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-10T18:08:45Z","receivedAt":"2023-03-10T18:08:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> When reading from stdin, we populate the filter.name_patterns array\n> dynamically as opposed to pointing to the 'argv' array directly. This\n> requires a careful cast while freeing the individual strings,\n> conditioned on the --stdin option.\n\nIndeed. Thanks for carefully describing the concerns you had.\n\nLooking good. I'll read on.\n"},{"id":"473377","messageId":"xmqqlek45sqb.fsf@gitster.g","threadId":"59347","inReplyTo":"82dd6f44a33279551bb638357df4bc82253283e5.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 7/8] for-each-ref: add ahead-behind format atom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-10T19:09:00Z","receivedAt":"2023-03-10T19:09:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> Test this tree\n> ---------------------------------------------------------------\n> 1500.2: ahead-behind counts: git for-each-ref 0.27(0.24+0.02)\n> 1500.3: ahead-behind counts: git branch 0.27(0.24+0.03)\n> 1500.4: ahead-behind counts: git tag 0.28(0.27+0.01)\n> 1500.5: ahead-behind counts: git rev-list 4.57(4.03+0.54)\n>\n> The 'git rev-list' test exists in this change as a demonstration, but it\n> will be removed in the next change to avoid wasting time on this\n> comparison.\n\nNice.\n\n> +ahead-behind:::\n> +\tTwo integers, separated by a space, demonstrating the number of\n> +\tcommits ahead and behind, respectively, when comparing the output\n> +\tref to the `` specified in the format.\n\nDon't we take any commit-ish, not necessarily a ref? In the context\nof for-each-ref documentation, I am afraid that the readers assume\nthat the word refers to a ref and %(ahead-behind:ea6e93913b) and the\nlike are forbidden, which is not what you wanted when you used\nlookup_commit_reference_by_name() in the implementation.\n\n> +\t# Use \" !\" at the end to demonstrate whitepsace\n\npsace.\n\n> +\t# around empty ahead-behind token for tag-blob.\n\n;-)\n\n> +\tcat >expect <<-EOF &&\n> +\trefs/tags/tag-blob !\n> +\trefs/tags/tag-left 1 1 !\n> +\trefs/tags/tag-lines 0 1 !\n> +\trefs/tags/tag-one-line 0 1 !\n> +\trefs/tags/tag-right 0 0 !\n> +\trefs/tags/tag-zero-lines 0 1 !\n> +\tEOF\n> +\tgit tag -l --format=\"%(refname) %(ahead-behind:HEAD) !\" >actual 2>err &&\n> +\tgrep \"refs/tags/tag\" actual >actual.focus &&\n> +\ttest_cmp expect actual.focus &&\n> +\n> +\t# Error reported for tags that point to non-commits.\n> +\tgrep \"error: object [0-9a-f]* is a blob, not a commit\" err\n> +'\n> +\n> # trying to verify annotated non-signed tags:\n> \n> test_expect_success GPG \\\n"},{"id":"473378","messageId":"xmqqedpw5se6.fsf@gitster.g","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-10T19:16:17Z","receivedAt":"2023-03-10T19:16:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> I was\n> initially concerned about the overhead of 'git for-each-ref' and its\n> generality and sorting, but I was not able to measure any important\n> difference between this implementation and our internal 'git ahead-behind'\n> implementation.\n\nThat certainly is nice to know.\n\n> However, for our specific uses, we like to batch a list of exact references\n> that could be very long. We introduce a new --stdin option here.\n>\n> To keep things close to the v1 outline, I replaced the existing patches with\n> closely-related ones, when possible.\n>\n> Patch 1 adds the --stdin option to 'git for-each-ref'. (This is similar to\n> the boilerplate patch from v1.)\n>\n> Patch 2 adds a test to explicitly check that 'git for-each-ref' will still\n> succeed when all input refs are missing. (This is similar to the\n> --ignore-missing patch from v1.)\n\nSensible.\n\n> Patches 3-5 introduce a new method: ensure_generations_valid(). Patch 3 does\n> some refactoring of the existing generation number computations to make it\n> more generic, and patch 4 updates the definition of\n> commit_graph_generation() slightly, making way for patch 5 to implement the\n> method. With an existing commit-graph file, the commits that are not present\n> in the file are considered as having generation number \"infinity\". This is\n> useful for most of our reachability queries to this point, since those\n> commits are \"above\" the ones tracked by the commit-graph. When these commits\n> are low in number, then there is very little performance cost and zero\n> correctness cost. (These patches match v1 exactly.)\n>\n> However, we will see that the ahead/behind computation requires accurate\n> generation numbers to avoid overcounting. Thus, ensure_generations_valid()\n> is a way to specify a list of commits that need generation numbers computed\n> before continuing. It's a no-op if all of those commits are in the\n> commit-graph file. It's expensive if the commit-graph doesn't exist.\n\nReasonable.\n\n> However, '%(ahead-behind:)' computations are likely to be slow no matter\n> what without a commit-graph, so assuming an existing commit-graph file is\n> reasonable. If we find sufficient desire to have an implementation that does\n> not have this requirement, we could create a second implementation and\n> toggle to it when generation_numbers_enabled() returns false.\n\nAt that point, it might make sense to find a way to make the work\nensure_generations_valid() had to spend cycles on not to go to\nwaste. Something like \"ah, you do not have commit-graph at all, so\nlet's try to create one if you can write into the repository\" at the\nbeginning of the function, or something? Just thinking aloud.\n\n> Patch 6 implements the ahead-behind algorithm, but it is not connected to a\n> builtin. It's a long commit message, so hopefully it explains the algorithm\n> sufficiently. (The difference from v1 is that it no longer integrates with a\n> builtin and there are no new tests. It also uses 'unsigned int' and is\n> correctly co-authored by Taylor.)\n\nNice.\n\n> Patch 7 integrates the ahead-behind algorithm with the ref-filter code,\n> including parsing the \"ahead-behind\" token. This finally adds tests that\n> check both ahead_behind() and ensure_generations_valid() via\n> t6600-test-reach.sh. (This patch is essentially completely new in v2.)\n>\n> Patch 8 implements the tips_reachable_from_base() method, and uses it within\n> the ref-filter code to speed up 'git for-each-ref --merged' and 'git branch\n> --merged'. (The interface is slightly different than v1, due to the needs of\n> the new caller.)\n\nVery nice.\n\nHaving read all the patches, I am very impressed and pleased, but\nare we losing anything by having the feature inside for-each-ref\ncompared to a new command ahead-behind? As far as I can tell, the\nnew \"for-each-ref --stdin\" would still want to match refs and work\nonly on refs, but there shouldn't be any reason for ahead-behind\ncomputation to limit to tips that are at the tip of a ref, so that\nmay be one downside in this updated design. For the intended use\ncase of \"let's find which branches are stale\", that downside does\nnot matter in practice, but for other use cases people will think\nof in the future, the limitation might matter (at which time we can\neasily resurrect the other subcommand, using the internal machinery\nwe have here, so it is not a huge deal, I presume).\n\nThanks.\n\n"},{"id":"473380","messageId":"74656b93-8c67-ffe3-4aea-4c184345384d@github.com","threadId":"59347","inReplyTo":"xmqqedpw5se6.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-10T19:25:52Z","receivedAt":"2023-03-10T19:26:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/10/2023 2:16 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n\n>> However, '%(ahead-behind:)' computations are likely to be slow no matter\n>> what without a commit-graph, so assuming an existing commit-graph file is\n>> reasonable. If we find sufficient desire to have an implementation that does\n>> not have this requirement, we could create a second implementation and\n>> toggle to it when generation_numbers_enabled() returns false.\n> \n> At that point, it might make sense to find a way to make the work\n> ensure_generations_valid() had to spend cycles on not to go to\n> waste. Something like \"ah, you do not have commit-graph at all, so\n> let's try to create one if you can write into the repository\" at the\n> beginning of the function, or something? Just thinking aloud.\n\nThis is a reasonable idea for helping users get out of a slow\nsituation. Let's see how often this is a problem to see if it\nis worth doing that extra work in another series.\n\n> Having read all the patches, I am very impressed and pleased, but\n> are we losing anything by having the feature inside for-each-ref\n> compared to a new command ahead-behind? As far as I can tell, the\n> new \"for-each-ref --stdin\" would still want to match refs and work\n> only on refs, but there shouldn't be any reason for ahead-behind\n> computation to limit to tips that are at the tip of a ref, so that\n> may be one downside in this updated design. For the intended use\n> case of \"let's find which branches are stale\", that downside does\n> not matter in practice, but for other use cases people will think\n> of in the future, the limitation might matter (at which time we can\n> easily resurrect the other subcommand, using the internal machinery\n> we have here, so it is not a huge deal, I presume).\n\nI think the for-each-ref implementation solves the use case we\nhad in mind, I think. I'll double-check to see if we ever use\nexact commit IDs instead of reference names, but I think these\ncallers are rarely interested in an exact commit ID but instead\nwant the latest version of refs.\n\nThe idea of using committish tips definitely changes the\nfunctionality boundary. You are right that we can introduce a\nnew builtin easily if that is necessary. Even without the\nahead-behind builtin, we are succeeding in reducing the diff\nbetween our fork and the core project.\n\nThanks,\n-Stolee\n"},{"id":"473449","messageId":"6badb697-aa74-cc2f-ba43-0bbf54b10973@dunelm.org.uk","threadId":"59347","inReplyTo":"a1d9e0f6ff6660c9264673be18bc24956f74eb9c.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-03-13T10:31:44Z","receivedAt":"2023-03-13T10:31:51Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Stolee\n\nOn 10/03/2023 17:20, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> When a user wishes to input a large list of patterns to 'git\n> for-each-ref' (likely a long list of exact refs) there are frequently\n> system limits on the number of command-line arguments.\n> \n> Add a new --stdin option to instead read the patterns from standard\n> input. Add tests that check that any unrecognized arguments are\n> considered an error when --stdin is provided. Also, an empty pattern\n> list is interpreted as the complete ref set.\n> \n> When reading from stdin, we populate the filter.name_patterns array\n> dynamically as opposed to pointing to the 'argv' array directly. This\n> requires a careful cast while freeing the individual strings,\n> conditioned on the --stdin option.\n\nI think what you've got here is fine, but if you wanted you could \nsimplify it by using an strvec. Something like\n\n\tstruct strvec vec = STRVEC_INIT;\n\n\t...\n\n\tif (from_stdin) {\n\t\tstruct strbuf buf = STRBUF_INIT;\n\n\t\tif (argv[0])\n\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n\n\t\twhile (strbuf_getline(&line, stdin) != EOF)\n\t\t\tstrvec_push(&vec, buf.buf);\n\n\t\tfilter.name_patters = vec.v;\n\t} else {\n\t\tfilter.name_patterns = argv;\n\t}\n\n\t...\n\n\tstrvec_clear(&vec);\n\ngets rid of the manual memory management with ALLOC_GROW() and the need \nto cast filter.name_patterns when free()ing. It is not immediately \nobvious from the name but struct strvec keeps the array NULL terminated.\n\nBest Wishes\n\nPhillip\n\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-for-each-ref.txt | 7 +++++-\n> builtin/for-each-ref.c | 29 ++++++++++++++++++++++-\n> t/t6300-for-each-ref.sh | 37 ++++++++++++++++++++++++++++++\n> 3 files changed, 71 insertions(+), 2 deletions(-)\n> \n> diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\n> index 6da899c6296..ccdc2911bb9 100644\n> --- a/Documentation/git-for-each-ref.txt\n> +++ b/Documentation/git-for-each-ref.txt\n> @@ -9,7 +9,8 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n> -\t\t [(--sort=)...] [--format=] [...]\n> +\t\t [(--sort=)...] [--format=]\n> +\t\t [ --stdin | ... ]\n> \t\t [--points-at=]\n> \t\t [--merged[=]] [--no-merged[=]]\n> \t\t [--contains[=]] [--no-contains[=]]\n> @@ -32,6 +33,10 @@ OPTIONS\n> \tliterally, in the latter case matching completely or from the\n> \tbeginning up to a slash.\n> \n> +--stdin::\n> +\tIf `--stdin` is supplied, then the list of patterns is read from\n> +\tstandard input instead of from the argument list.\n> +\n> --count=::\n> \tBy default the command shows all refs that match\n> \t``. This option makes it stop after showing\n> diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\n> index 6f62f40d126..e005a7ef3ce 100644\n> --- a/builtin/for-each-ref.c\n> +++ b/builtin/for-each-ref.c\n> @@ -25,6 +25,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \tstruct ref_format format = REF_FORMAT_INIT;\n> \tstruct strbuf output = STRBUF_INIT;\n> \tstruct strbuf err = STRBUF_INIT;\n> +\tint from_stdin = 0;\n> \n> \tstruct option opts[] = {\n> \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n> @@ -49,6 +50,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \t\tOPT_CONTAINS(&filter.with_commit, N_(\"print only refs which contain the commit\")),\n> \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n> \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n> +\t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -75,7 +77,27 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n> \tfilter.ignore_case = icase;\n> \n> -\tfilter.name_patterns = argv;\n> +\tif (from_stdin) {\n> +\t\tstruct strbuf line = STRBUF_INIT;\n> +\t\tsize_t nr = 0, alloc = 16;\n> +\n> +\t\tif (argv[0])\n> +\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n> +\n> +\t\tCALLOC_ARRAY(filter.name_patterns, alloc);\n> +\n> +\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n> +\t\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n> +\t\t\tfilter.name_patterns[nr++] = strbuf_detach(&line, NULL);\n> +\t\t}\n> +\n> +\t\t/* Add a terminating NULL string. */\n> +\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n> +\t\tfilter.name_patterns[nr + 1] = NULL;\n> +\t} else {\n> +\t\tfilter.name_patterns = argv;\n> +\t}\n> +\n> \tfilter.match_as_path = 1;\n> \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n> \tref_array_sort(sorting, &array);\n> @@ -97,5 +119,10 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \tfree_commit_list(filter.with_commit);\n> \tfree_commit_list(filter.no_commit);\n> \tref_sorting_release(sorting);\n> +\tif (from_stdin) {\n> +\t\tfor (size_t i = 0; filter.name_patterns[i]; i++)\n> +\t\t\tfree((char *)filter.name_patterns[i]);\n> +\t\tfree(filter.name_patterns);\n> +\t}\n> \treturn 0;\n> }\n> diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\n> index c466fd989f1..a58053a54c5 100755\n> --- a/t/t6300-for-each-ref.sh\n> +++ b/t/t6300-for-each-ref.sh\n> @@ -1464,4 +1464,41 @@ sig_crlf=\"$(printf \"%s\" \"$sig\" | append_cr; echo dummy)\"\n> sig_crlf=${sig_crlf%dummy}\n> test_atom refs/tags/fake-sig-crlf contents:signature \"$sig_crlf\"\n> \n> +test_expect_success 'git for-each-ref --stdin: empty' '\n> +\t>in &&\n> +\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n> +\tgit for-each-ref --format=\"%(refname)\" >expect &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git for-each-ref --stdin: fails if extra args' '\n> +\t>in &&\n> +\ttest_must_fail git for-each-ref --format=\"%(refname)\" \\\n> +\t\t--stdin refs/heads/extra err &&\n> +\tgrep \"unknown arguments supplied with --stdin\" err\n> +'\n> +\n> +test_expect_success 'git for-each-ref --stdin: matches' '\n> +\tcat >in <<-EOF &&\n> +\trefs/tags/multi*\n> +\trefs/heads/amb*\n> +\tEOF\n> +\n> +\tcat >expect <<-EOF &&\n> +\trefs/heads/ambiguous\n> +\trefs/tags/multi-ref1-100000-user1\n> +\trefs/tags/multi-ref1-100000-user2\n> +\trefs/tags/multi-ref1-200000-user1\n> +\trefs/tags/multi-ref1-200000-user2\n> +\trefs/tags/multi-ref2-100000-user1\n> +\trefs/tags/multi-ref2-100000-user2\n> +\trefs/tags/multi-ref2-200000-user1\n> +\trefs/tags/multi-ref2-200000-user2\n> +\trefs/tags/multiline\n> +\tEOF\n> +\n> +\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_done\n"},{"id":"473455","messageId":"cd61cd39-ba84-49ae-4b36-92c7be043740@github.com","threadId":"59347","inReplyTo":"6badb697-aa74-cc2f-ba43-0bbf54b10973@dunelm.org.uk","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-13T13:33:29Z","receivedAt":"2023-03-13T13:33:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/13/2023 6:31 AM, Phillip Wood wrote:\n> Hi Stolee\n> \n> On 10/03/2023 17:20, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> When a user wishes to input a large list of patterns to 'git\n>> for-each-ref' (likely a long list of exact refs) there are frequently\n>> system limits on the number of command-line arguments.\n>>\n>> Add a new --stdin option to instead read the patterns from standard\n>> input. Add tests that check that any unrecognized arguments are\n>> considered an error when --stdin is provided. Also, an empty pattern\n>> list is interpreted as the complete ref set.\n>>\n>> When reading from stdin, we populate the filter.name_patterns array\n>> dynamically as opposed to pointing to the 'argv' array directly. This\n>> requires a careful cast while freeing the individual strings,\n>> conditioned on the --stdin option.\n> \n> I think what you've got here is fine, but if you wanted you could simplify it by using an strvec. Something like\n> \n>     struct strvec vec = STRVEC_INIT;\n> \n>     ...\n> \n>     if (from_stdin) {\n>         struct strbuf buf = STRBUF_INIT;\n> \n>         if (argv[0])\n>             die(_(\"unknown arguments supplied with --stdin\"));\n> \n>         while (strbuf_getline(&line, stdin) != EOF)\n>             strvec_push(&vec, buf.buf);\n> \n>         filter.name_patters = vec.v;\n>     } else {\n>         filter.name_patterns = argv;\n>     }\n> \n>     ...\n> \n>     strvec_clear(&vec);\n> \n> gets rid of the manual memory management with ALLOC_GROW() and the need to cast filter.name_patterns when free()ing. It is not immediately obvious from the name but struct strvec keeps the array NULL terminated.\n\nThanks, Philip. I like your version a lot and will use\nit in the next version.\n\n-Stolee\n"},{"id":"473476","messageId":"ZA+RWc+eejXyPfO4@nand.local","threadId":"59347","inReplyTo":"cd61cd39-ba84-49ae-4b36-92c7be043740@github.com","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-03-13T21:10:49Z","receivedAt":"2023-03-13T21:11:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 13, 2023 at 09:33:29AM -0400, Derrick Stolee wrote:\n> Thanks, Philip. I like your version a lot and will use\n> it in the next version.\n\nDitto. Thanks, both.\n\nThanks,\nTaylor\n"},{"id":"473523","messageId":"230315.86lejyxi13.gmgdl@evledraar.gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-03-15T13:22:54Z","receivedAt":"2023-03-15T13:26:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n\n> At $DAYJOB, we have used a custom 'ahead-behind' builtin in our fork of Git\n> for lots of reasons. The main goal of the builtin is to compare multiple\n> references against a common base reference. The comparison is number of\n> commits that are in each side of the symmtric difference of their reachable\n> sets. A commit C is \"ahead\" of a commit B by the number of commits in B..C\n> (reachable from C but not reachable from B). Similarly, the commit C is\n> \"behind\" the commit B by the number of commits in C..B (reachable from B but\n> not reachable from C).\n\nI have a local change to get rid of the various \"the_repository\" macros,\nwhich a merge of this in \"seen\" conflicted with (semantically).\n\nThe below patch on top of \"seen\" will fix it, could you please squash it\nin in the appropriate places?\n\nAside from a desire to get rid of \"the_repository\" macros this also\nmakes your own code consistent, i.e. you use repo_clear_commit_marks(),\nbut then use parse_commit() instead of the repo_parse_commit() in the\nsame function.\n\nIt also makes the new API more future-proof, I don't think we should be\nadding new code that implicitly uses \"the_repository\" to our libraries\nif we can help it, much better to pass it down, even if all the current\nusers are built-in that end up using \"the_repository\".\n\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex 21526d9883a..7c7dba839cf 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -448,7 +448,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \n-\tfilter_ahead_behind(format, &array);\n+\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 1cdf8eb5a6b..e097f44e226 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -102,7 +102,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n-\tfilter_ahead_behind(&format, &array);\n+\tfilter_ahead_behind(the_repository, &format, &array);\n \n \tref_array_sort(sorting, &array);\n \ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex 8652d5edd47..7e2f686600a 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -67,7 +67,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n \tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n-\tfilter_ahead_behind(format, &array);\n+\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/commit-reach.c b/commit-reach.c\nindex 0abd43801fc..4a2216b8ae0 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -977,8 +977,9 @@ static void free_bit_array(struct commit *c)\n \t*bitmap = NULL;\n }\n \n-void ahead_behind(struct commit **commits, size_t commits_nr,\n-\t\t struct ahead_behind_count *counts, size_t counts_nr)\n+void ahead_behind(struct repository *r, struct commit **commits,\n+\t\t size_t commits_nr, struct ahead_behind_count *counts,\n+\t\t size_t counts_nr)\n {\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n@@ -1024,7 +1025,7 @@ void ahead_behind(struct commit **commits, size_t commits_nr,\n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tstruct bitmap *bitmap_p;\n \n-\t\t\tparse_commit(p->item);\n+\t\t\trepo_parse_commit(r, p->item);\n \n \t\t\tbitmap_p = init_bit_array(p->item, width);\n \t\t\tbitmap_or(bitmap_p, bitmap_c);\n@@ -1039,7 +1040,7 @@ void ahead_behind(struct commit **commits, size_t commits_nr,\n \t}\n \n \t/* STALE is used here, PARENT2 is used by insert_no_dup(). */\n-\trepo_clear_commit_marks(the_repository, PARENT2 | STALE);\n+\trepo_clear_commit_marks(r, PARENT2 | STALE);\n \tclear_bit_arrays(&bit_arrays);\n \tclear_prio_queue(&queue);\n }\ndiff --git a/commit-reach.h b/commit-reach.h\nindex f871b5dcce9..2269fab8261 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -131,7 +131,8 @@ struct ahead_behind_count {\n * Given an array of commits and an array of ahead_behind_count pairs,\n * compute the ahead/behind counts for each pair.\n */\n-void ahead_behind(struct commit **commits, size_t commits_nr,\n-\t\t struct ahead_behind_count *counts, size_t counts_nr);\n+void ahead_behind(struct repository *r, struct commit **commits,\n+\t\t size_t commits_nr, struct ahead_behind_count *counts,\n+\t\t size_t counts_nr);\n \n #endif\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 4125ec3fd3a..cdc054beede 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2462,7 +2462,7 @@ static void reach_filter(struct ref_array *array,\n \tfree(to_clear);\n }\n \n-void filter_ahead_behind(struct ref_format *format,\n+void filter_ahead_behind(struct repository *r, struct ref_format *format,\n \t\t\t struct ref_array *array)\n {\n \tstruct commit **commits;\n@@ -2502,7 +2502,7 @@ void filter_ahead_behind(struct ref_format *format,\n \t\tcommits_nr++;\n \t}\n \n-\tahead_behind(commits, commits_nr, array->counts, array->counts_nr);\n+\tahead_behind(r, commits, commits_nr, array->counts, array->counts_nr);\n \tfree(commits);\n }\n \ndiff --git a/ref-filter.h b/ref-filter.h\nindex 7e8bff3864e..1a757b49233 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -163,7 +163,7 @@ struct ref_array_item *ref_array_push(struct ref_array *array,\n *\n * If this is not called, then any ahead-behind atoms will be blank.\n */\n-void filter_ahead_behind(struct ref_format *format,\n+void filter_ahead_behind(struct repository *r, struct ref_format *format,\n \t\t\t struct ref_array *array);\n \n #endif /* REF_FILTER_H */\n"},{"id":"473525","messageId":"230315.86h6umxh7c.gmgdl@evledraar.gmail.com","threadId":"59347","inReplyTo":"a1d9e0f6ff6660c9264673be18bc24956f74eb9c.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-03-15T13:37:39Z","receivedAt":"2023-03-15T13:45:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> When a user wishes to input a large list of patterns to 'git\n> for-each-ref' (likely a long list of exact refs) there are frequently\n> system limits on the number of command-line arguments.\n\nOkey, and the current API assumes you just assign \"argv\" to this, but...\n\n> When reading from stdin, we populate the filter.name_patterns array\n> dynamically as opposed to pointing to the 'argv' array directly. This\n> requires a careful cast while freeing the individual strings,\n> conditioned on the --stdin option.\n\n..sounds potentially nasty...\n\n> @@ -75,7 +77,27 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n> \tfilter.ignore_case = icase;\n> \n> -\tfilter.name_patterns = argv;\n> +\tif (from_stdin) {\n> +\t\tstruct strbuf line = STRBUF_INIT;\n> +\t\tsize_t nr = 0, alloc = 16;\n> +\n> +\t\tif (argv[0])\n> +\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n> +\n> +\t\tCALLOC_ARRAY(filter.name_patterns, alloc);\n> +\n> +\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n> +\t\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n> +\t\t\tfilter.name_patterns[nr++] = strbuf_detach(&line, NULL);\n> +\t\t}\n> +\n> +\t\t/* Add a terminating NULL string. */\n> +\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n> +\t\tfilter.name_patterns[nr + 1] = NULL;\n> +\t} else {\n> +\t\tfilter.name_patterns = argv;\n> +\t}\n> +\n> \tfilter.match_as_path = 1;\n> \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n> \tref_array_sort(sorting, &array);\n> @@ -97,5 +119,10 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \tfree_commit_list(filter.with_commit);\n> \tfree_commit_list(filter.no_commit);\n> \tref_sorting_release(sorting);\n> +\tif (from_stdin) {\n> +\t\tfor (size_t i = 0; filter.name_patterns[i]; i++)\n> +\t\t\tfree((char *)filter.name_patterns[i]);\n> +\t\tfree(filter.name_patterns);\n> +\t}\n> \treturn 0;\n> }\n\nWhy do we need to seemingly re-invent a \"struct strvec\" here? I tried to\nsimplify this on top of this (well, \"seen\"), and we can get rid of all\nof this manual memory management & trailing NULL juggling as a result:\n\t\n\tdiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\n\tindex cf5ba6ffc12..13b75eff28c 100644\n\t--- a/builtin/for-each-ref.c\n\t+++ b/builtin/for-each-ref.c\n\t@@ -7,6 +7,7 @@\n\t #include \"parse-options.h\"\n\t #include \"ref-filter.h\"\n\t #include \"commit-reach.h\"\n\t+#include \"strvec.h\"\n\t \n\t static char const * const for_each_ref_usage[] = {\n\t \tN_(\"git for-each-ref [] []\"),\n\t@@ -27,6 +28,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n\t \tstruct ref_format format = REF_FORMAT_INIT;\n\t \tstruct strbuf output = STRBUF_INIT;\n\t \tstruct strbuf err = STRBUF_INIT;\n\t+\tstruct strvec stdin_pat = STRVEC_INIT;\n\t \tint from_stdin = 0;\n\t \n\t \tstruct option opts[] = {\n\t@@ -81,21 +83,13 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n\t \n\t \tif (from_stdin) {\n\t \t\tstruct strbuf line = STRBUF_INIT;\n\t-\t\tsize_t nr = 0, alloc = 16;\n\t \n\t \t\tif (argv[0])\n\t \t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n\t \n\t-\t\tCALLOC_ARRAY(filter.name_patterns, alloc);\n\t-\n\t-\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n\t-\t\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n\t-\t\t\tfilter.name_patterns[nr++] = strbuf_detach(&line, NULL);\n\t-\t\t}\n\t-\n\t-\t\t/* Add a terminating NULL string. */\n\t-\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n\t-\t\tfilter.name_patterns[nr + 1] = NULL;\n\t+\t\twhile (strbuf_getline(&line, stdin) != EOF)\n\t+\t\t\tstrvec_push(&stdin_pat, line.buf);\n\t+\t\tfilter.name_patterns = stdin_pat.v;\n\t \t} else {\n\t \t\tfilter.name_patterns = argv;\n\t \t}\n\t@@ -123,10 +117,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n\t \tfree_commit_list(filter.with_commit);\n\t \tfree_commit_list(filter.no_commit);\n\t \tref_sorting_release(sorting);\n\t-\tif (from_stdin) {\n\t-\t\tfor (size_t i = 0; filter.name_patterns[i]; i++)\n\t-\t\t\tfree(filter.name_patterns[i]);\n\t-\t\tfree(filter.name_patterns);\n\t-\t}\n\t+\tstrvec_clear(&stdin_pat);\n\t \treturn 0;\n\t }\n\nIt *is* an extra copy though, as your implementation re-uses the strbuf\nwe already allocated.\n\nBut presumably that's trivial in this case, and if we care I think we\nshould resurrect something like [1] instead, i.e. we could just teach\nthe strvec API to have a strvec_push_nodup(). But I doubt that in this\ncase it'll matter.\n\nIn any case, if you don't want to take this as-is, please fix this so\nthat we're not reaching into the \"filter.name_patterns\" and casting its\n\"const char\" to \"char\".\n\nIf we're going to add a hack here that API should instead know how to\nfree its own resources (so we could clean up the free_commit_list() here\nseen in the context), and we could carry some \"my argv needs free-ing\".\n\nBut none of that seems needed in this case, this is just another case\nwhere we can pretend that we have a \"normal\" argv, and then clean up our\nown strvec, no?\n\n1. https://lore.kernel.org/git/65a620b08ef359e29d678497f1b529e3ce6477b1.1673475190.git.gitgitgadget@gmail.com/\n"},{"id":"473526","messageId":"bfade752-6468-95a8-433b-44dc1d13413d@github.com","threadId":"59347","inReplyTo":"230315.86lejyxi13.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-15T13:54:30Z","receivedAt":"2023-03-15T13:54:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 9:22 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n> \n>> At $DAYJOB, we have used a custom 'ahead-behind' builtin in our fork of Git\n>> for lots of reasons. The main goal of the builtin is to compare multiple\n>> references against a common base reference. The comparison is number of\n>> commits that are in each side of the symmtric difference of their reachable\n>> sets. A commit C is \"ahead\" of a commit B by the number of commits in B..C\n>> (reachable from C but not reachable from B). Similarly, the commit C is\n>> \"behind\" the commit B by the number of commits in C..B (reachable from B but\n>> not reachable from C).\n> \n> I have a local change to get rid of the various \"the_repository\" macros,\n> which a merge of this in \"seen\" conflicted with (semantically).\n\nThanks for doing that important refactoring.\n \n> The below patch on top of \"seen\" will fix it, could you please squash it\n> in in the appropriate places?\n\nGot it. Thanks. v3 will arrive later today with those changes and\nthe recommended strvec changes.\n\nThanks,\n-Stolee\n"},{"id":"473527","messageId":"230315.86cz5axgny.gmgdl@evledraar.gmail.com","threadId":"59347","inReplyTo":"9831c23eadbcd990ca09628e5846056e4879ee3d.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/8] commit-reach: implement ahead_behind() logic","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-03-15T13:50:03Z","receivedAt":"2023-03-15T13:56:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> +void ahead_behind(struct commit **commits, size_t commits_nr,\n> +\t\t struct ahead_behind_count *counts, size_t counts_nr)\n> +{\n> +\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n\nI see we have some of this already in this file, so maybe we should\nleave it for a subsequent cleanup, but as DEVOPTS=extra-all notes here\nwe're using a positional initializer.\n\nWe could instead be more explicit, and do:\n\n\t{ .compare = compare_commits_by_gen_then_commit_date }\n\nBut as noted, maybe for asubsequent cleanup...\n\n> +\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n> +\tsize_t i;\n\nNit: Consider dropping this \"size_t i\" line, and instead...\n\n> +\n> +\tif (!commits_nr || !counts_nr)\n> +\t\treturn;\n> +\n> +\tfor (i = 0; i < counts_nr; i++) {\n\n...just make this \"for (size_t i = 0; ...\", ditto the 2x ones below.\n\n> +struct ahead_behind_count {\n> +\t/**\n> +\t * As input, the *_index members indicate which positions in\n> +\t * the 'tips' array correspond to the tip and base of this\n> +\t * comparison.\n> +\t */\n> +\tsize_t tip_index;\n> +\tsize_t base_index;\n> +\n> +\t/**\n> +\t * These values store the computed counts for each side of the\n> +\t * symmetric difference:\n> +\t *\n> +\t * 'ahead' stores the number of commits reachable from the tip\n> +\t * and not reachable from the base.\n> +\t *\n> +\t * 'behind' stores the number of commits reachable from the base\n> +\t * and not reachable from the tip.\n> +\t */\n> +\tunsigned int ahead;\n> +\tunsigned int behind;\n\nEven though this is the tip of the iceberg in terms of our codebase\noverall, can't we just use \"size_t\" for counts in new APIs?\n\nTrying to squash this into the end-state seems to work:\n\t\n\tdiff --git a/commit-reach.h b/commit-reach.h\n\tindex 2269fab8261..108651213d9 100644\n\t--- a/commit-reach.h\n\t+++ b/commit-reach.h\n\t@@ -123,8 +123,8 @@ struct ahead_behind_count {\n\t \t * 'behind' stores the number of commits reachable from the base\n\t \t * and not reachable from the tip.\n\t \t */\n\t-\tunsigned int ahead;\n\t-\tunsigned int behind;\n\t+\tsize_t ahead;\n\t+\tsize_t behind;\n\t };\n\t \n\t /*\n\tdiff --git a/ref-filter.c b/ref-filter.c\n\tindex cdc054beede..b328db696bf 100644\n\t--- a/ref-filter.c\n\t+++ b/ref-filter.c\n\t@@ -2013,7 +2013,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n\t \t\t\tif (ref->counts) {\n\t \t\t\t\tconst struct ahead_behind_count *count;\n\t \t\t\t\tcount = ref->counts[ahead_behind_atoms++];\n\t-\t\t\t\tv->s = xstrfmt(\"%d %d\", count->ahead, count->behind);\n\t+\t\t\t\tv->s = xstrfmt(\"%\"PRIuMAX\" \" \"%\"PRIuMAX, count->ahead, count->behind);\n\t \t\t\t} else {\n\t \t\t\t\t/* Not a commit. */\n\t \t\t\t\tv->s = xstrdup(\"\");\n\n"},{"id":"473528","messageId":"230315.868rfyxfus.gmgdl@evledraar.gmail.com","threadId":"59347","inReplyTo":"82dd6f44a33279551bb638357df4bc82253283e5.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 7/8] for-each-ref: add ahead-behind format atom","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-03-15T13:57:20Z","receivedAt":"2023-03-15T14:14:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> +ahead-behind:::\n> +\tTwo integers, separated by a space, demonstrating the number of\n> +\tcommits ahead and behind, respectively, when comparing the output\n> +\tref to the `` specified in the format.\n> +\n\nAs a potential (expert) user who hasn't read the code yet I'd think the\nthe \"\" here would be the same as \"update-ref\", but glancing ahead\nat your tests it seems that it does ref matching, so \"refs/heads/master\"\nand \"master\" are both accepted?\n\nSince nothing else uses \"\" here I think we should clearly define\nthe matching rules somehow, or maybe we do, and I missed it.\n\nIs there a reason we couldn't use the same \"\" as for-each-ref's\ntop-level accepts, with the limitation that if it matches more than one\nwe'll die?\n\nLater you have e.g. ahead-behind:HEAD, but do we have test coverage for\ne.g. the edge cases where a refs/heads/HEAD exists?\n\n> @@ -645,6 +656,7 @@ static struct {\n> \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n> \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n> \t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n> +\t[ATOM_AHEADBEHIND] = { \"ahead-behind\", SOURCE_OTHER, FIELD_STR, ahead_behind_atom_parser },\n> \t/*\n> \t * Please update $__git_ref_fieldlist in git-completion.bash\n> \t * when you add new atoms\n> @@ -1848,6 +1860,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> \tstruct object *obj;\n> \tint i;\n> \tstruct object_info empty = OBJECT_INFO_INIT;\n> +\tint ahead_behind_atoms = 0;\n> \n> \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n> \n> @@ -1978,6 +1991,16 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n> \t\t\telse\n> \t\t\t\tv->s = xstrdup(\"\");\n> \t\t\tcontinue;\n> +\t\t} else if (atom_type == ATOM_AHEADBEHIND) {\n> +\t\t\tif (ref->counts) {\n> +\t\t\t\tconst struct ahead_behind_count *count;\n> +\t\t\t\tcount = ref->counts[ahead_behind_atoms++];\n> +\t\t\t\tv->s = xstrfmt(\"%d %d\", count->ahead, count->behind);\n> +\t\t\t} else {\n> +\t\t\t\t/* Not a commit. */\n> +\t\t\t\tv->s = xstrdup(\"\");\n> +\t\t\t}\n> +\t\t\tcontinue;\n> \t\t} else\n> \t\t\tcontinue;\n\nHrm, so despite by earlier suggestion of using \"size_t\" it seems we\nreally are limited to \"int\" in the end, as our \"used_atom_cnt\" is an\n\"int\".\n\nBut anyway, better to implement that limitation here, so we only need to\nfix ref-filter.c to move beyond \"int\".\n\n> \n> @@ -2328,6 +2351,7 @@ static void free_array_item(struct ref_array_item *item)\n> \t\t\tfree((char *)item->value[i].s);\n> \t\tfree(item->value);\n> \t}\n> +\tfree(item->counts);\n> \tfree(item);\n> }\n> \n> @@ -2356,6 +2380,8 @@ void ref_array_clear(struct ref_array *array)\n> \t\tfree_worktrees(ref_to_worktree_map.worktrees);\n> \t\tref_to_worktree_map.worktrees = NULL;\n> \t}\n> +\n> +\tFREE_AND_NULL(array->counts);\n> }\n> \n\nFollows the exsiting pattern, so good, but FWIW I think we could do away\nwith all this \"and NULL\", it looks like the only users are built-ins\nwhich never look at this data again, but then we should probably rename\nit to ref_array_release() or something...\n\n> #define EXCLUDE_REACHED 0\n> @@ -2418,6 +2444,50 @@ static void reach_filter(struct ref_array *array,\n> \tfree(to_clear);\n> }\n> \n> +void filter_ahead_behind(struct ref_format *format,\n> +\t\t\t struct ref_array *array)\n> +{\n> +\tstruct commit **commits;\n> +\tsize_t commits_nr = format->bases.nr + array->nr;\n> +\n> +\tif (!format->bases.nr || !array->nr)\n> +\t\treturn;\n> +\n> +\tALLOC_ARRAY(commits, commits_nr);\n> +\tfor (size_t i = 0; i < format->bases.nr; i++) {\n\nEariler I suggested using this \"size_t\" in a \"for\", which is used here,\ngood, newer code than the other commit, presumably...\n\n> +\t\tconst char *name = format->bases.items[i].string;\n> +\t\tcommits[i] = lookup_commit_reference_by_name(name);\n> +\t\tif (!commits[i])\n> +\t\t\tdie(\"failed to find '%s'\", name);\n> +\t}\n> +\n> +\tALLOC_ARRAY(array->counts, st_mult(format->bases.nr, array->nr));\n> +\n> +\tcommits_nr = format->bases.nr;\n> +\tarray->counts_nr = 0;\n\nNot being very familiar with ref-filter.c, it seems odd that the API is\ntaking pains to clear things elsewhere, but we need to set \"counts_nr\"\nto 0 here before an iteration.\n\nIf I comment this assignment out all the tests pass, is this redundant,\nor left here for some future potential API use?\n\n> diff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\n> new file mode 100755\n> index 00000000000..439a448c2e6\n> --- /dev/null\n> +++ b/t/perf/p1500-graph-walks.sh\n> @@ -0,0 +1,45 @@\n> +#!/bin/sh\n> +\n> +test_description='Commit walk performance tests'\n> +. ./perf-lib.sh\n> +\n> +test_perf_large_repo\n> +\n> +test_expect_success 'setup' '\n> +\tgit for-each-ref --format=\"%(refname)\" \"refs/heads/*\" \"refs/tags/*\" >allrefs &&\n> +\tsort -r allrefs | head -n 50 >refs &&\n\nSome of the point of test_perf_large_repo is being able to point the\ntest to an arbitrary sized repo, why \"head -n 50\" here, instead of just\ndoing that filtering when preparing the test repo?\n\n> +test_expect_success 'ahead-behind requires an argument' '\n> +\ttest_must_fail git for-each-ref \\\n> +\t\t--format=\"%(ahead-behind)\" 2>err &&\n> +\tgrep \"expected format: %(ahead-behind:)\" err\n> +'\n> +\n> +test_expect_success 'missing ahead-behind base' '\n> +\ttest_must_fail git for-each-ref \\\n> +\t\t--format=\"%(ahead-behind:refs/heads/missing)\" 2>err &&\n> +\tgrep \"failed to find '\\''refs/heads/missing'\\''\" err\n> +'\n> +\n\nIs this grep instead of test_cmp for brevity, or because we'll catch\nthis late and spew out other output as well?\n\nI'd think it would be worth testing that we only emit an error. Even if\nyou don't want a full test_cmp we could check the line count too to\nassert that...\n\n> +# Run this before doing any signing, so the test has the same results\n> +# regardless of the GPG prereq.\n> +test_expect_success 'git tag --format with ahead-behind' '\n> +\ttest_when_finished git reset --hard tag-one-line &&\n> +\tgit commit --allow-empty -m \"left\" &&\n> +\tgit tag -a -m left tag-left &&\n> +\tgit reset --hard HEAD~1 &&\n> +\tgit commit --allow-empty -m \"right\" &&\n> +\tgit tag -a -m left tag-right &&\n\nDo we really need this --allow-empty insted of just using \"test_commit\"?\nI.e. is being TREESAME here important?\n\n> +\n> +\t# Use \" !\" at the end to demonstrate whitepsace\n> +\t# around empty ahead-behind token for tag-blob.\n> +\tcat >expect <<-EOF &&\n> +\trefs/tags/tag-blob !\n> +\trefs/tags/tag-left 1 1 !\n> +\trefs/tags/tag-lines 0 1 !\n> +\trefs/tags/tag-one-line 0 1 !\n> +\trefs/tags/tag-right 0 0 !\n> +\trefs/tags/tag-zero-lines 0 1 !\n> +\tEOF\n> +\tgit tag -l --format=\"%(refname) %(ahead-behind:HEAD) !\" >actual 2>err &&\n> +\tgrep \"refs/tags/tag\" actual >actual.focus &&\n> +\ttest_cmp expect actual.focus &&\n> +\n> +\t# Error reported for tags that point to non-commits.\n> +\tgrep \"error: object [0-9a-f]* is a blob, not a commit\" err\n\nMaybe, but at a glance it doesn't seem so, but maybe I'm missing something...\n"},{"id":"473529","messageId":"230315.864jqmxfd9.gmgdl@evledraar.gmail.com","threadId":"59347","inReplyTo":"f3fb6833bd71d559a3076d9617a235614ad9a5f8.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 8/8] commit-reach: add tips_reachable_from_bases()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-03-15T14:13:56Z","receivedAt":"2023-03-15T14:24:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n\n> +{\n> +\tsize_t i;\n\nDitto the decl suggestion in an earlier commit, i.e...\n\n> +\tstruct commit_and_index *commits;\n> +\tunsigned int min_generation_index = 0;\n> +\ttimestamp_t min_generation;\n> +\tstruct commit_list *stack = NULL;\n> +\n> +\tif (!bases || !tips || !tips_nr)\n> +\t\treturn;\n> +\n> +\t/*\n> +\t * Do a depth-first search starting at 'bases' to search for the\n> +\t * tips. Stop at the lowest (un-found) generation number. When\n> +\t * finding the lowest commit, increase the minimum generation\n> +\t * number to the next lowest (un-found) generation number.\n> +\t */\n> +\n> +\tCALLOC_ARRAY(commits, tips_nr);\n> +\n> +\tfor (i = 0; i < tips_nr; i++) {\n\n...move this here?\n\n> +\t\tcommits[i].commit = tips[i];\n> +\t\tcommits[i].index = i;\n> +\t\tcommits[i].generation = commit_graph_generation(tips[i]);\n> +\t}\n> +\n> +\t/* Sort with generation number ascending. */\n> +\tQSORT(commits, tips_nr, compare_commit_and_index_by_generation);\n> +\tmin_generation = commits[0].generation;\n> +\n> +\twhile (bases) {\n> +\t\tparse_commit(bases->item);\n> +\t\tcommit_list_insert(bases->item, &stack);\n> +\t\tbases = bases->next;\n> +\t}\n> +\n> +\twhile (stack) {\n> +\t\tunsigned int j;\n\n...ditto...\n\n> +\t\tint explored_all_parents = 1;\n> +\t\tstruct commit_list *p;\n> +\t\tstruct commit *c = stack->item;\n> +\t\ttimestamp_t c_gen = commit_graph_generation(c);\n> +\n> +\t\t/* Does it match any of our tips? */\n> +\t\tfor (j = min_generation_index; j < tips_nr; j++) {\n\n...to here...\n\n> +\t\t\tif (c_gen < commits[j].generation)\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tif (commits[j].commit == c) {\n> +\t\t\t\ttips[commits[j].index]->object.flags |= mark;\n> +\n> +\t\t\t\tif (j == min_generation_index) {\n> +\t\t\t\t\tunsigned int k = j + 1;\n> +\t\t\t\t\twhile (k < tips_nr &&\n> +\t\t\t\t\t (tips[commits[k].index]->object.flags & mark))\n> +\t\t\t\t\t\tk++;\n> +\n> +\t\t\t\t\t/* Terminate early if all found. */\n> +\t\t\t\t\tif (k >= tips_nr)\n> +\t\t\t\t\t\tgoto done;\n> +\n> +\t\t\t\t\tmin_generation_index = k;\n> +\t\t\t\t\tmin_generation = commits[k].generation;\n> +\t\t\t\t}\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tfor (p = c->parents; p; p = p->next) {\n> +\t\t\tparse_commit(p->item);\n> +\n> +\t\t\t/* Have we already explored this parent? */\n> +\t\t\tif (p->item->object.flags & SEEN)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\t/* Is it below the current minimum generation? */\n> +\t\t\tif (commit_graph_generation(p->item) < min_generation)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\t/* Ok, we will explore from here on. */\n> +\t\t\tp->item->object.flags |= SEEN;\n> +\t\t\texplored_all_parents = 0;\n> +\t\t\tcommit_list_insert(p->item, &stack);\n> +\t\t\tbreak;\n> +\t\t}\n> +\n> +\t\tif (explored_all_parents)\n> +\t\t\tpop_commit(&stack);\n> +\t}\n> +\n> +done:\n> +\tfree(commits);\n> +\trepo_clear_commit_marks(the_repository, SEEN);\n\nI didn't see this in my earlier suggestion for passing \"struct\nrepository\", but I think we should do the same here, i.e. have this\nfunction take a \"r\" argument.\n\n> [...]\n> @@ -2390,33 +2390,21 @@ static void reach_filter(struct ref_array *array,\n> \t\t\t struct commit_list *check_reachable,\n> \t\t\t int include_reached)\n> {\n> -\tstruct rev_info revs;\n> \tint i, old_nr;\n> \tstruct commit **to_clear;\n> -\tstruct commit_list *cr;\n> \n> \tif (!check_reachable)\n> \t\treturn;\n> \n> \tCALLOC_ARRAY(to_clear, array->nr);\n> -\n> -\trepo_init_revisions(the_repository, &revs, NULL);\n> -\n> \tfor (i = 0; i < array->nr; i++) {\n> \t\tstruct ref_array_item *item = array->items[i];\n> -\t\tadd_pending_object(&revs, &item->commit->object, item->refname);\n> \t\tto_clear[i] = item->commit;\n> \t}\n> \n> -\tfor (cr = check_reachable; cr; cr = cr->next) {\n> -\t\tstruct commit *merge_commit = cr->item;\n> -\t\tmerge_commit->object.flags |= UNINTERESTING;\n> -\t\tadd_pending_object(&revs, &merge_commit->object, \"\");\n> -\t}\n> -\n> -\trevs.limited = 1;\n> -\tif (prepare_revision_walk(&revs))\n> -\t\tdie(_(\"revision walk setup failed\"));\n> +\ttips_reachable_from_bases(check_reachable,\n> +\t\t\t\t to_clear, array->nr,\n> +\t\t\t\t UNINTERESTING);\n\nI.e. it's not ideal, but we had a the_repository in this function before\n(should probably have passed it from further up, but whatever), so we\ncould pass that to the new tips_reachable_from_bases() still.\n\n> -test_perf 'ahead-behind counts: git rev-list' '\n> -\tfor r in $(cat refs)\n> -\tdo\n> -\t\tgit rev-list --count \"HEAD..$r\" || return 1\n> -\tdone\n\nWhy does this change require deleting the old perf test? Your commit 7/8\nnotes this test, but here we're deleting it, let's keep it and instead\nnote if the results changed, or stayed the same?\n\nMore generally, your commit message says:\n\n> Add extra tests for this behavior in t6600-test-reach.sh as the\n> interesting data shape of that repository can sometimes demonstrate\n> corner case bugs.\n\nAnd here for a supposed optimization commit you're adding new tests, but\nwhen I try them with the C code at 7/8 they pass.\n\nSo it seems we should add them earlier, and this is a pure-optimization\ncommit, but one that's a bit confused about what goes where? :)\n"},{"id":"473535","messageId":"xmqqh6umovf9.fsf@gitster.g","threadId":"59347","inReplyTo":"230315.868rfyxfus.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 7/8] for-each-ref: add ahead-behind format atom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-15T16:01:46Z","receivedAt":"2023-03-15T16:02:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n>\n>> From: Derrick Stolee \n>> [...]\n>> +ahead-behind:::\n>> +\tTwo integers, separated by a space, demonstrating the number of\n>> +\tcommits ahead and behind, respectively, when comparing the output\n>> +\tref to the `` specified in the format.\n>> +\n>\n> As a potential (expert) user who hasn't read the code yet I'd think the\n> the \"\" here would be the same as \"update-ref\", but glancing ahead\n> at your tests it seems that it does ref matching, so \"refs/heads/master\"\n> and \"master\" are both accepted?\n>\n> Since nothing else uses \"\" here I think we should clearly define\n> the matching rules somehow, or maybe we do, and I missed it.\n\nI vaguely recall noticing this in the previous round, but doesn't\nthis only require a commit-ish, not even a ref? It is parsed with\nlookup_commit_reference_by_name().\n\n\n\n"},{"id":"473536","messageId":"xmqqcz5aovc6.fsf@gitster.g","threadId":"59347","inReplyTo":"230315.86cz5axgny.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 6/8] commit-reach: implement ahead_behind() logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-15T16:03:37Z","receivedAt":"2023-03-15T16:04:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>> +\t/**\n>> +\t * These values store the computed counts for each side of the\n>> +\t * symmetric difference:\n>> +\t *\n>> +\t * 'ahead' stores the number of commits reachable from the tip\n>> +\t * and not reachable from the base.\n>> +\t *\n>> +\t * 'behind' stores the number of commits reachable from the base\n>> +\t * and not reachable from the tip.\n>> +\t */\n>> +\tunsigned int ahead;\n>> +\tunsigned int behind;\n>\n> Even though this is the tip of the iceberg in terms of our codebase\n> overall, can't we just use \"size_t\" for counts in new APIs?\n\nI personally do not see a point in becoming so dogmatic. Plain\n(possibly) 32-bit integers have their places in the code.\n"},{"id":"473538","messageId":"17e8437c-4a48-9029-da7f-c858414f781c@github.com","threadId":"59347","inReplyTo":"230315.868rfyxfus.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 7/8] for-each-ref: add ahead-behind format atom","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-15T16:11:14Z","receivedAt":"2023-03-15T16:12:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 9:57 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n\n>> +test_description='Commit walk performance tests'\n>> +. ./perf-lib.sh\n>> +\n>> +test_perf_large_repo\n>> +\n>> +test_expect_success 'setup' '\n>> +\tgit for-each-ref --format=\"%(refname)\" \"refs/heads/*\" \"refs/tags/*\" >allrefs &&\n>> +\tsort -r allrefs | head -n 50 >refs &&\n> \n> Some of the point of test_perf_large_repo is being able to point the\n> test to an arbitrary sized repo, why \"head -n 50\" here, instead of just\n> doing that filtering when preparing the test repo?\n\nI think it's too much work to expect that the tester removes\nall but a small number of refs for testing here. Using all\nrefs on a repo with may refs would be too slow to be helpful.\n\nThis is especially important when running the entire perf\nsuite on a repo where a large number of refs is _desired_\nfor some of the other tests.\n \n>> +test_expect_success 'ahead-behind requires an argument' '\n>> +\ttest_must_fail git for-each-ref \\\n>> +\t\t--format=\"%(ahead-behind)\" 2>err &&\n>> +\tgrep \"expected format: %(ahead-behind:)\" err\n>> +'\n>> +\n>> +test_expect_success 'missing ahead-behind base' '\n>> +\ttest_must_fail git for-each-ref \\\n>> +\t\t--format=\"%(ahead-behind:refs/heads/missing)\" 2>err &&\n>> +\tgrep \"failed to find '\\''refs/heads/missing'\\''\" err\n>> +'\n>> +\n> \n> Is this grep instead of test_cmp for brevity, or because we'll catch\n> this late and spew out other output as well?\n> \n> I'd think it would be worth testing that we only emit an error. Even if\n> you don't want a full test_cmp we could check the line count too to\n> assert that...\n\nA full test_cmp is a little more annoying to write, but\nis a stronger test, so sure.\n\n>> +# Run this before doing any signing, so the test has the same results\n>> +# regardless of the GPG prereq.\n>> +test_expect_success 'git tag --format with ahead-behind' '\n>> +\ttest_when_finished git reset --hard tag-one-line &&\n>> +\tgit commit --allow-empty -m \"left\" &&\n>> +\tgit tag -a -m left tag-left &&\n>> +\tgit reset --hard HEAD~1 &&\n>> +\tgit commit --allow-empty -m \"right\" &&\n>> +\tgit tag -a -m left tag-right &&\n> \n> Do we really need this --allow-empty insted of just using \"test_commit\"?\n> I.e. is being TREESAME here important?\n\nYou missed this in the commit message:\n\n>> [...] Also, the\n>> test in t7004 is carefully located to avoid being dependent on the GPG\n>> prereq. It also avoids using the test_commit helper, as that will add\n>> ticks to the time and disrupt the expected timestampes in later tag\n>> tests.\n\n(And I see the \"timestampes\" typo now.)\n\nThanks,\n-Stolee\n"},{"id":"473539","messageId":"3f7fe6c4-197e-bde9-50b8-ecce705a9f22@github.com","threadId":"59347","inReplyTo":"xmqqh6umovf9.fsf@gitster.g","subject":"Re: [PATCH v2 7/8] for-each-ref: add ahead-behind format atom","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-15T16:12:45Z","receivedAt":"2023-03-15T16:13:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 12:01 PM, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n> \n>> On Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n>>\n>>> From: Derrick Stolee \n>>> [...]\n>>> +ahead-behind:::\n>>> +\tTwo integers, separated by a space, demonstrating the number of\n>>> +\tcommits ahead and behind, respectively, when comparing the output\n>>> +\tref to the `` specified in the format.\n>>> +\n>>\n>> As a potential (expert) user who hasn't read the code yet I'd think the\n>> the \"\" here would be the same as \"update-ref\", but glancing ahead\n>> at your tests it seems that it does ref matching, so \"refs/heads/master\"\n>> and \"master\" are both accepted?\n>>\n>> Since nothing else uses \"\" here I think we should clearly define\n>> the matching rules somehow, or maybe we do, and I missed it.\n> \n> I vaguely recall noticing this in the previous round, but doesn't\n> this only require a commit-ish, not even a ref? It is parsed with\n> lookup_commit_reference_by_name().\n\nYou noticed it in this round, but I haven't sent v3 yet. I have\nthis in my local copy:\n\nahead-behind:::\n\tTwo integers, separated by a space, demonstrating the number of\n\tcommits ahead and behind, respectively, when comparing the output\n\tref to the `` specified in the format.\n\nThanks,\n-Stolee\n"},{"id":"473540","messageId":"4d8f63c4-2e8c-fd60-f857-4e7971e8c0c0@github.com","threadId":"59347","inReplyTo":"xmqqcz5aovc6.fsf@gitster.g","subject":"Re: [PATCH v2 6/8] commit-reach: implement ahead_behind() logic","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-15T16:13:22Z","receivedAt":"2023-03-15T16:14:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 12:03 PM, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n> \n>>> +\t/**\n>>> +\t * These values store the computed counts for each side of the\n>>> +\t * symmetric difference:\n>>> +\t *\n>>> +\t * 'ahead' stores the number of commits reachable from the tip\n>>> +\t * and not reachable from the base.\n>>> +\t *\n>>> +\t * 'behind' stores the number of commits reachable from the base\n>>> +\t * and not reachable from the tip.\n>>> +\t */\n>>> +\tunsigned int ahead;\n>>> +\tunsigned int behind;\n>>\n>> Even though this is the tip of the iceberg in terms of our codebase\n>> overall, can't we just use \"size_t\" for counts in new APIs?\n> \n> I personally do not see a point in becoming so dogmatic. Plain\n> (possibly) 32-bit integers have their places in the code.\n\nIn particular, we have 32-bit limits on the commit-graph due\nto it being unreasonable to have billions of commits in a\nrepository.\n\nThanks,\n-Stolee\n"},{"id":"473542","messageId":"bbf4921b-851b-7630-c185-f7235f6b28d4@github.com","threadId":"59347","inReplyTo":"230315.864jqmxfd9.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 8/8] commit-reach: add tips_reachable_from_bases()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-15T16:17:53Z","receivedAt":"2023-03-15T16:18:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 10:13 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n\n(omitting the_repository stuff which will be reflected in v3)\n\n>> -test_perf 'ahead-behind counts: git rev-list' '\n>> -\tfor r in $(cat refs)\n>> -\tdo\n>> -\t\tgit rev-list --count \"HEAD..$r\" || return 1\n>> -\tdone\n> \n> Why does this change require deleting the old perf test? Your commit 7/8\n> notes this test, but here we're deleting it, let's keep it and instead\n> note if the results changed, or stayed the same?\n>\n> More generally, your commit message says:\n> \n>> Add extra tests for this behavior in t6600-test-reach.sh as the\n>> interesting data shape of that repository can sometimes demonstrate\n>> corner case bugs.\n\nThis note is about t6600-test-reach.sh, not p1500-graph-walks.sh.\n\nNot only does the previous message note this perf test, it has this to say:\n\n The 'git rev-list' test exists in this change as a demonstration, but it\n will be removed in the next change to avoid wasting time on this\n comparison.\n \n> And here for a supposed optimization commit you're adding new tests, but\n> when I try them with the C code at 7/8 they pass.\n> \n> So it seems we should add them earlier, and this is a pure-optimization\n> commit, but one that's a bit confused about what goes where? :)\n\nI can make it more clear why we are removing it in this commit.\n\nThanks,\n-Stolee\n"},{"id":"473543","messageId":"973033ff-940c-01d9-bb8f-5722591988de@github.com","threadId":"59347","inReplyTo":"bbf4921b-851b-7630-c185-f7235f6b28d4@github.com","subject":"Re: [PATCH v2 8/8] commit-reach: add tips_reachable_from_bases()","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-15T16:18:53Z","receivedAt":"2023-03-15T16:19:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 12:17 PM, Derrick Stolee wrote:\n> On 3/15/2023 10:13 AM, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Fri, Mar 10 2023, Derrick Stolee via GitGitGadget wrote:\n>>\n>>> From: Derrick Stolee \n\n> Not only does the previous message note this perf test, it has this to say:\n> \n> The 'git rev-list' test exists in this change as a demonstration, but it\n> will be removed in the next change to avoid wasting time on this\n> comparison.\n> \n>> And here for a supposed optimization commit you're adding new tests, but\n>> when I try them with the C code at 7/8 they pass.\n>>\n>> So it seems we should add them earlier, and this is a pure-optimization\n>> commit, but one that's a bit confused about what goes where? :)\n> \n> I can make it more clear why we are removing it in this commit.\n\nOh wait, I did:\n\n>> (Note that we remove the iterative 'git rev-list' test from p1500\n>> because it no longer makes sense as a comparison to 'git for-each-ref'\n>> and would just waste time running it for these comparisons.)\n\nThanks,\n-Stolee\n"},{"id":"473546","messageId":"ZBH9xxJDTT3jzm5u@coredump.intra.peff.net","threadId":"59347","inReplyTo":"230315.86h6umxh7c.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-15T17:17:59Z","receivedAt":"2023-03-15T17:18:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 15, 2023 at 02:37:39PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> \t-\t\tCALLOC_ARRAY(filter.name_patterns, alloc);\n> \t-\n> \t-\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n> \t-\t\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n> \t-\t\t\tfilter.name_patterns[nr++] = strbuf_detach(&line, NULL);\n> \t-\t\t}\n> \t-\n> \t-\t\t/* Add a terminating NULL string. */\n> \t-\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n> \t-\t\tfilter.name_patterns[nr + 1] = NULL;\n> \t+\t\twhile (strbuf_getline(&line, stdin) != EOF)\n> \t+\t\t\tstrvec_push(&stdin_pat, line.buf);\n> \t+\t\tfilter.name_patterns = stdin_pat.v;\n> \t \t} else {\n> \t \t\tfilter.name_patterns = argv;\n> \t \t}\n> \t@@ -123,10 +117,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \t \tfree_commit_list(filter.with_commit);\n> \t \tfree_commit_list(filter.no_commit);\n> \t \tref_sorting_release(sorting);\n> \t-\tif (from_stdin) {\n> \t-\t\tfor (size_t i = 0; filter.name_patterns[i]; i++)\n> \t-\t\t\tfree(filter.name_patterns[i]);\n> \t-\t\tfree(filter.name_patterns);\n> \t-\t}\n> \t+\tstrvec_clear(&stdin_pat);\n> \t \treturn 0;\n> \t }\n> \n> It *is* an extra copy though, as your implementation re-uses the strbuf\n> we already allocated.\n\nAt first I thought you meant \"extra allocation\" here. But you really do\nmean an extra copy of the bytes.\n\nThe number of allocations is the same either way. In the original, we\ndetach the strbuf in each iteration of the loop as it becomes the final\nentry in the array, but then have to allocate a new strbuf for the next\niteration. With a strvec, we can reuse the same strbuf over and over,\nbut make a new allocation when we add it to the strvec.\n\nSo yes, we end up with an extra memcpy() of the bytes. But the flip side\nis that the final allocations we store in the strvec are correctly\nsized, without the extra slop that the strbuf added while reading.\n\n> But presumably that's trivial in this case, and if we care I think we\n> should resurrect something like [1] instead, i.e. we could just teach\n> the strvec API to have a strvec_push_nodup(). But I doubt that in this\n> case it'll matter.\n\nYeah, I'd agree it is not important either way in this case. But I\nwanted to think it through above, just because it's not clear to me that\neven in a tight loop, the \"allocate buffer and then attach to the\nstrvec\" approach would be the better tradeoff.\n\nI guess it would make sense to wait for a case where it _does_ matter\nand then we could experiment with the two approaches. ;)\n\n-Peff\n"},{"id":"473547","messageId":"ZBIA7+85Lld+lpUS@coredump.intra.peff.net","threadId":"59347","inReplyTo":"74656b93-8c67-ffe3-4aea-4c184345384d@github.com","subject":"Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-15T17:31:27Z","receivedAt":"2023-03-15T17:31:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 10, 2023 at 02:25:52PM -0500, Derrick Stolee wrote:\n\n> > Having read all the patches, I am very impressed and pleased, but\n> > are we losing anything by having the feature inside for-each-ref\n> > compared to a new command ahead-behind? As far as I can tell, the\n> > new \"for-each-ref --stdin\" would still want to match refs and work\n> > only on refs, but there shouldn't be any reason for ahead-behind\n> > computation to limit to tips that are at the tip of a ref, so that\n> > may be one downside in this updated design. For the intended use\n> > case of \"let's find which branches are stale\", that downside does\n> > not matter in practice, but for other use cases people will think\n> > of in the future, the limitation might matter (at which time we can\n> > easily resurrect the other subcommand, using the internal machinery\n> > we have here, so it is not a huge deal, I presume).\n> \n> I think the for-each-ref implementation solves the use case we\n> had in mind, I think. I'll double-check to see if we ever use\n> exact commit IDs instead of reference names, but I think these\n> callers are rarely interested in an exact commit ID but instead\n> want the latest version of refs.\n\nOne thing I'd worry about here are race conditions.\n\nIf you have a porcelain-ish view (and I'd count \"showing a web page\" as\na porcelain view) that requires several commands to compute, it's\npossible for there to be simultaneous ref updates between your commands.\nIf each command is given a refname, then the results may not be\nconsistent.\n\nE.g., imagine resolving \"main\" to 1234abcd in step one, then somebody\nupdates it to 5678cdef, then you run \"for-each-ref\" to compute\nahead/behind, and now you show an inconsistent result: you say that\n\"main\" points to 1234abcd, but show the wrong ahead/behind information.\n\nShowing 1234abcd at all is out-of-date, of course, but the real problem\nis the lack of atomicity. Most porcelain scripts deal with this by\nresolving the refs immediately, assuming object ids are immutable (which\nthey are modulo games like refs/replace), and then working with them.\n\nI don't know if this is how your current application-level code calling\nahead-behind works, or if it just accepts the possibility of a race (or\nmaybe the call is not presented along with other information so it's\nsort-of atomic on its own). Presumably your double-checking will find\nout. :)\n\nI do otherwise like exposing this as an option of for-each-ref, as that\nis the way I'd expect most normal client users to want to get at the\ninformation. And if this is step 1 and that's good enough for now, and\nwe have a path forward to later expose it for general commits, that's OK\nwith me.\n\n-Peff\n"},{"id":"473548","messageId":"97a34499-7dc0-72f7-abd2-1bd0a78fa956@github.com","threadId":"59347","inReplyTo":"ZBIA7+85Lld+lpUS@coredump.intra.peff.net","subject":"Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-15T17:44:53Z","receivedAt":"2023-03-15T17:45:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 1:31 PM, Jeff King wrote:\n> On Fri, Mar 10, 2023 at 02:25:52PM -0500, Derrick Stolee wrote:\n> \n>>> Having read all the patches, I am very impressed and pleased, but\n>>> are we losing anything by having the feature inside for-each-ref\n>>> compared to a new command ahead-behind? As far as I can tell, the\n>>> new \"for-each-ref --stdin\" would still want to match refs and work\n>>> only on refs, but there shouldn't be any reason for ahead-behind\n>>> computation to limit to tips that are at the tip of a ref, so that\n>>> may be one downside in this updated design. For the intended use\n>>> case of \"let's find which branches are stale\", that downside does\n>>> not matter in practice, but for other use cases people will think\n>>> of in the future, the limitation might matter (at which time we can\n>>> easily resurrect the other subcommand, using the internal machinery\n>>> we have here, so it is not a huge deal, I presume).\n>>\n>> I think the for-each-ref implementation solves the use case we\n>> had in mind, I think. I'll double-check to see if we ever use\n>> exact commit IDs instead of reference names, but I think these\n>> callers are rarely interested in an exact commit ID but instead\n>> want the latest version of refs.\n> \n> One thing I'd worry about here are race conditions.\n> \n> If you have a porcelain-ish view (and I'd count \"showing a web page\" as\n> a porcelain view) that requires several commands to compute, it's\n> possible for there to be simultaneous ref updates between your commands.\n> If each command is given a refname, then the results may not be\n> consistent.\n\n> I don't know if this is how your current application-level code calling\n> ahead-behind works, or if it just accepts the possibility of a race (or\n> maybe the call is not presented along with other information so it's\n> sort-of atomic on its own). Presumably your double-checking will find\n> out. :)\n\nI completely agree on both of these points.\n\nThe major lift in this series is that the two commit walk algorithms\nare being contributed to the core project in a way that are easy to\nmodify our 'git ahead-behind' builtin to use the \"new\" internals\nwithout any UX change. Actually porting the application layer to use\n'git for-each-ref' instead would be a second step, where I'd plan to\ndo this deep dive. From what I understand, though, these race conditions\ndo exist already, but they are minor relative to the cost of doing a\nlookup of all the ref values and then calling this backend.\n \n> I do otherwise like exposing this as an option of for-each-ref, as that\n> is the way I'd expect most normal client users to want to get at the\n> information. And if this is step 1 and that's good enough for now, and\n> we have a path forward to later expose it for general commits, that's OK\n> with me.\n\nAnd if we truly need more general committish inputs for tips (HEAD~10,\ntoo), then a new builtin could be built on top. Modeling it after\nfor-each-ref (for-each-commit?) would be a good start to make the\nbehavior as similar as possible. Doing that in full generality might\nrequire strange updates to ref-filter.[c|h], but we can cross that\nbridge when we come to it.\n\nThanks,\n-Stolee\n"},{"id":"473549","messageId":"f9e80e233f1ca26f07f9db49261e54b7deaf83b8.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 1/8] for-each-ref: add --stdin option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:36Z","receivedAt":"2023-03-15T17:46:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a user wishes to input a large list of patterns to 'git\nfor-each-ref' (likely a long list of exact refs) there are frequently\nsystem limits on the number of command-line arguments.\n\nAdd a new --stdin option to instead read the patterns from standard\ninput. Add tests that check that any unrecognized arguments are\nconsidered an error when --stdin is provided. Also, an empty pattern\nlist is interpreted as the complete ref set.\n\nWhen reading from stdin, we populate the filter.name_patterns array\ndynamically as opposed to pointing to the 'argv' array directly. This is\nsimple when using a strvec, as it is NULL-terminated in the same way. We\nthen free the memory directly from the strvec.\n\nHelped-by: Phillip Wood \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-for-each-ref.txt | 7 +++++-\n builtin/for-each-ref.c | 21 ++++++++++++++++-\n t/t6300-for-each-ref.sh | 37 ++++++++++++++++++++++++++++++\n 3 files changed, 63 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 6da899c6296..ccdc2911bb9 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -9,7 +9,8 @@ SYNOPSIS\n --------\n [verse]\n 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n-\t\t [(--sort=)...] [--format=] [...]\n+\t\t [(--sort=)...] [--format=]\n+\t\t [ --stdin | ... ]\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n@@ -32,6 +33,10 @@ OPTIONS\n \tliterally, in the latter case matching completely or from the\n \tbeginning up to a slash.\n \n+--stdin::\n+\tIf `--stdin` is supplied, then the list of patterns is read from\n+\tstandard input instead of from the argument list.\n+\n --count=::\n \tBy default the command shows all refs that match\n \t``. This option makes it stop after showing\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 6f62f40d126..4c5f2324793 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n+#include \"strvec.h\"\n \n static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n@@ -25,6 +26,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n+\tint from_stdin = 0;\n+\tstruct strvec vec = STRVEC_INIT;\n \n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n@@ -49,6 +52,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_CONTAINS(&filter.with_commit, N_(\"print only refs which contain the commit\")),\n \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n+\t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n \t\tOPT_END(),\n \t};\n \n@@ -75,7 +79,21 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n \tfilter.ignore_case = icase;\n \n-\tfilter.name_patterns = argv;\n+\tif (from_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\tif (argv[0])\n+\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n+\n+\t\twhile (strbuf_getline(&line, stdin) != EOF)\n+\t\t\tstrvec_push(&vec, line.buf);\n+\n+\t\t/* vec.v is NULL-terminated, just like 'argv'. */\n+\t\tfilter.name_patterns = vec.v;\n+\t} else {\n+\t\tfilter.name_patterns = argv;\n+\t}\n+\n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n \tref_array_sort(sorting, &array);\n@@ -97,5 +115,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tfree_commit_list(filter.with_commit);\n \tfree_commit_list(filter.no_commit);\n \tref_sorting_release(sorting);\n+\tstrvec_clear(&vec);\n \treturn 0;\n }\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex c466fd989f1..a58053a54c5 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1464,4 +1464,41 @@ sig_crlf=\"$(printf \"%s\" \"$sig\" | append_cr; echo dummy)\"\n sig_crlf=${sig_crlf%dummy}\n test_atom refs/tags/fake-sig-crlf contents:signature \"$sig_crlf\"\n \n+test_expect_success 'git for-each-ref --stdin: empty' '\n+\t>in &&\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\tgit for-each-ref --format=\"%(refname)\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git for-each-ref --stdin: fails if extra args' '\n+\t>in &&\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" \\\n+\t\t--stdin refs/heads/extra err &&\n+\tgrep \"unknown arguments supplied with --stdin\" err\n+'\n+\n+test_expect_success 'git for-each-ref --stdin: matches' '\n+\tcat >in <<-EOF &&\n+\trefs/tags/multi*\n+\trefs/heads/amb*\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/tags/multi-ref1-100000-user1\n+\trefs/tags/multi-ref1-100000-user2\n+\trefs/tags/multi-ref1-200000-user1\n+\trefs/tags/multi-ref1-200000-user2\n+\trefs/tags/multi-ref2-100000-user1\n+\trefs/tags/multi-ref2-100000-user2\n+\trefs/tags/multi-ref2-200000-user1\n+\trefs/tags/multi-ref2-200000-user2\n+\trefs/tags/multiline\n+\tEOF\n+\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473550","messageId":"f56d6a64d24147b4fc59118d82029fbe7cd5af38.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 2/8] for-each-ref: explicitly test no matches","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:37Z","receivedAt":"2023-03-15T17:46:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe for-each-ref builtin can take a list of ref patterns, but if none\nmatch, it still succeeds (but with no output). Add an explicit test that\ndemonstrates that behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t6300-for-each-ref.sh | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex a58053a54c5..6614469d2d6 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1501,4 +1501,17 @@ test_expect_success 'git for-each-ref --stdin: matches' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git for-each-ref with non-existing refs' '\n+\tcat >in <<-EOF &&\n+\trefs/heads/this-ref-does-not-exist\n+\trefs/tags/bogus\n+\tEOF\n+\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\ttest_must_be_empty actual &&\n+\n+\txargs git for-each-ref --format=\"%(refname)\" actual &&\n+\ttest_must_be_empty actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473551","messageId":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v2.git.1678468863.gitgitgadget@gmail.com","subject":"[PATCH v3 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:35Z","receivedAt":"2023-03-15T17:46:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"At $DAYJOB, we have used a custom 'ahead-behind' builtin in our fork of Git\nfor lots of reasons. The main goal of the builtin is to compare multiple\nreferences against a common base reference. The comparison is number of\ncommits that are in each side of the symmtric difference of their reachable\nsets. A commit C is \"ahead\" of a commit B by the number of commits in B..C\n(reachable from C but not reachable from B). Similarly, the commit C is\n\"behind\" the commit B by the number of commits in C..B (reachable from B but\nnot reachable from C).\n\nThese numbers can be computed by 'git rev-list --count B..C' and 'git\nrev-list --count C..B', but there are common needs that benefit from having\nthe checks being done in the same process:\n\n 1. Our \"branches\" page lists ahead/behind counts for each listed branch as\n compared to the repo's default branch. This can be done with a single\n 'git ahead-behind' process.\n 2. When a branch is updated, a background job checks if any pull requests\n that target that branch should be closed because their branches were\n merged implicitly by that update. These queries can e batched into 'git\n ahead-behind' calls.\n\nIn that second example, we don't need the full ahead/behind counts (although\nit is sufficient to look for branches that are \"zero commits ahead\", meaning\nthey are reachable from the base), and instead reachability is the critical\npiece.\n\nThis series contributes the custom algorithms we used for our 'git\nahead-behind' builtin, but as extensions to 'git for-each-ref':\n\n * Add a new \"%(ahead-behind:)\" format token to for-each-ref which allows\n outputting the ahead/behind values in the format string for a matching\n ref.\n * Add a new algorithm that speeds up the 'git for-each-ref --merged='\n option. This also applies to the 'git branch --merged=' option.\n\nThe idea to use 'git for-each-ref' instead of creating a new builtin is from\nJunio, and simplifies this series significantly compared to v1. I was\ninitially concerned about the overhead of 'git for-each-ref' and its\ngenerality and sorting, but I was not able to measure any important\ndifference between this implementation and our internal 'git ahead-behind'\nimplementation. In particular, when a pattern is given to 'git for-each-ref'\nthat looks like an exact ref, it navigates directly to the ref instead of\nscanning all references for matches.\n\nHowever, for our specific uses, we like to batch a list of exact references\nthat could be very long. We introduce a new --stdin option here.\n\nTo keep things close to the v1 outline, I replaced the existing patches with\nclosely-related ones, when possible.\n\nPatch 1 adds the --stdin option to 'git for-each-ref'. (This is similar to\nthe boilerplate patch from v1.)\n\nPatch 2 adds a test to explicitly check that 'git for-each-ref' will still\nsucceed when all input refs are missing. (This is similar to the\n--ignore-missing patch from v1.)\n\nPatches 3-5 introduce a new method: ensure_generations_valid(). Patch 3 does\nsome refactoring of the existing generation number computations to make it\nmore generic, and patch 4 updates the definition of\ncommit_graph_generation() slightly, making way for patch 5 to implement the\nmethod. With an existing commit-graph file, the commits that are not present\nin the file are considered as having generation number \"infinity\". This is\nuseful for most of our reachability queries to this point, since those\ncommits are \"above\" the ones tracked by the commit-graph. When these commits\nare low in number, then there is very little performance cost and zero\ncorrectness cost. (These patches match v1 exactly.)\n\nHowever, we will see that the ahead/behind computation requires accurate\ngeneration numbers to avoid overcounting. Thus, ensure_generations_valid()\nis a way to specify a list of commits that need generation numbers computed\nbefore continuing. It's a no-op if all of those commits are in the\ncommit-graph file. It's expensive if the commit-graph doesn't exist.\nHowever, '%(ahead-behind:)' computations are likely to be slow no matter\nwhat without a commit-graph, so assuming an existing commit-graph file is\nreasonable. If we find sufficient desire to have an implementation that does\nnot have this requirement, we could create a second implementation and\ntoggle to it when generation_numbers_enabled() returns false.\n\nPatch 6 implements the ahead-behind algorithm, but it is not connected to a\nbuiltin. It's a long commit message, so hopefully it explains the algorithm\nsufficiently. (The difference from v1 is that it no longer integrates with a\nbuiltin and there are no new tests. It also uses 'unsigned int' and is\ncorrectly co-authored by Taylor.)\n\nPatch 7 integrates the ahead-behind algorithm with the ref-filter code,\nincluding parsing the \"ahead-behind\" token. This finally adds tests that\ncheck both ahead_behind() and ensure_generations_valid() via\nt6600-test-reach.sh. (This patch is essentially completely new in v2.)\n\nPatch 8 implements the tips_reachable_from_base() method, and uses it within\nthe ref-filter code to speed up 'git for-each-ref --merged' and 'git branch\n--merged'. (The interface is slightly different than v1, due to the needs of\nthe new caller.)\n\n\nUpdates in v3\n=============\n\n * The APIs are modified to take a 'struct repository *' and use them\n appropriately.\n * The --stdin option in 'git for-each-ref' now uses strvec instead of an\n ad-hoc array.\n\nThanks, -Stolee\n\nDerrick Stolee (7):\n for-each-ref: add --stdin option\n for-each-ref: explicitly test no matches\n commit-graph: combine generation computations\n commit-graph: return generation from memory\n commit-reach: implement ahead_behind() logic\n for-each-ref: add ahead-behind format atom\n commit-reach: add tips_reachable_from_bases()\n\nTaylor Blau (1):\n commit-graph: introduce `ensure_generations_valid()`\n\n Documentation/git-for-each-ref.txt | 12 +-\n builtin/branch.c | 1 +\n builtin/for-each-ref.c | 24 +++-\n builtin/tag.c | 1 +\n commit-graph.c | 208 ++++++++++++++++++----------\n commit-graph.h | 8 ++\n commit-reach.c | 209 +++++++++++++++++++++++++++++\n commit-reach.h | 40 ++++++\n ref-filter.c | 91 ++++++++++---\n ref-filter.h | 26 +++-\n t/perf/p1500-graph-walks.sh | 50 +++++++\n t/t3203-branch-output.sh | 14 ++\n t/t5318-commit-graph.sh | 2 +-\n t/t6300-for-each-ref.sh | 50 +++++++\n t/t6301-for-each-ref-errors.sh | 14 ++\n t/t6600-test-reach.sh | 169 +++++++++++++++++++++++\n t/t7004-tag.sh | 28 ++++\n 17 files changed, 858 insertions(+), 89 deletions(-)\n create mode 100755 t/perf/p1500-graph-walks.sh\n\n\nbase-commit: 725f57037d81e24eacfda6e59a19c60c0b4c8062\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1489%2Fderrickstolee%2Fstolee%2Fupstream-ahead-behind-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1489/derrickstolee/stolee/upstream-ahead-behind-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1489\n\nRange-diff vs v2:\n\n 1: a1d9e0f6ff6 ! 1: f9e80e233f1 for-each-ref: add --stdin option\n @@ Commit message\n list is interpreted as the complete ref set.\n \n When reading from stdin, we populate the filter.name_patterns array\n - dynamically as opposed to pointing to the 'argv' array directly. This\n - requires a careful cast while freeing the individual strings,\n - conditioned on the --stdin option.\n + dynamically as opposed to pointing to the 'argv' array directly. This is\n + simple when using a strvec, as it is NULL-terminated in the same way. We\n + then free the memory directly from the strvec.\n \n + Helped-by: Phillip Wood \n Signed-off-by: Derrick Stolee \n \n ## Documentation/git-for-each-ref.txt ##\n @@ Documentation/git-for-each-ref.txt: OPTIONS\n \t``. This option makes it stop after showing\n \n ## builtin/for-each-ref.c ##\n +@@\n + #include \"object.h\"\n + #include \"parse-options.h\"\n + #include \"ref-filter.h\"\n ++#include \"strvec.h\"\n + \n + static char const * const for_each_ref_usage[] = {\n + \tN_(\"git for-each-ref [] []\"),\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n +\tint from_stdin = 0;\n ++\tstruct strvec vec = STRVEC_INIT;\n \n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const\n -\tfilter.name_patterns = argv;\n +\tif (from_stdin) {\n +\t\tstruct strbuf line = STRBUF_INIT;\n -+\t\tsize_t nr = 0, alloc = 16;\n +\n +\t\tif (argv[0])\n +\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n +\n -+\t\tCALLOC_ARRAY(filter.name_patterns, alloc);\n -+\n -+\t\twhile (strbuf_getline(&line, stdin) != EOF) {\n -+\t\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n -+\t\t\tfilter.name_patterns[nr++] = strbuf_detach(&line, NULL);\n -+\t\t}\n ++\t\twhile (strbuf_getline(&line, stdin) != EOF)\n ++\t\t\tstrvec_push(&vec, line.buf);\n +\n -+\t\t/* Add a terminating NULL string. */\n -+\t\tALLOC_GROW(filter.name_patterns, nr + 1, alloc);\n -+\t\tfilter.name_patterns[nr + 1] = NULL;\n ++\t\t/* vec.v is NULL-terminated, just like 'argv'. */\n ++\t\tfilter.name_patterns = vec.v;\n +\t} else {\n +\t\tfilter.name_patterns = argv;\n +\t}\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const\n \tfree_commit_list(filter.with_commit);\n \tfree_commit_list(filter.no_commit);\n \tref_sorting_release(sorting);\n -+\tif (from_stdin) {\n -+\t\tfor (size_t i = 0; filter.name_patterns[i]; i++)\n -+\t\t\tfree((char *)filter.name_patterns[i]);\n -+\t\tfree(filter.name_patterns);\n -+\t}\n ++\tstrvec_clear(&vec);\n \treturn 0;\n }\n \n 2: 2f162a2f39f = 2: f56d6a64d24 for-each-ref: explicitly test no matches\n 3: db28e82d2a6 = 3: 3b15e9df770 commit-graph: combine generation computations\n 4: 3cf33801443 = 4: abd3e7a67be commit-graph: return generation from memory\n 5: 34dffd836b1 ! 5: e197bddcace commit-graph: introduce `ensure_generations_valid()`\n @@ commit-graph.c: static void compute_generation_numbers(struct write_commit_graph\n + * After this method, all commits reachable from those in the given\n + * list will have non-zero, non-infinite generation numbers.\n + */\n -+void ensure_generations_valid(struct commit **commits, size_t nr)\n ++void ensure_generations_valid(struct repository *r,\n ++\t\t\t struct commit **commits, size_t nr)\n +{\n -+\tstruct repository *r = the_repository;\n +\tint generation_version = get_configured_generation_version(r);\n +\tstruct packed_commit_list list = {\n +\t\t.list = commits,\n @@ commit-graph.h: struct commit_graph_data {\n + * After this method, all commits reachable from those in the given\n + * list will have non-zero, non-infinite generation numbers.\n + */\n -+void ensure_generations_valid(struct commit **commits, size_t nr);\n ++void ensure_generations_valid(struct repository *r,\n ++\t\t\t struct commit **commits, size_t nr);\n +\n #endif\n 6: 9831c23eadb ! 6: 0fb3913810b commit-reach: implement ahead_behind() logic\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t*bitmap = NULL;\n +}\n +\n -+void ahead_behind(struct commit **commits, size_t commits_nr,\n ++void ahead_behind(struct repository *r,\n ++\t\t struct commit **commits, size_t commits_nr,\n +\t\t struct ahead_behind_count *counts, size_t counts_nr)\n +{\n -+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n ++\tstruct prio_queue queue = { .compare = compare_commits_by_gen_then_commit_date };\n +\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n -+\tsize_t i;\n +\n +\tif (!commits_nr || !counts_nr)\n +\t\treturn;\n +\n -+\tfor (i = 0; i < counts_nr; i++) {\n ++\tfor (size_t i = 0; i < counts_nr; i++) {\n +\t\tcounts[i].ahead = 0;\n +\t\tcounts[i].behind = 0;\n +\t}\n +\n -+\tensure_generations_valid(commits, commits_nr);\n ++\tensure_generations_valid(r, commits, commits_nr);\n +\n +\tinit_bit_arrays(&bit_arrays);\n +\n -+\tfor (i = 0; i < commits_nr; i++) {\n ++\tfor (size_t i = 0; i < commits_nr; i++) {\n +\t\tstruct commit *c = commits[i];\n +\t\tstruct bitmap *bitmap = init_bit_array(c, width);\n +\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t\tstruct commit_list *p;\n +\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n +\n -+\t\tfor (i = 0; i < counts_nr; i++) {\n ++\t\tfor (size_t i = 0; i < counts_nr; i++) {\n +\t\t\tint reach_from_tip = !!bitmap_get(bitmap_c, counts[i].tip_index);\n +\t\t\tint reach_from_base = !!bitmap_get(bitmap_c, counts[i].base_index);\n +\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t\tfor (p = c->parents; p; p = p->next) {\n +\t\t\tstruct bitmap *bitmap_p;\n +\n -+\t\t\tparse_commit(p->item);\n ++\t\t\trepo_parse_commit(r, p->item);\n +\n +\t\t\tbitmap_p = init_bit_array(p->item, width);\n +\t\t\tbitmap_or(bitmap_p, bitmap_c);\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t}\n +\n +\t/* STALE is used here, PARENT2 is used by insert_no_dup(). */\n -+\trepo_clear_commit_marks(the_repository, PARENT2 | STALE);\n ++\trepo_clear_commit_marks(r, PARENT2 | STALE);\n +\tclear_bit_arrays(&bit_arrays);\n +\tclear_prio_queue(&queue);\n +}\n @@ commit-reach.h: struct commit_list *get_reachable_subset(struct commit **from, i\n + * Given an array of commits and an array of ahead_behind_count pairs,\n + * compute the ahead/behind counts for each pair.\n + */\n -+void ahead_behind(struct commit **commits, size_t commits_nr,\n ++void ahead_behind(struct repository *r,\n ++\t\t struct commit **commits, size_t commits_nr,\n +\t\t struct ahead_behind_count *counts, size_t counts_nr);\n +\n #endif\n 7: 82dd6f44a33 ! 7: 59cf6759e60 for-each-ref: add ahead-behind format atom\n @@ Commit message\n around commits_nr in the second loop of filter_ahead_behind(). Also, the\n test in t7004 is carefully located to avoid being dependent on the GPG\n prereq. It also avoids using the test_commit helper, as that will add\n - ticks to the time and disrupt the expected timestampes in later tag\n + ticks to the time and disrupt the expected timestamps in later tag\n tests.\n \n Also add performance tests in a new p1300-graph-walks.sh script. This\n @@ Documentation/git-for-each-ref.txt: worktreepath::\n \tout, if it is checked out in any linked worktree. Empty string\n \totherwise.\n \n -+ahead-behind:::\n ++ahead-behind:::\n +\tTwo integers, separated by a space, demonstrating the number of\n +\tcommits ahead and behind, respectively, when comparing the output\n -+\tref to the `` specified in the format.\n ++\tref to the `` specified in the format.\n +\n In addition to the above, for commit and tag objects, the header\n field names (`tree`, `parent`, `object`, `type`, and `tag`) can\n @@ builtin/branch.c: static void print_ref_list(struct ref_filter *filter, struct r\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \n -+\tfilter_ahead_behind(format, &array);\n ++\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\n \n ## builtin/for-each-ref.c ##\n @@\n - #include \"object.h\"\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n + #include \"strvec.h\"\n +#include \"commit-reach.h\"\n \n static char const * const for_each_ref_usage[] = {\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const\n \n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n -+\tfilter_ahead_behind(&format, &array);\n ++\tfilter_ahead_behind(the_repository, &format, &array);\n +\n \tref_array_sort(sorting, &array);\n \n @@ builtin/tag.c: static int list_tags(struct ref_filter *filter, struct ref_sortin\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n \tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n -+\tfilter_ahead_behind(format, &array);\n ++\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\n @@ ref-filter.c: static void reach_filter(struct ref_array *array,\n \tfree(to_clear);\n }\n \n -+void filter_ahead_behind(struct ref_format *format,\n ++void filter_ahead_behind(struct repository *r,\n ++\t\t\t struct ref_format *format,\n +\t\t\t struct ref_array *array)\n +{\n +\tstruct commit **commits;\n @@ ref-filter.c: static void reach_filter(struct ref_array *array,\n +\t\tcommits_nr++;\n +\t}\n +\n -+\tahead_behind(commits, commits_nr, array->counts, array->counts_nr);\n ++\tahead_behind(r, commits, commits_nr, array->counts, array->counts_nr);\n +\tfree(commits);\n +}\n +\n @@ ref-filter.h: struct ref_array_item *ref_array_push(struct ref_array *array,\n + *\n + * If this is not called, then any ahead-behind atoms will be blank.\n + */\n -+void filter_ahead_behind(struct ref_format *format,\n ++void filter_ahead_behind(struct repository *r,\n ++\t\t\t struct ref_format *format,\n +\t\t\t struct ref_array *array);\n +\n #endif /* REF_FILTER_H */\n @@ t/t6301-for-each-ref-errors.sh: test_expect_success 'Missing objects are reporte\n +test_expect_success 'ahead-behind requires an argument' '\n +\ttest_must_fail git for-each-ref \\\n +\t\t--format=\"%(ahead-behind)\" 2>err &&\n -+\tgrep \"expected format: %(ahead-behind:)\" err\n ++\techo \"fatal: expected format: %(ahead-behind:)\" >expect &&\n ++\ttest_cmp expect err\n +'\n +\n +test_expect_success 'missing ahead-behind base' '\n +\ttest_must_fail git for-each-ref \\\n +\t\t--format=\"%(ahead-behind:refs/heads/missing)\" 2>err &&\n -+\tgrep \"failed to find '\\''refs/heads/missing'\\''\" err\n ++\techo \"fatal: failed to find '\\''refs/heads/missing'\\''\" >expect &&\n ++\ttest_cmp expect err\n +'\n +\n test_done\n @@ t/t7004-tag.sh: test_expect_success 'annotations for blobs are empty' '\n +\tgit commit --allow-empty -m \"right\" &&\n +\tgit tag -a -m left tag-right &&\n +\n -+\t# Use \" !\" at the end to demonstrate whitepsace\n ++\t# Use \" !\" at the end to demonstrate whitespace\n +\t# around empty ahead-behind token for tag-blob.\n +\tcat >expect <<-EOF &&\n +\trefs/tags/tag-blob !\n 8: f3fb6833bd7 ! 8: 7476a39331e commit-reach: add tips_reachable_from_bases()\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## commit-reach.c ##\n -@@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +@@ commit-reach.c: void ahead_behind(struct repository *r,\n \tclear_bit_arrays(&bit_arrays);\n \tclear_prio_queue(&queue);\n }\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +\treturn 0;\n +}\n +\n -+void tips_reachable_from_bases(struct commit_list *bases,\n ++void tips_reachable_from_bases(struct repository *r,\n ++\t\t\t struct commit_list *bases,\n +\t\t\t struct commit **tips, size_t tips_nr,\n +\t\t\t int mark)\n +{\n -+\tsize_t i;\n +\tstruct commit_and_index *commits;\n -+\tunsigned int min_generation_index = 0;\n ++\tsize_t min_generation_index = 0;\n +\ttimestamp_t min_generation;\n +\tstruct commit_list *stack = NULL;\n +\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +\n +\tCALLOC_ARRAY(commits, tips_nr);\n +\n -+\tfor (i = 0; i < tips_nr; i++) {\n ++\tfor (size_t i = 0; i < tips_nr; i++) {\n +\t\tcommits[i].commit = tips[i];\n +\t\tcommits[i].index = i;\n +\t\tcommits[i].generation = commit_graph_generation(tips[i]);\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +\tmin_generation = commits[0].generation;\n +\n +\twhile (bases) {\n -+\t\tparse_commit(bases->item);\n ++\t\trepo_parse_commit(r, bases->item);\n +\t\tcommit_list_insert(bases->item, &stack);\n +\t\tbases = bases->next;\n +\t}\n +\n +\twhile (stack) {\n -+\t\tunsigned int j;\n +\t\tint explored_all_parents = 1;\n +\t\tstruct commit_list *p;\n +\t\tstruct commit *c = stack->item;\n +\t\ttimestamp_t c_gen = commit_graph_generation(c);\n +\n +\t\t/* Does it match any of our tips? */\n -+\t\tfor (j = min_generation_index; j < tips_nr; j++) {\n ++\t\tfor (size_t j = min_generation_index; j < tips_nr; j++) {\n +\t\t\tif (c_gen < commits[j].generation)\n +\t\t\t\tbreak;\n +\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +\t\t}\n +\n +\t\tfor (p = c->parents; p; p = p->next) {\n -+\t\t\tparse_commit(p->item);\n ++\t\t\trepo_parse_commit(r, p->item);\n +\n +\t\t\t/* Have we already explored this parent? */\n +\t\t\tif (p->item->object.flags & SEEN)\n @@ commit-reach.c: void ahead_behind(struct commit **commits, size_t commits_nr,\n +\n +done:\n +\tfree(commits);\n -+\trepo_clear_commit_marks(the_repository, SEEN);\n ++\trepo_clear_commit_marks(r, SEEN);\n +}\n \n ## commit-reach.h ##\n -@@ commit-reach.h: struct ahead_behind_count {\n - void ahead_behind(struct commit **commits, size_t commits_nr,\n +@@ commit-reach.h: void ahead_behind(struct repository *r,\n + \t\t struct commit **commits, size_t commits_nr,\n \t\t struct ahead_behind_count *counts, size_t counts_nr);\n \n +/*\n + * For all tip commits, add 'mark' to their flags if and only if they\n + * are reachable from one of the commits in 'bases'.\n + */\n -+void tips_reachable_from_bases(struct commit_list *bases,\n ++void tips_reachable_from_bases(struct repository *r,\n ++\t\t\t struct commit_list *bases,\n +\t\t\t struct commit **tips, size_t tips_nr,\n +\t\t\t int mark);\n +\n @@ ref-filter.c: static void reach_filter(struct ref_array *array,\n -\trevs.limited = 1;\n -\tif (prepare_revision_walk(&revs))\n -\t\tdie(_(\"revision walk setup failed\"));\n -+\ttips_reachable_from_bases(check_reachable,\n ++\ttips_reachable_from_bases(the_repository,\n ++\t\t\t\t check_reachable,\n +\t\t\t\t to_clear, array->nr,\n +\t\t\t\t UNINTERESTING);\n \n\n-- \ngitgitgadget\n"},{"id":"473552","messageId":"e197bddcace3fc4ca980d1a459d13c0f85c2de7c.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 5/8] commit-graph: introduce `ensure_generations_valid()`","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:40Z","receivedAt":"2023-03-15T17:46:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nUse the just-introduced compute_reachable_generation_numbers_1() to\nimplement a function which dynamically computes topological levels (or\ncorrected commit dates) for out-of-graph commits.\n\nThis will be useful for the ahead-behind algorithm we are about to\nintroduce, which needs accurate topological levels on _all_ commits\nreachable from the tips in order to avoid over-counting.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 +++++++++++++++++++++++++++++\n commit-graph.h | 8 ++++++++\n 2 files changed, 37 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b4da4e05067..0df8d27afc8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1609,6 +1609,35 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void set_generation_in_graph_data(struct commit *c, timestamp_t t,\n+\t\t\t\t\t void *data)\n+{\n+\tcommit_graph_data_at(c)->generation = t;\n+}\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct repository *r,\n+\t\t\t struct commit **commits, size_t nr)\n+{\n+\tint generation_version = get_configured_generation_version(r);\n+\tstruct packed_commit_list list = {\n+\t\t.list = commits,\n+\t\t.alloc = nr,\n+\t\t.nr = nr,\n+\t};\n+\tstruct compute_generation_info info = {\n+\t\t.r = r,\n+\t\t.commits = &list,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_in_graph_data,\n+\t};\n+\n+\tcompute_reachable_generation_numbers_1(&info, generation_version);\n+}\n+\n static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n {\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-computed\",\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 37faee6b66d..73e182ab2d0 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -190,4 +190,12 @@ struct commit_graph_data {\n */\n timestamp_t commit_graph_generation(const struct commit *);\n uint32_t commit_graph_position(const struct commit *);\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct repository *r,\n+\t\t\t struct commit **commits, size_t nr);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"473553","messageId":"3b15e9df770a118331a1b25f51de8ce97c1b7cab.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 3/8] commit-graph: combine generation computations","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:38Z","receivedAt":"2023-03-15T17:46:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis patch extracts the common code used to compute topological levels\nand corrected committer dates into a common routine,\ncompute_reachable_generation_numbers_1().\n\nThis new routine dispatches to call the necessary functions to get and\nset the generation number for a given commit through a vtable (the\ncompute_generation_info struct).\n\nComputing the generation number itself is done in\ncompute_generation_from_max(), which dispatches its implementation based\non the generation version requested, or issuing a BUG() for unrecognized\ngeneration versions.\n\nThis patch cleans up the two places that currently compute topological\nlevels and corrected commit dates by reducing the amount of duplicated\ncode. It also makes it possible to introduce a function which\ndynamically computes those values for commits that aren't stored in a\ncommit-graph, which will be required for the forthcoming ahead-behind\nrewrite.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 171 +++++++++++++++++++++++++++++++------------------\n 1 file changed, 107 insertions(+), 64 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b3..deccf984a0d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1446,24 +1446,53 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n-static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+struct compute_generation_info {\n+\tstruct repository *r;\n+\tstruct packed_commit_list *commits;\n+\tstruct progress *progress;\n+\tint progress_cnt;\n+\n+\ttimestamp_t (*get_generation)(struct commit *c, void *data);\n+\tvoid (*set_generation)(struct commit *c, timestamp_t gen, void *data);\n+\tvoid *data;\n+};\n+\n+static timestamp_t compute_generation_from_max(struct commit *c,\n+\t\t\t\t\t timestamp_t max_gen,\n+\t\t\t\t\t int generation_version)\n+{\n+\tswitch (generation_version) {\n+\tcase 1: /* topological levels */\n+\t\tif (max_gen > GENERATION_NUMBER_V1_MAX - 1)\n+\t\t\tmax_gen = GENERATION_NUMBER_V1_MAX - 1;\n+\t\treturn max_gen + 1;\n+\n+\tcase 2: /* corrected commit date */\n+\t\tif (c->date && c->date > max_gen)\n+\t\t\tmax_gen = c->date - 1;\n+\t\treturn max_gen + 1;\n+\n+\tdefault:\n+\t\tBUG(\"attempting unimplemented version\");\n+\t}\n+}\n+\n+static void compute_reachable_generation_numbers_1(\n+\t\t\tstruct compute_generation_info *info,\n+\t\t\tint generation_version)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n \n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n-\t\t\t\t\tctx->commits.nr);\n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\tuint32_t level;\n+\tfor (i = 0; i < info->commits->nr; i++) {\n+\t\tstruct commit *c = info->commits->list[i];\n+\t\ttimestamp_t gen;\n+\t\trepo_parse_commit(info->r, c);\n+\t\tgen = info->get_generation(c, info->data);\n \n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n+\t\tdisplay_progress(info->progress, info->progress_cnt + 1);\n \n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (level != GENERATION_NUMBER_ZERO)\n+\t\tif (gen != GENERATION_NUMBER_ZERO && gen != GENERATION_NUMBER_INFINITY)\n \t\t\tcontinue;\n \n \t\tcommit_list_insert(c, &list);\n@@ -1471,38 +1500,91 @@ static void compute_topological_levels(struct write_commit_graph_context *ctx)\n \t\t\tstruct commit *current = list->item;\n \t\t\tstruct commit_list *parent;\n \t\t\tint all_parents_computed = 1;\n-\t\t\tuint32_t max_level = 0;\n+\t\t\tuint32_t max_gen = 0;\n \n \t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tlevel = *topo_level_slab_at(ctx->topo_levels, parent->item);\n+\t\t\t\trepo_parse_commit(info->r, parent->item);\n+\t\t\t\tgen = info->get_generation(parent->item, info->data);\n \n-\t\t\t\tif (level == GENERATION_NUMBER_ZERO) {\n+\t\t\t\tif (gen == GENERATION_NUMBER_ZERO) {\n \t\t\t\t\tall_parents_computed = 0;\n \t\t\t\t\tcommit_list_insert(parent->item, &list);\n \t\t\t\t\tbreak;\n \t\t\t\t}\n \n-\t\t\t\tif (level > max_level)\n-\t\t\t\t\tmax_level = level;\n+\t\t\t\tif (gen > max_gen)\n+\t\t\t\t\tmax_gen = gen;\n \t\t\t}\n \n \t\t\tif (all_parents_computed) {\n \t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (max_level > GENERATION_NUMBER_V1_MAX - 1)\n-\t\t\t\t\tmax_level = GENERATION_NUMBER_V1_MAX - 1;\n-\t\t\t\t*topo_level_slab_at(ctx->topo_levels, current) = max_level + 1;\n+\t\t\t\tgen = compute_generation_from_max(\n+\t\t\t\t\t\tcurrent, max_gen,\n+\t\t\t\t\t\tgeneration_version);\n+\t\t\t\tinfo->set_generation(current, gen, info->data);\n \t\t\t}\n \t\t}\n \t}\n+}\n+\n+static timestamp_t get_topo_level(struct commit *c, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\treturn *topo_level_slab_at(ctx->topo_levels, c);\n+}\n+\n+static void set_topo_level(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\t*topo_level_slab_at(ctx->topo_levels, c) = (uint32_t)t;\n+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n+}\n+\n+static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+{\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.progress = ctx->progress,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_topo_level,\n+\t\t.set_generation = set_topo_level,\n+\t\t.data = ctx,\n+\t};\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tcompute_reachable_generation_numbers_1(&info, 1);\n+\n \tstop_progress(&ctx->progress);\n }\n \n+static timestamp_t get_generation_from_graph_data(struct commit *c, void *data)\n+{\n+\treturn commit_graph_data_at(c)->generation;\n+}\n+\n+static void set_generation_v2(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\tstruct commit_graph_data *g = commit_graph_data_at(c);\n+\tg->generation = (uint32_t)t;\n+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n+}\n+\n static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n-\tstruct commit_list *list = NULL;\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.progress = ctx->progress,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_v2,\n+\t\t.data = ctx,\n+\t};\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1517,47 +1599,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\t}\n \t}\n \n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\ttimestamp_t corrected_commit_date;\n-\n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tcorrected_commit_date = commit_graph_data_at(c)->generation;\n-\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (corrected_commit_date != GENERATION_NUMBER_ZERO)\n-\t\t\tcontinue;\n-\n-\t\tcommit_list_insert(c, &list);\n-\t\twhile (list) {\n-\t\t\tstruct commit *current = list->item;\n-\t\t\tstruct commit_list *parent;\n-\t\t\tint all_parents_computed = 1;\n-\t\t\ttimestamp_t max_corrected_commit_date = 0;\n-\n-\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tcorrected_commit_date = commit_graph_data_at(parent->item)->generation;\n-\n-\t\t\t\tif (corrected_commit_date == GENERATION_NUMBER_ZERO) {\n-\t\t\t\t\tall_parents_computed = 0;\n-\t\t\t\t\tcommit_list_insert(parent->item, &list);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\n-\t\t\t\tif (corrected_commit_date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = corrected_commit_date;\n-\t\t\t}\n-\n-\t\t\tif (all_parents_computed) {\n-\t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n-\t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n-\t\t\t}\n-\t\t}\n-\t}\n+\tcompute_reachable_generation_numbers_1(&info, 2);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n@@ -1565,6 +1607,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n \t\t\tctx->num_generation_data_overflows++;\n \t}\n+\n \tstop_progress(&ctx->progress);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"473554","messageId":"0fb3913810b7d4731707a0129df7f5d57c8ab39f.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 6/8] commit-reach: implement ahead_behind() logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:41Z","receivedAt":"2023-03-15T17:46:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nFully implement the commit-counting logic required to determine\nahead/behind counts for a batch of commit pairs. This is a new library\nmethod within commit-reach.h. This method will be linked to the\nfor-each-ref builtin in the next change.\n\nThe interface for ahead_behind() uses two arrays. The first array of\ncommits contains the list of all starting points for the walk. This\nincludes all tip commits _and_ base commits. The second array, using the\nnew ahead_behind_count struct, indicates which commits from that initial\narray form the base/tip pair for the ahead/behind count it will store.\n\nThis implementation of ahead_behind() allows multiple bases, if desired.\nEven with multiple bases, there is only one commit walk used for\ncounting the ahead/behind values, saving time when the base/tip ranges\noverlap significantly.\n\nThis interface for ahead_behind() also makes it very easy to call\nensure_generations_valid() on the entire array of bases and tips. This\ncall is necessary because it is critical that the walk that counts\nahead/behind values never walks a commit more than once. Without\ngeneration numbers on every commit, there is a possibility that a\ncommit date skew could cause the walk to revisit a commit and then\ndouble-count it. For this reason, it is strongly recommended that 'git\nahead-behind' is only run in a repository with a commit-graph file that\ncovers most of the reachable commits, storing precomputed generation\nnumbers. If no commit-graph exists, this walk will be much slower as it\nmust walk all reachable commits in ensure_generations_valid() before\nperforming the counting logic.\n\nIt is possible to detect if generation numbers are available at run time\nand redirect the implementation to another algorithm that does not\nrequire this property. However, that implementation requires a commit\nwalk per base/tip pair _and_ can be slower due to the commit date\nheuristics required. Such an implementation could be considered in the\nfuture if there is a reason to include it, but most Git hosts should\nalready be generating a commit-graph file as part of repository\nmaintenance. Most Git clients should also be generating commit-graph\nfiles as part of background maintenance or automatic GCs.\n\nNow, let's discuss the ahead/behind counting algorithm.\n\nEach commit in the input commit list is associated with a bit position\nindicating \"the ith commit can reach this commit\". Each of these commits\nis associated with a bitmap with its position flipped on and then\nplaced in a queue for walking commit history. We walk commits by popping\nthe commit with maximum generation number out of the queue, guaranteeing\nthat we will never walk a child of that commit in any future steps.\n\nAs we walk, we load the bitmap for the current commit and perform two\nmain steps. The _second_ step examines each parent of the current commit\nand adds the current commit's bitmap bits to each parent's bitmap. (We\ncreate a new bitmap for the parent if this is our first time seeing that\nparent.) After adding the bits to the parent's bitmap, the parent is\nadded to the walk queue. Due to this passing of bits to parents, the\ncurrent commit has a guarantee that the ith bit is enabled on its bitmap\nif and only if the ith commit can reach the current commit.\n\nThe first step of the walk is to examine the bitmask on the current\ncommit and decide which ranges the commit is in or not. Due to the \"bit\npushing\" in the second step, we have a guarantee that the ith bit of the\ncurrent commit's bitmap is on if and only if the ith starting commit can\nreach it. For each ahead_behind_count struct, check the base_index and\ntip_index to see if those bits are enabled on the current bitmap. If\nexactly one bit is enabled, then increment the corresponding 'ahead' or\n'behind' count. This increment is the reason we _absolutely need_ to\nwalk commits at most once.\n\nThe only subtle thing to do with this walk is to check to see if a\nparent has all bits on in its bitmap, in which case it becomes \"stale\"\nand is marked with the STALE bit. This allows queue_has_nonstale() to be\nthe terminating condition of the walk, which greatly reduces the number\nof commits walked if all of the commits are nearby in history. It avoids\nwalking a large number of common commits when there is a deep history.\nWe also use the helper method insert_no_dup() to add commits to the\npriority queue without adding them multiple times. This uses the PARENT2\nflag. Thus, we must clear both the STALE and PARENT2 bits of all\ncommits, in case ahead_behind() is called multiple times in the same\nprocess.\n\nCo-authored-by: Taylor Blau \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-reach.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++\n commit-reach.h | 31 ++++++++++++++++\n 2 files changed, 127 insertions(+)\n\ndiff --git a/commit-reach.c b/commit-reach.c\nindex 2e33c599a82..1e5a1c37fb7 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -8,6 +8,7 @@\n #include \"revision.h\"\n #include \"tag.h\"\n #include \"commit-reach.h\"\n+#include \"ewah/ewok.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define PARENT1\t\t(1u<<16)\n@@ -941,3 +942,98 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \n \treturn found_commits;\n }\n+\n+define_commit_slab(bit_arrays, struct bitmap *);\n+static struct bit_arrays bit_arrays;\n+\n+static void insert_no_dup(struct prio_queue *queue, struct commit *c)\n+{\n+\tif (c->object.flags & PARENT2)\n+\t\treturn;\n+\tprio_queue_put(queue, c);\n+\tc->object.flags |= PARENT2;\n+}\n+\n+static struct bitmap *init_bit_array(struct commit *c, int width)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\t*bitmap = bitmap_word_alloc(width);\n+\treturn *bitmap;\n+}\n+\n+static void free_bit_array(struct commit *c)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\treturn;\n+\tbitmap_free(*bitmap);\n+\t*bitmap = NULL;\n+}\n+\n+void ahead_behind(struct repository *r,\n+\t\t struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr)\n+{\n+\tstruct prio_queue queue = { .compare = compare_commits_by_gen_then_commit_date };\n+\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n+\n+\tif (!commits_nr || !counts_nr)\n+\t\treturn;\n+\n+\tfor (size_t i = 0; i < counts_nr; i++) {\n+\t\tcounts[i].ahead = 0;\n+\t\tcounts[i].behind = 0;\n+\t}\n+\n+\tensure_generations_valid(r, commits, commits_nr);\n+\n+\tinit_bit_arrays(&bit_arrays);\n+\n+\tfor (size_t i = 0; i < commits_nr; i++) {\n+\t\tstruct commit *c = commits[i];\n+\t\tstruct bitmap *bitmap = init_bit_array(c, width);\n+\n+\t\tbitmap_set(bitmap, i);\n+\t\tinsert_no_dup(&queue, c);\n+\t}\n+\n+\twhile (queue_has_nonstale(&queue)) {\n+\t\tstruct commit *c = prio_queue_get(&queue);\n+\t\tstruct commit_list *p;\n+\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n+\n+\t\tfor (size_t i = 0; i < counts_nr; i++) {\n+\t\t\tint reach_from_tip = !!bitmap_get(bitmap_c, counts[i].tip_index);\n+\t\t\tint reach_from_base = !!bitmap_get(bitmap_c, counts[i].base_index);\n+\n+\t\t\tif (reach_from_tip ^ reach_from_base) {\n+\t\t\t\tif (reach_from_base)\n+\t\t\t\t\tcounts[i].behind++;\n+\t\t\t\telse\n+\t\t\t\t\tcounts[i].ahead++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tstruct bitmap *bitmap_p;\n+\n+\t\t\trepo_parse_commit(r, p->item);\n+\n+\t\t\tbitmap_p = init_bit_array(p->item, width);\n+\t\t\tbitmap_or(bitmap_p, bitmap_c);\n+\n+\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n+\t\t\t\tp->item->object.flags |= STALE;\n+\n+\t\t\tinsert_no_dup(&queue, p->item);\n+\t\t}\n+\n+\t\tfree_bit_array(c);\n+\t}\n+\n+\t/* STALE is used here, PARENT2 is used by insert_no_dup(). */\n+\trepo_clear_commit_marks(r, PARENT2 | STALE);\n+\tclear_bit_arrays(&bit_arrays);\n+\tclear_prio_queue(&queue);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex 148b56fea50..f708c46e523 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -104,4 +104,35 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \t\t\t\t\t struct commit **to, int nr_to,\n \t\t\t\t\t unsigned int reachable_flag);\n \n+struct ahead_behind_count {\n+\t/**\n+\t * As input, the *_index members indicate which positions in\n+\t * the 'tips' array correspond to the tip and base of this\n+\t * comparison.\n+\t */\n+\tsize_t tip_index;\n+\tsize_t base_index;\n+\n+\t/**\n+\t * These values store the computed counts for each side of the\n+\t * symmetric difference:\n+\t *\n+\t * 'ahead' stores the number of commits reachable from the tip\n+\t * and not reachable from the base.\n+\t *\n+\t * 'behind' stores the number of commits reachable from the base\n+\t * and not reachable from the tip.\n+\t */\n+\tunsigned int ahead;\n+\tunsigned int behind;\n+};\n+\n+/*\n+ * Given an array of commits and an array of ahead_behind_count pairs,\n+ * compute the ahead/behind counts for each pair.\n+ */\n+void ahead_behind(struct repository *r,\n+\t\t struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"473555","messageId":"abd3e7a67beb80ef687253ddb8bb5fe7a769357f.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 4/8] commit-graph: return generation from memory","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:39Z","receivedAt":"2023-03-15T17:46:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit_graph_generation() method used to report a value of\nGENERATION_NUMBER_INFINITY if the commit_graph_data_slab had an instance\nfor the given commit but the graph_pos indicated the commit was not in\nthe commit-graph file.\n\nInstead, trust the 'generation' member if the commit has a value in the\nslab _and_ the 'generation' member is non-zero. Otherwise, treat it as\nGENERATION_NUMBER_INFINITY.\n\nThis only makes a difference for a very old case for the commit-graph:\nthe very first Git release to write commit-graph files wrote zeroes in\nthe topological level positions. If we are parsing a commit-graph with\nall zeroes, those commits will now appear to have\nGENERATION_NUMBER_INFINITY (as if they were not parsed from the\ncommit-graph).\n\nI attempted several variations to work around the need for providing an\nuninitialized 'generation' member, but this was the best one I found. It\ndoes require a change to a verification test in t5318 because it reports\na different error than the one about non-zero generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 8 +++-----\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 4 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex deccf984a0d..b4da4e05067 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -116,12 +116,10 @@ timestamp_t commit_graph_generation(const struct commit *c)\n \tstruct commit_graph_data *data =\n \t\tcommit_graph_data_slab_peek(&commit_graph_data_slab, c);\n \n-\tif (!data)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n-\telse if (data->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n+\tif (data && data->generation)\n+\t\treturn data->generation;\n \n-\treturn data->generation;\n+\treturn GENERATION_NUMBER_INFINITY;\n }\n \n static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 049c5fc8ead..b6e12115786 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -630,7 +630,7 @@ test_expect_success 'detect incorrect generation number' '\n \n test_expect_success 'detect incorrect generation number' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n-\t\t\"non-zero generation number\"\n+\t\t\"commit-graph generation for commit\"\n '\n \n test_expect_success 'detect incorrect commit date' '\n-- \ngitgitgadget\n\n"},{"id":"473556","messageId":"7476a39331eb1df33c62e8d62a059473d681fa3c.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 8/8] commit-reach: add tips_reachable_from_bases()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:43Z","receivedAt":"2023-03-15T17:46:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBoth 'git for-each-ref --merged=' and 'git branch --merged=' use\nthe ref-filter machinery to select references or branches (respectively)\nthat are reachable from a set of commits presented by one or more\n--merged arguments. This happens within reach_filter(), which uses the\nrevision-walk machinery to walk history in a standard way.\n\nHowever, the commit-reach.c file is full of custom searches that are\nmore efficient, especially for reachability queries that can terminate\nearly when reachability is discovered. Add a new\ntips_reachable_from_bases() method to commit-reach.c and call it from\nwithin reach_filter() in ref-filter.c. This affects both 'git branch'\nand 'git for-each-ref' as tested in p1500-graph-walks.sh.\n\nFor the Linux kernel repository, we take an already-fast algorithm and\nmake it even faster:\n\nTest HEAD~1 HEAD\n-------------------------------------------------------------------\n1500.5: contains: git for-each-ref --merged 0.13 0.02 -84.6%\n1500.6: contains: git branch --merged 0.14 0.02 -85.7%\n1500.7: contains: git tag --merged 0.15 0.03 -80.0%\n\n(Note that we remove the iterative 'git rev-list' test from p1500\nbecause it no longer makes sense as a comparison to 'git for-each-ref'\nand would just waste time running it for these comparisons.)\n\nThe algorithm is implemented in commit-reach.c in the method\ntips_reachable_from_base(). This method takes a string_list of tips and\nassigns the 'util' for each item with the value 1 if the base commit can\nreach those tips.\n\nLike other reachability queries in commit-reach.c, the fastest way to\nsearch for \"can A reach B?\" is to do a depth-first search up to the\ngeneration number of B, preferring to explore first parents before later\nparents. While we must walk all reachable commits up to that generation\nnumber when the answer is \"no\", the depth-first search can answer \"yes\"\nmuch faster than other approaches in most cases.\n\nThis search becomes trickier when there are multiple targets for the\ndepth-first search. The commits with lower generation number are more\nlikely to be within the history of the start commit, but we don't want\nto waste time searching commits of low generation number if the commit\ntarget with lowest generation number has already been found.\n\nThe trick here is to take the input commits and sort them by generation\nnumber in ascending order. Track the index within this order as\nmin_generation_index. When we find a commit, if its index in the list is\nequal to min_generation_index, then we can increase the generation\nnumber boundary of our search to the next-lowest value in the list.\n\nWith this mechanism, the number of commits to search is minimized with\nrespect to the depth-first search heuristic. We will walk all commits up\nto the minimum generation number of a commit that is _not_ reachable\nfrom the start, but we will walk only the necessary portion of the\ndepth-first search for the reachable commits of lower generation.\n\nAdd extra tests for this behavior in t6600-test-reach.sh as the\ninteresting data shape of that repository can sometimes demonstrate\ncorner case bugs.\n\nSigned-off-by: Derrick Stolee \n---\n commit-reach.c | 113 ++++++++++++++++++++++++++++++++++++\n commit-reach.h | 9 +++\n ref-filter.c | 20 ++-----\n t/perf/p1500-graph-walks.sh | 15 +++--\n t/t6600-test-reach.sh | 83 ++++++++++++++++++++++++++\n 5 files changed, 219 insertions(+), 21 deletions(-)\n\ndiff --git a/commit-reach.c b/commit-reach.c\nindex 1e5a1c37fb7..463e9e8fd0e 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -1037,3 +1037,116 @@ void ahead_behind(struct repository *r,\n \tclear_bit_arrays(&bit_arrays);\n \tclear_prio_queue(&queue);\n }\n+\n+struct commit_and_index {\n+\tstruct commit *commit;\n+\tunsigned int index;\n+\ttimestamp_t generation;\n+};\n+\n+static int compare_commit_and_index_by_generation(const void *va, const void *vb)\n+{\n+\tconst struct commit_and_index *a = (const struct commit_and_index *)va;\n+\tconst struct commit_and_index *b = (const struct commit_and_index *)vb;\n+\n+\tif (a->generation > b->generation)\n+\t\treturn 1;\n+\tif (a->generation < b->generation)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n+void tips_reachable_from_bases(struct repository *r,\n+\t\t\t struct commit_list *bases,\n+\t\t\t struct commit **tips, size_t tips_nr,\n+\t\t\t int mark)\n+{\n+\tstruct commit_and_index *commits;\n+\tsize_t min_generation_index = 0;\n+\ttimestamp_t min_generation;\n+\tstruct commit_list *stack = NULL;\n+\n+\tif (!bases || !tips || !tips_nr)\n+\t\treturn;\n+\n+\t/*\n+\t * Do a depth-first search starting at 'bases' to search for the\n+\t * tips. Stop at the lowest (un-found) generation number. When\n+\t * finding the lowest commit, increase the minimum generation\n+\t * number to the next lowest (un-found) generation number.\n+\t */\n+\n+\tCALLOC_ARRAY(commits, tips_nr);\n+\n+\tfor (size_t i = 0; i < tips_nr; i++) {\n+\t\tcommits[i].commit = tips[i];\n+\t\tcommits[i].index = i;\n+\t\tcommits[i].generation = commit_graph_generation(tips[i]);\n+\t}\n+\n+\t/* Sort with generation number ascending. */\n+\tQSORT(commits, tips_nr, compare_commit_and_index_by_generation);\n+\tmin_generation = commits[0].generation;\n+\n+\twhile (bases) {\n+\t\trepo_parse_commit(r, bases->item);\n+\t\tcommit_list_insert(bases->item, &stack);\n+\t\tbases = bases->next;\n+\t}\n+\n+\twhile (stack) {\n+\t\tint explored_all_parents = 1;\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = stack->item;\n+\t\ttimestamp_t c_gen = commit_graph_generation(c);\n+\n+\t\t/* Does it match any of our tips? */\n+\t\tfor (size_t j = min_generation_index; j < tips_nr; j++) {\n+\t\t\tif (c_gen < commits[j].generation)\n+\t\t\t\tbreak;\n+\n+\t\t\tif (commits[j].commit == c) {\n+\t\t\t\ttips[commits[j].index]->object.flags |= mark;\n+\n+\t\t\t\tif (j == min_generation_index) {\n+\t\t\t\t\tunsigned int k = j + 1;\n+\t\t\t\t\twhile (k < tips_nr &&\n+\t\t\t\t\t (tips[commits[k].index]->object.flags & mark))\n+\t\t\t\t\t\tk++;\n+\n+\t\t\t\t\t/* Terminate early if all found. */\n+\t\t\t\t\tif (k >= tips_nr)\n+\t\t\t\t\t\tgoto done;\n+\n+\t\t\t\t\tmin_generation_index = k;\n+\t\t\t\t\tmin_generation = commits[k].generation;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\trepo_parse_commit(r, p->item);\n+\n+\t\t\t/* Have we already explored this parent? */\n+\t\t\tif (p->item->object.flags & SEEN)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Is it below the current minimum generation? */\n+\t\t\tif (commit_graph_generation(p->item) < min_generation)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ok, we will explore from here on. */\n+\t\t\tp->item->object.flags |= SEEN;\n+\t\t\texplored_all_parents = 0;\n+\t\t\tcommit_list_insert(p->item, &stack);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (explored_all_parents)\n+\t\t\tpop_commit(&stack);\n+\t}\n+\n+done:\n+\tfree(commits);\n+\trepo_clear_commit_marks(r, SEEN);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex f708c46e523..d6321ae700e 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -135,4 +135,13 @@ void ahead_behind(struct repository *r,\n \t\t struct commit **commits, size_t commits_nr,\n \t\t struct ahead_behind_count *counts, size_t counts_nr);\n \n+/*\n+ * For all tip commits, add 'mark' to their flags if and only if they\n+ * are reachable from one of the commits in 'bases'.\n+ */\n+void tips_reachable_from_bases(struct repository *r,\n+\t\t\t struct commit_list *bases,\n+\t\t\t struct commit **tips, size_t tips_nr,\n+\t\t\t int mark);\n+\n #endif\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 5a94fea7981..c20010a6e94 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2390,33 +2390,22 @@ static void reach_filter(struct ref_array *array,\n \t\t\t struct commit_list *check_reachable,\n \t\t\t int include_reached)\n {\n-\tstruct rev_info revs;\n \tint i, old_nr;\n \tstruct commit **to_clear;\n-\tstruct commit_list *cr;\n \n \tif (!check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\n \tfor (i = 0; i < array->nr; i++) {\n \t\tstruct ref_array_item *item = array->items[i];\n-\t\tadd_pending_object(&revs, &item->commit->object, item->refname);\n \t\tto_clear[i] = item->commit;\n \t}\n \n-\tfor (cr = check_reachable; cr; cr = cr->next) {\n-\t\tstruct commit *merge_commit = cr->item;\n-\t\tmerge_commit->object.flags |= UNINTERESTING;\n-\t\tadd_pending_object(&revs, &merge_commit->object, \"\");\n-\t}\n-\n-\trevs.limited = 1;\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n+\ttips_reachable_from_bases(the_repository,\n+\t\t\t\t check_reachable,\n+\t\t\t\t to_clear, array->nr,\n+\t\t\t\t UNINTERESTING);\n \n \told_nr = array->nr;\n \tarray->nr = 0;\n@@ -2440,7 +2429,6 @@ static void reach_filter(struct ref_array *array,\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n-\trelease_revisions(&revs);\n \tfree(to_clear);\n }\n \ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nindex 439a448c2e6..e14e7620cce 100755\n--- a/t/perf/p1500-graph-walks.sh\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -35,11 +35,16 @@ test_perf 'ahead-behind counts: git tag' '\n \txargs git tag -l --format=\"%(ahead-behind:HEAD)\" input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\trefs/heads/commit-2-1\n+\trefs/heads/commit-5-1\n+\trefs/heads/commit-9-1\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-1-9 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:all' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-5-5 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-9-6 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:some, multibase' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-7-8\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-5-3\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--merged=commit-5-8 \\\n+\t\t--merged=commit-8-5 \\\n+\t\t--format=\"%(refname)\" \\\n+\t\t--stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:none' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-7-5\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\t>expect &&\n+\trun_all_modes git for-each-ref --merged=commit-8-4 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"473557","messageId":"59cf6759e60cca9dd67a105e65df46f5acbae2fc.1678902343.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v3 7/8] for-each-ref: add ahead-behind format atom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-15T17:45:42Z","receivedAt":"2023-03-15T17:46:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change implemented the ahead_behind() method, including an\nalgorithm to compute the ahead/behind values for a number of commit tips\nrelative to a number of commit bases. Now, integrate that algorithm as\npart of 'git for-each-ref' hidden behind a new format atom,\nahead-behind. This naturally extends to 'git branch' and 'git tag'\nbuiltins, as well.\n\nThis format allows specifying multiple bases, if so desired, and all\nmatching references are compared against all of those bases. For this\nreason, failing to read a reference provided from these atoms results in\nan error.\n\nIn order to translate the ahead_behind() method information to the\nformat output code in ref-filter.c, we must populate arrays of\nahead_behind_count structs. In struct ref_array, we store the full array\nthat will be passed to ahead_behind(). In struct ref_array_item, we\nstore an array of pointers that point to the relvant items within the\nfull array. In this way, we can pull all relevant ahead/behind values\ndirectly when formatting output for a specific item. It also ensures the\nlifetime of the ahead_behind_count structs matches the time that the\narray is being used.\n\nAdd specific tests of the ahead/behind counts in t6600-test-reach.sh, as\nit has an interesting repository shape. In particular, its merging\nstrategy and its use of different commit-graphs would demonstrate over-\ncounting if the ahead_behind() method did not already account for that\npossibility.\n\nAlso add tests for the specific for-each-ref, branch, and tag builtins.\nIn the case of 'git tag', there are intersting cases that happen when\nsome of the selected tips are not commits. This requires careful logic\naround commits_nr in the second loop of filter_ahead_behind(). Also, the\ntest in t7004 is carefully located to avoid being dependent on the GPG\nprereq. It also avoids using the test_commit helper, as that will add\nticks to the time and disrupt the expected timestamps in later tag\ntests.\n\nAlso add performance tests in a new p1300-graph-walks.sh script. This\nwill be useful for more uses in the future, but for now compare the\nahead-behind counting algorithm in 'git for-each-ref' to the naive\nimplementation by running 'git rev-list --count' processes for each\ninput.\n\nFor the Git source code repository, the improvement is already obvious:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git for-each-ref 0.07(0.07+0.00)\n1500.3: ahead-behind counts: git branch 0.07(0.06+0.00)\n1500.4: ahead-behind counts: git tag 0.07(0.06+0.00)\n1500.5: ahead-behind counts: git rev-list 1.32(1.04+0.27)\n\nBut the standard performance benchmark is the Linux kernel repository,\nwhich demosntrates a significant improvement:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git for-each-ref 0.27(0.24+0.02)\n1500.3: ahead-behind counts: git branch 0.27(0.24+0.03)\n1500.4: ahead-behind counts: git tag 0.28(0.27+0.01)\n1500.5: ahead-behind counts: git rev-list 4.57(4.03+0.54)\n\nThe 'git rev-list' test exists in this change as a demonstration, but it\nwill be removed in the next change to avoid wasting time on this\ncomparison.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-for-each-ref.txt | 5 ++\n builtin/branch.c | 1 +\n builtin/for-each-ref.c | 3 ++\n builtin/tag.c | 1 +\n ref-filter.c | 71 ++++++++++++++++++++++++\n ref-filter.h | 26 ++++++++-\n t/perf/p1500-graph-walks.sh | 45 ++++++++++++++++\n t/t3203-branch-output.sh | 14 +++++\n t/t6301-for-each-ref-errors.sh | 14 +++++\n t/t6600-test-reach.sh | 86 ++++++++++++++++++++++++++++++\n t/t7004-tag.sh | 28 ++++++++++\n 11 files changed, 293 insertions(+), 1 deletion(-)\n create mode 100755 t/perf/p1500-graph-walks.sh\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex ccdc2911bb9..0713e49b499 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -222,6 +222,11 @@ worktreepath::\n \tout, if it is checked out in any linked worktree. Empty string\n \totherwise.\n \n+ahead-behind:::\n+\tTwo integers, separated by a space, demonstrating the number of\n+\tcommits ahead and behind, respectively, when comparing the output\n+\tref to the `` specified in the format.\n+\n In addition to the above, for commit and tag objects, the header\n field names (`tree`, `parent`, `object`, `type`, and `tag`) can\n be used to specify the value in the header field.\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex f63fd45edb9..0554d7cebb3 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -448,6 +448,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \n+\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 4c5f2324793..bb559c343ed 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -6,6 +6,7 @@\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n #include \"strvec.h\"\n+#include \"commit-reach.h\"\n \n static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n@@ -96,6 +97,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n+\tfilter_ahead_behind(the_repository, &format, &array);\n+\n \tref_array_sort(sorting, &array);\n \n \tif (!maxcount || array.nr < maxcount)\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex d428c45dc8d..1b3f49d7b4c 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -66,6 +66,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n \tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n+\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f8203c6b052..5a94fea7981 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -158,6 +158,7 @@ enum atom_type {\n \tATOM_THEN,\n \tATOM_ELSE,\n \tATOM_REST,\n+\tATOM_AHEADBEHIND,\n };\n \n /*\n@@ -586,6 +587,16 @@ static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int ahead_behind_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tif (!arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"expected format: %%(ahead-behind:)\"));\n+\n+\tstring_list_append(&format->bases, arg);\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -645,6 +656,7 @@ static struct {\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n \t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n+\t[ATOM_AHEADBEHIND] = { \"ahead-behind\", SOURCE_OTHER, FIELD_STR, ahead_behind_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -1848,6 +1860,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \tstruct object *obj;\n \tint i;\n \tstruct object_info empty = OBJECT_INFO_INIT;\n+\tint ahead_behind_atoms = 0;\n \n \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n \n@@ -1978,6 +1991,16 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\telse\n \t\t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_AHEADBEHIND) {\n+\t\t\tif (ref->counts) {\n+\t\t\t\tconst struct ahead_behind_count *count;\n+\t\t\t\tcount = ref->counts[ahead_behind_atoms++];\n+\t\t\t\tv->s = xstrfmt(\"%d %d\", count->ahead, count->behind);\n+\t\t\t} else {\n+\t\t\t\t/* Not a commit. */\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\t}\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2328,6 +2351,7 @@ static void free_array_item(struct ref_array_item *item)\n \t\t\tfree((char *)item->value[i].s);\n \t\tfree(item->value);\n \t}\n+\tfree(item->counts);\n \tfree(item);\n }\n \n@@ -2356,6 +2380,8 @@ void ref_array_clear(struct ref_array *array)\n \t\tfree_worktrees(ref_to_worktree_map.worktrees);\n \t\tref_to_worktree_map.worktrees = NULL;\n \t}\n+\n+\tFREE_AND_NULL(array->counts);\n }\n \n #define EXCLUDE_REACHED 0\n@@ -2418,6 +2444,51 @@ static void reach_filter(struct ref_array *array,\n \tfree(to_clear);\n }\n \n+void filter_ahead_behind(struct repository *r,\n+\t\t\t struct ref_format *format,\n+\t\t\t struct ref_array *array)\n+{\n+\tstruct commit **commits;\n+\tsize_t commits_nr = format->bases.nr + array->nr;\n+\n+\tif (!format->bases.nr || !array->nr)\n+\t\treturn;\n+\n+\tALLOC_ARRAY(commits, commits_nr);\n+\tfor (size_t i = 0; i < format->bases.nr; i++) {\n+\t\tconst char *name = format->bases.items[i].string;\n+\t\tcommits[i] = lookup_commit_reference_by_name(name);\n+\t\tif (!commits[i])\n+\t\t\tdie(\"failed to find '%s'\", name);\n+\t}\n+\n+\tALLOC_ARRAY(array->counts, st_mult(format->bases.nr, array->nr));\n+\n+\tcommits_nr = format->bases.nr;\n+\tarray->counts_nr = 0;\n+\tfor (size_t i = 0; i < array->nr; i++) {\n+\t\tconst char *name = array->items[i]->refname;\n+\t\tcommits[commits_nr] = lookup_commit_reference_by_name(name);\n+\n+\t\tif (!commits[commits_nr])\n+\t\t\tcontinue;\n+\n+\t\tCALLOC_ARRAY(array->items[i]->counts, format->bases.nr);\n+\t\tfor (size_t j = 0; j < format->bases.nr; j++) {\n+\t\t\tstruct ahead_behind_count *count;\n+\t\t\tcount = &array->counts[array->counts_nr++];\n+\t\t\tcount->tip_index = commits_nr;\n+\t\t\tcount->base_index = j;\n+\n+\t\t\tarray->items[i]->counts[j] = count;\n+\t\t}\n+\t\tcommits_nr++;\n+\t}\n+\n+\tahead_behind(r, commits, commits_nr, array->counts, array->counts_nr);\n+\tfree(commits);\n+}\n+\n /*\n * API for filtering a set of refs. Based on the type of refs the user\n * has requested, we iterate through those refs and apply filters\ndiff --git a/ref-filter.h b/ref-filter.h\nindex aa0eea4ecf5..c9a11495177 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -5,6 +5,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"parse-options.h\"\n+#include \"string-list.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -24,6 +25,7 @@\n \n struct atom_value;\n struct ref_sorting;\n+struct ahead_behind_count;\n \n enum ref_sorting_order {\n \tREF_SORTING_REVERSE = 1<<0,\n@@ -40,6 +42,8 @@ struct ref_array_item {\n \tconst char *symref;\n \tstruct commit *commit;\n \tstruct atom_value *value;\n+\tstruct ahead_behind_count **counts;\n+\n \tchar refname[FLEX_ARRAY];\n };\n \n@@ -47,6 +51,9 @@ struct ref_array {\n \tint nr, alloc;\n \tstruct ref_array_item **items;\n \tstruct rev_info *revs;\n+\n+\tstruct ahead_behind_count *counts;\n+\tsize_t counts_nr;\n };\n \n struct ref_filter {\n@@ -80,9 +87,15 @@ struct ref_format {\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n+\n+\t/* List of bases for ahead-behind counts. */\n+\tstruct string_list bases;\n };\n \n-#define REF_FORMAT_INIT { .use_color = -1 }\n+#define REF_FORMAT_INIT { \\\n+\t.use_color = -1, \\\n+\t.bases = STRING_LIST_INIT_DUP, \\\n+}\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\n@@ -143,4 +156,15 @@ struct ref_array_item *ref_array_push(struct ref_array *array,\n \t\t\t\t const char *refname,\n \t\t\t\t const struct object_id *oid);\n \n+/*\n+ * If the provided format includes ahead-behind atoms, then compute the\n+ * ahead-behind values for the array of filtered references. Must be\n+ * called after filter_refs() but before outputting the formatted refs.\n+ *\n+ * If this is not called, then any ahead-behind atoms will be blank.\n+ */\n+void filter_ahead_behind(struct repository *r,\n+\t\t\t struct ref_format *format,\n+\t\t\t struct ref_array *array);\n+\n #endif /* REF_FILTER_H */\ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nnew file mode 100755\nindex 00000000000..439a448c2e6\n--- /dev/null\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='Commit walk performance tests'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit for-each-ref --format=\"%(refname)\" \"refs/heads/*\" \"refs/tags/*\" >allrefs &&\n+\tsort -r allrefs | head -n 50 >refs &&\n+\tfor ref in $(cat refs)\n+\tdo\n+\t\tgit branch -f ref-$ref $ref &&\n+\t\techo ref-$ref ||\n+\t\treturn 1\n+\tdone >branches &&\n+\tfor ref in $(cat refs)\n+\tdo\n+\t\tgit tag -f tag-$ref $ref &&\n+\t\techo tag-$ref ||\n+\t\treturn 1\n+\tdone >tags &&\n+\tgit commit-graph write --reachable\n+'\n+\n+test_perf 'ahead-behind counts: git for-each-ref' '\n+\tgit for-each-ref --format=\"%(ahead-behind:HEAD)\" --stdin expect <<-\\EOF &&\n+\t(HEAD detached from fromtag) 0 0\n+\trefs/heads/ambiguous 0 0\n+\trefs/heads/branch-one 1 0\n+\trefs/heads/branch-two 0 0\n+\trefs/heads/main 1 0\n+\trefs/heads/ref-to-branch 1 0\n+\trefs/heads/ref-to-remote 1 0\n+\tEOF\n+\tgit branch --format=\"%(refname) %(ahead-behind:HEAD)\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'git branch with --format=%(rest) must fail' '\n \ttest_must_fail git branch --format=\"%(rest)\" >actual\n '\ndiff --git a/t/t6301-for-each-ref-errors.sh b/t/t6301-for-each-ref-errors.sh\nindex bfda1f46ad2..47c2c183c30 100755\n--- a/t/t6301-for-each-ref-errors.sh\n+++ b/t/t6301-for-each-ref-errors.sh\n@@ -54,4 +54,18 @@ test_expect_success 'Missing objects are reported correctly' '\n \ttest_must_be_empty brief-err\n '\n \n+test_expect_success 'ahead-behind requires an argument' '\n+\ttest_must_fail git for-each-ref \\\n+\t\t--format=\"%(ahead-behind)\" 2>err &&\n+\techo \"fatal: expected format: %(ahead-behind:)\" >expect &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'missing ahead-behind base' '\n+\ttest_must_fail git for-each-ref \\\n+\t\t--format=\"%(ahead-behind:refs/heads/missing)\" 2>err &&\n+\techo \"fatal: failed to find '\\''refs/heads/missing'\\''\" >expect &&\n+\ttest_cmp expect err\n+'\n+\n test_done\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 338a9c46a24..0cb50797ef7 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -443,4 +443,90 @@ test_expect_success 'get_reachable_subset:none' '\n \ttest_all_modes get_reachable_subset\n '\n \n+test_expect_success 'for-each-ref ahead-behind:linear' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 8\n+\trefs/heads/commit-1-3 0 6\n+\trefs/heads/commit-1-5 0 4\n+\trefs/heads/commit-1-8 0 1\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-1-9)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:all' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 24\n+\trefs/heads/commit-2-4 0 17\n+\trefs/heads/commit-4-2 0 17\n+\trefs/heads/commit-4-4 0 9\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-5-5)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 53\n+\trefs/heads/commit-4-8 8 30\n+\trefs/heads/commit-5-3 0 39\n+\trefs/heads/commit-9-9 27 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-9-6)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some, multibase' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-7-8\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 53 0 53\n+\trefs/heads/commit-4-8 8 30 0 22\n+\trefs/heads/commit-5-3 0 39 0 39\n+\trefs/heads/commit-7-8 14 12 8 6\n+\trefs/heads/commit-9-9 27 0 27 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-9-6) %(ahead-behind:commit-6-9)\" \\\n+\t\t--stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:none' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-7-5\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-4-8 16 16\n+\trefs/heads/commit-7-5 7 4\n+\trefs/heads/commit-9-9 49 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-8-4)\" --stdin\n+'\n+\n test_done\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 9aa1660651b..04a4b44183d 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -792,6 +792,34 @@ test_expect_success 'annotations for blobs are empty' '\n \ttest_cmp expect actual\n '\n \n+# Run this before doing any signing, so the test has the same results\n+# regardless of the GPG prereq.\n+test_expect_success 'git tag --format with ahead-behind' '\n+\ttest_when_finished git reset --hard tag-one-line &&\n+\tgit commit --allow-empty -m \"left\" &&\n+\tgit tag -a -m left tag-left &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit commit --allow-empty -m \"right\" &&\n+\tgit tag -a -m left tag-right &&\n+\n+\t# Use \" !\" at the end to demonstrate whitespace\n+\t# around empty ahead-behind token for tag-blob.\n+\tcat >expect <<-EOF &&\n+\trefs/tags/tag-blob !\n+\trefs/tags/tag-left 1 1 !\n+\trefs/tags/tag-lines 0 1 !\n+\trefs/tags/tag-one-line 0 1 !\n+\trefs/tags/tag-right 0 0 !\n+\trefs/tags/tag-zero-lines 0 1 !\n+\tEOF\n+\tgit tag -l --format=\"%(refname) %(ahead-behind:HEAD) !\" >actual 2>err &&\n+\tgrep \"refs/tags/tag\" actual >actual.focus &&\n+\ttest_cmp expect actual.focus &&\n+\n+\t# Error reported for tags that point to non-commits.\n+\tgrep \"error: object [0-9a-f]* is a blob, not a commit\" err\n+'\n+\n # trying to verify annotated non-signed tags:\n \n test_expect_success GPG \\\n-- \ngitgitgadget\n\n"},{"id":"473558","messageId":"ZBIFEXuE5Um9Z4zM@coredump.intra.peff.net","threadId":"59347","inReplyTo":"a1d9e0f6ff6660c9264673be18bc24956f74eb9c.1678468864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-15T17:49:05Z","receivedAt":"2023-03-15T17:49:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 10, 2023 at 05:20:56PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> When a user wishes to input a large list of patterns to 'git\n> for-each-ref' (likely a long list of exact refs) there are frequently\n> system limits on the number of command-line arguments.\n> \n> Add a new --stdin option to instead read the patterns from standard\n> input. Add tests that check that any unrecognized arguments are\n> considered an error when --stdin is provided. Also, an empty pattern\n> list is interpreted as the complete ref set.\n> \n> When reading from stdin, we populate the filter.name_patterns array\n> dynamically as opposed to pointing to the 'argv' array directly. This\n> requires a careful cast while freeing the individual strings,\n> conditioned on the --stdin option.\n\nThis is a nice feature to have, but I suspect like other pattern\nfeatures in Git (e.g., pathspecs), the matching is linear, and thus\npre-expanding the set of refs you're interested in becomes accidentally\nquadratic.\n\nAnd that seems to be the case here. If I have N refs and feed the whole\nset as patterns via --stdin:\n\n-- >8 --\nfor i in 4000 8000 16000 32000; do\n rm -rf repo\n git init -q repo\n (\n cd repo\n git commit --allow-empty -qm foo\n perl -e '\n my ($oid, $n) = @ARGV;\n print \"create refs/heads/branch$_ $oid\\n\" for (1..$n);\n ' $(git rev-parse HEAD) $i |\n git update-ref --stdin\n git for-each-ref --format='%(refname)' >refs\n echo -n \"$i: \"\n command time -f %U \\\n git.compile for-each-ref --stdin &1 >/dev/null\n )\ndone\n-- 8< --\n\nthen the result quadruples for every doubling of the refs.\n\n 4000: 0.32\n 8000: 1.33\n 16000: 5.10\n 32000: 20.90\n\nThat may or may not be a show-stopper for your use case, and if not,\nI don't think it's something we need to address immediately. But we may\nwant some kind of \"literal\" mode, that takes in a list of refs rather\nthan a list of patterns, and does a sorted-merge with the list of\navailable refs (or uses a hash table, I guess, but for-each-ref also\ntries to avoid even being linear in the total number of refs, so you'd\nstill want to find the lowest/highest to bound the iteration).\n\n-Peff\n"},{"id":"473561","messageId":"ZBIJKrgI0+Qxd3pG@coredump.intra.peff.net","threadId":"59347","inReplyTo":"f9e80e233f1ca26f07f9db49261e54b7deaf83b8.1678902343.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/8] for-each-ref: add --stdin option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-15T18:06:34Z","receivedAt":"2023-03-15T18:06:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 15, 2023 at 05:45:36PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> @@ -75,7 +79,21 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n> \tfilter.ignore_case = icase;\n> \n> -\tfilter.name_patterns = argv;\n> +\tif (from_stdin) {\n> +\t\tstruct strbuf line = STRBUF_INIT;\n> +\n> +\t\tif (argv[0])\n> +\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n> +\n> +\t\twhile (strbuf_getline(&line, stdin) != EOF)\n> +\t\t\tstrvec_push(&vec, line.buf);\n> +\n> +\t\t/* vec.v is NULL-terminated, just like 'argv'. */\n> +\t\tfilter.name_patterns = vec.v;\n> +\t} else {\n> +\t\tfilter.name_patterns = argv;\n> +\t}\n\nNow that you aren't detaching the \"line\" strbuf in each iteration of the\nloop, it needs to eventually be cleaned up. strbuf_getline() will\n_reset() it, which is good, but at the end we'd need a strbuf_release()\nor it will leak.\n\n-Peff\n"},{"id":"473563","messageId":"xmqqttylomi1.fsf@gitster.g","threadId":"59347","inReplyTo":"ZBIJKrgI0+Qxd3pG@coredump.intra.peff.net","subject":"Re: [PATCH v3 1/8] for-each-ref: add --stdin option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-15T19:14:30Z","receivedAt":"2023-03-15T19:14:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Wed, Mar 15, 2023 at 05:45:36PM +0000, Derrick Stolee via GitGitGadget wrote:\n>\n>> @@ -75,7 +79,21 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n>> \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n>> \tfilter.ignore_case = icase;\n>> \n>> -\tfilter.name_patterns = argv;\n>> +\tif (from_stdin) {\n>> +\t\tstruct strbuf line = STRBUF_INIT;\n>> +\n>> +\t\tif (argv[0])\n>> +\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n>> +\n>> +\t\twhile (strbuf_getline(&line, stdin) != EOF)\n>> +\t\t\tstrvec_push(&vec, line.buf);\n>> +\n>> +\t\t/* vec.v is NULL-terminated, just like 'argv'. */\n>> +\t\tfilter.name_patterns = vec.v;\n>> +\t} else {\n>> +\t\tfilter.name_patterns = argv;\n>> +\t}\n>\n> Now that you aren't detaching the \"line\" strbuf in each iteration of the\n> loop, it needs to eventually be cleaned up. strbuf_getline() will\n> _reset() it, which is good, but at the end we'd need a strbuf_release()\n> or it will leak.\n\nNicely spotted.\n\nThanks.\n"},{"id":"473564","messageId":"xmqqpm99om1p.fsf@gitster.g","threadId":"59347","inReplyTo":"ZBIFEXuE5Um9Z4zM@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-15T19:24:18Z","receivedAt":"2023-03-15T19:25:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> ... we may\n> want some kind of \"literal\" mode, that takes in a list of refs rather\n> than a list of patterns, and does a sorted-merge with the list of\n> available refs (or uses a hash table, I guess, but for-each-ref also\n> tries to avoid even being linear in the total number of refs, so you'd\n> still want to find the lowest/highest to bound the iteration).\n\nExactly.\n\nI actually was wondering if \"literal\" mode can just take a list of\n, and when a is not a refname, use it as if it\nwere. I.e. %(refname) would parrot it, while %(refname:short) would\nnot shorten and still parrot it, if the is 73876f4861c, but\nsomething like %(subject) would still work.\n\nFor that, I suspect ref-filter.c::filter_refs() would need to learn\na different kind fo finter->kind that iterates over the literal\n\"refs\" that was fed from the standard input, instead of calling\nfor_each_fullref_in() for the given hierarchy, but the new iterator\nshould be able to reuse the ref_filter_hander() for the heavy\nlifting.\n\n\n"},{"id":"473565","messageId":"xmqqilf1oll9.fsf@gitster.g","threadId":"59347","inReplyTo":"ZBIA7+85Lld+lpUS@coredump.intra.peff.net","subject":"Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-15T19:34:10Z","receivedAt":"2023-03-15T19:34:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> E.g., imagine resolving \"main\" to 1234abcd in step one, then somebody\n> updates it to 5678cdef, then you run \"for-each-ref\" to compute\n> ahead/behind, and now you show an inconsistent result: you say that\n> \"main\" points to 1234abcd, but show the wrong ahead/behind information.\n>\n> Showing 1234abcd at all is out-of-date, of course, but the real problem\n> is the lack of atomicity. Most porcelain scripts deal with this by\n> resolving the refs immediately, assuming object ids are immutable (which\n> they are modulo games like refs/replace), and then working with them.\n\nA really paranoid caller can use %(ahead-behind-detail:refs/heads/main)\nand get a report on refs/heads/topic, something that conveys\n\n refs/heads/topic (at 67f9f40d) is ahead by 2 commits and behind\n by 4 commits relative to refs/heads/main (at d7c3a768).\n\nin a machine readable form. And when the \"ahead by 2 commits\"\ndisappears, we know 67f9f40d is merged to main sometime before\nd7c3a768. Then it can say \"update-ref -d refs/heads/topic 67f9f40d\"\nto avoid racing with simultanous updaters.\n\n"},{"id":"473566","messageId":"ZBIgOwP5HhR1RJZp@coredump.intra.peff.net","threadId":"59347","inReplyTo":"xmqqpm99om1p.fsf@gitster.g","subject":"Re: [PATCH v2 1/8] for-each-ref: add --stdin option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-15T19:44:59Z","receivedAt":"2023-03-15T19:45:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 15, 2023 at 12:24:18PM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > ... we may\n> > want some kind of \"literal\" mode, that takes in a list of refs rather\n> > than a list of patterns, and does a sorted-merge with the list of\n> > available refs (or uses a hash table, I guess, but for-each-ref also\n> > tries to avoid even being linear in the total number of refs, so you'd\n> > still want to find the lowest/highest to bound the iteration).\n> \n> Exactly.\n> \n> I actually was wondering if \"literal\" mode can just take a list of\n> , and when a is not a refname, use it as if it\n> were. I.e. %(refname) would parrot it, while %(refname:short) would\n> not shorten and still parrot it, if the is 73876f4861c, but\n> something like %(subject) would still work.\n\nYeah, I think that would nicely solve the quadratic issue _and_ the\n\"we are stuck using only ref tips\" issue. I like it.\n\n> For that, I suspect ref-filter.c::filter_refs() would need to learn\n> a different kind fo finter->kind that iterates over the literal\n> \"refs\" that was fed from the standard input, instead of calling\n> for_each_fullref_in() for the given hierarchy, but the new iterator\n> should be able to reuse the ref_filter_hander() for the heavy\n> lifting.\n\nYeah, that sounds about right from my recollection of the code. I\nsuspect there may be other sharp edges (e.g., asking for %(upstream)\nisn't meaningful for a non-ref). But softening those is actually\nsomething I think we want to do in the long run, as it helps with the\nlong-term goal of sharing pretty-printing code between ref-filter,\ncat-file, and pretty.c.\n\n-Peff\n"},{"id":"473569","messageId":"20230315224157.167705-1-jonathantanmy@google.com","threadId":"59347","inReplyTo":"f9e80e233f1ca26f07f9db49261e54b7deaf83b8.1678902343.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/8] for-each-ref: add --stdin option","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-03-15T22:41:57Z","receivedAt":"2023-03-15T22:42:09Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n> @@ -75,7 +79,21 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n> \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n> \tfilter.ignore_case = icase;\n> \n> -\tfilter.name_patterns = argv;\n> +\tif (from_stdin) {\n> +\t\tstruct strbuf line = STRBUF_INIT;\n> +\n> +\t\tif (argv[0])\n> +\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n\nAs a reference point, both fetch and send-pack accept input from stdin\ntoo, and use them to augment what's provided on the CLI (and not only\naccept them when nothing's provided on the CLI).\n\nHaving said that, it's also reasonable to start by prohibiting CLI\narguments when --stdin is given, and we can always relax this later\nif needed.\n"},{"id":"473571","messageId":"20230315224958.169443-1-jonathantanmy@google.com","threadId":"59347","inReplyTo":"3b15e9df770a118331a1b25f51de8ce97c1b7cab.1678902343.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/8] commit-graph: combine generation computations","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-03-15T22:49:58Z","receivedAt":"2023-03-15T22:50:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n> +static void compute_reachable_generation_numbers_1(\n> +\t\t\tstruct compute_generation_info *info,\n> +\t\t\tint generation_version)\n> {\n> \tint i;\n> \tstruct commit_list *list = NULL;\n> \n> -\tif (ctx->report_progress)\n> -\t\tctx->progress = start_delayed_progress(\n> -\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n> -\t\t\t\t\tctx->commits.nr);\n> -\tfor (i = 0; i < ctx->commits.nr; i++) {\n> -\t\tstruct commit *c = ctx->commits.list[i];\n> -\t\tuint32_t level;\n> +\tfor (i = 0; i < info->commits->nr; i++) {\n> +\t\tstruct commit *c = info->commits->list[i];\n> +\t\ttimestamp_t gen;\n> +\t\trepo_parse_commit(info->r, c);\n> +\t\tgen = info->get_generation(c, info->data);\n> \n> -\t\trepo_parse_commit(ctx->r, c);\n> -\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n> +\t\tdisplay_progress(info->progress, info->progress_cnt + 1);\n> \n> -\t\tdisplay_progress(ctx->progress, i + 1);\n> -\t\tif (level != GENERATION_NUMBER_ZERO)\n> +\t\tif (gen != GENERATION_NUMBER_ZERO && gen != GENERATION_NUMBER_INFINITY)\n> \t\t\tcontinue;\n> \n> \t\tcommit_list_insert(c, &list);\n\nSo this replaces a call to display_progress with another...\n\n> \t\t\tif (all_parents_computed) {\n> \t\t\t\tpop_commit(&list);\n> -\n> -\t\t\t\tif (max_level > GENERATION_NUMBER_V1_MAX - 1)\n> -\t\t\t\t\tmax_level = GENERATION_NUMBER_V1_MAX - 1;\n> -\t\t\t\t*topo_level_slab_at(ctx->topo_levels, current) = max_level + 1;\n> +\t\t\t\tgen = compute_generation_from_max(\n> +\t\t\t\t\t\tcurrent, max_gen,\n> +\t\t\t\t\t\tgeneration_version);\n> +\t\t\t\tinfo->set_generation(current, gen, info->data);\n> \t\t\t}\n\n...here is where set_generation is called...\n\n> +static void set_topo_level(struct commit *c, timestamp_t t, void *data)\n> +{\n> +\tstruct write_commit_graph_context *ctx = data;\n> +\t*topo_level_slab_at(ctx->topo_levels, c) = (uint32_t)t;\n> +\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n> +}\n\n...is this display_progress() redundant? (set_topo_level() is one of the\npossibilities that set_generation could be assigned to.) There already\nseems to be one at the top. Further supporting my query is the fact that\nin the hunk containing set_generation, there is no progress report on\nthe LHS of the diff.\n\n> +static void set_generation_v2(struct commit *c, timestamp_t t, void *data)\n> +{\n> +\tstruct write_commit_graph_context *ctx = data;\n> +\tstruct commit_graph_data *g = commit_graph_data_at(c);\n> +\tg->generation = (uint32_t)t;\n> +\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n> +}\n\nLikewise for this function.\n\nEverything else up to and including this patch looks good.\n"},{"id":"473572","messageId":"20230315225815.170437-1-jonathantanmy@google.com","threadId":"59347","inReplyTo":"abd3e7a67beb80ef687253ddb8bb5fe7a769357f.1678902343.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/8] commit-graph: return generation from memory","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-03-15T22:58:15Z","receivedAt":"2023-03-15T22:58:22Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n> From: Derrick Stolee \n> \n> The commit_graph_generation() method used to report a value of\n> GENERATION_NUMBER_INFINITY if the commit_graph_data_slab had an instance\n> for the given commit but the graph_pos indicated the commit was not in\n> the commit-graph file.\n> \n> Instead, trust the 'generation' member if the commit has a value in the\n> slab _and_ the 'generation' member is non-zero. Otherwise, treat it as\n> GENERATION_NUMBER_INFINITY.\n\nI would replace \"Instead\" with \"However, a future commit intends to\ncompute and use commit generation numbers even for commits that are\nnot in the commit-graph file (and thus have no graph_pos). Therefore,\nwe need a new criterion for deciding if a generation number can be\ntrusted:\" (or something to that effect).\n\n> This only makes a difference for a very old case for the commit-graph:\n> the very first Git release to write commit-graph files wrote zeroes in\n> the topological level positions. If we are parsing a commit-graph with\n> all zeroes, those commits will now appear to have\n> GENERATION_NUMBER_INFINITY (as if they were not parsed from the\n> commit-graph).\n> \n> I attempted several variations to work around the need for providing an\n> uninitialized 'generation' member, but this was the best one I found. It\n> does require a change to a verification test in t5318 because it reports\n> a different error than the one about non-zero generation numbers.\n\nThanks for investigating, and I think the method in this patch would\nwork. As you have stated, this only affects the commit-graph files that\nonce upon a time were written with no generation numbers, and this patch\nmakes those behave as if there were no generation numbers in the first\nplace (which is exactly what happened).\n\n"},{"id":"473575","messageId":"20230315232835.173584-1-jonathantanmy@google.com","threadId":"59347","inReplyTo":"0fb3913810b7d4731707a0129df7f5d57c8ab39f.1678902343.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/8] commit-reach: implement ahead_behind() logic","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-03-15T23:28:35Z","receivedAt":"2023-03-15T23:28:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"First of all, thanks to Taylor and Stolee for this algorithm and code\n- it is straightforwardly written and looks correct to me. I have some\ncommit message and code comment suggestions that if taken, would have\nhelped me on my first reading, but these are subjective so feel free\nto ignore them if you think they would add unnecessary detail (I did\nunderstand the algorithm in the end, after all).\n\n\"Derrick Stolee via GitGitGadget\" writes:\n> The second array, using the\n> new ahead_behind_count struct, indicates which commits from that initial\n> array form the base/tip pair for the ahead/behind count it will store.\n\nI would have preferred: The second array contains base/tip pairs\ndesignating pairs of commits for which ahead/behind counts need to be\ncomputed, each pair being a pair of indexes into the first array.\n\n> Each commit in the input commit list is associated with a bit position\n> indicating \"the ith commit can reach this commit\". Each of these commits\n> is associated with a bitmap with its position flipped on and then\n> placed in a queue for walking commit history. \n\n\"this commit\" is not necessarily a commit in the input commit list (it\nis actually the commit that we're currently at in our iteration) and I\nthink that the association of bitmaps with commits in the input commit\nlist could be more clearly described. So I would have preferred: Each\ncommit in the priority queue is associated with a bitmap of width N\n(N being the count of commits in the first array), in which a bit is\nset iff the commit can be reached by the corresponding commit in the\nfirst array. This is different from packfile or MIDX bitmaps in that\na commit's bitmap stores what can reach it, not what it can reach.\nThe priority queue is initialized with N commits, each commit being\nassociated with a bitmap in which a single bit is set (indicating that\nthe commit can be reached by itself).\n\n> +void ahead_behind(struct repository *r,\n> +\t\t struct commit **commits, size_t commits_nr,\n> +\t\t struct ahead_behind_count *counts, size_t counts_nr)\n> +{\n> +\tstruct prio_queue queue = { .compare = compare_commits_by_gen_then_commit_date };\n> +\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n\nAs we discussed in our Review Club, DIV_ROUND_UP can be used for this.\n\n(For those reading who do not know what Review Club is, search the\narchives and/or look out for future announcements!)\n\n> +\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n> +\t\t\t\tp->item->object.flags |= STALE;\n\nMight be worth adding a comment above the STALE line: this parent commit\nand all its ancestors can be reached by every commit in the commits\nlist and thus can never be \"ahead\" or \"behind\" in any pair; mark this\nSTALE so that, as an optimization, we can stop iteration if only STALE\ncommits remain (since further iteration would never change any \"ahead\"\nor \"behind\" value).\n\n"},{"id":"473658","messageId":"d4ac5ab2-e5e5-e894-04e0-b99249dfeced@github.com","threadId":"59347","inReplyTo":"20230315224958.169443-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 3/8] commit-graph: combine generation computations","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-17T18:30:35Z","receivedAt":"2023-03-17T18:30:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 6:49 PM, Jonathan Tan wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>> +static void compute_reachable_generation_numbers_1(\n>> +\t\t\tstruct compute_generation_info *info,\n>> +\t\t\tint generation_version)\n>> {\n>> \tint i;\n>> \tstruct commit_list *list = NULL;\n>> \n>> -\tif (ctx->report_progress)\n>> -\t\tctx->progress = start_delayed_progress(\n>> -\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n>> -\t\t\t\t\tctx->commits.nr);\n>> -\tfor (i = 0; i < ctx->commits.nr; i++) {\n>> -\t\tstruct commit *c = ctx->commits.list[i];\n>> -\t\tuint32_t level;\n>> +\tfor (i = 0; i < info->commits->nr; i++) {\n>> +\t\tstruct commit *c = info->commits->list[i];\n>> +\t\ttimestamp_t gen;\n>> +\t\trepo_parse_commit(info->r, c);\n>> +\t\tgen = info->get_generation(c, info->data);\n>> \n>> -\t\trepo_parse_commit(ctx->r, c);\n>> -\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n>> +\t\tdisplay_progress(info->progress, info->progress_cnt + 1);\n>> \n>> -\t\tdisplay_progress(ctx->progress, i + 1);\n>> -\t\tif (level != GENERATION_NUMBER_ZERO)\n>> +\t\tif (gen != GENERATION_NUMBER_ZERO && gen != GENERATION_NUMBER_INFINITY)\n>> \t\t\tcontinue;\n>> \n>> \t\tcommit_list_insert(c, &list);\n> \n> So this replaces a call to display_progress with another...\n> \n>> \t\t\tif (all_parents_computed) {\n>> \t\t\t\tpop_commit(&list);\n>> -\n>> -\t\t\t\tif (max_level > GENERATION_NUMBER_V1_MAX - 1)\n>> -\t\t\t\t\tmax_level = GENERATION_NUMBER_V1_MAX - 1;\n>> -\t\t\t\t*topo_level_slab_at(ctx->topo_levels, current) = max_level + 1;\n>> +\t\t\t\tgen = compute_generation_from_max(\n>> +\t\t\t\t\t\tcurrent, max_gen,\n>> +\t\t\t\t\t\tgeneration_version);\n>> +\t\t\t\tinfo->set_generation(current, gen, info->data);\n>> \t\t\t}\n> \n> ...here is where set_generation is called...\n> \n>> +static void set_topo_level(struct commit *c, timestamp_t t, void *data)\n>> +{\n>> +\tstruct write_commit_graph_context *ctx = data;\n>> +\t*topo_level_slab_at(ctx->topo_levels, c) = (uint32_t)t;\n>> +\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n>> +}\n> \n> ...is this display_progress() redundant? (set_topo_level() is one of the\n> possibilities that set_generation could be assigned to.) There already\n> seems to be one at the top. Further supporting my query is the fact that\n> in the hunk containing set_generation, there is no progress report on\n> the LHS of the diff.\n\nIt turns out the progress is a bit redundant here, but not entirely in\nthe case of ensure_generations_valid() (if progress was enabled).\n\nLet's break down the iteration, which has nested loops:\n\n 1. for all commits in the initial list.\n 2. perform DFS until generation can be computed. (while loop)\n\nWhen writing a commit-graph file, that initial list is _every commit\nin the commit-graph_, so having a display_progress() in the for loop\nis sufficient to get the exact number.\n\nIn the case of ensure_generations_valid(), the number of assignments\nin the while loop can be much larger than the initial input list.\n\nHowever, ensure_generations_valid() does not use progress _and_ even\nif it did, it would make sense to signal progress based on the number\nof tips that need to be computed. I'll remove these progress counts\ninside the mutators.\n\nThanks,\n-Stolee\n"},{"id":"473659","messageId":"9bd7943c-a8a3-5022-bd7e-f95be8dc2ab5@github.com","threadId":"59347","inReplyTo":"20230315232835.173584-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 6/8] commit-reach: implement ahead_behind() logic","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-03-17T18:44:12Z","receivedAt":"2023-03-17T18:44:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/15/2023 7:28 PM, Jonathan Tan wrote:\n> First of all, thanks to Taylor and Stolee for this algorithm and code\n> - it is straightforwardly written and looks correct to me. I have some\n> commit message and code comment suggestions that if taken, would have\n> helped me on my first reading, but these are subjective so feel free\n> to ignore them if you think they would add unnecessary detail (I did\n> understand the algorithm in the end, after all).\n\nI appreciate your comments here. I've done some reworking of the\nmessage based on what you say here, as well as the verbal feedback from\nreview club.\n\n>> +void ahead_behind(struct repository *r,\n>> +\t\t struct commit **commits, size_t commits_nr,\n>> +\t\t struct ahead_behind_count *counts, size_t counts_nr)\n>> +{\n>> +\tstruct prio_queue queue = { .compare = compare_commits_by_gen_then_commit_date };\n>> +\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n> \n> As we discussed in our Review Club, DIV_ROUND_UP can be used for this.\n\nGot it!\n\n>> +\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n>> +\t\t\t\tp->item->object.flags |= STALE;\n> \n> Might be worth adding a comment above the STALE line: this parent commit\n> and all its ancestors can be reached by every commit in the commits\n> list and thus can never be \"ahead\" or \"behind\" in any pair; mark this\n> STALE so that, as an optimization, we can stop iteration if only STALE\n> commits remain (since further iteration would never change any \"ahead\"\n> or \"behind\" value).\n\nThis is a helpful thing to point out, so a comment is appropriate.\n\nOverall, maybe algorithms like this should have more inline comments\nthan we typically expect in the Git codebase. We want to make sure that\nthese things are readable in the future, hopefully without digging too\nfar in the history to find the lengthy commit message about it.\n\nI'll delay sending v4 until giving a little time to hear back on this\npoint. My default is to not add the comments, but I'd be happy to, if\nwe think this is an appropriate time to deviate from the standard.\n\nThanks,\n-Stolee\n"},{"id":"473752","messageId":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v3.git.1678902343.gitgitgadget@gmail.com","subject":"[PATCH v4 0/9] ref-filter: ahead/behind counting, faster --merged option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:46Z","receivedAt":"2023-03-20T11:27:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"At $DAYJOB, we have used a custom 'ahead-behind' builtin in our fork of Git\nfor lots of reasons. The main goal of the builtin is to compare multiple\nreferences against a common base reference. The comparison is number of\ncommits that are in each side of the symmtric difference of their reachable\nsets. A commit C is \"ahead\" of a commit B by the number of commits in B..C\n(reachable from C but not reachable from B). Similarly, the commit C is\n\"behind\" the commit B by the number of commits in C..B (reachable from B but\nnot reachable from C).\n\nThese numbers can be computed by 'git rev-list --count B..C' and 'git\nrev-list --count C..B', but there are common needs that benefit from having\nthe checks being done in the same process:\n\n 1. Our \"branches\" page lists ahead/behind counts for each listed branch as\n compared to the repo's default branch. This can be done with a single\n 'git ahead-behind' process.\n 2. When a branch is updated, a background job checks if any pull requests\n that target that branch should be closed because their branches were\n merged implicitly by that update. These queries can be batched into 'git\n ahead-behind' calls.\n\nIn that second example, we don't need the full ahead/behind counts (although\nit is sufficient to look for branches that are \"zero commits ahead\", meaning\nthey are reachable from the base), and instead reachability is the critical\npiece.\n\nThis series contributes the custom algorithms we used for our 'git\nahead-behind' builtin, but as extensions to 'git for-each-ref':\n\n * Add a new \"%(ahead-behind:)\" format token to for-each-ref which\n allows outputting the ahead/behind values in the format string for a\n matching ref.\n * Add a new algorithm that speeds up the 'git for-each-ref --merged='\n option. This also applies to the 'git branch --merged=' option.\n\nThe idea to use 'git for-each-ref' instead of creating a new builtin is from\nJunio, and simplifies this series significantly compared to v1. I was\ninitially concerned about the overhead of 'git for-each-ref' and its\ngenerality and sorting, but I was not able to measure any important\ndifference between this implementation and our internal 'git ahead-behind'\nimplementation. In particular, when a pattern is given to 'git for-each-ref'\nthat looks like an exact ref, it navigates directly to the ref instead of\nscanning all references for matches.\n\nHowever, for our specific uses, we like to batch a list of exact references\nthat could be very long. We introduce a new --stdin option here.\n\nTo keep things close to the v1 outline, I replaced the existing patches with\nclosely-related ones, when possible.\n\nPatch 1 adds the --stdin option to 'git for-each-ref'. (This is similar to\nthe boilerplate patch from v1.)\n\nPatch 2 adds a test to explicitly check that 'git for-each-ref' will still\nsucceed when all input refs are missing. (This is similar to the\n--ignore-missing patch from v1.)\n\nPatches 3-6 introduce a new method: ensure_generations_valid(). Patch 3\nrefactors the existing compute_topological_levels() to make it more generic\nwhile Patch 4 ports compute_generation_numbers() onto that generic base.\nPatch 5 updates the definition of commit_graph_generation() slightly, making\nway for patch 6 to implement the in-memory computation of generation\nnumbers. With an existing commit-graph file, the commits that are not\npresent in the file are considered as having generation number \"infinity\".\nThis is useful for most of our reachability queries to this point, since\nthose commits are \"above\" the ones tracked by the commit-graph. When these\ncommits are low in number, then there is very little performance cost and\nzero correctness cost. (These patches match v1 exactly.)\n\nHowever, we will see that the ahead/behind computation requires accurate\ngeneration numbers to avoid overcounting. Thus, ensure_generations_valid()\nisa way to specify a list of commits that need generation numbers computed\nbefore continuing. It's a no-op if all of those commits are in the\ncommit-graph file. It's expensive if the commit-graph doesn't exist.\nHowever, '%(ahead-behind:)' computations are likely to be slow no matter\nwhat without a commit-graph, so assuming an existing commit-graph file is\nreasonable. If we find sufficient desire to have an implementation that does\nnot have this requirement, we could create a second implementation and\ntoggle to it when generation_numbers_enabled() returns false.\n\nPatch 7 implements the ahead-behind algorithm, but it is not connected to a\nbuiltin. It's a long commit message, so hopefully it explains the algorithm\nsufficiently. (The difference from v1 is that it no longer integrates with a\nbuiltin and there are no new tests. It also uses 'unsigned int' and is\ncorrectly co-authored by Taylor.)\n\nPatch 8 integrates the ahead-behind algorithm with the ref-filter code,\nincluding parsing the \"ahead-behind\" token. This finally adds tests that\ncheck both ahead_behind() and ensure_generations_valid() via\nt6600-test-reach.sh. (This patch is essentially completely new in v2.)\n\nPatch 9 implements the tips_reachable_from_base() method, and uses it within\nthe ref-filter code to speed up 'git for-each-ref --merged' and 'git branch\n--merged'. (The interface is slightly different than v1, due to the needs of\nthe new caller.)\n\n\nUpdates in v4\n=============\n\n * A string leak in Patch 1 is remedied.\n * v3's patch 3 is split into v4's patches 3 and 4. Patch 3 now only\n refactors compute_topological_levels(), leaving Patch 4 to only port\n compute_generation_numbers() to the new\n compute_reachable_generation_numbers().\n * Other edits in Patch 3:\n * compute_reachable_generation_numbers() drops the \"_1\" in its name.\n * Redundant progress counters are removed.\n * The progress struct is passed down into 'struct\n compute_generation_info' and not just the 'struct\n write_commit_graph_context'\n * Patch 6 adds context around why we care about in-memory generation\n number.\n * Patch 7 has several changes:\n * Includes extra context around the input data to ahead_behind() as well\n as some rewrite of the algorithm description.\n * uses DIV_ROUND_UP() to initialize the bitmap width.\n * renames init_bit_array() to get_bit_array() to make it clear that it\n will not regenerate the bitmap if it already exists.\n * adds a new comment before assigning the STALE bit\n * Patch 8 changes the expected format to output and checks the\n given base during the token parsing instead of delaying.\n\n\nUpdates in v3\n=============\n\n * The APIs are modified to take a 'struct repository *' and use them\n appropriately.\n * The --stdin option in 'git for-each-ref' now uses strvec instead of an\n ad-hoc array.\n\nThanks, -Stolee\n\nDerrick Stolee (8):\n for-each-ref: add --stdin option\n for-each-ref: explicitly test no matches\n commit-graph: refactor compute_topological_levels()\n commit-graph: simplify compute_generation_numbers()\n commit-graph: return generation from memory\n commit-reach: implement ahead_behind() logic\n for-each-ref: add ahead-behind format atom\n commit-reach: add tips_reachable_from_bases()\n\nTaylor Blau (1):\n commit-graph: introduce `ensure_generations_valid()`\n\n Documentation/git-for-each-ref.txt | 12 +-\n builtin/branch.c | 1 +\n builtin/for-each-ref.c | 26 +++-\n builtin/tag.c | 1 +\n commit-graph.c | 207 +++++++++++++++++----------\n commit-graph.h | 8 ++\n commit-reach.c | 216 +++++++++++++++++++++++++++++\n commit-reach.h | 40 ++++++\n ref-filter.c | 93 ++++++++++---\n ref-filter.h | 26 +++-\n t/perf/p1500-graph-walks.sh | 50 +++++++\n t/t3203-branch-output.sh | 14 ++\n t/t5318-commit-graph.sh | 2 +-\n t/t6300-for-each-ref.sh | 50 +++++++\n t/t6301-for-each-ref-errors.sh | 14 ++\n t/t6600-test-reach.sh | 169 ++++++++++++++++++++++\n t/t7004-tag.sh | 28 ++++\n 17 files changed, 866 insertions(+), 91 deletions(-)\n create mode 100755 t/perf/p1500-graph-walks.sh\n\n\nbase-commit: 725f57037d81e24eacfda6e59a19c60c0b4c8062\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1489%2Fderrickstolee%2Fstolee%2Fupstream-ahead-behind-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1489/derrickstolee/stolee/upstream-ahead-behind-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1489\n\nRange-diff vs v3:\n\n 1: f9e80e233f1 ! 1: 27d94077aa9 for-each-ref: add --stdin option\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc, const char **argv, const\n +\t\twhile (strbuf_getline(&line, stdin) != EOF)\n +\t\t\tstrvec_push(&vec, line.buf);\n +\n ++\t\tstrbuf_release(&line);\n ++\n +\t\t/* vec.v is NULL-terminated, just like 'argv'. */\n +\t\tfilter.name_patterns = vec.v;\n +\t} else {\n 2: f56d6a64d24 = 2: 1e3d499431a for-each-ref: explicitly test no matches\n 3: 3b15e9df770 ! 3: 79a57f30a85 commit-graph: combine generation computations\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - commit-graph: combine generation computations\n + commit-graph: refactor compute_topological_levels()\n \n This patch extracts the common code used to compute topological levels\n and corrected committer dates into a common routine,\n - compute_reachable_generation_numbers_1().\n + compute_reachable_generation_numbers(). For ease of reading, it only\n + modifies compute_topological_levels() to use this new routine, leaving\n + compute_generation_numbers() to be modified in the next change.\n \n This new routine dispatches to call the necessary functions to get and\n set the generation number for a given commit through a vtable (the\n @@ Commit message\n Computing the generation number itself is done in\n compute_generation_from_max(), which dispatches its implementation based\n on the generation version requested, or issuing a BUG() for unrecognized\n - generation versions.\n + generation versions. This does not use a vtable because the logic\n + depends only on the generation number version, not where the data is\n + being loaded from or being stored to. This is a subtle point that will\n + make more sense in a future change that modifies the in-memory\n + generation values instead of just preparing values for writing to a\n + commit-graph file.\n \n - This patch cleans up the two places that currently compute topological\n - levels and corrected commit dates by reducing the amount of duplicated\n - code. It also makes it possible to introduce a function which\n - dynamically computes those values for commits that aren't stored in a\n - commit-graph, which will be required for the forthcoming ahead-behind\n - rewrite.\n + This change looks like it adds a lot of new code. However, two upcoming\n + changes will be quite small due to the work being done in this change.\n \n + Co-authored-by: Taylor Blau \n Signed-off-by: Taylor Blau \n Signed-off-by: Derrick Stolee \n \n @@ commit-graph.c: static void close_reachable(struct write_commit_graph_context *c\n +\t}\n +}\n +\n -+static void compute_reachable_generation_numbers_1(\n ++static void compute_reachable_generation_numbers(\n +\t\t\tstruct compute_generation_info *info,\n +\t\t\tint generation_version)\n {\n @@ commit-graph.c: static void close_reachable(struct write_commit_graph_context *c\n -\tfor (i = 0; i < ctx->commits.nr; i++) {\n -\t\tstruct commit *c = ctx->commits.list[i];\n -\t\tuint32_t level;\n +-\n +-\t\trepo_parse_commit(ctx->r, c);\n +-\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n +\tfor (i = 0; i < info->commits->nr; i++) {\n +\t\tstruct commit *c = info->commits->list[i];\n +\t\ttimestamp_t gen;\n +\t\trepo_parse_commit(info->r, c);\n +\t\tgen = info->get_generation(c, info->data);\n - \n --\t\trepo_parse_commit(ctx->r, c);\n --\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n +\t\tdisplay_progress(info->progress, info->progress_cnt + 1);\n \n -\t\tdisplay_progress(ctx->progress, i + 1);\n @@ commit-graph.c: static void compute_topological_levels(struct write_commit_graph\n +{\n +\tstruct write_commit_graph_context *ctx = data;\n +\t*topo_level_slab_at(ctx->topo_levels, c) = (uint32_t)t;\n -+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n +}\n +\n +static void compute_topological_levels(struct write_commit_graph_context *ctx)\n +{\n +\tstruct compute_generation_info info = {\n +\t\t.r = ctx->r,\n -+\t\t.progress = ctx->progress,\n +\t\t.commits = &ctx->commits,\n +\t\t.get_generation = get_topo_level,\n +\t\t.set_generation = set_topo_level,\n @@ commit-graph.c: static void compute_topological_levels(struct write_commit_graph\n +\t};\n +\n +\tif (ctx->report_progress)\n -+\t\tctx->progress = start_delayed_progress(\n ++\t\tinfo.progress = ctx->progress\n ++\t\t\t = start_delayed_progress(\n +\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n +\t\t\t\t\tctx->commits.nr);\n +\n -+\tcompute_reachable_generation_numbers_1(&info, 1);\n -+\n - \tstop_progress(&ctx->progress);\n - }\n - \n -+static timestamp_t get_generation_from_graph_data(struct commit *c, void *data)\n -+{\n -+\treturn commit_graph_data_at(c)->generation;\n -+}\n -+\n -+static void set_generation_v2(struct commit *c, timestamp_t t, void *data)\n -+{\n -+\tstruct write_commit_graph_context *ctx = data;\n -+\tstruct commit_graph_data *g = commit_graph_data_at(c);\n -+\tg->generation = (uint32_t)t;\n -+\tdisplay_progress(ctx->progress, ctx->progress_cnt + 1);\n -+}\n -+\n - static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n - {\n - \tint i;\n --\tstruct commit_list *list = NULL;\n -+\tstruct compute_generation_info info = {\n -+\t\t.r = ctx->r,\n -+\t\t.progress = ctx->progress,\n -+\t\t.commits = &ctx->commits,\n -+\t\t.get_generation = get_generation_from_graph_data,\n -+\t\t.set_generation = set_generation_v2,\n -+\t\t.data = ctx,\n -+\t};\n - \n - \tif (ctx->report_progress)\n - \t\tctx->progress = start_delayed_progress(\n -@@ commit-graph.c: static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n - \t\t}\n - \t}\n - \n --\tfor (i = 0; i < ctx->commits.nr; i++) {\n --\t\tstruct commit *c = ctx->commits.list[i];\n --\t\ttimestamp_t corrected_commit_date;\n --\n --\t\trepo_parse_commit(ctx->r, c);\n --\t\tcorrected_commit_date = commit_graph_data_at(c)->generation;\n --\n --\t\tdisplay_progress(ctx->progress, i + 1);\n --\t\tif (corrected_commit_date != GENERATION_NUMBER_ZERO)\n --\t\t\tcontinue;\n --\n --\t\tcommit_list_insert(c, &list);\n --\t\twhile (list) {\n --\t\t\tstruct commit *current = list->item;\n --\t\t\tstruct commit_list *parent;\n --\t\t\tint all_parents_computed = 1;\n --\t\t\ttimestamp_t max_corrected_commit_date = 0;\n --\n --\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n --\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n --\t\t\t\tcorrected_commit_date = commit_graph_data_at(parent->item)->generation;\n --\n --\t\t\t\tif (corrected_commit_date == GENERATION_NUMBER_ZERO) {\n --\t\t\t\t\tall_parents_computed = 0;\n --\t\t\t\t\tcommit_list_insert(parent->item, &list);\n --\t\t\t\t\tbreak;\n --\t\t\t\t}\n --\n --\t\t\t\tif (corrected_commit_date > max_corrected_commit_date)\n --\t\t\t\t\tmax_corrected_commit_date = corrected_commit_date;\n --\t\t\t}\n --\n --\t\t\tif (all_parents_computed) {\n --\t\t\t\tpop_commit(&list);\n --\n --\t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n --\t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n --\t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n --\t\t\t}\n --\t\t}\n --\t}\n -+\tcompute_reachable_generation_numbers_1(&info, 2);\n - \n - \tfor (i = 0; i < ctx->commits.nr; i++) {\n - \t\tstruct commit *c = ctx->commits.list[i];\n -@@ commit-graph.c: static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n - \t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n - \t\t\tctx->num_generation_data_overflows++;\n - \t}\n ++\tcompute_reachable_generation_numbers(&info, 1);\n +\n \tstop_progress(&ctx->progress);\n }\n -: ----------- > 4: 3fd6c758129 commit-graph: simplify compute_generation_numbers()\n 4: abd3e7a67be ! 5: fed76f0f08e commit-graph: return generation from memory\n @@ Commit message\n for the given commit but the graph_pos indicated the commit was not in\n the commit-graph file.\n \n + However, an upcoming change will introduce the ability to set generation\n + values in-memory without writing the commit-graph file. Thus, we can no\n + longer trust 'graph_pos' to indicate whether or not the generation\n + member can be trusted.\n +\n Instead, trust the 'generation' member if the commit has a value in the\n slab _and_ the 'generation' member is non-zero. Otherwise, treat it as\n GENERATION_NUMBER_INFINITY.\n 5: e197bddcace ! 6: 17a1fc9b15e commit-graph: introduce `ensure_generations_valid()`\n @@ commit-graph.c: static void compute_generation_numbers(struct write_commit_graph\n +\t\t.set_generation = set_generation_in_graph_data,\n +\t};\n +\n -+\tcompute_reachable_generation_numbers_1(&info, generation_version);\n ++\tcompute_reachable_generation_numbers(&info, generation_version);\n +}\n +\n static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n 6: 0fb3913810b ! 7: 5d937184a0e commit-reach: implement ahead_behind() logic\n @@ Commit message\n \n The interface for ahead_behind() uses two arrays. The first array of\n commits contains the list of all starting points for the walk. This\n - includes all tip commits _and_ base commits. The second array, using the\n - new ahead_behind_count struct, indicates which commits from that initial\n - array form the base/tip pair for the ahead/behind count it will store.\n + includes all tip commits _and_ base commits. The second array specifies\n + base/tip pairs by pointing to commits within the first array, by index.\n + The second array also stores the resulting ahead/behind counts for each\n + of these pairs.\n \n This implementation of ahead_behind() allows multiple bases, if desired.\n Even with multiple bases, there is only one commit walk used for\n @@ Commit message\n \n Now, let's discuss the ahead/behind counting algorithm.\n \n - Each commit in the input commit list is associated with a bit position\n - indicating \"the ith commit can reach this commit\". Each of these commits\n - is associated with a bitmap with its position flipped on and then\n - placed in a queue for walking commit history. We walk commits by popping\n - the commit with maximum generation number out of the queue, guaranteeing\n - that we will never walk a child of that commit in any future steps.\n + The first array of commits are considered the starting commits. The\n + index within that array will play a critical role.\n +\n + We create a new commit slab that maps commits to a bitmap. For a given\n + commit (anywhere in the history), its bitmap stores information relative\n + to which of the input commits can reach that commit. The ith bit will be\n + on if the ith commit from the starting list can reach that commit. It is\n + important to notice that these bitmaps are not the typical \"reachability\n + bitmaps\" that are stored in .bitmap files. Instead of signalling which\n + objects are reachable from the current commit, they instead signal\n + \"which starting commits can reach me?\" It is also important to know that\n + the bitmap is not necessarily \"complete\" until we walk that commit. We\n + will perform a commit walk by generation number in such a way that we\n + can guarantee the bitmap is correct when we visit that commit.\n +\n + At the beginning of the ahead_behind() method, we initialize the bitmaps\n + for each of the starting commits. By enabling the ith bit for the ith\n + starting commit, we signal \"the ith commit can reach itself.\"\n +\n + We walk commits by popping the commit with maximum generation number out\n + of the queue, guaranteeing that we will never walk a child of that\n + commit in any future steps.\n \n As we walk, we load the bitmap for the current commit and perform two\n main steps. The _second_ step examines each parent of the current commit\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\tc->object.flags |= PARENT2;\n +}\n +\n -+static struct bitmap *init_bit_array(struct commit *c, int width)\n ++static struct bitmap *get_bit_array(struct commit *c, int width)\n +{\n +\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n +\tif (!*bitmap)\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\t\t struct ahead_behind_count *counts, size_t counts_nr)\n +{\n +\tstruct prio_queue queue = { .compare = compare_commits_by_gen_then_commit_date };\n -+\tsize_t width = (commits_nr + BITS_IN_EWORD - 1) / BITS_IN_EWORD;\n ++\tsize_t width = DIV_ROUND_UP(commits_nr, BITS_IN_EWORD);\n +\n +\tif (!commits_nr || !counts_nr)\n +\t\treturn;\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\n +\tfor (size_t i = 0; i < commits_nr; i++) {\n +\t\tstruct commit *c = commits[i];\n -+\t\tstruct bitmap *bitmap = init_bit_array(c, width);\n ++\t\tstruct bitmap *bitmap = get_bit_array(c, width);\n +\n +\t\tbitmap_set(bitmap, i);\n +\t\tinsert_no_dup(&queue, c);\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\twhile (queue_has_nonstale(&queue)) {\n +\t\tstruct commit *c = prio_queue_get(&queue);\n +\t\tstruct commit_list *p;\n -+\t\tstruct bitmap *bitmap_c = init_bit_array(c, width);\n ++\t\tstruct bitmap *bitmap_c = get_bit_array(c, width);\n +\n +\t\tfor (size_t i = 0; i < counts_nr; i++) {\n +\t\t\tint reach_from_tip = !!bitmap_get(bitmap_c, counts[i].tip_index);\n @@ commit-reach.c: struct commit_list *get_reachable_subset(struct commit **from, i\n +\n +\t\t\trepo_parse_commit(r, p->item);\n +\n -+\t\t\tbitmap_p = init_bit_array(p->item, width);\n ++\t\t\tbitmap_p = get_bit_array(p->item, width);\n +\t\t\tbitmap_or(bitmap_p, bitmap_c);\n +\n ++\t\t\t/*\n ++\t\t\t * If this parent is reachable from every starting\n ++\t\t\t * commit, then none of its ancestors can contribute\n ++\t\t\t * to the ahead/behind count. Mark it as STALE, so\n ++\t\t\t * we can stop the walk when every commit in the\n ++\t\t\t * queue is STALE.\n ++\t\t\t */\n +\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n +\t\t\t\tp->item->object.flags |= STALE;\n +\n 7: 59cf6759e60 ! 8: b8523e2be0b for-each-ref: add ahead-behind format atom\n @@ ref-filter.c: static int rest_atom_parser(struct ref_format *format, struct used\n +static int ahead_behind_atom_parser(struct ref_format *format, struct used_atom *atom,\n +\t\t\t\t const char *arg, struct strbuf *err)\n +{\n ++\tstruct string_list_item *item;\n ++\n +\tif (!arg)\n -+\t\treturn strbuf_addf_ret(err, -1, _(\"expected format: %%(ahead-behind:)\"));\n ++\t\treturn strbuf_addf_ret(err, -1, _(\"expected format: %%(ahead-behind:)\"));\n ++\n ++\titem = string_list_append(&format->bases, arg);\n ++\titem->util = lookup_commit_reference_by_name(arg);\n ++\tif (!item->util)\n ++\t\tdie(\"failed to find '%s'\", arg);\n +\n -+\tstring_list_append(&format->bases, arg);\n +\treturn 0;\n +}\n +\n @@ ref-filter.c: static void reach_filter(struct ref_array *array,\n +\t\treturn;\n +\n +\tALLOC_ARRAY(commits, commits_nr);\n -+\tfor (size_t i = 0; i < format->bases.nr; i++) {\n -+\t\tconst char *name = format->bases.items[i].string;\n -+\t\tcommits[i] = lookup_commit_reference_by_name(name);\n -+\t\tif (!commits[i])\n -+\t\t\tdie(\"failed to find '%s'\", name);\n -+\t}\n ++\tfor (size_t i = 0; i < format->bases.nr; i++)\n ++\t\tcommits[i] = format->bases.items[i].util;\n +\n +\tALLOC_ARRAY(array->counts, st_mult(format->bases.nr, array->nr));\n +\n @@ t/t6301-for-each-ref-errors.sh: test_expect_success 'Missing objects are reporte\n +test_expect_success 'ahead-behind requires an argument' '\n +\ttest_must_fail git for-each-ref \\\n +\t\t--format=\"%(ahead-behind)\" 2>err &&\n -+\techo \"fatal: expected format: %(ahead-behind:)\" >expect &&\n ++\techo \"fatal: expected format: %(ahead-behind:)\" >expect &&\n +\ttest_cmp expect err\n +'\n +\n 8: 7476a39331e = 9: 87fe9676aec commit-reach: add tips_reachable_from_bases()\n\n-- \ngitgitgadget\n"},{"id":"473753","messageId":"27d94077aa98c7ea76787ac8c2a8fd15915c9eab.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 1/9] for-each-ref: add --stdin option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:47Z","receivedAt":"2023-03-20T11:27:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a user wishes to input a large list of patterns to 'git\nfor-each-ref' (likely a long list of exact refs) there are frequently\nsystem limits on the number of command-line arguments.\n\nAdd a new --stdin option to instead read the patterns from standard\ninput. Add tests that check that any unrecognized arguments are\nconsidered an error when --stdin is provided. Also, an empty pattern\nlist is interpreted as the complete ref set.\n\nWhen reading from stdin, we populate the filter.name_patterns array\ndynamically as opposed to pointing to the 'argv' array directly. This is\nsimple when using a strvec, as it is NULL-terminated in the same way. We\nthen free the memory directly from the strvec.\n\nHelped-by: Phillip Wood \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-for-each-ref.txt | 7 +++++-\n builtin/for-each-ref.c | 23 ++++++++++++++++++-\n t/t6300-for-each-ref.sh | 37 ++++++++++++++++++++++++++++++\n 3 files changed, 65 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex 6da899c6296..ccdc2911bb9 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -9,7 +9,8 @@ SYNOPSIS\n --------\n [verse]\n 'git for-each-ref' [--count=] [--shell|--perl|--python|--tcl]\n-\t\t [(--sort=)...] [--format=] [...]\n+\t\t [(--sort=)...] [--format=]\n+\t\t [ --stdin | ... ]\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n@@ -32,6 +33,10 @@ OPTIONS\n \tliterally, in the latter case matching completely or from the\n \tbeginning up to a slash.\n \n+--stdin::\n+\tIf `--stdin` is supplied, then the list of patterns is read from\n+\tstandard input instead of from the argument list.\n+\n --count=::\n \tBy default the command shows all refs that match\n \t``. This option makes it stop after showing\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 6f62f40d126..9df16cfb854 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -5,6 +5,7 @@\n #include \"object.h\"\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n+#include \"strvec.h\"\n \n static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n@@ -25,6 +26,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tstruct ref_format format = REF_FORMAT_INIT;\n \tstruct strbuf output = STRBUF_INIT;\n \tstruct strbuf err = STRBUF_INIT;\n+\tint from_stdin = 0;\n+\tstruct strvec vec = STRVEC_INIT;\n \n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n@@ -49,6 +52,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \t\tOPT_CONTAINS(&filter.with_commit, N_(\"print only refs which contain the commit\")),\n \t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n \t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n+\t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n \t\tOPT_END(),\n \t};\n \n@@ -75,7 +79,23 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n \tfilter.ignore_case = icase;\n \n-\tfilter.name_patterns = argv;\n+\tif (from_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\tif (argv[0])\n+\t\t\tdie(_(\"unknown arguments supplied with --stdin\"));\n+\n+\t\twhile (strbuf_getline(&line, stdin) != EOF)\n+\t\t\tstrvec_push(&vec, line.buf);\n+\n+\t\tstrbuf_release(&line);\n+\n+\t\t/* vec.v is NULL-terminated, just like 'argv'. */\n+\t\tfilter.name_patterns = vec.v;\n+\t} else {\n+\t\tfilter.name_patterns = argv;\n+\t}\n+\n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n \tref_array_sort(sorting, &array);\n@@ -97,5 +117,6 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \tfree_commit_list(filter.with_commit);\n \tfree_commit_list(filter.no_commit);\n \tref_sorting_release(sorting);\n+\tstrvec_clear(&vec);\n \treturn 0;\n }\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex c466fd989f1..a58053a54c5 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1464,4 +1464,41 @@ sig_crlf=\"$(printf \"%s\" \"$sig\" | append_cr; echo dummy)\"\n sig_crlf=${sig_crlf%dummy}\n test_atom refs/tags/fake-sig-crlf contents:signature \"$sig_crlf\"\n \n+test_expect_success 'git for-each-ref --stdin: empty' '\n+\t>in &&\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\tgit for-each-ref --format=\"%(refname)\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git for-each-ref --stdin: fails if extra args' '\n+\t>in &&\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" \\\n+\t\t--stdin refs/heads/extra err &&\n+\tgrep \"unknown arguments supplied with --stdin\" err\n+'\n+\n+test_expect_success 'git for-each-ref --stdin: matches' '\n+\tcat >in <<-EOF &&\n+\trefs/tags/multi*\n+\trefs/heads/amb*\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\trefs/heads/ambiguous\n+\trefs/tags/multi-ref1-100000-user1\n+\trefs/tags/multi-ref1-100000-user2\n+\trefs/tags/multi-ref1-200000-user1\n+\trefs/tags/multi-ref1-200000-user2\n+\trefs/tags/multi-ref2-100000-user1\n+\trefs/tags/multi-ref2-100000-user2\n+\trefs/tags/multi-ref2-200000-user1\n+\trefs/tags/multi-ref2-200000-user2\n+\trefs/tags/multiline\n+\tEOF\n+\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473754","messageId":"1e3d499431aee563f6104f5ef1192658ece4ae4b.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 2/9] for-each-ref: explicitly test no matches","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:48Z","receivedAt":"2023-03-20T11:27:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe for-each-ref builtin can take a list of ref patterns, but if none\nmatch, it still succeeds (but with no output). Add an explicit test that\ndemonstrates that behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t6300-for-each-ref.sh | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh\nindex a58053a54c5..6614469d2d6 100755\n--- a/t/t6300-for-each-ref.sh\n+++ b/t/t6300-for-each-ref.sh\n@@ -1501,4 +1501,17 @@ test_expect_success 'git for-each-ref --stdin: matches' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'git for-each-ref with non-existing refs' '\n+\tcat >in <<-EOF &&\n+\trefs/heads/this-ref-does-not-exist\n+\trefs/tags/bogus\n+\tEOF\n+\n+\tgit for-each-ref --format=\"%(refname)\" --stdin actual &&\n+\ttest_must_be_empty actual &&\n+\n+\txargs git for-each-ref --format=\"%(refname)\" actual &&\n+\ttest_must_be_empty actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"473755","messageId":"3fd6c75812924e90075dba4f5eadba44f1eb03cb.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 4/9] commit-graph: simplify compute_generation_numbers()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:50Z","receivedAt":"2023-03-20T11:27:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change introduced the generic algorithm\ncompute_reachable_generation_numbers() and used it as the core\nfunctionality of compute_topological_levels(). Now, use it as the core\nfunctionality of compute_generation_numbers().\n\nThe main difference here is that we use generation version 2, which is\nused in to toggle the logic in compute_generation_from_max() for\ncomputing the corrected commit date based on the corrected commit dates\nof the parent commits (and the commit date of the current commit). It\nalso uses different methods for (get|set)_generation in the vtable in\norder to store and access the value in the correct places.\n\nCo-authored-by: Taylor Blau \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 64 +++++++++++++++++---------------------------------\n 1 file changed, 21 insertions(+), 43 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 4356c8c1f4b..d1c98681e88 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1559,13 +1559,31 @@ static void compute_topological_levels(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static timestamp_t get_generation_from_graph_data(struct commit *c, void *data)\n+{\n+\treturn commit_graph_data_at(c)->generation;\n+}\n+\n+static void set_generation_v2(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct commit_graph_data *g = commit_graph_data_at(c);\n+\tg->generation = (uint32_t)t;\n+}\n+\n static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n-\tstruct commit_list *list = NULL;\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_v2,\n+\t\t.data = ctx,\n+\t};\n \n \tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n+\t\tinfo.progress = ctx->progress\n+\t\t\t = start_delayed_progress(\n \t\t\t\t\t_(\"Computing commit graph generation numbers\"),\n \t\t\t\t\tctx->commits.nr);\n \n@@ -1577,47 +1595,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\t}\n \t}\n \n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\ttimestamp_t corrected_commit_date;\n-\n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tcorrected_commit_date = commit_graph_data_at(c)->generation;\n-\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (corrected_commit_date != GENERATION_NUMBER_ZERO)\n-\t\t\tcontinue;\n-\n-\t\tcommit_list_insert(c, &list);\n-\t\twhile (list) {\n-\t\t\tstruct commit *current = list->item;\n-\t\t\tstruct commit_list *parent;\n-\t\t\tint all_parents_computed = 1;\n-\t\t\ttimestamp_t max_corrected_commit_date = 0;\n-\n-\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tcorrected_commit_date = commit_graph_data_at(parent->item)->generation;\n-\n-\t\t\t\tif (corrected_commit_date == GENERATION_NUMBER_ZERO) {\n-\t\t\t\t\tall_parents_computed = 0;\n-\t\t\t\t\tcommit_list_insert(parent->item, &list);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\n-\t\t\t\tif (corrected_commit_date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = corrected_commit_date;\n-\t\t\t}\n-\n-\t\t\tif (all_parents_computed) {\n-\t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n-\t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n-\t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n-\t\t\t}\n-\t\t}\n-\t}\n+\tcompute_reachable_generation_numbers(&info, 2);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n-- \ngitgitgadget\n\n"},{"id":"473756","messageId":"79a57f30a850177ce6400b9e898f24981635541a.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 3/9] commit-graph: refactor compute_topological_levels()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:49Z","receivedAt":"2023-03-20T11:27:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis patch extracts the common code used to compute topological levels\nand corrected committer dates into a common routine,\ncompute_reachable_generation_numbers(). For ease of reading, it only\nmodifies compute_topological_levels() to use this new routine, leaving\ncompute_generation_numbers() to be modified in the next change.\n\nThis new routine dispatches to call the necessary functions to get and\nset the generation number for a given commit through a vtable (the\ncompute_generation_info struct).\n\nComputing the generation number itself is done in\ncompute_generation_from_max(), which dispatches its implementation based\non the generation version requested, or issuing a BUG() for unrecognized\ngeneration versions. This does not use a vtable because the logic\ndepends only on the generation number version, not where the data is\nbeing loaded from or being stored to. This is a subtle point that will\nmake more sense in a future change that modifies the in-memory\ngeneration values instead of just preparing values for writing to a\ncommit-graph file.\n\nThis change looks like it adds a lot of new code. However, two upcoming\nchanges will be quite small due to the work being done in this change.\n\nCo-authored-by: Taylor Blau \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 106 ++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 83 insertions(+), 23 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b3..4356c8c1f4b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1446,24 +1446,52 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n-static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+struct compute_generation_info {\n+\tstruct repository *r;\n+\tstruct packed_commit_list *commits;\n+\tstruct progress *progress;\n+\tint progress_cnt;\n+\n+\ttimestamp_t (*get_generation)(struct commit *c, void *data);\n+\tvoid (*set_generation)(struct commit *c, timestamp_t gen, void *data);\n+\tvoid *data;\n+};\n+\n+static timestamp_t compute_generation_from_max(struct commit *c,\n+\t\t\t\t\t timestamp_t max_gen,\n+\t\t\t\t\t int generation_version)\n+{\n+\tswitch (generation_version) {\n+\tcase 1: /* topological levels */\n+\t\tif (max_gen > GENERATION_NUMBER_V1_MAX - 1)\n+\t\t\tmax_gen = GENERATION_NUMBER_V1_MAX - 1;\n+\t\treturn max_gen + 1;\n+\n+\tcase 2: /* corrected commit date */\n+\t\tif (c->date && c->date > max_gen)\n+\t\t\tmax_gen = c->date - 1;\n+\t\treturn max_gen + 1;\n+\n+\tdefault:\n+\t\tBUG(\"attempting unimplemented version\");\n+\t}\n+}\n+\n+static void compute_reachable_generation_numbers(\n+\t\t\tstruct compute_generation_info *info,\n+\t\t\tint generation_version)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n \n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n-\t\t\t\t\tctx->commits.nr);\n-\tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = ctx->commits.list[i];\n-\t\tuint32_t level;\n-\n-\t\trepo_parse_commit(ctx->r, c);\n-\t\tlevel = *topo_level_slab_at(ctx->topo_levels, c);\n+\tfor (i = 0; i < info->commits->nr; i++) {\n+\t\tstruct commit *c = info->commits->list[i];\n+\t\ttimestamp_t gen;\n+\t\trepo_parse_commit(info->r, c);\n+\t\tgen = info->get_generation(c, info->data);\n+\t\tdisplay_progress(info->progress, info->progress_cnt + 1);\n \n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (level != GENERATION_NUMBER_ZERO)\n+\t\tif (gen != GENERATION_NUMBER_ZERO && gen != GENERATION_NUMBER_INFINITY)\n \t\t\tcontinue;\n \n \t\tcommit_list_insert(c, &list);\n@@ -1471,31 +1499,63 @@ static void compute_topological_levels(struct write_commit_graph_context *ctx)\n \t\t\tstruct commit *current = list->item;\n \t\t\tstruct commit_list *parent;\n \t\t\tint all_parents_computed = 1;\n-\t\t\tuint32_t max_level = 0;\n+\t\t\tuint32_t max_gen = 0;\n \n \t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n-\t\t\t\trepo_parse_commit(ctx->r, parent->item);\n-\t\t\t\tlevel = *topo_level_slab_at(ctx->topo_levels, parent->item);\n+\t\t\t\trepo_parse_commit(info->r, parent->item);\n+\t\t\t\tgen = info->get_generation(parent->item, info->data);\n \n-\t\t\t\tif (level == GENERATION_NUMBER_ZERO) {\n+\t\t\t\tif (gen == GENERATION_NUMBER_ZERO) {\n \t\t\t\t\tall_parents_computed = 0;\n \t\t\t\t\tcommit_list_insert(parent->item, &list);\n \t\t\t\t\tbreak;\n \t\t\t\t}\n \n-\t\t\t\tif (level > max_level)\n-\t\t\t\t\tmax_level = level;\n+\t\t\t\tif (gen > max_gen)\n+\t\t\t\t\tmax_gen = gen;\n \t\t\t}\n \n \t\t\tif (all_parents_computed) {\n \t\t\t\tpop_commit(&list);\n-\n-\t\t\t\tif (max_level > GENERATION_NUMBER_V1_MAX - 1)\n-\t\t\t\t\tmax_level = GENERATION_NUMBER_V1_MAX - 1;\n-\t\t\t\t*topo_level_slab_at(ctx->topo_levels, current) = max_level + 1;\n+\t\t\t\tgen = compute_generation_from_max(\n+\t\t\t\t\t\tcurrent, max_gen,\n+\t\t\t\t\t\tgeneration_version);\n+\t\t\t\tinfo->set_generation(current, gen, info->data);\n \t\t\t}\n \t\t}\n \t}\n+}\n+\n+static timestamp_t get_topo_level(struct commit *c, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\treturn *topo_level_slab_at(ctx->topo_levels, c);\n+}\n+\n+static void set_topo_level(struct commit *c, timestamp_t t, void *data)\n+{\n+\tstruct write_commit_graph_context *ctx = data;\n+\t*topo_level_slab_at(ctx->topo_levels, c) = (uint32_t)t;\n+}\n+\n+static void compute_topological_levels(struct write_commit_graph_context *ctx)\n+{\n+\tstruct compute_generation_info info = {\n+\t\t.r = ctx->r,\n+\t\t.commits = &ctx->commits,\n+\t\t.get_generation = get_topo_level,\n+\t\t.set_generation = set_topo_level,\n+\t\t.data = ctx,\n+\t};\n+\n+\tif (ctx->report_progress)\n+\t\tinfo.progress = ctx->progress\n+\t\t\t = start_delayed_progress(\n+\t\t\t\t\t_(\"Computing commit graph topological levels\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tcompute_reachable_generation_numbers(&info, 1);\n+\n \tstop_progress(&ctx->progress);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"473757","messageId":"fed76f0f08ea42af6f6207c49e8801d867d01313.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 5/9] commit-graph: return generation from memory","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:51Z","receivedAt":"2023-03-20T11:27:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit_graph_generation() method used to report a value of\nGENERATION_NUMBER_INFINITY if the commit_graph_data_slab had an instance\nfor the given commit but the graph_pos indicated the commit was not in\nthe commit-graph file.\n\nHowever, an upcoming change will introduce the ability to set generation\nvalues in-memory without writing the commit-graph file. Thus, we can no\nlonger trust 'graph_pos' to indicate whether or not the generation\nmember can be trusted.\n\nInstead, trust the 'generation' member if the commit has a value in the\nslab _and_ the 'generation' member is non-zero. Otherwise, treat it as\nGENERATION_NUMBER_INFINITY.\n\nThis only makes a difference for a very old case for the commit-graph:\nthe very first Git release to write commit-graph files wrote zeroes in\nthe topological level positions. If we are parsing a commit-graph with\nall zeroes, those commits will now appear to have\nGENERATION_NUMBER_INFINITY (as if they were not parsed from the\ncommit-graph).\n\nI attempted several variations to work around the need for providing an\nuninitialized 'generation' member, but this was the best one I found. It\ndoes require a change to a verification test in t5318 because it reports\na different error than the one about non-zero generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 8 +++-----\n t/t5318-commit-graph.sh | 2 +-\n 2 files changed, 4 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d1c98681e88..63a56483cf6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -116,12 +116,10 @@ timestamp_t commit_graph_generation(const struct commit *c)\n \tstruct commit_graph_data *data =\n \t\tcommit_graph_data_slab_peek(&commit_graph_data_slab, c);\n \n-\tif (!data)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n-\telse if (data->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\treturn GENERATION_NUMBER_INFINITY;\n+\tif (data && data->generation)\n+\t\treturn data->generation;\n \n-\treturn data->generation;\n+\treturn GENERATION_NUMBER_INFINITY;\n }\n \n static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 049c5fc8ead..b6e12115786 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -630,7 +630,7 @@ test_expect_success 'detect incorrect generation number' '\n \n test_expect_success 'detect incorrect generation number' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n-\t\t\"non-zero generation number\"\n+\t\t\"commit-graph generation for commit\"\n '\n \n test_expect_success 'detect incorrect commit date' '\n-- \ngitgitgadget\n\n"},{"id":"473758","messageId":"17a1fc9b15e27dab71c123876fb6ced535d7d4d8.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 6/9] commit-graph: introduce `ensure_generations_valid()`","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:52Z","receivedAt":"2023-03-20T11:27:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nUse the just-introduced compute_reachable_generation_numbers_1() to\nimplement a function which dynamically computes topological levels (or\ncorrected commit dates) for out-of-graph commits.\n\nThis will be useful for the ahead-behind algorithm we are about to\nintroduce, which needs accurate topological levels on _all_ commits\nreachable from the tips in order to avoid over-counting.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 +++++++++++++++++++++++++++++\n commit-graph.h | 8 ++++++++\n 2 files changed, 37 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 63a56483cf6..172e679db19 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1604,6 +1604,35 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void set_generation_in_graph_data(struct commit *c, timestamp_t t,\n+\t\t\t\t\t void *data)\n+{\n+\tcommit_graph_data_at(c)->generation = t;\n+}\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct repository *r,\n+\t\t\t struct commit **commits, size_t nr)\n+{\n+\tint generation_version = get_configured_generation_version(r);\n+\tstruct packed_commit_list list = {\n+\t\t.list = commits,\n+\t\t.alloc = nr,\n+\t\t.nr = nr,\n+\t};\n+\tstruct compute_generation_info info = {\n+\t\t.r = r,\n+\t\t.commits = &list,\n+\t\t.get_generation = get_generation_from_graph_data,\n+\t\t.set_generation = set_generation_in_graph_data,\n+\t};\n+\n+\tcompute_reachable_generation_numbers(&info, generation_version);\n+}\n+\n static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n {\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-computed\",\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 37faee6b66d..73e182ab2d0 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -190,4 +190,12 @@ struct commit_graph_data {\n */\n timestamp_t commit_graph_generation(const struct commit *);\n uint32_t commit_graph_position(const struct commit *);\n+\n+/*\n+ * After this method, all commits reachable from those in the given\n+ * list will have non-zero, non-infinite generation numbers.\n+ */\n+void ensure_generations_valid(struct repository *r,\n+\t\t\t struct commit **commits, size_t nr);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"473759","messageId":"5d937184a0eba9176d97423fb450850fc482e4de.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 7/9] commit-reach: implement ahead_behind() logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:53Z","receivedAt":"2023-03-20T11:27:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nFully implement the commit-counting logic required to determine\nahead/behind counts for a batch of commit pairs. This is a new library\nmethod within commit-reach.h. This method will be linked to the\nfor-each-ref builtin in the next change.\n\nThe interface for ahead_behind() uses two arrays. The first array of\ncommits contains the list of all starting points for the walk. This\nincludes all tip commits _and_ base commits. The second array specifies\nbase/tip pairs by pointing to commits within the first array, by index.\nThe second array also stores the resulting ahead/behind counts for each\nof these pairs.\n\nThis implementation of ahead_behind() allows multiple bases, if desired.\nEven with multiple bases, there is only one commit walk used for\ncounting the ahead/behind values, saving time when the base/tip ranges\noverlap significantly.\n\nThis interface for ahead_behind() also makes it very easy to call\nensure_generations_valid() on the entire array of bases and tips. This\ncall is necessary because it is critical that the walk that counts\nahead/behind values never walks a commit more than once. Without\ngeneration numbers on every commit, there is a possibility that a\ncommit date skew could cause the walk to revisit a commit and then\ndouble-count it. For this reason, it is strongly recommended that 'git\nahead-behind' is only run in a repository with a commit-graph file that\ncovers most of the reachable commits, storing precomputed generation\nnumbers. If no commit-graph exists, this walk will be much slower as it\nmust walk all reachable commits in ensure_generations_valid() before\nperforming the counting logic.\n\nIt is possible to detect if generation numbers are available at run time\nand redirect the implementation to another algorithm that does not\nrequire this property. However, that implementation requires a commit\nwalk per base/tip pair _and_ can be slower due to the commit date\nheuristics required. Such an implementation could be considered in the\nfuture if there is a reason to include it, but most Git hosts should\nalready be generating a commit-graph file as part of repository\nmaintenance. Most Git clients should also be generating commit-graph\nfiles as part of background maintenance or automatic GCs.\n\nNow, let's discuss the ahead/behind counting algorithm.\n\nThe first array of commits are considered the starting commits. The\nindex within that array will play a critical role.\n\nWe create a new commit slab that maps commits to a bitmap. For a given\ncommit (anywhere in the history), its bitmap stores information relative\nto which of the input commits can reach that commit. The ith bit will be\non if the ith commit from the starting list can reach that commit. It is\nimportant to notice that these bitmaps are not the typical \"reachability\nbitmaps\" that are stored in .bitmap files. Instead of signalling which\nobjects are reachable from the current commit, they instead signal\n\"which starting commits can reach me?\" It is also important to know that\nthe bitmap is not necessarily \"complete\" until we walk that commit. We\nwill perform a commit walk by generation number in such a way that we\ncan guarantee the bitmap is correct when we visit that commit.\n\nAt the beginning of the ahead_behind() method, we initialize the bitmaps\nfor each of the starting commits. By enabling the ith bit for the ith\nstarting commit, we signal \"the ith commit can reach itself.\"\n\nWe walk commits by popping the commit with maximum generation number out\nof the queue, guaranteeing that we will never walk a child of that\ncommit in any future steps.\n\nAs we walk, we load the bitmap for the current commit and perform two\nmain steps. The _second_ step examines each parent of the current commit\nand adds the current commit's bitmap bits to each parent's bitmap. (We\ncreate a new bitmap for the parent if this is our first time seeing that\nparent.) After adding the bits to the parent's bitmap, the parent is\nadded to the walk queue. Due to this passing of bits to parents, the\ncurrent commit has a guarantee that the ith bit is enabled on its bitmap\nif and only if the ith commit can reach the current commit.\n\nThe first step of the walk is to examine the bitmask on the current\ncommit and decide which ranges the commit is in or not. Due to the \"bit\npushing\" in the second step, we have a guarantee that the ith bit of the\ncurrent commit's bitmap is on if and only if the ith starting commit can\nreach it. For each ahead_behind_count struct, check the base_index and\ntip_index to see if those bits are enabled on the current bitmap. If\nexactly one bit is enabled, then increment the corresponding 'ahead' or\n'behind' count. This increment is the reason we _absolutely need_ to\nwalk commits at most once.\n\nThe only subtle thing to do with this walk is to check to see if a\nparent has all bits on in its bitmap, in which case it becomes \"stale\"\nand is marked with the STALE bit. This allows queue_has_nonstale() to be\nthe terminating condition of the walk, which greatly reduces the number\nof commits walked if all of the commits are nearby in history. It avoids\nwalking a large number of common commits when there is a deep history.\nWe also use the helper method insert_no_dup() to add commits to the\npriority queue without adding them multiple times. This uses the PARENT2\nflag. Thus, we must clear both the STALE and PARENT2 bits of all\ncommits, in case ahead_behind() is called multiple times in the same\nprocess.\n\nCo-authored-by: Taylor Blau \nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n commit-reach.c | 103 +++++++++++++++++++++++++++++++++++++++++++++++++\n commit-reach.h | 31 +++++++++++++++\n 2 files changed, 134 insertions(+)\n\ndiff --git a/commit-reach.c b/commit-reach.c\nindex 2e33c599a82..cd990dce16a 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -8,6 +8,7 @@\n #include \"revision.h\"\n #include \"tag.h\"\n #include \"commit-reach.h\"\n+#include \"ewah/ewok.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define PARENT1\t\t(1u<<16)\n@@ -941,3 +942,105 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \n \treturn found_commits;\n }\n+\n+define_commit_slab(bit_arrays, struct bitmap *);\n+static struct bit_arrays bit_arrays;\n+\n+static void insert_no_dup(struct prio_queue *queue, struct commit *c)\n+{\n+\tif (c->object.flags & PARENT2)\n+\t\treturn;\n+\tprio_queue_put(queue, c);\n+\tc->object.flags |= PARENT2;\n+}\n+\n+static struct bitmap *get_bit_array(struct commit *c, int width)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\t*bitmap = bitmap_word_alloc(width);\n+\treturn *bitmap;\n+}\n+\n+static void free_bit_array(struct commit *c)\n+{\n+\tstruct bitmap **bitmap = bit_arrays_at(&bit_arrays, c);\n+\tif (!*bitmap)\n+\t\treturn;\n+\tbitmap_free(*bitmap);\n+\t*bitmap = NULL;\n+}\n+\n+void ahead_behind(struct repository *r,\n+\t\t struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr)\n+{\n+\tstruct prio_queue queue = { .compare = compare_commits_by_gen_then_commit_date };\n+\tsize_t width = DIV_ROUND_UP(commits_nr, BITS_IN_EWORD);\n+\n+\tif (!commits_nr || !counts_nr)\n+\t\treturn;\n+\n+\tfor (size_t i = 0; i < counts_nr; i++) {\n+\t\tcounts[i].ahead = 0;\n+\t\tcounts[i].behind = 0;\n+\t}\n+\n+\tensure_generations_valid(r, commits, commits_nr);\n+\n+\tinit_bit_arrays(&bit_arrays);\n+\n+\tfor (size_t i = 0; i < commits_nr; i++) {\n+\t\tstruct commit *c = commits[i];\n+\t\tstruct bitmap *bitmap = get_bit_array(c, width);\n+\n+\t\tbitmap_set(bitmap, i);\n+\t\tinsert_no_dup(&queue, c);\n+\t}\n+\n+\twhile (queue_has_nonstale(&queue)) {\n+\t\tstruct commit *c = prio_queue_get(&queue);\n+\t\tstruct commit_list *p;\n+\t\tstruct bitmap *bitmap_c = get_bit_array(c, width);\n+\n+\t\tfor (size_t i = 0; i < counts_nr; i++) {\n+\t\t\tint reach_from_tip = !!bitmap_get(bitmap_c, counts[i].tip_index);\n+\t\t\tint reach_from_base = !!bitmap_get(bitmap_c, counts[i].base_index);\n+\n+\t\t\tif (reach_from_tip ^ reach_from_base) {\n+\t\t\t\tif (reach_from_base)\n+\t\t\t\t\tcounts[i].behind++;\n+\t\t\t\telse\n+\t\t\t\t\tcounts[i].ahead++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tstruct bitmap *bitmap_p;\n+\n+\t\t\trepo_parse_commit(r, p->item);\n+\n+\t\t\tbitmap_p = get_bit_array(p->item, width);\n+\t\t\tbitmap_or(bitmap_p, bitmap_c);\n+\n+\t\t\t/*\n+\t\t\t * If this parent is reachable from every starting\n+\t\t\t * commit, then none of its ancestors can contribute\n+\t\t\t * to the ahead/behind count. Mark it as STALE, so\n+\t\t\t * we can stop the walk when every commit in the\n+\t\t\t * queue is STALE.\n+\t\t\t */\n+\t\t\tif (bitmap_popcount(bitmap_p) == commits_nr)\n+\t\t\t\tp->item->object.flags |= STALE;\n+\n+\t\t\tinsert_no_dup(&queue, p->item);\n+\t\t}\n+\n+\t\tfree_bit_array(c);\n+\t}\n+\n+\t/* STALE is used here, PARENT2 is used by insert_no_dup(). */\n+\trepo_clear_commit_marks(r, PARENT2 | STALE);\n+\tclear_bit_arrays(&bit_arrays);\n+\tclear_prio_queue(&queue);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex 148b56fea50..f708c46e523 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -104,4 +104,35 @@ struct commit_list *get_reachable_subset(struct commit **from, int nr_from,\n \t\t\t\t\t struct commit **to, int nr_to,\n \t\t\t\t\t unsigned int reachable_flag);\n \n+struct ahead_behind_count {\n+\t/**\n+\t * As input, the *_index members indicate which positions in\n+\t * the 'tips' array correspond to the tip and base of this\n+\t * comparison.\n+\t */\n+\tsize_t tip_index;\n+\tsize_t base_index;\n+\n+\t/**\n+\t * These values store the computed counts for each side of the\n+\t * symmetric difference:\n+\t *\n+\t * 'ahead' stores the number of commits reachable from the tip\n+\t * and not reachable from the base.\n+\t *\n+\t * 'behind' stores the number of commits reachable from the base\n+\t * and not reachable from the tip.\n+\t */\n+\tunsigned int ahead;\n+\tunsigned int behind;\n+};\n+\n+/*\n+ * Given an array of commits and an array of ahead_behind_count pairs,\n+ * compute the ahead/behind counts for each pair.\n+ */\n+void ahead_behind(struct repository *r,\n+\t\t struct commit **commits, size_t commits_nr,\n+\t\t struct ahead_behind_count *counts, size_t counts_nr);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"473760","messageId":"87fe9676aec5224d00b786515c822975b2fce9b6.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 9/9] commit-reach: add tips_reachable_from_bases()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:55Z","receivedAt":"2023-03-20T11:27:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBoth 'git for-each-ref --merged=' and 'git branch --merged=' use\nthe ref-filter machinery to select references or branches (respectively)\nthat are reachable from a set of commits presented by one or more\n--merged arguments. This happens within reach_filter(), which uses the\nrevision-walk machinery to walk history in a standard way.\n\nHowever, the commit-reach.c file is full of custom searches that are\nmore efficient, especially for reachability queries that can terminate\nearly when reachability is discovered. Add a new\ntips_reachable_from_bases() method to commit-reach.c and call it from\nwithin reach_filter() in ref-filter.c. This affects both 'git branch'\nand 'git for-each-ref' as tested in p1500-graph-walks.sh.\n\nFor the Linux kernel repository, we take an already-fast algorithm and\nmake it even faster:\n\nTest HEAD~1 HEAD\n-------------------------------------------------------------------\n1500.5: contains: git for-each-ref --merged 0.13 0.02 -84.6%\n1500.6: contains: git branch --merged 0.14 0.02 -85.7%\n1500.7: contains: git tag --merged 0.15 0.03 -80.0%\n\n(Note that we remove the iterative 'git rev-list' test from p1500\nbecause it no longer makes sense as a comparison to 'git for-each-ref'\nand would just waste time running it for these comparisons.)\n\nThe algorithm is implemented in commit-reach.c in the method\ntips_reachable_from_base(). This method takes a string_list of tips and\nassigns the 'util' for each item with the value 1 if the base commit can\nreach those tips.\n\nLike other reachability queries in commit-reach.c, the fastest way to\nsearch for \"can A reach B?\" is to do a depth-first search up to the\ngeneration number of B, preferring to explore first parents before later\nparents. While we must walk all reachable commits up to that generation\nnumber when the answer is \"no\", the depth-first search can answer \"yes\"\nmuch faster than other approaches in most cases.\n\nThis search becomes trickier when there are multiple targets for the\ndepth-first search. The commits with lower generation number are more\nlikely to be within the history of the start commit, but we don't want\nto waste time searching commits of low generation number if the commit\ntarget with lowest generation number has already been found.\n\nThe trick here is to take the input commits and sort them by generation\nnumber in ascending order. Track the index within this order as\nmin_generation_index. When we find a commit, if its index in the list is\nequal to min_generation_index, then we can increase the generation\nnumber boundary of our search to the next-lowest value in the list.\n\nWith this mechanism, the number of commits to search is minimized with\nrespect to the depth-first search heuristic. We will walk all commits up\nto the minimum generation number of a commit that is _not_ reachable\nfrom the start, but we will walk only the necessary portion of the\ndepth-first search for the reachable commits of lower generation.\n\nAdd extra tests for this behavior in t6600-test-reach.sh as the\ninteresting data shape of that repository can sometimes demonstrate\ncorner case bugs.\n\nSigned-off-by: Derrick Stolee \n---\n commit-reach.c | 113 ++++++++++++++++++++++++++++++++++++\n commit-reach.h | 9 +++\n ref-filter.c | 20 ++-----\n t/perf/p1500-graph-walks.sh | 15 +++--\n t/t6600-test-reach.sh | 83 ++++++++++++++++++++++++++\n 5 files changed, 219 insertions(+), 21 deletions(-)\n\ndiff --git a/commit-reach.c b/commit-reach.c\nindex cd990dce16a..c1edeb46106 100644\n--- a/commit-reach.c\n+++ b/commit-reach.c\n@@ -1044,3 +1044,116 @@ void ahead_behind(struct repository *r,\n \tclear_bit_arrays(&bit_arrays);\n \tclear_prio_queue(&queue);\n }\n+\n+struct commit_and_index {\n+\tstruct commit *commit;\n+\tunsigned int index;\n+\ttimestamp_t generation;\n+};\n+\n+static int compare_commit_and_index_by_generation(const void *va, const void *vb)\n+{\n+\tconst struct commit_and_index *a = (const struct commit_and_index *)va;\n+\tconst struct commit_and_index *b = (const struct commit_and_index *)vb;\n+\n+\tif (a->generation > b->generation)\n+\t\treturn 1;\n+\tif (a->generation < b->generation)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n+void tips_reachable_from_bases(struct repository *r,\n+\t\t\t struct commit_list *bases,\n+\t\t\t struct commit **tips, size_t tips_nr,\n+\t\t\t int mark)\n+{\n+\tstruct commit_and_index *commits;\n+\tsize_t min_generation_index = 0;\n+\ttimestamp_t min_generation;\n+\tstruct commit_list *stack = NULL;\n+\n+\tif (!bases || !tips || !tips_nr)\n+\t\treturn;\n+\n+\t/*\n+\t * Do a depth-first search starting at 'bases' to search for the\n+\t * tips. Stop at the lowest (un-found) generation number. When\n+\t * finding the lowest commit, increase the minimum generation\n+\t * number to the next lowest (un-found) generation number.\n+\t */\n+\n+\tCALLOC_ARRAY(commits, tips_nr);\n+\n+\tfor (size_t i = 0; i < tips_nr; i++) {\n+\t\tcommits[i].commit = tips[i];\n+\t\tcommits[i].index = i;\n+\t\tcommits[i].generation = commit_graph_generation(tips[i]);\n+\t}\n+\n+\t/* Sort with generation number ascending. */\n+\tQSORT(commits, tips_nr, compare_commit_and_index_by_generation);\n+\tmin_generation = commits[0].generation;\n+\n+\twhile (bases) {\n+\t\trepo_parse_commit(r, bases->item);\n+\t\tcommit_list_insert(bases->item, &stack);\n+\t\tbases = bases->next;\n+\t}\n+\n+\twhile (stack) {\n+\t\tint explored_all_parents = 1;\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = stack->item;\n+\t\ttimestamp_t c_gen = commit_graph_generation(c);\n+\n+\t\t/* Does it match any of our tips? */\n+\t\tfor (size_t j = min_generation_index; j < tips_nr; j++) {\n+\t\t\tif (c_gen < commits[j].generation)\n+\t\t\t\tbreak;\n+\n+\t\t\tif (commits[j].commit == c) {\n+\t\t\t\ttips[commits[j].index]->object.flags |= mark;\n+\n+\t\t\t\tif (j == min_generation_index) {\n+\t\t\t\t\tunsigned int k = j + 1;\n+\t\t\t\t\twhile (k < tips_nr &&\n+\t\t\t\t\t (tips[commits[k].index]->object.flags & mark))\n+\t\t\t\t\t\tk++;\n+\n+\t\t\t\t\t/* Terminate early if all found. */\n+\t\t\t\t\tif (k >= tips_nr)\n+\t\t\t\t\t\tgoto done;\n+\n+\t\t\t\t\tmin_generation_index = k;\n+\t\t\t\t\tmin_generation = commits[k].generation;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\trepo_parse_commit(r, p->item);\n+\n+\t\t\t/* Have we already explored this parent? */\n+\t\t\tif (p->item->object.flags & SEEN)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Is it below the current minimum generation? */\n+\t\t\tif (commit_graph_generation(p->item) < min_generation)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ok, we will explore from here on. */\n+\t\t\tp->item->object.flags |= SEEN;\n+\t\t\texplored_all_parents = 0;\n+\t\t\tcommit_list_insert(p->item, &stack);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (explored_all_parents)\n+\t\t\tpop_commit(&stack);\n+\t}\n+\n+done:\n+\tfree(commits);\n+\trepo_clear_commit_marks(r, SEEN);\n+}\ndiff --git a/commit-reach.h b/commit-reach.h\nindex f708c46e523..d6321ae700e 100644\n--- a/commit-reach.h\n+++ b/commit-reach.h\n@@ -135,4 +135,13 @@ void ahead_behind(struct repository *r,\n \t\t struct commit **commits, size_t commits_nr,\n \t\t struct ahead_behind_count *counts, size_t counts_nr);\n \n+/*\n+ * For all tip commits, add 'mark' to their flags if and only if they\n+ * are reachable from one of the commits in 'bases'.\n+ */\n+void tips_reachable_from_bases(struct repository *r,\n+\t\t\t struct commit_list *bases,\n+\t\t\t struct commit **tips, size_t tips_nr,\n+\t\t\t int mark);\n+\n #endif\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 62135f649ec..c724ff94113 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2396,33 +2396,22 @@ static void reach_filter(struct ref_array *array,\n \t\t\t struct commit_list *check_reachable,\n \t\t\t int include_reached)\n {\n-\tstruct rev_info revs;\n \tint i, old_nr;\n \tstruct commit **to_clear;\n-\tstruct commit_list *cr;\n \n \tif (!check_reachable)\n \t\treturn;\n \n \tCALLOC_ARRAY(to_clear, array->nr);\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\n \tfor (i = 0; i < array->nr; i++) {\n \t\tstruct ref_array_item *item = array->items[i];\n-\t\tadd_pending_object(&revs, &item->commit->object, item->refname);\n \t\tto_clear[i] = item->commit;\n \t}\n \n-\tfor (cr = check_reachable; cr; cr = cr->next) {\n-\t\tstruct commit *merge_commit = cr->item;\n-\t\tmerge_commit->object.flags |= UNINTERESTING;\n-\t\tadd_pending_object(&revs, &merge_commit->object, \"\");\n-\t}\n-\n-\trevs.limited = 1;\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n+\ttips_reachable_from_bases(the_repository,\n+\t\t\t\t check_reachable,\n+\t\t\t\t to_clear, array->nr,\n+\t\t\t\t UNINTERESTING);\n \n \told_nr = array->nr;\n \tarray->nr = 0;\n@@ -2446,7 +2435,6 @@ static void reach_filter(struct ref_array *array,\n \t\tclear_commit_marks(merge_commit, ALL_REV_FLAGS);\n \t}\n \n-\trelease_revisions(&revs);\n \tfree(to_clear);\n }\n \ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nindex 439a448c2e6..e14e7620cce 100755\n--- a/t/perf/p1500-graph-walks.sh\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -35,11 +35,16 @@ test_perf 'ahead-behind counts: git tag' '\n \txargs git tag -l --format=\"%(ahead-behind:HEAD)\" input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\trefs/heads/commit-2-1\n+\trefs/heads/commit-5-1\n+\trefs/heads/commit-9-1\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-1-9 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:all' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-5-5 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\tEOF\n+\trun_all_modes git for-each-ref --merged=commit-9-6 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:some, multibase' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-7-8\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-5-3\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--merged=commit-5-8 \\\n+\t\t--merged=commit-8-5 \\\n+\t\t--format=\"%(refname)\" \\\n+\t\t--stdin\n+'\n+\n+test_expect_success 'for-each-ref merged:none' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-7-5\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\t>expect &&\n+\trun_all_modes git for-each-ref --merged=commit-8-4 \\\n+\t\t--format=\"%(refname)\" --stdin\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"473761","messageId":"b8523e2be0bf78a782ee3b8591b4b40d4372259d.1679311616.git.gitgitgadget@gmail.com","threadId":"59347","inReplyTo":"pull.1489.v4.git.1679311615.gitgitgadget@gmail.com","subject":"[PATCH v4 8/9] for-each-ref: add ahead-behind format atom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-03-20T11:26:54Z","receivedAt":"2023-03-20T11:27:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change implemented the ahead_behind() method, including an\nalgorithm to compute the ahead/behind values for a number of commit tips\nrelative to a number of commit bases. Now, integrate that algorithm as\npart of 'git for-each-ref' hidden behind a new format atom,\nahead-behind. This naturally extends to 'git branch' and 'git tag'\nbuiltins, as well.\n\nThis format allows specifying multiple bases, if so desired, and all\nmatching references are compared against all of those bases. For this\nreason, failing to read a reference provided from these atoms results in\nan error.\n\nIn order to translate the ahead_behind() method information to the\nformat output code in ref-filter.c, we must populate arrays of\nahead_behind_count structs. In struct ref_array, we store the full array\nthat will be passed to ahead_behind(). In struct ref_array_item, we\nstore an array of pointers that point to the relvant items within the\nfull array. In this way, we can pull all relevant ahead/behind values\ndirectly when formatting output for a specific item. It also ensures the\nlifetime of the ahead_behind_count structs matches the time that the\narray is being used.\n\nAdd specific tests of the ahead/behind counts in t6600-test-reach.sh, as\nit has an interesting repository shape. In particular, its merging\nstrategy and its use of different commit-graphs would demonstrate over-\ncounting if the ahead_behind() method did not already account for that\npossibility.\n\nAlso add tests for the specific for-each-ref, branch, and tag builtins.\nIn the case of 'git tag', there are intersting cases that happen when\nsome of the selected tips are not commits. This requires careful logic\naround commits_nr in the second loop of filter_ahead_behind(). Also, the\ntest in t7004 is carefully located to avoid being dependent on the GPG\nprereq. It also avoids using the test_commit helper, as that will add\nticks to the time and disrupt the expected timestamps in later tag\ntests.\n\nAlso add performance tests in a new p1300-graph-walks.sh script. This\nwill be useful for more uses in the future, but for now compare the\nahead-behind counting algorithm in 'git for-each-ref' to the naive\nimplementation by running 'git rev-list --count' processes for each\ninput.\n\nFor the Git source code repository, the improvement is already obvious:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git for-each-ref 0.07(0.07+0.00)\n1500.3: ahead-behind counts: git branch 0.07(0.06+0.00)\n1500.4: ahead-behind counts: git tag 0.07(0.06+0.00)\n1500.5: ahead-behind counts: git rev-list 1.32(1.04+0.27)\n\nBut the standard performance benchmark is the Linux kernel repository,\nwhich demosntrates a significant improvement:\n\nTest this tree\n---------------------------------------------------------------\n1500.2: ahead-behind counts: git for-each-ref 0.27(0.24+0.02)\n1500.3: ahead-behind counts: git branch 0.27(0.24+0.03)\n1500.4: ahead-behind counts: git tag 0.28(0.27+0.01)\n1500.5: ahead-behind counts: git rev-list 4.57(4.03+0.54)\n\nThe 'git rev-list' test exists in this change as a demonstration, but it\nwill be removed in the next change to avoid wasting time on this\ncomparison.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-for-each-ref.txt | 5 ++\n builtin/branch.c | 1 +\n builtin/for-each-ref.c | 3 ++\n builtin/tag.c | 1 +\n ref-filter.c | 73 +++++++++++++++++++++++++\n ref-filter.h | 26 ++++++++-\n t/perf/p1500-graph-walks.sh | 45 ++++++++++++++++\n t/t3203-branch-output.sh | 14 +++++\n t/t6301-for-each-ref-errors.sh | 14 +++++\n t/t6600-test-reach.sh | 86 ++++++++++++++++++++++++++++++\n t/t7004-tag.sh | 28 ++++++++++\n 11 files changed, 295 insertions(+), 1 deletion(-)\n create mode 100755 t/perf/p1500-graph-walks.sh\n\ndiff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt\nindex ccdc2911bb9..0713e49b499 100644\n--- a/Documentation/git-for-each-ref.txt\n+++ b/Documentation/git-for-each-ref.txt\n@@ -222,6 +222,11 @@ worktreepath::\n \tout, if it is checked out in any linked worktree. Empty string\n \totherwise.\n \n+ahead-behind:::\n+\tTwo integers, separated by a space, demonstrating the number of\n+\tcommits ahead and behind, respectively, when comparing the output\n+\tref to the `` specified in the format.\n+\n In addition to the above, for commit and tag objects, the header\n field names (`tree`, `parent`, `object`, `type`, and `tag`) can\n be used to specify the value in the header field.\ndiff --git a/builtin/branch.c b/builtin/branch.c\nindex f63fd45edb9..0554d7cebb3 100644\n--- a/builtin/branch.c\n+++ b/builtin/branch.c\n@@ -448,6 +448,7 @@ static void print_ref_list(struct ref_filter *filter, struct ref_sorting *sortin\n \tif (verify_ref_format(format))\n \t\tdie(_(\"unable to parse format string\"));\n \n+\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 9df16cfb854..6b3d07ef409 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -6,6 +6,7 @@\n #include \"parse-options.h\"\n #include \"ref-filter.h\"\n #include \"strvec.h\"\n+#include \"commit-reach.h\"\n \n static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n@@ -98,6 +99,8 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)\n \n \tfilter.match_as_path = 1;\n \tfilter_refs(&array, &filter, FILTER_REFS_ALL);\n+\tfilter_ahead_behind(the_repository, &format, &array);\n+\n \tref_array_sort(sorting, &array);\n \n \tif (!maxcount || array.nr < maxcount)\ndiff --git a/builtin/tag.c b/builtin/tag.c\nindex d428c45dc8d..1b3f49d7b4c 100644\n--- a/builtin/tag.c\n+++ b/builtin/tag.c\n@@ -66,6 +66,7 @@ static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,\n \t\tdie(_(\"unable to parse format string\"));\n \tfilter->with_commit_tag_algo = 1;\n \tfilter_refs(&array, filter, FILTER_REFS_TAGS);\n+\tfilter_ahead_behind(the_repository, format, &array);\n \tref_array_sort(sorting, &array);\n \n \tfor (i = 0; i < array.nr; i++) {\ndiff --git a/ref-filter.c b/ref-filter.c\nindex f8203c6b052..62135f649ec 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -158,6 +158,7 @@ enum atom_type {\n \tATOM_THEN,\n \tATOM_ELSE,\n \tATOM_REST,\n+\tATOM_AHEADBEHIND,\n };\n \n /*\n@@ -586,6 +587,22 @@ static int rest_atom_parser(struct ref_format *format, struct used_atom *atom,\n \treturn 0;\n }\n \n+static int ahead_behind_atom_parser(struct ref_format *format, struct used_atom *atom,\n+\t\t\t\t const char *arg, struct strbuf *err)\n+{\n+\tstruct string_list_item *item;\n+\n+\tif (!arg)\n+\t\treturn strbuf_addf_ret(err, -1, _(\"expected format: %%(ahead-behind:)\"));\n+\n+\titem = string_list_append(&format->bases, arg);\n+\titem->util = lookup_commit_reference_by_name(arg);\n+\tif (!item->util)\n+\t\tdie(\"failed to find '%s'\", arg);\n+\n+\treturn 0;\n+}\n+\n static int head_atom_parser(struct ref_format *format, struct used_atom *atom,\n \t\t\t const char *arg, struct strbuf *err)\n {\n@@ -645,6 +662,7 @@ static struct {\n \t[ATOM_THEN] = { \"then\", SOURCE_NONE },\n \t[ATOM_ELSE] = { \"else\", SOURCE_NONE },\n \t[ATOM_REST] = { \"rest\", SOURCE_NONE, FIELD_STR, rest_atom_parser },\n+\t[ATOM_AHEADBEHIND] = { \"ahead-behind\", SOURCE_OTHER, FIELD_STR, ahead_behind_atom_parser },\n \t/*\n \t * Please update $__git_ref_fieldlist in git-completion.bash\n \t * when you add new atoms\n@@ -1848,6 +1866,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \tstruct object *obj;\n \tint i;\n \tstruct object_info empty = OBJECT_INFO_INIT;\n+\tint ahead_behind_atoms = 0;\n \n \tCALLOC_ARRAY(ref->value, used_atom_cnt);\n \n@@ -1978,6 +1997,16 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)\n \t\t\telse\n \t\t\t\tv->s = xstrdup(\"\");\n \t\t\tcontinue;\n+\t\t} else if (atom_type == ATOM_AHEADBEHIND) {\n+\t\t\tif (ref->counts) {\n+\t\t\t\tconst struct ahead_behind_count *count;\n+\t\t\t\tcount = ref->counts[ahead_behind_atoms++];\n+\t\t\t\tv->s = xstrfmt(\"%d %d\", count->ahead, count->behind);\n+\t\t\t} else {\n+\t\t\t\t/* Not a commit. */\n+\t\t\t\tv->s = xstrdup(\"\");\n+\t\t\t}\n+\t\t\tcontinue;\n \t\t} else\n \t\t\tcontinue;\n \n@@ -2328,6 +2357,7 @@ static void free_array_item(struct ref_array_item *item)\n \t\t\tfree((char *)item->value[i].s);\n \t\tfree(item->value);\n \t}\n+\tfree(item->counts);\n \tfree(item);\n }\n \n@@ -2356,6 +2386,8 @@ void ref_array_clear(struct ref_array *array)\n \t\tfree_worktrees(ref_to_worktree_map.worktrees);\n \t\tref_to_worktree_map.worktrees = NULL;\n \t}\n+\n+\tFREE_AND_NULL(array->counts);\n }\n \n #define EXCLUDE_REACHED 0\n@@ -2418,6 +2450,47 @@ static void reach_filter(struct ref_array *array,\n \tfree(to_clear);\n }\n \n+void filter_ahead_behind(struct repository *r,\n+\t\t\t struct ref_format *format,\n+\t\t\t struct ref_array *array)\n+{\n+\tstruct commit **commits;\n+\tsize_t commits_nr = format->bases.nr + array->nr;\n+\n+\tif (!format->bases.nr || !array->nr)\n+\t\treturn;\n+\n+\tALLOC_ARRAY(commits, commits_nr);\n+\tfor (size_t i = 0; i < format->bases.nr; i++)\n+\t\tcommits[i] = format->bases.items[i].util;\n+\n+\tALLOC_ARRAY(array->counts, st_mult(format->bases.nr, array->nr));\n+\n+\tcommits_nr = format->bases.nr;\n+\tarray->counts_nr = 0;\n+\tfor (size_t i = 0; i < array->nr; i++) {\n+\t\tconst char *name = array->items[i]->refname;\n+\t\tcommits[commits_nr] = lookup_commit_reference_by_name(name);\n+\n+\t\tif (!commits[commits_nr])\n+\t\t\tcontinue;\n+\n+\t\tCALLOC_ARRAY(array->items[i]->counts, format->bases.nr);\n+\t\tfor (size_t j = 0; j < format->bases.nr; j++) {\n+\t\t\tstruct ahead_behind_count *count;\n+\t\t\tcount = &array->counts[array->counts_nr++];\n+\t\t\tcount->tip_index = commits_nr;\n+\t\t\tcount->base_index = j;\n+\n+\t\t\tarray->items[i]->counts[j] = count;\n+\t\t}\n+\t\tcommits_nr++;\n+\t}\n+\n+\tahead_behind(r, commits, commits_nr, array->counts, array->counts_nr);\n+\tfree(commits);\n+}\n+\n /*\n * API for filtering a set of refs. Based on the type of refs the user\n * has requested, we iterate through those refs and apply filters\ndiff --git a/ref-filter.h b/ref-filter.h\nindex aa0eea4ecf5..c9a11495177 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -5,6 +5,7 @@\n #include \"refs.h\"\n #include \"commit.h\"\n #include \"parse-options.h\"\n+#include \"string-list.h\"\n \n /* Quoting styles */\n #define QUOTE_NONE 0\n@@ -24,6 +25,7 @@\n \n struct atom_value;\n struct ref_sorting;\n+struct ahead_behind_count;\n \n enum ref_sorting_order {\n \tREF_SORTING_REVERSE = 1<<0,\n@@ -40,6 +42,8 @@ struct ref_array_item {\n \tconst char *symref;\n \tstruct commit *commit;\n \tstruct atom_value *value;\n+\tstruct ahead_behind_count **counts;\n+\n \tchar refname[FLEX_ARRAY];\n };\n \n@@ -47,6 +51,9 @@ struct ref_array {\n \tint nr, alloc;\n \tstruct ref_array_item **items;\n \tstruct rev_info *revs;\n+\n+\tstruct ahead_behind_count *counts;\n+\tsize_t counts_nr;\n };\n \n struct ref_filter {\n@@ -80,9 +87,15 @@ struct ref_format {\n \n \t/* Internal state to ref-filter */\n \tint need_color_reset_at_eol;\n+\n+\t/* List of bases for ahead-behind counts. */\n+\tstruct string_list bases;\n };\n \n-#define REF_FORMAT_INIT { .use_color = -1 }\n+#define REF_FORMAT_INIT { \\\n+\t.use_color = -1, \\\n+\t.bases = STRING_LIST_INIT_DUP, \\\n+}\n \n /* Macros for checking --merged and --no-merged options */\n #define _OPT_MERGED_NO_MERGED(option, filter, h) \\\n@@ -143,4 +156,15 @@ struct ref_array_item *ref_array_push(struct ref_array *array,\n \t\t\t\t const char *refname,\n \t\t\t\t const struct object_id *oid);\n \n+/*\n+ * If the provided format includes ahead-behind atoms, then compute the\n+ * ahead-behind values for the array of filtered references. Must be\n+ * called after filter_refs() but before outputting the formatted refs.\n+ *\n+ * If this is not called, then any ahead-behind atoms will be blank.\n+ */\n+void filter_ahead_behind(struct repository *r,\n+\t\t\t struct ref_format *format,\n+\t\t\t struct ref_array *array);\n+\n #endif /* REF_FILTER_H */\ndiff --git a/t/perf/p1500-graph-walks.sh b/t/perf/p1500-graph-walks.sh\nnew file mode 100755\nindex 00000000000..439a448c2e6\n--- /dev/null\n+++ b/t/perf/p1500-graph-walks.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='Commit walk performance tests'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit for-each-ref --format=\"%(refname)\" \"refs/heads/*\" \"refs/tags/*\" >allrefs &&\n+\tsort -r allrefs | head -n 50 >refs &&\n+\tfor ref in $(cat refs)\n+\tdo\n+\t\tgit branch -f ref-$ref $ref &&\n+\t\techo ref-$ref ||\n+\t\treturn 1\n+\tdone >branches &&\n+\tfor ref in $(cat refs)\n+\tdo\n+\t\tgit tag -f tag-$ref $ref &&\n+\t\techo tag-$ref ||\n+\t\treturn 1\n+\tdone >tags &&\n+\tgit commit-graph write --reachable\n+'\n+\n+test_perf 'ahead-behind counts: git for-each-ref' '\n+\tgit for-each-ref --format=\"%(ahead-behind:HEAD)\" --stdin expect <<-\\EOF &&\n+\t(HEAD detached from fromtag) 0 0\n+\trefs/heads/ambiguous 0 0\n+\trefs/heads/branch-one 1 0\n+\trefs/heads/branch-two 0 0\n+\trefs/heads/main 1 0\n+\trefs/heads/ref-to-branch 1 0\n+\trefs/heads/ref-to-remote 1 0\n+\tEOF\n+\tgit branch --format=\"%(refname) %(ahead-behind:HEAD)\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'git branch with --format=%(rest) must fail' '\n \ttest_must_fail git branch --format=\"%(rest)\" >actual\n '\ndiff --git a/t/t6301-for-each-ref-errors.sh b/t/t6301-for-each-ref-errors.sh\nindex bfda1f46ad2..2667dd13fe3 100755\n--- a/t/t6301-for-each-ref-errors.sh\n+++ b/t/t6301-for-each-ref-errors.sh\n@@ -54,4 +54,18 @@ test_expect_success 'Missing objects are reported correctly' '\n \ttest_must_be_empty brief-err\n '\n \n+test_expect_success 'ahead-behind requires an argument' '\n+\ttest_must_fail git for-each-ref \\\n+\t\t--format=\"%(ahead-behind)\" 2>err &&\n+\techo \"fatal: expected format: %(ahead-behind:)\" >expect &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'missing ahead-behind base' '\n+\ttest_must_fail git for-each-ref \\\n+\t\t--format=\"%(ahead-behind:refs/heads/missing)\" 2>err &&\n+\techo \"fatal: failed to find '\\''refs/heads/missing'\\''\" >expect &&\n+\ttest_cmp expect err\n+'\n+\n test_done\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 338a9c46a24..0cb50797ef7 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -443,4 +443,90 @@ test_expect_success 'get_reachable_subset:none' '\n \ttest_all_modes get_reachable_subset\n '\n \n+test_expect_success 'for-each-ref ahead-behind:linear' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-1-3\n+\trefs/heads/commit-1-5\n+\trefs/heads/commit-1-8\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 8\n+\trefs/heads/commit-1-3 0 6\n+\trefs/heads/commit-1-5 0 4\n+\trefs/heads/commit-1-8 0 1\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-1-9)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:all' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-2-4\n+\trefs/heads/commit-4-2\n+\trefs/heads/commit-4-4\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 24\n+\trefs/heads/commit-2-4 0 17\n+\trefs/heads/commit-4-2 0 17\n+\trefs/heads/commit-4-4 0 9\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-5-5)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 53\n+\trefs/heads/commit-4-8 8 30\n+\trefs/heads/commit-5-3 0 39\n+\trefs/heads/commit-9-9 27 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-9-6)\" --stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:some, multibase' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-1-1\n+\trefs/heads/commit-5-3\n+\trefs/heads/commit-7-8\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-1-1 0 53 0 53\n+\trefs/heads/commit-4-8 8 30 0 22\n+\trefs/heads/commit-5-3 0 39 0 39\n+\trefs/heads/commit-7-8 14 12 8 6\n+\trefs/heads/commit-9-9 27 0 27 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-9-6) %(ahead-behind:commit-6-9)\" \\\n+\t\t--stdin\n+'\n+\n+test_expect_success 'for-each-ref ahead-behind:none' '\n+\tcat >input <<-\\EOF &&\n+\trefs/heads/commit-7-5\n+\trefs/heads/commit-4-8\n+\trefs/heads/commit-9-9\n+\tEOF\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/commit-4-8 16 16\n+\trefs/heads/commit-7-5 7 4\n+\trefs/heads/commit-9-9 49 0\n+\tEOF\n+\trun_all_modes git for-each-ref \\\n+\t\t--format=\"%(refname) %(ahead-behind:commit-8-4)\" --stdin\n+'\n+\n test_done\ndiff --git a/t/t7004-tag.sh b/t/t7004-tag.sh\nindex 9aa1660651b..04a4b44183d 100755\n--- a/t/t7004-tag.sh\n+++ b/t/t7004-tag.sh\n@@ -792,6 +792,34 @@ test_expect_success 'annotations for blobs are empty' '\n \ttest_cmp expect actual\n '\n \n+# Run this before doing any signing, so the test has the same results\n+# regardless of the GPG prereq.\n+test_expect_success 'git tag --format with ahead-behind' '\n+\ttest_when_finished git reset --hard tag-one-line &&\n+\tgit commit --allow-empty -m \"left\" &&\n+\tgit tag -a -m left tag-left &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit commit --allow-empty -m \"right\" &&\n+\tgit tag -a -m left tag-right &&\n+\n+\t# Use \" !\" at the end to demonstrate whitespace\n+\t# around empty ahead-behind token for tag-blob.\n+\tcat >expect <<-EOF &&\n+\trefs/tags/tag-blob !\n+\trefs/tags/tag-left 1 1 !\n+\trefs/tags/tag-lines 0 1 !\n+\trefs/tags/tag-one-line 0 1 !\n+\trefs/tags/tag-right 0 0 !\n+\trefs/tags/tag-zero-lines 0 1 !\n+\tEOF\n+\tgit tag -l --format=\"%(refname) %(ahead-behind:HEAD) !\" >actual 2>err &&\n+\tgrep \"refs/tags/tag\" actual >actual.focus &&\n+\ttest_cmp expect actual.focus &&\n+\n+\t# Error reported for tags that point to non-commits.\n+\tgrep \"error: object [0-9a-f]* is a blob, not a commit\" err\n+'\n+\n # trying to verify annotated non-signed tags:\n \n test_expect_success GPG \\\n-- \ngitgitgadget\n\n"},{"id":"473812","messageId":"20230320204040.1446345-1-jonathantanmy@google.com","threadId":"59347","inReplyTo":"5d937184a0eba9176d97423fb450850fc482e4de.1679311616.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 7/9] commit-reach: implement ahead_behind() logic","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-03-20T20:40:40Z","receivedAt":"2023-03-20T20:41:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n> From: Derrick Stolee \n> \n> Fully implement the commit-counting logic required to determine\n> ahead/behind counts for a batch of commit pairs. This is a new library\n> method within commit-reach.h. This method will be linked to the\n> for-each-ref builtin in the next change.\n\nThanks. I see that all my review comments have been addressed, so up to\nand including this patch looks good. I haven't had time to look at the\nlast 2 patches, but it seems that other reviewers are already looking\nat those.\n"}]}