{"thread":{"id":"49229","subject":"[PATCH 0/6] Use generation numbers for --topo-order","startedAt":"2018-08-27T20:41:25Z","lastAt":"2019-11-18T23:04:38Z","messageCount":87,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano","Ævar Arnfjörð Bjarmason","SZEDER Gábor","Derrick Stolee","Jakub Narebski","Jeff King","Stefan Beller","Johannes Sixt","Mike Hommey"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"356621","messageId":"pull.25.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":null,"subject":"[PATCH 0/6] Use generation numbers for --topo-order","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-08-27T20:41:21Z","receivedAt":"2018-08-27T20:41:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series performs a decently-sized refactoring of the revision-walk\nmachinery. Well, \"refactoring\" is probably the wrong word, as I don't\nactually remove the old code. Instead, when we see certain options in the\n'rev_info' struct, we redirect the commit-walk logic to a new set of methods\nthat distribute the workload differently. By using generation numbers in the\ncommit-graph, we can significantly improve 'git log --graph' commands (and\nthe underlying 'git rev-list --topo-order').\n\nOn the Linux repository, I got the following performance results when\ncomparing to the previous version with or without a commit-graph:\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nIf you want to read this series but are unfamiliar with the commit-graph and\ngeneration numbers, then I recommend reading \nDocumentation/technical/commit-graph.txt or a blob post [1] I wrote on the\nsubject. In particular, the three-part walk described in \"revision.c:\nrefactor basic topo-order logic\" is present (but underexplained) as an\nanimated PNG [2].\n\nSince revision.c is an incredibly important (and old) portion of the\ncodebase -- and because there are so many orthogonal options in 'struct\nrev_info' -- I consider this submission to be \"RFC quality\". That is, I am\nnot confident that I am not missing anything, or that my solution is the\nbest it can be. I did merge this branch with ds/commit-graph-with-grafts and\nthe \"DO-NOT-MERGE: write and read commit-graph always\" commit that computes\na commit-graph with every 'git commit' command. The test suite passed with\nthat change, available on GitHub [3]. To ensure that I cover at least the\ncase I think are interesting, I added tests to t6600-test-reach.sh to verify\nthe walks report the correct results for the three cases there (no\ncommit-graph, full commit-graph, and a partial commit-graph so the walk\nstarts at GENERATION_NUMBER_INFINITY).\n\nOne notable case that is not included in this series is the case of a\nhistory comparison such as 'git rev-list --topo-order A..B'. The existing\ncode in limit_list() has ways to cut the walk short when all pending commits\nare UNINTERESTING. Since this code depends on commit_list instead of the\nprio_queue we are using here, I chose to leave it untouched for now. We can\nrevisit it in a separate series later. Since handle_commit() turns on\nrevs->limited when a commit is UNINTERESTING, we do not hit the new code in\nthis case. Removing this 'revs->limited = 1;' line yields correct results,\nbut the performance is worse.\n\nThis series is based on ds/reachable.\n\nThanks, -Stolee\n\n[1] \nhttps://blogs.msdn.microsoft.com/devops/2018/07/09/supercharging-the-git-commit-graph-iii-generations/\nSupercharging the Git Commit Graph III: Generations and Graph Algorithms\n\n[2] \nhttps://msdnshared.blob.core.windows.net/media/2018/06/commit-graph-topo-order-b-a.png\nAnimation showing three-part walk\n\n[3] https://github.com/derrickstolee/git/tree/topo-order/testA branch\ncontaining this series along with commits to compute commit-graph in entire\ntest suite.\n\nDerrick Stolee (6):\n  prio-queue: add 'peek' operation\n  test-reach: add run_three_modes method\n  test-reach: add rev-list tests\n  revision.c: begin refactoring --topo-order logic\n  commit/revisions: bookkeeping before refactoring\n  revision.c: refactor basic topo-order logic\n\n commit.c                   |  11 +-\n commit.h                   |   8 ++\n object.h                   |   4 +-\n prio-queue.c               |   9 ++\n prio-queue.h               |   6 +\n revision.c                 | 232 ++++++++++++++++++++++++++++++++++++-\n revision.h                 |   6 +\n t/helper/test-prio-queue.c |  10 +-\n t/t6600-test-reach.sh      |  98 +++++++++++++++-\n 9 files changed, 361 insertions(+), 23 deletions(-)\n\n\nbase-commit: 6cc017431c1c48f80d1c6512fdcc9866cf4b7f55\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-25%2Fderrickstolee%2Ftopo-order%2Fprogress-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-25/derrickstolee/topo-order/progress-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/25\n-- \ngitgitgadget\n"},{"id":"356622","messageId":"5e55669f4d035eee7f839a7b6559907f97e6b7c6.1535402479.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"[PATCH 1/6] prio-queue: add 'peek' operation","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-08-27T20:41:23Z","receivedAt":"2018-08-27T20:41:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen consuming a priority queue, it can be convenient to inspect\nthe next object that will be dequeued without actually dequeueing\nit. Our existing library did not have such a 'peek' operation, so\nadd it as prio_queue_peek().\n\nAdd a reference-level comparison in t/helper/test-prio-queue.c\nso this method is exercised by t0009-prio-queue.sh.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n prio-queue.c               |  9 +++++++++\n prio-queue.h               |  6 ++++++\n t/helper/test-prio-queue.c | 10 +++++++---\n 3 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/prio-queue.c b/prio-queue.c\nindex a078451872..d3f488cb05 100644\n--- a/prio-queue.c\n+++ b/prio-queue.c\n@@ -85,3 +85,12 @@ void *prio_queue_get(struct prio_queue *queue)\n \t}\n \treturn result;\n }\n+\n+void *prio_queue_peek(struct prio_queue *queue)\n+{\n+\tif (!queue->nr)\n+\t\treturn NULL;\n+\tif (!queue->compare)\n+\t\treturn queue->array[queue->nr - 1].data;\n+\treturn queue->array[0].data;\n+}\ndiff --git a/prio-queue.h b/prio-queue.h\nindex d030ec9dd6..682e51867a 100644\n--- a/prio-queue.h\n+++ b/prio-queue.h\n@@ -46,6 +46,12 @@ extern void prio_queue_put(struct prio_queue *, void *thing);\n  */\n extern void *prio_queue_get(struct prio_queue *);\n \n+/*\n+ * Gain access to the \"thing\" that would be returned by\n+ * prio_queue_get, but do not remove it from the queue.\n+ */\n+extern void *prio_queue_peek(struct prio_queue *);\n+\n extern void clear_prio_queue(struct prio_queue *);\n \n /* Reverse the LIFO elements */\ndiff --git a/t/helper/test-prio-queue.c b/t/helper/test-prio-queue.c\nindex 9807b649b1..e817bbf464 100644\n--- a/t/helper/test-prio-queue.c\n+++ b/t/helper/test-prio-queue.c\n@@ -22,9 +22,13 @@ int cmd__prio_queue(int argc, const char **argv)\n \tstruct prio_queue pq = { intcmp };\n \n \twhile (*++argv) {\n-\t\tif (!strcmp(*argv, \"get\"))\n-\t\t\tshow(prio_queue_get(&pq));\n-\t\telse if (!strcmp(*argv, \"dump\")) {\n+\t\tif (!strcmp(*argv, \"get\")) {\n+\t\t\tvoid *peek = prio_queue_peek(&pq);\n+\t\t\tvoid *get = prio_queue_get(&pq);\n+\t\t\tif (peek != get)\n+\t\t\t\tBUG(\"peek and get results do not match\");\n+\t\t\tshow(get);\n+\t\t} else if (!strcmp(*argv, \"dump\")) {\n \t\t\tint *v;\n \t\t\twhile ((v = prio_queue_get(&pq)))\n \t\t\t       show(v);\n-- \ngitgitgadget\n\n"},{"id":"356623","messageId":"9628396af1d3ba24c6c13c57b4aa6e0c659a324c.1535402479.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"[PATCH 2/6] test-reach: add run_three_modes method","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-08-27T20:41:24Z","receivedAt":"2018-08-27T20:41:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe 'test_three_modes' method assumes we are using the 'test-tool\nreach' command for our test. However, we may want to use the data\nshape of our commit graph and the three modes (no commit-graph,\nfull commit-graph, partial commit-graph) for other git commands.\n\nSplit test_three_modes to be a simple translation on a more general\nrun_three_modes method that executes the given command and tests\nthe actual output to the expected output.\n\nWhile inspecting this code, I realized that the final test for\n'commit_contains --tag' is silently dropping the '--tag' argument.\nIt should be quoted to include both.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 14 +++++++++-----\n 1 file changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex d139a00d1d..1b18e12a4e 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -53,18 +53,22 @@ test_expect_success 'setup' '\n \tgit config core.commitGraph true\n '\n \n-test_three_modes () {\n+run_three_modes () {\n \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t$1 <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-full .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t$1 <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-half .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t$1 <input >actual &&\n \ttest_cmp expect actual\n }\n \n+test_three_modes () {\n+\trun_three_modes \"test-tool reach $1\"\n+}\n+\n test_expect_success 'ref_newer:miss' '\n \tcat >input <<-\\EOF &&\n \tA:commit-5-7\n@@ -219,7 +223,7 @@ test_expect_success 'commit_contains:hit' '\n \tEOF\n \techo \"commit_contains(_,A,X,_):1\" >expect &&\n \ttest_three_modes commit_contains &&\n-\ttest_three_modes commit_contains --tag\n+\ttest_three_modes \"commit_contains --tag\"\n '\n \n test_expect_success 'commit_contains:miss' '\n-- \ngitgitgadget\n\n"},{"id":"356624","messageId":"708b4550a115c3905de688dae8c479e7c2636d98.1535402479.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"[PATCH 3/6] test-reach: add rev-list tests","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-08-27T20:41:25Z","receivedAt":"2018-08-27T20:41:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe rev-list command is critical to Git's functionality. Ensure it\nworks in the three commit-graph environments constructed in\nt6600-test-reach.sh. Here are a few important types of rev-list\noperations:\n\n* Basic: git rev-list --topo-order HEAD\n* Range: git rev-list --topo-order compare..HEAD\n* Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n* Symmetric Difference: git rev-list --topo-order compare...HEAD\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 84 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 84 insertions(+)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 1b18e12a4e..2fcaa39077 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -243,4 +243,88 @@ test_expect_success 'commit_contains:miss' '\n \ttest_three_modes commit_contains --tag\n '\n \n+test_expect_success 'rev-list: basic topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 commit-3-3 commit-2-3 commit-1-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: first-parent topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --first-parent --topo-order commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-3-3..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-3-8..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: first-parent range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --first-parent --topo-order commit-3-8..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: ancestry-path topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: symmetric difference topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t\tcommit-3-8 commit-2-8 commit-1-8 \\\n+\t\tcommit-3-7 commit-2-7 commit-1-7 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-3-8...commit-6-6\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"356626","messageId":"908442417d9d0e0d861fa7ac47a97eca63294596.1535402479.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"[PATCH 4/6] revision.c: begin refactoring --topo-order logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-08-27T20:41:27Z","receivedAt":"2018-08-27T20:41:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen running 'git rev-list --topo-order' and its kin, the topo_order\nsetting in struct rev_info implies the limited setting. This means\nthat the following things happen during prepare_revision_walk():\n\n* revs->limited implies we run limit_list() to walk the entire\n  reachable set. There are some short-cuts here, such as if we\n  perform a range query like 'git rev-list COMPARE..HEAD' and we\n  can stop limit_list() when all queued commits are uninteresting.\n\n* revs->topo_order implies we run sort_in_topological_order(). See\n  the implementation of that method in commit.c. It implies that\n  the full set of commits to order is in the given commit_list.\n\nThese two methods imply that a 'git rev-list --topo-order HEAD'\ncommand must walk the entire reachable set of commits _twice_ before\nreturning a single result.\n\nIf we have a commit-graph file with generation numbers computed, then\nthere is a better way. This patch introduces some necessary logic\nredirection when we are in this situation.\n\nIn v2.18.0, the commit-graph file contains zero-valued bytes in the\npositions where the generation number is stored in v2.19.0 and later.\nThus, we use generation_numbers_enabled() to check if the commit-graph\nis available and has non-zero generation numbers.\n\nWhen setting revs->limited only because revs->topo_order is true,\nonly do so if generation numbers are not available. There is no\nreason to use the new logic as it will behave similarly when all\ngeneration numbers are INFINITY or ZERO.\n\nIn prepare_revision_walk(), if we have revs->topo_order but not\nrevs->limited, then we trigger the new logic. It breaks the logic\ninto three pieces, to fit with the existing framework:\n\n1. init_topo_walk() fills a new struct topo_walk_info in the rev_info\n   struct. We use the presence of this struct as a signal to use the\n   new methods during our walk. In this patch, this method simply\n   calls limit_list() and sort_in_topological_order(). In the future,\n   this method will set up a new data structure to perform that logic\n   in-line.\n\n2. next_topo_commit() provides get_revision_1() with the next topo-\n   ordered commit in the list. Currently, this simply pops the commit\n   from revs->commits.\n\n3. expand_topo_walk() provides get_revision_1() with a way to signal\n   walking beyond the latest commit. Currently, this calls\n   add_parents_to_list() exactly like the old logic.\n\nWhile this commit presents method redirection for performing the\nexact same logic as before, it allows the next commit to focus only\non the new logic.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 42 ++++++++++++++++++++++++++++++++++++++----\n revision.h |  4 ++++\n 2 files changed, 42 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 3205a3947a..1db70dc951 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -25,6 +25,7 @@\n #include \"worktree.h\"\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n+#include \"commit-graph.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2451,7 +2452,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \tif (revs->diffopt.objfind)\n \t\trevs->simplify_history = 0;\n \n-\tif (revs->topo_order)\n+\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n \t\trevs->limited = 1;\n \n \tif (revs->prune_data.nr) {\n@@ -2889,6 +2890,33 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n+struct topo_walk_info {};\n+\n+static void init_topo_walk(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info;\n+\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n+\tinfo = revs->topo_walk_info;\n+\tmemset(info, 0, sizeof(struct topo_walk_info));\n+\n+\tlimit_list(revs);\n+\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+}\n+\n+static struct commit *next_topo_commit(struct rev_info *revs)\n+{\n+\treturn pop_commit(&revs->commits);\n+}\n+\n+static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n+{\n+\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\t\tif (!revs->ignore_missing_links)\n+\t\t\tdie(\"Failed to traverse parents of commit %s\",\n+\t\t\t    oid_to_hex(&commit->object.oid));\n+\t}\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -2925,11 +2953,13 @@ int prepare_revision_walk(struct rev_info *revs)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\n-\tif (revs->limited)\n+\tif (revs->limited) {\n \t\tif (limit_list(revs) < 0)\n \t\t\treturn -1;\n-\tif (revs->topo_order)\n-\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t\tif (revs->topo_order)\n+\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t} else if (revs->topo_order)\n+\t\tinit_topo_walk(revs);\n \tif (revs->line_level_traverse)\n \t\tline_log_filter(revs);\n \tif (revs->simplify_merges)\n@@ -3254,6 +3284,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\tif (revs->reflog_info)\n \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n+\t\telse if (revs->topo_walk_info)\n+\t\t\tcommit = next_topo_commit(revs);\n \t\telse\n \t\t\tcommit = pop_commit(&revs->commits);\n \n@@ -3275,6 +3307,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\t\tif (revs->reflog_info)\n \t\t\t\ttry_to_simplify_commit(revs, commit);\n+\t\t\telse if (revs->topo_walk_info)\n+\t\t\t\texpand_topo_walk(revs, commit);\n \t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n \t\t\t\tif (!revs->ignore_missing_links)\n \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\ndiff --git a/revision.h b/revision.h\nindex bf2239f876..e48181673d 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -54,6 +54,8 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct topo_walk_info;\n+\n struct rev_info {\n \t/* Starting list */\n \tstruct commit_list *commits;\n@@ -227,6 +229,8 @@ struct rev_info {\n \tconst char *break_bar;\n \n \tstruct revision_sources *sources;\n+\n+\tstruct topo_walk_info *topo_walk_info;\n };\n \n extern int ref_excluded(struct string_list *, const char *path);\n-- \ngitgitgadget\n\n"},{"id":"356625","messageId":"a7272f2799d8a04e3ba0479a6ed988322665f436.1535402479.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"[PATCH 5/6] commit/revisions: bookkeeping before refactoring","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-08-27T20:41:28Z","receivedAt":"2018-08-27T20:41:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThere are a few things that need to move around a little before\nmaking a big refactoring in the topo-order logic:\n\n1. We need access to record_author_date() and\n   compare_commits_by_author_date() in revision.c. These are used\n   currently by sort_in_topological_order() in commit.c.\n\n2. Moving these methods to commit.h requires adding the author_slab\n   definition to commit.h.\n\n3. The add_parents_to_list() method in revision.c performs logic\n   around the UNINTERESTING flag and other special cases depending\n   on the struct rev_info. Allow this method to ignore a NULL 'list'\n   parameter, as we will not be populating the list for our walk.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n commit.c   | 11 ++++-------\n commit.h   |  8 ++++++++\n revision.c |  6 ++++--\n 3 files changed, 16 insertions(+), 9 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 32d1234bd7..2dbe187b8c 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -655,11 +655,8 @@ struct commit *pop_commit(struct commit_list **stack)\n /* count number of children that have not been emitted */\n define_commit_slab(indegree_slab, int);\n \n-/* record author-date for each commit object */\n-define_commit_slab(author_date_slab, unsigned long);\n-\n-static void record_author_date(struct author_date_slab *author_date,\n-\t\t\t       struct commit *commit)\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit)\n {\n \tconst char *buffer = get_commit_buffer(commit, NULL);\n \tstruct ident_split ident;\n@@ -684,8 +681,8 @@ fail_exit:\n \tunuse_commit_buffer(commit, buffer);\n }\n \n-static int compare_commits_by_author_date(const void *a_, const void *b_,\n-\t\t\t\t\t  void *cb_data)\n+int compare_commits_by_author_date(const void *a_, const void *b_,\n+\t\t\t\t   void *cb_data)\n {\n \tconst struct commit *a = a_, *b = b_;\n \tstruct author_date_slab *author_date = cb_data;\ndiff --git a/commit.h b/commit.h\nindex e2c99d9b04..51de10e698 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -8,6 +8,7 @@\n #include \"gpg-interface.h\"\n #include \"string-list.h\"\n #include \"pretty.h\"\n+#include \"commit-slab.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n #define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n@@ -328,6 +329,13 @@ extern int remove_signature(struct strbuf *buf);\n  */\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n+/* record author-date for each commit object */\n+define_commit_slab(author_date_slab, timestamp_t);\n+\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit);\n+\n+int compare_commits_by_author_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \ndiff --git a/revision.c b/revision.c\nindex 1db70dc951..565f903e46 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -804,7 +804,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\treturn 0;\n \t}\n@@ -843,7 +844,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\tif (revs->first_parent_only)\n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"356627","messageId":"73713bcbee42417f909241dbd5a2a916f316baa7.1535402480.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"[PATCH 6/6] revision.c: refactor basic topo-order logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-08-27T20:41:29Z","receivedAt":"2018-08-27T20:41:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen running a command like 'git rev-list --topo-order HEAD',\nGit performed the following steps:\n\n1. Run limit_list(), which parses all reachable commits,\n   adds them to a linked list, and distributes UNINTERESTING\n   flags. If all unprocessed commits are UNINTERESTING, then\n   it may terminate without walking all reachable commits.\n   This does not occur if we do not specify UNINTERESTING\n   commits.\n\n2. Run sort_in_topological_order(), which is an implementation\n   of Kahn's algorithm. It first iterates through the entire\n   set of important commits and computes the in-degree of each\n   (plus one, as we use 'zero' as a special value here). Then,\n   we walk the commits in priority order, adding them to the\n   priority queue if and only if their in-degree is one. As\n   we remove commits from this priority queue, we decrement the\n   in-degree of their parents.\n\n3. While we are peeling commits for output, get_revision_1()\n   uses pop_commit on the full list of commits computed by\n   sort_in_topological_order().\n\nIn the new algorithm, these three steps correspond to three\ndifferent commit walks. We run these walks simultaneously,\nand advance each only as far as necessary to satisfy the\nrequirements of the 'higher order' walk. We know when we can\npause each walk by using generation numbers from the commit-\ngraph feature.\n\nRecall that the generation number of a commit satisfies:\n\n* If the commit has at least one parent, then the generation\n  number is one more than the maximum generation number among\n  its parents.\n\n* If the commit has no parent, then the generation number is one.\n\nThere are two special generation numbers:\n\n* GENERATION_NUMBER_INFINITY: this value is 0xffffffff and\n  indicates that the commit is not stored in the commit-graph and\n  the generation number was not previously calculated.\n\n* GENERATION_NUMBER_ZERO: this value (0) is a special indicator\n  to say that the commit-graph was generated by a version of Git\n  that does not compute generation numbers (such as v2.18.0).\n\nSince we use generation_numbers_enabled() before using the new\nalgorithm, we do not need to worry about GENERATION_NUMBER_ZERO.\nHowever, the existence of GENERATION_NUMBER_INFINITY implies the\nfollowing weaker statement than the usual we expect from\ngeneration numbers:\n\n    If A and B are commits with generation numbers gen(A) and\n    gen(B) and gen(A) < gen(B), then A cannot reach B.\n\nThus, we will walk in each of our stages until the \"maximum\nunexpanded generation number\" is strictly lower than the\ngeneration number of a commit we are about to use.\n\nThe walks are as follows:\n\n1. EXPLORE: using the explore_queue priority queue (ordered by\n   maximizing the generation number), parse each reachable\n   commit until all commits in the queue have generation\n   number strictly lower than needed. During this walk, update\n   the UNINTERESTING flags as necessary.\n\n2. INDEGREE: using the indegree_queue priority queue (ordered\n   by maximizing the generation number), add one to the in-\n   degree of each parent for each commit that is walked. Since\n   we walk in order of decreasing generation number, we know\n   that discovering an in-degree value of 0 means the value for\n   that commit was not initialized, so should be initialized to\n   two. (Recall that in-degree value \"1\" is what we use to say a\n   commit is ready for output.) As we iterate the parents of a\n   commit during this walk, ensure the EXPLORE walk has walked\n   beyond their generation numbers.\n\n3. TOPO: using the topo_queue priority queue (ordered based on\n   the sort_order given, which could be commit-date, author-\n   date, or typical topo-order which treats the queue as a LIFO\n   stack), remove a commit from the queue and decrement the\n   in-degree of each parent. If a parent has an in-degree of\n   one, then we add it to the topo_queue. Before we decrement\n   the in-degree, however, ensure the INDEGREE walk has walked\n   beyond that generation number.\n\nThe implementations of these walks are in the following methods:\n\n* explore_walk_step and explore_to_depth\n* indegree_walk_step and compute_indegrees_to_depth\n* next_topo_commit and expand_topo_walk\n\nThese methods have some patterns that may seem strange at first,\nbut they are probably carry-overs from their equivalents in\nlimit_list and sort_in_topological_order.\n\nOne thing that is missing from this implementation is a proper\nway to stop walking when the entire queue is UNINTERESTING, so\nthis implementation is not enabled by comparisions, such as in\n'git rev-list --topo-order A..B'. This can be updated in the\nfuture.\n\nIn my local testing, I used the following Git commands on the\nLinux repository in three modes: HEAD~1 with no commit-graph,\nHEAD~1 with a commit-graph, and HEAD with a commit-graph. This\nallows comparing the benefits we get from parsing commits from\nthe commit-graph and then again the benefits we get by\nrestricting the set of commits we walk.\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nThis speedup is due to a few things. First, the new generation-\nnumber-enabled algorithm walks commits on order of the number of\nresults output (subject to some branching structure expectations).\nSince we limit to 100 results, we are running a query similar to\nfilling a single page of results. Second, when specifying a path,\nwe must parse the root tree object for each commit we walk. The\nprevious benefits from the commit-graph are entirely from reading\nthe commit-graph instead of parsing commits. Since we need to\nparse trees for the same number of commits as before, we slow\ndown significantly from the non-path-based query.\n\nFor the test above, I specifically selected a path that is changed\nfrequently, including by merge commits. A less-frequently-changed\npath (such as 'README') has similar end-to-end time since we need\nto walk the same number of commits (before determining we do not\nhave 100 hits). However, get get the benefit that the output is\npresented to the user as it is discovered, much the same as a\nnormal 'git log' command (no '--topo-order'). This is an improved\nuser experience, even if the command has the same runtime.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n object.h   |   4 +-\n revision.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n revision.h |   2 +\n 3 files changed, 194 insertions(+), 8 deletions(-)\n\ndiff --git a/object.h b/object.h\nindex b132944c51..a84eea61d2 100644\n--- a/object.h\n+++ b/object.h\n@@ -57,7 +57,7 @@ struct object_array {\n \n /*\n  * object flag allocation:\n- * revision.h:               0---------10                                26\n+ * revision.h:               0---------10                                26--28\n  * fetch-pack.c:             0----5\n  * walker.c:                 0-2\n  * upload-pack.c:                4       11-----14  16-----19\n@@ -75,7 +75,7 @@ struct object_array {\n  * builtin/show-branch.c:    0-------------------------------------------26\n  * builtin/unpack-objects.c:                                 2021\n  */\n-#define FLAG_BITS  27\n+#define FLAG_BITS  29\n \n /*\n  * The object type is stored in 3 bits.\ndiff --git a/revision.c b/revision.c\nindex 565f903e46..7b4beb9978 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -26,6 +26,7 @@\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"prio-queue.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2892,30 +2893,213 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct topo_walk_info {};\n+define_commit_slab(indegree_slab, int);\n+\n+struct topo_walk_info {\n+\tuint32_t min_generation;\n+\tstruct prio_queue explore_queue;\n+\tstruct prio_queue indegree_queue;\n+\tstruct prio_queue topo_queue;\n+\tstruct indegree_slab indegree;\n+\tstruct author_date_slab author_date;\n+};\n+\n+static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n+{\n+\tif (c->object.flags & flag)\n+\t\treturn;\n+\n+\tc->object.flags |= flag;\n+\tprio_queue_put(q, c);\n+}\n+\n+static void explore_walk_step(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit_list *p;\n+\tstruct commit *c = prio_queue_get(&info->explore_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tif (revs->max_age != -1 && (c->date < revs->max_age))\n+\t\tc->object.flags |= UNINTERESTING;\n+\n+\tif (add_parents_to_list(revs, c, NULL, NULL) < 0)\n+\t\treturn;\n+\n+\tif (c->object.flags & UNINTERESTING)\n+\t\tmark_parents_uninteresting(c);\n+\n+\tfor (p = c->parents; p; p = p->next)\n+\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n+}\n+\n+static void explore_to_depth(struct rev_info *revs,\n+\t\t\t     uint32_t gen)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n+\t       c->generation >= gen)\n+\t\texplore_walk_step(revs);\n+}\n+\n+static void indegree_walk_step(struct rev_info *revs)\n+{\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c = prio_queue_get(&info->indegree_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\texplore_to_depth(revs, c->generation);\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tfor (p = c->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\tif (*pi)\n+\t\t\t(*pi)++;\n+\t\telse\n+\t\t\t*pi = 2;\n+\n+\t\ttest_flag_and_insert(&info->indegree_queue, parent, TOPO_WALK_INDEGREE);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n+\t}\n+}\n+\n+static void compute_indegrees_to_depth(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n+\t       c->generation >= info->min_generation)\n+\t\tindegree_walk_step(revs);\n+}\n \n static void init_topo_walk(struct rev_info *revs)\n {\n \tstruct topo_walk_info *info;\n+\tstruct commit_list *list;\n \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n \tinfo = revs->topo_walk_info;\n \tmemset(info, 0, sizeof(struct topo_walk_info));\n \n-\tlimit_list(revs);\n-\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\tinit_indegree_slab(&info->indegree);\n+\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n+\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n+\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n+\n+\tswitch (revs->sort_order) {\n+\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n+\t\tinfo->topo_queue.compare = NULL;\n+\t\tbreak;\n+\tcase REV_SORT_BY_COMMIT_DATE:\n+\t\tinfo->topo_queue.compare = compare_commits_by_commit_date;\n+\t\tbreak;\n+\tcase REV_SORT_BY_AUTHOR_DATE:\n+\t\tinit_author_date_slab(&info->author_date);\n+\t\tinfo->topo_queue.compare = compare_commits_by_author_date;\n+\t\tinfo->topo_queue.cb_data = &info->author_date;\n+\t\tbreak;\n+\t}\n+\n+\tinfo->explore_queue.compare = compare_commits_by_gen_then_commit_date;\n+\tinfo->indegree_queue.compare = compare_commits_by_gen_then_commit_date;\n+\n+\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n+\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n+\n+\t\tif (parse_commit_gently(c, 1))\n+\t\t\tcontinue;\n+\t\tif (c->generation < info->min_generation)\n+\t\t\tinfo->min_generation = c->generation;\n+\t}\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n+\n+\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n+\t\t\trecord_author_date(&info->author_date, c);\n+\t}\n+\tcompute_indegrees_to_depth(revs);\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\n+\t\tif (*(indegree_slab_at(&info->indegree, c)) == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, c);\n+\t}\n+\n+\t/*\n+\t * This is unfortunate; the initial tips need to be shown\n+\t * in the order given from the revision traversal machinery.\n+\t */\n+\tif (revs->sort_order == REV_SORT_IN_GRAPH_ORDER)\n+\t\tprio_queue_reverse(&info->topo_queue);\n }\n \n static struct commit *next_topo_commit(struct rev_info *revs)\n {\n-\treturn pop_commit(&revs->commits);\n+\tstruct commit *c;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\n+\t/* pop next off of topo_queue */\n+\tc = prio_queue_get(&info->topo_queue);\n+\n+\tif (c)\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 0;\n+\n+\treturn c;\n }\n \n static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n {\n-\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tif (add_parents_to_list(revs, commit, NULL, NULL) < 0) {\n \t\tif (!revs->ignore_missing_links)\n \t\t\tdie(\"Failed to traverse parents of commit %s\",\n-\t\t\t    oid_to_hex(&commit->object.oid));\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t}\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi;\n+\n+\t\tif (parse_commit_gently(parent, 1) < 0)\n+\t\t\tcontinue;\n+\n+\t\tif (parent->generation < info->min_generation) {\n+\t\t\tinfo->min_generation = parent->generation;\n+\t\t\tcompute_indegrees_to_depth(revs);\n+\t\t}\n+\n+\t\tpi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\t(*pi)--;\n+\t\tif (*pi == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, parent);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n \t}\n }\n \ndiff --git a/revision.h b/revision.h\nindex e48181673d..ca10392021 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -21,6 +21,8 @@\n #define PATCHSAME\t(1u<<9)\n #define BOTTOM\t\t(1u<<10)\n #define TRACK_LINEAR\t(1u<<26)\n+#define TOPO_WALK_EXPLORED (1u<<27)\n+#define TOPO_WALK_INDEGREE (1u<<28)\n #define ALL_REV_FLAGS\t(((1u<<11)-1) | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n-- \ngitgitgadget\n"},{"id":"356630","messageId":"xmqqh8jfijv6.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] Use generation numbers for --topo-order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-08-27T21:23:57Z","receivedAt":"2018-08-27T21:24:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> This patch series performs a decently-sized refactoring of the revision-walk\n> machinery. Well, \"refactoring\" is probably the wrong word, as I don't\n> actually remove the old code. Instead, when we see certain options in the\n> 'rev_info' struct, we redirect the commit-walk logic to a new set of methods\n> that distribute the workload differently. By using generation numbers in the\n> commit-graph, we can significantly improve 'git log --graph' commands (and\n> the underlying 'git rev-list --topo-order').\n\nFinally ;-).\n"},{"id":"358377","messageId":"pull.25.v2.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.git.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] Use generation numbers for --topo-order","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-18T04:08:41Z","receivedAt":"2018-09-18T04:08:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series performs a decently-sized refactoring of the revision-walk\nmachinery. Well, \"refactoring\" is probably the wrong word, as I don't\nactually remove the old code. Instead, when we see certain options in the\n'rev_info' struct, we redirect the commit-walk logic to a new set of methods\nthat distribute the workload differently. By using generation numbers in the\ncommit-graph, we can significantly improve 'git log --graph' commands (and\nthe underlying 'git rev-list --topo-order').\n\nOn the Linux repository, I got the following performance results when\ncomparing to the previous version with or without a commit-graph:\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nIf you want to read this series but are unfamiliar with the commit-graph and\ngeneration numbers, then I recommend reading \nDocumentation/technical/commit-graph.txt or a blob post [1] I wrote on the\nsubject. In particular, the three-part walk described in \"revision.c:\nrefactor basic topo-order logic\" is present (but underexplained) as an\nanimated PNG [2].\n\nSince revision.c is an incredibly important (and old) portion of the\ncodebase -- and because there are so many orthogonal options in 'struct\nrev_info' -- I consider this submission to be \"RFC quality\". That is, I am\nnot confident that I am not missing anything, or that my solution is the\nbest it can be. I did merge this branch with ds/commit-graph-with-grafts and\nthe \"DO-NOT-MERGE: write and read commit-graph always\" commit that computes\na commit-graph with every 'git commit' command. The test suite passed with\nthat change, available on GitHub [3]. To ensure that I cover at least the\ncase I think are interesting, I added tests to t6600-test-reach.sh to verify\nthe walks report the correct results for the three cases there (no\ncommit-graph, full commit-graph, and a partial commit-graph so the walk\nstarts at GENERATION_NUMBER_INFINITY).\n\nOne notable case that is not included in this series is the case of a\nhistory comparison such as 'git rev-list --topo-order A..B'. The existing\ncode in limit_list() has ways to cut the walk short when all pending commits\nare UNINTERESTING. Since this code depends on commit_list instead of the\nprio_queue we are using here, I chose to leave it untouched for now. We can\nrevisit it in a separate series later. Since handle_commit() turns on\nrevs->limited when a commit is UNINTERESTING, we do not hit the new code in\nthis case. Removing this 'revs->limited = 1;' line yields correct results,\nbut the performance is worse.\n\nThis series was based on ds/reachable, but is now based on 'master' to not\nconflict with 182070 \"commit: use timestamp_t for author_date_slab\". There\nis a small conflict with md/filter-trees, because it renamed a flag in\nrevisions.h in the line before I add new flags. Hopefully this conflict is\nnot too difficult to resolve.\n\nThanks, -Stolee\n\n[1] \nhttps://blogs.msdn.microsoft.com/devops/2018/07/09/supercharging-the-git-commit-graph-iii-generations/\nSupercharging the Git Commit Graph III: Generations and Graph Algorithms\n\n[2] \nhttps://msdnshared.blob.core.windows.net/media/2018/06/commit-graph-topo-order-b-a.png\nAnimation showing three-part walk\n\n[3] https://github.com/derrickstolee/git/tree/topo-order/testA branch\ncontaining this series along with commits to compute commit-graph in entire\ntest suite.\n\nDerrick Stolee (6):\n  prio-queue: add 'peek' operation\n  test-reach: add run_three_modes method\n  test-reach: add rev-list tests\n  revision.c: begin refactoring --topo-order logic\n  commit/revisions: bookkeeping before refactoring\n  revision.c: refactor basic topo-order logic\n\n commit.c                   |  11 +-\n commit.h                   |   8 ++\n object.h                   |   4 +-\n prio-queue.c               |   9 ++\n prio-queue.h               |   6 +\n revision.c                 | 232 ++++++++++++++++++++++++++++++++++++-\n revision.h                 |   6 +\n t/helper/test-prio-queue.c |  10 +-\n t/t6600-test-reach.sh      |  98 +++++++++++++++-\n 9 files changed, 361 insertions(+), 23 deletions(-)\n\n\nbase-commit: 2d3b1c576c85b7f5db1f418907af00ab88e0c303\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-25%2Fderrickstolee%2Ftopo-order%2Fprogress-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-25/derrickstolee/topo-order/progress-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/25\n\nRange-diff vs v1:\n\n 1:  5e55669f4d = 1:  cc1ec4c270 prio-queue: add 'peek' operation\n 2:  9628396af1 = 2:  404c918608 test-reach: add run_three_modes method\n 3:  708b4550a1 = 3:  30dee58c61 test-reach: add rev-list tests\n 4:  908442417d ! 4:  a74ae13d4e revision.c: begin refactoring --topo-order logic\n     @@ -168,4 +168,4 @@\n      +\tstruct topo_walk_info *topo_walk_info;\n       };\n       \n     - extern int ref_excluded(struct string_list *, const char *path);\n     + int ref_excluded(struct string_list *, const char *path);\n 5:  a7272f2799 ! 5:  0e64fc144c commit/revisions: bookkeeping before refactoring\n     @@ -27,7 +27,7 @@\n       define_commit_slab(indegree_slab, int);\n       \n      -/* record author-date for each commit object */\n     --define_commit_slab(author_date_slab, unsigned long);\n     +-define_commit_slab(author_date_slab, timestamp_t);\n      -\n      -static void record_author_date(struct author_date_slab *author_date,\n      -\t\t\t       struct commit *commit)\n 6:  73713bcbee ! 6:  3b185ac3b1 revision.c: refactor basic topo-order logic\n     @@ -153,11 +153,11 @@\n       \n       /*\n        * object flag allocation:\n     -- * revision.h:               0---------10                                26\n     -+ * revision.h:               0---------10                                26--28\n     -  * fetch-pack.c:             0----5\n     +- * revision.h:               0---------10                              2526\n     ++ * revision.h:               0---------10                              25----28\n     +  * fetch-pack.c:             01\n     +  * negotiator/default.c:       2--5\n        * walker.c:                 0-2\n     -  * upload-pack.c:                4       11-----14  16-----19\n      @@\n        * builtin/show-branch.c:    0-------------------------------------------26\n        * builtin/unpack-objects.c:                                 2021\n     @@ -404,11 +404,11 @@\n      --- a/revision.h\n      +++ b/revision.h\n      @@\n     - #define PATCHSAME\t(1u<<9)\n     - #define BOTTOM\t\t(1u<<10)\n     + #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n       #define TRACK_LINEAR\t(1u<<26)\n     + #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n      +#define TOPO_WALK_EXPLORED (1u<<27)\n      +#define TOPO_WALK_INDEGREE (1u<<28)\n     - #define ALL_REV_FLAGS\t(((1u<<11)-1) | TRACK_LINEAR)\n       \n       #define DECORATE_SHORT_REFS\t1\n     + #define DECORATE_FULL_REFS\t2\n\n-- \ngitgitgadget\n"},{"id":"358378","messageId":"cc1ec4c2702d8ba35873600d321015bb0430d92e.1537243720.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] prio-queue: add 'peek' operation","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-18T04:08:43Z","receivedAt":"2018-09-18T04:08:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen consuming a priority queue, it can be convenient to inspect\nthe next object that will be dequeued without actually dequeueing\nit. Our existing library did not have such a 'peek' operation, so\nadd it as prio_queue_peek().\n\nAdd a reference-level comparison in t/helper/test-prio-queue.c\nso this method is exercised by t0009-prio-queue.sh.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n prio-queue.c               |  9 +++++++++\n prio-queue.h               |  6 ++++++\n t/helper/test-prio-queue.c | 10 +++++++---\n 3 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/prio-queue.c b/prio-queue.c\nindex a078451872..d3f488cb05 100644\n--- a/prio-queue.c\n+++ b/prio-queue.c\n@@ -85,3 +85,12 @@ void *prio_queue_get(struct prio_queue *queue)\n \t}\n \treturn result;\n }\n+\n+void *prio_queue_peek(struct prio_queue *queue)\n+{\n+\tif (!queue->nr)\n+\t\treturn NULL;\n+\tif (!queue->compare)\n+\t\treturn queue->array[queue->nr - 1].data;\n+\treturn queue->array[0].data;\n+}\ndiff --git a/prio-queue.h b/prio-queue.h\nindex d030ec9dd6..682e51867a 100644\n--- a/prio-queue.h\n+++ b/prio-queue.h\n@@ -46,6 +46,12 @@ extern void prio_queue_put(struct prio_queue *, void *thing);\n  */\n extern void *prio_queue_get(struct prio_queue *);\n \n+/*\n+ * Gain access to the \"thing\" that would be returned by\n+ * prio_queue_get, but do not remove it from the queue.\n+ */\n+extern void *prio_queue_peek(struct prio_queue *);\n+\n extern void clear_prio_queue(struct prio_queue *);\n \n /* Reverse the LIFO elements */\ndiff --git a/t/helper/test-prio-queue.c b/t/helper/test-prio-queue.c\nindex 9807b649b1..e817bbf464 100644\n--- a/t/helper/test-prio-queue.c\n+++ b/t/helper/test-prio-queue.c\n@@ -22,9 +22,13 @@ int cmd__prio_queue(int argc, const char **argv)\n \tstruct prio_queue pq = { intcmp };\n \n \twhile (*++argv) {\n-\t\tif (!strcmp(*argv, \"get\"))\n-\t\t\tshow(prio_queue_get(&pq));\n-\t\telse if (!strcmp(*argv, \"dump\")) {\n+\t\tif (!strcmp(*argv, \"get\")) {\n+\t\t\tvoid *peek = prio_queue_peek(&pq);\n+\t\t\tvoid *get = prio_queue_get(&pq);\n+\t\t\tif (peek != get)\n+\t\t\t\tBUG(\"peek and get results do not match\");\n+\t\t\tshow(get);\n+\t\t} else if (!strcmp(*argv, \"dump\")) {\n \t\t\tint *v;\n \t\t\twhile ((v = prio_queue_get(&pq)))\n \t\t\t       show(v);\n-- \ngitgitgadget\n\n"},{"id":"358379","messageId":"404c9186080ecee6c1cc39a6dcd17deaaa7a620a.1537243720.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] test-reach: add run_three_modes method","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-18T04:08:44Z","receivedAt":"2018-09-18T04:08:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe 'test_three_modes' method assumes we are using the 'test-tool\nreach' command for our test. However, we may want to use the data\nshape of our commit graph and the three modes (no commit-graph,\nfull commit-graph, partial commit-graph) for other git commands.\n\nSplit test_three_modes to be a simple translation on a more general\nrun_three_modes method that executes the given command and tests\nthe actual output to the expected output.\n\nWhile inspecting this code, I realized that the final test for\n'commit_contains --tag' is silently dropping the '--tag' argument.\nIt should be quoted to include both.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 14 +++++++++-----\n 1 file changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex d139a00d1d..1b18e12a4e 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -53,18 +53,22 @@ test_expect_success 'setup' '\n \tgit config core.commitGraph true\n '\n \n-test_three_modes () {\n+run_three_modes () {\n \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t$1 <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-full .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t$1 <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-half .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t$1 <input >actual &&\n \ttest_cmp expect actual\n }\n \n+test_three_modes () {\n+\trun_three_modes \"test-tool reach $1\"\n+}\n+\n test_expect_success 'ref_newer:miss' '\n \tcat >input <<-\\EOF &&\n \tA:commit-5-7\n@@ -219,7 +223,7 @@ test_expect_success 'commit_contains:hit' '\n \tEOF\n \techo \"commit_contains(_,A,X,_):1\" >expect &&\n \ttest_three_modes commit_contains &&\n-\ttest_three_modes commit_contains --tag\n+\ttest_three_modes \"commit_contains --tag\"\n '\n \n test_expect_success 'commit_contains:miss' '\n-- \ngitgitgadget\n\n"},{"id":"358380","messageId":"30dee58c615701c8810d055008a687fca278888a.1537243720.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] test-reach: add rev-list tests","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-18T04:08:46Z","receivedAt":"2018-09-18T04:08:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe rev-list command is critical to Git's functionality. Ensure it\nworks in the three commit-graph environments constructed in\nt6600-test-reach.sh. Here are a few important types of rev-list\noperations:\n\n* Basic: git rev-list --topo-order HEAD\n* Range: git rev-list --topo-order compare..HEAD\n* Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n* Symmetric Difference: git rev-list --topo-order compare...HEAD\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 84 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 84 insertions(+)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 1b18e12a4e..2fcaa39077 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -243,4 +243,88 @@ test_expect_success 'commit_contains:miss' '\n \ttest_three_modes commit_contains --tag\n '\n \n+test_expect_success 'rev-list: basic topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 commit-3-3 commit-2-3 commit-1-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: first-parent topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --first-parent --topo-order commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-3-3..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-3-8..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: first-parent range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --first-parent --topo-order commit-3-8..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: ancestry-path topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\"\n+'\n+\n+test_expect_success 'rev-list: symmetric difference topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t\tcommit-3-8 commit-2-8 commit-1-8 \\\n+\t\tcommit-3-7 commit-2-7 commit-1-7 \\\n+\t>expect &&\n+\trun_three_modes \"git rev-list --topo-order commit-3-8...commit-6-6\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"358381","messageId":"a74ae13d4e207acec20b53bf48f24a2de192ad97.1537243720.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] revision.c: begin refactoring --topo-order logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-18T04:08:47Z","receivedAt":"2018-09-18T04:08:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen running 'git rev-list --topo-order' and its kin, the topo_order\nsetting in struct rev_info implies the limited setting. This means\nthat the following things happen during prepare_revision_walk():\n\n* revs->limited implies we run limit_list() to walk the entire\n  reachable set. There are some short-cuts here, such as if we\n  perform a range query like 'git rev-list COMPARE..HEAD' and we\n  can stop limit_list() when all queued commits are uninteresting.\n\n* revs->topo_order implies we run sort_in_topological_order(). See\n  the implementation of that method in commit.c. It implies that\n  the full set of commits to order is in the given commit_list.\n\nThese two methods imply that a 'git rev-list --topo-order HEAD'\ncommand must walk the entire reachable set of commits _twice_ before\nreturning a single result.\n\nIf we have a commit-graph file with generation numbers computed, then\nthere is a better way. This patch introduces some necessary logic\nredirection when we are in this situation.\n\nIn v2.18.0, the commit-graph file contains zero-valued bytes in the\npositions where the generation number is stored in v2.19.0 and later.\nThus, we use generation_numbers_enabled() to check if the commit-graph\nis available and has non-zero generation numbers.\n\nWhen setting revs->limited only because revs->topo_order is true,\nonly do so if generation numbers are not available. There is no\nreason to use the new logic as it will behave similarly when all\ngeneration numbers are INFINITY or ZERO.\n\nIn prepare_revision_walk(), if we have revs->topo_order but not\nrevs->limited, then we trigger the new logic. It breaks the logic\ninto three pieces, to fit with the existing framework:\n\n1. init_topo_walk() fills a new struct topo_walk_info in the rev_info\n   struct. We use the presence of this struct as a signal to use the\n   new methods during our walk. In this patch, this method simply\n   calls limit_list() and sort_in_topological_order(). In the future,\n   this method will set up a new data structure to perform that logic\n   in-line.\n\n2. next_topo_commit() provides get_revision_1() with the next topo-\n   ordered commit in the list. Currently, this simply pops the commit\n   from revs->commits.\n\n3. expand_topo_walk() provides get_revision_1() with a way to signal\n   walking beyond the latest commit. Currently, this calls\n   add_parents_to_list() exactly like the old logic.\n\nWhile this commit presents method redirection for performing the\nexact same logic as before, it allows the next commit to focus only\non the new logic.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 42 ++++++++++++++++++++++++++++++++++++++----\n revision.h |  4 ++++\n 2 files changed, 42 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex e18bd530e4..2dcde8a8ac 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -25,6 +25,7 @@\n #include \"worktree.h\"\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n+#include \"commit-graph.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2454,7 +2455,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \tif (revs->diffopt.objfind)\n \t\trevs->simplify_history = 0;\n \n-\tif (revs->topo_order)\n+\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n \t\trevs->limited = 1;\n \n \tif (revs->prune_data.nr) {\n@@ -2892,6 +2893,33 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n+struct topo_walk_info {};\n+\n+static void init_topo_walk(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info;\n+\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n+\tinfo = revs->topo_walk_info;\n+\tmemset(info, 0, sizeof(struct topo_walk_info));\n+\n+\tlimit_list(revs);\n+\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+}\n+\n+static struct commit *next_topo_commit(struct rev_info *revs)\n+{\n+\treturn pop_commit(&revs->commits);\n+}\n+\n+static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n+{\n+\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\t\tif (!revs->ignore_missing_links)\n+\t\t\tdie(\"Failed to traverse parents of commit %s\",\n+\t\t\t    oid_to_hex(&commit->object.oid));\n+\t}\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -2928,11 +2956,13 @@ int prepare_revision_walk(struct rev_info *revs)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\n-\tif (revs->limited)\n+\tif (revs->limited) {\n \t\tif (limit_list(revs) < 0)\n \t\t\treturn -1;\n-\tif (revs->topo_order)\n-\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t\tif (revs->topo_order)\n+\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t} else if (revs->topo_order)\n+\t\tinit_topo_walk(revs);\n \tif (revs->line_level_traverse)\n \t\tline_log_filter(revs);\n \tif (revs->simplify_merges)\n@@ -3257,6 +3287,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\tif (revs->reflog_info)\n \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n+\t\telse if (revs->topo_walk_info)\n+\t\t\tcommit = next_topo_commit(revs);\n \t\telse\n \t\t\tcommit = pop_commit(&revs->commits);\n \n@@ -3278,6 +3310,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\t\tif (revs->reflog_info)\n \t\t\t\ttry_to_simplify_commit(revs, commit);\n+\t\t\telse if (revs->topo_walk_info)\n+\t\t\t\texpand_topo_walk(revs, commit);\n \t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n \t\t\t\tif (!revs->ignore_missing_links)\n \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\ndiff --git a/revision.h b/revision.h\nindex 2b30ac270d..fd4154ff75 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -56,6 +56,8 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct topo_walk_info;\n+\n struct rev_info {\n \t/* Starting list */\n \tstruct commit_list *commits;\n@@ -245,6 +247,8 @@ struct rev_info {\n \tconst char *break_bar;\n \n \tstruct revision_sources *sources;\n+\n+\tstruct topo_walk_info *topo_walk_info;\n };\n \n int ref_excluded(struct string_list *, const char *path);\n-- \ngitgitgadget\n\n"},{"id":"358383","messageId":"0e64fc144cf4eb8b15c9e8453cf8af83330930e2.1537243720.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 5/6] commit/revisions: bookkeeping before refactoring","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-18T04:08:48Z","receivedAt":"2018-09-18T04:08:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThere are a few things that need to move around a little before\nmaking a big refactoring in the topo-order logic:\n\n1. We need access to record_author_date() and\n   compare_commits_by_author_date() in revision.c. These are used\n   currently by sort_in_topological_order() in commit.c.\n\n2. Moving these methods to commit.h requires adding the author_slab\n   definition to commit.h.\n\n3. The add_parents_to_list() method in revision.c performs logic\n   around the UNINTERESTING flag and other special cases depending\n   on the struct rev_info. Allow this method to ignore a NULL 'list'\n   parameter, as we will not be populating the list for our walk.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n commit.c   | 11 ++++-------\n commit.h   |  8 ++++++++\n revision.c |  6 ++++--\n 3 files changed, 16 insertions(+), 9 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex d0f199e122..f68e04b2f1 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -655,11 +655,8 @@ struct commit *pop_commit(struct commit_list **stack)\n /* count number of children that have not been emitted */\n define_commit_slab(indegree_slab, int);\n \n-/* record author-date for each commit object */\n-define_commit_slab(author_date_slab, timestamp_t);\n-\n-static void record_author_date(struct author_date_slab *author_date,\n-\t\t\t       struct commit *commit)\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit)\n {\n \tconst char *buffer = get_commit_buffer(commit, NULL);\n \tstruct ident_split ident;\n@@ -684,8 +681,8 @@ fail_exit:\n \tunuse_commit_buffer(commit, buffer);\n }\n \n-static int compare_commits_by_author_date(const void *a_, const void *b_,\n-\t\t\t\t\t  void *cb_data)\n+int compare_commits_by_author_date(const void *a_, const void *b_,\n+\t\t\t\t   void *cb_data)\n {\n \tconst struct commit *a = a_, *b = b_;\n \tstruct author_date_slab *author_date = cb_data;\ndiff --git a/commit.h b/commit.h\nindex 2b1a734388..ff0eb5f8ef 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -8,6 +8,7 @@\n #include \"gpg-interface.h\"\n #include \"string-list.h\"\n #include \"pretty.h\"\n+#include \"commit-slab.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n #define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n@@ -328,6 +329,13 @@ extern int remove_signature(struct strbuf *buf);\n  */\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n+/* record author-date for each commit object */\n+define_commit_slab(author_date_slab, timestamp_t);\n+\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit);\n+\n+int compare_commits_by_author_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \ndiff --git a/revision.c b/revision.c\nindex 2dcde8a8ac..92012d5f45 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -808,7 +808,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\treturn 0;\n \t}\n@@ -847,7 +848,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\tif (revs->first_parent_only)\n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"358382","messageId":"3b185ac3b1cd54e0ffee8e5b42cc0ea3d1fac3c7.1537243720.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] revision.c: refactor basic topo-order logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-18T04:08:50Z","receivedAt":"2018-09-18T04:08:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen running a command like 'git rev-list --topo-order HEAD',\nGit performed the following steps:\n\n1. Run limit_list(), which parses all reachable commits,\n   adds them to a linked list, and distributes UNINTERESTING\n   flags. If all unprocessed commits are UNINTERESTING, then\n   it may terminate without walking all reachable commits.\n   This does not occur if we do not specify UNINTERESTING\n   commits.\n\n2. Run sort_in_topological_order(), which is an implementation\n   of Kahn's algorithm. It first iterates through the entire\n   set of important commits and computes the in-degree of each\n   (plus one, as we use 'zero' as a special value here). Then,\n   we walk the commits in priority order, adding them to the\n   priority queue if and only if their in-degree is one. As\n   we remove commits from this priority queue, we decrement the\n   in-degree of their parents.\n\n3. While we are peeling commits for output, get_revision_1()\n   uses pop_commit on the full list of commits computed by\n   sort_in_topological_order().\n\nIn the new algorithm, these three steps correspond to three\ndifferent commit walks. We run these walks simultaneously,\nand advance each only as far as necessary to satisfy the\nrequirements of the 'higher order' walk. We know when we can\npause each walk by using generation numbers from the commit-\ngraph feature.\n\nRecall that the generation number of a commit satisfies:\n\n* If the commit has at least one parent, then the generation\n  number is one more than the maximum generation number among\n  its parents.\n\n* If the commit has no parent, then the generation number is one.\n\nThere are two special generation numbers:\n\n* GENERATION_NUMBER_INFINITY: this value is 0xffffffff and\n  indicates that the commit is not stored in the commit-graph and\n  the generation number was not previously calculated.\n\n* GENERATION_NUMBER_ZERO: this value (0) is a special indicator\n  to say that the commit-graph was generated by a version of Git\n  that does not compute generation numbers (such as v2.18.0).\n\nSince we use generation_numbers_enabled() before using the new\nalgorithm, we do not need to worry about GENERATION_NUMBER_ZERO.\nHowever, the existence of GENERATION_NUMBER_INFINITY implies the\nfollowing weaker statement than the usual we expect from\ngeneration numbers:\n\n    If A and B are commits with generation numbers gen(A) and\n    gen(B) and gen(A) < gen(B), then A cannot reach B.\n\nThus, we will walk in each of our stages until the \"maximum\nunexpanded generation number\" is strictly lower than the\ngeneration number of a commit we are about to use.\n\nThe walks are as follows:\n\n1. EXPLORE: using the explore_queue priority queue (ordered by\n   maximizing the generation number), parse each reachable\n   commit until all commits in the queue have generation\n   number strictly lower than needed. During this walk, update\n   the UNINTERESTING flags as necessary.\n\n2. INDEGREE: using the indegree_queue priority queue (ordered\n   by maximizing the generation number), add one to the in-\n   degree of each parent for each commit that is walked. Since\n   we walk in order of decreasing generation number, we know\n   that discovering an in-degree value of 0 means the value for\n   that commit was not initialized, so should be initialized to\n   two. (Recall that in-degree value \"1\" is what we use to say a\n   commit is ready for output.) As we iterate the parents of a\n   commit during this walk, ensure the EXPLORE walk has walked\n   beyond their generation numbers.\n\n3. TOPO: using the topo_queue priority queue (ordered based on\n   the sort_order given, which could be commit-date, author-\n   date, or typical topo-order which treats the queue as a LIFO\n   stack), remove a commit from the queue and decrement the\n   in-degree of each parent. If a parent has an in-degree of\n   one, then we add it to the topo_queue. Before we decrement\n   the in-degree, however, ensure the INDEGREE walk has walked\n   beyond that generation number.\n\nThe implementations of these walks are in the following methods:\n\n* explore_walk_step and explore_to_depth\n* indegree_walk_step and compute_indegrees_to_depth\n* next_topo_commit and expand_topo_walk\n\nThese methods have some patterns that may seem strange at first,\nbut they are probably carry-overs from their equivalents in\nlimit_list and sort_in_topological_order.\n\nOne thing that is missing from this implementation is a proper\nway to stop walking when the entire queue is UNINTERESTING, so\nthis implementation is not enabled by comparisions, such as in\n'git rev-list --topo-order A..B'. This can be updated in the\nfuture.\n\nIn my local testing, I used the following Git commands on the\nLinux repository in three modes: HEAD~1 with no commit-graph,\nHEAD~1 with a commit-graph, and HEAD with a commit-graph. This\nallows comparing the benefits we get from parsing commits from\nthe commit-graph and then again the benefits we get by\nrestricting the set of commits we walk.\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nThis speedup is due to a few things. First, the new generation-\nnumber-enabled algorithm walks commits on order of the number of\nresults output (subject to some branching structure expectations).\nSince we limit to 100 results, we are running a query similar to\nfilling a single page of results. Second, when specifying a path,\nwe must parse the root tree object for each commit we walk. The\nprevious benefits from the commit-graph are entirely from reading\nthe commit-graph instead of parsing commits. Since we need to\nparse trees for the same number of commits as before, we slow\ndown significantly from the non-path-based query.\n\nFor the test above, I specifically selected a path that is changed\nfrequently, including by merge commits. A less-frequently-changed\npath (such as 'README') has similar end-to-end time since we need\nto walk the same number of commits (before determining we do not\nhave 100 hits). However, get get the benefit that the output is\npresented to the user as it is discovered, much the same as a\nnormal 'git log' command (no '--topo-order'). This is an improved\nuser experience, even if the command has the same runtime.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n object.h   |   4 +-\n revision.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n revision.h |   2 +\n 3 files changed, 194 insertions(+), 8 deletions(-)\n\ndiff --git a/object.h b/object.h\nindex 0feb90ae61..796792cb32 100644\n--- a/object.h\n+++ b/object.h\n@@ -59,7 +59,7 @@ struct object_array {\n \n /*\n  * object flag allocation:\n- * revision.h:               0---------10                              2526\n+ * revision.h:               0---------10                              25----28\n  * fetch-pack.c:             01\n  * negotiator/default.c:       2--5\n  * walker.c:                 0-2\n@@ -78,7 +78,7 @@ struct object_array {\n  * builtin/show-branch.c:    0-------------------------------------------26\n  * builtin/unpack-objects.c:                                 2021\n  */\n-#define FLAG_BITS  27\n+#define FLAG_BITS  29\n \n /*\n  * The object type is stored in 3 bits.\ndiff --git a/revision.c b/revision.c\nindex 92012d5f45..c5d0cb6599 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -26,6 +26,7 @@\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"prio-queue.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2895,30 +2896,213 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct topo_walk_info {};\n+define_commit_slab(indegree_slab, int);\n+\n+struct topo_walk_info {\n+\tuint32_t min_generation;\n+\tstruct prio_queue explore_queue;\n+\tstruct prio_queue indegree_queue;\n+\tstruct prio_queue topo_queue;\n+\tstruct indegree_slab indegree;\n+\tstruct author_date_slab author_date;\n+};\n+\n+static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n+{\n+\tif (c->object.flags & flag)\n+\t\treturn;\n+\n+\tc->object.flags |= flag;\n+\tprio_queue_put(q, c);\n+}\n+\n+static void explore_walk_step(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit_list *p;\n+\tstruct commit *c = prio_queue_get(&info->explore_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tif (revs->max_age != -1 && (c->date < revs->max_age))\n+\t\tc->object.flags |= UNINTERESTING;\n+\n+\tif (add_parents_to_list(revs, c, NULL, NULL) < 0)\n+\t\treturn;\n+\n+\tif (c->object.flags & UNINTERESTING)\n+\t\tmark_parents_uninteresting(c);\n+\n+\tfor (p = c->parents; p; p = p->next)\n+\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n+}\n+\n+static void explore_to_depth(struct rev_info *revs,\n+\t\t\t     uint32_t gen)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n+\t       c->generation >= gen)\n+\t\texplore_walk_step(revs);\n+}\n+\n+static void indegree_walk_step(struct rev_info *revs)\n+{\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c = prio_queue_get(&info->indegree_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\texplore_to_depth(revs, c->generation);\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tfor (p = c->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\tif (*pi)\n+\t\t\t(*pi)++;\n+\t\telse\n+\t\t\t*pi = 2;\n+\n+\t\ttest_flag_and_insert(&info->indegree_queue, parent, TOPO_WALK_INDEGREE);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n+\t}\n+}\n+\n+static void compute_indegrees_to_depth(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n+\t       c->generation >= info->min_generation)\n+\t\tindegree_walk_step(revs);\n+}\n \n static void init_topo_walk(struct rev_info *revs)\n {\n \tstruct topo_walk_info *info;\n+\tstruct commit_list *list;\n \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n \tinfo = revs->topo_walk_info;\n \tmemset(info, 0, sizeof(struct topo_walk_info));\n \n-\tlimit_list(revs);\n-\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\tinit_indegree_slab(&info->indegree);\n+\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n+\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n+\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n+\n+\tswitch (revs->sort_order) {\n+\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n+\t\tinfo->topo_queue.compare = NULL;\n+\t\tbreak;\n+\tcase REV_SORT_BY_COMMIT_DATE:\n+\t\tinfo->topo_queue.compare = compare_commits_by_commit_date;\n+\t\tbreak;\n+\tcase REV_SORT_BY_AUTHOR_DATE:\n+\t\tinit_author_date_slab(&info->author_date);\n+\t\tinfo->topo_queue.compare = compare_commits_by_author_date;\n+\t\tinfo->topo_queue.cb_data = &info->author_date;\n+\t\tbreak;\n+\t}\n+\n+\tinfo->explore_queue.compare = compare_commits_by_gen_then_commit_date;\n+\tinfo->indegree_queue.compare = compare_commits_by_gen_then_commit_date;\n+\n+\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n+\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n+\n+\t\tif (parse_commit_gently(c, 1))\n+\t\t\tcontinue;\n+\t\tif (c->generation < info->min_generation)\n+\t\t\tinfo->min_generation = c->generation;\n+\t}\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n+\n+\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n+\t\t\trecord_author_date(&info->author_date, c);\n+\t}\n+\tcompute_indegrees_to_depth(revs);\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\n+\t\tif (*(indegree_slab_at(&info->indegree, c)) == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, c);\n+\t}\n+\n+\t/*\n+\t * This is unfortunate; the initial tips need to be shown\n+\t * in the order given from the revision traversal machinery.\n+\t */\n+\tif (revs->sort_order == REV_SORT_IN_GRAPH_ORDER)\n+\t\tprio_queue_reverse(&info->topo_queue);\n }\n \n static struct commit *next_topo_commit(struct rev_info *revs)\n {\n-\treturn pop_commit(&revs->commits);\n+\tstruct commit *c;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\n+\t/* pop next off of topo_queue */\n+\tc = prio_queue_get(&info->topo_queue);\n+\n+\tif (c)\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 0;\n+\n+\treturn c;\n }\n \n static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n {\n-\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tif (add_parents_to_list(revs, commit, NULL, NULL) < 0) {\n \t\tif (!revs->ignore_missing_links)\n \t\t\tdie(\"Failed to traverse parents of commit %s\",\n-\t\t\t    oid_to_hex(&commit->object.oid));\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t}\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi;\n+\n+\t\tif (parse_commit_gently(parent, 1) < 0)\n+\t\t\tcontinue;\n+\n+\t\tif (parent->generation < info->min_generation) {\n+\t\t\tinfo->min_generation = parent->generation;\n+\t\t\tcompute_indegrees_to_depth(revs);\n+\t\t}\n+\n+\t\tpi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\t(*pi)--;\n+\t\tif (*pi == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, parent);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n \t}\n }\n \ndiff --git a/revision.h b/revision.h\nindex fd4154ff75..b20c16c0e0 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -24,6 +24,8 @@\n #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n #define TRACK_LINEAR\t(1u<<26)\n #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define TOPO_WALK_EXPLORED (1u<<27)\n+#define TOPO_WALK_INDEGREE (1u<<28)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \ngitgitgadget\n"},{"id":"358390","messageId":"87y3bzjqtm.fsf@evledraar.gmail.com","threadId":"49229","inReplyTo":"3b185ac3b1cd54e0ffee8e5b42cc0ea3d1fac3c7.1537243720.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/6] revision.c: refactor basic topo-order logic","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-18T05:51:49Z","receivedAt":"2018-09-18T05:51:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Sep 18 2018, Derrick Stolee via GitGitGadget wrote:\n\n> diff --git a/revision.h b/revision.h\n> index fd4154ff75..b20c16c0e0 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -24,6 +24,8 @@\n>  #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n>  #define TRACK_LINEAR\t(1u<<26)\n>  #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n> +#define TOPO_WALK_EXPLORED (1u<<27)\n> +#define TOPO_WALK_INDEGREE (1u<<28)\n\nMaybe lead with a commit to indent these bitfield defines so this change\ndoesn't end up making these two new flags (due to the length of the\nname) misaligned.\n"},{"id":"358391","messageId":"87worjjq6o.fsf@evledraar.gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/6] Use generation numbers for --topo-order","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-18T06:05:35Z","receivedAt":"2018-09-18T06:05:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Sep 18 2018, Derrick Stolee via GitGitGadget wrote:\n\nThanks. Good to see the commit graph used for more stuff.\n\n> On the Linux repository, I got the following performance results when\n> comparing to the previous version with or without a commit-graph:\n>\n> Test: git rev-list --topo-order -100 HEAD\n> HEAD~1, no commit-graph: 6.80 s\n> HEAD~1, w/ commit-graph: 0.77 s\n>   HEAD, w/ commit-graph: 0.02 s\n>\n> Test: git rev-list --topo-order -100 HEAD -- tools\n> HEAD~1, no commit-graph: 9.63 s\n> HEAD~1, w/ commit-graph: 6.06 s\n>   HEAD, w/ commit-graph: 0.06 s\n\nIt would be great if this were made into a t/perf/ test shipped with\nthis series, that would be later quoted in a commit, as in\ne.g. 3b41fb0cb2 (\"fsck: use oidset instead of oid_array for skipList\",\n2018-09-03).\n\nAlthough generalizing that \"-- tools\" part (i.e. finding a candidate\ndir) will require some heuristic, but would make it useful when running\nthis against other erpos.\n\n> If you want to read this series but are unfamiliar with the commit-graph and\n> generation numbers, then I recommend reading\n> Documentation/technical/commit-graph.txt or a blob post [1] I wrote on the\n> subject. In particular, the three-part walk described in \"revision.c:\n> refactor basic topo-order logic\" is present (but underexplained) as an\n> animated PNG [2].\n\nWe discussed some of this in private E-Mail, and this isn't really\nfeedback on *this* series in particular, just on the general\ncommit-graph work.\n\nRight now git-config(1) just matter-of-factly says how to enable it, and\npoints to git-commit-graph(1) for further info, which just shows how to\nrun the tool. But nothing's describing what stuff is sped up, and those\nsorts of docs aren't being updated as new optimizations (e.g. this\n--topo-order walk) are added.\n\nFor that you need to scour a combination of your blogpost & commits in\ngit.git (with quoted perf numbers).\n"},{"id":"358409","messageId":"20180918180200.GD27036@localhost","threadId":"49229","inReplyTo":"404c9186080ecee6c1cc39a6dcd17deaaa7a620a.1537243720.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/6] test-reach: add run_three_modes method","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-09-18T18:02:00Z","receivedAt":"2018-09-18T18:02:08Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Sep 17, 2018 at 09:08:44PM -0700, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <dstolee@microsoft.com>\n> \n> The 'test_three_modes' method assumes we are using the 'test-tool\n> reach' command for our test. However, we may want to use the data\n> shape of our commit graph and the three modes (no commit-graph,\n> full commit-graph, partial commit-graph) for other git commands.\n> \n> Split test_three_modes to be a simple translation on a more general\n> run_three_modes method that executes the given command and tests\n> the actual output to the expected output.\n> \n> While inspecting this code, I realized that the final test for\n> 'commit_contains --tag' is silently dropping the '--tag' argument.\n> It should be quoted to include both.\n\nNit: while quoting the function's arguments does fix the issue, it\nleaves the tests prone to the same issue in the future.  Wouldn't it\nbe better to use $@ inside the function to refer to all its arguments?\n\n\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  t/t6600-test-reach.sh | 14 +++++++++-----\n>  1 file changed, 9 insertions(+), 5 deletions(-)\n> \n> diff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\n> index d139a00d1d..1b18e12a4e 100755\n> --- a/t/t6600-test-reach.sh\n> +++ b/t/t6600-test-reach.sh\n> @@ -53,18 +53,22 @@ test_expect_success 'setup' '\n>  \tgit config core.commitGraph true\n>  '\n>  \n> -test_three_modes () {\n> +run_three_modes () {\n>  \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n> -\ttest-tool reach $1 <input >actual &&\n> +\t$1 <input >actual &&\n>  \ttest_cmp expect actual &&\n>  \tcp commit-graph-full .git/objects/info/commit-graph &&\n> -\ttest-tool reach $1 <input >actual &&\n> +\t$1 <input >actual &&\n>  \ttest_cmp expect actual &&\n>  \tcp commit-graph-half .git/objects/info/commit-graph &&\n> -\ttest-tool reach $1 <input >actual &&\n> +\t$1 <input >actual &&\n>  \ttest_cmp expect actual\n>  }\n>  \n> +test_three_modes () {\n> +\trun_three_modes \"test-tool reach $1\"\n> +}\n> +\n>  test_expect_success 'ref_newer:miss' '\n>  \tcat >input <<-\\EOF &&\n>  \tA:commit-5-7\n> @@ -219,7 +223,7 @@ test_expect_success 'commit_contains:hit' '\n>  \tEOF\n>  \techo \"commit_contains(_,A,X,_):1\" >expect &&\n>  \ttest_three_modes commit_contains &&\n> -\ttest_three_modes commit_contains --tag\n> +\ttest_three_modes \"commit_contains --tag\"\n>  '\n>  \n>  test_expect_success 'commit_contains:miss' '\n> -- \n> gitgitgadget\n> \n"},{"id":"358480","messageId":"xmqqbm8t473q.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"20180918180200.GD27036@localhost","subject":"Re: [PATCH v2 2/6] test-reach: add run_three_modes method","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-19T19:31:21Z","receivedAt":"2018-09-19T19:31:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor <szeder.dev@gmail.com> writes:\n\n>> While inspecting this code, I realized that the final test for\n>> 'commit_contains --tag' is silently dropping the '--tag' argument.\n>> It should be quoted to include both.\n>\n> Nit: while quoting the function's arguments does fix the issue, it\n> leaves the tests prone to the same issue in the future.  Wouldn't it\n> be better to use $@ inside the function to refer to all its arguments?\n\nIOW, do it more like this?\n\n>> -test_three_modes () {\n>> +run_three_modes () {\n>>  \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n>> -\ttest-tool reach $1 <input >actual &&\n>> +\t$1 <input >actual &&\n\n\t\"$@\" <input >actual\n\ni.e. treat each parameter as separate things without further getting\nsplit at $IFS and ...\n\n>> +test_three_modes () {\n>> +\trun_three_modes \"test-tool reach $1\"\n\n\trun_three_modes test-tool reach \"$1\"\n\n... make sure there three things are sent as separate, by quoting\n\"$1\" inside dq.\n\nI think that makes sense.\n\n"},{"id":"358482","messageId":"xmqq7ejh46rm.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"xmqqbm8t473q.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 2/6] test-reach: add run_three_modes method","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-19T19:38:37Z","receivedAt":"2018-09-19T19:38:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> SZEDER Gábor <szeder.dev@gmail.com> writes:\n>\n>>> While inspecting this code, I realized that the final test for\n>>> 'commit_contains --tag' is silently dropping the '--tag' argument.\n>>> It should be quoted to include both.\n>>\n>> Nit: while quoting the function's arguments does fix the issue, it\n>> leaves the tests prone to the same issue in the future.  Wouldn't it\n>> be better to use $@ inside the function to refer to all its arguments?\n>\n> IOW, do it more like this?\n>\n>>> -test_three_modes () {\n>>> +run_three_modes () {\n>>>  \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n>>> -\ttest-tool reach $1 <input >actual &&\n>>> +\t$1 <input >actual &&\n>\n> \t\"$@\" <input >actual\n>\n> i.e. treat each parameter as separate things without further getting\n> split at $IFS and ...\n>\n>>> +test_three_modes () {\n>>> +\trun_three_modes \"test-tool reach $1\"\n>\n> \trun_three_modes test-tool reach \"$1\"\n>\n> ... make sure there three things are sent as separate, by quoting\n> \"$1\" inside dq.\n>\n> I think that makes sense.\n\nI also noticed that 2/6 made \"commti_contains --tag\" enclosed in dq\npair for one test, but the next test after it has the identical one.\n\nHere is what I queued in the meantime.\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 1b18e12a4e..1377849bf8 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -55,18 +55,18 @@ test_expect_success 'setup' '\n \n run_three_modes () {\n \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n-\t$1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-full .git/objects/info/commit-graph &&\n-\t$1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-half .git/objects/info/commit-graph &&\n-\t$1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual\n }\n \n test_three_modes () {\n-\trun_three_modes \"test-tool reach $1\"\n+\trun_three_modes test-tool reach \"$1\"\n }\n \n test_expect_success 'ref_newer:miss' '\n@@ -223,7 +223,7 @@ test_expect_success 'commit_contains:hit' '\n \tEOF\n \techo \"commit_contains(_,A,X,_):1\" >expect &&\n \ttest_three_modes commit_contains &&\n-\ttest_three_modes \"commit_contains --tag\"\n+\ttest_three_modes commit_contains --tag\n '\n \n test_expect_success 'commit_contains:miss' '\n-- \n2.19.0-216-g2d3b1c576c\n\n"},{"id":"358541","messageId":"xmqq7ejfzx3u.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"xmqq7ejh46rm.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 2/6] test-reach: add run_three_modes method","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-20T21:18:29Z","receivedAt":"2018-09-20T21:18:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> I also noticed that 2/6 made \"commti_contains --tag\" enclosed in dq\n> pair for one test, but the next test after it has the identical one.\n>\n> Here is what I queued in the meantime.\n> ...\n\nAnd of course, I find out that 3/6 needs a matching update after\nI've almost finished day's integration cycle, and need to redo the\nwhole thing X-<.\n\nHere is a squashable update for 3/6 to match the proposed change.\n\n-- >8 --\nSubject: fixup! test-reach: add rev-list tests\n\n t/t6600-test-reach.sh | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 990ab56e7a..cf9179bdb8 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -252,7 +252,7 @@ test_expect_success 'rev-list: basic topo-order' '\n \t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n \t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n \t>expect &&\n-\trun_three_modes \"git rev-list --topo-order commit-6-6\"\n+\trun_three_modes git rev-list --topo-order commit-6-6\n '\n \n test_expect_success 'rev-list: first-parent topo-order' '\n@@ -264,7 +264,7 @@ test_expect_success 'rev-list: first-parent topo-order' '\n \t\tcommit-6-2 \\\n \t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n \t>expect &&\n-\trun_three_modes \"git rev-list --first-parent --topo-order commit-6-6\"\n+\trun_three_modes git rev-list --first-parent --topo-order commit-6-6\n '\n \n test_expect_success 'rev-list: range topo-order' '\n@@ -276,7 +276,7 @@ test_expect_success 'rev-list: range topo-order' '\n \t\tcommit-6-2 commit-5-2 commit-4-2 \\\n \t\tcommit-6-1 commit-5-1 commit-4-1 \\\n \t>expect &&\n-\trun_three_modes \"git rev-list --topo-order commit-3-3..commit-6-6\"\n+\trun_three_modes git rev-list --topo-order commit-3-3..commit-6-6\n '\n \n test_expect_success 'rev-list: range topo-order' '\n@@ -288,7 +288,7 @@ test_expect_success 'rev-list: range topo-order' '\n \t\tcommit-6-2 commit-5-2 commit-4-2 \\\n \t\tcommit-6-1 commit-5-1 commit-4-1 \\\n \t>expect &&\n-\trun_three_modes \"git rev-list --topo-order commit-3-8..commit-6-6\"\n+\trun_three_modes git rev-list --topo-order commit-3-8..commit-6-6\n '\n \n test_expect_success 'rev-list: first-parent range topo-order' '\n@@ -300,7 +300,7 @@ test_expect_success 'rev-list: first-parent range topo-order' '\n \t\tcommit-6-2 \\\n \t\tcommit-6-1 commit-5-1 commit-4-1 \\\n \t>expect &&\n-\trun_three_modes \"git rev-list --first-parent --topo-order commit-3-8..commit-6-6\"\n+\trun_three_modes git rev-list --first-parent --topo-order commit-3-8..commit-6-6\n '\n \n test_expect_success 'rev-list: ancestry-path topo-order' '\n@@ -310,7 +310,7 @@ test_expect_success 'rev-list: ancestry-path topo-order' '\n \t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 \\\n \t\tcommit-6-3 commit-5-3 commit-4-3 \\\n \t>expect &&\n-\trun_three_modes \"git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\"\n+\trun_three_modes git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\n '\n \n test_expect_success 'rev-list: symmetric difference topo-order' '\n@@ -324,7 +324,7 @@ test_expect_success 'rev-list: symmetric difference topo-order' '\n \t\tcommit-3-8 commit-2-8 commit-1-8 \\\n \t\tcommit-3-7 commit-2-7 commit-1-7 \\\n \t>expect &&\n-\trun_three_modes \"git rev-list --topo-order commit-3-8...commit-6-6\"\n+\trun_three_modes git rev-list --topo-order commit-3-8...commit-6-6\n '\n \n test_done\n"},{"id":"358577","messageId":"2d72ae31-774e-d0a4-8b6c-ea125377eac5@gmail.com","threadId":"49229","inReplyTo":"87worjjq6o.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/6] Use generation numbers for --topo-order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-21T15:47:45Z","receivedAt":"2018-09-21T15:47:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/18/2018 2:05 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Tue, Sep 18 2018, Derrick Stolee via GitGitGadget wrote:\n>\n> Thanks. Good to see the commit graph used for more stuff.\n>\n>> On the Linux repository, I got the following performance results when\n>> comparing to the previous version with or without a commit-graph:\n>>\n>> Test: git rev-list --topo-order -100 HEAD\n>> HEAD~1, no commit-graph: 6.80 s\n>> HEAD~1, w/ commit-graph: 0.77 s\n>>    HEAD, w/ commit-graph: 0.02 s\n>>\n>> Test: git rev-list --topo-order -100 HEAD -- tools\n>> HEAD~1, no commit-graph: 9.63 s\n>> HEAD~1, w/ commit-graph: 6.06 s\n>>    HEAD, w/ commit-graph: 0.06 s\n> It would be great if this were made into a t/perf/ test shipped with\n> this series, that would be later quoted in a commit, as in\n> e.g. 3b41fb0cb2 (\"fsck: use oidset instead of oid_array for skipList\",\n> 2018-09-03).\n>\n> Although generalizing that \"-- tools\" part (i.e. finding a candidate\n> dir) will require some heuristic, but would make it useful when running\n> this against other erpos.\n\nt/perf/p4211-line-log.sh has the following test:\n\n\n     test_perf 'git log --oneline --raw --parents -1000' '\n             git log --oneline --raw --parents -1000 >/dev/null\n     '\n\nWe could add the following to the end of that script to get similar \nvalues, since it already selects a file randomly at the top of the script:\n\n     test_perf 'git log --oneline --raw --parents -1000 -- <file>' '\n             git log --oneline --raw --parents -1000 -- $file >/dev/null\n     '\n\n>\n>> If you want to read this series but are unfamiliar with the commit-graph and\n>> generation numbers, then I recommend reading\n>> Documentation/technical/commit-graph.txt or a blob post [1] I wrote on the\n>> subject. In particular, the three-part walk described in \"revision.c:\n>> refactor basic topo-order logic\" is present (but underexplained) as an\n>> animated PNG [2].\n> We discussed some of this in private E-Mail, and this isn't really\n> feedback on *this* series in particular, just on the general\n> commit-graph work.\n>\n> Right now git-config(1) just matter-of-factly says how to enable it, and\n> points to git-commit-graph(1) for further info, which just shows how to\n> run the tool. But nothing's describing what stuff is sped up, and those\n> sorts of docs aren't being updated as new optimizations (e.g. this\n> --topo-order walk) are added.\n>\n> For that you need to scour a combination of your blogpost & commits in\n> git.git (with quoted perf numbers).\n\nThanks for reminding me. I have this on my list of TODOs.\n\n-Stolee\n\n"},{"id":"358620","messageId":"pull.25.v3.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v3 0/7] Use generation numbers for --topo-order","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:26Z","receivedAt":"2018-09-21T17:39:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series performs a decently-sized refactoring of the revision-walk\nmachinery. Well, \"refactoring\" is probably the wrong word, as I don't\nactually remove the old code. Instead, when we see certain options in the\n'rev_info' struct, we redirect the commit-walk logic to a new set of methods\nthat distribute the workload differently. By using generation numbers in the\ncommit-graph, we can significantly improve 'git log --graph' commands (and\nthe underlying 'git rev-list --topo-order').\n\nOn the Linux repository, I got the following performance results when\ncomparing to the previous version with or without a commit-graph:\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nIf you want to read this series but are unfamiliar with the commit-graph and\ngeneration numbers, then I recommend reading \nDocumentation/technical/commit-graph.txt or a blob post [1] I wrote on the\nsubject. In particular, the three-part walk described in \"revision.c:\nrefactor basic topo-order logic\" is present (but underexplained) as an\nanimated PNG [2].\n\nSince revision.c is an incredibly important (and old) portion of the\ncodebase -- and because there are so many orthogonal options in 'struct\nrev_info' -- I consider this submission to be \"RFC quality\". That is, I am\nnot confident that I am not missing anything, or that my solution is the\nbest it can be. I did merge this branch with ds/commit-graph-with-grafts and\nthe \"DO-NOT-MERGE: write and read commit-graph always\" commit that computes\na commit-graph with every 'git commit' command. The test suite passed with\nthat change, available on GitHub [3]. To ensure that I cover at least the\ncase I think are interesting, I added tests to t6600-test-reach.sh to verify\nthe walks report the correct results for the three cases there (no\ncommit-graph, full commit-graph, and a partial commit-graph so the walk\nstarts at GENERATION_NUMBER_INFINITY).\n\nOne notable case that is not included in this series is the case of a\nhistory comparison such as 'git rev-list --topo-order A..B'. The existing\ncode in limit_list() has ways to cut the walk short when all pending commits\nare UNINTERESTING. Since this code depends on commit_list instead of the\nprio_queue we are using here, I chose to leave it untouched for now. We can\nrevisit it in a separate series later. Since handle_commit() turns on\nrevs->limited when a commit is UNINTERESTING, we do not hit the new code in\nthis case. Removing this 'revs->limited = 1;' line yields correct results,\nbut the performance is worse.\n\nThis series was based on ds/reachable, but is now based on 'master' to not\nconflict with 182070 \"commit: use timestamp_t for author_date_slab\". There\nis a small conflict with md/filter-trees, because it renamed a flag in\nrevisions.h in the line before I add new flags. Hopefully this conflict is\nnot too difficult to resolve.\n\nChanges in V3: I added a new patch that updates the tab-alignment for flags\nin revision.h before adding new ones (Thanks, Ævar!). Also, I squashed the\nrecommended changes to run_three_modes and test_three_modes from Szeder and\nJunio. Thanks!\n\nThanks, -Stolee\n\n[1] \nhttps://blogs.msdn.microsoft.com/devops/2018/07/09/supercharging-the-git-commit-graph-iii-generations/\nSupercharging the Git Commit Graph III: Generations and Graph Algorithms\n\n[2] \nhttps://msdnshared.blob.core.windows.net/media/2018/06/commit-graph-topo-order-b-a.png\nAnimation showing three-part walk\n\n[3] https://github.com/derrickstolee/git/tree/topo-order/testA branch\ncontaining this series along with commits to compute commit-graph in entire\ntest suite.\n\nCc: avarab@gmail.comCc: szeder.dev@gmail.com\n\nDerrick Stolee (7):\n  prio-queue: add 'peek' operation\n  test-reach: add run_three_modes method\n  test-reach: add rev-list tests\n  revision.c: begin refactoring --topo-order logic\n  commit/revisions: bookkeeping before refactoring\n  revision.h: add whitespace in flag definitions\n  revision.c: refactor basic topo-order logic\n\n commit.c                   |  11 +-\n commit.h                   |   8 ++\n object.h                   |   4 +-\n prio-queue.c               |   9 ++\n prio-queue.h               |   6 +\n revision.c                 | 232 ++++++++++++++++++++++++++++++++++++-\n revision.h                 |  34 +++---\n t/helper/test-prio-queue.c |  10 +-\n t/t6600-test-reach.sh      |  96 ++++++++++++++-\n 9 files changed, 374 insertions(+), 36 deletions(-)\n\n\nbase-commit: 2d3b1c576c85b7f5db1f418907af00ab88e0c303\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-25%2Fderrickstolee%2Ftopo-order%2Fprogress-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-25/derrickstolee/topo-order/progress-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/25\n\nRange-diff vs v2:\n\n 1:  cc1ec4c270 = 1:  cc1ec4c270 prio-queue: add 'peek' operation\n 2:  404c918608 ! 2:  b2a1ade148 test-reach: add run_three_modes method\n     @@ -11,10 +11,6 @@\n          run_three_modes method that executes the given command and tests\n          the actual output to the expected output.\n      \n     -    While inspecting this code, I realized that the final test for\n     -    'commit_contains --tag' is silently dropping the '--tag' argument.\n     -    It should be quoted to include both.\n     -\n          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n      \n      diff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\n     @@ -28,31 +24,22 @@\n      +run_three_modes () {\n       \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n      -\ttest-tool reach $1 <input >actual &&\n     -+\t$1 <input >actual &&\n     ++\t\"$@\" <input >actual &&\n       \ttest_cmp expect actual &&\n       \tcp commit-graph-full .git/objects/info/commit-graph &&\n      -\ttest-tool reach $1 <input >actual &&\n     -+\t$1 <input >actual &&\n     ++\t\"$@\" <input >actual &&\n       \ttest_cmp expect actual &&\n       \tcp commit-graph-half .git/objects/info/commit-graph &&\n      -\ttest-tool reach $1 <input >actual &&\n     -+\t$1 <input >actual &&\n     ++\t\"$@\" <input >actual &&\n       \ttest_cmp expect actual\n       }\n       \n      +test_three_modes () {\n     -+\trun_three_modes \"test-tool reach $1\"\n     ++\trun_three_modes test-tool reach \"$@\"\n      +}\n      +\n       test_expect_success 'ref_newer:miss' '\n       \tcat >input <<-\\EOF &&\n       \tA:commit-5-7\n     -@@\n     - \tEOF\n     - \techo \"commit_contains(_,A,X,_):1\" >expect &&\n     - \ttest_three_modes commit_contains &&\n     --\ttest_three_modes commit_contains --tag\n     -+\ttest_three_modes \"commit_contains --tag\"\n     - '\n     - \n     - test_expect_success 'commit_contains:miss' '\n 3:  30dee58c61 ! 3:  b0ceb96076 test-reach: add rev-list tests\n     @@ -30,7 +30,7 @@\n      +\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n      +\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n      +\t>expect &&\n     -+\trun_three_modes \"git rev-list --topo-order commit-6-6\"\n     ++\trun_three_modes git rev-list --topo-order commit-6-6\n      +'\n      +\n      +test_expect_success 'rev-list: first-parent topo-order' '\n     @@ -42,7 +42,7 @@\n      +\t\tcommit-6-2 \\\n      +\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n      +\t>expect &&\n     -+\trun_three_modes \"git rev-list --first-parent --topo-order commit-6-6\"\n     ++\trun_three_modes git rev-list --first-parent --topo-order commit-6-6\n      +'\n      +\n      +test_expect_success 'rev-list: range topo-order' '\n     @@ -54,7 +54,7 @@\n      +\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n      +\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n      +\t>expect &&\n     -+\trun_three_modes \"git rev-list --topo-order commit-3-3..commit-6-6\"\n     ++\trun_three_modes git rev-list --topo-order commit-3-3..commit-6-6\n      +'\n      +\n      +test_expect_success 'rev-list: range topo-order' '\n     @@ -66,7 +66,7 @@\n      +\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n      +\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n      +\t>expect &&\n     -+\trun_three_modes \"git rev-list --topo-order commit-3-8..commit-6-6\"\n     ++\trun_three_modes git rev-list --topo-order commit-3-8..commit-6-6\n      +'\n      +\n      +test_expect_success 'rev-list: first-parent range topo-order' '\n     @@ -78,7 +78,7 @@\n      +\t\tcommit-6-2 \\\n      +\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n      +\t>expect &&\n     -+\trun_three_modes \"git rev-list --first-parent --topo-order commit-3-8..commit-6-6\"\n     ++\trun_three_modes git rev-list --first-parent --topo-order commit-3-8..commit-6-6\n      +'\n      +\n      +test_expect_success 'rev-list: ancestry-path topo-order' '\n     @@ -88,7 +88,7 @@\n      +\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 \\\n      +\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n      +\t>expect &&\n     -+\trun_three_modes \"git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\"\n     ++\trun_three_modes git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\n      +'\n      +\n      +test_expect_success 'rev-list: symmetric difference topo-order' '\n     @@ -102,7 +102,7 @@\n      +\t\tcommit-3-8 commit-2-8 commit-1-8 \\\n      +\t\tcommit-3-7 commit-2-7 commit-1-7 \\\n      +\t>expect &&\n     -+\trun_three_modes \"git rev-list --topo-order commit-3-8...commit-6-6\"\n     ++\trun_three_modes git rev-list --topo-order commit-3-8...commit-6-6\n      +'\n      +\n       test_done\n 4:  a74ae13d4e = 4:  fd1a0ab7cd revision.c: begin refactoring --topo-order logic\n 5:  0e64fc144c = 5:  e86f304082 commit/revisions: bookkeeping before refactoring\n -:  ---------- > 6:  fa6d5ef152 revision.h: add whitespace in flag definitions\n 6:  3b185ac3b1 ! 7:  020b2f50c5 revision.c: refactor basic topo-order logic\n     @@ -404,11 +404,11 @@\n      --- a/revision.h\n      +++ b/revision.h\n      @@\n     - #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n     - #define TRACK_LINEAR\t(1u<<26)\n     - #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n     -+#define TOPO_WALK_EXPLORED (1u<<27)\n     -+#define TOPO_WALK_INDEGREE (1u<<28)\n     + #define USER_GIVEN\t\t(1u<<25) /* given directly by the user */\n     + #define TRACK_LINEAR\t\t(1u<<26)\n     + #define ALL_REV_FLAGS\t\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n     ++#define TOPO_WALK_EXPLORED\t(1u<<27)\n     ++#define TOPO_WALK_INDEGREE\t(1u<<28)\n       \n       #define DECORATE_SHORT_REFS\t1\n       #define DECORATE_FULL_REFS\t2\n\n-- \ngitgitgadget\n"},{"id":"358621","messageId":"cc1ec4c2702d8ba35873600d321015bb0430d92e.1537551564.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 1/7] prio-queue: add 'peek' operation","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:27Z","receivedAt":"2018-09-21T17:39:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen consuming a priority queue, it can be convenient to inspect\nthe next object that will be dequeued without actually dequeueing\nit. Our existing library did not have such a 'peek' operation, so\nadd it as prio_queue_peek().\n\nAdd a reference-level comparison in t/helper/test-prio-queue.c\nso this method is exercised by t0009-prio-queue.sh.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n prio-queue.c               |  9 +++++++++\n prio-queue.h               |  6 ++++++\n t/helper/test-prio-queue.c | 10 +++++++---\n 3 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/prio-queue.c b/prio-queue.c\nindex a078451872..d3f488cb05 100644\n--- a/prio-queue.c\n+++ b/prio-queue.c\n@@ -85,3 +85,12 @@ void *prio_queue_get(struct prio_queue *queue)\n \t}\n \treturn result;\n }\n+\n+void *prio_queue_peek(struct prio_queue *queue)\n+{\n+\tif (!queue->nr)\n+\t\treturn NULL;\n+\tif (!queue->compare)\n+\t\treturn queue->array[queue->nr - 1].data;\n+\treturn queue->array[0].data;\n+}\ndiff --git a/prio-queue.h b/prio-queue.h\nindex d030ec9dd6..682e51867a 100644\n--- a/prio-queue.h\n+++ b/prio-queue.h\n@@ -46,6 +46,12 @@ extern void prio_queue_put(struct prio_queue *, void *thing);\n  */\n extern void *prio_queue_get(struct prio_queue *);\n \n+/*\n+ * Gain access to the \"thing\" that would be returned by\n+ * prio_queue_get, but do not remove it from the queue.\n+ */\n+extern void *prio_queue_peek(struct prio_queue *);\n+\n extern void clear_prio_queue(struct prio_queue *);\n \n /* Reverse the LIFO elements */\ndiff --git a/t/helper/test-prio-queue.c b/t/helper/test-prio-queue.c\nindex 9807b649b1..e817bbf464 100644\n--- a/t/helper/test-prio-queue.c\n+++ b/t/helper/test-prio-queue.c\n@@ -22,9 +22,13 @@ int cmd__prio_queue(int argc, const char **argv)\n \tstruct prio_queue pq = { intcmp };\n \n \twhile (*++argv) {\n-\t\tif (!strcmp(*argv, \"get\"))\n-\t\t\tshow(prio_queue_get(&pq));\n-\t\telse if (!strcmp(*argv, \"dump\")) {\n+\t\tif (!strcmp(*argv, \"get\")) {\n+\t\t\tvoid *peek = prio_queue_peek(&pq);\n+\t\t\tvoid *get = prio_queue_get(&pq);\n+\t\t\tif (peek != get)\n+\t\t\t\tBUG(\"peek and get results do not match\");\n+\t\t\tshow(get);\n+\t\t} else if (!strcmp(*argv, \"dump\")) {\n \t\t\tint *v;\n \t\t\twhile ((v = prio_queue_get(&pq)))\n \t\t\t       show(v);\n-- \ngitgitgadget\n\n"},{"id":"358622","messageId":"b2a1ade148bc5ef175338ba116bd33b6b5e8d386.1537551564.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 2/7] test-reach: add run_three_modes method","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:29Z","receivedAt":"2018-09-21T17:39:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe 'test_three_modes' method assumes we are using the 'test-tool\nreach' command for our test. However, we may want to use the data\nshape of our commit graph and the three modes (no commit-graph,\nfull commit-graph, partial commit-graph) for other git commands.\n\nSplit test_three_modes to be a simple translation on a more general\nrun_three_modes method that executes the given command and tests\nthe actual output to the expected output.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex d139a00d1d..9d65b8b946 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -53,18 +53,22 @@ test_expect_success 'setup' '\n \tgit config core.commitGraph true\n '\n \n-test_three_modes () {\n+run_three_modes () {\n \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-full .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-half .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual\n }\n \n+test_three_modes () {\n+\trun_three_modes test-tool reach \"$@\"\n+}\n+\n test_expect_success 'ref_newer:miss' '\n \tcat >input <<-\\EOF &&\n \tA:commit-5-7\n-- \ngitgitgadget\n\n"},{"id":"358623","messageId":"b0ceb960761329179d14e613343019e7ac207e4d.1537551564.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 3/7] test-reach: add rev-list tests","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:30Z","receivedAt":"2018-09-21T17:39:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe rev-list command is critical to Git's functionality. Ensure it\nworks in the three commit-graph environments constructed in\nt6600-test-reach.sh. Here are a few important types of rev-list\noperations:\n\n* Basic: git rev-list --topo-order HEAD\n* Range: git rev-list --topo-order compare..HEAD\n* Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n* Symmetric Difference: git rev-list --topo-order compare...HEAD\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 84 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 84 insertions(+)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 9d65b8b946..288f703b7b 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -243,4 +243,88 @@ test_expect_success 'commit_contains:miss' '\n \ttest_three_modes commit_contains --tag\n '\n \n+test_expect_success 'rev-list: basic topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 commit-3-3 commit-2-3 commit-1-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-6-6\n+'\n+\n+test_expect_success 'rev-list: first-parent topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --first-parent --topo-order commit-6-6\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-3..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-8..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: first-parent range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --first-parent --topo-order commit-3-8..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: ancestry-path topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: symmetric difference topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t\tcommit-3-8 commit-2-8 commit-1-8 \\\n+\t\tcommit-3-7 commit-2-7 commit-1-7 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-8...commit-6-6\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"358624","messageId":"fd1a0ab7cdaa06fd99f86fec51b483238f588296.1537551564.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:32Z","receivedAt":"2018-09-21T17:39:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen running 'git rev-list --topo-order' and its kin, the topo_order\nsetting in struct rev_info implies the limited setting. This means\nthat the following things happen during prepare_revision_walk():\n\n* revs->limited implies we run limit_list() to walk the entire\n  reachable set. There are some short-cuts here, such as if we\n  perform a range query like 'git rev-list COMPARE..HEAD' and we\n  can stop limit_list() when all queued commits are uninteresting.\n\n* revs->topo_order implies we run sort_in_topological_order(). See\n  the implementation of that method in commit.c. It implies that\n  the full set of commits to order is in the given commit_list.\n\nThese two methods imply that a 'git rev-list --topo-order HEAD'\ncommand must walk the entire reachable set of commits _twice_ before\nreturning a single result.\n\nIf we have a commit-graph file with generation numbers computed, then\nthere is a better way. This patch introduces some necessary logic\nredirection when we are in this situation.\n\nIn v2.18.0, the commit-graph file contains zero-valued bytes in the\npositions where the generation number is stored in v2.19.0 and later.\nThus, we use generation_numbers_enabled() to check if the commit-graph\nis available and has non-zero generation numbers.\n\nWhen setting revs->limited only because revs->topo_order is true,\nonly do so if generation numbers are not available. There is no\nreason to use the new logic as it will behave similarly when all\ngeneration numbers are INFINITY or ZERO.\n\nIn prepare_revision_walk(), if we have revs->topo_order but not\nrevs->limited, then we trigger the new logic. It breaks the logic\ninto three pieces, to fit with the existing framework:\n\n1. init_topo_walk() fills a new struct topo_walk_info in the rev_info\n   struct. We use the presence of this struct as a signal to use the\n   new methods during our walk. In this patch, this method simply\n   calls limit_list() and sort_in_topological_order(). In the future,\n   this method will set up a new data structure to perform that logic\n   in-line.\n\n2. next_topo_commit() provides get_revision_1() with the next topo-\n   ordered commit in the list. Currently, this simply pops the commit\n   from revs->commits.\n\n3. expand_topo_walk() provides get_revision_1() with a way to signal\n   walking beyond the latest commit. Currently, this calls\n   add_parents_to_list() exactly like the old logic.\n\nWhile this commit presents method redirection for performing the\nexact same logic as before, it allows the next commit to focus only\non the new logic.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 42 ++++++++++++++++++++++++++++++++++++++----\n revision.h |  4 ++++\n 2 files changed, 42 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex e18bd530e4..2dcde8a8ac 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -25,6 +25,7 @@\n #include \"worktree.h\"\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n+#include \"commit-graph.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2454,7 +2455,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \tif (revs->diffopt.objfind)\n \t\trevs->simplify_history = 0;\n \n-\tif (revs->topo_order)\n+\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n \t\trevs->limited = 1;\n \n \tif (revs->prune_data.nr) {\n@@ -2892,6 +2893,33 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n+struct topo_walk_info {};\n+\n+static void init_topo_walk(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info;\n+\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n+\tinfo = revs->topo_walk_info;\n+\tmemset(info, 0, sizeof(struct topo_walk_info));\n+\n+\tlimit_list(revs);\n+\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+}\n+\n+static struct commit *next_topo_commit(struct rev_info *revs)\n+{\n+\treturn pop_commit(&revs->commits);\n+}\n+\n+static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n+{\n+\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\t\tif (!revs->ignore_missing_links)\n+\t\t\tdie(\"Failed to traverse parents of commit %s\",\n+\t\t\t    oid_to_hex(&commit->object.oid));\n+\t}\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -2928,11 +2956,13 @@ int prepare_revision_walk(struct rev_info *revs)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\n-\tif (revs->limited)\n+\tif (revs->limited) {\n \t\tif (limit_list(revs) < 0)\n \t\t\treturn -1;\n-\tif (revs->topo_order)\n-\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t\tif (revs->topo_order)\n+\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t} else if (revs->topo_order)\n+\t\tinit_topo_walk(revs);\n \tif (revs->line_level_traverse)\n \t\tline_log_filter(revs);\n \tif (revs->simplify_merges)\n@@ -3257,6 +3287,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\tif (revs->reflog_info)\n \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n+\t\telse if (revs->topo_walk_info)\n+\t\t\tcommit = next_topo_commit(revs);\n \t\telse\n \t\t\tcommit = pop_commit(&revs->commits);\n \n@@ -3278,6 +3310,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\t\tif (revs->reflog_info)\n \t\t\t\ttry_to_simplify_commit(revs, commit);\n+\t\t\telse if (revs->topo_walk_info)\n+\t\t\t\texpand_topo_walk(revs, commit);\n \t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n \t\t\t\tif (!revs->ignore_missing_links)\n \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\ndiff --git a/revision.h b/revision.h\nindex 2b30ac270d..fd4154ff75 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -56,6 +56,8 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct topo_walk_info;\n+\n struct rev_info {\n \t/* Starting list */\n \tstruct commit_list *commits;\n@@ -245,6 +247,8 @@ struct rev_info {\n \tconst char *break_bar;\n \n \tstruct revision_sources *sources;\n+\n+\tstruct topo_walk_info *topo_walk_info;\n };\n \n int ref_excluded(struct string_list *, const char *path);\n-- \ngitgitgadget\n\n"},{"id":"358625","messageId":"e86f30408240da12a52858f836134b037e85f7ae.1537551564.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 5/7] commit/revisions: bookkeeping before refactoring","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:33Z","receivedAt":"2018-09-21T17:39:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThere are a few things that need to move around a little before\nmaking a big refactoring in the topo-order logic:\n\n1. We need access to record_author_date() and\n   compare_commits_by_author_date() in revision.c. These are used\n   currently by sort_in_topological_order() in commit.c.\n\n2. Moving these methods to commit.h requires adding the author_slab\n   definition to commit.h.\n\n3. The add_parents_to_list() method in revision.c performs logic\n   around the UNINTERESTING flag and other special cases depending\n   on the struct rev_info. Allow this method to ignore a NULL 'list'\n   parameter, as we will not be populating the list for our walk.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n commit.c   | 11 ++++-------\n commit.h   |  8 ++++++++\n revision.c |  6 ++++--\n 3 files changed, 16 insertions(+), 9 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex d0f199e122..f68e04b2f1 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -655,11 +655,8 @@ struct commit *pop_commit(struct commit_list **stack)\n /* count number of children that have not been emitted */\n define_commit_slab(indegree_slab, int);\n \n-/* record author-date for each commit object */\n-define_commit_slab(author_date_slab, timestamp_t);\n-\n-static void record_author_date(struct author_date_slab *author_date,\n-\t\t\t       struct commit *commit)\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit)\n {\n \tconst char *buffer = get_commit_buffer(commit, NULL);\n \tstruct ident_split ident;\n@@ -684,8 +681,8 @@ fail_exit:\n \tunuse_commit_buffer(commit, buffer);\n }\n \n-static int compare_commits_by_author_date(const void *a_, const void *b_,\n-\t\t\t\t\t  void *cb_data)\n+int compare_commits_by_author_date(const void *a_, const void *b_,\n+\t\t\t\t   void *cb_data)\n {\n \tconst struct commit *a = a_, *b = b_;\n \tstruct author_date_slab *author_date = cb_data;\ndiff --git a/commit.h b/commit.h\nindex 2b1a734388..ff0eb5f8ef 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -8,6 +8,7 @@\n #include \"gpg-interface.h\"\n #include \"string-list.h\"\n #include \"pretty.h\"\n+#include \"commit-slab.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n #define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n@@ -328,6 +329,13 @@ extern int remove_signature(struct strbuf *buf);\n  */\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n+/* record author-date for each commit object */\n+define_commit_slab(author_date_slab, timestamp_t);\n+\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit);\n+\n+int compare_commits_by_author_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \ndiff --git a/revision.c b/revision.c\nindex 2dcde8a8ac..92012d5f45 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -808,7 +808,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\treturn 0;\n \t}\n@@ -847,7 +848,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\tif (revs->first_parent_only)\n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"358626","messageId":"fa6d5ef1523676694a04b2d17c4d2ce043ccf5b7.1537551564.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 6/7] revision.h: add whitespace in flag definitions","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:35Z","receivedAt":"2018-09-21T17:39:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nIn anticipation of adding longer flag names in the next change, add\nan extra tab to each flag definition in revision.h.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.h | 28 ++++++++++++++--------------\n 1 file changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/revision.h b/revision.h\nindex fd4154ff75..e7bd059d80 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -10,20 +10,20 @@\n #include \"commit-slab-decl.h\"\n \n /* Remember to update object flag allocation in object.h */\n-#define SEEN\t\t(1u<<0)\n-#define UNINTERESTING   (1u<<1)\n-#define TREESAME\t(1u<<2)\n-#define SHOWN\t\t(1u<<3)\n-#define TMP_MARK\t(1u<<4) /* for isolated cases; clean after use */\n-#define BOUNDARY\t(1u<<5)\n-#define CHILD_SHOWN\t(1u<<6)\n-#define ADDED\t\t(1u<<7)\t/* Parents already parsed and added? */\n-#define SYMMETRIC_LEFT\t(1u<<8)\n-#define PATCHSAME\t(1u<<9)\n-#define BOTTOM\t\t(1u<<10)\n-#define USER_GIVEN\t(1u<<25) /* given directly by the user */\n-#define TRACK_LINEAR\t(1u<<26)\n-#define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define SEEN\t\t\t(1u<<0)\n+#define UNINTERESTING\t\t(1u<<1)\n+#define TREESAME\t\t(1u<<2)\n+#define SHOWN\t\t\t(1u<<3)\n+#define TMP_MARK\t\t(1u<<4) /* for isolated cases; clean after use */\n+#define BOUNDARY\t\t(1u<<5)\n+#define CHILD_SHOWN\t\t(1u<<6)\n+#define ADDED\t\t\t(1u<<7)\t/* Parents already parsed and added? */\n+#define SYMMETRIC_LEFT\t\t(1u<<8)\n+#define PATCHSAME\t\t(1u<<9)\n+#define BOTTOM\t\t\t(1u<<10)\n+#define USER_GIVEN\t\t(1u<<25) /* given directly by the user */\n+#define TRACK_LINEAR\t\t(1u<<26)\n+#define ALL_REV_FLAGS\t\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \ngitgitgadget\n\n"},{"id":"358627","messageId":"020b2f50c5703e8291577b008fdfa567093c6eab.1537551564.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-21T17:39:36Z","receivedAt":"2018-09-21T17:39:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen running a command like 'git rev-list --topo-order HEAD',\nGit performed the following steps:\n\n1. Run limit_list(), which parses all reachable commits,\n   adds them to a linked list, and distributes UNINTERESTING\n   flags. If all unprocessed commits are UNINTERESTING, then\n   it may terminate without walking all reachable commits.\n   This does not occur if we do not specify UNINTERESTING\n   commits.\n\n2. Run sort_in_topological_order(), which is an implementation\n   of Kahn's algorithm. It first iterates through the entire\n   set of important commits and computes the in-degree of each\n   (plus one, as we use 'zero' as a special value here). Then,\n   we walk the commits in priority order, adding them to the\n   priority queue if and only if their in-degree is one. As\n   we remove commits from this priority queue, we decrement the\n   in-degree of their parents.\n\n3. While we are peeling commits for output, get_revision_1()\n   uses pop_commit on the full list of commits computed by\n   sort_in_topological_order().\n\nIn the new algorithm, these three steps correspond to three\ndifferent commit walks. We run these walks simultaneously,\nand advance each only as far as necessary to satisfy the\nrequirements of the 'higher order' walk. We know when we can\npause each walk by using generation numbers from the commit-\ngraph feature.\n\nRecall that the generation number of a commit satisfies:\n\n* If the commit has at least one parent, then the generation\n  number is one more than the maximum generation number among\n  its parents.\n\n* If the commit has no parent, then the generation number is one.\n\nThere are two special generation numbers:\n\n* GENERATION_NUMBER_INFINITY: this value is 0xffffffff and\n  indicates that the commit is not stored in the commit-graph and\n  the generation number was not previously calculated.\n\n* GENERATION_NUMBER_ZERO: this value (0) is a special indicator\n  to say that the commit-graph was generated by a version of Git\n  that does not compute generation numbers (such as v2.18.0).\n\nSince we use generation_numbers_enabled() before using the new\nalgorithm, we do not need to worry about GENERATION_NUMBER_ZERO.\nHowever, the existence of GENERATION_NUMBER_INFINITY implies the\nfollowing weaker statement than the usual we expect from\ngeneration numbers:\n\n    If A and B are commits with generation numbers gen(A) and\n    gen(B) and gen(A) < gen(B), then A cannot reach B.\n\nThus, we will walk in each of our stages until the \"maximum\nunexpanded generation number\" is strictly lower than the\ngeneration number of a commit we are about to use.\n\nThe walks are as follows:\n\n1. EXPLORE: using the explore_queue priority queue (ordered by\n   maximizing the generation number), parse each reachable\n   commit until all commits in the queue have generation\n   number strictly lower than needed. During this walk, update\n   the UNINTERESTING flags as necessary.\n\n2. INDEGREE: using the indegree_queue priority queue (ordered\n   by maximizing the generation number), add one to the in-\n   degree of each parent for each commit that is walked. Since\n   we walk in order of decreasing generation number, we know\n   that discovering an in-degree value of 0 means the value for\n   that commit was not initialized, so should be initialized to\n   two. (Recall that in-degree value \"1\" is what we use to say a\n   commit is ready for output.) As we iterate the parents of a\n   commit during this walk, ensure the EXPLORE walk has walked\n   beyond their generation numbers.\n\n3. TOPO: using the topo_queue priority queue (ordered based on\n   the sort_order given, which could be commit-date, author-\n   date, or typical topo-order which treats the queue as a LIFO\n   stack), remove a commit from the queue and decrement the\n   in-degree of each parent. If a parent has an in-degree of\n   one, then we add it to the topo_queue. Before we decrement\n   the in-degree, however, ensure the INDEGREE walk has walked\n   beyond that generation number.\n\nThe implementations of these walks are in the following methods:\n\n* explore_walk_step and explore_to_depth\n* indegree_walk_step and compute_indegrees_to_depth\n* next_topo_commit and expand_topo_walk\n\nThese methods have some patterns that may seem strange at first,\nbut they are probably carry-overs from their equivalents in\nlimit_list and sort_in_topological_order.\n\nOne thing that is missing from this implementation is a proper\nway to stop walking when the entire queue is UNINTERESTING, so\nthis implementation is not enabled by comparisions, such as in\n'git rev-list --topo-order A..B'. This can be updated in the\nfuture.\n\nIn my local testing, I used the following Git commands on the\nLinux repository in three modes: HEAD~1 with no commit-graph,\nHEAD~1 with a commit-graph, and HEAD with a commit-graph. This\nallows comparing the benefits we get from parsing commits from\nthe commit-graph and then again the benefits we get by\nrestricting the set of commits we walk.\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nThis speedup is due to a few things. First, the new generation-\nnumber-enabled algorithm walks commits on order of the number of\nresults output (subject to some branching structure expectations).\nSince we limit to 100 results, we are running a query similar to\nfilling a single page of results. Second, when specifying a path,\nwe must parse the root tree object for each commit we walk. The\nprevious benefits from the commit-graph are entirely from reading\nthe commit-graph instead of parsing commits. Since we need to\nparse trees for the same number of commits as before, we slow\ndown significantly from the non-path-based query.\n\nFor the test above, I specifically selected a path that is changed\nfrequently, including by merge commits. A less-frequently-changed\npath (such as 'README') has similar end-to-end time since we need\nto walk the same number of commits (before determining we do not\nhave 100 hits). However, get get the benefit that the output is\npresented to the user as it is discovered, much the same as a\nnormal 'git log' command (no '--topo-order'). This is an improved\nuser experience, even if the command has the same runtime.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n object.h   |   4 +-\n revision.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n revision.h |   2 +\n 3 files changed, 194 insertions(+), 8 deletions(-)\n\ndiff --git a/object.h b/object.h\nindex 0feb90ae61..796792cb32 100644\n--- a/object.h\n+++ b/object.h\n@@ -59,7 +59,7 @@ struct object_array {\n \n /*\n  * object flag allocation:\n- * revision.h:               0---------10                              2526\n+ * revision.h:               0---------10                              25----28\n  * fetch-pack.c:             01\n  * negotiator/default.c:       2--5\n  * walker.c:                 0-2\n@@ -78,7 +78,7 @@ struct object_array {\n  * builtin/show-branch.c:    0-------------------------------------------26\n  * builtin/unpack-objects.c:                                 2021\n  */\n-#define FLAG_BITS  27\n+#define FLAG_BITS  29\n \n /*\n  * The object type is stored in 3 bits.\ndiff --git a/revision.c b/revision.c\nindex 92012d5f45..c5d0cb6599 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -26,6 +26,7 @@\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"prio-queue.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2895,30 +2896,213 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct topo_walk_info {};\n+define_commit_slab(indegree_slab, int);\n+\n+struct topo_walk_info {\n+\tuint32_t min_generation;\n+\tstruct prio_queue explore_queue;\n+\tstruct prio_queue indegree_queue;\n+\tstruct prio_queue topo_queue;\n+\tstruct indegree_slab indegree;\n+\tstruct author_date_slab author_date;\n+};\n+\n+static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n+{\n+\tif (c->object.flags & flag)\n+\t\treturn;\n+\n+\tc->object.flags |= flag;\n+\tprio_queue_put(q, c);\n+}\n+\n+static void explore_walk_step(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit_list *p;\n+\tstruct commit *c = prio_queue_get(&info->explore_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tif (revs->max_age != -1 && (c->date < revs->max_age))\n+\t\tc->object.flags |= UNINTERESTING;\n+\n+\tif (add_parents_to_list(revs, c, NULL, NULL) < 0)\n+\t\treturn;\n+\n+\tif (c->object.flags & UNINTERESTING)\n+\t\tmark_parents_uninteresting(c);\n+\n+\tfor (p = c->parents; p; p = p->next)\n+\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n+}\n+\n+static void explore_to_depth(struct rev_info *revs,\n+\t\t\t     uint32_t gen)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n+\t       c->generation >= gen)\n+\t\texplore_walk_step(revs);\n+}\n+\n+static void indegree_walk_step(struct rev_info *revs)\n+{\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c = prio_queue_get(&info->indegree_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\texplore_to_depth(revs, c->generation);\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tfor (p = c->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\tif (*pi)\n+\t\t\t(*pi)++;\n+\t\telse\n+\t\t\t*pi = 2;\n+\n+\t\ttest_flag_and_insert(&info->indegree_queue, parent, TOPO_WALK_INDEGREE);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n+\t}\n+}\n+\n+static void compute_indegrees_to_depth(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n+\t       c->generation >= info->min_generation)\n+\t\tindegree_walk_step(revs);\n+}\n \n static void init_topo_walk(struct rev_info *revs)\n {\n \tstruct topo_walk_info *info;\n+\tstruct commit_list *list;\n \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n \tinfo = revs->topo_walk_info;\n \tmemset(info, 0, sizeof(struct topo_walk_info));\n \n-\tlimit_list(revs);\n-\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\tinit_indegree_slab(&info->indegree);\n+\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n+\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n+\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n+\n+\tswitch (revs->sort_order) {\n+\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n+\t\tinfo->topo_queue.compare = NULL;\n+\t\tbreak;\n+\tcase REV_SORT_BY_COMMIT_DATE:\n+\t\tinfo->topo_queue.compare = compare_commits_by_commit_date;\n+\t\tbreak;\n+\tcase REV_SORT_BY_AUTHOR_DATE:\n+\t\tinit_author_date_slab(&info->author_date);\n+\t\tinfo->topo_queue.compare = compare_commits_by_author_date;\n+\t\tinfo->topo_queue.cb_data = &info->author_date;\n+\t\tbreak;\n+\t}\n+\n+\tinfo->explore_queue.compare = compare_commits_by_gen_then_commit_date;\n+\tinfo->indegree_queue.compare = compare_commits_by_gen_then_commit_date;\n+\n+\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n+\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n+\n+\t\tif (parse_commit_gently(c, 1))\n+\t\t\tcontinue;\n+\t\tif (c->generation < info->min_generation)\n+\t\t\tinfo->min_generation = c->generation;\n+\t}\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n+\n+\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n+\t\t\trecord_author_date(&info->author_date, c);\n+\t}\n+\tcompute_indegrees_to_depth(revs);\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\n+\t\tif (*(indegree_slab_at(&info->indegree, c)) == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, c);\n+\t}\n+\n+\t/*\n+\t * This is unfortunate; the initial tips need to be shown\n+\t * in the order given from the revision traversal machinery.\n+\t */\n+\tif (revs->sort_order == REV_SORT_IN_GRAPH_ORDER)\n+\t\tprio_queue_reverse(&info->topo_queue);\n }\n \n static struct commit *next_topo_commit(struct rev_info *revs)\n {\n-\treturn pop_commit(&revs->commits);\n+\tstruct commit *c;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\n+\t/* pop next off of topo_queue */\n+\tc = prio_queue_get(&info->topo_queue);\n+\n+\tif (c)\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 0;\n+\n+\treturn c;\n }\n \n static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n {\n-\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tif (add_parents_to_list(revs, commit, NULL, NULL) < 0) {\n \t\tif (!revs->ignore_missing_links)\n \t\t\tdie(\"Failed to traverse parents of commit %s\",\n-\t\t\t    oid_to_hex(&commit->object.oid));\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t}\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi;\n+\n+\t\tif (parse_commit_gently(parent, 1) < 0)\n+\t\t\tcontinue;\n+\n+\t\tif (parent->generation < info->min_generation) {\n+\t\t\tinfo->min_generation = parent->generation;\n+\t\t\tcompute_indegrees_to_depth(revs);\n+\t\t}\n+\n+\t\tpi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\t(*pi)--;\n+\t\tif (*pi == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, parent);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n \t}\n }\n \ndiff --git a/revision.h b/revision.h\nindex e7bd059d80..7cc3bf5fc0 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -24,6 +24,8 @@\n #define USER_GIVEN\t\t(1u<<25) /* given directly by the user */\n #define TRACK_LINEAR\t\t(1u<<26)\n #define ALL_REV_FLAGS\t\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define TOPO_WALK_EXPLORED\t(1u<<27)\n+#define TOPO_WALK_INDEGREE\t(1u<<28)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \ngitgitgadget\n"},{"id":"358659","messageId":"xmqq7ejev946.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/7] Use generation numbers for --topo-order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-21T21:22:33Z","receivedAt":"2018-09-21T21:22:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> Changes in V3: I added a new patch that updates the tab-alignment for flags\n> in revision.h before adding new ones (Thanks, Ævar!).\n\nThis is most unwelcome while other topics are in flight that caused\nunnecessary conflict.  It would have been very welcomed if the\ncodebase was dormant, though.\n\nI'll live, and there is no need to resend, but this change may not\nappear in today's pushout (I'll have to push out the result of\nintegration before I saw this new reroll with all the other topics).\n\n\n"},{"id":"358982","messageId":"04c9cd31-81cb-878c-85d4-dfa93220cf8b@gmail.com","threadId":"49229","inReplyTo":"cc1ec4c2702d8ba35873600d321015bb0430d92e.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/7] prio-queue: add 'peek' operation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-26T19:15:37Z","receivedAt":"2018-09-26T19:15:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/21/2018 1:39 PM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> When consuming a priority queue, it can be convenient to inspect\n> the next object that will be dequeued without actually dequeueing\n> it. Our existing library did not have such a 'peek' operation, so\n> add it as prio_queue_peek().\n>\n> Add a reference-level comparison in t/helper/test-prio-queue.c\n> so this method is exercised by t0009-prio-queue.sh.\n>\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>   prio-queue.c               |  9 +++++++++\n>   prio-queue.h               |  6 ++++++\n>   t/helper/test-prio-queue.c | 10 +++++++---\n>   3 files changed, 22 insertions(+), 3 deletions(-)\n>\n> diff --git a/prio-queue.c b/prio-queue.c\n> index a078451872..d3f488cb05 100644\n> --- a/prio-queue.c\n> +++ b/prio-queue.c\n> @@ -85,3 +85,12 @@ void *prio_queue_get(struct prio_queue *queue)\n>   \t}\n>   \treturn result;\n>   }\n> +\n> +void *prio_queue_peek(struct prio_queue *queue)\n> +{\n> +\tif (!queue->nr)\n> +\t\treturn NULL;\n> +\tif (!queue->compare)\n> +\t\treturn queue->array[queue->nr - 1].data;\n> +\treturn queue->array[0].data;\n> +}\n\nThe second branch here is never run by the test suite, as the only \nconsumers never have compare== NULL. I'll add an ability to test this \n\"stack\" behavior into t0009-prio-queue.sh.\n\n-Stolee\n\n"},{"id":"359076","messageId":"7b1640a0-4002-9e5f-ea06-4504a5cc9f43@gmail.com","threadId":"49229","inReplyTo":"020b2f50c5703e8291577b008fdfa567093c6eab.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-27T17:57:21Z","receivedAt":"2018-09-27T17:57:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/21/2018 1:39 PM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> When running a command like 'git rev-list --topo-order HEAD',\n> Git performed the following steps:\n>\n> 1. Run limit_list(), which parses all reachable commits,\n>     adds them to a linked list, and distributes UNINTERESTING\n>     flags. If all unprocessed commits are UNINTERESTING, then\n>     it may terminate without walking all reachable commits.\n>     This does not occur if we do not specify UNINTERESTING\n>     commits.\n>\n> 2. Run sort_in_topological_order(), which is an implementation\n>     of Kahn's algorithm. It first iterates through the entire\n>     set of important commits and computes the in-degree of each\n>     (plus one, as we use 'zero' as a special value here). Then,\n>     we walk the commits in priority order, adding them to the\n>     priority queue if and only if their in-degree is one. As\n>     we remove commits from this priority queue, we decrement the\n>     in-degree of their parents.\n>\n> 3. While we are peeling commits for output, get_revision_1()\n>     uses pop_commit on the full list of commits computed by\n>     sort_in_topological_order().\n>\n> In the new algorithm, these three steps correspond to three\n> different commit walks. We run these walks simultaneously,\n> and advance each only as far as necessary to satisfy the\n> requirements of the 'higher order' walk. We know when we can\n> pause each walk by using generation numbers from the commit-\n> graph feature.\nHello, Git contributors.\n\nI understand that this commit message and patch are pretty daunting. \nThere is a lot to read and digest. I would like to see if anyone is \nwilling to put the work in to review this patch, as I quite like what it \ndoes, and the performance numbers below.\n> In my local testing, I used the following Git commands on the\n> Linux repository in three modes: HEAD~1 with no commit-graph,\n> HEAD~1 with a commit-graph, and HEAD with a commit-graph. This\n> allows comparing the benefits we get from parsing commits from\n> the commit-graph and then again the benefits we get by\n> restricting the set of commits we walk.\n>\n> Test: git rev-list --topo-order -100 HEAD\n> HEAD~1, no commit-graph: 6.80 s\n> HEAD~1, w/ commit-graph: 0.77 s\n>    HEAD, w/ commit-graph: 0.02 s\n>\n> Test: git rev-list --topo-order -100 HEAD -- tools\n> HEAD~1, no commit-graph: 9.63 s\n> HEAD~1, w/ commit-graph: 6.06 s\n>    HEAD, w/ commit-graph: 0.06 s\n\nIf there is something I can do to make this easier to review, then \nplease let me know.\n\nThanks,\n-Stolee\n"},{"id":"359772","messageId":"86r2h3xbcx.fsf@gmail.com","threadId":"49229","inReplyTo":"7b1640a0-4002-9e5f-ea06-4504a5cc9f43@gmail.com","subject":"Re: [PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-06T16:56:30Z","receivedAt":"2018-10-06T16:56:35Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n> On 9/21/2018 1:39 PM, Derrick Stolee via GitGitGadget wrote:\n\n> Hello, Git contributors.\n>\n> I understand that this commit message and patch are pretty\n> daunting. There is a lot to read and digest. I would like to see if\n> anyone is willing to put the work in to review this patch, as I quite\n> like what it does, and the performance numbers below.\n\nI'll try to find time to review v3 of this patch series this week.\n\n>> In my local testing, I used the following Git commands on the\n>> Linux repository in three modes: HEAD~1 with no commit-graph,\n>> HEAD~1 with a commit-graph, and HEAD with a commit-graph. This\n>> allows comparing the benefits we get from parsing commits from\n>> the commit-graph and then again the benefits we get by\n>> restricting the set of commits we walk.\n>>\n>> Test: git rev-list --topo-order -100 HEAD\n>> HEAD~1, no commit-graph: 6.80 s\n>> HEAD~1, w/ commit-graph: 0.77 s\n>>    HEAD, w/ commit-graph: 0.02 s\n>>\n>> Test: git rev-list --topo-order -100 HEAD -- tools\n>> HEAD~1, no commit-graph: 9.63 s\n>> HEAD~1, w/ commit-graph: 6.06 s\n>>    HEAD, w/ commit-graph: 0.06 s\n>\n> If there is something I can do to make this easier to review, then\n> please let me know.\n>\n> Thanks,\n> -Stolee\n"},{"id":"360183","messageId":"20181011135459.GA27312@sigill.intra.peff.net","threadId":"49229","inReplyTo":"cc1ec4c2702d8ba35873600d321015bb0430d92e.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/7] prio-queue: add 'peek' operation","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T13:54:59Z","receivedAt":"2018-10-11T13:55:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 21, 2018 at 10:39:27AM -0700, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n> \n> When consuming a priority queue, it can be convenient to inspect\n> the next object that will be dequeued without actually dequeueing\n> it. Our existing library did not have such a 'peek' operation, so\n> add it as prio_queue_peek().\n\nMakes sense.\n\n> +void *prio_queue_peek(struct prio_queue *queue)\n> +{\n> +\tif (!queue->nr)\n> +\t\treturn NULL;\n> +\tif (!queue->compare)\n> +\t\treturn queue->array[queue->nr - 1].data;\n> +\treturn queue->array[0].data;\n\nThe non-compare version of get() treats this like a LIFO, and you do the\nsame here. Looks good.\n\nIn theory get() could be implemented in terms of peek(), but the result\nis not actually shorter because we have to check those same conditions\nto decide how to remove the item anyway.\n\n> diff --git a/t/helper/test-prio-queue.c b/t/helper/test-prio-queue.c\n> index 9807b649b1..e817bbf464 100644\n> --- a/t/helper/test-prio-queue.c\n> +++ b/t/helper/test-prio-queue.c\n> @@ -22,9 +22,13 @@ int cmd__prio_queue(int argc, const char **argv)\n>  \tstruct prio_queue pq = { intcmp };\n>  \n>  \twhile (*++argv) {\n> -\t\tif (!strcmp(*argv, \"get\"))\n> -\t\t\tshow(prio_queue_get(&pq));\n> -\t\telse if (!strcmp(*argv, \"dump\")) {\n> +\t\tif (!strcmp(*argv, \"get\")) {\n> +\t\t\tvoid *peek = prio_queue_peek(&pq);\n> +\t\t\tvoid *get = prio_queue_get(&pq);\n> +\t\t\tif (peek != get)\n> +\t\t\t\tBUG(\"peek and get results do not match\");\n> +\t\t\tshow(get);\n> +\t\t} else if (!strcmp(*argv, \"dump\")) {\n\nThis is a nice cheap way of piggy-backing on the existing get tests.\n\n-Peff\n"},{"id":"360184","messageId":"20181011135659.GB27312@sigill.intra.peff.net","threadId":"49229","inReplyTo":"b2a1ade148bc5ef175338ba116bd33b6b5e8d386.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/7] test-reach: add run_three_modes method","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T13:57:00Z","receivedAt":"2018-10-11T13:57:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 21, 2018 at 10:39:29AM -0700, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n> \n> The 'test_three_modes' method assumes we are using the 'test-tool\n> reach' command for our test. However, we may want to use the data\n> shape of our commit graph and the three modes (no commit-graph,\n> full commit-graph, partial commit-graph) for other git commands.\n> \n> Split test_three_modes to be a simple translation on a more general\n> run_three_modes method that executes the given command and tests\n> the actual output to the expected output.\n>\n> [...]\n> +test_three_modes () {\n> +\trun_three_modes test-tool reach \"$@\"\n> +}\n\nMakes sense. Sometimes in the test suite we want to be able to pass a\nwhole shell snippet to eval, but unless we specifically need that for\nthis series, running \"$@\" directly is simpler.\n\n-Peff\n"},{"id":"360185","messageId":"20181011135850.GC27312@sigill.intra.peff.net","threadId":"49229","inReplyTo":"b0ceb960761329179d14e613343019e7ac207e4d.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/7] test-reach: add rev-list tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T13:58:51Z","receivedAt":"2018-10-11T13:58:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 21, 2018 at 10:39:30AM -0700, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n> \n> The rev-list command is critical to Git's functionality. Ensure it\n> works in the three commit-graph environments constructed in\n> t6600-test-reach.sh. Here are a few important types of rev-list\n> operations:\n> \n> * Basic: git rev-list --topo-order HEAD\n> * Range: git rev-list --topo-order compare..HEAD\n> * Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n> * Symmetric Difference: git rev-list --topo-order compare...HEAD\n\nMakes sense. I'll assume you filled out all those \"expect\" blocks\ncorrectly.  ;)\n\n-Peff\n"},{"id":"360186","messageId":"20181011140610.GD27312@sigill.intra.peff.net","threadId":"49229","inReplyTo":"fd1a0ab7cdaa06fd99f86fec51b483238f588296.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T14:06:10Z","receivedAt":"2018-10-11T14:06:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 21, 2018 at 10:39:32AM -0700, Derrick Stolee via GitGitGadget wrote:\n\n> [..]\n> When setting revs->limited only because revs->topo_order is true,\n> only do so if generation numbers are not available. There is no\n> reason to use the new logic as it will behave similarly when all\n> generation numbers are INFINITY or ZERO.\n> \n> In prepare_revision_walk(), if we have revs->topo_order but not\n> revs->limited, then we trigger the new logic. It breaks the logic\n> into three pieces, to fit with the existing framework:\n\nNicely explained. Your abstracted init/next/expand API seems sane, but\nof course the real test will be reading the later patches that make use\nof it. :)\n\nThe patch matches my understanding of your explanation.\n\n-Peff\n"},{"id":"360187","messageId":"20181011142112.GE27312@sigill.intra.peff.net","threadId":"49229","inReplyTo":"e86f30408240da12a52858f836134b037e85f7ae.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/7] commit/revisions: bookkeeping before refactoring","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T14:21:12Z","receivedAt":"2018-10-11T14:21:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 21, 2018 at 10:39:33AM -0700, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n> \n> There are a few things that need to move around a little before\n> making a big refactoring in the topo-order logic:\n> \n> 1. We need access to record_author_date() and\n>    compare_commits_by_author_date() in revision.c. These are used\n>    currently by sort_in_topological_order() in commit.c.\n> \n> 2. Moving these methods to commit.h requires adding the author_slab\n>    definition to commit.h.\n\nThe overall goal makes sense. Do we really need to define the whole slab\nin the header file? We're going to end up with multiple copies of the\nfunctions, since they're declared static in each file that includes\ncommit.h.\n\nFrom what's here, I think you could get away with just:\n\n  struct author_date_slab;\n  void record_author_date(struct author_date_slab *author_date,\n                          struct commit *commit);\n\nin the header file. But presumably callers would eventually want to\nallocate their own author dates. If that's all we need, then these days\nyou can do:\n\n  declare_commit_slab(author_date, timestamp_t);\n\nto get the type declaration.\n\nIf they really do need the functions accessible outside of commit.c,\nthen perhaps:\n\n  define_shared_commit_slab(author_date, timestamp_t);\n\nin commit.h, and:\n\n  implement_shared_commit_slab(author_date, timestamp_t);\n\nin commit.c (the type repetition is not too bad, as the compiler would\ncatch any mistakes).\n\nThe only downside of this approach is that we're less likely to be able\nto inline element access (though \"peek\" is big enough that I'm not sure\nit ends up inlined anyway).\n\n> 3. The add_parents_to_list() method in revision.c performs logic\n>    around the UNINTERESTING flag and other special cases depending\n>    on the struct rev_info. Allow this method to ignore a NULL 'list'\n>    parameter, as we will not be populating the list for our walk.\n\nSo now you can add_parents_to_list() without a list? That sounds\nconfusing. :)\n\nIs it possible to split the function into two? Some\nhandle_uninteresting_parents() logic, and then an add_parents_to_list()\nthat calls that, but also adds to the list?\n\nA cursory look at the function suggests it's actually kind of tricky.\nPerhaps as an alternative, add_parents_to_list() could just get a more\ndescriptive name?\n\n> ---\n>  commit.c   | 11 ++++-------\n>  commit.h   |  8 ++++++++\n>  revision.c |  6 ++++--\n>  3 files changed, 16 insertions(+), 9 deletions(-)\n\nThe patch itself seems straight-forward based on those explanations.\n\n-Peff\n"},{"id":"360189","messageId":"20181011153510.GF27312@sigill.intra.peff.net","threadId":"49229","inReplyTo":"020b2f50c5703e8291577b008fdfa567093c6eab.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T15:35:11Z","receivedAt":"2018-10-11T15:35:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 21, 2018 at 10:39:36AM -0700, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n> \n> When running a command like 'git rev-list --topo-order HEAD',\n> Git performed the following steps:\n> [...]\n> In the new algorithm, these three steps correspond to three\n> different commit walks. We run these walks simultaneously,\n\nA minor nit, but this commit message doesn't mention the most basic\nthing up front: that its main purpose is to introduce a new algorithm\nfor topo-order. ;)\n\nIt's obvious in the context of reviewing the series, but somebody\nreading \"git log\" later may want a bit more. Perhaps:\n\n  revision.c: implement generation-based topo-order algorithm\n\nas a subject, and/or an introductory paragraph like:\n\n  The current --topo-order algorithm requires walking all commits we\n  are going to output up front, topo-sorting them, all before\n  outputting the first value. This patch introduces a new algorithm\n  which uses stored generation numbers to incrementally walk in\n  topo-order, outputting commits as we go.\n\nOther than that, I find this to be a wonderfully explanatory commit\nmessage. :)\n\n> The walks are as follows:\n> \n> 1. EXPLORE: using the explore_queue priority queue (ordered by\n>    maximizing the generation number), parse each reachable\n>    commit until all commits in the queue have generation\n>    number strictly lower than needed. During this walk, update\n>    the UNINTERESTING flags as necessary.\n\nOK, this makes sense. If we know that everybody else in our queue is at\ngeneration X, then it is safe to output a commit at generation greater\nthan X.\n\nI think this by itself would allow us to implement \"show no parents\nbefore all of its children are shown\", right? But --topo-order promises\na bit more: \"avoid showing commits no multiple lines of history\nintermixed\".\n\nI guess also INFINITY generation numbers need more. For a real\ngeneration number, we know that \"gen(A) == gen(B)\" implies that there is\nno ancestry relationship between the two. But not so for INFINITY.\n\n> 2. INDEGREE: using the indegree_queue priority queue (ordered\n>    by maximizing the generation number), add one to the in-\n>    degree of each parent for each commit that is walked. Since\n>    we walk in order of decreasing generation number, we know\n>    that discovering an in-degree value of 0 means the value for\n>    that commit was not initialized, so should be initialized to\n>    two. (Recall that in-degree value \"1\" is what we use to say a\n>    commit is ready for output.) As we iterate the parents of a\n>    commit during this walk, ensure the EXPLORE walk has walked\n>    beyond their generation numbers.\n\nI wondered how this would work for INFINITY. We can't know the order of\na bunch of INFINITY nodes at all, so we never know when their in-degree\nvalues are \"done\". But if I understand the EXPLORE walk, we'd basically\nwalk all of INFINITY down to something with a real generation number. Is\nthat right?\n\nBut after that, I'm not totally clear on why we need this INDEGREE walk.\n\n> 3. TOPO: using the topo_queue priority queue (ordered based on\n>    the sort_order given, which could be commit-date, author-\n>    date, or typical topo-order which treats the queue as a LIFO\n>    stack), remove a commit from the queue and decrement the\n>    in-degree of each parent. If a parent has an in-degree of\n>    one, then we add it to the topo_queue. Before we decrement\n>    the in-degree, however, ensure the INDEGREE walk has walked\n>    beyond that generation number.\n\nOK, this makes sense to make --author-date-order, etc, work. Potentially\nthose numbers might have no relationship at all with the graph\nstructure, but we promise \"no parent before its children are shown\", so\nthis is really just a tie-breaker after the topo-sort anyway. As long as\nsteps 1 and 2 are correct and produce a complete set of commits for one\n\"layer\", this should be OK.\n\nI guess I'm not 100% convinced that we don't have a case where we\nhaven't yet parsed or considered some commit that we know cannot have an\nancestry relationship with commits we are outputting. But it may have an\nauthor-date-order relationship.\n\n(I'm not at all convinced that this _is_ a problem, and I suspect it\nisn't; I'm only suggesting I haven't fully grokked the proof).\n\n> ---\n>  object.h   |   4 +-\n>  revision.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n>  revision.h |   2 +\n>  3 files changed, 194 insertions(+), 8 deletions(-)\n\nI'll pause here on evaluating the actual code. It looks sane from a\ncursory read, but there's no point in digging further until I'm sure I\nfully understand the algorithm. I think that needs a little more brain\npower from me, and hopefully discussion around my comments above will\nhelp trigger that.\n\n-Peff\n"},{"id":"360191","messageId":"1c0eb7d3-7f31-9377-d42f-4ac2f36ac26a@gmail.com","threadId":"49229","inReplyTo":"20181011153510.GF27312@sigill.intra.peff.net","subject":"Re: [PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-11T16:21:44Z","receivedAt":"2018-10-11T16:21:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/11/2018 11:35 AM, Jeff King wrote:\n> On Fri, Sep 21, 2018 at 10:39:36AM -0700, Derrick Stolee via GitGitGadget wrote:\n>\n>> From: Derrick Stolee <dstolee@microsoft.com>\n>>\n>> When running a command like 'git rev-list --topo-order HEAD',\n>> Git performed the following steps:\n>> [...]\n>> In the new algorithm, these three steps correspond to three\n>> different commit walks. We run these walks simultaneously,\n> A minor nit, but this commit message doesn't mention the most basic\n> thing up front: that its main purpose is to introduce a new algorithm\n> for topo-order. ;)\n>\n> It's obvious in the context of reviewing the series, but somebody\n> reading \"git log\" later may want a bit more. Perhaps:\n>\n>    revision.c: implement generation-based topo-order algorithm\n>\n> as a subject, and/or an introductory paragraph like:\n>\n>    The current --topo-order algorithm requires walking all commits we\n>    are going to output up front, topo-sorting them, all before\n>    outputting the first value. This patch introduces a new algorithm\n>    which uses stored generation numbers to incrementally walk in\n>    topo-order, outputting commits as we go.\n>\n> Other than that, I find this to be a wonderfully explanatory commit\n> message. :)\n\nGood idea. I'll make that change.\n\n>\n>> The walks are as follows:\n>>\n>> 1. EXPLORE: using the explore_queue priority queue (ordered by\n>>     maximizing the generation number), parse each reachable\n>>     commit until all commits in the queue have generation\n>>     number strictly lower than needed. During this walk, update\n>>     the UNINTERESTING flags as necessary.\n> OK, this makes sense. If we know that everybody else in our queue is at\n> generation X, then it is safe to output a commit at generation greater\n> than X.\n>\n> I think this by itself would allow us to implement \"show no parents\n> before all of its children are shown\", right? But --topo-order promises\n> a bit more: \"avoid showing commits no multiple lines of history\n> intermixed\".\n>\n> I guess also INFINITY generation numbers need more. For a real\n> generation number, we know that \"gen(A) == gen(B)\" implies that there is\n> no ancestry relationship between the two. But not so for INFINITY.\n\nYeah, to deal with INFINITY (and ZERO, but that won't happen if \ngeneration_numbers_enabled() returns true), we treat gen(A) == gen(B) as \na \"no information\" state. So, to output a commit at generation X, we \nneed to have our maximum generation number in the unexplored area to be \nat most X - 1. You'll see strict inequality when checking generations.\n\n\n>> 2. INDEGREE: using the indegree_queue priority queue (ordered\n>>     by maximizing the generation number), add one to the in-\n>>     degree of each parent for each commit that is walked. Since\n>>     we walk in order of decreasing generation number, we know\n>>     that discovering an in-degree value of 0 means the value for\n>>     that commit was not initialized, so should be initialized to\n>>     two. (Recall that in-degree value \"1\" is what we use to say a\n>>     commit is ready for output.) As we iterate the parents of a\n>>     commit during this walk, ensure the EXPLORE walk has walked\n>>     beyond their generation numbers.\n> I wondered how this would work for INFINITY. We can't know the order of\n> a bunch of INFINITY nodes at all, so we never know when their in-degree\n> values are \"done\". But if I understand the EXPLORE walk, we'd basically\n> walk all of INFINITY down to something with a real generation number. Is\n> that right?\n>\n> But after that, I'm not totally clear on why we need this INDEGREE walk.\n\nThe INDEGREE walk is an important element for Kahn's algorithm. The \nfinal output order is dictated by peeling commits of \"indegree zero\" to \nensure all children are output before their parents. (Note: since we use \nliteral 0 to mean \"uninitialized\", we peel commits when the indegree \nslab has value 1.)\n\nThis walk replaces the indegree logic from sort_in_topological_order(). \nThat method performs one walk that fills the indegree slab, then another \nwalk that peels the commits with indegree 0 and inserts them into a list.\n\n>> 3. TOPO: using the topo_queue priority queue (ordered based on\n>>     the sort_order given, which could be commit-date, author-\n>>     date, or typical topo-order which treats the queue as a LIFO\n>>     stack), remove a commit from the queue and decrement the\n>>     in-degree of each parent. If a parent has an in-degree of\n>>     one, then we add it to the topo_queue. Before we decrement\n>>     the in-degree, however, ensure the INDEGREE walk has walked\n>>     beyond that generation number.\n> OK, this makes sense to make --author-date-order, etc, work. Potentially\n> those numbers might have no relationship at all with the graph\n> structure, but we promise \"no parent before its children are shown\", so\n> this is really just a tie-breaker after the topo-sort anyway. As long as\n> steps 1 and 2 are correct and produce a complete set of commits for one\n> \"layer\", this should be OK.\n>\n> I guess I'm not 100% convinced that we don't have a case where we\n> haven't yet parsed or considered some commit that we know cannot have an\n> ancestry relationship with commits we are outputting. But it may have an\n> author-date-order relationship.\n>\n> (I'm not at all convinced that this _is_ a problem, and I suspect it\n> isn't; I'm only suggesting I haven't fully grokked the proof).\nThe INDEGREE walk should not stop until it has explored at least to the \npoint that all indegree 0 commits are exposed (relative to the current \nstate of the walk).\n\nAt initialization, we walk from all starting positions until the maximum \ngeneration number in our queue is less than the minimum generation of a \nstarting commit. The starting positions that have indegree 0 are then \nadded to the topo_queue, and the sort order dictates which is the best. \n From this point on, we can only create a new \"indegree 0\" commit by \nremoving a commit from the topo_queue and decrementing the indegree of \nits parents. Those parents with indegree 0 are inserted into topo_queue \nand compared to all other indegree 0 commits. Thus, we will always \nexplore enough to make the right choice relative our sort order.\n\n>\n>> ---\n>>   object.h   |   4 +-\n>>   revision.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n>>   revision.h |   2 +\n>>   3 files changed, 194 insertions(+), 8 deletions(-)\n> I'll pause here on evaluating the actual code. It looks sane from a\n> cursory read, but there's no point in digging further until I'm sure I\n> fully understand the algorithm. I think that needs a little more brain\n> power from me, and hopefully discussion around my comments above will\n> help trigger that.\nThanks for reading! I understand that reading the code is useless \nwithout understanding the high-level concepts. I'm happy to iterate on \nthis. If I can find a better way to explain the algorithm in the commit \nmessage to avoid the \"huh?\" moments above, then I will.\n\nThanks,\n-Stolee\n"},{"id":"360237","messageId":"CAGZ79kYttgMUeQk7fvXGk2+SF10PS8yqsgNH47HBQLfNqShpMQ@mail.gmail.com","threadId":"49229","inReplyTo":"020b2f50c5703e8291577b008fdfa567093c6eab.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-10-11T22:32:01Z","receivedAt":"2018-10-11T22:32:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Sep 21, 2018 at 10:39 AM Derrick Stolee via GitGitGadget\n<gitgitgadget@gmail.com> wrote:\n>\n> From: Derrick Stolee <dstolee@microsoft.com>\n[...]\n> For the test above, I specifically selected a path that is changed\n> frequently, including by merge commits. A less-frequently-changed\n> path (such as 'README') has similar end-to-end time since we need\n> to walk the same number of commits (before determining we do not\n> have 100 hits). However, get get the benefit that the output is\n\n\"get get\"\n"},{"id":"360270","messageId":"xmqqefcvdbqm.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"20181011135850.GC27312@sigill.intra.peff.net","subject":"Re: [PATCH v3 3/7] test-reach: add rev-list tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-12T04:34:25Z","receivedAt":"2018-10-12T04:34:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Fri, Sep 21, 2018 at 10:39:30AM -0700, Derrick Stolee via GitGitGadget wrote:\n>\n>> From: Derrick Stolee <dstolee@microsoft.com>\n>> \n>> The rev-list command is critical to Git's functionality. Ensure it\n>> works in the three commit-graph environments constructed in\n>> t6600-test-reach.sh. Here are a few important types of rev-list\n>> operations:\n>> \n>> * Basic: git rev-list --topo-order HEAD\n>> * Range: git rev-list --topo-order compare..HEAD\n>> * Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n>> * Symmetric Difference: git rev-list --topo-order compare...HEAD\n>\n> Makes sense. I'll assume you filled out all those \"expect\" blocks\n> correctly.  ;)\n\nWell, otherwise three-modes test would barf at least when it is\nrunning in its \"no graph\" mode, so I'd assume we are covered.\n"},{"id":"360276","messageId":"xmqqwoqnbrmw.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"fd1a0ab7cdaa06fd99f86fec51b483238f588296.1537551564.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-12T06:33:59Z","receivedAt":"2018-10-12T06:34:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> * revs->limited implies we run limit_list() to walk the entire\n>   reachable set. There are some short-cuts here, such as if we\n>   perform a range query like 'git rev-list COMPARE..HEAD' and we\n>   can stop limit_list() when all queued commits are uninteresting.\n>\n> * revs->topo_order implies we run sort_in_topological_order(). See\n>   the implementation of that method in commit.c. It implies that\n>   the full set of commits to order is in the given commit_list.\n>\n> These two methods imply that a 'git rev-list --topo-order HEAD'\n> command must walk the entire reachable set of commits _twice_ before\n> returning a single result.\n\nWith or without \"--topo-order\", running rev-list without any\nnegative commit means we must dig down to the roots that can be\nreached from the positive commits we have.\n\nI am to sure if having to run the \"sort\" of order N counts as \"walk\nthe entire reachable set once\" (in addition to the enumeration that\nmust be done to prepare that N commits, performed in limit_list()).\n\n> In v2.18.0, the commit-graph file contains zero-valued bytes in the\n> positions where the generation number is stored in v2.19.0 and later.\n> Thus, we use generation_numbers_enabled() to check if the commit-graph\n> is available and has non-zero generation numbers.\n>\n> When setting revs->limited only because revs->topo_order is true,\n> only do so if generation numbers are not available. There is no\n> reason to use the new logic as it will behave similarly when all\n> generation numbers are INFINITY or ZERO.\n\n> In prepare_revision_walk(), if we have revs->topo_order but not\n> revs->limited, then we trigger the new logic. It breaks the logic\n> into three pieces, to fit with the existing framework:\n>\n> 1. init_topo_walk() fills a new struct topo_walk_info in the rev_info\n>    struct. We use the presence of this struct as a signal to use the\n>    new methods during our walk. In this patch, this method simply\n>    calls limit_list() and sort_in_topological_order(). In the future,\n>    this method will set up a new data structure to perform that logic\n>    in-line.\n>\n> 2. next_topo_commit() provides get_revision_1() with the next topo-\n>    ordered commit in the list. Currently, this simply pops the commit\n>    from revs->commits.\n\n... because everything is already done in #1 above.  Which makes sense.\n\n> 3. expand_topo_walk() provides get_revision_1() with a way to signal\n>    walking beyond the latest commit. Currently, this calls\n>    add_parents_to_list() exactly like the old logic.\n\n\"latest\"?  We dig down the history from newer to older, so at some\npoint we hit an old commit and need to find the parents to keep\nwalking towards even older parts of the history.  Did you mean\n\"earliest\" instead?\n\n> While this commit presents method redirection for performing the\n> exact same logic as before, it allows the next commit to focus only\n> on the new logic.\n\nOK.\n\n> diff --git a/revision.c b/revision.c\n> index e18bd530e4..2dcde8a8ac 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -25,6 +25,7 @@\n>  #include \"worktree.h\"\n>  #include \"argv-array.h\"\n>  #include \"commit-reach.h\"\n> +#include \"commit-graph.h\"\n>  \n>  volatile show_early_output_fn_t show_early_output;\n>  \n> @@ -2454,7 +2455,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n>  \tif (revs->diffopt.objfind)\n>  \t\trevs->simplify_history = 0;\n>  \n> -\tif (revs->topo_order)\n> +\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n>  \t\trevs->limited = 1;\n\nAre we expecting that this is always a bool?  Can there be new\ncommits for which generation numbers are not computed and stored\nwhile all the old, stable and packed commits have generation\nnumbers?\n\n> @@ -2892,6 +2893,33 @@ static int mark_uninteresting(const struct object_id *oid,\n>  \treturn 0;\n>  }\n>  \n> +struct topo_walk_info {};\n> +\n> +static void init_topo_walk(struct rev_info *revs)\n> +{\n> +\tstruct topo_walk_info *info;\n> +\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n> +\tinfo = revs->topo_walk_info;\n> +\tmemset(info, 0, sizeof(struct topo_walk_info));\n\nThere is no member in the struct at this point.  Are we sure this is\nsafe?  Just being curious.  I know xmalloc() gives us at least one\nbyte and info won't be NULL.  I just do not know offhand if we have\na guarantee that memset() acts sensibly to fill the first 0 bytes.\n\n> +\tlimit_list(revs);\n> +\tsort_in_topological_order(&revs->commits, revs->sort_order);\n> +}\n> +\n> +static struct commit *next_topo_commit(struct rev_info *revs)\n> +{\n> +\treturn pop_commit(&revs->commits);\n> +}\n> +\n> +static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n> +{\n> +\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n> +\t\tif (!revs->ignore_missing_links)\n> +\t\t\tdie(\"Failed to traverse parents of commit %s\",\n> +\t\t\t    oid_to_hex(&commit->object.oid));\n> +\t}\n> +}\n> +\n>  int prepare_revision_walk(struct rev_info *revs)\n>  {\n>  \tint i;\n> @@ -2928,11 +2956,13 @@ int prepare_revision_walk(struct rev_info *revs)\n>  \t\tcommit_list_sort_by_date(&revs->commits);\n>  \tif (revs->no_walk)\n>  \t\treturn 0;\n> -\tif (revs->limited)\n> +\tif (revs->limited) {\n>  \t\tif (limit_list(revs) < 0)\n>  \t\t\treturn -1;\n> -\tif (revs->topo_order)\n> -\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n> +\t\tif (revs->topo_order)\n> +\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n> +\t} else if (revs->topo_order)\n> +\t\tinit_topo_walk(revs);\n>  \tif (revs->line_level_traverse)\n>  \t\tline_log_filter(revs);\n>  \tif (revs->simplify_merges)\n\nThe diff is a bit hard to grok around here, but \n\n - when limited *and* topo_order, we do the sort here, as we know we\n   already have called limit_list(), i.e. we behave identically as\n   the code before this patch in that case.\n\n - when not limited but topo_order, then we do init_topo_walk();\n   currently we do limit_list() and sort_in_topological_order(),\n   which means we do the same as above.\n\nAs long as limit_list() and sort_in_topological_order() does not\nlook at revs->limited bit, this patch cannot cause any regression.\n\n> @@ -3257,6 +3287,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n>  \n>  \t\tif (revs->reflog_info)\n>  \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n> +\t\telse if (revs->topo_walk_info)\n> +\t\t\tcommit = next_topo_commit(revs);\n>  \t\telse\n>  \t\t\tcommit = pop_commit(&revs->commits);\n\nSo this get_revision_1() always grabs the commit from next_topo_commit()\nwhen topo-order is in effect.\n\n> @@ -3278,6 +3310,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n>  \n>  \t\t\tif (revs->reflog_info)\n>  \t\t\t\ttry_to_simplify_commit(revs, commit);\n> +\t\t\telse if (revs->topo_walk_info)\n> +\t\t\t\texpand_topo_walk(revs, commit);\n>  \t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n>  \t\t\t\tif (!revs->ignore_missing_links)\n>  \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\n\nAnd this add-parents-or-barf is replicated in expand_topo_walk() at\nthis step, so there is no change in behaviour.\n\nLooks like a cleanly done preparation that is a no-op.\n\n> diff --git a/revision.h b/revision.h\n> index 2b30ac270d..fd4154ff75 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -56,6 +56,8 @@ struct rev_cmdline_info {\n>  #define REVISION_WALK_NO_WALK_SORTED 1\n>  #define REVISION_WALK_NO_WALK_UNSORTED 2\n>  \n> +struct topo_walk_info;\n> +\n>  struct rev_info {\n>  \t/* Starting list */\n>  \tstruct commit_list *commits;\n> @@ -245,6 +247,8 @@ struct rev_info {\n>  \tconst char *break_bar;\n>  \n>  \tstruct revision_sources *sources;\n> +\n> +\tstruct topo_walk_info *topo_walk_info;\n>  };\n>  \n>  int ref_excluded(struct string_list *, const char *path);\n"},{"id":"360300","messageId":"0407d50d-aabd-e85e-3c48-f6b6678f9884@gmail.com","threadId":"49229","inReplyTo":"xmqqwoqnbrmw.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-12T12:32:43Z","receivedAt":"2018-10-12T12:32:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/12/2018 2:33 AM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n>> * revs->limited implies we run limit_list() to walk the entire\n>>    reachable set. There are some short-cuts here, such as if we\n>>    perform a range query like 'git rev-list COMPARE..HEAD' and we\n>>    can stop limit_list() when all queued commits are uninteresting.\n>>\n>> * revs->topo_order implies we run sort_in_topological_order(). See\n>>    the implementation of that method in commit.c. It implies that\n>>    the full set of commits to order is in the given commit_list.\n>>\n>> These two methods imply that a 'git rev-list --topo-order HEAD'\n>> command must walk the entire reachable set of commits _twice_ before\n>> returning a single result.\n> With or without \"--topo-order\", running rev-list without any\n> negative commit means we must dig down to the roots that can be\n> reached from the positive commits we have.\nIf we use default order in 'git log', we don't walk all the way to the \nroot commits, and instead trust the commit-date. (This is different than \n--date-order, which does guarantee parents after children.) In this \ncase, revs->limited is false.\n> I am to sure if having to run the \"sort\" of order N counts as \"walk\n> the entire reachable set once\" (in addition to the enumeration that\n> must be done to prepare that N commits, performed in limit_list()).\n\nsort_in_topological_order() does actually _two_ walks (the in-degree \ncomputation plus the walk that peels commits of in-degree zero), but \nthose walks are cheaper because we've already parsed the commits in \nlimit_list().\n>> 3. expand_topo_walk() provides get_revision_1() with a way to signal\n>>     walking beyond the latest commit. Currently, this calls\n>>     add_parents_to_list() exactly like the old logic.\n> \"latest\"?  We dig down the history from newer to older, so at some\n> point we hit an old commit and need to find the parents to keep\n> walking towards even older parts of the history.  Did you mean\n> \"earliest\" instead?\nI mean \"latest\" in terms of the algorithm, so \"the commit that was \nreturned by get_revision_1() most recently\". This could use some \nrewriting for clarity.\n>>   \n>> @@ -2454,7 +2455,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n>>   \tif (revs->diffopt.objfind)\n>>   \t\trevs->simplify_history = 0;\n>>   \n>> -\tif (revs->topo_order)\n>> +\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n>>   \t\trevs->limited = 1;\n> Are we expecting that this is always a bool?  Can there be new\n> commits for which generation numbers are not computed and stored\n> while all the old, stable and packed commits have generation\n> numbers?\n\nFor this algorithm to work, we only care that _some_ commits have \ngeneration numbers. We expect that if a commit-graph file exists with \ngeneration numbers, then the majority of commits have generation \nnumbers. The commits that were added or fetched since the commit-graph \nwas written will have generation number INFINITY, but the topo-order \nalgorithm will still work and be efficient in those cases. (This is also \nwhy we have the \"half graph\" case in test_three_modes.)\n\n>> @@ -2892,6 +2893,33 @@ static int mark_uninteresting(const struct object_id *oid,\n>>   \treturn 0;\n>>   }\n>>   \n>> +struct topo_walk_info {};\n>> +\n>> +static void init_topo_walk(struct rev_info *revs)\n>> +{\n>> +\tstruct topo_walk_info *info;\n>> +\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n>> +\tinfo = revs->topo_walk_info;\n>> +\tmemset(info, 0, sizeof(struct topo_walk_info));\n> There is no member in the struct at this point.  Are we sure this is\n> safe?  Just being curious.  I know xmalloc() gives us at least one\n> byte and info won't be NULL.  I just do not know offhand if we have\n> a guarantee that memset() acts sensibly to fill the first 0 bytes.\nThis is a good question. It seems to work for me when I check out your \nversion of this commit (6c04ff30 \"revision.c: begin refactoring \n--topo-order logic\") and run all tests.\n>> +\tlimit_list(revs);\n>> +\tsort_in_topological_order(&revs->commits, revs->sort_order);\n>> +}\n>> +\n>> +static struct commit *next_topo_commit(struct rev_info *revs)\n>> +{\n>> +\treturn pop_commit(&revs->commits);\n>> +}\n>> +\n>> +static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n>> +{\n>> +\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n>> +\t\tif (!revs->ignore_missing_links)\n>> +\t\t\tdie(\"Failed to traverse parents of commit %s\",\n>> +\t\t\t    oid_to_hex(&commit->object.oid));\n>> +\t}\n>> +}\n>> +\n>>   int prepare_revision_walk(struct rev_info *revs)\n>>   {\n>>   \tint i;\n>> @@ -2928,11 +2956,13 @@ int prepare_revision_walk(struct rev_info *revs)\n>>   \t\tcommit_list_sort_by_date(&revs->commits);\n>>   \tif (revs->no_walk)\n>>   \t\treturn 0;\n>> -\tif (revs->limited)\n>> +\tif (revs->limited) {\n>>   \t\tif (limit_list(revs) < 0)\n>>   \t\t\treturn -1;\n>> -\tif (revs->topo_order)\n>> -\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n>> +\t\tif (revs->topo_order)\n>> +\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n>> +\t} else if (revs->topo_order)\n>> +\t\tinit_topo_walk(revs);\n>>   \tif (revs->line_level_traverse)\n>>   \t\tline_log_filter(revs);\n>>   \tif (revs->simplify_merges)\n> The diff is a bit hard to grok around here, but\n>\n>   - when limited *and* topo_order, we do the sort here, as we know we\n>     already have called limit_list(), i.e. we behave identically as\n>     the code before this patch in that case.\n>\n>   - when not limited but topo_order, then we do init_topo_walk();\n>     currently we do limit_list() and sort_in_topological_order(),\n>     which means we do the same as above.\n>\n> As long as limit_list() and sort_in_topological_order() does not\n> look at revs->limited bit, this patch cannot cause any regression.\n>\n>> @@ -3257,6 +3287,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n>>   \n>>   \t\tif (revs->reflog_info)\n>>   \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n>> +\t\telse if (revs->topo_walk_info)\n>> +\t\t\tcommit = next_topo_commit(revs);\n>>   \t\telse\n>>   \t\t\tcommit = pop_commit(&revs->commits);\n> So this get_revision_1() always grabs the commit from next_topo_commit()\n> when topo-order is in effect.\nAnd specifically, when the conditions for our new topo-walk algorithm \nare in effect. If the commit-graph doesn't exist, the old logic will \nstill go through for \"git log --topo-order\".\n\nThanks for the careful look!\n-Stolee\n"},{"id":"360323","messageId":"fa365691-e8ae-4aa8-5c76-b5c8c925c2f8@kdbg.org","threadId":"49229","inReplyTo":"xmqqwoqnbrmw.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2018-10-12T16:15:24Z","receivedAt":"2018-10-12T16:15:29Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 12.10.18 um 08:33 schrieb Junio C Hamano:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>> +struct topo_walk_info {};\n>> +\n>> +static void init_topo_walk(struct rev_info *revs)\n>> +{\n>> +\tstruct topo_walk_info *info;\n>> +\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n>> +\tinfo = revs->topo_walk_info;\n>> +\tmemset(info, 0, sizeof(struct topo_walk_info));\n> \n> There is no member in the struct at this point.  Are we sure this is\n> safe?  Just being curious.\n\nsizeof cannot return 0. sizeof(struct topo_walk_info) will be 1 here.\n\n-- Hannes\n"},{"id":"360383","messageId":"xmqqva668e69.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"fa365691-e8ae-4aa8-5c76-b5c8c925c2f8@kdbg.org","subject":"Re: [PATCH v3 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-13T08:05:18Z","receivedAt":"2018-10-13T08:05:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt <j6t@kdbg.org> writes:\n\n> Am 12.10.18 um 08:33 schrieb Junio C Hamano:\n>> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>>> +struct topo_walk_info {};\n>>> +\n>>> +static void init_topo_walk(struct rev_info *revs)\n>>> +{\n>>> +\tstruct topo_walk_info *info;\n>>> +\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n>>> +\tinfo = revs->topo_walk_info;\n>>> +\tmemset(info, 0, sizeof(struct topo_walk_info));\n>>\n>> There is no member in the struct at this point.  Are we sure this is\n>> safe?  Just being curious.\n>\n> sizeof cannot return 0. sizeof(struct topo_walk_info) will be 1 here.\n\nThanks.\n"},{"id":"360679","messageId":"pull.25.v4.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v4 0/7] Use generation numbers for --topo-order","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:35Z","receivedAt":"2018-10-16T22:36:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series performs a decently-sized refactoring of the revision-walk\nmachinery. Well, \"refactoring\" is probably the wrong word, as I don't\nactually remove the old code. Instead, when we see certain options in the\n'rev_info' struct, we redirect the commit-walk logic to a new set of methods\nthat distribute the workload differently. By using generation numbers in the\ncommit-graph, we can significantly improve 'git log --graph' commands (and\nthe underlying 'git rev-list --topo-order').\n\nOn the Linux repository, I got the following performance results when\ncomparing to the previous version with or without a commit-graph:\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nIf you want to read this series but are unfamiliar with the commit-graph and\ngeneration numbers, then I recommend reading \nDocumentation/technical/commit-graph.txt or a blob post [1] I wrote on the\nsubject. In particular, the three-part walk described in \"revision.c:\nrefactor basic topo-order logic\" is present (but underexplained) as an\nanimated PNG [2].\n\nSince revision.c is an incredibly important (and old) portion of the\ncodebase -- and because there are so many orthogonal options in 'struct\nrev_info' -- I consider this submission to be \"RFC quality\". That is, I am\nnot confident that I am not missing anything, or that my solution is the\nbest it can be. I did merge this branch with ds/commit-graph-with-grafts and\nthe \"DO-NOT-MERGE: write and read commit-graph always\" commit that computes\na commit-graph with every 'git commit' command. The test suite passed with\nthat change, available on GitHub [3]. To ensure that I cover at least the\ncase I think are interesting, I added tests to t6600-test-reach.sh to verify\nthe walks report the correct results for the three cases there (no\ncommit-graph, full commit-graph, and a partial commit-graph so the walk\nstarts at GENERATION_NUMBER_INFINITY).\n\nOne notable case that is not included in this series is the case of a\nhistory comparison such as 'git rev-list --topo-order A..B'. The existing\ncode in limit_list() has ways to cut the walk short when all pending commits\nare UNINTERESTING. Since this code depends on commit_list instead of the\nprio_queue we are using here, I chose to leave it untouched for now. We can\nrevisit it in a separate series later. Since handle_commit() turns on\nrevs->limited when a commit is UNINTERESTING, we do not hit the new code in\nthis case. Removing this 'revs->limited = 1;' line yields correct results,\nbut the performance is worse.\n\nThis series was based on ds/reachable, but is now based on 'master' to not\nconflict with 182070 \"commit: use timestamp_t for author_date_slab\". There\nis a small conflict with md/filter-trees, because it renamed a flag in\nrevisions.h in the line before I add new flags. Hopefully this conflict is\nnot too difficult to resolve.\n\nChanges in V3: I added a new patch that updates the tab-alignment for flags\nin revision.h before adding new ones (Thanks, Ævar!). Also, I squashed the\nrecommended changes to run_three_modes and test_three_modes from Szeder and\nJunio. Thanks!\n\nChanges in V4: I'm sending a V4 to respond to the feedback so far. Still\nlooking forward to more on the really big commit!\n\n * Removed the whitespace changes to the flags in revision.c that caused\n   merge pain. \n   \n   \n * The prio-queue peek function is now covered by tests when in \"stack\"\n   mode.\n   \n   \n * The \"add_parents_to_list()\" function is now renamed to\n   \"process_parents()\"\n   \n   \n * Added a new commit that expands test coverage with alternate orders and\n   file history (use GIT_TEST_COMMIT_GRAPH to have\n   t6012-rev-list-simplify.sh cover the new logic). These tests found a\n   problem with author dates (I forgot to record them during the explore\n   walk).\n   \n   \n * Commit message edits.\n   \n   \n\nThanks, -Stolee\n\n[1] \nhttps://blogs.msdn.microsoft.com/devops/2018/07/09/supercharging-the-git-commit-graph-iii-generations/\nSupercharging the Git Commit Graph III: Generations and Graph Algorithms\n\n[2] \nhttps://msdnshared.blob.core.windows.net/media/2018/06/commit-graph-topo-order-b-a.png\nAnimation showing three-part walk\n\n[3] https://github.com/derrickstolee/git/tree/topo-order/testA branch\ncontaining this series along with commits to compute commit-graph in entire\ntest suite.\n\nCc: avarab@gmail.comCc: szeder.dev@gmail.com\n\nDerrick Stolee (7):\n  prio-queue: add 'peek' operation\n  test-reach: add run_three_modes method\n  test-reach: add rev-list tests\n  revision.c: begin refactoring --topo-order logic\n  commit/revisions: bookkeeping before refactoring\n  revision.c: generation-based topo-order algorithm\n  t6012: make rev-list tests more interesting\n\n commit.c                     |  11 +-\n commit.h                     |   8 ++\n object.h                     |   4 +-\n prio-queue.c                 |   9 ++\n prio-queue.h                 |   6 +\n revision.c                   | 245 +++++++++++++++++++++++++++++++++--\n revision.h                   |   6 +\n t/helper/test-prio-queue.c   |  26 ++--\n t/t0009-prio-queue.sh        |  14 ++\n t/t6012-rev-list-simplify.sh |  45 +++++--\n t/t6600-test-reach.sh        |  96 +++++++++++++-\n 11 files changed, 430 insertions(+), 40 deletions(-)\n\n\nbase-commit: 2d3b1c576c85b7f5db1f418907af00ab88e0c303\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-25%2Fderrickstolee%2Ftopo-order%2Fprogress-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-25/derrickstolee/topo-order/progress-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/25\n\nRange-diff vs v3:\n\n 1:  cc1ec4c270 ! 1:  2358cfd5ed prio-queue: add 'peek' operation\n     @@ -8,7 +8,9 @@\n          add it as prio_queue_peek().\n      \n          Add a reference-level comparison in t/helper/test-prio-queue.c\n     -    so this method is exercised by t0009-prio-queue.sh.\n     +    so this method is exercised by t0009-prio-queue.sh. Further, add\n     +    a test that checks the behavior when the compare function is NULL\n     +    (i.e. the queue becomes a stack).\n      \n          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n      \n     @@ -56,6 +58,11 @@\n      -\t\tif (!strcmp(*argv, \"get\"))\n      -\t\t\tshow(prio_queue_get(&pq));\n      -\t\telse if (!strcmp(*argv, \"dump\")) {\n     +-\t\t\tint *v;\n     +-\t\t\twhile ((v = prio_queue_get(&pq)))\n     +-\t\t\t       show(v);\n     +-\t\t}\n     +-\t\telse {\n      +\t\tif (!strcmp(*argv, \"get\")) {\n      +\t\t\tvoid *peek = prio_queue_peek(&pq);\n      +\t\t\tvoid *get = prio_queue_get(&pq);\n     @@ -63,6 +70,40 @@\n      +\t\t\t\tBUG(\"peek and get results do not match\");\n      +\t\t\tshow(get);\n      +\t\t} else if (!strcmp(*argv, \"dump\")) {\n     - \t\t\tint *v;\n     - \t\t\twhile ((v = prio_queue_get(&pq)))\n     - \t\t\t       show(v);\n     ++\t\t\tvoid *peek;\n     ++\t\t\tvoid *get;\n     ++\t\t\twhile ((peek = prio_queue_peek(&pq))) {\n     ++\t\t\t\tget = prio_queue_get(&pq);\n     ++\t\t\t\tif (peek != get)\n     ++\t\t\t\t\tBUG(\"peek and get results do not match\");\n     ++\t\t\t\tshow(get);\n     ++\t\t\t}\n     ++\t\t} else if (!strcmp(*argv, \"stack\")) {\n     ++\t\t\tpq.compare = NULL;\n     ++\t\t} else {\n     + \t\t\tint *v = malloc(sizeof(*v));\n     + \t\t\t*v = atoi(*argv);\n     + \t\t\tprio_queue_put(&pq, v);\n     +\n     +diff --git a/t/t0009-prio-queue.sh b/t/t0009-prio-queue.sh\n     +--- a/t/t0009-prio-queue.sh\n     ++++ b/t/t0009-prio-queue.sh\n     +@@\n     + \ttest_cmp expect actual\n     + '\n     + \n     ++cat >expect <<'EOF'\n     ++3\n     ++2\n     ++6\n     ++4\n     ++5\n     ++1\n     ++8\n     ++EOF\n     ++test_expect_success 'stack order' '\n     ++\ttest-tool prio-queue stack 8 1 5 4 6 2 3 dump >actual &&\n     ++\ttest_cmp expect actual\n     ++'\n     ++\n     + test_done\n 2:  b2a1ade148 = 2:  3a4b68e479 test-reach: add run_three_modes method\n 3:  b0ceb96076 = 3:  12a3f6d367 test-reach: add rev-list tests\n 4:  fd1a0ab7cd = 4:  cd9eef9688 revision.c: begin refactoring --topo-order logic\n 5:  e86f304082 ! 5:  f3e291665d commit/revisions: bookkeeping before refactoring\n     @@ -16,7 +16,11 @@\n             around the UNINTERESTING flag and other special cases depending\n             on the struct rev_info. Allow this method to ignore a NULL 'list'\n             parameter, as we will not be populating the list for our walk.\n     +       Also rename the method to the slightly more generic name\n     +       process_parents() to make clear that this method does more than\n     +       add to a list (and no list is required anymore).\n      \n     +    Helped-by: Jeff King <peff@peff.net>\n          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n      \n      diff --git a/commit.c b/commit.c\n     @@ -28,7 +32,8 @@\n       \n      -/* record author-date for each commit object */\n      -define_commit_slab(author_date_slab, timestamp_t);\n     --\n     ++implement_shared_commit_slab(author_date_slab, timestamp_t);\n     + \n      -static void record_author_date(struct author_date_slab *author_date,\n      -\t\t\t       struct commit *commit)\n      +void record_author_date(struct author_date_slab *author_date,\n     @@ -64,7 +69,7 @@\n       extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n       \n      +/* record author-date for each commit object */\n     -+define_commit_slab(author_date_slab, timestamp_t);\n     ++define_shared_commit_slab(author_date_slab, timestamp_t);\n      +\n      +void record_author_date(struct author_date_slab *author_date,\n      +\t\t\tstruct commit *commit);\n     @@ -77,6 +82,17 @@\n      diff --git a/revision.c b/revision.c\n      --- a/revision.c\n      +++ b/revision.c\n     +@@\n     + \t\t*cache = new_entry;\n     + }\n     + \n     +-static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n     +-\t\t    struct commit_list **list, struct commit_list **cache_ptr)\n     ++static int process_parents(struct rev_info *revs, struct commit *commit,\n     ++\t\t\t   struct commit_list **list, struct commit_list **cache_ptr)\n     + {\n     + \tstruct commit_list *parent = commit->parents;\n     + \tunsigned left_flag;\n      @@\n       \t\t\tif (p->object.flags & SEEN)\n       \t\t\t\tcontinue;\n     @@ -97,3 +113,39 @@\n       \t\t}\n       \t\tif (revs->first_parent_only)\n       \t\t\tbreak;\n     +@@\n     + \n     + \t\tif (revs->max_age != -1 && (commit->date < revs->max_age))\n     + \t\t\tobj->flags |= UNINTERESTING;\n     +-\t\tif (add_parents_to_list(revs, commit, &list, NULL) < 0)\n     ++\t\tif (process_parents(revs, commit, &list, NULL) < 0)\n     + \t\t\treturn -1;\n     + \t\tif (obj->flags & UNINTERESTING) {\n     + \t\t\tmark_parents_uninteresting(commit);\n     +@@\n     + \n     + static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n     + {\n     +-\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n     ++\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n     + \t\tif (!revs->ignore_missing_links)\n     + \t\t\tdie(\"Failed to traverse parents of commit %s\",\n     + \t\t\t    oid_to_hex(&commit->object.oid));\n     +@@\n     + \tfor (;;) {\n     + \t\tstruct commit *p = *pp;\n     + \t\tif (!revs->limited)\n     +-\t\t\tif (add_parents_to_list(revs, p, &revs->commits, &cache) < 0)\n     ++\t\t\tif (process_parents(revs, p, &revs->commits, &cache) < 0)\n     + \t\t\t\treturn rewrite_one_error;\n     + \t\tif (p->object.flags & UNINTERESTING)\n     + \t\t\treturn rewrite_one_ok;\n     +@@\n     + \t\t\t\ttry_to_simplify_commit(revs, commit);\n     + \t\t\telse if (revs->topo_walk_info)\n     + \t\t\t\texpand_topo_walk(revs, commit);\n     +-\t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n     ++\t\t\telse if (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n     + \t\t\t\tif (!revs->ignore_missing_links)\n     + \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\n     + \t\t\t\t\t\toid_to_hex(&commit->object.oid));\n 6:  fa6d5ef152 < -:  ---------- revision.h: add whitespace in flag definitions\n 7:  020b2f50c5 ! 6:  aa0bb2221d revision.c: refactor basic topo-order logic\n     @@ -1,6 +1,15 @@\n      Author: Derrick Stolee <dstolee@microsoft.com>\n      \n     -    revision.c: refactor basic topo-order logic\n     +    revision.c: generation-based topo-order algorithm\n     +\n     +    The current --topo-order algorithm requires walking all\n     +    reachable commits up front, topo-sorting them, all before\n     +    outputting the first value. This patch introduces a new\n     +    algorithm which uses stored generation numbers to\n     +    incrementally walk in topo-order, outputting commits as\n     +    we go. This can dramatically reduce the computation time\n     +    to write a fixed number of commits, such as when limiting\n     +    with \"-n <N>\" or filling the first page of a pager.\n      \n          When running a command like 'git rev-list --topo-order HEAD',\n          Git performed the following steps:\n     @@ -139,11 +148,12 @@\n          frequently, including by merge commits. A less-frequently-changed\n          path (such as 'README') has similar end-to-end time since we need\n          to walk the same number of commits (before determining we do not\n     -    have 100 hits). However, get get the benefit that the output is\n     +    have 100 hits). However, get the benefit that the output is\n          presented to the user as it is discovered, much the same as a\n          normal 'git log' command (no '--topo-order'). This is an improved\n          user experience, even if the command has the same runtime.\n      \n     +    Helped-by: Jeff King <peff@peff.net>\n          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n      \n      diff --git a/object.h b/object.h\n     @@ -216,10 +226,13 @@\n      +\tif (parse_commit_gently(c, 1) < 0)\n      +\t\treturn;\n      +\n     ++\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n     ++\t\trecord_author_date(&info->author_date, c);\n     ++\n      +\tif (revs->max_age != -1 && (c->date < revs->max_age))\n      +\t\tc->object.flags |= UNINTERESTING;\n      +\n     -+\tif (add_parents_to_list(revs, c, NULL, NULL) < 0)\n     ++\tif (process_parents(revs, c, NULL, NULL) < 0)\n      +\t\treturn;\n      +\n      +\tif (c->object.flags & UNINTERESTING)\n     @@ -366,10 +379,10 @@\n       \n       static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n       {\n     --\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n     +-\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n      +\tstruct commit_list *p;\n      +\tstruct topo_walk_info *info = revs->topo_walk_info;\n     -+\tif (add_parents_to_list(revs, commit, NULL, NULL) < 0) {\n     ++\tif (process_parents(revs, commit, NULL, NULL) < 0) {\n       \t\tif (!revs->ignore_missing_links)\n       \t\t\tdie(\"Failed to traverse parents of commit %s\",\n      -\t\t\t    oid_to_hex(&commit->object.oid));\n     @@ -404,9 +417,9 @@\n      --- a/revision.h\n      +++ b/revision.h\n      @@\n     - #define USER_GIVEN\t\t(1u<<25) /* given directly by the user */\n     - #define TRACK_LINEAR\t\t(1u<<26)\n     - #define ALL_REV_FLAGS\t\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n     + #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n     + #define TRACK_LINEAR\t(1u<<26)\n     + #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n      +#define TOPO_WALK_EXPLORED\t(1u<<27)\n      +#define TOPO_WALK_INDEGREE\t(1u<<28)\n       \n -:  ---------- > 7:  a21febe112 t6012: make rev-list tests more interesting\n\n-- \ngitgitgadget\n"},{"id":"360681","messageId":"2358cfd5edde4f749ee237f6b9f643444c62b900.1539729393.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 1/7] prio-queue: add 'peek' operation","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:37Z","receivedAt":"2018-10-16T22:36:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen consuming a priority queue, it can be convenient to inspect\nthe next object that will be dequeued without actually dequeueing\nit. Our existing library did not have such a 'peek' operation, so\nadd it as prio_queue_peek().\n\nAdd a reference-level comparison in t/helper/test-prio-queue.c\nso this method is exercised by t0009-prio-queue.sh. Further, add\na test that checks the behavior when the compare function is NULL\n(i.e. the queue becomes a stack).\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n prio-queue.c               |  9 +++++++++\n prio-queue.h               |  6 ++++++\n t/helper/test-prio-queue.c | 26 ++++++++++++++++++--------\n t/t0009-prio-queue.sh      | 14 ++++++++++++++\n 4 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/prio-queue.c b/prio-queue.c\nindex a078451872..d3f488cb05 100644\n--- a/prio-queue.c\n+++ b/prio-queue.c\n@@ -85,3 +85,12 @@ void *prio_queue_get(struct prio_queue *queue)\n \t}\n \treturn result;\n }\n+\n+void *prio_queue_peek(struct prio_queue *queue)\n+{\n+\tif (!queue->nr)\n+\t\treturn NULL;\n+\tif (!queue->compare)\n+\t\treturn queue->array[queue->nr - 1].data;\n+\treturn queue->array[0].data;\n+}\ndiff --git a/prio-queue.h b/prio-queue.h\nindex d030ec9dd6..682e51867a 100644\n--- a/prio-queue.h\n+++ b/prio-queue.h\n@@ -46,6 +46,12 @@ extern void prio_queue_put(struct prio_queue *, void *thing);\n  */\n extern void *prio_queue_get(struct prio_queue *);\n \n+/*\n+ * Gain access to the \"thing\" that would be returned by\n+ * prio_queue_get, but do not remove it from the queue.\n+ */\n+extern void *prio_queue_peek(struct prio_queue *);\n+\n extern void clear_prio_queue(struct prio_queue *);\n \n /* Reverse the LIFO elements */\ndiff --git a/t/helper/test-prio-queue.c b/t/helper/test-prio-queue.c\nindex 9807b649b1..5bc9c46ea5 100644\n--- a/t/helper/test-prio-queue.c\n+++ b/t/helper/test-prio-queue.c\n@@ -22,14 +22,24 @@ int cmd__prio_queue(int argc, const char **argv)\n \tstruct prio_queue pq = { intcmp };\n \n \twhile (*++argv) {\n-\t\tif (!strcmp(*argv, \"get\"))\n-\t\t\tshow(prio_queue_get(&pq));\n-\t\telse if (!strcmp(*argv, \"dump\")) {\n-\t\t\tint *v;\n-\t\t\twhile ((v = prio_queue_get(&pq)))\n-\t\t\t       show(v);\n-\t\t}\n-\t\telse {\n+\t\tif (!strcmp(*argv, \"get\")) {\n+\t\t\tvoid *peek = prio_queue_peek(&pq);\n+\t\t\tvoid *get = prio_queue_get(&pq);\n+\t\t\tif (peek != get)\n+\t\t\t\tBUG(\"peek and get results do not match\");\n+\t\t\tshow(get);\n+\t\t} else if (!strcmp(*argv, \"dump\")) {\n+\t\t\tvoid *peek;\n+\t\t\tvoid *get;\n+\t\t\twhile ((peek = prio_queue_peek(&pq))) {\n+\t\t\t\tget = prio_queue_get(&pq);\n+\t\t\t\tif (peek != get)\n+\t\t\t\t\tBUG(\"peek and get results do not match\");\n+\t\t\t\tshow(get);\n+\t\t\t}\n+\t\t} else if (!strcmp(*argv, \"stack\")) {\n+\t\t\tpq.compare = NULL;\n+\t\t} else {\n \t\t\tint *v = malloc(sizeof(*v));\n \t\t\t*v = atoi(*argv);\n \t\t\tprio_queue_put(&pq, v);\ndiff --git a/t/t0009-prio-queue.sh b/t/t0009-prio-queue.sh\nindex e56dfce668..3941ad2528 100755\n--- a/t/t0009-prio-queue.sh\n+++ b/t/t0009-prio-queue.sh\n@@ -47,4 +47,18 @@ test_expect_success 'notice empty queue' '\n \ttest_cmp expect actual\n '\n \n+cat >expect <<'EOF'\n+3\n+2\n+6\n+4\n+5\n+1\n+8\n+EOF\n+test_expect_success 'stack order' '\n+\ttest-tool prio-queue stack 8 1 5 4 6 2 3 dump >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"360680","messageId":"12a3f6d3670834f19b38f5e23ef83cdf80a58c33.1539729393.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 3/7] test-reach: add rev-list tests","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:41Z","receivedAt":"2018-10-16T22:36:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe rev-list command is critical to Git's functionality. Ensure it\nworks in the three commit-graph environments constructed in\nt6600-test-reach.sh. Here are a few important types of rev-list\noperations:\n\n* Basic: git rev-list --topo-order HEAD\n* Range: git rev-list --topo-order compare..HEAD\n* Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n* Symmetric Difference: git rev-list --topo-order compare...HEAD\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 84 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 84 insertions(+)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 9d65b8b946..288f703b7b 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -243,4 +243,88 @@ test_expect_success 'commit_contains:miss' '\n \ttest_three_modes commit_contains --tag\n '\n \n+test_expect_success 'rev-list: basic topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 commit-3-3 commit-2-3 commit-1-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-6-6\n+'\n+\n+test_expect_success 'rev-list: first-parent topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --first-parent --topo-order commit-6-6\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-3..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-8..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: first-parent range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --first-parent --topo-order commit-3-8..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: ancestry-path topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: symmetric difference topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t\tcommit-3-8 commit-2-8 commit-1-8 \\\n+\t\tcommit-3-7 commit-2-7 commit-1-7 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-8...commit-6-6\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"360682","messageId":"cd9eef96885a811196ab0b851a98de4455b950ab.1539729393.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:42Z","receivedAt":"2018-10-16T22:36:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen running 'git rev-list --topo-order' and its kin, the topo_order\nsetting in struct rev_info implies the limited setting. This means\nthat the following things happen during prepare_revision_walk():\n\n* revs->limited implies we run limit_list() to walk the entire\n  reachable set. There are some short-cuts here, such as if we\n  perform a range query like 'git rev-list COMPARE..HEAD' and we\n  can stop limit_list() when all queued commits are uninteresting.\n\n* revs->topo_order implies we run sort_in_topological_order(). See\n  the implementation of that method in commit.c. It implies that\n  the full set of commits to order is in the given commit_list.\n\nThese two methods imply that a 'git rev-list --topo-order HEAD'\ncommand must walk the entire reachable set of commits _twice_ before\nreturning a single result.\n\nIf we have a commit-graph file with generation numbers computed, then\nthere is a better way. This patch introduces some necessary logic\nredirection when we are in this situation.\n\nIn v2.18.0, the commit-graph file contains zero-valued bytes in the\npositions where the generation number is stored in v2.19.0 and later.\nThus, we use generation_numbers_enabled() to check if the commit-graph\nis available and has non-zero generation numbers.\n\nWhen setting revs->limited only because revs->topo_order is true,\nonly do so if generation numbers are not available. There is no\nreason to use the new logic as it will behave similarly when all\ngeneration numbers are INFINITY or ZERO.\n\nIn prepare_revision_walk(), if we have revs->topo_order but not\nrevs->limited, then we trigger the new logic. It breaks the logic\ninto three pieces, to fit with the existing framework:\n\n1. init_topo_walk() fills a new struct topo_walk_info in the rev_info\n   struct. We use the presence of this struct as a signal to use the\n   new methods during our walk. In this patch, this method simply\n   calls limit_list() and sort_in_topological_order(). In the future,\n   this method will set up a new data structure to perform that logic\n   in-line.\n\n2. next_topo_commit() provides get_revision_1() with the next topo-\n   ordered commit in the list. Currently, this simply pops the commit\n   from revs->commits.\n\n3. expand_topo_walk() provides get_revision_1() with a way to signal\n   walking beyond the latest commit. Currently, this calls\n   add_parents_to_list() exactly like the old logic.\n\nWhile this commit presents method redirection for performing the\nexact same logic as before, it allows the next commit to focus only\non the new logic.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 42 ++++++++++++++++++++++++++++++++++++++----\n revision.h |  4 ++++\n 2 files changed, 42 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex e18bd530e4..2dcde8a8ac 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -25,6 +25,7 @@\n #include \"worktree.h\"\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n+#include \"commit-graph.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2454,7 +2455,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \tif (revs->diffopt.objfind)\n \t\trevs->simplify_history = 0;\n \n-\tif (revs->topo_order)\n+\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n \t\trevs->limited = 1;\n \n \tif (revs->prune_data.nr) {\n@@ -2892,6 +2893,33 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n+struct topo_walk_info {};\n+\n+static void init_topo_walk(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info;\n+\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n+\tinfo = revs->topo_walk_info;\n+\tmemset(info, 0, sizeof(struct topo_walk_info));\n+\n+\tlimit_list(revs);\n+\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+}\n+\n+static struct commit *next_topo_commit(struct rev_info *revs)\n+{\n+\treturn pop_commit(&revs->commits);\n+}\n+\n+static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n+{\n+\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\t\tif (!revs->ignore_missing_links)\n+\t\t\tdie(\"Failed to traverse parents of commit %s\",\n+\t\t\t    oid_to_hex(&commit->object.oid));\n+\t}\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -2928,11 +2956,13 @@ int prepare_revision_walk(struct rev_info *revs)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\n-\tif (revs->limited)\n+\tif (revs->limited) {\n \t\tif (limit_list(revs) < 0)\n \t\t\treturn -1;\n-\tif (revs->topo_order)\n-\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t\tif (revs->topo_order)\n+\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t} else if (revs->topo_order)\n+\t\tinit_topo_walk(revs);\n \tif (revs->line_level_traverse)\n \t\tline_log_filter(revs);\n \tif (revs->simplify_merges)\n@@ -3257,6 +3287,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\tif (revs->reflog_info)\n \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n+\t\telse if (revs->topo_walk_info)\n+\t\t\tcommit = next_topo_commit(revs);\n \t\telse\n \t\t\tcommit = pop_commit(&revs->commits);\n \n@@ -3278,6 +3310,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\t\tif (revs->reflog_info)\n \t\t\t\ttry_to_simplify_commit(revs, commit);\n+\t\t\telse if (revs->topo_walk_info)\n+\t\t\t\texpand_topo_walk(revs, commit);\n \t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n \t\t\t\tif (!revs->ignore_missing_links)\n \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\ndiff --git a/revision.h b/revision.h\nindex 2b30ac270d..fd4154ff75 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -56,6 +56,8 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct topo_walk_info;\n+\n struct rev_info {\n \t/* Starting list */\n \tstruct commit_list *commits;\n@@ -245,6 +247,8 @@ struct rev_info {\n \tconst char *break_bar;\n \n \tstruct revision_sources *sources;\n+\n+\tstruct topo_walk_info *topo_walk_info;\n };\n \n int ref_excluded(struct string_list *, const char *path);\n-- \ngitgitgadget\n\n"},{"id":"360683","messageId":"3a4b68e479aa6c270ef1af3aae3222c3310c2935.1539729393.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 2/7] test-reach: add run_three_modes method","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:39Z","receivedAt":"2018-10-16T22:36:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe 'test_three_modes' method assumes we are using the 'test-tool\nreach' command for our test. However, we may want to use the data\nshape of our commit graph and the three modes (no commit-graph,\nfull commit-graph, partial commit-graph) for other git commands.\n\nSplit test_three_modes to be a simple translation on a more general\nrun_three_modes method that executes the given command and tests\nthe actual output to the expected output.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex d139a00d1d..9d65b8b946 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -53,18 +53,22 @@ test_expect_success 'setup' '\n \tgit config core.commitGraph true\n '\n \n-test_three_modes () {\n+run_three_modes () {\n \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-full .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-half .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual\n }\n \n+test_three_modes () {\n+\trun_three_modes test-tool reach \"$@\"\n+}\n+\n test_expect_success 'ref_newer:miss' '\n \tcat >input <<-\\EOF &&\n \tA:commit-5-7\n-- \ngitgitgadget\n\n"},{"id":"360684","messageId":"f3e291665dae94b7347621ec78721f7e0dcc86d8.1539729393.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 5/7] commit/revisions: bookkeeping before refactoring","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:43Z","receivedAt":"2018-10-16T22:36:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThere are a few things that need to move around a little before\nmaking a big refactoring in the topo-order logic:\n\n1. We need access to record_author_date() and\n   compare_commits_by_author_date() in revision.c. These are used\n   currently by sort_in_topological_order() in commit.c.\n\n2. Moving these methods to commit.h requires adding the author_slab\n   definition to commit.h.\n\n3. The add_parents_to_list() method in revision.c performs logic\n   around the UNINTERESTING flag and other special cases depending\n   on the struct rev_info. Allow this method to ignore a NULL 'list'\n   parameter, as we will not be populating the list for our walk.\n   Also rename the method to the slightly more generic name\n   process_parents() to make clear that this method does more than\n   add to a list (and no list is required anymore).\n\nHelped-by: Jeff King <peff@peff.net>\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n commit.c   | 11 +++++------\n commit.h   |  8 ++++++++\n revision.c | 18 ++++++++++--------\n 3 files changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex d0f199e122..861a485e93 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -655,11 +655,10 @@ struct commit *pop_commit(struct commit_list **stack)\n /* count number of children that have not been emitted */\n define_commit_slab(indegree_slab, int);\n \n-/* record author-date for each commit object */\n-define_commit_slab(author_date_slab, timestamp_t);\n+implement_shared_commit_slab(author_date_slab, timestamp_t);\n \n-static void record_author_date(struct author_date_slab *author_date,\n-\t\t\t       struct commit *commit)\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit)\n {\n \tconst char *buffer = get_commit_buffer(commit, NULL);\n \tstruct ident_split ident;\n@@ -684,8 +683,8 @@ fail_exit:\n \tunuse_commit_buffer(commit, buffer);\n }\n \n-static int compare_commits_by_author_date(const void *a_, const void *b_,\n-\t\t\t\t\t  void *cb_data)\n+int compare_commits_by_author_date(const void *a_, const void *b_,\n+\t\t\t\t   void *cb_data)\n {\n \tconst struct commit *a = a_, *b = b_;\n \tstruct author_date_slab *author_date = cb_data;\ndiff --git a/commit.h b/commit.h\nindex 2b1a734388..977d397356 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -8,6 +8,7 @@\n #include \"gpg-interface.h\"\n #include \"string-list.h\"\n #include \"pretty.h\"\n+#include \"commit-slab.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n #define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n@@ -328,6 +329,13 @@ extern int remove_signature(struct strbuf *buf);\n  */\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n+/* record author-date for each commit object */\n+define_shared_commit_slab(author_date_slab, timestamp_t);\n+\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit);\n+\n+int compare_commits_by_author_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \ndiff --git a/revision.c b/revision.c\nindex 2dcde8a8ac..36458265a0 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -768,8 +768,8 @@ static void commit_list_insert_by_date_cached(struct commit *p, struct commit_li\n \t\t*cache = new_entry;\n }\n \n-static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n-\t\t    struct commit_list **list, struct commit_list **cache_ptr)\n+static int process_parents(struct rev_info *revs, struct commit *commit,\n+\t\t\t   struct commit_list **list, struct commit_list **cache_ptr)\n {\n \tstruct commit_list *parent = commit->parents;\n \tunsigned left_flag;\n@@ -808,7 +808,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\treturn 0;\n \t}\n@@ -847,7 +848,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\tif (revs->first_parent_only)\n \t\t\tbreak;\n@@ -1091,7 +1093,7 @@ static int limit_list(struct rev_info *revs)\n \n \t\tif (revs->max_age != -1 && (commit->date < revs->max_age))\n \t\t\tobj->flags |= UNINTERESTING;\n-\t\tif (add_parents_to_list(revs, commit, &list, NULL) < 0)\n+\t\tif (process_parents(revs, commit, &list, NULL) < 0)\n \t\t\treturn -1;\n \t\tif (obj->flags & UNINTERESTING) {\n \t\t\tmark_parents_uninteresting(commit);\n@@ -2913,7 +2915,7 @@ static struct commit *next_topo_commit(struct rev_info *revs)\n \n static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n {\n-\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n \t\tif (!revs->ignore_missing_links)\n \t\t\tdie(\"Failed to traverse parents of commit %s\",\n \t\t\t    oid_to_hex(&commit->object.oid));\n@@ -2979,7 +2981,7 @@ static enum rewrite_result rewrite_one(struct rev_info *revs, struct commit **pp\n \tfor (;;) {\n \t\tstruct commit *p = *pp;\n \t\tif (!revs->limited)\n-\t\t\tif (add_parents_to_list(revs, p, &revs->commits, &cache) < 0)\n+\t\t\tif (process_parents(revs, p, &revs->commits, &cache) < 0)\n \t\t\t\treturn rewrite_one_error;\n \t\tif (p->object.flags & UNINTERESTING)\n \t\t\treturn rewrite_one_ok;\n@@ -3312,7 +3314,7 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \t\t\t\ttry_to_simplify_commit(revs, commit);\n \t\t\telse if (revs->topo_walk_info)\n \t\t\t\texpand_topo_walk(revs, commit);\n-\t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\t\t\telse if (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n \t\t\t\tif (!revs->ignore_missing_links)\n \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\n \t\t\t\t\t\toid_to_hex(&commit->object.oid));\n-- \ngitgitgadget\n\n"},{"id":"360685","messageId":"aa0bb2221d437f595f6d0c16b125072fde289c15.1539729393.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 6/7] revision.c: generation-based topo-order algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:45Z","receivedAt":"2018-10-16T22:36:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe current --topo-order algorithm requires walking all\nreachable commits up front, topo-sorting them, all before\noutputting the first value. This patch introduces a new\nalgorithm which uses stored generation numbers to\nincrementally walk in topo-order, outputting commits as\nwe go. This can dramatically reduce the computation time\nto write a fixed number of commits, such as when limiting\nwith \"-n <N>\" or filling the first page of a pager.\n\nWhen running a command like 'git rev-list --topo-order HEAD',\nGit performed the following steps:\n\n1. Run limit_list(), which parses all reachable commits,\n   adds them to a linked list, and distributes UNINTERESTING\n   flags. If all unprocessed commits are UNINTERESTING, then\n   it may terminate without walking all reachable commits.\n   This does not occur if we do not specify UNINTERESTING\n   commits.\n\n2. Run sort_in_topological_order(), which is an implementation\n   of Kahn's algorithm. It first iterates through the entire\n   set of important commits and computes the in-degree of each\n   (plus one, as we use 'zero' as a special value here). Then,\n   we walk the commits in priority order, adding them to the\n   priority queue if and only if their in-degree is one. As\n   we remove commits from this priority queue, we decrement the\n   in-degree of their parents.\n\n3. While we are peeling commits for output, get_revision_1()\n   uses pop_commit on the full list of commits computed by\n   sort_in_topological_order().\n\nIn the new algorithm, these three steps correspond to three\ndifferent commit walks. We run these walks simultaneously,\nand advance each only as far as necessary to satisfy the\nrequirements of the 'higher order' walk. We know when we can\npause each walk by using generation numbers from the commit-\ngraph feature.\n\nRecall that the generation number of a commit satisfies:\n\n* If the commit has at least one parent, then the generation\n  number is one more than the maximum generation number among\n  its parents.\n\n* If the commit has no parent, then the generation number is one.\n\nThere are two special generation numbers:\n\n* GENERATION_NUMBER_INFINITY: this value is 0xffffffff and\n  indicates that the commit is not stored in the commit-graph and\n  the generation number was not previously calculated.\n\n* GENERATION_NUMBER_ZERO: this value (0) is a special indicator\n  to say that the commit-graph was generated by a version of Git\n  that does not compute generation numbers (such as v2.18.0).\n\nSince we use generation_numbers_enabled() before using the new\nalgorithm, we do not need to worry about GENERATION_NUMBER_ZERO.\nHowever, the existence of GENERATION_NUMBER_INFINITY implies the\nfollowing weaker statement than the usual we expect from\ngeneration numbers:\n\n    If A and B are commits with generation numbers gen(A) and\n    gen(B) and gen(A) < gen(B), then A cannot reach B.\n\nThus, we will walk in each of our stages until the \"maximum\nunexpanded generation number\" is strictly lower than the\ngeneration number of a commit we are about to use.\n\nThe walks are as follows:\n\n1. EXPLORE: using the explore_queue priority queue (ordered by\n   maximizing the generation number), parse each reachable\n   commit until all commits in the queue have generation\n   number strictly lower than needed. During this walk, update\n   the UNINTERESTING flags as necessary.\n\n2. INDEGREE: using the indegree_queue priority queue (ordered\n   by maximizing the generation number), add one to the in-\n   degree of each parent for each commit that is walked. Since\n   we walk in order of decreasing generation number, we know\n   that discovering an in-degree value of 0 means the value for\n   that commit was not initialized, so should be initialized to\n   two. (Recall that in-degree value \"1\" is what we use to say a\n   commit is ready for output.) As we iterate the parents of a\n   commit during this walk, ensure the EXPLORE walk has walked\n   beyond their generation numbers.\n\n3. TOPO: using the topo_queue priority queue (ordered based on\n   the sort_order given, which could be commit-date, author-\n   date, or typical topo-order which treats the queue as a LIFO\n   stack), remove a commit from the queue and decrement the\n   in-degree of each parent. If a parent has an in-degree of\n   one, then we add it to the topo_queue. Before we decrement\n   the in-degree, however, ensure the INDEGREE walk has walked\n   beyond that generation number.\n\nThe implementations of these walks are in the following methods:\n\n* explore_walk_step and explore_to_depth\n* indegree_walk_step and compute_indegrees_to_depth\n* next_topo_commit and expand_topo_walk\n\nThese methods have some patterns that may seem strange at first,\nbut they are probably carry-overs from their equivalents in\nlimit_list and sort_in_topological_order.\n\nOne thing that is missing from this implementation is a proper\nway to stop walking when the entire queue is UNINTERESTING, so\nthis implementation is not enabled by comparisions, such as in\n'git rev-list --topo-order A..B'. This can be updated in the\nfuture.\n\nIn my local testing, I used the following Git commands on the\nLinux repository in three modes: HEAD~1 with no commit-graph,\nHEAD~1 with a commit-graph, and HEAD with a commit-graph. This\nallows comparing the benefits we get from parsing commits from\nthe commit-graph and then again the benefits we get by\nrestricting the set of commits we walk.\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nThis speedup is due to a few things. First, the new generation-\nnumber-enabled algorithm walks commits on order of the number of\nresults output (subject to some branching structure expectations).\nSince we limit to 100 results, we are running a query similar to\nfilling a single page of results. Second, when specifying a path,\nwe must parse the root tree object for each commit we walk. The\nprevious benefits from the commit-graph are entirely from reading\nthe commit-graph instead of parsing commits. Since we need to\nparse trees for the same number of commits as before, we slow\ndown significantly from the non-path-based query.\n\nFor the test above, I specifically selected a path that is changed\nfrequently, including by merge commits. A less-frequently-changed\npath (such as 'README') has similar end-to-end time since we need\nto walk the same number of commits (before determining we do not\nhave 100 hits). However, get the benefit that the output is\npresented to the user as it is discovered, much the same as a\nnormal 'git log' command (no '--topo-order'). This is an improved\nuser experience, even if the command has the same runtime.\n\nHelped-by: Jeff King <peff@peff.net>\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n object.h   |   4 +-\n revision.c | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n revision.h |   2 +\n 3 files changed, 197 insertions(+), 8 deletions(-)\n\ndiff --git a/object.h b/object.h\nindex 0feb90ae61..796792cb32 100644\n--- a/object.h\n+++ b/object.h\n@@ -59,7 +59,7 @@ struct object_array {\n \n /*\n  * object flag allocation:\n- * revision.h:               0---------10                              2526\n+ * revision.h:               0---------10                              25----28\n  * fetch-pack.c:             01\n  * negotiator/default.c:       2--5\n  * walker.c:                 0-2\n@@ -78,7 +78,7 @@ struct object_array {\n  * builtin/show-branch.c:    0-------------------------------------------26\n  * builtin/unpack-objects.c:                                 2021\n  */\n-#define FLAG_BITS  27\n+#define FLAG_BITS  29\n \n /*\n  * The object type is stored in 3 bits.\ndiff --git a/revision.c b/revision.c\nindex 36458265a0..472f3994e3 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -26,6 +26,7 @@\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"prio-queue.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2895,30 +2896,216 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct topo_walk_info {};\n+define_commit_slab(indegree_slab, int);\n+\n+struct topo_walk_info {\n+\tuint32_t min_generation;\n+\tstruct prio_queue explore_queue;\n+\tstruct prio_queue indegree_queue;\n+\tstruct prio_queue topo_queue;\n+\tstruct indegree_slab indegree;\n+\tstruct author_date_slab author_date;\n+};\n+\n+static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n+{\n+\tif (c->object.flags & flag)\n+\t\treturn;\n+\n+\tc->object.flags |= flag;\n+\tprio_queue_put(q, c);\n+}\n+\n+static void explore_walk_step(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit_list *p;\n+\tstruct commit *c = prio_queue_get(&info->explore_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n+\t\trecord_author_date(&info->author_date, c);\n+\n+\tif (revs->max_age != -1 && (c->date < revs->max_age))\n+\t\tc->object.flags |= UNINTERESTING;\n+\n+\tif (process_parents(revs, c, NULL, NULL) < 0)\n+\t\treturn;\n+\n+\tif (c->object.flags & UNINTERESTING)\n+\t\tmark_parents_uninteresting(c);\n+\n+\tfor (p = c->parents; p; p = p->next)\n+\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n+}\n+\n+static void explore_to_depth(struct rev_info *revs,\n+\t\t\t     uint32_t gen)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n+\t       c->generation >= gen)\n+\t\texplore_walk_step(revs);\n+}\n+\n+static void indegree_walk_step(struct rev_info *revs)\n+{\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c = prio_queue_get(&info->indegree_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\texplore_to_depth(revs, c->generation);\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tfor (p = c->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\tif (*pi)\n+\t\t\t(*pi)++;\n+\t\telse\n+\t\t\t*pi = 2;\n+\n+\t\ttest_flag_and_insert(&info->indegree_queue, parent, TOPO_WALK_INDEGREE);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n+\t}\n+}\n+\n+static void compute_indegrees_to_depth(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n+\t       c->generation >= info->min_generation)\n+\t\tindegree_walk_step(revs);\n+}\n \n static void init_topo_walk(struct rev_info *revs)\n {\n \tstruct topo_walk_info *info;\n+\tstruct commit_list *list;\n \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n \tinfo = revs->topo_walk_info;\n \tmemset(info, 0, sizeof(struct topo_walk_info));\n \n-\tlimit_list(revs);\n-\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\tinit_indegree_slab(&info->indegree);\n+\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n+\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n+\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n+\n+\tswitch (revs->sort_order) {\n+\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n+\t\tinfo->topo_queue.compare = NULL;\n+\t\tbreak;\n+\tcase REV_SORT_BY_COMMIT_DATE:\n+\t\tinfo->topo_queue.compare = compare_commits_by_commit_date;\n+\t\tbreak;\n+\tcase REV_SORT_BY_AUTHOR_DATE:\n+\t\tinit_author_date_slab(&info->author_date);\n+\t\tinfo->topo_queue.compare = compare_commits_by_author_date;\n+\t\tinfo->topo_queue.cb_data = &info->author_date;\n+\t\tbreak;\n+\t}\n+\n+\tinfo->explore_queue.compare = compare_commits_by_gen_then_commit_date;\n+\tinfo->indegree_queue.compare = compare_commits_by_gen_then_commit_date;\n+\n+\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n+\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n+\n+\t\tif (parse_commit_gently(c, 1))\n+\t\t\tcontinue;\n+\t\tif (c->generation < info->min_generation)\n+\t\t\tinfo->min_generation = c->generation;\n+\t}\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n+\n+\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n+\t\t\trecord_author_date(&info->author_date, c);\n+\t}\n+\tcompute_indegrees_to_depth(revs);\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\n+\t\tif (*(indegree_slab_at(&info->indegree, c)) == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, c);\n+\t}\n+\n+\t/*\n+\t * This is unfortunate; the initial tips need to be shown\n+\t * in the order given from the revision traversal machinery.\n+\t */\n+\tif (revs->sort_order == REV_SORT_IN_GRAPH_ORDER)\n+\t\tprio_queue_reverse(&info->topo_queue);\n }\n \n static struct commit *next_topo_commit(struct rev_info *revs)\n {\n-\treturn pop_commit(&revs->commits);\n+\tstruct commit *c;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\n+\t/* pop next off of topo_queue */\n+\tc = prio_queue_get(&info->topo_queue);\n+\n+\tif (c)\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 0;\n+\n+\treturn c;\n }\n \n static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n {\n-\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tif (process_parents(revs, commit, NULL, NULL) < 0) {\n \t\tif (!revs->ignore_missing_links)\n \t\t\tdie(\"Failed to traverse parents of commit %s\",\n-\t\t\t    oid_to_hex(&commit->object.oid));\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t}\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi;\n+\n+\t\tif (parse_commit_gently(parent, 1) < 0)\n+\t\t\tcontinue;\n+\n+\t\tif (parent->generation < info->min_generation) {\n+\t\t\tinfo->min_generation = parent->generation;\n+\t\t\tcompute_indegrees_to_depth(revs);\n+\t\t}\n+\n+\t\tpi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\t(*pi)--;\n+\t\tif (*pi == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, parent);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n \t}\n }\n \ndiff --git a/revision.h b/revision.h\nindex fd4154ff75..b0b3bb8025 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -24,6 +24,8 @@\n #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n #define TRACK_LINEAR\t(1u<<26)\n #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define TOPO_WALK_EXPLORED\t(1u<<27)\n+#define TOPO_WALK_INDEGREE\t(1u<<28)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \ngitgitgadget\n\n"},{"id":"360686","messageId":"a21febe1128cbaf69f2201598d9efee6f6a4b2d8.1539729393.git.gitgitgadget@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 7/7] t6012: make rev-list tests more interesting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-16T22:36:46Z","receivedAt":"2018-10-16T22:36:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nAs we are working to rewrite some of the revision-walk machinery,\nthere could easily be some interesting interactions between the\noptions that force topological constraints (--topo-order,\n--date-order, and --author-date-order) along with specifying a\npath.\n\nAdd extra tests to t6012-rev-list-simplify.sh to add coverage of\nthese interactions. To ensure interesting things occur, alter the\nrepo data shape to have different orders depending on topo-, date-,\nor author-date-order.\n\nWhen testing using GIT_TEST_COMMIT_GRAPH, this assists in covering\nthe new logic for topo-order walks using generation numbers. The\nextra tests can be added indepently.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6012-rev-list-simplify.sh | 45 ++++++++++++++++++++++++++++--------\n 1 file changed, 36 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b5a1190ffe..a10f0df02b 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -12,6 +12,22 @@ unnote () {\n \tgit name-rev --tags --stdin | sed -e \"s|$OID_REGEX (tags/\\([^)]*\\)) |\\1 |g\"\n }\n \n+#\n+# Create a test repo with interesting commit graph:\n+#\n+# A--B----------G--H--I--K--L\n+#  \\  \\           /     /\n+#   \\  \\         /     /\n+#    C------E---F     J\n+#        \\_/\n+#\n+# The commits are laid out from left-to-right starting with\n+# the root commit A and terminating at the tip commit L.\n+#\n+# There are a few places where we adjust the commit date or\n+# author date to make the --topo-order, --date-order, and\n+# --author-date-order flags produce different output.\n+\n test_expect_success setup '\n \techo \"Hi there\" >file &&\n \techo \"initial\" >lost &&\n@@ -21,10 +37,18 @@ test_expect_success setup '\n \n \tgit branch other-branch &&\n \n+\tgit symbolic-ref HEAD refs/heads/unrelated &&\n+\tgit rm -f \"*\" &&\n+\techo \"Unrelated branch\" >side &&\n+\tgit add side &&\n+\ttest_tick && git commit -m \"Side root\" &&\n+\tnote J &&\n+\tgit checkout master &&\n+\n \techo \"Hello\" >file &&\n \techo \"second\" >lost &&\n \tgit add file lost &&\n-\ttest_tick && git commit -m \"Modified file and lost\" &&\n+\ttest_tick && GIT_AUTHOR_DATE=$(($test_tick + 120)) git commit -m \"Modified file and lost\" &&\n \tnote B &&\n \n \tgit checkout other-branch &&\n@@ -63,13 +87,6 @@ test_expect_success setup '\n \ttest_tick && git commit -a -m \"Final change\" &&\n \tnote I &&\n \n-\tgit symbolic-ref HEAD refs/heads/unrelated &&\n-\tgit rm -f \"*\" &&\n-\techo \"Unrelated branch\" >side &&\n-\tgit add side &&\n-\ttest_tick && git commit -m \"Side root\" &&\n-\tnote J &&\n-\n \tgit checkout master &&\n \ttest_tick && git merge --allow-unrelated-histories -m \"Coolest\" unrelated &&\n \tnote K &&\n@@ -103,14 +120,24 @@ check_result () {\n \tcheck_outcome success \"$@\"\n }\n \n-check_result 'L K J I H G F E D C B A' --full-history\n+check_result 'L K J I H F E D C G B A' --full-history --topo-order\n+check_result 'L K I H G F E D C B J A' --full-history\n+check_result 'L K I H G F E D C B J A' --full-history --date-order\n+check_result 'L K I H G F E D B C J A' --full-history --author-date-order\n check_result 'K I H E C B A' --full-history -- file\n check_result 'K I H E C B A' --full-history --topo-order -- file\n check_result 'K I H E C B A' --full-history --date-order -- file\n+check_result 'K I H E B C A' --full-history --author-date-order -- file\n check_result 'I E C B A' --simplify-merges -- file\n+check_result 'I E C B A' --simplify-merges --topo-order -- file\n+check_result 'I E C B A' --simplify-merges --date-order -- file\n+check_result 'I E B C A' --simplify-merges --author-date-order -- file\n check_result 'I B A' -- file\n check_result 'I B A' --topo-order -- file\n+check_result 'I B A' --date-order -- file\n+check_result 'I B A' --author-date-order -- file\n check_result 'H' --first-parent -- another-file\n+check_result 'H' --first-parent --topo-order -- another-file\n \n check_result 'E C B A' --full-history E -- lost\n test_expect_success 'full history simplification without parent' '\n-- \ngitgitgadget\n"},{"id":"361089","messageId":"865zxvior3.fsf@gmail.com","threadId":"49229","inReplyTo":"12a3f6d3670834f19b38f5e23ef83cdf80a58c33.1539729393.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 3/7] test-reach: add rev-list tests","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-21T10:21:36Z","receivedAt":"2018-10-21T10:25:51Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> The rev-list command is critical to Git's functionality. Ensure it\n> works in the three commit-graph environments constructed in\n> t6600-test-reach.sh. Here are a few important types of rev-list\n> operations:\n>\n> * Basic: git rev-list --topo-order HEAD\n> * Range: git rev-list --topo-order compare..HEAD\n> * Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n> * Symmetric Difference: git rev-list --topo-order compare...HEAD\n\nCould you remind us here which of those operations will be using\ngeneration numbers after this patch series?\n\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  t/t6600-test-reach.sh | 84 +++++++++++++++++++++++++++++++++++++++++++\n>  1 file changed, 84 insertions(+)\n>\n> diff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\n> index 9d65b8b946..288f703b7b 100755\n> --- a/t/t6600-test-reach.sh\n> +++ b/t/t6600-test-reach.sh\n> @@ -243,4 +243,88 @@ test_expect_success 'commit_contains:miss' '\n>  \ttest_three_modes commit_contains --tag\n>  '\n>  \n> +test_expect_success 'rev-list: basic topo-order' '\n> +\tgit rev-parse \\\n> +\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n> +\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n> +\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n> +\t\tcommit-6-3 commit-5-3 commit-4-3 commit-3-3 commit-2-3 commit-1-3 \\\n> +\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n> +\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n> +\t>expect &&\n> +\trun_three_modes git rev-list --topo-order commit-6-6\n> +'\n\nI wonder if this test could be make easier to write and less error\nprone, e.g. creating it from ASCII-art graphics.\n\nBut it is good enough.\n\n[...]\n\n--\nJakub Narębski\n"},{"id":"361091","messageId":"86zhv7h2y9.fsf@gmail.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/7] Use generation numbers for --topo-order","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-21T12:57:50Z","receivedAt":"2018-10-21T12:58:00Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> This patch series performs a decently-sized refactoring of the revision-walk\n> machinery. Well, \"refactoring\" is probably the wrong word, as I don't\n> actually remove the old code. Instead, when we see certain options in the\n> 'rev_info' struct, we redirect the commit-walk logic to a new set of methods\n> that distribute the workload differently. By using generation numbers in the\n> commit-graph, we can significantly improve 'git log --graph' commands (and\n> the underlying 'git rev-list --topo-order').\n>\n> On the Linux repository, I got the following performance results when\n> comparing to the previous version with or without a commit-graph:\n>\n> Test: git rev-list --topo-order -100 HEAD\n> HEAD~1, no commit-graph: 6.80 s\n> HEAD~1, w/ commit-graph: 0.77 s\n>   HEAD, w/ commit-graph: 0.02 s\n>\n> Test: git rev-list --topo-order -100 HEAD -- tools\n> HEAD~1, no commit-graph: 9.63 s\n> HEAD~1, w/ commit-graph: 6.06 s\n>   HEAD, w/ commit-graph: 0.06 s\n\nI wonder if we could make use of existing infrstructure in 't/perf/' to\nperform those benchmarks for us (perhaps augmented with large\nrepository, and only if requested -- similarly to how long tests are\nhandled).\n\n--\nJakub Narębski\n"},{"id":"361095","messageId":"4d0587d1-d204-2507-3ac4-20358ed70ad0@gmail.com","threadId":"49229","inReplyTo":"865zxvior3.fsf@gmail.com","subject":"Re: [PATCH v4 3/7] test-reach: add rev-list tests","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-21T15:28:20Z","receivedAt":"2018-10-21T15:33:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/21/2018 6:21 AM, Jakub Narebski wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n>> From: Derrick Stolee <dstolee@microsoft.com>\n>>\n>> The rev-list command is critical to Git's functionality. Ensure it\n>> works in the three commit-graph environments constructed in\n>> t6600-test-reach.sh. Here are a few important types of rev-list\n>> operations:\n>>\n>> * Basic: git rev-list --topo-order HEAD\n>> * Range: git rev-list --topo-order compare..HEAD\n>> * Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n>> * Symmetric Difference: git rev-list --topo-order compare...HEAD\n> Could you remind us here which of those operations will be using\n> generation numbers after this patch series?\n\nFor this series, we are focused only on the --topo-order with a single \nstart position. The versions that use a compare branch still use the old \nlogic. In the future, I would like to use the new logic for these other \nmodes.\n\n>>   \n>> +test_expect_success 'rev-list: basic topo-order' '\n>> +\tgit rev-parse \\\n>> +\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n>> +\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n>> +\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n>> +\t\tcommit-6-3 commit-5-3 commit-4-3 commit-3-3 commit-2-3 commit-1-3 \\\n>> +\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n>> +\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n>> +\t>expect &&\n>> +\trun_three_modes git rev-list --topo-order commit-6-6\n>> +'\n> I wonder if this test could be make easier to write and less error\n> prone, e.g. creating it from ASCII-art graphics.\n>\n> But it is good enough.\n\nI did lay out the branch names in a grid layout similar to the \ncommit-graph layout. It's easier to see the purposeful layout in the \ncomparison sections where some commits don't appear in the output.\n\nThanks,\n\n-Stolee\n\n"},{"id":"361096","messageId":"86h8hfguqc.fsf@gmail.com","threadId":"49229","inReplyTo":"cd9eef96885a811196ab0b851a98de4455b950ab.1539729393.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-21T15:55:23Z","receivedAt":"2018-10-21T15:55:29Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> When running 'git rev-list --topo-order' and its kin, the topo_order\n> setting in struct rev_info implies the limited setting. This means\n> that the following things happen during prepare_revision_walk():\n>\n> * revs->limited implies we run limit_list() to walk the entire\n>   reachable set. There are some short-cuts here, such as if we\n>   perform a range query like 'git rev-list COMPARE..HEAD' and we\n>   can stop limit_list() when all queued commits are uninteresting.\n\nAnd if revs->topo_order is set, then (with current implementation) we\nneed limit_list() to run to generate commit_list with commits to be\ntopologically sorted, which is done by setting revs->limited.\n\nIn short, with current code revs->topo_order implies revs->limited.\n\n>\n> * revs->topo_order implies we run sort_in_topological_order(). See\n>   the implementation of that method in commit.c. It implies that\n>   the full set of commits to order is in the given commit_list.\n\nSo the current code uses \"generate list of commits, then sort it\"\napproach...\n\n>\n> These two methods imply that a 'git rev-list --topo-order HEAD'\n> command must walk the entire reachable set of commits _twice_ before\n> returning a single result.\n>\n> If we have a commit-graph file with generation numbers computed, then\n> there is a better way.\n\n...instead of generating commits in topological order as you go.\n\n>                        This patch introduces some necessary logic\n> redirection when we are in this situation.\n\nO.K., this should make main commit smaller.  All right.\n\n> In v2.18.0, the commit-graph file contains zero-valued bytes in the\n> positions where the generation number is stored in v2.19.0 and later.\n> Thus, we use generation_numbers_enabled() to check if the commit-graph\n> is available and has non-zero generation numbers.\n>\n> When setting revs->limited only because revs->topo_order is true,\n> only do so if generation numbers are not available. There is no\n> reason to use the new logic as it will behave similarly when all\n> generation numbers are INFINITY or ZERO.\n\nO.K. we will be using new algorithm only when there actually are some\ngeneration numbers.\n\n\n> In prepare_revision_walk(), if we have revs->topo_order but not\n> revs->limited, then we trigger the new logic. It breaks the logic\n> into three pieces, to fit with the existing framework:\n\nSo if revs->limited is set (but not because revs->topo_order is set),\nwhich means A..B queries, we will be still using the old algorithm.\nAll right, though I wonder if it could be improved in the future\n(perhaps with the help of other graph labelling / indices than\ngeneration numbers, maybe a positive-cut index).\n\nDo you have an idea why there is no improvement with the new code in\nthis case?\n\n> 1. init_topo_walk() fills a new struct topo_walk_info in the rev_info\n>    struct. We use the presence of this struct as a signal to use the\n>    new methods during our walk. In this patch, this method simply\n>    calls limit_list() and sort_in_topological_order(). In the future,\n>    this method will set up a new data structure to perform that logic\n>    in-line.\n>\n> 2. next_topo_commit() provides get_revision_1() with the next topo-\n>    ordered commit in the list. Currently, this simply pops the commit\n>    from revs->commits.\n>\n> 3. expand_topo_walk() provides get_revision_1() with a way to signal\n>    walking beyond the latest commit. Currently, this calls\n>    add_parents_to_list() exactly like the old logic.\n\nSo all three new functions should perform exactly like the old logic,\nisn't it?\n\n> While this commit presents method redirection for performing the\n> exact same logic as before, it allows the next commit to focus only\n> on the new logic.\n\nAll right, it's logical.\n\n>\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  revision.c | 42 ++++++++++++++++++++++++++++++++++++++----\n>  revision.h |  4 ++++\n>  2 files changed, 42 insertions(+), 4 deletions(-)\n>\n> diff --git a/revision.c b/revision.c\n> index e18bd530e4..2dcde8a8ac 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -25,6 +25,7 @@\n>  #include \"worktree.h\"\n>  #include \"argv-array.h\"\n>  #include \"commit-reach.h\"\n> +#include \"commit-graph.h\"\n>  \n>  volatile show_early_output_fn_t show_early_output;\n>  \n> @@ -2454,7 +2455,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n>  \tif (revs->diffopt.objfind)\n>  \t\trevs->simplify_history = 0;\n>  \n> -\tif (revs->topo_order)\n> +\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n>  \t\trevs->limited = 1;\n\nAll right, with --topo-order and existing generation numbers don't force\nthe revs->limited code (i.e. explicit not wrapped use of limit_list()).\n\nSo with --topo-order and A..B, we have revs->limited set, with\n--topo-order and no generation numbers we have revs->limited set.\n\n>  \n>  \tif (revs->prune_data.nr) {\n> @@ -2892,6 +2893,33 @@ static int mark_uninteresting(const struct object_id *oid,\n>  \treturn 0;\n>  }\n>  \n> +struct topo_walk_info {};\n\nNice trick with using NULL-ness of the pointer to the currently empty\nstruct as a boolean flag denoting whether to use new generation number\nusing algorithm for topological sorting.\n\n> +\n> +static void init_topo_walk(struct rev_info *revs)\n> +{\n> +\tstruct topo_walk_info *info;\n\nI guess this helper variables is here for next revisions, as we could\nhave made without it...\n\n> +\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n> +\tinfo = revs->topo_walk_info;\n> +\tmemset(info, 0, sizeof(struct topo_walk_info));\n\n...by using\n\n  +\tmemset(revs->topo_walk_info, 0, sizeof(struct topo_walk_info));\n\n> +\n> +\tlimit_list(revs);\n> +\tsort_in_topological_order(&revs->commits, revs->sort_order);\n\nThis is not exactly identical to the old code, which has\n\n\tif (limit_list(revs) < 0)\n\t\treturn -1;\n\tif (revs->topo_order)\n\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n\nWe know that init_topo_walk() would be invoked, as the name implies,\nonly when revs->topo_order is set, but do we know that limit_list()\nwould not return an error?\n\n> +}\n> +\n> +static struct commit *next_topo_commit(struct rev_info *revs)\n> +{\n> +\treturn pop_commit(&revs->commits);\n> +}\n\nAll right, identical to the old code.\n\n> +\n> +static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n> +{\n> +\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n> +\t\tif (!revs->ignore_missing_links)\n> +\t\t\tdie(\"Failed to traverse parents of commit %s\",\n> +\t\t\t    oid_to_hex(&commit->object.oid));\n> +\t}\n> +}\n\nAll right, identical to the old code.\n\nWhile at it, should this message be marked up for translation, or is it\nsomething so low-level (and rare) that should be kept untranslated?  But\nthis would be better left for separate commit series, to not entangle\nthis one with spurious changes.\n\n> +\n>  int prepare_revision_walk(struct rev_info *revs)\n>  {\n>  \tint i;\n> @@ -2928,11 +2956,13 @@ int prepare_revision_walk(struct rev_info *revs)\n>  \t\tcommit_list_sort_by_date(&revs->commits);\n>  \tif (revs->no_walk)\n>  \t\treturn 0;\n> -\tif (revs->limited)\n> +\tif (revs->limited) {\n>  \t\tif (limit_list(revs) < 0)\n>  \t\t\treturn -1;\n> -\tif (revs->topo_order)\n> -\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n> +\t\tif (revs->topo_order)\n> +\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n> +\t} else if (revs->topo_order)\n> +\t\tinit_topo_walk(revs);\n\nPreviously when revs->topo_order was set, Git called\nsort_in_topological_order(), because revs->limited got always set to\ntruthy value if revs->topo_order was true.\n\nNow running sort_in_topological_order() is done only if revs->limited is\nset (because of A..B); if it is not, init_topo_walk() is called.\n\nAll right, identical to the old code, up to checking the return value of\nlimit_list(), see previous comments.\n\n>  \tif (revs->line_level_traverse)\n>  \t\tline_log_filter(revs);\n>  \tif (revs->simplify_merges)\n> @@ -3257,6 +3287,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n>  \n>  \t\tif (revs->reflog_info)\n>  \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n> +\t\telse if (revs->topo_walk_info)\n> +\t\t\tcommit = next_topo_commit(revs);\n>  \t\telse\n>  \t\t\tcommit = pop_commit(&revs->commits);\n\nAll right, identical to the old code.\n\n> @@ -3278,6 +3310,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n>  \n>  \t\t\tif (revs->reflog_info)\n>  \t\t\t\ttry_to_simplify_commit(revs, commit);\n> +\t\t\telse if (revs->topo_walk_info)\n> +\t\t\t\texpand_topo_walk(revs, commit);\n>  \t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n>  \t\t\t\tif (!revs->ignore_missing_links)\n>  \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\n\nAll right, identical to the old code.\n\n> diff --git a/revision.h b/revision.h\n> index 2b30ac270d..fd4154ff75 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -56,6 +56,8 @@ struct rev_cmdline_info {\n>  #define REVISION_WALK_NO_WALK_SORTED 1\n>  #define REVISION_WALK_NO_WALK_UNSORTED 2\n>  \n> +struct topo_walk_info;\n> +\n>  struct rev_info {\n>  \t/* Starting list */\n>  \tstruct commit_list *commits;\n> @@ -245,6 +247,8 @@ struct rev_info {\n>  \tconst char *break_bar;\n>  \n>  \tstruct revision_sources *sources;\n> +\n> +\tstruct topo_walk_info *topo_walk_info;\n>  };\n>  \n>  int ref_excluded(struct string_list *, const char *path);\n"},{"id":"361102","messageId":"865zxvgftq.fsf@gmail.com","threadId":"49229","inReplyTo":"f3e291665dae94b7347621ec78721f7e0dcc86d8.1539729393.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 5/7] commit/revisions: bookkeeping before refactoring","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-21T21:17:21Z","receivedAt":"2018-10-21T21:17:27Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> There are a few things that need to move around a little before\n> making a big refactoring in the topo-order logic:\n>\n> 1. We need access to record_author_date() and\n>    compare_commits_by_author_date() in revision.c. These are used\n>    currently by sort_in_topological_order() in commit.c.\n>\n> 2. Moving these methods to commit.h requires adding the author_slab\n>    definition to commit.h.\n\nThose two changes are connected, and must be kept together.\n\n> 3. The add_parents_to_list() method in revision.c performs logic\n>    around the UNINTERESTING flag and other special cases depending\n>    on the struct rev_info. Allow this method to ignore a NULL 'list'\n>    parameter, as we will not be populating the list for our walk.\n>    Also rename the method to the slightly more generic name\n>    process_parents() to make clear that this method does more than\n>    add to a list (and no list is required anymore).\n\nBut as far as I can understand, this change is independent, and it could\nbe put into a separate commmit.\n\nThe change of function name to process_parents() and allowing for 'list'\nparameter to be NULL are related, though.\n\n>\n> Helped-by: Jeff King <peff@peff.net>\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n\nNo need to split, unless there would be v5 anyway, in my opinion.\n\n> ---\n>  commit.c   | 11 +++++------\n>  commit.h   |  8 ++++++++\n>  revision.c | 18 ++++++++++--------\n>  3 files changed, 23 insertions(+), 14 deletions(-)\n>\n> diff --git a/commit.c b/commit.c\n> index d0f199e122..861a485e93 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -655,11 +655,10 @@ struct commit *pop_commit(struct commit_list **stack)\n>  /* count number of children that have not been emitted */\n>  define_commit_slab(indegree_slab, int);\n>  \n> -/* record author-date for each commit object */\n> -define_commit_slab(author_date_slab, timestamp_t);\n> +implement_shared_commit_slab(author_date_slab, timestamp_t);\n\nI see that the comment got moved to the site with\ndefine_shared_commit_slab(), i.e. to commit.h, instead of duplicting\nit.  All right.\n\nSidenote: Ugh, small_caps preprocessor macros [trickery].\n\n>  \n> -static void record_author_date(struct author_date_slab *author_date,\n> -\t\t\t       struct commit *commit)\n> +void record_author_date(struct author_date_slab *author_date,\n> +\t\t\tstruct commit *commit)\n>  {\n>  \tconst char *buffer = get_commit_buffer(commit, NULL);\n>  \tstruct ident_split ident;\n> @@ -684,8 +683,8 @@ fail_exit:\n>  \tunuse_commit_buffer(commit, buffer);\n>  }\n>  \n> -static int compare_commits_by_author_date(const void *a_, const void *b_,\n> -\t\t\t\t\t  void *cb_data)\n> +int compare_commits_by_author_date(const void *a_, const void *b_,\n> +\t\t\t\t   void *cb_data)\n\nAll right, this is straighforward changing record_author_date() and\ncompare_commits_by_author_date() from static (file-local) functions to\nexported functions.\n\n>  {\n>  \tconst struct commit *a = a_, *b = b_;\n>  \tstruct author_date_slab *author_date = cb_data;\n> diff --git a/commit.h b/commit.h\n> index 2b1a734388..977d397356 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -8,6 +8,7 @@\n>  #include \"gpg-interface.h\"\n>  #include \"string-list.h\"\n>  #include \"pretty.h\"\n> +#include \"commit-slab.h\"\n>  \n>  #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n>  #define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n> @@ -328,6 +329,13 @@ extern int remove_signature(struct strbuf *buf);\n>   */\n>  extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n>  \n> +/* record author-date for each commit object */\n> +define_shared_commit_slab(author_date_slab, timestamp_t);\n\nAll right, this is needed for record_author_date() function, which is\nnow exported.\n\n> +\n> +void record_author_date(struct author_date_slab *author_date,\n> +\t\t\tstruct commit *commit);\n> +\n> +int compare_commits_by_author_date(const void *a_, const void *b_, void *unused);\n\nO.K., this is simply exporting previously static functions.\n\n>  int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n>  int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n>  \n> diff --git a/revision.c b/revision.c\n> index 2dcde8a8ac..36458265a0 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -768,8 +768,8 @@ static void commit_list_insert_by_date_cached(struct commit *p, struct commit_li\n>  \t\t*cache = new_entry;\n>  }\n>  \n> -static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n> -\t\t    struct commit_list **list, struct commit_list **cache_ptr)\n> +static int process_parents(struct rev_info *revs, struct commit *commit,\n> +\t\t\t   struct commit_list **list, struct commit_list **cache_ptr)\n\nAll right, straighforward rename.\n\n>  {\n>  \tstruct commit_list *parent = commit->parents;\n>  \tunsigned left_flag;\n> @@ -808,7 +808,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n>  \t\t\tif (p->object.flags & SEEN)\n>  \t\t\t\tcontinue;\n>  \t\t\tp->object.flags |= SEEN;\n> -\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n> +\t\t\tif (list)\n> +\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n>  \t\t}\n>  \t\treturn 0;\n>  \t}\n> @@ -847,7 +848,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n>  \t\tp->object.flags |= left_flag;\n>  \t\tif (!(p->object.flags & SEEN)) {\n>  \t\t\tp->object.flags |= SEEN;\n> -\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n> +\t\t\tif (list)\n> +\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n\nAll right, both of those is about allowing 'list' parameter to be NULL,\nand invoking commit_list_insert_by_date_cached() only if it's not NULL.\n\n>  \t\t}\n>  \t\tif (revs->first_parent_only)\n>  \t\t\tbreak;\n> @@ -1091,7 +1093,7 @@ static int limit_list(struct rev_info *revs)\n>  \n>  \t\tif (revs->max_age != -1 && (commit->date < revs->max_age))\n>  \t\t\tobj->flags |= UNINTERESTING;\n> -\t\tif (add_parents_to_list(revs, commit, &list, NULL) < 0)\n> +\t\tif (process_parents(revs, commit, &list, NULL) < 0)\n>  \t\t\treturn -1;\n>  \t\tif (obj->flags & UNINTERESTING) {\n>  \t\t\tmark_parents_uninteresting(commit);\n> @@ -2913,7 +2915,7 @@ static struct commit *next_topo_commit(struct rev_info *revs)\n>  \n>  static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n>  {\n> -\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n> +\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n>  \t\tif (!revs->ignore_missing_links)\n>  \t\t\tdie(\"Failed to traverse parents of commit %s\",\n>  \t\t\t    oid_to_hex(&commit->object.oid));\n> @@ -2979,7 +2981,7 @@ static enum rewrite_result rewrite_one(struct rev_info *revs, struct commit **pp\n>  \tfor (;;) {\n>  \t\tstruct commit *p = *pp;\n>  \t\tif (!revs->limited)\n> -\t\t\tif (add_parents_to_list(revs, p, &revs->commits, &cache) < 0)\n> +\t\t\tif (process_parents(revs, p, &revs->commits, &cache) < 0)\n>  \t\t\t\treturn rewrite_one_error;\n>  \t\tif (p->object.flags & UNINTERESTING)\n>  \t\t\treturn rewrite_one_ok;\n> @@ -3312,7 +3314,7 @@ static struct commit *get_revision_1(struct rev_info *revs)\n>  \t\t\t\ttry_to_simplify_commit(revs, commit);\n>  \t\t\telse if (revs->topo_walk_info)\n>  \t\t\t\texpand_topo_walk(revs, commit);\n> -\t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n> +\t\t\telse if (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n>  \t\t\t\tif (!revs->ignore_missing_links)\n>  \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\n>  \t\t\t\t\t\toid_to_hex(&commit->object.oid));\n\nAll those is just changing the calling convention due to function\nrename.\n\n(I wonder if such simple refactoring could have been done via Coccinelle\npatch).\n\n\nAnyway, looks good to me.\n--\nJakub Narębski\n"},{"id":"361104","messageId":"xmqqa7n6zswl.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"86h8hfguqc.fsf@gmail.com","subject":"Re: [PATCH v4 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-22T01:12:10Z","receivedAt":"2018-10-22T01:12:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> So if revs->limited is set (but not because revs->topo_order is set),\n> which means A..B queries, we will be still using the old algorithm.\n> All right, though I wonder if it could be improved in the future\n> (perhaps with the help of other graph labelling / indices than\n> generation numbers, maybe a positive-cut index).\n>\n> Do you have an idea why there is no improvement with the new code in\n> this case?\n\nI didn't get the impression that it would not be possible to improve\nthe \"--topo A..B\" case by using generation numbers from this series.\nIsn't it just because the necessary code has not been written yet?\nIn addition to what is needed for \"--topo P1 P2 P3...\" (all\npositive), limited walk needs to notice the bottom boundary and stop\ntraversal.  Having generation numbers would make it slightly easier\nthan without, as you know that a positive commit you have will not\nbe marked UNINTERESTING due to a negative commit whose ancestors\nhave not been explored, as long as that negative commit has a higher\ngeneration number.  But you still need to adjust the traversal logic\nto properly terminate upon hitting UNINTERESTING one, and also\npropagate the bit down the history, which is not needed at all if\nyou only want to support the \"positive only\" case.\n\n"},{"id":"361107","messageId":"bb91d5bb-4220-97b9-dd78-5aed39033e58@gmail.com","threadId":"49229","inReplyTo":"xmqqa7n6zswl.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-22T01:51:10Z","receivedAt":"2018-10-22T01:51:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/21/2018 9:12 PM, Junio C Hamano wrote:\n> Jakub Narebski <jnareb@gmail.com> writes:\n>\n>> So if revs->limited is set (but not because revs->topo_order is set),\n>> which means A..B queries, we will be still using the old algorithm.\n>> All right, though I wonder if it could be improved in the future\n>> (perhaps with the help of other graph labelling / indices than\n>> generation numbers, maybe a positive-cut index).\n>>\n>> Do you have an idea why there is no improvement with the new code in\n>> this case?\n> I didn't get the impression that it would not be possible to improve\n> the \"--topo A..B\" case by using generation numbers from this series.\n> Isn't it just because the necessary code has not been written yet?\n> In addition to what is needed for \"--topo P1 P2 P3...\" (all\n> positive), limited walk needs to notice the bottom boundary and stop\n> traversal.  Having generation numbers would make it slightly easier\n> than without, as you know that a positive commit you have will not\n> be marked UNINTERESTING due to a negative commit whose ancestors\n> have not been explored, as long as that negative commit has a higher\n> generation number.  But you still need to adjust the traversal logic\n> to properly terminate upon hitting UNINTERESTING one, and also\n> propagate the bit down the history, which is not needed at all if\n> you only want to support the \"positive only\" case.\n\nActually, the code has been written, but the problem is the same as the \nperformance issue when I made paint_down_to_common() use generation \nnumbers: the algorithm for deciding what is in the set \"reachable from A \nbut not reachable from B\" uses commit-date order as a heuristic to avoid \nwalking the entire graph. Yes, the revision parameters specify \"limited\" \nin order to call \"limit_list()\", but it uses the same algorithm to \ndetermine the reachable set difference.\n\nYou can test this yourself! Run the following two commands in the Git \nrepository using v2.19.1:\n\n     time git log --topo-order -10 master >/dev/null\n\n     time git log --topo-order -10 maint..master >/dev/null\n\nI get 0.39s for the first call and 0.01s for the second. (Note: I \nspecified \"-10\" to ensure we are only writing 10 commits and the output \nsize does not factor into the time.) This is because the first walks the \nentire history, while the second uses the heuristic walk to identify a \nmuch smaller subgraph that the topo-order algorithm uses.\n\nJust as before, by using this algorithm for the B..A case, we are adding \nan extra restriction on the algorithm: always be correct. This results \nin us walking a larger set (everything reachable from B or A with \ngeneration number at least the smallest generation of a commit reachable \nfrom only one).\n\nI believe this can be handled by using a smarter generation number (one \nthat relies on commit date as a heuristic, but still have enough \ninformation to guarantee topological relationships), and I've already \nstarted testing a few of these directions. It is possible now that we \nhave concrete graph algorithms to use on real repositories. I hope to \nshare a report on my findings in a couple weeks. I'll include how using \nthis algorithm compares to the old algorithm in the B..A case.\n\nThanks,\n\n-Stolee\n\n"},{"id":"361108","messageId":"20181022015528.141238-1-dstolee@microsoft.com","threadId":"49229","inReplyTo":"bb91d5bb-4220-97b9-dd78-5aed39033e58@gmail.com","subject":"[RFC PATCH] revision.c: use new algorithm in A..B case","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-22T01:55:28Z","receivedAt":"2018-10-22T01:55:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n\nI just wanted to mention that in order to use the new logic for 'git log\n--topo-order A..B', we just need the following patch. It is an extra\ntime that sets 'revs->limited' to 1, triggering the old logic.\n\nYou can use this for comparison purposes, but I'm not ready to do this\nuntil more performance testing is ready in this case. Since these\ncomparison commands are already pretty fast when the diff is small,\nthere is less urgency to improve performance here.\n\nThanks,\n-Stolee\n\n revision.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 472f3994e3..8e5656f7b4 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -278,10 +278,8 @@ static struct commit *handle_commit(struct rev_info *revs,\n \n \t\tif (parse_commit(commit) < 0)\n \t\t\tdie(\"unable to parse commit %s\", name);\n-\t\tif (flags & UNINTERESTING) {\n+\t\tif (flags & UNINTERESTING)\n \t\t\tmark_parents_uninteresting(commit);\n-\t\t\trevs->limited = 1;\n-\t\t}\n \t\tif (revs->sources) {\n \t\t\tchar **slot = revision_sources_at(revs->sources, commit);\n \n-- \n2.19.1\n\n"},{"id":"361159","messageId":"86sh0y5cgk.fsf@gmail.com","threadId":"49229","inReplyTo":"aa0bb2221d437f595f6d0c16b125072fde289c15.1539729393.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 6/7] revision.c: generation-based topo-order algorithm","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-22T13:37:47Z","receivedAt":"2018-10-22T13:37:54Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> The current --topo-order algorithm requires walking all\n> reachable commits up front, topo-sorting them, all before\n> outputting the first value. This patch introduces a new\n> algorithm which uses stored generation numbers to\n> incrementally walk in topo-order, outputting commits as\n> we go. This can dramatically reduce the computation time\n> to write a fixed number of commits, such as when limiting\n> with \"-n <N>\" or filling the first page of a pager.\n>\n> When running a command like 'git rev-list --topo-order HEAD',\n> Git performed the following steps:\n>\n> 1. Run limit_list(), which parses all reachable commits,\n>    adds them to a linked list, and distributes UNINTERESTING\n>    flags. If all unprocessed commits are UNINTERESTING, then\n>    it may terminate without walking all reachable commits.\n>    This does not occur if we do not specify UNINTERESTING\n>    commits.\n>\n> 2. Run sort_in_topological_order(), which is an implementation\n>    of Kahn's algorithm. It first iterates through the entire\n>    set of important commits and computes the in-degree of each\n>    (plus one, as we use 'zero' as a special value here). Then,\n>    we walk the commits in priority order, adding them to the\n>    priority queue if and only if their in-degree is one. As\n>    we remove commits from this priority queue, we decrement the\n>    in-degree of their parents.\n\nBecause in-degree has very specific defined meaning of number of\nchildren, i.e. the number of _incoming_ edges, I would say \"if and only\nif their in-degree-plus-one is one\".  It is more exact, even if it looks\na bit funny.\n\n> 3. While we are peeling commits for output, get_revision_1()\n>    uses pop_commit on the full list of commits computed by\n>    sort_in_topological_order().\n\nAll right, so those are separate steps (separate walks): prepare and\nparse commits, topologically sort list of commits from previous step,\noutput sorted list of commits from previous step.\n\n> In the new algorithm, these three steps correspond to three\n> different commit walks. We run these walks simultaneously,\n> and advance each only as far as necessary to satisfy the\n> requirements of the 'higher order' walk.\n\nWhat does 'higher order' walk means: steps 3, 2, 1, in this order,\ni.e. output being the highest order, or something different?\n\nSidenote: the new algorithm looks a bit like Unix pipeline, where each\nstep of pipeline does not output much more than next step needs /\nrequests.\n\n>                                          We know when we can\n> pause each walk by using generation numbers from the commit-\n> graph feature.\n\nDo I understand it correctly that this is mainly used in Kahn's\nalgorithm to find out through the negative-cut index of generation\nnumber which commits in the to-be-sorted list cannot have an in-degree\nof zero (or otherise cannot be next commit to be shown in output)?\n\n> Recall that the generation number of a commit satisfies:\n>\n> * If the commit has at least one parent, then the generation\n>   number is one more than the maximum generation number among\n>   its parents.\n>\n> * If the commit has no parent, then the generation number is one.\n>\n> There are two special generation numbers:\n>\n> * GENERATION_NUMBER_INFINITY: this value is 0xffffffff and\n>   indicates that the commit is not stored in the commit-graph and\n>   the generation number was not previously calculated.\n>\n> * GENERATION_NUMBER_ZERO: this value (0) is a special indicator\n>   to say that the commit-graph was generated by a version of Git\n>   that does not compute generation numbers (such as v2.18.0).\n>\n> Since we use generation_numbers_enabled() before using the new\n> algorithm, we do not need to worry about GENERATION_NUMBER_ZERO.\n> However, the existence of GENERATION_NUMBER_INFINITY implies the\n> following weaker statement than the usual we expect from\n> generation numbers:\n>\n>     If A and B are commits with generation numbers gen(A) and\n>     gen(B) and gen(A) < gen(B), then A cannot reach B.\n>\n> Thus, we will walk in each of our stages until the \"maximum\n> unexpanded generation number\" is strictly lower than the\n> generation number of a commit we are about to use.\n\nAnd this \"maximum unexpanded generation number\" must be greater or equal\nto 1, thanks to assuming generation_numbers_enabled().\n\n\nLet's start by writing down the original version of the Kahn's algorith\n(which is not the only way to calculate topological ordering; another\nmethod is to use depth-first searches).\n\n  L ← Empty list that will contain the sorted elements\n  S ← Set of all nodes with no incoming edge\n  while S is non-empty do\n      remove a node n from S\n      add n to tail of L\n      for each node m with an edge e from n to m do\n          remove edge e from the graph\n          if m has no other incoming edges then\n              insert m into S\n  if graph has edges then\n      return error   _(graph has at least one cycle)_\n  else \n      return L       _(a topologically sorted order)_\n\nIn the case of Git, we display only commits reachable from the starting\ncommits, so only those starting commits can have no incoming edge, by\nthe definition of the reachable commit (note that some starting commits\ncan be reachable from other starting commits).\n\nNote that in Git by construction we cannot have cycles in the objects\ngraph, and that 'remove edge e [= n -> m] from the graph' simply means\ndecreasing the [effective] in-degree of node m.\n\n> The walks are as follows:\n>\n> 1. EXPLORE: using the explore_queue priority queue (ordered by\n>    maximizing the generation number), parse each reachable\n>    commit until all commits in the queue have generation\n>    number strictly lower than needed. During this walk, update\n>    the UNINTERESTING flags as necessary.\n\nAll right, that looks sensible.  Parse commits and update the\nUNINTERESTING flags only up to what might be needed.\n\nThough I would add for each walk what are post-conditions, i.e. what\nrequirements list of returned commits does fullfill.  In the case of the\nEXPLORE walk it would be that commits are in the \"reachable from start\ncommts\" set, parsed and not UNINTERESTING.  And that there are all such\ncommits there with generation number greater or equal if needed (and\ntheir parents).\n\n\nWouldn't this though make the output always start at the commit with\nmaximal generation number (such commit or commits would need to have an\nin-degree of zero, i.e. no incoming edges), instead of whatever order is\nrequested (if date order contradicts generation number order) or in the\ncommand line arguments order?\n\n> 2. INDEGREE: using the indegree_queue priority queue (ordered\n>    by maximizing the generation number), add one to the in-\n>    degree of each parent for each commit that is walked. Since\n>    we walk in order of decreasing generation number, we know\n>    that discovering an in-degree value of 0 means the value for\n>    that commit was not initialized, so should be initialized to\n>    two. (Recall that in-degree value \"1\" is what we use to say a\n>    commit is ready for output.)\n\nThe post-condition is that all returned commits have their in-degree\nplus one calculated.\n\nMixing actual in-degree (number of incoming edges, zero means no\nincoming edge and candidate for the next commit in topological order),\nand details of implementation (using value of zero for uninitialized,\nand thus actually storing in-degree plus one) makes this description a\nbit hard to follow.\n\nNote that the additional complication is that if commits have generation\nnumber INFINITY, then we cannot say anything about reachability among\ncommits with this special generation number.  That means that until we\nprocess all commits with generation number INFINITY, we don't know which\nones have no incoming edges (a real in-degree of zero).  If they are not\nINFINITY, the stronger version of the reachability condition for\ngeneration number holds, and thus we know that if we pop the commit and\nit has uninitialized in-degree and generation number not INFINITY, then\nit has no-incoming edges (in-degree of zero).\n\nThat does not matter much, but for the fact that before outputting list\nof commits / returning from the function we need to ensure that all out\ncommits have defined in-degree value.  All that have in-degree undefined\nwhen indegree_queue is empty, because of reachability and generation\nnumbers constraints, actually have an in-degree of zero (no incoming\nedges).\n\n>                                  As we iterate the parents of a\n>    commit during this walk, ensure the EXPLORE walk has walked\n>    beyond their generation numbers.\n\nAll right. looks sensible from the point of view of trying to do\nstreaming of sorted commits.\n\n>\n> 3. TOPO: using the topo_queue priority queue (ordered based on\n>    the sort_order given, which could be commit-date, author-\n>    date, or typical topo-order which treats the queue as a LIFO\n>    stack), remove a commit from the queue and decrement the\n>    in-degree of each parent. If a parent has an in-degree of\n>    one, then we add it to the topo_queue. Before we decrement\n>    the in-degree, however, ensure the INDEGREE walk has walked\n>    beyond that generation number.\n\nThis description missed an important constraint, namely that all commits\nin the topo_order queue have real in-degree of zero, i.e. no incoming\nedges.  The topo_order queue is set S in the Kahn's algorithm.\n\nAlso, we need to know how to populate the topo_queue at start with at\nleast one commit.  How it is initially populated?  That is very\nimportant question.\n\n>\n> The implementations of these walks are in the following methods:\n>\n> * explore_walk_step and explore_to_depth\n> * indegree_walk_step and compute_indegrees_to_depth\n> * next_topo_commit and expand_topo_walk\n\nAll right, one one hand: good calling convention.  On the other hand:\nwhy the difference in naming?\n\n>\n> These methods have some patterns that may seem strange at first,\n> but they are probably carry-overs from their equivalents in\n> limit_list and sort_in_topological_order.\n>\n> One thing that is missing from this implementation is a proper\n> way to stop walking when the entire queue is UNINTERESTING, so\n> this implementation is not enabled by comparisions, such as in\n> 'git rev-list --topo-order A..B'. This can be updated in the\n> future.\n\nAll right, lets start with easier step.\n\n>\n> In my local testing, I used the following Git commands on the\n> Linux repository in three modes: HEAD~1 with no commit-graph,\n> HEAD~1 with a commit-graph, and HEAD with a commit-graph. This\n> allows comparing the benefits we get from parsing commits from\n> the commit-graph and then again the benefits we get by\n> restricting the set of commits we walk.\n>\n> Test: git rev-list --topo-order -100 HEAD\n> HEAD~1, no commit-graph: 6.80 s\n> HEAD~1, w/ commit-graph: 0.77 s\n>   HEAD, w/ commit-graph: 0.02 s\n>\n> Test: git rev-list --topo-order -100 HEAD -- tools\n> HEAD~1, no commit-graph: 9.63 s\n> HEAD~1, w/ commit-graph: 6.06 s\n>   HEAD, w/ commit-graph: 0.06 s\n>\n> This speedup is due to a few things. First, the new generation-\n> number-enabled algorithm walks commits on order of the number of\n> results output (subject to some branching structure expectations).\n> Since we limit to 100 results, we are running a query similar to\n> filling a single page of results. Second, when specifying a path,\n> we must parse the root tree object for each commit we walk. The\n> previous benefits from the commit-graph are entirely from reading\n> the commit-graph instead of parsing commits. Since we need to\n> parse trees for the same number of commits as before, we slow\n> down significantly from the non-path-based query.\n>\n> For the test above, I specifically selected a path that is changed\n> frequently, including by merge commits. A less-frequently-changed\n> path (such as 'README') has similar end-to-end time since we need\n> to walk the same number of commits (before determining we do not\n> have 100 hits). However, get the benefit that the output is\n> presented to the user as it is discovered, much the same as a\n> normal 'git log' command (no '--topo-order'). This is an improved\n> user experience, even if the command has the same runtime.\n\nFirst, do I understand it correctly that in first case the gains from\nnew algorithms are so slim because with commit-graph file and no path\nlimiting we don't hit repository anyway; we walk less commits, but\nreading commit data from commit-graph file is fast/\n\nSecond, I wonder if there is some easy way to perform automatic latency\ntests, i.e. how fast does Git show the first page of output...\n\n> Helped-by: Jeff King <peff@peff.net>\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  object.h   |   4 +-\n>  revision.c | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n>  revision.h |   2 +\n>  3 files changed, 197 insertions(+), 8 deletions(-)\n\nDaunting change to review.\n\n> diff --git a/object.h b/object.h\n> index 0feb90ae61..796792cb32 100644\n> --- a/object.h\n> +++ b/object.h\n> @@ -59,7 +59,7 @@ struct object_array {\n>  \n>  /*\n>   * object flag allocation:\n> - * revision.h:               0---------10                              2526\n> + * revision.h:               0---------10                              25----28\n>   * fetch-pack.c:             01\n>   * negotiator/default.c:       2--5\n>   * walker.c:                 0-2\n> @@ -78,7 +78,7 @@ struct object_array {\n>   * builtin/show-branch.c:    0-------------------------------------------26\n>   * builtin/unpack-objects.c:                                 2021\n>   */\n> -#define FLAG_BITS  27\n> +#define FLAG_BITS  29\n\nWhat are those two additional object flags needed for revision.h /\nrevision.c after this change?\n\nAh, those are TOPO_WALK_EXPLORED and TOPO_WALK_INDEGREE.\n\n>  \n>  /*\n>   * The object type is stored in 3 bits.\n> diff --git a/revision.c b/revision.c\n> index 36458265a0..472f3994e3 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -26,6 +26,7 @@\n>  #include \"argv-array.h\"\n>  #include \"commit-reach.h\"\n>  #include \"commit-graph.h\"\n> +#include \"prio-queue.h\"\n>  \n>  volatile show_early_output_fn_t show_early_output;\n>  \n> @@ -2895,30 +2896,216 @@ static int mark_uninteresting(const struct object_id *oid,\n>  \treturn 0;\n>  }\n>  \n> -struct topo_walk_info {};\n> +define_commit_slab(indegree_slab, int);\n> +\n> +struct topo_walk_info {\n> +\tuint32_t min_generation;\n> +\tstruct prio_queue explore_queue;\n> +\tstruct prio_queue indegree_queue;\n> +\tstruct prio_queue topo_queue;\n> +\tstruct indegree_slab indegree;\n\nAll right.\n\n> +\tstruct author_date_slab author_date;\n\nWhy this slab is needed in topo_walk_info struct?\n\n> +};\n> +\n> +static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n> +{\n> +\tif (c->object.flags & flag)\n> +\t\treturn;\n> +\n> +\tc->object.flags |= flag;\n> +\tprio_queue_put(q, c);\n> +}\n\nThis is an independent change, though I see that it is quite specific\n(as opposed to quite generic prio_queue_peek() operation added earlier\nin this series), so it does not make much sense as standalone change.\n\nIt inserts commit into priority queue only if it didn't have flags set,\nand sets the flag (so we won't add it to the queue again, not without\nunsetting the flag), am I correct?\n\n> +\n> +static void explore_walk_step(struct rev_info *revs)\n> +{\n> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n> +\tstruct commit_list *p;\n> +\tstruct commit *c = prio_queue_get(&info->explore_queue);\n> +\n> +\tif (!c)\n> +\t\treturn;\n> +\n> +\tif (parse_commit_gently(c, 1) < 0)\n> +\t\treturn;\n\nAll right, all commits taken out of explore_queue are parsed.  This is\nused to ensure that all commits qith generation number greater than some\nset cutoff are parsed.\n\n> +\n> +\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n> +\t\trecord_author_date(&info->author_date, c);\n> +\n> +\tif (revs->max_age != -1 && (c->date < revs->max_age))\n> +\t\tc->object.flags |= UNINTERESTING;\n\nThese two conditionals looks a bit strange to me; they are hardcoded\nspecific cases of query.  But that might be just me...\n\n> +\n> +\tif (process_parents(revs, c, NULL, NULL) < 0)\n> +\t\treturn;\n\nI see that we are using process_parents(), formerly i.e. before patch\n5/7 add_parents_to_list(), with NULL 'list' parameter for the first\ntime.\n\n> +\n> +\tif (c->object.flags & UNINTERESTING)\n> +\t\tmark_parents_uninteresting(c);\n> +\n> +\tfor (p = c->parents; p; p = p->next)\n> +\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n\nDo we need to insert parents to the queue even if they were marked\nUNINTERESTING?\n\nI guess that we use test_flag_and_insert() instead of prio_queue_put()\nto avoid duplicate entries in the queue.  I think the queue is initially\npopulated with the starting commits, but those need not to be\nunreachable from each other, and walking down parents we can encounter\nstarting commit already in the queue.  Am I correct?\n\n> +}\n\nLet's compare this new function with the limit_list() used in the old\nalgorithm for --topo-order walk (and even now for A..B walks), or to be\nmore exact with the contents of the while loop.\n\n1. limit_list() doesn't have the check if the commit exists, and\n   does not use parse_commit_gently().  Why the difference, i.e. where\n   revs->commits gets parsed, and why explore_walk_step() cannot rely on\n   this?\n\n   I get that the goal is to not have parse commits if not needed, so it\n   is good that it is moved to explore_walk_step().\n\n2. limit_list() is also missing running record_author_date() when\n   sorting output by author date.  I guess that explore_walk_step()\n   needs this because commit-graph file does not include this\n   information.\n\n3. Handling of revs->max_age by marking commit as UNINTERESTING if\n   needed is the same in limit_list() and in explore_walk_step().\n\n4. limit_list() but not explore_walk_step() handles revs->min_age near\n   the end pf the loop by terminating the loop.  I guess for this case\n   we have revs->limited set, and we use old algorithm, isn't it?\n\n   Something to remember when adding A..B handling to new algorithm.\n\n5. add_parents_to_list() / process_parents() is nearly the same in\n   limit_list() and in explore_walk_step(), but for the fact that the\n   new function doesn't use 'list' parameter.\n\n6. Both limit_list() and explore_walk_step() use\n   mark_parents_uninteresting() on uninteresting commits.\n\n   However, limit_list() breaks out of the loop, and uses\n   interesting_cache with slop.  I guess that those two facts are\n   connected, right?\n\n7. Then explore_walk_step() inserts parents to the priority queue if\n   they are not present there already, with test_flag_and_insert(),\n   which rough equivalent in limit_list() would be using\n   commit_list_insert().\n\n8. limit_list() has also some code for show_early_output, which I guess\n   explore_walk_step() does not need to handle.\n\n> +\n> +static void explore_to_depth(struct rev_info *revs,\n> +\t\t\t     uint32_t gen)\n> +{\n> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n> +\tstruct commit *c;\n> +\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n> +\t       c->generation >= gen)\n\nI have originally thought that if we extract prio_queue_get() and\ntest_flag_and_insert() / prio_queue_put() out of explore_walk_step() and\nput it into this loop, i.e. into the calling function, we could avoid\ncode duplication between explore_walk_step() and limit_list()... but I\nguess that is impossible anyway.\n\n> +\t\texplore_walk_step(revs);\n> +}\n\nNice, tight, and easy to understand function.  Though perhaps 'gen'\ncould be called 'gen_cutoff' or 'min_gen', or 'min_gen_cufott'.\n\n> +\n> +static void indegree_walk_step(struct rev_info *revs)\n> +{\n> +\tstruct commit_list *p;\n> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n> +\tstruct commit *c = prio_queue_get(&info->indegree_queue);\n> +\n> +\tif (!c)\n> +\t\treturn;\n> +\n> +\tif (parse_commit_gently(c, 1) < 0)\n> +\t\treturn;\n\nAll right, we need to parse commit 'c' to have its generation number,\nand we need to do the same in explore_walk_step() because we walk\npossibly unparsed parents.\n\n> +\n> +\texplore_to_depth(revs, c->generation);\n\nIf we walk everything up to current commit depth, then we have walked\nall commits that can affect in-degree of current commit.  Good.\n\n> +\n> +\tif (parse_commit_gently(c, 1) < 0)\n> +\t\treturn;\n\nWhy do we parse the same commit again???\n\n> +\n> +\tfor (p = c->parents; p; p = p->next) {\n> +\t\tstruct commit *parent = p->item;\n> +\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n\nSidenote: I would call this 'indegree_plus_one', not 'indegree'.  But\nmaybe I am too pedantic here.\n\n> +\n> +\t\tif (*pi)\n> +\t\t\t(*pi)++;\n\nIf in-degree of parent is defined, then increase it.\n\n> +\t\telse\n> +\t\t\t*pi = 2;\n\nIf in-degree of parent is not defined, then it is first incoming edge,\nand in-degree plus one is thus 2 (i.e. 1 + INDEGREE_ZERO).\n\n> +\n> +\t\ttest_flag_and_insert(&info->indegree_queue, parent, TOPO_WALK_INDEGREE);\n> +\n> +\t\tif (revs->first_parent_only)\n> +\t\t\treturn;\n> +\t}\n\nThis loop looks all right to me: we insert the parents if they do not\nexist in the queue, and we handle --first-parent correctly.\n\n> +}\n> +\n> +static void compute_indegrees_to_depth(struct rev_info *revs)\n> +{\n> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n> +\tstruct commit *c;\n> +\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n> +\t       c->generation >= info->min_generation)\n> +\t\tindegree_walk_step(revs);\n> +}\n\nAll right, this looks correct.  It is identical with explore_to_depth(),\nbut for the change of queue member of topo_walk_info and step function.\n\nSidenote: if C had true macros (higher-order functions), then it might\nbe worth encoding this structure in a macro.  Preprocessor macros though\nwould make the code more obscure, not less.\n\n>  \n>  static void init_topo_walk(struct rev_info *revs)\n>  {\n>  \tstruct topo_walk_info *info;\n> +\tstruct commit_list *list;\n\nHmmm, I wonder what do we need this 'list' for.\n\n>  \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n>  \tinfo = revs->topo_walk_info;\n>  \tmemset(info, 0, sizeof(struct topo_walk_info));\n>  \n> -\tlimit_list(revs);\n> -\tsort_in_topological_order(&revs->commits, revs->sort_order);\n> +\tinit_indegree_slab(&info->indegree);\n> +\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n> +\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n> +\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n\nWhy this memset uses '\\0' as a filler value and not 0?  The queues are\nnot strings.\n\n> +\n> +\tswitch (revs->sort_order) {\n> +\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n> +\t\tinfo->topo_queue.compare = NULL;\n> +\t\tbreak;\n> +\tcase REV_SORT_BY_COMMIT_DATE:\n> +\t\tinfo->topo_queue.compare = compare_commits_by_commit_date;\n> +\t\tbreak;\n> +\tcase REV_SORT_BY_AUTHOR_DATE:\n> +\t\tinit_author_date_slab(&info->author_date);\n> +\t\tinfo->topo_queue.compare = compare_commits_by_author_date;\n> +\t\tinfo->topo_queue.cb_data = &info->author_date;\n> +\t\tbreak;\n> +\t}\n\nO.K., that are all possible values for revs->sort_order (all possible\nvalues of the rev_sort_order enum).\n\n> +\n> +\tinfo->explore_queue.compare = compare_commits_by_gen_then_commit_date;\n> +\tinfo->indegree_queue.compare = compare_commits_by_gen_then_commit_date;\n\nAll right, those lower level priority queues are sorted by generation\nnumber (with commit date as tie breaker).\n\n> +\n> +\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n> +\tfor (list = revs->commits; list; list = list->next) {\n\nThis list loops over all starting commits, isn't it.\n\n> +\t\tstruct commit *c = list->item;\n> +\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n> +\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n> +\n> +\t\tif (parse_commit_gently(c, 1))\n> +\t\t\tcontinue;\n\nWhy do we insert commits that cannot be parsed to those two queues?\n\n> +\t\tif (c->generation < info->min_generation)\n> +\t\t\tinfo->min_generation = c->generation;\n\nAll right, we have parsed commit 'c' so we know its generation numbers.\n\n> +\t}\n\nHere all starting commits are inserted into both expore_queue (for\nparsing and walk), and to indegree_queue (for in-degree calculations).\nAll right.\n\n> +\n> +\tfor (list = revs->commits; list; list = list->next) {\n> +\t\tstruct commit *c = list->item;\n> +\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n> +\n> +\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n> +\t\t\trecord_author_date(&info->author_date, c);\n> +\t}\n\nThis is a separate loop to initialize and possibly record data in slabs\nfor indegree and author_date info.\n\nI wonder why it is in a separate loop.  Is it to make code cleaner, to\nseparate different concerns into separate loops?\n\n> +\tcompute_indegrees_to_depth(revs);\n\nIt looks a bit strange that depth is not passed as a parameter, but its\nvalue is embedded inside revs structure, but I guess it is done this way\nto keep it in sync.\n\nThough it is a bit *inconsistent* to have explore_to_depth() having\n'gen' parameter, but compute_indegrees_to_depth() not having it.  There\nis '_to_depth()' in a name, and there is no 'depth' parameter...\n\n\nHere we have computed indegrees of all starting commits, walking the\ncommit graph if necessary.\n\n> +\n> +\tfor (list = revs->commits; list; list = list->next) {\n> +\t\tstruct commit *c = list->item;\n> +\n> +\t\tif (*(indegree_slab_at(&info->indegree, c)) == 1)\n> +\t\t\tprio_queue_put(&info->topo_queue, c);\n> +\t}\n\nAnd here we add all commits with no incoming edges, i.e. with real\nin-degree of zero, and \"indegree plus one\" equal 1, or INDEGREE_ZERO.\n\nThis is the starting point of Kahn's algorithm (assuming that in-degrees\nwill be calculated correctly while running it).  All right.\n\n> +\n> +\t/*\n> +\t * This is unfortunate; the initial tips need to be shown\n> +\t * in the order given from the revision traversal machinery.\n> +\t */\n> +\tif (revs->sort_order == REV_SORT_IN_GRAPH_ORDER)\n> +\t\tprio_queue_reverse(&info->topo_queue);\n\nRight, with REV_SORT_IN_GRAPH_ORDER the priority queue is actually a\nstack, and access through this stack reverses the order of commits as it\nwas originally in the list (last commit was added last, and stack is\nLIFO structure, last added element is retrieved first).\n\nI think thet here some sort of complication with regards to\nREV_SORT_IN_GRAPH_ORDER is unavoidable, unless priority queue is\nenhanced to work as an ordinary FIFO queue in addition to making it work\nas LIFO stack.\n\n>  }\n>  \n>  static struct commit *next_topo_commit(struct rev_info *revs)\n>  {\n> -\treturn pop_commit(&revs->commits);\n> +\tstruct commit *c;\n> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n> +\n> +\t/* pop next off of topo_queue */\n> +\tc = prio_queue_get(&info->topo_queue);\n\nAll right, pop_commit() transforms straighforwardly to\nprio_queue_get().\n\n> +\n> +\tif (c)\n> +\t\t*(indegree_slab_at(&info->indegree, c)) = 0;\n\nWhy do we need to mark indegree of commit to be returned as undefined\nhere (INDEGREE_UNINITIALIZED)?\n\n> +\n> +\treturn c;\n>  }\n>\n\nBefore the change, expand_topo_walk() simply added parents to the list,\nand actual sorting was done by sort_in_topological_order().\n\n>  static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n>  {\n> -\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n> +\tstruct commit_list *p;\n> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n> +\tif (process_parents(revs, commit, NULL, NULL) < 0) {\n\nAll right, here we remove storing commits in revs->commits list, the\nthird parameter changed from &revs->commits to NULL.\n\n>  \t\tif (!revs->ignore_missing_links)\n>  \t\t\tdie(\"Failed to traverse parents of commit %s\",\n> -\t\t\t    oid_to_hex(&commit->object.oid));\n> +\t\t\t\toid_to_hex(&commit->object.oid));\n\nThe above looks like spurious and accidental whitespace change, isn't\nit?\n\n> +\t}\n> +\n\nAll right, the loop below looks like the inner loop of the Kahn's\nalgorithm, i.e.:\n\n      for each node m with an edge e from n to m do\n          remove edge e from the graph\n          if m has no other incoming edges then\n              insert m into S\n\n\n> +\tfor (p = commit->parents; p; p = p->next) {\n> +\t\tstruct commit *parent = p->item;\n> +\t\tint *pi;\n> +\n> +\t\tif (parse_commit_gently(parent, 1) < 0)\n> +\t\t\tcontinue;\n\nAll right, we need to parse parent commit to ensure that we can access\nits generation number.\n\n> +\n> +\t\tif (parent->generation < info->min_generation) {\n> +\t\t\tinfo->min_generation = parent->generation;\n> +\t\t\tcompute_indegrees_to_depth(revs);\n> +\t\t}\n\nThe above ensures that the parent will have correctly calculated\nin-degree.  Looks all right.\n\n> +\n> +\t\tpi = indegree_slab_at(&info->indegree, parent);\n> +\n> +\t\t(*pi)--;\n\n          remove edge e from the graph\n\n> +\t\tif (*pi == 1)\n> +\t\t\tprio_queue_put(&info->topo_queue, parent);\n\nIf parent has no incoming edges (indegree == 1 == INDEGREE_ZERO), then\ninsert it into topo_queue.\n\n          if m has no other incoming edges then\n              insert m into S\n\n> +\n> +\t\tif (revs->first_parent_only)\n> +\t\t\treturn;\n>  \t}\n>  }\n\nLooks all right.\n\n>  \n> diff --git a/revision.h b/revision.h\n> index fd4154ff75..b0b3bb8025 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -24,6 +24,8 @@\n>  #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n>  #define TRACK_LINEAR\t(1u<<26)\n>  #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n> +#define TOPO_WALK_EXPLORED\t(1u<<27)\n> +#define TOPO_WALK_INDEGREE\t(1u<<28)\n\nTo be more exact, this flag does not mean that the commit has been\nexplored, or has its in-degree calculated, but that it was added to the\nqueue for exploring, or for having its in-degree calculated.\n\nCurrent names of those two new preprocessor constants might be\nconsidered mildly misleading, absent context.\n\n>  \n>  #define DECORATE_SHORT_REFS\t1\n>  #define DECORATE_FULL_REFS\t2\n\n--\nJakub Narębski\n"},{"id":"361287","messageId":"6501930f-4097-1b81-6d0d-be54f050a5a4@gmail.com","threadId":"49229","inReplyTo":"86sh0y5cgk.fsf@gmail.com","subject":"Re: [PATCH v4 6/7] revision.c: generation-based topo-order algorithm","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-23T13:54:30Z","receivedAt":"2018-10-23T13:54:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/22/2018 9:37 AM, Jakub Narebski wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n>> From: Derrick Stolee <dstolee@microsoft.com>\n>>\n>> The current --topo-order algorithm requires walking all\n>> reachable commits up front, topo-sorting them, all before\n>> outputting the first value. This patch introduces a new\n>> algorithm which uses stored generation numbers to\n>> incrementally walk in topo-order, outputting commits as\n>> we go. This can dramatically reduce the computation time\n>> to write a fixed number of commits, such as when limiting\n>> with \"-n <N>\" or filling the first page of a pager.\n>>\n>> When running a command like 'git rev-list --topo-order HEAD',\n>> Git performed the following steps:\n>>\n>> 1. Run limit_list(), which parses all reachable commits,\n>>     adds them to a linked list, and distributes UNINTERESTING\n>>     flags. If all unprocessed commits are UNINTERESTING, then\n>>     it may terminate without walking all reachable commits.\n>>     This does not occur if we do not specify UNINTERESTING\n>>     commits.\n>>\n>> 2. Run sort_in_topological_order(), which is an implementation\n>>     of Kahn's algorithm. It first iterates through the entire\n>>     set of important commits and computes the in-degree of each\n>>     (plus one, as we use 'zero' as a special value here). Then,\n>>     we walk the commits in priority order, adding them to the\n>>     priority queue if and only if their in-degree is one. As\n>>     we remove commits from this priority queue, we decrement the\n>>     in-degree of their parents.\n> Because in-degree has very specific defined meaning of number of\n> children, i.e. the number of _incoming_ edges, I would say \"if and only\n> if their in-degree-plus-one is one\".  It is more exact, even if it looks\n> a bit funny.\n>\n>> 3. While we are peeling commits for output, get_revision_1()\n>>     uses pop_commit on the full list of commits computed by\n>>     sort_in_topological_order().\n> All right, so those are separate steps (separate walks): prepare and\n> parse commits, topologically sort list of commits from previous step,\n> output sorted list of commits from previous step.\n\nI would rephrase your explanation above as: prepare and parse commits, \ncompute in-degrees, and peel commits of in-degree zero.\n\n>> In the new algorithm, these three steps correspond to three\n>> different commit walks. We run these walks simultaneously,\n>> and advance each only as far as necessary to satisfy the\n>> requirements of the 'higher order' walk.\n> What does 'higher order' walk means: steps 3, 2, 1, in this order,\n> i.e. output being the highest order, or something different?\n\nYes. We only walk \"level 2\" in order to satisfy how far we are in \"level 3\".\n\n> Sidenote: the new algorithm looks a bit like Unix pipeline, where each\n> step of pipeline does not output much more than next step needs /\n> requests.\n\nThat's essentially the idea.\n\n>>                                           We know when we can\n>> pause each walk by using generation numbers from the commit-\n>> graph feature.\n> Do I understand it correctly that this is mainly used in Kahn's\n> algorithm to find out through the negative-cut index of generation\n> number which commits in the to-be-sorted list cannot have an in-degree\n> of zero (or otherise cannot be next commit to be shown in output)?\n\nIn each step of the algorithm, we operate under the assumption that \ncertain vertices have \"all necessary information\".\n\nIn the case of \"level 3\", we need to know that all descendants were \nwalked and our in-degree calculation is correct. We guarantee this by \nensuring that \"level 2\" has walked beyond that commit's generation number.\n\nIn the case of \"level 2\", we need to know that we have parsed all \ndescendants and determined their simplifications (if necessary, such as \nin file-history) and if they are UNINTERESTING. We guarantee this by \nensuring that \"level 1\" has walked beyond that commit's generation number.\n\nIn the previous algorithm, these guarantees were handled by doing each \nstep on all reachable commits before moving to the next level.\n\n>> Recall that the generation number of a commit satisfies:\n>>\n>> * If the commit has at least one parent, then the generation\n>>    number is one more than the maximum generation number among\n>>    its parents.\n>>\n>> * If the commit has no parent, then the generation number is one.\n>>\n>> There are two special generation numbers:\n>>\n>> * GENERATION_NUMBER_INFINITY: this value is 0xffffffff and\n>>    indicates that the commit is not stored in the commit-graph and\n>>    the generation number was not previously calculated.\n>>\n>> * GENERATION_NUMBER_ZERO: this value (0) is a special indicator\n>>    to say that the commit-graph was generated by a version of Git\n>>    that does not compute generation numbers (such as v2.18.0).\n>>\n>> Since we use generation_numbers_enabled() before using the new\n>> algorithm, we do not need to worry about GENERATION_NUMBER_ZERO.\n>> However, the existence of GENERATION_NUMBER_INFINITY implies the\n>> following weaker statement than the usual we expect from\n>> generation numbers:\n>>\n>>      If A and B are commits with generation numbers gen(A) and\n>>      gen(B) and gen(A) < gen(B), then A cannot reach B.\n>>\n>> Thus, we will walk in each of our stages until the \"maximum\n>> unexpanded generation number\" is strictly lower than the\n>> generation number of a commit we are about to use.\n> And this \"maximum unexpanded generation number\" must be greater or equal\n> to 1, thanks to assuming generation_numbers_enabled().\n>\n>\n> Let's start by writing down the original version of the Kahn's algorith\n> (which is not the only way to calculate topological ordering; another\n> method is to use depth-first searches).\n>\n>    L ← Empty list that will contain the sorted elements\n>    S ← Set of all nodes with no incoming edge\n>    while S is non-empty do\n>        remove a node n from S\n>        add n to tail of L\n>        for each node m with an edge e from n to m do\n>            remove edge e from the graph\n>            if m has no other incoming edges then\n>                insert m into S\n>    if graph has edges then\n>        return error   _(graph has at least one cycle)_\n>    else\n>        return L       _(a topologically sorted order)_\n>\n> In the case of Git, we display only commits reachable from the starting\n> commits, so only those starting commits can have no incoming edge, by\n> the definition of the reachable commit (note that some starting commits\n> can be reachable from other starting commits).\n>\n> Note that in Git by construction we cannot have cycles in the objects\n> graph, and that 'remove edge e [= n -> m] from the graph' simply means\n> decreasing the [effective] in-degree of node m.\n>\n>> The walks are as follows:\n>>\n>> 1. EXPLORE: using the explore_queue priority queue (ordered by\n>>     maximizing the generation number), parse each reachable\n>>     commit until all commits in the queue have generation\n>>     number strictly lower than needed. During this walk, update\n>>     the UNINTERESTING flags as necessary.\n> All right, that looks sensible.  Parse commits and update the\n> UNINTERESTING flags only up to what might be needed.\n>\n> Though I would add for each walk what are post-conditions, i.e. what\n> requirements list of returned commits does fullfill.  In the case of the\n> EXPLORE walk it would be that commits are in the \"reachable from start\n> commts\" set, parsed and not UNINTERESTING.  And that there are all such\n> commits there with generation number greater or equal if needed (and\n> their parents).\n>\n>\n> Wouldn't this though make the output always start at the commit with\n> maximal generation number (such commit or commits would need to have an\n> in-degree of zero, i.e. no incoming edges), instead of whatever order is\n> requested (if date order contradicts generation number order) or in the\n> command line arguments order?\n\nThe final order is prioritized in the \"level 3\" walk, which either uses \nan incrementing counter (--topo-order), commit date (--date-order), or \nauthor date (--author-date-order) as the priority.\n\n>> 2. INDEGREE: using the indegree_queue priority queue (ordered\n>>     by maximizing the generation number), add one to the in-\n>>     degree of each parent for each commit that is walked. Since\n>>     we walk in order of decreasing generation number, we know\n>>     that discovering an in-degree value of 0 means the value for\n>>     that commit was not initialized, so should be initialized to\n>>     two. (Recall that in-degree value \"1\" is what we use to say a\n>>     commit is ready for output.)\n> The post-condition is that all returned commits have their in-degree\n> plus one calculated.\n>\n> Mixing actual in-degree (number of incoming edges, zero means no\n> incoming edge and candidate for the next commit in topological order),\n> and details of implementation (using value of zero for uninitialized,\n> and thus actually storing in-degree plus one) makes this description a\n> bit hard to follow.\n>\n> Note that the additional complication is that if commits have generation\n> number INFINITY, then we cannot say anything about reachability among\n> commits with this special generation number.  That means that until we\n> process all commits with generation number INFINITY, we don't know which\n> ones have no incoming edges (a real in-degree of zero).  If they are not\n> INFINITY, the stronger version of the reachability condition for\n> generation number holds, and thus we know that if we pop the commit and\n> it has uninitialized in-degree and generation number not INFINITY, then\n> it has no-incoming edges (in-degree of zero).\n>\n> That does not matter much, but for the fact that before outputting list\n> of commits / returning from the function we need to ensure that all out\n> commits have defined in-degree value.  All that have in-degree undefined\n> when indegree_queue is empty, because of reachability and generation\n> numbers constraints, actually have an in-degree of zero (no incoming\n> edges).\n\nThis is why we walk until exploring _beyond_ a given generation number. \nGeneration number INFINITY is not special with that restriction, as made \nclear in the earlier discussion of generation numbers.\n\nWe expect there to be commits with generation number INFINITY, because \nusers will not be updating their commit-graph with every single 'git \ncommit' command. This mode is covered in our test cases.\n\n>\n>>                                   As we iterate the parents of a\n>>     commit during this walk, ensure the EXPLORE walk has walked\n>>     beyond their generation numbers.\n> All right. looks sensible from the point of view of trying to do\n> streaming of sorted commits.\n>\n>> 3. TOPO: using the topo_queue priority queue (ordered based on\n>>     the sort_order given, which could be commit-date, author-\n>>     date, or typical topo-order which treats the queue as a LIFO\n>>     stack), remove a commit from the queue and decrement the\n>>     in-degree of each parent. If a parent has an in-degree of\n>>     one, then we add it to the topo_queue. Before we decrement\n>>     the in-degree, however, ensure the INDEGREE walk has walked\n>>     beyond that generation number.\n> This description missed an important constraint, namely that all commits\n> in the topo_order queue have real in-degree of zero, i.e. no incoming\n> edges.  The topo_order queue is set S in the Kahn's algorithm.\n>\n> Also, we need to know how to populate the topo_queue at start with at\n> least one commit.  How it is initially populated?  That is very\n> important question.\n>\n>> The implementations of these walks are in the following methods:\n>>\n>> * explore_walk_step and explore_to_depth\n>> * indegree_walk_step and compute_indegrees_to_depth\n>> * next_topo_commit and expand_topo_walk\n> All right, one one hand: good calling convention.  On the other hand:\n> why the difference in naming?\n>\n>> These methods have some patterns that may seem strange at first,\n>> but they are probably carry-overs from their equivalents in\n>> limit_list and sort_in_topological_order.\n>>\n>> One thing that is missing from this implementation is a proper\n>> way to stop walking when the entire queue is UNINTERESTING, so\n>> this implementation is not enabled by comparisions, such as in\n>> 'git rev-list --topo-order A..B'. This can be updated in the\n>> future.\n> All right, lets start with easier step.\n>\n>> In my local testing, I used the following Git commands on the\n>> Linux repository in three modes: HEAD~1 with no commit-graph,\n>> HEAD~1 with a commit-graph, and HEAD with a commit-graph. This\n>> allows comparing the benefits we get from parsing commits from\n>> the commit-graph and then again the benefits we get by\n>> restricting the set of commits we walk.\n>>\n>> Test: git rev-list --topo-order -100 HEAD\n>> HEAD~1, no commit-graph: 6.80 s\n>> HEAD~1, w/ commit-graph: 0.77 s\n>>    HEAD, w/ commit-graph: 0.02 s\n>>\n>> Test: git rev-list --topo-order -100 HEAD -- tools\n>> HEAD~1, no commit-graph: 9.63 s\n>> HEAD~1, w/ commit-graph: 6.06 s\n>>    HEAD, w/ commit-graph: 0.06 s\n>>\n>> This speedup is due to a few things. First, the new generation-\n>> number-enabled algorithm walks commits on order of the number of\n>> results output (subject to some branching structure expectations).\n>> Since we limit to 100 results, we are running a query similar to\n>> filling a single page of results. Second, when specifying a path,\n>> we must parse the root tree object for each commit we walk. The\n>> previous benefits from the commit-graph are entirely from reading\n>> the commit-graph instead of parsing commits. Since we need to\n>> parse trees for the same number of commits as before, we slow\n>> down significantly from the non-path-based query.\n>>\n>> For the test above, I specifically selected a path that is changed\n>> frequently, including by merge commits. A less-frequently-changed\n>> path (such as 'README') has similar end-to-end time since we need\n>> to walk the same number of commits (before determining we do not\n>> have 100 hits). However, get the benefit that the output is\n>> presented to the user as it is discovered, much the same as a\n>> normal 'git log' command (no '--topo-order'). This is an improved\n>> user experience, even if the command has the same runtime.\n> First, do I understand it correctly that in first case the gains from\n> new algorithms are so slim because with commit-graph file and no path\n> limiting we don't hit repository anyway; we walk less commits, but\n> reading commit data from commit-graph file is fast/\n\nIf you mean 0.77s to 0.02s is \"slim\" then yes, it is because the \ncommit-graph command already made a full walk of the commit history \nfaster. (I'm only poking at this because the _relative_ improvement is \nsignificant, even if the command was already sub-second.)\n\n> Second, I wonder if there is some easy way to perform automatic latency\n> tests, i.e. how fast does Git show the first page of output...\n\nI have talked with Jeff Hostetler about this, to see if we can have a \n\"time to first page\" traced with trace2, but we don't seem to have \naccess to that information within Git. We would need to insert it into \nthe pager. The \"-100\" is used instead.\n\n>\n>> Helped-by: Jeff King <peff@peff.net>\n>> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n>> ---\n>>   object.h   |   4 +-\n>>   revision.c | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n>>   revision.h |   2 +\n>>   3 files changed, 197 insertions(+), 8 deletions(-)\n> Daunting change to review.\n>\n>> diff --git a/object.h b/object.h\n>> index 0feb90ae61..796792cb32 100644\n>> --- a/object.h\n>> +++ b/object.h\n>> @@ -59,7 +59,7 @@ struct object_array {\n>>   \n>>   /*\n>>    * object flag allocation:\n>> - * revision.h:               0---------10                              2526\n>> + * revision.h:               0---------10                              25----28\n>>    * fetch-pack.c:             01\n>>    * negotiator/default.c:       2--5\n>>    * walker.c:                 0-2\n>> @@ -78,7 +78,7 @@ struct object_array {\n>>    * builtin/show-branch.c:    0-------------------------------------------26\n>>    * builtin/unpack-objects.c:                                 2021\n>>    */\n>> -#define FLAG_BITS  27\n>> +#define FLAG_BITS  29\n> What are those two additional object flags needed for revision.h /\n> revision.c after this change?\n>\n> Ah, those are TOPO_WALK_EXPLORED and TOPO_WALK_INDEGREE.\n>\n>>   \n>>   /*\n>>    * The object type is stored in 3 bits.\n>> diff --git a/revision.c b/revision.c\n>> index 36458265a0..472f3994e3 100644\n>> --- a/revision.c\n>> +++ b/revision.c\n>> @@ -26,6 +26,7 @@\n>>   #include \"argv-array.h\"\n>>   #include \"commit-reach.h\"\n>>   #include \"commit-graph.h\"\n>> +#include \"prio-queue.h\"\n>>   \n>>   volatile show_early_output_fn_t show_early_output;\n>>   \n>> @@ -2895,30 +2896,216 @@ static int mark_uninteresting(const struct object_id *oid,\n>>   \treturn 0;\n>>   }\n>>   \n>> -struct topo_walk_info {};\n>> +define_commit_slab(indegree_slab, int);\n>> +\n>> +struct topo_walk_info {\n>> +\tuint32_t min_generation;\n>> +\tstruct prio_queue explore_queue;\n>> +\tstruct prio_queue indegree_queue;\n>> +\tstruct prio_queue topo_queue;\n>> +\tstruct indegree_slab indegree;\n> All right.\n>\n>> +\tstruct author_date_slab author_date;\n> Why this slab is needed in topo_walk_info struct?\n>\n>> +};\n>> +\n>> +static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n>> +{\n>> +\tif (c->object.flags & flag)\n>> +\t\treturn;\n>> +\n>> +\tc->object.flags |= flag;\n>> +\tprio_queue_put(q, c);\n>> +}\n> This is an independent change, though I see that it is quite specific\n> (as opposed to quite generic prio_queue_peek() operation added earlier\n> in this series), so it does not make much sense as standalone change.\n>\n> It inserts commit into priority queue only if it didn't have flags set,\n> and sets the flag (so we won't add it to the queue again, not without\n> unsetting the flag), am I correct?\n\nYes, this pattern of using a flag to avoid duplicate entries in the \npriority queue appears in multiple walks. It wasn't needed before. We \ncall it four times in the code below.\n>> +\n>> +static void explore_walk_step(struct rev_info *revs)\n>> +{\n>> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n>> +\tstruct commit_list *p;\n>> +\tstruct commit *c = prio_queue_get(&info->explore_queue);\n>> +\n>> +\tif (!c)\n>> +\t\treturn;\n>> +\n>> +\tif (parse_commit_gently(c, 1) < 0)\n>> +\t\treturn;\n> All right, all commits taken out of explore_queue are parsed.  This is\n> used to ensure that all commits qith generation number greater than some\n> set cutoff are parsed.\n>\n>> +\n>> +\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n>> +\t\trecord_author_date(&info->author_date, c);\n>> +\n>> +\tif (revs->max_age != -1 && (c->date < revs->max_age))\n>> +\t\tc->object.flags |= UNINTERESTING;\n> These two conditionals looks a bit strange to me; they are hardcoded\n> specific cases of query.  But that might be just me...\n\nThese special cases are important for making all of the different option \nflags to rev-list work with the algorithm. They are pulled directly from \nlimit_list().\n\n>\n>> +\n>> +\tif (process_parents(revs, c, NULL, NULL) < 0)\n>> +\t\treturn;\n> I see that we are using process_parents(), formerly i.e. before patch\n> 5/7 add_parents_to_list(), with NULL 'list' parameter for the first\n> time.\n>> +\n>> +\tif (c->object.flags & UNINTERESTING)\n>> +\t\tmark_parents_uninteresting(c);\n>> +\n>> +\tfor (p = c->parents; p; p = p->next)\n>> +\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n> Do we need to insert parents to the queue even if they were marked\n> UNINTERESTING?\n\nWe need to propagate the UNINTERESTING flag to our parents. That \npropagation happens in process_parents().\n\n>\n> I guess that we use test_flag_and_insert() instead of prio_queue_put()\n> to avoid duplicate entries in the queue.  I think the queue is initially\n> populated with the starting commits, but those need not to be\n> unreachable from each other, and walking down parents we can encounter\n> starting commit already in the queue.  Am I correct?\n\nWe can also reach commits in multiple ways, so the initial conditions \nare not the only ways to insert duplicates.\n\n>> +}\n> Let's compare this new function with the limit_list() used in the old\n> algorithm for --topo-order walk (and even now for A..B walks), or to be\n> more exact with the contents of the while loop.\n>\n> 1. limit_list() doesn't have the check if the commit exists, and\n>     does not use parse_commit_gently().  Why the difference, i.e. where\n>     revs->commits gets parsed, and why explore_walk_step() cannot rely on\n>     this?\n>\n>     I get that the goal is to not have parse commits if not needed, so it\n>     is good that it is moved to explore_walk_step().\n>\n> 2. limit_list() is also missing running record_author_date() when\n>     sorting output by author date.  I guess that explore_walk_step()\n>     needs this because commit-graph file does not include this\n>     information.\n>\n> 3. Handling of revs->max_age by marking commit as UNINTERESTING if\n>     needed is the same in limit_list() and in explore_walk_step().\n>\n> 4. limit_list() but not explore_walk_step() handles revs->min_age near\n>     the end pf the loop by terminating the loop.  I guess for this case\n>     we have revs->limited set, and we use old algorithm, isn't it?\n>\n>     Something to remember when adding A..B handling to new algorithm.\n>\n> 5. add_parents_to_list() / process_parents() is nearly the same in\n>     limit_list() and in explore_walk_step(), but for the fact that the\n>     new function doesn't use 'list' parameter.\n>\n> 6. Both limit_list() and explore_walk_step() use\n>     mark_parents_uninteresting() on uninteresting commits.\n>\n>     However, limit_list() breaks out of the loop, and uses\n>     interesting_cache with slop.  I guess that those two facts are\n>     connected, right?\n>\n> 7. Then explore_walk_step() inserts parents to the priority queue if\n>     they are not present there already, with test_flag_and_insert(),\n>     which rough equivalent in limit_list() would be using\n>     commit_list_insert().\n>\n> 8. limit_list() has also some code for show_early_output, which I guess\n>     explore_walk_step() does not need to handle.\n>\n>> +\n>> +static void explore_to_depth(struct rev_info *revs,\n>> +\t\t\t     uint32_t gen)\n>> +{\n>> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n>> +\tstruct commit *c;\n>> +\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n>> +\t       c->generation >= gen)\n> I have originally thought that if we extract prio_queue_get() and\n> test_flag_and_insert() / prio_queue_put() out of explore_walk_step() and\n> put it into this loop, i.e. into the calling function, we could avoid\n> code duplication between explore_walk_step() and limit_list()... but I\n> guess that is impossible anyway.\n>\n>> +\t\texplore_walk_step(revs);\n>> +}\n> Nice, tight, and easy to understand function.  Though perhaps 'gen'\n> could be called 'gen_cutoff' or 'min_gen', or 'min_gen_cufott'.\n>\n>> +\n>> +static void indegree_walk_step(struct rev_info *revs)\n>> +{\n>> +\tstruct commit_list *p;\n>> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n>> +\tstruct commit *c = prio_queue_get(&info->indegree_queue);\n>> +\n>> +\tif (!c)\n>> +\t\treturn;\n>> +\n>> +\tif (parse_commit_gently(c, 1) < 0)\n>> +\t\treturn;\n> All right, we need to parse commit 'c' to have its generation number,\n> and we need to do the same in explore_walk_step() because we walk\n> possibly unparsed parents.\n>\n>> +\n>> +\texplore_to_depth(revs, c->generation);\n> If we walk everything up to current commit depth, then we have walked\n> all commits that can affect in-degree of current commit.  Good.\n>\n>> +\n>> +\tif (parse_commit_gently(c, 1) < 0)\n>> +\t\treturn;\n> Why do we parse the same commit again???\n\nGood point! Accidental duplicate lines.\n\n>> +\n>> +\tfor (p = c->parents; p; p = p->next) {\n>> +\t\tstruct commit *parent = p->item;\n>> +\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n> Sidenote: I would call this 'indegree_plus_one', not 'indegree'.  But\n> maybe I am too pedantic here.\n>\n>> +\n>> +\t\tif (*pi)\n>> +\t\t\t(*pi)++;\n> If in-degree of parent is defined, then increase it.\n>\n>> +\t\telse\n>> +\t\t\t*pi = 2;\n> If in-degree of parent is not defined, then it is first incoming edge,\n> and in-degree plus one is thus 2 (i.e. 1 + INDEGREE_ZERO).\n>\n>> +\n>> +\t\ttest_flag_and_insert(&info->indegree_queue, parent, TOPO_WALK_INDEGREE);\n>> +\n>> +\t\tif (revs->first_parent_only)\n>> +\t\t\treturn;\n>> +\t}\n> This loop looks all right to me: we insert the parents if they do not\n> exist in the queue, and we handle --first-parent correctly.\n>\n>> +}\n>> +\n>> +static void compute_indegrees_to_depth(struct rev_info *revs)\n>> +{\n>> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n>> +\tstruct commit *c;\n>> +\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n>> +\t       c->generation >= info->min_generation)\n>> +\t\tindegree_walk_step(revs);\n>> +}\n> All right, this looks correct.  It is identical with explore_to_depth(),\n> but for the change of queue member of topo_walk_info and step function.\n>\n> Sidenote: if C had true macros (higher-order functions), then it might\n> be worth encoding this structure in a macro.  Preprocessor macros though\n> would make the code more obscure, not less.\n>\n>>   \n>>   static void init_topo_walk(struct rev_info *revs)\n>>   {\n>>   \tstruct topo_walk_info *info;\n>> +\tstruct commit_list *list;\n> Hmmm, I wonder what do we need this 'list' for.\n>\n>>   \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n>>   \tinfo = revs->topo_walk_info;\n>>   \tmemset(info, 0, sizeof(struct topo_walk_info));\n>>   \n>> -\tlimit_list(revs);\n>> -\tsort_in_topological_order(&revs->commits, revs->sort_order);\n>> +\tinit_indegree_slab(&info->indegree);\n>> +\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n>> +\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n>> +\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n> Why this memset uses '\\0' as a filler value and not 0?  The queues are\n> not strings.\n>\n>> +\n>> +\tswitch (revs->sort_order) {\n>> +\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n>> +\t\tinfo->topo_queue.compare = NULL;\n>> +\t\tbreak;\n>> +\tcase REV_SORT_BY_COMMIT_DATE:\n>> +\t\tinfo->topo_queue.compare = compare_commits_by_commit_date;\n>> +\t\tbreak;\n>> +\tcase REV_SORT_BY_AUTHOR_DATE:\n>> +\t\tinit_author_date_slab(&info->author_date);\n>> +\t\tinfo->topo_queue.compare = compare_commits_by_author_date;\n>> +\t\tinfo->topo_queue.cb_data = &info->author_date;\n>> +\t\tbreak;\n>> +\t}\n> O.K., that are all possible values for revs->sort_order (all possible\n> values of the rev_sort_order enum).\n>\n>> +\n>> +\tinfo->explore_queue.compare = compare_commits_by_gen_then_commit_date;\n>> +\tinfo->indegree_queue.compare = compare_commits_by_gen_then_commit_date;\n> All right, those lower level priority queues are sorted by generation\n> number (with commit date as tie breaker).\n>\n>> +\n>> +\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n>> +\tfor (list = revs->commits; list; list = list->next) {\n> This list loops over all starting commits, isn't it.\n>\n>> +\t\tstruct commit *c = list->item;\n>> +\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n>> +\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n>> +\n>> +\t\tif (parse_commit_gently(c, 1))\n>> +\t\t\tcontinue;\n> Why do we insert commits that cannot be parsed to those two queues?\n>\n>> +\t\tif (c->generation < info->min_generation)\n>> +\t\t\tinfo->min_generation = c->generation;\n> All right, we have parsed commit 'c' so we know its generation numbers.\n>\n>> +\t}\n> Here all starting commits are inserted into both expore_queue (for\n> parsing and walk), and to indegree_queue (for in-degree calculations).\n> All right.\n>\n>> +\n>> +\tfor (list = revs->commits; list; list = list->next) {\n>> +\t\tstruct commit *c = list->item;\n>> +\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n>> +\n>> +\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n>> +\t\t\trecord_author_date(&info->author_date, c);\n>> +\t}\n> This is a separate loop to initialize and possibly record data in slabs\n> for indegree and author_date info.\n>\n> I wonder why it is in a separate loop.  Is it to make code cleaner, to\n> separate different concerns into separate loops?\n>\n>> +\tcompute_indegrees_to_depth(revs);\n> It looks a bit strange that depth is not passed as a parameter, but its\n> value is embedded inside revs structure, but I guess it is done this way\n> to keep it in sync.\n>\n> Though it is a bit *inconsistent* to have explore_to_depth() having\n> 'gen' parameter, but compute_indegrees_to_depth() not having it.  There\n> is '_to_depth()' in a name, and there is no 'depth' parameter...\n>\n>\n> Here we have computed indegrees of all starting commits, walking the\n> commit graph if necessary.\n>\n>> +\n>> +\tfor (list = revs->commits; list; list = list->next) {\n>> +\t\tstruct commit *c = list->item;\n>> +\n>> +\t\tif (*(indegree_slab_at(&info->indegree, c)) == 1)\n>> +\t\t\tprio_queue_put(&info->topo_queue, c);\n>> +\t}\n> And here we add all commits with no incoming edges, i.e. with real\n> in-degree of zero, and \"indegree plus one\" equal 1, or INDEGREE_ZERO.\n>\n> This is the starting point of Kahn's algorithm (assuming that in-degrees\n> will be calculated correctly while running it).  All right.\n>\n>> +\n>> +\t/*\n>> +\t * This is unfortunate; the initial tips need to be shown\n>> +\t * in the order given from the revision traversal machinery.\n>> +\t */\n>> +\tif (revs->sort_order == REV_SORT_IN_GRAPH_ORDER)\n>> +\t\tprio_queue_reverse(&info->topo_queue);\n> Right, with REV_SORT_IN_GRAPH_ORDER the priority queue is actually a\n> stack, and access through this stack reverses the order of commits as it\n> was originally in the list (last commit was added last, and stack is\n> LIFO structure, last added element is retrieved first).\n>\n> I think thet here some sort of complication with regards to\n> REV_SORT_IN_GRAPH_ORDER is unavoidable, unless priority queue is\n> enhanced to work as an ordinary FIFO queue in addition to making it work\n> as LIFO stack.\n>\n>>   }\n>>   \n>>   static struct commit *next_topo_commit(struct rev_info *revs)\n>>   {\n>> -\treturn pop_commit(&revs->commits);\n>> +\tstruct commit *c;\n>> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n>> +\n>> +\t/* pop next off of topo_queue */\n>> +\tc = prio_queue_get(&info->topo_queue);\n> All right, pop_commit() transforms straighforwardly to\n> prio_queue_get().\n>\n>> +\n>> +\tif (c)\n>> +\t\t*(indegree_slab_at(&info->indegree, c)) = 0;\n> Why do we need to mark indegree of commit to be returned as undefined\n> here (INDEGREE_UNINITIALIZED)?\n>\n>> +\n>> +\treturn c;\n>>   }\n>>\n> Before the change, expand_topo_walk() simply added parents to the list,\n> and actual sorting was done by sort_in_topological_order().\n>\n>>   static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n>>   {\n>> -\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n>> +\tstruct commit_list *p;\n>> +\tstruct topo_walk_info *info = revs->topo_walk_info;\n>> +\tif (process_parents(revs, commit, NULL, NULL) < 0) {\n> All right, here we remove storing commits in revs->commits list, the\n> third parameter changed from &revs->commits to NULL.\n>\n>>   \t\tif (!revs->ignore_missing_links)\n>>   \t\t\tdie(\"Failed to traverse parents of commit %s\",\n>> -\t\t\t    oid_to_hex(&commit->object.oid));\n>> +\t\t\t\toid_to_hex(&commit->object.oid));\n> The above looks like spurious and accidental whitespace change, isn't\n> it?\n\nCorrect. Thanks for finding it.\n\n>\n>> +\t}\n>> +\n> All right, the loop below looks like the inner loop of the Kahn's\n> algorithm, i.e.:\n>\n>        for each node m with an edge e from n to m do\n>            remove edge e from the graph\n>            if m has no other incoming edges then\n>                insert m into S\n>\n>\n>> +\tfor (p = commit->parents; p; p = p->next) {\n>> +\t\tstruct commit *parent = p->item;\n>> +\t\tint *pi;\n>> +\n>> +\t\tif (parse_commit_gently(parent, 1) < 0)\n>> +\t\t\tcontinue;\n> All right, we need to parse parent commit to ensure that we can access\n> its generation number.\n>\n>> +\n>> +\t\tif (parent->generation < info->min_generation) {\n>> +\t\t\tinfo->min_generation = parent->generation;\n>> +\t\t\tcompute_indegrees_to_depth(revs);\n>> +\t\t}\n> The above ensures that the parent will have correctly calculated\n> in-degree.  Looks all right.\n>\n>> +\n>> +\t\tpi = indegree_slab_at(&info->indegree, parent);\n>> +\n>> +\t\t(*pi)--;\n>            remove edge e from the graph\n>\n>> +\t\tif (*pi == 1)\n>> +\t\t\tprio_queue_put(&info->topo_queue, parent);\n> If parent has no incoming edges (indegree == 1 == INDEGREE_ZERO), then\n> insert it into topo_queue.\n>\n>            if m has no other incoming edges then\n>                insert m into S\n>\n>> +\n>> +\t\tif (revs->first_parent_only)\n>> +\t\t\treturn;\n>>   \t}\n>>   }\n> Looks all right.\n\nThanks for taking the time on this huge patch!\n\nThanks,\n-Stolee\n"},{"id":"361294","messageId":"86lg6o64w7.fsf@gmail.com","threadId":"49229","inReplyTo":"a21febe1128cbaf69f2201598d9efee6f6a4b2d8.1539729393.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 7/7] t6012: make rev-list tests more interesting","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-23T15:48:08Z","receivedAt":"2018-10-23T15:48:14Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> As we are working to rewrite some of the revision-walk machinery,\n> there could easily be some interesting interactions between the\n> options that force topological constraints (--topo-order,\n> --date-order, and --author-date-order) along with specifying a\n> path.\n>\n> Add extra tests to t6012-rev-list-simplify.sh to add coverage of\n> these interactions. To ensure interesting things occur, alter the\n> repo data shape to have different orders depending on topo-, date-,\n> or author-date-order.\n\nVery nice, though I have noticed (please correct me if I am wrong) that\nin all cases path limited query always have the same result for\n--topo-order and for --date-order; as opposed to three different results\nfor three different revision sorting modes for path-less query.\n\n>\n> When testing using GIT_TEST_COMMIT_GRAPH, this assists in covering\n> the new logic for topo-order walks using generation numbers. The\n> extra tests can be added indepently.\n\nGood.  I guess we are mainly interested in tests without limits and\nexclusions, i.e. A or A B and not A..B or A...B walks.\n\n>\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  t/t6012-rev-list-simplify.sh | 45 ++++++++++++++++++++++++++++--------\n>  1 file changed, 36 insertions(+), 9 deletions(-)\n>\n> diff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\n> index b5a1190ffe..a10f0df02b 100755\n> --- a/t/t6012-rev-list-simplify.sh\n> +++ b/t/t6012-rev-list-simplify.sh\n> @@ -12,6 +12,22 @@ unnote () {\n>  \tgit name-rev --tags --stdin | sed -e \"s|$OID_REGEX (tags/\\([^)]*\\)) |\\1 |g\"\n>  }\n>  \n> +#\n> +# Create a test repo with interesting commit graph:\n> +#\n> +# A--B----------G--H--I--K--L\n> +#  \\  \\           /     /\n> +#   \\  \\         /     /\n> +#    C------E---F     J\n> +#        \\_/\n> +#\n> +# The commits are laid out from left-to-right starting with\n> +# the root commit A and terminating at the tip commit L.\n\nDo I understand it correctly that it is a visualization of history\ncreated by existing code (which is a very nice to have)?\n\n> +#\n> +# There are a few places where we adjust the commit date or\n> +# author date to make the --topo-order, --date-order, and\n> +# --author-date-order flags produce different output.\n\nSidenote: it looks like \"a few places\" is \"one place\" for now...\n\n> +\n>  test_expect_success setup '\n>  \techo \"Hi there\" >file &&\n>  \techo \"initial\" >lost &&\n> @@ -21,10 +37,18 @@ test_expect_success setup '\n>  \n>  \tgit branch other-branch &&\n>  \n> +\tgit symbolic-ref HEAD refs/heads/unrelated &&\n> +\tgit rm -f \"*\" &&\n> +\techo \"Unrelated branch\" >side &&\n> +\tgit add side &&\n> +\ttest_tick && git commit -m \"Side root\" &&\n> +\tnote J &&\n> +\tgit checkout master &&\n\nI see that this fragment is moved earlier, but I don't know what\nconsequences it does have.\n\n> +\n>  \techo \"Hello\" >file &&\n>  \techo \"second\" >lost &&\n>  \tgit add file lost &&\n> -\ttest_tick && git commit -m \"Modified file and lost\" &&\n> +\ttest_tick && GIT_AUTHOR_DATE=$(($test_tick + 120)) git commit -m \"Modified file and lost\" &&\n>  \tnote B &&\n\nNice trick, though I think it produces slightly unrealistic history (at\nleast in absence of the clock skew).  Author dates are ordinarily\nearlier or equal to commit dates, and commits can be authored in\ndifferent order that they were committed.\n\n>  \n>  \tgit checkout other-branch &&\n> @@ -63,13 +87,6 @@ test_expect_success setup '\n>  \ttest_tick && git commit -a -m \"Final change\" &&\n>  \tnote I &&\n>  \n> -\tgit symbolic-ref HEAD refs/heads/unrelated &&\n> -\tgit rm -f \"*\" &&\n> -\techo \"Unrelated branch\" >side &&\n> -\tgit add side &&\n> -\ttest_tick && git commit -m \"Side root\" &&\n> -\tnote J &&\n> -\n>  \tgit checkout master &&\n>  \ttest_tick && git merge --allow-unrelated-histories -m \"Coolest\" unrelated &&\n>  \tnote K &&\n> @@ -103,14 +120,24 @@ check_result () {\n>  \tcheck_outcome success \"$@\"\n>  }\n>  \n> -check_result 'L K J I H G F E D C B A' --full-history\n> +check_result 'L K J I H F E D C G B A' --full-history --topo-order\n> +check_result 'L K I H G F E D C B J A' --full-history\n> +check_result 'L K I H G F E D C B J A' --full-history --date-order\n> +check_result 'L K I H G F E D B C J A' --full-history --author-date-order\n>  check_result 'K I H E C B A' --full-history -- file\n>  check_result 'K I H E C B A' --full-history --topo-order -- file\n>  check_result 'K I H E C B A' --full-history --date-order -- file\n> +check_result 'K I H E B C A' --full-history --author-date-order -- file\n>  check_result 'I E C B A' --simplify-merges -- file\n> +check_result 'I E C B A' --simplify-merges --topo-order -- file\n> +check_result 'I E C B A' --simplify-merges --date-order -- file\n> +check_result 'I E B C A' --simplify-merges --author-date-order -- file\n>  check_result 'I B A' -- file\n>  check_result 'I B A' --topo-order -- file\n> +check_result 'I B A' --date-order -- file\n> +check_result 'I B A' --author-date-order -- file\n>  check_result 'H' --first-parent -- another-file\n> +check_result 'H' --first-parent --topo-order -- another-file\n>  \n>  check_result 'E C B A' --full-history E -- lost\n>  test_expect_success 'full history simplification without parent' '\n\nMore tests, looks good.\n"},{"id":"361487","messageId":"xmqqk1m6ig52.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"bb91d5bb-4220-97b9-dd78-5aed39033e58@gmail.com","subject":"Re: [PATCH v4 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-25T08:28:57Z","receivedAt":"2018-10-25T08:29:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n>     time git log --topo-order -10 master >/dev/null\n>\n>     time git log --topo-order -10 maint..master >/dev/null\n>\n> I get 0.39s for the first call and 0.01s for the second. (Note: I\n> specified \"-10\" to ensure we are only writing 10 commits and the\n> output size does not factor into the time.) This is because the first\n> walks the entire history, while the second uses the heuristic walk to\n> identify a much smaller subgraph that the topo-order algorithm uses.\n\nThe algorithm can be fooled by skewed timestamps (i.e. that SLOP\nthing tries to work around), but is helped by being able to leave\nearly, and it will give us the correct answer as long as there is no\ntimestamp inversion.\n\nBut monotonically increasing \"timestamp\" without inversion is what\nwe invented \"generation numbers\" for, no?  When there is no\ntimestamp inversion, would a walk based on commit timestamps walk\nsmaller set than a walk based on commit generation numbers?\n\n> Just as before, by using this algorithm for the B..A case, we are\n> adding an extra restriction on the algorithm: always be correct. This\n> results in us walking a larger set (everything reachable from B or A\n> with generation number at least the smallest generation of a commit\n> reachable from only one).\n>\n> I believe this can be handled by using a smarter generation number\n> (one that relies on commit date as a heuristic, but still have enough\n> information to guarantee topological relationships), and I've already\n> started testing a few of these directions.\n\nGood ot hear.\n"},{"id":"361501","messageId":"20181025094345.GA18794@sigill.intra.peff.net","threadId":"49229","inReplyTo":"1c0eb7d3-7f31-9377-d42f-4ac2f36ac26a@gmail.com","subject":"Re: [PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-25T09:43:46Z","receivedAt":"2018-10-25T09:43:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 11, 2018 at 12:21:44PM -0400, Derrick Stolee wrote:\n\n> > > 2. INDEGREE: using the indegree_queue priority queue (ordered\n> > >     by maximizing the generation number), add one to the in-\n> > >     degree of each parent for each commit that is walked. Since\n> > >     we walk in order of decreasing generation number, we know\n> > >     that discovering an in-degree value of 0 means the value for\n> > >     that commit was not initialized, so should be initialized to\n> > >     two. (Recall that in-degree value \"1\" is what we use to say a\n> > >     commit is ready for output.) As we iterate the parents of a\n> > >     commit during this walk, ensure the EXPLORE walk has walked\n> > >     beyond their generation numbers.\n> > I wondered how this would work for INFINITY. We can't know the order of\n> > a bunch of INFINITY nodes at all, so we never know when their in-degree\n> > values are \"done\". But if I understand the EXPLORE walk, we'd basically\n> > walk all of INFINITY down to something with a real generation number. Is\n> > that right?\n> > \n> > But after that, I'm not totally clear on why we need this INDEGREE walk.\n> \n> The INDEGREE walk is an important element for Kahn's algorithm. The final\n> output order is dictated by peeling commits of \"indegree zero\" to ensure all\n> children are output before their parents. (Note: since we use literal 0 to\n> mean \"uninitialized\", we peel commits when the indegree slab has value 1.)\n> \n> This walk replaces the indegree logic from sort_in_topological_order(). That\n> method performs one walk that fills the indegree slab, then another walk\n> that peels the commits with indegree 0 and inserts them into a list.\n\nI guess my big question here was: if we have generation numbers, do we\nneed Kahn's algorithm? That is, in a fully populated set of generation\nnumbers (i.e., no INFINITY), we could always just pick a commit with the\nhighest generation number to show.\n\nSo if we EXPLORE down to a real generation number in phase 1, why do we\nneed to care about INDEGREE anymore? Or am I wrong that we always have a\nreal generation number (i.e., not INFINITY) after EXPLORE? (And if so,\nwhy is exploring to a real generation number a bad idea; presumably\nit's due to a worst-case that goes deeper than we'd otherwise need to\nhere).\n\n> [...]\n\nEverything else you said here made perfect sense.\n\n-Peff\n"},{"id":"361512","messageId":"8dfc5be8-e3f8-51ab-cdf8-1f083c503a58@gmail.com","threadId":"49229","inReplyTo":"20181025094345.GA18794@sigill.intra.peff.net","subject":"Re: [PATCH v3 7/7] revision.c: refactor basic topo-order logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-25T13:00:44Z","receivedAt":"2018-10-25T13:00:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/25/2018 5:43 AM, Jeff King wrote:\n> On Thu, Oct 11, 2018 at 12:21:44PM -0400, Derrick Stolee wrote:\n>\n>>>> 2. INDEGREE: using the indegree_queue priority queue (ordered\n>>>>      by maximizing the generation number), add one to the in-\n>>>>      degree of each parent for each commit that is walked. Since\n>>>>      we walk in order of decreasing generation number, we know\n>>>>      that discovering an in-degree value of 0 means the value for\n>>>>      that commit was not initialized, so should be initialized to\n>>>>      two. (Recall that in-degree value \"1\" is what we use to say a\n>>>>      commit is ready for output.) As we iterate the parents of a\n>>>>      commit during this walk, ensure the EXPLORE walk has walked\n>>>>      beyond their generation numbers.\n>>> I wondered how this would work for INFINITY. We can't know the order of\n>>> a bunch of INFINITY nodes at all, so we never know when their in-degree\n>>> values are \"done\". But if I understand the EXPLORE walk, we'd basically\n>>> walk all of INFINITY down to something with a real generation number. Is\n>>> that right?\n>>>\n>>> But after that, I'm not totally clear on why we need this INDEGREE walk.\n>> The INDEGREE walk is an important element for Kahn's algorithm. The final\n>> output order is dictated by peeling commits of \"indegree zero\" to ensure all\n>> children are output before their parents. (Note: since we use literal 0 to\n>> mean \"uninitialized\", we peel commits when the indegree slab has value 1.)\n>>\n>> This walk replaces the indegree logic from sort_in_topological_order(). That\n>> method performs one walk that fills the indegree slab, then another walk\n>> that peels the commits with indegree 0 and inserts them into a list.\n> I guess my big question here was: if we have generation numbers, do we\n> need Kahn's algorithm? That is, in a fully populated set of generation\n> numbers (i.e., no INFINITY), we could always just pick a commit with the\n> highest generation number to show.\n>\n> So if we EXPLORE down to a real generation number in phase 1, why do we\n> need to care about INDEGREE anymore? Or am I wrong that we always have a\n> real generation number (i.e., not INFINITY) after EXPLORE? (And if so,\n> why is exploring to a real generation number a bad idea; presumably\n> it's due to a worst-case that goes deeper than we'd otherwise need to\n> here).\n\nThe issue is that we our final order (used by level 3) is unrelated to \ngeneration number. Yes, if we prioritized by generation number then we \ncould output the commits in _some_ order that doesn't violate \ntopological constraints. However, we are asking for a different \npriority, which is different than the generation number priority.\n\nIn the case of \"--topo-order\", we want to output the commits reachable \nfrom the second parent of a merge before the commits reachable from the \nfirst parent. However, in most cases the generation number of the first \nparent is higher than the second parent (there are more things in the \nmerge chain than in a small topic that got merged). The INDEGREE is what \nallows us to know when we can peel these commits while still respecting \nthe priority we want at the end.\n\nThanks,\n-Stolee\n"},{"id":"361629","messageId":"86bm7g641x.fsf@gmail.com","threadId":"49229","inReplyTo":"6501930f-4097-1b81-6d0d-be54f050a5a4@gmail.com","subject":"Re: [PATCH v4 6/7] revision.c: generation-based topo-order algorithm","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-26T16:55:22Z","receivedAt":"2018-10-26T16:55:30Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n> On 10/22/2018 9:37 AM, Jakub Narebski wrote:\n>> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n[...]\n>> Sidenote: the new algorithm looks a bit like Unix pipeline, where each\n>> step of pipeline does not output much more than next step needs /\n>> requests.\n>\n> That's essentially the idea.\n\nSome of the newer languages have built-in support for similar kind of\npipeline for connecting processes, be it channels in Go, supplies and\nsuppliers in Perl6.  I wonder if there exists some library implementing\nthis kind of construct in C.\n\nThat aside, I wonder if when there would be support for more\nreachability indices than generation numbers, if it wouldn't be better\nto pass a commit as a limiter (up to this commit), than specific indices\nlike current passing of generation number.  Just food for thoughs...\n\n[...]\n>>> In my local testing, I used the following Git commands on the\n>>> Linux repository in three modes: HEAD~1 with no commit-graph,\n>>> HEAD~1 with a commit-graph, and HEAD with a commit-graph. This\n>>> allows comparing the benefits we get from parsing commits from\n>>> the commit-graph and then again the benefits we get by\n>>> restricting the set of commits we walk.\n>>>\n>>> Test: git rev-list --topo-order -100 HEAD\n>>> HEAD~1, no commit-graph: 6.80 s\n>>> HEAD~1, w/ commit-graph: 0.77 s\n>>>    HEAD, w/ commit-graph: 0.02 s\n>>>\n>>> Test: git rev-list --topo-order -100 HEAD -- tools\n>>> HEAD~1, no commit-graph: 9.63 s\n>>> HEAD~1, w/ commit-graph: 6.06 s\n>>>    HEAD, w/ commit-graph: 0.06 s\n>>>\n>>> This speedup is due to a few things. First, the new generation-\n>>> number-enabled algorithm walks commits on order of the number of\n>>> results output (subject to some branching structure expectations).\n>>> Since we limit to 100 results, we are running a query similar to\n>>> filling a single page of results. Second, when specifying a path,\n>>> we must parse the root tree object for each commit we walk. The\n>>> previous benefits from the commit-graph are entirely from reading\n>>> the commit-graph instead of parsing commits. Since we need to\n>>> parse trees for the same number of commits as before, we slow\n>>> down significantly from the non-path-based query.\n>>>\n>>> For the test above, I specifically selected a path that is changed\n>>> frequently, including by merge commits. A less-frequently-changed\n>>> path (such as 'README') has similar end-to-end time since we need\n>>> to walk the same number of commits (before determining we do not\n>>> have 100 hits). However, get the benefit that the output is\n>>> presented to the user as it is discovered, much the same as a\n>>> normal 'git log' command (no '--topo-order'). This is an improved\n>>> user experience, even if the command has the same runtime.\n>>>\n>> First, do I understand it correctly that in first case the gains from\n>> new algorithms are so slim because with commit-graph file and no path\n>> limiting we don't hit repository anyway; we walk less commits, but\n>> reading commit data from commit-graph file is fast/\n>\n> If you mean 0.77s to 0.02s is \"slim\" then yes, it is because the\n> commit-graph command already made a full walk of the commit history\n> faster. (I'm only poking at this because the _relative_ improvement is\n> significant, even if the command was already sub-second.)\n\nFirst, you didn't provide us with percentages, i.e. relative improvement\n(and I am lazy).  Second, 0.02s can be within the margin of error,\ndepending on how it is measured, and how stable this measurement is.\n\n>> Second, I wonder if there is some easy way to perform automatic latency\n>> tests, i.e. how fast does Git show the first page of output...\n>\n> I have talked with Jeff Hostetler about this, to see if we can have a\n> \"time to first page\" traced with trace2, but we don't seem to have\n> access to that information within Git. We would need to insert it into\n> the pager. The \"-100\" is used instead.\n\nPerhaps another solution to the problem of \"first page of output\"\nlatency tests could be feasible, namely create a helper test-pager-1p\n\"pager\" program that would automatically quit after first page of\noutput; or perhaps even one that benchmarks each page of output\nautomatically.\n\nThere exists 'pv' (pipe viewer) program for pipes, so I think it would\nbe possible to do equivalent, but as a pager.\n\n[...]\n>>> +static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n>>> +{\n>>> +\tif (c->object.flags & flag)\n>>> +\t\treturn;\n>>> +\n>>> +\tc->object.flags |= flag;\n>>> +\tprio_queue_put(q, c);\n>>> +}\n>>\n>> This is an independent change, though I see that it is quite specific\n>> (as opposed to quite generic prio_queue_peek() operation added earlier\n>> in this series), so it does not make much sense as standalone change.\n>>\n>> It inserts commit into priority queue only if it didn't have flags set,\n>> and sets the flag (so we won't add it to the queue again, not without\n>> unsetting the flag), am I correct?\n>\n> Yes, this pattern of using a flag to avoid duplicate entries in the\n> priority queue appears in multiple walks. It wasn't needed before. We\n> call it four times in the code below.\n\n>> I guess that we use test_flag_and_insert() instead of prio_queue_put()\n>> to avoid duplicate entries in the queue.  I think the queue is initially\n>> populated with the starting commits, but those need not to be\n>> unreachable from each other, and walking down parents we can encounter\n>> starting commit already in the queue.  Am I correct?\n>\n> We can also reach commits in multiple ways, so the initial conditions\n> are not the only ways to insert duplicates.\n\nRight.\n\n[...]\n>>> +\tif (c->object.flags & UNINTERESTING)\n>>> +\t\tmark_parents_uninteresting(c);\n>>> +\n>>> +\tfor (p = c->parents; p; p = p->next)\n>>> +\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n>>\n>> Do we need to insert parents to the queue even if they were marked\n>> UNINTERESTING?\n>\n> We need to propagate the UNINTERESTING flag to our parents. That\n> propagation happens in process_parents().\n\nI think I understand.  We need to propagate UNINTERESTING flag down the\nchain, isn't it?\n\n[...]\n>>>     static void init_topo_walk(struct rev_info *revs)\n>>>   {\n>>>   \tstruct topo_walk_info *info;\n>>> +\tstruct commit_list *list;\n>>>   \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n>>>   \tinfo = revs->topo_walk_info;\n>>>   \tmemset(info, 0, sizeof(struct topo_walk_info));\n>>>   -\tlimit_list(revs);\n>>> -\tsort_in_topological_order(&revs->commits, revs->sort_order);\n>>> +\tinit_indegree_slab(&info->indegree);\n>>> +\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n>>> +\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n>>> +\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n>>\n>> Why this memset uses '\\0' as a filler value and not 0?  The queues are\n>> not strings [and you use 0 in other places].\n\nI think you missed answering about this issue.\n\n[...]\n>> Looks all right.\n>\n> Thanks for taking the time on this huge patch!\n\nYou are welcome.\n\n-- \nJakub Narębski\n"},{"id":"361648","messageId":"8636ss5swm.fsf@gmail.com","threadId":"49229","inReplyTo":"xmqqk1m6ig52.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-10-26T20:56:09Z","receivedAt":"2018-10-26T20:56:15Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Derrick Stolee <stolee@gmail.com> writes:\n>\n>>     time git log --topo-order -10 master >/dev/null\n>>\n>>     time git log --topo-order -10 maint..master >/dev/null\n>>\n>> I get 0.39s for the first call and 0.01s for the second. (Note: I\n>> specified \"-10\" to ensure we are only writing 10 commits and the\n>> output size does not factor into the time.) This is because the first\n>> walks the entire history, while the second uses the heuristic walk to\n>> identify a much smaller subgraph that the topo-order algorithm uses.\n>\n> The algorithm can be fooled by skewed timestamps (i.e. that SLOP\n> thing tries to work around), but is helped by being able to leave\n> early, and it will give us the correct answer as long as there is no\n> timestamp inversion.\n>\n> But monotonically increasing \"timestamp\" without inversion is what\n> we invented \"generation numbers\" for, no?  When there is no\n> timestamp inversion, would a walk based on commit timestamps walk\n> smaller set than a walk based on commit generation numbers?\n\nThe problem, as far as I understand it, is in B..A case, to be more\nexact with the walk from the exclusion set.  There can be cases when\nthere are two paths from the commit in the exclusion set, and sorting by\ngeneration number will always walk the longer one, while date-order\nbased heuristic walk traverses smaller subgraph.\n\nThis problem was described in \"[PATCH 1/1] commit: don't use generation\nnumbers if not needed\" [1]; relevant fragment cited below:\n\nDS> For instance, computing the merge-base between consecutive versions of\nDS> the Linux kernel has no effect for versions after v4.9, but 'git\nDS> merge-base v4.8 v4.9' presents a performance regression [...]\nDS> \nDS> The topology of this case can be described in a simplified way\nDS> here:\nDS> \nDS>   v4.9\nDS>    |  \\\nDS>    |   \\\nDS>   v4.8  \\\nDS>    | \\   \\\nDS>    |  \\   |\nDS>   ...  A  B\nDS>    |  /  /\nDS>    | /  /\nDS>    |/__/\nDS>    C\nDS> \nDS> Here, the \"...\" means \"a very long line of commits\". By generation\nDS> number, A and B have generation one more than C. However, A and B\nDS> have commit date higher than most of the commits reachable from\nDS> v4.8. When the walk reaches v4.8, we realize that it has PARENT1\nDS> and PARENT2 flags, so everything it can reach is marked as STALE,\nDS> including A. B has only the PARENT1 flag, so is not STALE.\nDS> \nDS> When paint_down_to_common() is run using\nDS> compare_commits_by_commit_date, A and B are removed from the queue\nDS> early and C is inserted into the queue. At this point, C and the\nDS> rest of the queue entries are marked as STALE. The loop then\nDS> terminates.\nDS> \nDS> When paint_down_to_common() is run using\nDS> compare_commits_by_gen_then_commit_date, B is removed from the\nDS> queue only after the many commits reachable from v4.8 are explored.\nDS> This causes the loop to run longer. The reason for this regression\nDS> is simple: the queue order is intended to not explore a commit\nDS> until everything that _could_ reach that commit is explored. From\nDS> the information gathered by the original ordering, we have no\nDS> guarantee that there is not a commit D reachable from v4.8 that\nDS> can also reach B. We gained absolute correctness in exchange for\nDS> a performance regression.\n\n[1]: https://public-inbox.org/git/efa3720fb40638e5d61c6130b55e3348d8e4339e.1535633886.git.gitgitgadget@gmail.com/T/#u\n\n>> Just as before, by using this algorithm for the B..A case, we are\n>> adding an extra restriction on the algorithm: always be correct. This\n>> results in us walking a larger set (everything reachable from B or A\n>> with generation number at least the smallest generation of a commit\n>> reachable from only one).\n>>\n>> I believe this can be handled by using a smarter generation number\n>> (one that relies on commit date as a heuristic, but still have enough\n>> information to guarantee topological relationships), and I've already\n>> started testing a few of these directions.\n>\n> Good to hear.\n\nI'm not sure if you can enhance generation numbers, or rather using /\nsorting of generation numbers in such way.\n\nWith generation numbers, if you have two possible paths to the merge\ncommit, and one is much longer than the other, then the commits on this\nlonger path will have larger generation numbers.\n\n\n        0     1     2     3     4     5     6     7       = gen(c) - gen(B)\n    --- B<----*<----*<----*<----*<----*<----*<----M ---\n        ^                                        /\n         \\--------------------------x<--------x</         = gen(c) - gen(B)\n                                    1         2\n\n    ===================== commit date ======================>\n\nUsing priority-queue which selects max generation number would then\nalways walk the longer path (or walk longer path first).\n\nThis does not matter for the walk from the inclusive set (A in B..A), as\nwe want to walk all commits anyway; we walk both paths, it does not\nmatter which we walk first (it may change the unimportant parts of\ntopological sort, though).\n\nFor the walk from the exclusive set (B in B..A), any path is good; we\ndon't need and do not want to walk all paths.  Sorting by generation\nnumbers will always choose the longer path, while sorting by commit date\nmay walk the \"shortcut\" first, thus marking commits reachable from the\ninclusive subset (from A) STALE earlier, thus earlier notice of\nall-stale, and end of walk.\n\nI think that in the case of walking from the exclusion subset, using\npositive-cut reachbility index would be more useful than trying to come\nup with \"smarter generation number\" (though I am not saying that it\nwould be impossible).  For example using reachability bitmap indices, or\npost-order in spanning tree (the latter descibed in [2]) to mark more\ncommits stale than just parents, may help more.\n\nActually, this is something independent from \"smarter generation\nnumbers\", and can only help...\n\n[2]: \"[RFC] Other chunks for commit-graph, part 2 - reachability indexes\"\n     https://public-inbox.org/git/86muxcuyod.fsf@gmail.com/\n\n-- \nJakub Narębski\n"},{"id":"362148","messageId":"xmqqpnvptlta.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/7] Use generation numbers for --topo-order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-01T05:21:21Z","receivedAt":"2018-11-01T05:21:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> This patch series performs a decently-sized refactoring of the revision-walk\n> machinery. Well, \"refactoring\" is probably the wrong word, as I don't\n> actually remove the old code. Instead, when we see certain options in the\n> 'rev_info' struct, we redirect the commit-walk logic to a new set of methods\n> that distribute the workload differently. By using generation numbers in the\n> commit-graph, we can significantly improve 'git log --graph' commands (and\n> the underlying 'git rev-list --topo-order').\n\nReview discussions seem to have petered out.  Would we merge this to\n'next' and start cooking, perhaps for the remainder of this cycle?\n"},{"id":"362178","messageId":"20181101134623.84055-1-dstolee@microsoft.com","threadId":"49229","inReplyTo":"pull.25.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v5 0/7] Use generation numbers for --topo-order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:16Z","receivedAt":"2018-11-01T13:46:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series performs a decently-sized refactoring of the\nrevision-walk machinery. Well, \"refactoring\" is probably the wrong word,\nas I don't actually remove the old code. Instead, when we see certain\noptions in the 'rev_info' struct, we redirect the commit-walk logic to\na new set of methods that distribute the workload differently. By using\ngeneration numbers in the commit-graph, we can significantly improve\n'git log --graph' commands (and the underlying 'git rev-list --topo-order').\n\nOn the Linux repository, I got the following performance results when\ncomparing to the previous version with or without a commit-graph:\n\n    Test: git rev-list --topo-order -100 HEAD\n    HEAD~1, no commit-graph: 6.80 s\n    HEAD~1, w/ commit-graph: 0.77 s\n      HEAD, w/ commit-graph: 0.02 s\n\n    Test: git rev-list --topo-order -100 HEAD -- tools\n    HEAD~1, no commit-graph: 9.63 s\n    HEAD~1, w/ commit-graph: 6.06 s\n      HEAD, w/ commit-graph: 0.06 s\n\nIf you want to read this series but are unfamiliar with the commit-graph\nand generation numbers, then I recommend reading\n`Documentation/technical/commit-graph.txt` or a blog post [1] I wrote on\nthe subject. In particular, the three-part walk described in \"revision.c:\nrefactor basic topo-order logic\" is present (but underexplained) as an\nanimated PNG [2].\n\n**UPDATED** Now that we have had some review and some dogfooding, I'm\nremoving the paragraph I had here about \"RFC quality\". I think this is\nready to merge!\n\nOne notable case that is not included in this series is the case of a\nhistory comparison such as 'git rev-list --topo-order A..B'. The existing\ncode in limit_list() has ways to cut the walk short when all pending\ncommits are UNINTERESTING. Since this code depends on commit_list instead\nof the prio_queue we are using here, I chose to leave it untouched for now.\nWe can revisit it in a separate series later. Since handle_commit() turns\non revs->limited when a commit is UNINTERESTING, we do not hit the new\ncode in this case. Removing this 'revs->limited = 1;' line yields correct\nresults, but the performance can be worse.\n\n**UPDATED** See the discussion about Generation Number V2 [4] for more\non this topic.\n\nChanges in V5: Thanks Jakub for feedback on the huge commit! I think\nI've responded to all the code feedback. See the range-diff at the\nend of this cover-page.\n\nThanks,\n-Stolee\n\n[1] https://blogs.msdn.microsoft.com/devops/2018/07/09/supercharging-the-git-commit-graph-iii-generations/\n   Supercharging the Git Commit Graph III: Generations and Graph Algorithms\n\n[2] https://msdnshared.blob.core.windows.net/media/2018/06/commit-graph-topo-order-b-a.png\n    Animation showing three-part walk\n\n[3] https://github.com/derrickstolee/git/tree/topo-order/test\n    A branch containing this series along with commits to compute commit-graph in entire test suite.\n\n[4] https://public-inbox.org/git/6367e30a-1b3a-4fe9-611b-d931f51effef@gmail.com/\n    [RFC] Generation Number v2\n\nNote: I'm not submitting this version via GitGitGadget because it's\ncurrently struggling with how to handle a PR in a conflict state.\nThe new flags in revision.h have a conflict with recent changes in\nmaster.\n\nDerrick Stolee (7):\n  prio-queue: add 'peek' operation\n  test-reach: add run_three_modes method\n  test-reach: add rev-list tests\n  revision.c: begin refactoring --topo-order logic\n  commit/revisions: bookkeeping before refactoring\n  revision.c: generation-based topo-order algorithm\n  t6012: make rev-list tests more interesting\n\n commit.c                     |   9 +-\n commit.h                     |   7 +\n object.h                     |   4 +-\n prio-queue.c                 |   9 ++\n prio-queue.h                 |   6 +\n revision.c                   | 243 +++++++++++++++++++++++++++++++++--\n revision.h                   |   6 +\n t/helper/test-prio-queue.c   |  26 ++--\n t/t0009-prio-queue.sh        |  14 ++\n t/t6012-rev-list-simplify.sh |  45 +++++--\n t/t6600-test-reach.sh        |  96 +++++++++++++-\n 11 files changed, 426 insertions(+), 39 deletions(-)\n\n\nbase-commit: 2d3b1c576c85b7f5db1f418907af00ab88e0c303\n-- \n2.19.1.542.gc4df23f792\n\n-->8--\n\n1:  2358cfd5ed = 1:  7c75a56505 prio-queue: add 'peek' operation\n2:  3a4b68e479 = 2:  686c4370de test-reach: add run_three_modes method\n3:  12a3f6d367 = 3:  7410c00982 test-reach: add rev-list tests\n4:  cd9eef9688 = 4:  5439e11e37 revision.c: begin refactoring --topo-order logic\n5:  f3e291665d ! 5:  71554deb9b commit/revisions: bookkeeping before refactoring\n    @@ -9,8 +9,8 @@\n            compare_commits_by_author_date() in revision.c. These are used\n            currently by sort_in_topological_order() in commit.c.\n     \n    -    2. Moving these methods to commit.h requires adding the author_slab\n    -       definition to commit.h.\n    +    2. Moving these methods to commit.h requires adding an author_date_slab\n    +       declaration to commit.h. Consumers will need their own implementation.\n     \n         3. The add_parents_to_list() method in revision.c performs logic\n            around the UNINTERESTING flag and other special cases depending\n    @@ -31,8 +31,7 @@\n      define_commit_slab(indegree_slab, int);\n      \n     -/* record author-date for each commit object */\n    --define_commit_slab(author_date_slab, timestamp_t);\n    -+implement_shared_commit_slab(author_date_slab, timestamp_t);\n    + define_commit_slab(author_date_slab, timestamp_t);\n      \n     -static void record_author_date(struct author_date_slab *author_date,\n     -\t\t\t       struct commit *commit)\n    @@ -69,8 +68,7 @@\n      extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n      \n     +/* record author-date for each commit object */\n    -+define_shared_commit_slab(author_date_slab, timestamp_t);\n    -+\n    ++struct author_date_slab;\n     +void record_author_date(struct author_date_slab *author_date,\n     +\t\t\tstruct commit *commit);\n     +\n6:  aa0bb2221d ! 6:  84c142e0bc revision.c: generation-based topo-order algorithm\n    @@ -195,6 +195,7 @@\n      \n     -struct topo_walk_info {};\n     +define_commit_slab(indegree_slab, int);\n    ++define_commit_slab(author_date_slab, timestamp_t);\n     +\n     +struct topo_walk_info {\n     +\tuint32_t min_generation;\n    @@ -243,12 +244,12 @@\n     +}\n     +\n     +static void explore_to_depth(struct rev_info *revs,\n    -+\t\t\t     uint32_t gen)\n    ++\t\t\t     uint32_t gen_cutoff)\n     +{\n     +\tstruct topo_walk_info *info = revs->topo_walk_info;\n     +\tstruct commit *c;\n     +\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n    -+\t       c->generation >= gen)\n    ++\t       c->generation >= gen_cutoff)\n     +\t\texplore_walk_step(revs);\n     +}\n     +\n    @@ -266,9 +267,6 @@\n     +\n     +\texplore_to_depth(revs, c->generation);\n     +\n    -+\tif (parse_commit_gently(c, 1) < 0)\n    -+\t\treturn;\n    -+\n     +\tfor (p = c->parents; p; p = p->next) {\n     +\t\tstruct commit *parent = p->item;\n     +\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n    @@ -285,12 +283,13 @@\n     +\t}\n     +}\n     +\n    -+static void compute_indegrees_to_depth(struct rev_info *revs)\n    ++static void compute_indegrees_to_depth(struct rev_info *revs,\n    ++\t\t\t\t       uint32_t gen_cutoff)\n     +{\n     +\tstruct topo_walk_info *info = revs->topo_walk_info;\n     +\tstruct commit *c;\n     +\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n    -+\t       c->generation >= info->min_generation)\n    ++\t       c->generation >= gen_cutoff)\n     +\t\tindegree_walk_step(revs);\n     +}\n      \n    @@ -305,9 +304,9 @@\n     -\tlimit_list(revs);\n     -\tsort_in_topological_order(&revs->commits, revs->sort_order);\n     +\tinit_indegree_slab(&info->indegree);\n    -+\tmemset(&info->explore_queue, '\\0', sizeof(info->explore_queue));\n    -+\tmemset(&info->indegree_queue, '\\0', sizeof(info->indegree_queue));\n    -+\tmemset(&info->topo_queue, '\\0', sizeof(info->topo_queue));\n    ++\tmemset(&info->explore_queue, 0, sizeof(info->explore_queue));\n    ++\tmemset(&info->indegree_queue, 0, sizeof(info->indegree_queue));\n    ++\tmemset(&info->topo_queue, 0, sizeof(info->topo_queue));\n     +\n     +\tswitch (revs->sort_order) {\n     +\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n    @@ -329,23 +328,22 @@\n     +\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n     +\tfor (list = revs->commits; list; list = list->next) {\n     +\t\tstruct commit *c = list->item;\n    -+\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n    -+\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n     +\n     +\t\tif (parse_commit_gently(c, 1))\n     +\t\t\tcontinue;\n    ++\n    ++\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n    ++\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n    ++\n     +\t\tif (c->generation < info->min_generation)\n     +\t\t\tinfo->min_generation = c->generation;\n    -+\t}\n     +\n    -+\tfor (list = revs->commits; list; list = list->next) {\n    -+\t\tstruct commit *c = list->item;\n     +\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n     +\n     +\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n     +\t\t\trecord_author_date(&info->author_date, c);\n     +\t}\n    -+\tcompute_indegrees_to_depth(revs);\n    ++\tcompute_indegrees_to_depth(revs, info->min_generation);\n     +\n     +\tfor (list = revs->commits; list; list = list->next) {\n     +\t\tstruct commit *c = list->item;\n    @@ -385,9 +383,8 @@\n     +\tif (process_parents(revs, commit, NULL, NULL) < 0) {\n      \t\tif (!revs->ignore_missing_links)\n      \t\t\tdie(\"Failed to traverse parents of commit %s\",\n    --\t\t\t    oid_to_hex(&commit->object.oid));\n    -+\t\t\t\toid_to_hex(&commit->object.oid));\n    -+\t}\n    + \t\t\t    oid_to_hex(&commit->object.oid));\n    + \t}\n     +\n     +\tfor (p = commit->parents; p; p = p->next) {\n     +\t\tstruct commit *parent = p->item;\n    @@ -398,7 +395,7 @@\n     +\n     +\t\tif (parent->generation < info->min_generation) {\n     +\t\t\tinfo->min_generation = parent->generation;\n    -+\t\t\tcompute_indegrees_to_depth(revs);\n    ++\t\t\tcompute_indegrees_to_depth(revs, info->min_generation);\n     +\t\t}\n     +\n     +\t\tpi = indegree_slab_at(&info->indegree, parent);\n    @@ -409,9 +406,10 @@\n     +\n     +\t\tif (revs->first_parent_only)\n     +\t\t\treturn;\n    - \t}\n    ++\t}\n      }\n      \n    + int prepare_revision_walk(struct rev_info *revs)\n     \n      diff --git a/revision.h b/revision.h\n      --- a/revision.h\n7:  a21febe112 = 7:  5479087812 t6012: make rev-list tests more interesting\n"},{"id":"362179","messageId":"20181101134623.84055-2-dstolee@microsoft.com","threadId":"49229","inReplyTo":"20181101134623.84055-1-dstolee@microsoft.com","subject":"[PATCH v5 1/7] prio-queue: add 'peek' operation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:17Z","receivedAt":"2018-11-01T13:46:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When consuming a priority queue, it can be convenient to inspect\nthe next object that will be dequeued without actually dequeueing\nit. Our existing library did not have such a 'peek' operation, so\nadd it as prio_queue_peek().\n\nAdd a reference-level comparison in t/helper/test-prio-queue.c\nso this method is exercised by t0009-prio-queue.sh. Further, add\na test that checks the behavior when the compare function is NULL\n(i.e. the queue becomes a stack).\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n prio-queue.c               |  9 +++++++++\n prio-queue.h               |  6 ++++++\n t/helper/test-prio-queue.c | 26 ++++++++++++++++++--------\n t/t0009-prio-queue.sh      | 14 ++++++++++++++\n 4 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/prio-queue.c b/prio-queue.c\nindex a078451872..d3f488cb05 100644\n--- a/prio-queue.c\n+++ b/prio-queue.c\n@@ -85,3 +85,12 @@ void *prio_queue_get(struct prio_queue *queue)\n \t}\n \treturn result;\n }\n+\n+void *prio_queue_peek(struct prio_queue *queue)\n+{\n+\tif (!queue->nr)\n+\t\treturn NULL;\n+\tif (!queue->compare)\n+\t\treturn queue->array[queue->nr - 1].data;\n+\treturn queue->array[0].data;\n+}\ndiff --git a/prio-queue.h b/prio-queue.h\nindex d030ec9dd6..682e51867a 100644\n--- a/prio-queue.h\n+++ b/prio-queue.h\n@@ -46,6 +46,12 @@ extern void prio_queue_put(struct prio_queue *, void *thing);\n  */\n extern void *prio_queue_get(struct prio_queue *);\n \n+/*\n+ * Gain access to the \"thing\" that would be returned by\n+ * prio_queue_get, but do not remove it from the queue.\n+ */\n+extern void *prio_queue_peek(struct prio_queue *);\n+\n extern void clear_prio_queue(struct prio_queue *);\n \n /* Reverse the LIFO elements */\ndiff --git a/t/helper/test-prio-queue.c b/t/helper/test-prio-queue.c\nindex 9807b649b1..5bc9c46ea5 100644\n--- a/t/helper/test-prio-queue.c\n+++ b/t/helper/test-prio-queue.c\n@@ -22,14 +22,24 @@ int cmd__prio_queue(int argc, const char **argv)\n \tstruct prio_queue pq = { intcmp };\n \n \twhile (*++argv) {\n-\t\tif (!strcmp(*argv, \"get\"))\n-\t\t\tshow(prio_queue_get(&pq));\n-\t\telse if (!strcmp(*argv, \"dump\")) {\n-\t\t\tint *v;\n-\t\t\twhile ((v = prio_queue_get(&pq)))\n-\t\t\t       show(v);\n-\t\t}\n-\t\telse {\n+\t\tif (!strcmp(*argv, \"get\")) {\n+\t\t\tvoid *peek = prio_queue_peek(&pq);\n+\t\t\tvoid *get = prio_queue_get(&pq);\n+\t\t\tif (peek != get)\n+\t\t\t\tBUG(\"peek and get results do not match\");\n+\t\t\tshow(get);\n+\t\t} else if (!strcmp(*argv, \"dump\")) {\n+\t\t\tvoid *peek;\n+\t\t\tvoid *get;\n+\t\t\twhile ((peek = prio_queue_peek(&pq))) {\n+\t\t\t\tget = prio_queue_get(&pq);\n+\t\t\t\tif (peek != get)\n+\t\t\t\t\tBUG(\"peek and get results do not match\");\n+\t\t\t\tshow(get);\n+\t\t\t}\n+\t\t} else if (!strcmp(*argv, \"stack\")) {\n+\t\t\tpq.compare = NULL;\n+\t\t} else {\n \t\t\tint *v = malloc(sizeof(*v));\n \t\t\t*v = atoi(*argv);\n \t\t\tprio_queue_put(&pq, v);\ndiff --git a/t/t0009-prio-queue.sh b/t/t0009-prio-queue.sh\nindex e56dfce668..3941ad2528 100755\n--- a/t/t0009-prio-queue.sh\n+++ b/t/t0009-prio-queue.sh\n@@ -47,4 +47,18 @@ test_expect_success 'notice empty queue' '\n \ttest_cmp expect actual\n '\n \n+cat >expect <<'EOF'\n+3\n+2\n+6\n+4\n+5\n+1\n+8\n+EOF\n+test_expect_success 'stack order' '\n+\ttest-tool prio-queue stack 8 1 5 4 6 2 3 dump >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\nbase-commit: 2d3b1c576c85b7f5db1f418907af00ab88e0c303\n-- \n2.19.1.542.gc4df23f792\n\n"},{"id":"362180","messageId":"20181101134623.84055-3-dstolee@microsoft.com","threadId":"49229","inReplyTo":"20181101134623.84055-1-dstolee@microsoft.com","subject":"[PATCH v5 2/7] test-reach: add run_three_modes method","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:18Z","receivedAt":"2018-11-01T13:46:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'test_three_modes' method assumes we are using the 'test-tool\nreach' command for our test. However, we may want to use the data\nshape of our commit graph and the three modes (no commit-graph,\nfull commit-graph, partial commit-graph) for other git commands.\n\nSplit test_three_modes to be a simple translation on a more general\nrun_three_modes method that executes the given command and tests\nthe actual output to the expected output.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex d139a00d1d..9d65b8b946 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -53,18 +53,22 @@ test_expect_success 'setup' '\n \tgit config core.commitGraph true\n '\n \n-test_three_modes () {\n+run_three_modes () {\n \ttest_when_finished rm -rf .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-full .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual &&\n \tcp commit-graph-half .git/objects/info/commit-graph &&\n-\ttest-tool reach $1 <input >actual &&\n+\t\"$@\" <input >actual &&\n \ttest_cmp expect actual\n }\n \n+test_three_modes () {\n+\trun_three_modes test-tool reach \"$@\"\n+}\n+\n test_expect_success 'ref_newer:miss' '\n \tcat >input <<-\\EOF &&\n \tA:commit-5-7\n-- \n2.19.1.542.gc4df23f792\n\n"},{"id":"362181","messageId":"20181101134623.84055-4-dstolee@microsoft.com","threadId":"49229","inReplyTo":"20181101134623.84055-1-dstolee@microsoft.com","subject":"[PATCH v5 3/7] test-reach: add rev-list tests","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:19Z","receivedAt":"2018-11-01T13:46:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The rev-list command is critical to Git's functionality. Ensure it\nworks in the three commit-graph environments constructed in\nt6600-test-reach.sh. Here are a few important types of rev-list\noperations:\n\n* Basic: git rev-list --topo-order HEAD\n* Range: git rev-list --topo-order compare..HEAD\n* Ancestry: git rev-list --topo-order --ancestry-path compare..HEAD\n* Symmetric Difference: git rev-list --topo-order compare...HEAD\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6600-test-reach.sh | 84 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 84 insertions(+)\n\ndiff --git a/t/t6600-test-reach.sh b/t/t6600-test-reach.sh\nindex 9d65b8b946..288f703b7b 100755\n--- a/t/t6600-test-reach.sh\n+++ b/t/t6600-test-reach.sh\n@@ -243,4 +243,88 @@ test_expect_success 'commit_contains:miss' '\n \ttest_three_modes commit_contains --tag\n '\n \n+test_expect_success 'rev-list: basic topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 commit-3-3 commit-2-3 commit-1-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 commit-3-2 commit-2-2 commit-1-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-6-6\n+'\n+\n+test_expect_success 'rev-list: first-parent topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 commit-3-1 commit-2-1 commit-1-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --first-parent --topo-order commit-6-6\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 commit-2-6 commit-1-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 commit-2-5 commit-1-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 commit-2-4 commit-1-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-3..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-8..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: first-parent range topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 \\\n+\t\tcommit-6-5 \\\n+\t\tcommit-6-4 \\\n+\t\tcommit-6-3 \\\n+\t\tcommit-6-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --first-parent --topo-order commit-3-8..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: ancestry-path topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 commit-3-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 commit-3-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 commit-3-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order --ancestry-path commit-3-3..commit-6-6\n+'\n+\n+test_expect_success 'rev-list: symmetric difference topo-order' '\n+\tgit rev-parse \\\n+\t\tcommit-6-6 commit-5-6 commit-4-6 \\\n+\t\tcommit-6-5 commit-5-5 commit-4-5 \\\n+\t\tcommit-6-4 commit-5-4 commit-4-4 \\\n+\t\tcommit-6-3 commit-5-3 commit-4-3 \\\n+\t\tcommit-6-2 commit-5-2 commit-4-2 \\\n+\t\tcommit-6-1 commit-5-1 commit-4-1 \\\n+\t\tcommit-3-8 commit-2-8 commit-1-8 \\\n+\t\tcommit-3-7 commit-2-7 commit-1-7 \\\n+\t>expect &&\n+\trun_three_modes git rev-list --topo-order commit-3-8...commit-6-6\n+'\n+\n test_done\n-- \n2.19.1.542.gc4df23f792\n\n"},{"id":"362182","messageId":"20181101134623.84055-5-dstolee@microsoft.com","threadId":"49229","inReplyTo":"20181101134623.84055-1-dstolee@microsoft.com","subject":"[PATCH v5 4/7] revision.c: begin refactoring --topo-order logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:20Z","receivedAt":"2018-11-01T13:46:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git rev-list --topo-order' and its kin, the topo_order\nsetting in struct rev_info implies the limited setting. This means\nthat the following things happen during prepare_revision_walk():\n\n* revs->limited implies we run limit_list() to walk the entire\n  reachable set. There are some short-cuts here, such as if we\n  perform a range query like 'git rev-list COMPARE..HEAD' and we\n  can stop limit_list() when all queued commits are uninteresting.\n\n* revs->topo_order implies we run sort_in_topological_order(). See\n  the implementation of that method in commit.c. It implies that\n  the full set of commits to order is in the given commit_list.\n\nThese two methods imply that a 'git rev-list --topo-order HEAD'\ncommand must walk the entire reachable set of commits _twice_ before\nreturning a single result.\n\nIf we have a commit-graph file with generation numbers computed, then\nthere is a better way. This patch introduces some necessary logic\nredirection when we are in this situation.\n\nIn v2.18.0, the commit-graph file contains zero-valued bytes in the\npositions where the generation number is stored in v2.19.0 and later.\nThus, we use generation_numbers_enabled() to check if the commit-graph\nis available and has non-zero generation numbers.\n\nWhen setting revs->limited only because revs->topo_order is true,\nonly do so if generation numbers are not available. There is no\nreason to use the new logic as it will behave similarly when all\ngeneration numbers are INFINITY or ZERO.\n\nIn prepare_revision_walk(), if we have revs->topo_order but not\nrevs->limited, then we trigger the new logic. It breaks the logic\ninto three pieces, to fit with the existing framework:\n\n1. init_topo_walk() fills a new struct topo_walk_info in the rev_info\n   struct. We use the presence of this struct as a signal to use the\n   new methods during our walk. In this patch, this method simply\n   calls limit_list() and sort_in_topological_order(). In the future,\n   this method will set up a new data structure to perform that logic\n   in-line.\n\n2. next_topo_commit() provides get_revision_1() with the next topo-\n   ordered commit in the list. Currently, this simply pops the commit\n   from revs->commits.\n\n3. expand_topo_walk() provides get_revision_1() with a way to signal\n   walking beyond the latest commit. Currently, this calls\n   add_parents_to_list() exactly like the old logic.\n\nWhile this commit presents method redirection for performing the\nexact same logic as before, it allows the next commit to focus only\non the new logic.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 42 ++++++++++++++++++++++++++++++++++++++----\n revision.h |  4 ++++\n 2 files changed, 42 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex e18bd530e4..2dcde8a8ac 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -25,6 +25,7 @@\n #include \"worktree.h\"\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n+#include \"commit-graph.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2454,7 +2455,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \tif (revs->diffopt.objfind)\n \t\trevs->simplify_history = 0;\n \n-\tif (revs->topo_order)\n+\tif (revs->topo_order && !generation_numbers_enabled(the_repository))\n \t\trevs->limited = 1;\n \n \tif (revs->prune_data.nr) {\n@@ -2892,6 +2893,33 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n+struct topo_walk_info {};\n+\n+static void init_topo_walk(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info;\n+\trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n+\tinfo = revs->topo_walk_info;\n+\tmemset(info, 0, sizeof(struct topo_walk_info));\n+\n+\tlimit_list(revs);\n+\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+}\n+\n+static struct commit *next_topo_commit(struct rev_info *revs)\n+{\n+\treturn pop_commit(&revs->commits);\n+}\n+\n+static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n+{\n+\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\t\tif (!revs->ignore_missing_links)\n+\t\t\tdie(\"Failed to traverse parents of commit %s\",\n+\t\t\t    oid_to_hex(&commit->object.oid));\n+\t}\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -2928,11 +2956,13 @@ int prepare_revision_walk(struct rev_info *revs)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\n-\tif (revs->limited)\n+\tif (revs->limited) {\n \t\tif (limit_list(revs) < 0)\n \t\t\treturn -1;\n-\tif (revs->topo_order)\n-\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t\tif (revs->topo_order)\n+\t\t\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\t} else if (revs->topo_order)\n+\t\tinit_topo_walk(revs);\n \tif (revs->line_level_traverse)\n \t\tline_log_filter(revs);\n \tif (revs->simplify_merges)\n@@ -3257,6 +3287,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\tif (revs->reflog_info)\n \t\t\tcommit = next_reflog_entry(revs->reflog_info);\n+\t\telse if (revs->topo_walk_info)\n+\t\t\tcommit = next_topo_commit(revs);\n \t\telse\n \t\t\tcommit = pop_commit(&revs->commits);\n \n@@ -3278,6 +3310,8 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \n \t\t\tif (revs->reflog_info)\n \t\t\t\ttry_to_simplify_commit(revs, commit);\n+\t\t\telse if (revs->topo_walk_info)\n+\t\t\t\texpand_topo_walk(revs, commit);\n \t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n \t\t\t\tif (!revs->ignore_missing_links)\n \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\ndiff --git a/revision.h b/revision.h\nindex 2b30ac270d..fd4154ff75 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -56,6 +56,8 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct topo_walk_info;\n+\n struct rev_info {\n \t/* Starting list */\n \tstruct commit_list *commits;\n@@ -245,6 +247,8 @@ struct rev_info {\n \tconst char *break_bar;\n \n \tstruct revision_sources *sources;\n+\n+\tstruct topo_walk_info *topo_walk_info;\n };\n \n int ref_excluded(struct string_list *, const char *path);\n-- \n2.19.1.542.gc4df23f792\n\n"},{"id":"362183","messageId":"20181101134623.84055-6-dstolee@microsoft.com","threadId":"49229","inReplyTo":"20181101134623.84055-1-dstolee@microsoft.com","subject":"[PATCH v5 5/7] commit/revisions: bookkeeping before refactoring","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:21Z","receivedAt":"2018-11-01T13:46:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"There are a few things that need to move around a little before\nmaking a big refactoring in the topo-order logic:\n\n1. We need access to record_author_date() and\n   compare_commits_by_author_date() in revision.c. These are used\n   currently by sort_in_topological_order() in commit.c.\n\n2. Moving these methods to commit.h requires adding an author_date_slab\n   declaration to commit.h. Consumers will need their own implementation.\n\n3. The add_parents_to_list() method in revision.c performs logic\n   around the UNINTERESTING flag and other special cases depending\n   on the struct rev_info. Allow this method to ignore a NULL 'list'\n   parameter, as we will not be populating the list for our walk.\n   Also rename the method to the slightly more generic name\n   process_parents() to make clear that this method does more than\n   add to a list (and no list is required anymore).\n\nHelped-by: Jeff King <peff@peff.net>\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n commit.c   |  9 ++++-----\n commit.h   |  7 +++++++\n revision.c | 18 ++++++++++--------\n 3 files changed, 21 insertions(+), 13 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex d0f199e122..a025a0db60 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -655,11 +655,10 @@ struct commit *pop_commit(struct commit_list **stack)\n /* count number of children that have not been emitted */\n define_commit_slab(indegree_slab, int);\n \n-/* record author-date for each commit object */\n define_commit_slab(author_date_slab, timestamp_t);\n \n-static void record_author_date(struct author_date_slab *author_date,\n-\t\t\t       struct commit *commit)\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit)\n {\n \tconst char *buffer = get_commit_buffer(commit, NULL);\n \tstruct ident_split ident;\n@@ -684,8 +683,8 @@ static void record_author_date(struct author_date_slab *author_date,\n \tunuse_commit_buffer(commit, buffer);\n }\n \n-static int compare_commits_by_author_date(const void *a_, const void *b_,\n-\t\t\t\t\t  void *cb_data)\n+int compare_commits_by_author_date(const void *a_, const void *b_,\n+\t\t\t\t   void *cb_data)\n {\n \tconst struct commit *a = a_, *b = b_;\n \tstruct author_date_slab *author_date = cb_data;\ndiff --git a/commit.h b/commit.h\nindex 2b1a734388..ec5b9093ad 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -8,6 +8,7 @@\n #include \"gpg-interface.h\"\n #include \"string-list.h\"\n #include \"pretty.h\"\n+#include \"commit-slab.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n #define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n@@ -328,6 +329,12 @@ extern int remove_signature(struct strbuf *buf);\n  */\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n+/* record author-date for each commit object */\n+struct author_date_slab;\n+void record_author_date(struct author_date_slab *author_date,\n+\t\t\tstruct commit *commit);\n+\n+int compare_commits_by_author_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \ndiff --git a/revision.c b/revision.c\nindex 2dcde8a8ac..36458265a0 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -768,8 +768,8 @@ static void commit_list_insert_by_date_cached(struct commit *p, struct commit_li\n \t\t*cache = new_entry;\n }\n \n-static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n-\t\t    struct commit_list **list, struct commit_list **cache_ptr)\n+static int process_parents(struct rev_info *revs, struct commit *commit,\n+\t\t\t   struct commit_list **list, struct commit_list **cache_ptr)\n {\n \tstruct commit_list *parent = commit->parents;\n \tunsigned left_flag;\n@@ -808,7 +808,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\treturn 0;\n \t}\n@@ -847,7 +848,8 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n \t\t\tp->object.flags |= SEEN;\n-\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n+\t\t\tif (list)\n+\t\t\t\tcommit_list_insert_by_date_cached(p, list, cached_base, cache_ptr);\n \t\t}\n \t\tif (revs->first_parent_only)\n \t\t\tbreak;\n@@ -1091,7 +1093,7 @@ static int limit_list(struct rev_info *revs)\n \n \t\tif (revs->max_age != -1 && (commit->date < revs->max_age))\n \t\t\tobj->flags |= UNINTERESTING;\n-\t\tif (add_parents_to_list(revs, commit, &list, NULL) < 0)\n+\t\tif (process_parents(revs, commit, &list, NULL) < 0)\n \t\t\treturn -1;\n \t\tif (obj->flags & UNINTERESTING) {\n \t\t\tmark_parents_uninteresting(commit);\n@@ -2913,7 +2915,7 @@ static struct commit *next_topo_commit(struct rev_info *revs)\n \n static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n {\n-\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n \t\tif (!revs->ignore_missing_links)\n \t\t\tdie(\"Failed to traverse parents of commit %s\",\n \t\t\t    oid_to_hex(&commit->object.oid));\n@@ -2979,7 +2981,7 @@ static enum rewrite_result rewrite_one(struct rev_info *revs, struct commit **pp\n \tfor (;;) {\n \t\tstruct commit *p = *pp;\n \t\tif (!revs->limited)\n-\t\t\tif (add_parents_to_list(revs, p, &revs->commits, &cache) < 0)\n+\t\t\tif (process_parents(revs, p, &revs->commits, &cache) < 0)\n \t\t\t\treturn rewrite_one_error;\n \t\tif (p->object.flags & UNINTERESTING)\n \t\t\treturn rewrite_one_ok;\n@@ -3312,7 +3314,7 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \t\t\t\ttry_to_simplify_commit(revs, commit);\n \t\t\telse if (revs->topo_walk_info)\n \t\t\t\texpand_topo_walk(revs, commit);\n-\t\t\telse if (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0) {\n+\t\t\telse if (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n \t\t\t\tif (!revs->ignore_missing_links)\n \t\t\t\t\tdie(\"Failed to traverse parents of commit %s\",\n \t\t\t\t\t\toid_to_hex(&commit->object.oid));\n-- \n2.19.1.542.gc4df23f792\n\n"},{"id":"362184","messageId":"20181101134623.84055-7-dstolee@microsoft.com","threadId":"49229","inReplyTo":"20181101134623.84055-1-dstolee@microsoft.com","subject":"[PATCH v5 6/7] revision.c: generation-based topo-order algorithm","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:22Z","receivedAt":"2018-11-01T13:46:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The current --topo-order algorithm requires walking all\nreachable commits up front, topo-sorting them, all before\noutputting the first value. This patch introduces a new\nalgorithm which uses stored generation numbers to\nincrementally walk in topo-order, outputting commits as\nwe go. This can dramatically reduce the computation time\nto write a fixed number of commits, such as when limiting\nwith \"-n <N>\" or filling the first page of a pager.\n\nWhen running a command like 'git rev-list --topo-order HEAD',\nGit performed the following steps:\n\n1. Run limit_list(), which parses all reachable commits,\n   adds them to a linked list, and distributes UNINTERESTING\n   flags. If all unprocessed commits are UNINTERESTING, then\n   it may terminate without walking all reachable commits.\n   This does not occur if we do not specify UNINTERESTING\n   commits.\n\n2. Run sort_in_topological_order(), which is an implementation\n   of Kahn's algorithm. It first iterates through the entire\n   set of important commits and computes the in-degree of each\n   (plus one, as we use 'zero' as a special value here). Then,\n   we walk the commits in priority order, adding them to the\n   priority queue if and only if their in-degree is one. As\n   we remove commits from this priority queue, we decrement the\n   in-degree of their parents.\n\n3. While we are peeling commits for output, get_revision_1()\n   uses pop_commit on the full list of commits computed by\n   sort_in_topological_order().\n\nIn the new algorithm, these three steps correspond to three\ndifferent commit walks. We run these walks simultaneously,\nand advance each only as far as necessary to satisfy the\nrequirements of the 'higher order' walk. We know when we can\npause each walk by using generation numbers from the commit-\ngraph feature.\n\nRecall that the generation number of a commit satisfies:\n\n* If the commit has at least one parent, then the generation\n  number is one more than the maximum generation number among\n  its parents.\n\n* If the commit has no parent, then the generation number is one.\n\nThere are two special generation numbers:\n\n* GENERATION_NUMBER_INFINITY: this value is 0xffffffff and\n  indicates that the commit is not stored in the commit-graph and\n  the generation number was not previously calculated.\n\n* GENERATION_NUMBER_ZERO: this value (0) is a special indicator\n  to say that the commit-graph was generated by a version of Git\n  that does not compute generation numbers (such as v2.18.0).\n\nSince we use generation_numbers_enabled() before using the new\nalgorithm, we do not need to worry about GENERATION_NUMBER_ZERO.\nHowever, the existence of GENERATION_NUMBER_INFINITY implies the\nfollowing weaker statement than the usual we expect from\ngeneration numbers:\n\n    If A and B are commits with generation numbers gen(A) and\n    gen(B) and gen(A) < gen(B), then A cannot reach B.\n\nThus, we will walk in each of our stages until the \"maximum\nunexpanded generation number\" is strictly lower than the\ngeneration number of a commit we are about to use.\n\nThe walks are as follows:\n\n1. EXPLORE: using the explore_queue priority queue (ordered by\n   maximizing the generation number), parse each reachable\n   commit until all commits in the queue have generation\n   number strictly lower than needed. During this walk, update\n   the UNINTERESTING flags as necessary.\n\n2. INDEGREE: using the indegree_queue priority queue (ordered\n   by maximizing the generation number), add one to the in-\n   degree of each parent for each commit that is walked. Since\n   we walk in order of decreasing generation number, we know\n   that discovering an in-degree value of 0 means the value for\n   that commit was not initialized, so should be initialized to\n   two. (Recall that in-degree value \"1\" is what we use to say a\n   commit is ready for output.) As we iterate the parents of a\n   commit during this walk, ensure the EXPLORE walk has walked\n   beyond their generation numbers.\n\n3. TOPO: using the topo_queue priority queue (ordered based on\n   the sort_order given, which could be commit-date, author-\n   date, or typical topo-order which treats the queue as a LIFO\n   stack), remove a commit from the queue and decrement the\n   in-degree of each parent. If a parent has an in-degree of\n   one, then we add it to the topo_queue. Before we decrement\n   the in-degree, however, ensure the INDEGREE walk has walked\n   beyond that generation number.\n\nThe implementations of these walks are in the following methods:\n\n* explore_walk_step and explore_to_depth\n* indegree_walk_step and compute_indegrees_to_depth\n* next_topo_commit and expand_topo_walk\n\nThese methods have some patterns that may seem strange at first,\nbut they are probably carry-overs from their equivalents in\nlimit_list and sort_in_topological_order.\n\nOne thing that is missing from this implementation is a proper\nway to stop walking when the entire queue is UNINTERESTING, so\nthis implementation is not enabled by comparisions, such as in\n'git rev-list --topo-order A..B'. This can be updated in the\nfuture.\n\nIn my local testing, I used the following Git commands on the\nLinux repository in three modes: HEAD~1 with no commit-graph,\nHEAD~1 with a commit-graph, and HEAD with a commit-graph. This\nallows comparing the benefits we get from parsing commits from\nthe commit-graph and then again the benefits we get by\nrestricting the set of commits we walk.\n\nTest: git rev-list --topo-order -100 HEAD\nHEAD~1, no commit-graph: 6.80 s\nHEAD~1, w/ commit-graph: 0.77 s\n  HEAD, w/ commit-graph: 0.02 s\n\nTest: git rev-list --topo-order -100 HEAD -- tools\nHEAD~1, no commit-graph: 9.63 s\nHEAD~1, w/ commit-graph: 6.06 s\n  HEAD, w/ commit-graph: 0.06 s\n\nThis speedup is due to a few things. First, the new generation-\nnumber-enabled algorithm walks commits on order of the number of\nresults output (subject to some branching structure expectations).\nSince we limit to 100 results, we are running a query similar to\nfilling a single page of results. Second, when specifying a path,\nwe must parse the root tree object for each commit we walk. The\nprevious benefits from the commit-graph are entirely from reading\nthe commit-graph instead of parsing commits. Since we need to\nparse trees for the same number of commits as before, we slow\ndown significantly from the non-path-based query.\n\nFor the test above, I specifically selected a path that is changed\nfrequently, including by merge commits. A less-frequently-changed\npath (such as 'README') has similar end-to-end time since we need\nto walk the same number of commits (before determining we do not\nhave 100 hits). However, get the benefit that the output is\npresented to the user as it is discovered, much the same as a\nnormal 'git log' command (no '--topo-order'). This is an improved\nuser experience, even if the command has the same runtime.\n\nHelped-by: Jeff King <peff@peff.net>\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n object.h   |   4 +-\n revision.c | 195 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n revision.h |   2 +\n 3 files changed, 194 insertions(+), 7 deletions(-)\n\ndiff --git a/object.h b/object.h\nindex 0feb90ae61..796792cb32 100644\n--- a/object.h\n+++ b/object.h\n@@ -59,7 +59,7 @@ struct object_array {\n \n /*\n  * object flag allocation:\n- * revision.h:               0---------10                              2526\n+ * revision.h:               0---------10                              25----28\n  * fetch-pack.c:             01\n  * negotiator/default.c:       2--5\n  * walker.c:                 0-2\n@@ -78,7 +78,7 @@ struct object_array {\n  * builtin/show-branch.c:    0-------------------------------------------26\n  * builtin/unpack-objects.c:                                 2021\n  */\n-#define FLAG_BITS  27\n+#define FLAG_BITS  29\n \n /*\n  * The object type is stored in 3 bits.\ndiff --git a/revision.c b/revision.c\nindex 36458265a0..4ef47d2fb4 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -26,6 +26,7 @@\n #include \"argv-array.h\"\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n+#include \"prio-queue.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2895,31 +2896,215 @@ static int mark_uninteresting(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct topo_walk_info {};\n+define_commit_slab(indegree_slab, int);\n+define_commit_slab(author_date_slab, timestamp_t);\n+\n+struct topo_walk_info {\n+\tuint32_t min_generation;\n+\tstruct prio_queue explore_queue;\n+\tstruct prio_queue indegree_queue;\n+\tstruct prio_queue topo_queue;\n+\tstruct indegree_slab indegree;\n+\tstruct author_date_slab author_date;\n+};\n+\n+static inline void test_flag_and_insert(struct prio_queue *q, struct commit *c, int flag)\n+{\n+\tif (c->object.flags & flag)\n+\t\treturn;\n+\n+\tc->object.flags |= flag;\n+\tprio_queue_put(q, c);\n+}\n+\n+static void explore_walk_step(struct rev_info *revs)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit_list *p;\n+\tstruct commit *c = prio_queue_get(&info->explore_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n+\t\trecord_author_date(&info->author_date, c);\n+\n+\tif (revs->max_age != -1 && (c->date < revs->max_age))\n+\t\tc->object.flags |= UNINTERESTING;\n+\n+\tif (process_parents(revs, c, NULL, NULL) < 0)\n+\t\treturn;\n+\n+\tif (c->object.flags & UNINTERESTING)\n+\t\tmark_parents_uninteresting(c);\n+\n+\tfor (p = c->parents; p; p = p->next)\n+\t\ttest_flag_and_insert(&info->explore_queue, p->item, TOPO_WALK_EXPLORED);\n+}\n+\n+static void explore_to_depth(struct rev_info *revs,\n+\t\t\t     uint32_t gen_cutoff)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->explore_queue)) &&\n+\t       c->generation >= gen_cutoff)\n+\t\texplore_walk_step(revs);\n+}\n+\n+static void indegree_walk_step(struct rev_info *revs)\n+{\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c = prio_queue_get(&info->indegree_queue);\n+\n+\tif (!c)\n+\t\treturn;\n+\n+\tif (parse_commit_gently(c, 1) < 0)\n+\t\treturn;\n+\n+\texplore_to_depth(revs, c->generation);\n+\n+\tfor (p = c->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\tif (*pi)\n+\t\t\t(*pi)++;\n+\t\telse\n+\t\t\t*pi = 2;\n+\n+\t\ttest_flag_and_insert(&info->indegree_queue, parent, TOPO_WALK_INDEGREE);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n+\t}\n+}\n+\n+static void compute_indegrees_to_depth(struct rev_info *revs,\n+\t\t\t\t       uint32_t gen_cutoff)\n+{\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tstruct commit *c;\n+\twhile ((c = prio_queue_peek(&info->indegree_queue)) &&\n+\t       c->generation >= gen_cutoff)\n+\t\tindegree_walk_step(revs);\n+}\n \n static void init_topo_walk(struct rev_info *revs)\n {\n \tstruct topo_walk_info *info;\n+\tstruct commit_list *list;\n \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n \tinfo = revs->topo_walk_info;\n \tmemset(info, 0, sizeof(struct topo_walk_info));\n \n-\tlimit_list(revs);\n-\tsort_in_topological_order(&revs->commits, revs->sort_order);\n+\tinit_indegree_slab(&info->indegree);\n+\tmemset(&info->explore_queue, 0, sizeof(info->explore_queue));\n+\tmemset(&info->indegree_queue, 0, sizeof(info->indegree_queue));\n+\tmemset(&info->topo_queue, 0, sizeof(info->topo_queue));\n+\n+\tswitch (revs->sort_order) {\n+\tdefault: /* REV_SORT_IN_GRAPH_ORDER */\n+\t\tinfo->topo_queue.compare = NULL;\n+\t\tbreak;\n+\tcase REV_SORT_BY_COMMIT_DATE:\n+\t\tinfo->topo_queue.compare = compare_commits_by_commit_date;\n+\t\tbreak;\n+\tcase REV_SORT_BY_AUTHOR_DATE:\n+\t\tinit_author_date_slab(&info->author_date);\n+\t\tinfo->topo_queue.compare = compare_commits_by_author_date;\n+\t\tinfo->topo_queue.cb_data = &info->author_date;\n+\t\tbreak;\n+\t}\n+\n+\tinfo->explore_queue.compare = compare_commits_by_gen_then_commit_date;\n+\tinfo->indegree_queue.compare = compare_commits_by_gen_then_commit_date;\n+\n+\tinfo->min_generation = GENERATION_NUMBER_INFINITY;\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\n+\t\tif (parse_commit_gently(c, 1))\n+\t\t\tcontinue;\n+\n+\t\ttest_flag_and_insert(&info->explore_queue, c, TOPO_WALK_EXPLORED);\n+\t\ttest_flag_and_insert(&info->indegree_queue, c, TOPO_WALK_INDEGREE);\n+\n+\t\tif (c->generation < info->min_generation)\n+\t\t\tinfo->min_generation = c->generation;\n+\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 1;\n+\n+\t\tif (revs->sort_order == REV_SORT_BY_AUTHOR_DATE)\n+\t\t\trecord_author_date(&info->author_date, c);\n+\t}\n+\tcompute_indegrees_to_depth(revs, info->min_generation);\n+\n+\tfor (list = revs->commits; list; list = list->next) {\n+\t\tstruct commit *c = list->item;\n+\n+\t\tif (*(indegree_slab_at(&info->indegree, c)) == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, c);\n+\t}\n+\n+\t/*\n+\t * This is unfortunate; the initial tips need to be shown\n+\t * in the order given from the revision traversal machinery.\n+\t */\n+\tif (revs->sort_order == REV_SORT_IN_GRAPH_ORDER)\n+\t\tprio_queue_reverse(&info->topo_queue);\n }\n \n static struct commit *next_topo_commit(struct rev_info *revs)\n {\n-\treturn pop_commit(&revs->commits);\n+\tstruct commit *c;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\n+\t/* pop next off of topo_queue */\n+\tc = prio_queue_get(&info->topo_queue);\n+\n+\tif (c)\n+\t\t*(indegree_slab_at(&info->indegree, c)) = 0;\n+\n+\treturn c;\n }\n \n static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n {\n-\tif (process_parents(revs, commit, &revs->commits, NULL) < 0) {\n+\tstruct commit_list *p;\n+\tstruct topo_walk_info *info = revs->topo_walk_info;\n+\tif (process_parents(revs, commit, NULL, NULL) < 0) {\n \t\tif (!revs->ignore_missing_links)\n \t\t\tdie(\"Failed to traverse parents of commit %s\",\n \t\t\t    oid_to_hex(&commit->object.oid));\n \t}\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tstruct commit *parent = p->item;\n+\t\tint *pi;\n+\n+\t\tif (parse_commit_gently(parent, 1) < 0)\n+\t\t\tcontinue;\n+\n+\t\tif (parent->generation < info->min_generation) {\n+\t\t\tinfo->min_generation = parent->generation;\n+\t\t\tcompute_indegrees_to_depth(revs, info->min_generation);\n+\t\t}\n+\n+\t\tpi = indegree_slab_at(&info->indegree, parent);\n+\n+\t\t(*pi)--;\n+\t\tif (*pi == 1)\n+\t\t\tprio_queue_put(&info->topo_queue, parent);\n+\n+\t\tif (revs->first_parent_only)\n+\t\t\treturn;\n+\t}\n }\n \n int prepare_revision_walk(struct rev_info *revs)\ndiff --git a/revision.h b/revision.h\nindex fd4154ff75..b0b3bb8025 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -24,6 +24,8 @@\n #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n #define TRACK_LINEAR\t(1u<<26)\n #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n+#define TOPO_WALK_EXPLORED\t(1u<<27)\n+#define TOPO_WALK_INDEGREE\t(1u<<28)\n \n #define DECORATE_SHORT_REFS\t1\n #define DECORATE_FULL_REFS\t2\n-- \n2.19.1.542.gc4df23f792\n\n"},{"id":"362185","messageId":"20181101134623.84055-8-dstolee@microsoft.com","threadId":"49229","inReplyTo":"20181101134623.84055-1-dstolee@microsoft.com","subject":"[PATCH v5 7/7] t6012: make rev-list tests more interesting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:46:23Z","receivedAt":"2018-11-01T13:46:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As we are working to rewrite some of the revision-walk machinery,\nthere could easily be some interesting interactions between the\noptions that force topological constraints (--topo-order,\n--date-order, and --author-date-order) along with specifying a\npath.\n\nAdd extra tests to t6012-rev-list-simplify.sh to add coverage of\nthese interactions. To ensure interesting things occur, alter the\nrepo data shape to have different orders depending on topo-, date-,\nor author-date-order.\n\nWhen testing using GIT_TEST_COMMIT_GRAPH, this assists in covering\nthe new logic for topo-order walks using generation numbers. The\nextra tests can be added indepently.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t6012-rev-list-simplify.sh | 45 ++++++++++++++++++++++++++++--------\n 1 file changed, 36 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b5a1190ffe..a10f0df02b 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -12,6 +12,22 @@ unnote () {\n \tgit name-rev --tags --stdin | sed -e \"s|$OID_REGEX (tags/\\([^)]*\\)) |\\1 |g\"\n }\n \n+#\n+# Create a test repo with interesting commit graph:\n+#\n+# A--B----------G--H--I--K--L\n+#  \\  \\           /     /\n+#   \\  \\         /     /\n+#    C------E---F     J\n+#        \\_/\n+#\n+# The commits are laid out from left-to-right starting with\n+# the root commit A and terminating at the tip commit L.\n+#\n+# There are a few places where we adjust the commit date or\n+# author date to make the --topo-order, --date-order, and\n+# --author-date-order flags produce different output.\n+\n test_expect_success setup '\n \techo \"Hi there\" >file &&\n \techo \"initial\" >lost &&\n@@ -21,10 +37,18 @@ test_expect_success setup '\n \n \tgit branch other-branch &&\n \n+\tgit symbolic-ref HEAD refs/heads/unrelated &&\n+\tgit rm -f \"*\" &&\n+\techo \"Unrelated branch\" >side &&\n+\tgit add side &&\n+\ttest_tick && git commit -m \"Side root\" &&\n+\tnote J &&\n+\tgit checkout master &&\n+\n \techo \"Hello\" >file &&\n \techo \"second\" >lost &&\n \tgit add file lost &&\n-\ttest_tick && git commit -m \"Modified file and lost\" &&\n+\ttest_tick && GIT_AUTHOR_DATE=$(($test_tick + 120)) git commit -m \"Modified file and lost\" &&\n \tnote B &&\n \n \tgit checkout other-branch &&\n@@ -63,13 +87,6 @@ test_expect_success setup '\n \ttest_tick && git commit -a -m \"Final change\" &&\n \tnote I &&\n \n-\tgit symbolic-ref HEAD refs/heads/unrelated &&\n-\tgit rm -f \"*\" &&\n-\techo \"Unrelated branch\" >side &&\n-\tgit add side &&\n-\ttest_tick && git commit -m \"Side root\" &&\n-\tnote J &&\n-\n \tgit checkout master &&\n \ttest_tick && git merge --allow-unrelated-histories -m \"Coolest\" unrelated &&\n \tnote K &&\n@@ -103,14 +120,24 @@ check_result () {\n \tcheck_outcome success \"$@\"\n }\n \n-check_result 'L K J I H G F E D C B A' --full-history\n+check_result 'L K J I H F E D C G B A' --full-history --topo-order\n+check_result 'L K I H G F E D C B J A' --full-history\n+check_result 'L K I H G F E D C B J A' --full-history --date-order\n+check_result 'L K I H G F E D B C J A' --full-history --author-date-order\n check_result 'K I H E C B A' --full-history -- file\n check_result 'K I H E C B A' --full-history --topo-order -- file\n check_result 'K I H E C B A' --full-history --date-order -- file\n+check_result 'K I H E B C A' --full-history --author-date-order -- file\n check_result 'I E C B A' --simplify-merges -- file\n+check_result 'I E C B A' --simplify-merges --topo-order -- file\n+check_result 'I E C B A' --simplify-merges --date-order -- file\n+check_result 'I E B C A' --simplify-merges --author-date-order -- file\n check_result 'I B A' -- file\n check_result 'I B A' --topo-order -- file\n+check_result 'I B A' --date-order -- file\n+check_result 'I B A' --author-date-order -- file\n check_result 'H' --first-parent -- another-file\n+check_result 'H' --first-parent --topo-order -- another-file\n \n check_result 'E C B A' --full-history E -- lost\n test_expect_success 'full history simplification without parent' '\n-- \n2.19.1.542.gc4df23f792\n\n"},{"id":"362187","messageId":"9ffd8ecd-5595-ad19-cfe1-a268e3fa1bd3@gmail.com","threadId":"49229","inReplyTo":"xmqqpnvptlta.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 0/7] Use generation numbers for --topo-order","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T13:49:22Z","receivedAt":"2018-11-01T13:49:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/2018 1:21 AM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n>> This patch series performs a decently-sized refactoring of the revision-walk\n>> machinery. Well, \"refactoring\" is probably the wrong word, as I don't\n>> actually remove the old code. Instead, when we see certain options in the\n>> 'rev_info' struct, we redirect the commit-walk logic to a new set of methods\n>> that distribute the workload differently. By using generation numbers in the\n>> commit-graph, we can significantly improve 'git log --graph' commands (and\n>> the underlying 'git rev-list --topo-order').\n> Review discussions seem to have petered out.  Would we merge this to\n> 'next' and start cooking, perhaps for the remainder of this cycle?\n\nThanks, but I've just sent a v5 responding to Jakub's feedback on v4. [1]\n\nI'd be happy to let it sit in next until you feel it has cooked long \nenough. I'm available to respond to feedback in the form of new topics.\n\nThanks,\n-Stolee\n\n[1] \nhttps://public-inbox.org/git/20181101134623.84055-1-dstolee@microsoft.com/T/#u\n"},{"id":"362196","messageId":"20181101154857.GW30222@szeder.dev","threadId":"49229","inReplyTo":"20181101134623.84055-7-dstolee@microsoft.com","subject":"Re: [PATCH v5 6/7] revision.c: generation-based topo-order algorithm","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-01T15:48:57Z","receivedAt":"2018-11-01T15:49:04Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Nov 01, 2018 at 01:46:22PM +0000, Derrick Stolee wrote:\n> 1. EXPLORE: using the explore_queue priority queue (ordered by\n>    maximizing the generation number)\n\n> 2. INDEGREE: using the indegree_queue priority queue (ordered\n>    by maximizing the generation number)\n\nNit: I've been pondering for a while what exactly does \"order by\nmaximizing ...\" mean.  Highest to lowest or lowest to highest?  If I\nunderstand the rest of the descriptions (that I snipped) correctly,\nthen it's the former, but I find that phrase in itself too ambiguous.\n\n"},{"id":"362198","messageId":"7ee1b2b4-b748-85d7-ab0f-4446a466d196@gmail.com","threadId":"49229","inReplyTo":"20181101154857.GW30222@szeder.dev","subject":"Re: [PATCH v5 6/7] revision.c: generation-based topo-order algorithm","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-01T16:12:10Z","receivedAt":"2018-11-01T16:12:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/2018 11:48 AM, SZEDER Gábor wrote:\n> On Thu, Nov 01, 2018 at 01:46:22PM +0000, Derrick Stolee wrote:\n>> 1. EXPLORE: using the explore_queue priority queue (ordered by\n>>     maximizing the generation number)\n>> 2. INDEGREE: using the indegree_queue priority queue (ordered\n>>     by maximizing the generation number)\n> Nit: I've been pondering for a while what exactly does \"order by\n> maximizing ...\" mean.  Highest to lowest or lowest to highest?  If I\n> understand the rest of the descriptions (that I snipped) correctly,\n> then it's the former, but I find that phrase in itself too ambiguous.\n\nIt means that our priority-queue \"get\" operation selects the item in the\nqueue that is largest by our comparison function (first generation number,\nthencommit-date for ties).This means we walk commits that have high\ngeneration number before those with lower generation number, guaranteeing\nthat we walk all children of a commit before walking that commit.\n\nThanks,\n-Stolee\n"},{"id":"362219","messageId":"xmqqva5gs69n.fsf@gitster-ct.c.googlers.com","threadId":"49229","inReplyTo":"9ffd8ecd-5595-ad19-cfe1-a268e3fa1bd3@gmail.com","subject":"Re: [PATCH v4 0/7] Use generation numbers for --topo-order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-01T23:54:44Z","receivedAt":"2018-11-01T23:54:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n>> Review discussions seem to have petered out.  Would we merge this to\n>> 'next' and start cooking, perhaps for the remainder of this cycle?\n>\n> Thanks, but I've just sent a v5 responding to Jakub's feedback on v4. [1]\n>\n> I'd be happy to let it sit in next until you feel it has cooked long\n> enough. I'm available to respond to feedback in the form of new\n> topics.\n\nOK.  I'm quite happy to see this round of review helped greatly by\nJakub, by the way.\n\nTHanks, both.\n"},{"id":"385746","messageId":"20191108025007.bphr7ynvskeoe6tb@glandium.org","threadId":"49229","inReplyTo":"20181101134623.84055-7-dstolee@microsoft.com","subject":"Re: [PATCH v5 6/7] revision.c: generation-based topo-order algorithm","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2019-11-08T02:50:07Z","receivedAt":"2019-11-08T02:50:14Z","isPatch":true,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Replying to this old thread because I have questions regarding the\npatch, in the context of problems I had downstream, in git-cinnabar.\n\nOn Thu, Nov 01, 2018 at 01:46:22PM +0000, Derrick Stolee wrote:\n>  static void init_topo_walk(struct rev_info *revs)\n>  {\n>  \tstruct topo_walk_info *info;\n> +\tstruct commit_list *list;\n>  \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n\nNot directly from this patch, but there's nothing that frees this memory\nAFAICS, but then, that's also true for most of the things in struct\nrev_info.\n\n> diff --git a/revision.h b/revision.h\n> index fd4154ff75..b0b3bb8025 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -24,6 +24,8 @@\n>  #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n>  #define TRACK_LINEAR\t(1u<<26)\n>  #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n> +#define TOPO_WALK_EXPLORED\t(1u<<27)\n> +#define TOPO_WALK_INDEGREE\t(1u<<28)\n\nShould these two flags be included in ALL_REV_FLAGS?\nShould they be reset by reset_revision_walk?\n\nAt least for the latter, I'd say yes, otherwise you can end up with\nmissing revs in a subsequent topo-order revwalk.\n\nMike\n"},{"id":"385854","messageId":"ea07fd13-7d0a-4636-1aa4-453f6b837735@gmail.com","threadId":"49229","inReplyTo":"20191108025007.bphr7ynvskeoe6tb@glandium.org","subject":"Re: [PATCH v5 6/7] revision.c: generation-based topo-order algorithm","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-11T01:07:31Z","receivedAt":"2019-11-11T01:07:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/7/2019 9:50 PM, Mike Hommey wrote:\n> Replying to this old thread because I have questions regarding the\n> patch, in the context of problems I had downstream, in git-cinnabar.\n> \n> On Thu, Nov 01, 2018 at 01:46:22PM +0000, Derrick Stolee wrote:\n>>  static void init_topo_walk(struct rev_info *revs)\n>>  {\n>>  \tstruct topo_walk_info *info;\n>> +\tstruct commit_list *list;\n>>  \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n> \n> Not directly from this patch, but there's nothing that frees this memory\n> AFAICS, but then, that's also true for most of the things in struct\n> rev_info.\n\nThis is true, the 'struct rev_info' doesn't get cleaned up at the end.\nIt is probably a lot of work to find all the consumers and get them to\nclean everything up, and the value is rather low. I believe the expectation\nis that each process will only run a revision walk at most once.\n\n>> diff --git a/revision.h b/revision.h\n>> index fd4154ff75..b0b3bb8025 100644\n>> --- a/revision.h\n>> +++ b/revision.h\n>> @@ -24,6 +24,8 @@\n>>  #define USER_GIVEN\t(1u<<25) /* given directly by the user */\n>>  #define TRACK_LINEAR\t(1u<<26)\n>>  #define ALL_REV_FLAGS\t(((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR)\n>> +#define TOPO_WALK_EXPLORED\t(1u<<27)\n>> +#define TOPO_WALK_INDEGREE\t(1u<<28)\n> \n> Should these two flags be included in ALL_REV_FLAGS?\n> Should they be reset by reset_revision_walk?\n> \n> At least for the latter, I'd say yes, otherwise you can end up with\n> missing revs in a subsequent topo-order revwalk.\n\nThis is probably true. Sounds like a quick contribution could\nbe in order?\n\nThanks,\n-Stolee\n\n"},{"id":"386474","messageId":"20191118230431.GC23183@szeder.dev","threadId":"49229","inReplyTo":"ea07fd13-7d0a-4636-1aa4-453f6b837735@gmail.com","subject":"Re: [PATCH v5 6/7] revision.c: generation-based topo-order algorithm","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-18T23:04:31Z","receivedAt":"2019-11-18T23:04:38Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sun, Nov 10, 2019 at 08:07:31PM -0500, Derrick Stolee wrote:\n> On 11/7/2019 9:50 PM, Mike Hommey wrote:\n> > Replying to this old thread because I have questions regarding the\n> > patch, in the context of problems I had downstream, in git-cinnabar.\n> > \n> > On Thu, Nov 01, 2018 at 01:46:22PM +0000, Derrick Stolee wrote:\n> >>  static void init_topo_walk(struct rev_info *revs)\n> >>  {\n> >>  \tstruct topo_walk_info *info;\n> >> +\tstruct commit_list *list;\n> >>  \trevs->topo_walk_info = xmalloc(sizeof(struct topo_walk_info));\n> > \n> > Not directly from this patch, but there's nothing that frees this memory\n> > AFAICS, but then, that's also true for most of the things in struct\n> > rev_info.\n> \n> This is true, the 'struct rev_info' doesn't get cleaned up at the end.\n> It is probably a lot of work to find all the consumers and get them to\n> clean everything up, and the value is rather low. I believe the expectation\n> is that each process will only run a revision walk at most once.\n\nI don't think that's a valid expectation.\n\nSeveral commands must do multiple revision walks in a single process,\ne.g. 'describe' or 'name-rev', but they tend to do so by rolling their\nown low-level revision walking (e.g. by putting all ~SEEN parents into\na 'commit_list' and iterating until the list becomes empty) instead of\na higher-level 'while ((commit = get_revision(revs)))' loop.\n\nAlas, some of those commands are buggy, or at least 'git describe' is\n[1], and AFAICT the only way to fix that bug is to walk the history in\ntopo-order.  And of course we should not roll its own topo-order\nrevision walk for each of those commands, but rather should convert\nthem to use get_revision(), so they can all rely on the magic of the\ncommit-graph-based on-the-fly topo-order, especially since the\ncommit-graph is now enabled by default.  However, all this means a lot\nof separate get_revision()-based revision walks in a single process.\n\n[1]  https://public-inbox.org/git/20191008123156.GG11529@szeder.dev/\n\n"}]}