{"thread":{"id":"48212","subject":"[PATCH 0/6] Compute and consume generation numbers","startedAt":"2018-04-03T16:52:15Z","lastAt":"2018-05-03T18:45:37Z","messageCount":162,"participants":["Derrick Stolee","Brandon Williams","Jonathan Tan","Jeff King","Stefan Beller","Ramsay Jones","Jakub Narebski","Junio C Hamano","Eric Sunshine"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"343721","messageId":"20180403165143.80661-1-dstolee@microsoft.com","threadId":"48212","inReplyTo":null,"subject":"[PATCH 0/6] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-03T16:51:37Z","receivedAt":"2018-04-03T16:52:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first of several \"small\" patches that follow the serialized\nGit commit graph patch (ds/commit-graph).\n\nAs described in Documentation/technical/commit-graph.txt, the generation\nnumber of a commit is one more than the maximum generation number among\nits parents (trivially, a commit with no parents has generation number\none).\n\nThis series makes the computation of generation numbers part of the\ncommit-graph write process.\n\nFinally, generation numbers are used to order commits in the priority\nqueue in paint_down_to_common(). This allows a constant-time check in\nqueue_has_nonstale() instead of the previous linear-time check.\n\nThis does not have a significant performance benefit in repositories\nof normal size, but in the Windows repository, some merge-base\ncalculations improve from 3.1s to 2.9s. A modest speedup, but provides\nan actual consumer of generation numbers as a starting point.\n\nA more substantial refactoring of revision.c is required before making\n'git log --graph' use generation numbers effectively.\n\nThis patch series depends on v7 of ds/commit-graph.\n\nDerrick Stolee (6):\n object.c: parse commit in graph first\n commit: add generation number to struct commmit\n commit-graph: compute generation numbers\n commit: sort by generation number in paint_down_to_common()\n commit.c: use generation number to stop merge-base walks\n commit-graph.txt: update design doc with generation numbers\n\n Documentation/technical/commit-graph.txt | 7 +---\n alloc.c | 1 +\n commit-graph.c | 48 +++++++++++++++++++++\n commit.c | 53 ++++++++++++++++++++----\n commit.h | 7 +++-\n object.c | 4 +-\n 6 files changed, 104 insertions(+), 16 deletions(-)\n\n-- \n2.17.0.20.g9f30ba16e1\n\n"},{"id":"343722","messageId":"20180403165143.80661-2-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"[PATCH 1/6] object.c: parse commit in graph first","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-03T16:51:38Z","receivedAt":"2018-04-03T16:52:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Most code paths load commits using lookup_commit() and then\nparse_commit(). In some cases, including some branch lookups, the commit\nis parsed using parse_object_buffer() which side-steps parse_commit() in\nfavor of parse_commit_buffer().\n\nBefore adding generation numbers to the commit-graph, we need to ensure\nthat any commit that exists in the graph is loaded from the graph, so\ncheck parse_commit_in_graph() before calling parse_commit_buffer().\n\nSigned-off-by: Derrick Stolee \n---\n object.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/object.c b/object.c\nindex e6ad3f61f0..4cd3e98e04 100644\n--- a/object.c\n+++ b/object.c\n@@ -3,6 +3,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"commit.h\"\n+#include \"commit-graph.h\"\n #include \"tag.h\"\n \n static struct object **obj_hash;\n@@ -207,7 +208,8 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n \t} else if (type == OBJ_COMMIT) {\n \t\tstruct commit *commit = lookup_commit(oid);\n \t\tif (commit) {\n-\t\t\tif (parse_commit_buffer(commit, buffer, size))\n+\t\t\tif (!parse_commit_in_graph(commit) &&\n+\t\t\t parse_commit_buffer(commit, buffer, size))\n \t\t\t\treturn NULL;\n \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n \t\t\t\tset_commit_buffer(commit, buffer, size);\n-- \n2.17.0.rc0\n\n"},{"id":"343723","messageId":"20180403165143.80661-7-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"[PATCH 6/6] commit-graph.txt: update future work","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-03T16:51:43Z","receivedAt":"2018-04-03T16:52:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We now calculate generation numbers in the commit-graph file and use\nthem in paint_down_to_common().\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 7 +------\n 1 file changed, 1 insertion(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 0550c6d0dc..be68bee43d 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -98,17 +98,12 @@ Future Work\n - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n necessary for integration with fsck.\n \n-- The file format includes room for precomputed generation numbers. These\n- are not currently computed, so all generation numbers will be marked as\n- 0 (or \"uncomputed\"). A later patch will include this calculation.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n priority queue with one ordered by generation number. The following\n- operations are important candidates:\n+ operation is an important candidate:\n \n- - paint_down_to_common()\n - 'log --topo-order'\n \n - Currently, parse_commit_gently() requires filling in the root tree\n-- \n2.17.0.rc0\n\n"},{"id":"343724","messageId":"20180403165143.80661-6-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"[PATCH 5/6] commit.c: use generation to halt paint walk","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-03T16:51:42Z","receivedAt":"2018-04-03T16:52:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In paint_down_to_common(), the walk is halted when the queue contains\nonly stale commits. The queue_has_nonstale() method iterates over the\nentire queue looking for a nonstale commit. In a wide commit graph where\nthe two sides share many commits in common, but have deep sets of\ndifferent commits, this method may inspect many elements before finding\na nonstale commit. In the worst case, this can give quadratic\nperformance in paint_down_to_common().\n\nConvert queue_has_nonstale() to use generation numbers for an O(1)\ntermination condition. To properly take advantage of this condition,\ntrack the minimum generation number of a commit that enters the queue\nwith nonstale status.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 95ae7e13a3..858f4fdbc9 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -776,14 +776,22 @@ void sort_in_topological_order(struct commit_list **list, enum rev_sort_order so\n \n static const unsigned all_flags = (PARENT1 | PARENT2 | STALE | RESULT);\n \n-static int queue_has_nonstale(struct prio_queue *queue)\n+static int queue_has_nonstale(struct prio_queue *queue, uint32_t min_gen)\n {\n-\tint i;\n-\tfor (i = 0; i < queue->nr; i++) {\n-\t\tstruct commit *commit = queue->array[i].data;\n-\t\tif (!(commit->object.flags & STALE))\n-\t\t\treturn 1;\n+\tif (min_gen != GENERATION_NUMBER_UNDEF) {\n+\t\tif (queue->nr > 0) {\n+\t\t\tstruct commit *commit = queue->array[0].data;\n+\t\t\treturn commit->generation >= min_gen;\n+\t\t}\n+\t} else {\n+\t\tint i;\n+\t\tfor (i = 0; i < queue->nr; i++) {\n+\t\t\tstruct commit *commit = queue->array[i].data;\n+\t\t\tif (!(commit->object.flags & STALE))\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n+\n \treturn 0;\n }\n \n@@ -793,6 +801,8 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n+\tuint32_t last_gen = GENERATION_NUMBER_UNDEF;\n+\tuint32_t min_nonstale_gen = GENERATION_NUMBER_UNDEF;\n \n \tone->object.flags |= PARENT1;\n \tif (!n) {\n@@ -800,17 +810,26 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \t\treturn result;\n \t}\n \tprio_queue_put(&queue, one);\n+\tif (one->generation < min_nonstale_gen)\n+\t\tmin_nonstale_gen = one->generation;\n \n \tfor (i = 0; i < n; i++) {\n \t\ttwos[i]->object.flags |= PARENT2;\n \t\tprio_queue_put(&queue, twos[i]);\n+\t\tif (twos[i]->generation < min_nonstale_gen)\n+\t\t\tmin_nonstale_gen = twos[i]->generation;\n \t}\n \n-\twhile (queue_has_nonstale(&queue)) {\n+\twhile (queue_has_nonstale(&queue, min_nonstale_gen)) {\n \t\tstruct commit *commit = prio_queue_get(&queue);\n \t\tstruct commit_list *parents;\n \t\tint flags;\n \n+\t\tif (commit->generation > last_gen)\n+\t\t\tBUG(\"bad generation skip\");\n+\n+\t\tlast_gen = commit->generation;\n+\n \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n \t\tif (flags == (PARENT1 | PARENT2)) {\n \t\t\tif (!(commit->object.flags & RESULT)) {\n@@ -830,6 +849,10 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \t\t\t\treturn NULL;\n \t\t\tp->object.flags |= flags;\n \t\t\tprio_queue_put(&queue, p);\n+\n+\t\t\tif (!(flags & STALE) &&\n+\t\t\t p->generation < min_nonstale_gen)\n+\t\t\t\tmin_nonstale_gen = p->generation;\n \t\t}\n \t}\n \n-- \n2.17.0.rc0\n\n"},{"id":"343725","messageId":"20180403165143.80661-5-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"[PATCH 4/6] commit: use generations in paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-03T16:51:41Z","receivedAt":"2018-04-03T16:52:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Define compare_commits_by_gen_then_commit_date(), which uses generation\nnumbers as a primary comparison and commit date to break ties (or as a\ncomparison when both commits do not have computed generation numbers).\n\nSince the commit-graph file is closed under reachability, we know that\nall commits in the file have generation at most GENERATION_NUMBER_MAX\nwhich is less than GENERATION_NUMBER_UNDEF.\n\nThis change does not affect the number of commits that are walked during\nthe execution of paint_down_to_common(), only the order that those\ncommits are inspected. In the case that commit dates violate topological\norder (i.e. a parent is \"newer\" than a child), the previous code could\nwalk a commit twice: if a commit is reached with the PARENT1 bit, but\nlater is re-visited with the PARENT2 bit, then that PARENT2 bit must be\npropagated to its parents. Using generation numbers avoids this extra\neffort, even if it is somewhat rare.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 19 ++++++++++++++++++-\n commit.h | 1 +\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 3e39c86abf..95ae7e13a3 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -624,6 +624,23 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n \treturn 0;\n }\n \n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n+{\n+\tconst struct commit *a = a_, *b = b_;\n+\n+\tif (a->generation < b->generation)\n+\t\treturn 1;\n+\telse if (a->generation > b->generation)\n+\t\treturn -1;\n+\n+\t/* newer commits with larger date first */\n+\tif (a->date < b->date)\n+\t\treturn 1;\n+\telse if (a->date > b->date)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n {\n \tconst struct commit *a = a_, *b = b_;\n@@ -773,7 +790,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n /* all input commits in one and twos[] must have been parsed! */\n static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n {\n-\tstruct prio_queue queue = { compare_commits_by_commit_date };\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n \ndiff --git a/commit.h b/commit.h\nindex bc7a3186c5..cb97b7636a 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -332,6 +332,7 @@ extern int remove_signature(struct strbuf *buf);\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \n LAST_ARG_MUST_BE_NULL\n extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n-- \n2.17.0.rc0\n\n"},{"id":"343726","messageId":"20180403165143.80661-3-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"[PATCH 2/6] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-03T16:51:39Z","receivedAt":"2018-04-03T16:52:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The generation number of a commit is defined recursively as follows:\n\n* If a commit A has no parents, then the generation number of A is one.\n* If a commit A has parents, then the generation number of A is one\n more than the maximum generation number among the parents of A.\n\nAdd a uint32_t generation field to struct commit so we can pass this\ninformation to revision walks. We use two special values to signal\nthe generation number is invalid:\n\nGENERATION_NUMBER_UNDEF 0xFFFFFFFF\nGENERATION_NUMBER_NONE 0\n\nThe first (_UNDEF) means the generation number has not been loaded or\ncomputed. The second (_NONE) means the generation number was loaded\nfrom a commit graph file that was stored before generation numbers\nwere computed.\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 2 ++\n commit.h | 3 +++\n 3 files changed, 6 insertions(+)\n\ndiff --git a/alloc.c b/alloc.c\nindex cf4f8b61e1..1a62e85ac3 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n+\tc->generation = GENERATION_NUMBER_UNDEF;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1fc63d541b..d24b947525 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -264,6 +264,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \tdate_low = get_be32(commit_data + g->hash_len + 12);\n \titem->date = (timestamp_t)((date_high << 32) | date_low);\n \n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+\n \tpptr = &item->parents;\n \n \tedge_value = get_be32(commit_data + g->hash_len);\ndiff --git a/commit.h b/commit.h\nindex e57ae4b583..3cadd386f3 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -10,6 +10,8 @@\n #include \"pretty.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+#define GENERATION_NUMBER_UNDEF 0xFFFFFFFF\n+#define GENERATION_NUMBER_NONE 0\n \n struct commit_list {\n \tstruct commit *item;\n@@ -24,6 +26,7 @@ struct commit {\n \tstruct commit_list *parents;\n \tstruct tree *tree;\n \tuint32_t graph_pos;\n+\tuint32_t generation;\n };\n \n extern int save_commit_buffer;\n-- \n2.17.0.rc0\n\n"},{"id":"343727","messageId":"20180403165143.80661-4-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"[PATCH 3/6] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-03T16:51:40Z","receivedAt":"2018-04-03T16:52:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While preparing commits to be written into a commit-graph file, compute\nthe generation numbers using a depth-first strategy.\n\nThe only commits that are walked in this depth-first search are those\nwithout a precomputed generation number. Thus, computation time will be\nrelative to the number of new commits to the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++\n commit.h | 1 +\n 2 files changed, 47 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d24b947525..b80c8ad80e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -419,6 +419,13 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\telse\n \t\t\tpackedDate[0] = 0;\n \n+\t\tif ((*list)->generation != GENERATION_NUMBER_UNDEF) {\n+\t\t\tif ((*list)->generation > GENERATION_NUMBER_MAX)\n+\t\t\t\tdie(\"generation number %u is too large to store in commit-graph\",\n+\t\t\t\t (*list)->generation);\n+\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n+\t\t}\n+\n \t\tpackedDate[1] = htonl((*list)->date);\n \t\thashwrite(f, packedDate, 8);\n \n@@ -551,6 +558,43 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n }\n \n+static void compute_generation_numbers(struct commit** commits,\n+\t\t\t\t int nr_commits)\n+{\n+\tint i;\n+\tstruct commit_list *list = NULL;\n+\n+\tfor (i = 0; i < nr_commits; i++) {\n+\t\tif (commits[i]->generation != GENERATION_NUMBER_UNDEF &&\n+\t\t commits[i]->generation != GENERATION_NUMBER_NONE)\n+\t\t\tcontinue;\n+\n+\t\tcommit_list_insert(commits[i], &list);\n+\t\twhile (list) {\n+\t\t\tstruct commit *current = list->item;\n+\t\t\tstruct commit_list *parent;\n+\t\t\tint all_parents_computed = 1;\n+\t\t\tuint32_t max_generation = 0;\n+\n+\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n+\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_UNDEF ||\n+\t\t\t\t parent->item->generation == GENERATION_NUMBER_NONE) {\n+\t\t\t\t\tall_parents_computed = 0;\n+\t\t\t\t\tcommit_list_insert(parent->item, &list);\n+\t\t\t\t\tbreak;\n+\t\t\t\t} else if (parent->item->generation > max_generation) {\n+\t\t\t\t\tmax_generation = parent->item->generation;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (all_parents_computed) {\n+\t\t\t\tcurrent->generation = max_generation + 1;\n+\t\t\t\tpop_commit(&list);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tconst char **pack_indexes,\n \t\t\tint nr_packs,\n@@ -674,6 +718,8 @@ void write_commit_graph(const char *obj_dir,\n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n \n+\tcompute_generation_numbers(commits.list, commits.nr);\n+\n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n \ndiff --git a/commit.h b/commit.h\nindex 3cadd386f3..bc7a3186c5 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -11,6 +11,7 @@\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n #define GENERATION_NUMBER_UNDEF 0xFFFFFFFF\n+#define GENERATION_NUMBER_MAX 0x3FFFFFFF\n #define GENERATION_NUMBER_NONE 0\n \n struct commit_list {\n-- \n2.17.0.rc0\n\n"},{"id":"343728","messageId":"f68ca797-8ea6-ca3f-b940-2f5aa6633938@gmail.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-03T16:56:41Z","receivedAt":"2018-04-03T16:56:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/3/2018 12:51 PM, Derrick Stolee wrote:\n> This is the first of several \"small\" patches that follow the serialized\n> Git commit graph patch (ds/commit-graph).\n>\n> As described in Documentation/technical/commit-graph.txt, the generation\n> number of a commit is one more than the maximum generation number among\n> its parents (trivially, a commit with no parents has generation number\n> one).\n>\n> This series makes the computation of generation numbers part of the\n> commit-graph write process.\n>\n> Finally, generation numbers are used to order commits in the priority\n> queue in paint_down_to_common(). This allows a constant-time check in\n> queue_has_nonstale() instead of the previous linear-time check.\n>\n> This does not have a significant performance benefit in repositories\n> of normal size, but in the Windows repository, some merge-base\n> calculations improve from 3.1s to 2.9s. A modest speedup, but provides\n> an actual consumer of generation numbers as a starting point.\n>\n> A more substantial refactoring of revision.c is required before making\n> 'git log --graph' use generation numbers effectively.\n>\n> This patch series depends on v7 of ds/commit-graph.\n>\n> Derrick Stolee (6):\n> object.c: parse commit in graph first\n> commit: add generation number to struct commmit\n> commit-graph: compute generation numbers\n> commit: sort by generation number in paint_down_to_common()\n> commit.c: use generation number to stop merge-base walks\n> commit-graph.txt: update design doc with generation numbers\n\nThis patch is also available as a GitHub pull request [1]\n\n[1] https://github.com/derrickstolee/git/pull/5\n"},{"id":"343734","messageId":"20180403180336.GA100220@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-04-03T18:03:36Z","receivedAt":"2018-04-03T18:03:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/03, Derrick Stolee wrote:\n> This is the first of several \"small\" patches that follow the serialized\n> Git commit graph patch (ds/commit-graph).\n> \n> As described in Documentation/technical/commit-graph.txt, the generation\n> number of a commit is one more than the maximum generation number among\n> its parents (trivially, a commit with no parents has generation number\n> one).\n\nThanks for ensuring that this is defined and documented somewhere :)\n\n> \n> This series makes the computation of generation numbers part of the\n> commit-graph write process.\n> \n> Finally, generation numbers are used to order commits in the priority\n> queue in paint_down_to_common(). This allows a constant-time check in\n> queue_has_nonstale() instead of the previous linear-time check.\n> \n> This does not have a significant performance benefit in repositories\n> of normal size, but in the Windows repository, some merge-base\n> calculations improve from 3.1s to 2.9s. A modest speedup, but provides\n> an actual consumer of generation numbers as a starting point.\n> \n> A more substantial refactoring of revision.c is required before making\n> 'git log --graph' use generation numbers effectively.\n\nlog --graph should benefit a lot more from this correct? I know we've\ntalked a bit about negotiation and I wonder if these generation numbers\nshould be able to help out a little bit with that some day.\n\n> \n> This patch series depends on v7 of ds/commit-graph.\n> \n> Derrick Stolee (6):\n> object.c: parse commit in graph first\n> commit: add generation number to struct commmit\n> commit-graph: compute generation numbers\n> commit: sort by generation number in paint_down_to_common()\n> commit.c: use generation number to stop merge-base walks\n> commit-graph.txt: update design doc with generation numbers\n> \n> Documentation/technical/commit-graph.txt | 7 +---\n> alloc.c | 1 +\n> commit-graph.c | 48 +++++++++++++++++++++\n> commit.c | 53 ++++++++++++++++++++----\n> commit.h | 7 +++-\n> object.c | 4 +-\n> 6 files changed, 104 insertions(+), 16 deletions(-)\n> \n> -- \n> 2.17.0.20.g9f30ba16e1\n> \n\n-- \nBrandon Williams\n"},{"id":"343735","messageId":"20180403180536.GB100220@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-3-dstolee@microsoft.com","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-04-03T18:05:36Z","receivedAt":"2018-04-03T18:05:42Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/03, Derrick Stolee wrote:\n> The generation number of a commit is defined recursively as follows:\n> \n> * If a commit A has no parents, then the generation number of A is one.\n> * If a commit A has parents, then the generation number of A is one\n> more than the maximum generation number among the parents of A.\n> \n> Add a uint32_t generation field to struct commit so we can pass this\n\nIs there any reason to believe this would be too small of a value in the\nfuture? Or is a 32 bit unsigned good enough?\n\n> information to revision walks. We use two special values to signal\n> the generation number is invalid:\n> \n> GENERATION_NUMBER_UNDEF 0xFFFFFFFF\n> GENERATION_NUMBER_NONE 0\n> \n> The first (_UNDEF) means the generation number has not been loaded or\n> computed. The second (_NONE) means the generation number was loaded\n> from a commit graph file that was stored before generation numbers\n> were computed.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> alloc.c | 1 +\n> commit-graph.c | 2 ++\n> commit.h | 3 +++\n> 3 files changed, 6 insertions(+)\n> \n> diff --git a/alloc.c b/alloc.c\n> index cf4f8b61e1..1a62e85ac3 100644\n> --- a/alloc.c\n> +++ b/alloc.c\n> @@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n> \tc->object.type = OBJ_COMMIT;\n> \tc->index = alloc_commit_index();\n> \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n> +\tc->generation = GENERATION_NUMBER_UNDEF;\n> \treturn c;\n> }\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 1fc63d541b..d24b947525 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -264,6 +264,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \tdate_low = get_be32(commit_data + g->hash_len + 12);\n> \titem->date = (timestamp_t)((date_high << 32) | date_low);\n> \n> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n> +\n> \tpptr = &item->parents;\n> \n> \tedge_value = get_be32(commit_data + g->hash_len);\n> diff --git a/commit.h b/commit.h\n> index e57ae4b583..3cadd386f3 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -10,6 +10,8 @@\n> #include \"pretty.h\"\n> \n> #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n> +#define GENERATION_NUMBER_UNDEF 0xFFFFFFFF\n> +#define GENERATION_NUMBER_NONE 0\n> \n> struct commit_list {\n> \tstruct commit *item;\n> @@ -24,6 +26,7 @@ struct commit {\n> \tstruct commit_list *parents;\n> \tstruct tree *tree;\n> \tuint32_t graph_pos;\n> +\tuint32_t generation;\n> };\n> \n> extern int save_commit_buffer;\n> -- \n> 2.17.0.rc0\n> \n\n-- \nBrandon Williams\n"},{"id":"343736","messageId":"20180403112136.db2aeda65afe0c09f262dfb9@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-2-dstolee@microsoft.com","subject":"Re: [PATCH 1/6] object.c: parse commit in graph first","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-03T18:21:36Z","receivedAt":"2018-04-03T18:21:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Apr 2018 12:51:38 -0400\nDerrick Stolee wrote:\n\n> Most code paths load commits using lookup_commit() and then\n> parse_commit(). In some cases, including some branch lookups, the commit\n> is parsed using parse_object_buffer() which side-steps parse_commit() in\n> favor of parse_commit_buffer().\n> \n> Before adding generation numbers to the commit-graph, we need to ensure\n> that any commit that exists in the graph is loaded from the graph, so\n> check parse_commit_in_graph() before calling parse_commit_buffer().\n> \n> Signed-off-by: Derrick Stolee \n\nModifying parse_object_buffer() is the most pragmatic way to accomplish\nthis, but this also means that parse_object_buffer() now potentially\nreads from the local object store (instead of only relying on what's in\nmemory and what's in the provided buffer). parse_object_buffer() is\ncalled by several callers including in builtin/fsck.c. I would feel more\ncomfortable if the relevant [1] caller to parse_object_buffer() was\nmodified instead of parse_object_buffer(), but I'll let others give\ntheir opinions too.\n\n[1] The caller which, if modified, will result in the speedup to\nthe merge-base calculations in the Windows repository you describe in\nyour cover letter.\n"},{"id":"343738","messageId":"20180403112452.96ebcfcdfd68181d1d269171@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-3-dstolee@microsoft.com","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-03T18:24:52Z","receivedAt":"2018-04-03T18:25:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Apr 2018 12:51:39 -0400\nDerrick Stolee wrote:\n\n> The generation number of a commit is defined recursively as follows:\n> \n> * If a commit A has no parents, then the generation number of A is one.\n> * If a commit A has parents, then the generation number of A is one\n> more than the maximum generation number among the parents of A.\n> \n> Add a uint32_t generation field to struct commit so we can pass this\n> information to revision walks. We use two special values to signal\n> the generation number is invalid:\n> \n> GENERATION_NUMBER_UNDEF 0xFFFFFFFF\n> GENERATION_NUMBER_NONE 0\n> \n> The first (_UNDEF) means the generation number has not been loaded or\n> computed. The second (_NONE) means the generation number was loaded\n> from a commit graph file that was stored before generation numbers\n> were computed.\n> \n> Signed-off-by: Derrick Stolee \n\nThis looks straightforward and correct, thanks. I think some of the\ndescription above should appear as code comments.\n\n> +#define GENERATION_NUMBER_UNDEF 0xFFFFFFFF\n> +#define GENERATION_NUMBER_NONE 0\n\nI would include the description above here as documentation, and would\nreplace \"was stored before generation numbers were computed\" by \"was\nwritten by a version of Git that did not support generation numbers\".\n"},{"id":"343739","messageId":"20180403182800.GA8377@sigill.intra.peff.net","threadId":"48212","inReplyTo":"20180403180536.GB100220@google.com","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-03T18:28:00Z","receivedAt":"2018-04-03T18:28:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 11:05:36AM -0700, Brandon Williams wrote:\n\n> On 04/03, Derrick Stolee wrote:\n> > The generation number of a commit is defined recursively as follows:\n> > \n> > * If a commit A has no parents, then the generation number of A is one.\n> > * If a commit A has parents, then the generation number of A is one\n> > more than the maximum generation number among the parents of A.\n> > \n> > Add a uint32_t generation field to struct commit so we can pass this\n> \n> Is there any reason to believe this would be too small of a value in the\n> future? Or is a 32 bit unsigned good enough?\n\nThe linux kernel took ~10 years to produce 500k commits. Even assuming\nthose were all linear (and they're not), that gives us ~80,000 years of\nleeway. So even if the pace of development speeds up or we have a\nquicker project, it still seems we have a pretty reasonable safety\nmargin.\n\n-Peff\n"},{"id":"343740","messageId":"20180403182846.GB8377@sigill.intra.peff.net","threadId":"48212","inReplyTo":"20180403112136.db2aeda65afe0c09f262dfb9@google.com","subject":"Re: [PATCH 1/6] object.c: parse commit in graph first","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-03T18:28:46Z","receivedAt":"2018-04-03T18:28:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 11:21:36AM -0700, Jonathan Tan wrote:\n\n> On Tue, 3 Apr 2018 12:51:38 -0400\n> Derrick Stolee wrote:\n> \n> > Most code paths load commits using lookup_commit() and then\n> > parse_commit(). In some cases, including some branch lookups, the commit\n> > is parsed using parse_object_buffer() which side-steps parse_commit() in\n> > favor of parse_commit_buffer().\n> > \n> > Before adding generation numbers to the commit-graph, we need to ensure\n> > that any commit that exists in the graph is loaded from the graph, so\n> > check parse_commit_in_graph() before calling parse_commit_buffer().\n> > \n> > Signed-off-by: Derrick Stolee \n> \n> Modifying parse_object_buffer() is the most pragmatic way to accomplish\n> this, but this also means that parse_object_buffer() now potentially\n> reads from the local object store (instead of only relying on what's in\n> memory and what's in the provided buffer). parse_object_buffer() is\n> called by several callers including in builtin/fsck.c. I would feel more\n> comfortable if the relevant [1] caller to parse_object_buffer() was\n> modified instead of parse_object_buffer(), but I'll let others give\n> their opinions too.\n\nIt's not just you. This seems like a really odd place to put it.\nEspecially because if we have the buffer to pass to this function, then\nwe'd already have incurred the cost to inflate the object.\n\n-Peff\n"},{"id":"343741","messageId":"f8e171f9-0936-1412-81f9-1149fd2c5a23@gmail.com","threadId":"48212","inReplyTo":"20180403180336.GA100220@google.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-03T18:29:01Z","receivedAt":"2018-04-03T18:29:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/3/2018 2:03 PM, Brandon Williams wrote:\n> On 04/03, Derrick Stolee wrote:\n>> This is the first of several \"small\" patches that follow the serialized\n>> Git commit graph patch (ds/commit-graph).\n>>\n>> As described in Documentation/technical/commit-graph.txt, the generation\n>> number of a commit is one more than the maximum generation number among\n>> its parents (trivially, a commit with no parents has generation number\n>> one).\n> Thanks for ensuring that this is defined and documented somewhere :)\n>\n>> This series makes the computation of generation numbers part of the\n>> commit-graph write process.\n>>\n>> Finally, generation numbers are used to order commits in the priority\n>> queue in paint_down_to_common(). This allows a constant-time check in\n>> queue_has_nonstale() instead of the previous linear-time check.\n>>\n>> This does not have a significant performance benefit in repositories\n>> of normal size, but in the Windows repository, some merge-base\n>> calculations improve from 3.1s to 2.9s. A modest speedup, but provides\n>> an actual consumer of generation numbers as a starting point.\n>>\n>> A more substantial refactoring of revision.c is required before making\n>> 'git log --graph' use generation numbers effectively.\n> log --graph should benefit a lot more from this correct? I know we've\n> talked a bit about negotiation and I wonder if these generation numbers\n> should be able to help out a little bit with that some day.\n\n'log --graph' should be a HUGE speedup, when it is refactored. Since the \ntopo-order can \"stream\" commits to the pager, it can be very responsive \nto return the graph in almost all conditions. (The case where generation \nnumbers are not enough is when filters reduce the set of displayed \ncommits to be very sparse, so many commits are walked anyway.)\n\nIf we have generic \"can X reach Y?\" queries, then we can also use \ngeneration numbers there to great effect (by not walking commits Z with \ngen(Z) <= gen(Y)). Perhaps I should look at that \"git branch --contains\" \nthread for ideas.\n\nFor negotiation, there are some things we can do here. VSTS uses \ngeneration numbers as a heuristic for determining \"all wants connected \nto haves\" which is a condition for halting negotiation. The idea is very \nsimple, and I'd be happy to discuss it on a separate thread.\n\nThanks,\n-Stolee\n"},{"id":"343742","messageId":"20180403113010.170ef6a14b0c126f99651bc4@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-4-dstolee@microsoft.com","subject":"Re: [PATCH 3/6] commit-graph: compute generation numbers","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-03T18:30:10Z","receivedAt":"2018-04-03T18:30:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Apr 2018 12:51:40 -0400\nDerrick Stolee wrote:\n\n> +\t\tif ((*list)->generation != GENERATION_NUMBER_UNDEF) {\n> +\t\t\tif ((*list)->generation > GENERATION_NUMBER_MAX)\n> +\t\t\t\tdie(\"generation number %u is too large to store in commit-graph\",\n> +\t\t\t\t (*list)->generation);\n> +\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n> +\t\t}\n\nThe die() should have \"BUG:\" if you agree with my comment below.\n\n> +static void compute_generation_numbers(struct commit** commits,\n> +\t\t\t\t int nr_commits)\n\nStyle: space before **, not after.\n\n> +\t\t\tif (all_parents_computed) {\n> +\t\t\t\tcurrent->generation = max_generation + 1;\n> +\t\t\t\tpop_commit(&list);\n> +\t\t\t}\n\nI think the current->generation should be clamped to _MAX here. If we do, then\nthe die() I mentioned in my first comment will have \"BUG:\", since we are never\nmeant to write any number larger than _MAX in ->generation.\n"},{"id":"343743","messageId":"2b5a08da-6b15-194e-4336-ce7c84240a9b@gmail.com","threadId":"48212","inReplyTo":"20180403182800.GA8377@sigill.intra.peff.net","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-03T18:31:18Z","receivedAt":"2018-04-03T18:31:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/3/2018 2:28 PM, Jeff King wrote:\n> On Tue, Apr 03, 2018 at 11:05:36AM -0700, Brandon Williams wrote:\n>\n>> On 04/03, Derrick Stolee wrote:\n>>> The generation number of a commit is defined recursively as follows:\n>>>\n>>> * If a commit A has no parents, then the generation number of A is one.\n>>> * If a commit A has parents, then the generation number of A is one\n>>> more than the maximum generation number among the parents of A.\n>>>\n>>> Add a uint32_t generation field to struct commit so we can pass this\n>> Is there any reason to believe this would be too small of a value in the\n>> future? Or is a 32 bit unsigned good enough?\n> The linux kernel took ~10 years to produce 500k commits. Even assuming\n> those were all linear (and they're not), that gives us ~80,000 years of\n> leeway. So even if the pace of development speeds up or we have a\n> quicker project, it still seems we have a pretty reasonable safety\n> margin.\n\nThat, and larger projects do not have linear histories. Despite having \nalmost 2 million reachable commits, the Windows repository has maximum \ngeneration number ~100,000.\n\nThanks,\n-Stolee\n"},{"id":"343744","messageId":"CAGZ79kYt0VaLN=97+BT-T28W5bpTsach4XcB0_-p3ac5h1Vdgg@mail.gmail.com","threadId":"48212","inReplyTo":"20180403165143.80661-5-dstolee@microsoft.com","subject":"Re: [PATCH 4/6] commit: use generations in paint_down_to_common()","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-03T18:31:50Z","receivedAt":"2018-04-03T18:31:56Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Apr 3, 2018 at 9:51 AM, Derrick Stolee wrote:\n> Define compare_commits_by_gen_then_commit_date(), which uses generation\n> numbers as a primary comparison and commit date to break ties (or as a\n> comparison when both commits do not have computed generation numbers).\n>\n> Since the commit-graph file is closed under reachability, we know that\n> all commits in the file have generation at most GENERATION_NUMBER_MAX\n> which is less than GENERATION_NUMBER_UNDEF.\n>\n> This change does not affect the number of commits that are walked during\n> the execution of paint_down_to_common(), only the order that those\n> commits are inspected. In the case that commit dates violate topological\n> order (i.e. a parent is \"newer\" than a child), the previous code could\n> walk a commit twice: if a commit is reached with the PARENT1 bit, but\n> later is re-visited with the PARENT2 bit, then that PARENT2 bit must be\n> propagated to its parents. Using generation numbers avoids this extra\n> effort, even if it is somewhat rare.\n\n\nThis patch (or later in this series) may want to touch\nDocumentation/technical/commit-graph.txt, that mentions this in\nthe section of Future Work:\n\n- After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n priority queue with one ordered by generation number. The following\n operations are important candidates:\n\n - paint_down_to_common()\n - 'log --topo-order'\n\nThe paint down to common is only internal, not exposed to the user\nfor ordering, i.e. the topological ordering is still ordering commits in\na branch adjacent?\n\nThanks,\nStefan\n"},{"id":"343745","messageId":"20180403113154.4d81719baba74aae53c9a58c@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-5-dstolee@microsoft.com","subject":"Re: [PATCH 4/6] commit: use generations in paint_down_to_common()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-03T18:31:54Z","receivedAt":"2018-04-03T18:32:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Apr 2018 12:51:41 -0400\nDerrick Stolee wrote:\n\n> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n> +{\n> +\tconst struct commit *a = a_, *b = b_;\n> +\n> +\tif (a->generation < b->generation)\n> +\t\treturn 1;\n> +\telse if (a->generation > b->generation)\n> +\t\treturn -1;\n> +\n> +\t/* newer commits with larger date first */\n> +\tif (a->date < b->date)\n> +\t\treturn 1;\n> +\telse if (a->date > b->date)\n> +\t\treturn -1;\n> +\treturn 0;\n> +}\n\nI think it would be clearer if you commented above the first block\n\"newer commits first\", then on the second block, \"use date as a\nheuristic to determine newer commit\".\n\nOther than that, this looks good.\n"},{"id":"343746","messageId":"20180403183201.GC100220@google.com","threadId":"48212","inReplyTo":"20180403182800.GA8377@sigill.intra.peff.net","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-04-03T18:32:01Z","receivedAt":"2018-04-03T18:32:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/03, Jeff King wrote:\n> On Tue, Apr 03, 2018 at 11:05:36AM -0700, Brandon Williams wrote:\n> \n> > On 04/03, Derrick Stolee wrote:\n> > > The generation number of a commit is defined recursively as follows:\n> > > \n> > > * If a commit A has no parents, then the generation number of A is one.\n> > > * If a commit A has parents, then the generation number of A is one\n> > > more than the maximum generation number among the parents of A.\n> > > \n> > > Add a uint32_t generation field to struct commit so we can pass this\n> > \n> > Is there any reason to believe this would be too small of a value in the\n> > future? Or is a 32 bit unsigned good enough?\n> \n> The linux kernel took ~10 years to produce 500k commits. Even assuming\n> those were all linear (and they're not), that gives us ~80,000 years of\n> leeway. So even if the pace of development speeds up or we have a\n> quicker project, it still seems we have a pretty reasonable safety\n> margin.\n> \n> -Peff\n\nI figured as much, but just wanted to check since the windows folks\nseems to produce commits pretty quickly.\n\n-- \nBrandon Williams\n"},{"id":"343747","messageId":"04f9d212-58b0-df15-43e1-ce6320585291@gmail.com","threadId":"48212","inReplyTo":"20180403182846.GB8377@sigill.intra.peff.net","subject":"Re: [PATCH 1/6] object.c: parse commit in graph first","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-03T18:32:19Z","receivedAt":"2018-04-03T18:33:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 4/3/2018 2:28 PM, Jeff King wrote:\n> On Tue, Apr 03, 2018 at 11:21:36AM -0700, Jonathan Tan wrote:\n>\n>> On Tue, 3 Apr 2018 12:51:38 -0400\n>> Derrick Stolee wrote:\n>>\n>>> Most code paths load commits using lookup_commit() and then\n>>> parse_commit(). In some cases, including some branch lookups, the commit\n>>> is parsed using parse_object_buffer() which side-steps parse_commit() in\n>>> favor of parse_commit_buffer().\n>>>\n>>> Before adding generation numbers to the commit-graph, we need to ensure\n>>> that any commit that exists in the graph is loaded from the graph, so\n>>> check parse_commit_in_graph() before calling parse_commit_buffer().\n>>>\n>>> Signed-off-by: Derrick Stolee \n>> Modifying parse_object_buffer() is the most pragmatic way to accomplish\n>> this, but this also means that parse_object_buffer() now potentially\n>> reads from the local object store (instead of only relying on what's in\n>> memory and what's in the provided buffer). parse_object_buffer() is\n>> called by several callers including in builtin/fsck.c. I would feel more\n>> comfortable if the relevant [1] caller to parse_object_buffer() was\n>> modified instead of parse_object_buffer(), but I'll let others give\n>> their opinions too.\n> It's not just you. This seems like a really odd place to put it.\n> Especially because if we have the buffer to pass to this function, then\n> we'd already have incurred the cost to inflate the object.\n>\n\nOK. Thanks. I'll try to find the better place to put this check.\n\n-Stolee\n"},{"id":"343749","messageId":"CAGZ79kYeKmWidi0SqV_R6NhWrH1rT-rTXetRfZ_YBpycieFxgw@mail.gmail.com","threadId":"48212","inReplyTo":"20180403182800.GA8377@sigill.intra.peff.net","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-03T18:44:19Z","receivedAt":"2018-04-03T18:44:25Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Apr 3, 2018 at 11:28 AM, Jeff King wrote:\n> On Tue, Apr 03, 2018 at 11:05:36AM -0700, Brandon Williams wrote:\n>\n>> On 04/03, Derrick Stolee wrote:\n>> > The generation number of a commit is defined recursively as follows:\n>> >\n>> > * If a commit A has no parents, then the generation number of A is one.\n>> > * If a commit A has parents, then the generation number of A is one\n>> > more than the maximum generation number among the parents of A.\n>> >\n>> > Add a uint32_t generation field to struct commit so we can pass this\n>>\n>> Is there any reason to believe this would be too small of a value in the\n>> future? Or is a 32 bit unsigned good enough?\n>\n> The linux kernel took ~10 years to produce 500k commits. Even assuming\n> those were all linear (and they're not),\n\n... which you meant in terms of DAG, where a linear history is the worst case\nfor generation numbers.\n\nI first read it the other way round, as the best case w.r.t. timing\n\n~/linux$ git log --oneline |wc -l\n721223\n$ git log --oneline --since 2012 |wc -l\n421853\n$ git log --oneline --since 2011 |wc -l\n477155\n\nThe number of commits is growing exponentially, though the exponential\npart is very small and the YoY growth can be estimated using linear\ninterpolation.\n\nIn linux, the release is a natural synchronization point IIUC as well\nas on a regular schedule. So an interesting question to ask there would\nbe whether the delta in generation number goes up over time, or if the\nDAG just gets wider (=more parallel)\n\n> that gives us ~80,000 years of\n> leeway. So even if the pace of development speeds up or we have a\n> quicker project, it still seems we have a pretty reasonable safety\n> margin.\n\nThanks for the estimate.\nStefan\n"},{"id":"343750","messageId":"20180403184727.GC8377@sigill.intra.peff.net","threadId":"48212","inReplyTo":"f8e171f9-0936-1412-81f9-1149fd2c5a23@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-03T18:47:27Z","receivedAt":"2018-04-03T18:47:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 02:29:01PM -0400, Derrick Stolee wrote:\n\n> If we have generic \"can X reach Y?\" queries, then we can also use generation\n> numbers there to great effect (by not walking commits Z with gen(Z) <=\n> gen(Y)). Perhaps I should look at that \"git branch --contains\" thread for\n> ideas.\n\nI think the gist of it is the patch below. Which I hastily adapted from\nthe patch we run at GitHub that uses timestamps as a proxy. So it's\npossible I completely flubbed the logic. I'm assuming unavailable\ngeneration numbers are set to 0; the logic is actually a bit simpler if\nthey end up as (uint32_t)-1.\n\nAssuming it works, that would cover for-each-ref and tag. You'd probably\nwant to drop the \"with_commit_tag_algo\" flag in ref-filter.h, and just\nuse always use it by default (and that would cover \"git branch\").\n\n---\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 45fc56216a..6bea6173d1 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1584,7 +1584,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n */\n static enum contains_result contains_test(struct commit *candidate,\n \t\t\t\t\t const struct commit_list *want,\n-\t\t\t\t\t struct contains_cache *cache)\n+\t\t\t\t\t struct contains_cache *cache,\n+\t\t\t\t\t uint32_t cutoff)\n {\n \tenum contains_result *cached = contains_cache_at(cache, candidate);\n \n@@ -1598,8 +1599,11 @@ static enum contains_result contains_test(struct commit *candidate,\n \t\treturn CONTAINS_YES;\n \t}\n \n-\t/* Otherwise, we don't know; prepare to recurse */\n \tparse_commit_or_die(candidate);\n+\n+\tif (candidate->generation && candidate->generation < cutoff)\n+\t\treturn CONTAINS_NO;\n+\n \treturn CONTAINS_UNKNOWN;\n }\n \n@@ -1615,8 +1619,20 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t\t\t\t struct contains_cache *cache)\n {\n \tstruct contains_stack contains_stack = { 0, 0, NULL };\n-\tenum contains_result result = contains_test(candidate, want, cache);\n+\tenum contains_result result;\n+\tuint32_t cutoff = -1;\n+\tconst struct commit_list *p;\n+\n+\tfor (p = want; p; p = p->next) {\n+\t\tstruct commit *c = p->item;\n+\t\tparse_commit_or_die(c);\n+\t\tif (c->generation && c->generation < cutoff )\n+\t\t\tcutoff = c->generation;\n+\t}\n+\tif (cutoff == -1)\n+\t\tcutoff = 0;\n \n+\tresult = contains_test(candidate, want, cache, cutoff);\n \tif (result != CONTAINS_UNKNOWN)\n \t\treturn result;\n \n@@ -1634,7 +1650,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t * If we just popped the stack, parents->item has been marked,\n \t\t * therefore contains_test will return a meaningful yes/no.\n \t\t */\n-\t\telse switch (contains_test(parents->item, want, cache)) {\n+\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n \t\tcase CONTAINS_YES:\n \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n \t\t\tcontains_stack.nr--;\n@@ -1648,7 +1664,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t}\n \t}\n \tfree(contains_stack.contains_stack);\n-\treturn contains_test(candidate, want, cache);\n+\treturn contains_test(candidate, want, cache, cutoff);\n }\n \n static int commit_contains(struct ref_filter *filter, struct commit *commit,\n"},{"id":"343751","messageId":"CAGZ79kY1z5gYph3S4uiLZddjSmwPMAxyTCEWGaUsOZLbAu01xw@mail.gmail.com","threadId":"48212","inReplyTo":"20180403113010.170ef6a14b0c126f99651bc4@google.com","subject":"Re: [PATCH 3/6] commit-graph: compute generation numbers","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-03T18:49:27Z","receivedAt":"2018-04-03T18:49:33Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Apr 3, 2018 at 11:30 AM, Jonathan Tan wrote:\n> On Tue, 3 Apr 2018 12:51:40 -0400\n> Derrick Stolee wrote:\n>\n>> + if ((*list)->generation != GENERATION_NUMBER_UNDEF) {\n>> + if ((*list)->generation > GENERATION_NUMBER_MAX)\n>> + die(\"generation number %u is too large to store in commit-graph\",\n>> + (*list)->generation);\n>> + packedDate[0] |= htonl((*list)->generation << 2);\n>> + }\n>\n> The die() should have \"BUG:\" if you agree with my comment below.\n\nI would remove the BUG/die() altogether and keep going.\n(But do not write it out, i.e. warn and skip the next line)\n\nA degraded commit graph with partial generation numbers is better\nthan Git refusing to write any part of the commit graph (which later on\nwill be part of many maintenance operations I would think, leading to\nmore immediate headache rather than \"working but slightly slower\")\n\n>\n>> +static void compute_generation_numbers(struct commit** commits,\n>> + int nr_commits)\n>\n> Style: space before **, not after.\n>\n>> + if (all_parents_computed) {\n>> + current->generation = max_generation + 1;\n>> + pop_commit(&list);\n>> + }\n>\n> I think the current->generation should be clamped to _MAX here. If we do, then\n> the die() I mentioned in my first comment will have \"BUG:\", since we are never\n> meant to write any number larger than _MAX in ->generation.\n\nWhen we clamp here, we'd have to treat the _MAX specially\nin all our use cases or we'd encounter funny bugs due to miss ordered\ncommits later?\n"},{"id":"343755","messageId":"20180403120133.a1327fae0475a6ad1467e364@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-6-dstolee@microsoft.com","subject":"Re: [PATCH 5/6] commit.c: use generation to halt paint walk","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-03T19:01:33Z","receivedAt":"2018-04-03T19:01:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Apr 2018 12:51:42 -0400\nDerrick Stolee wrote:\n\n> -static int queue_has_nonstale(struct prio_queue *queue)\n> +static int queue_has_nonstale(struct prio_queue *queue, uint32_t min_gen)\n> {\n> -\tint i;\n> -\tfor (i = 0; i < queue->nr; i++) {\n> -\t\tstruct commit *commit = queue->array[i].data;\n> -\t\tif (!(commit->object.flags & STALE))\n> -\t\t\treturn 1;\n> +\tif (min_gen != GENERATION_NUMBER_UNDEF) {\n> +\t\tif (queue->nr > 0) {\n> +\t\t\tstruct commit *commit = queue->array[0].data;\n> +\t\t\treturn commit->generation >= min_gen;\n> +\t\t}\n\nThis only works if the prio_queue has\ncompare_commits_by_gen_then_commit_date. Also, I don't think that the\nmin_gen != GENERATION_NUMBER_UNDEF check is necessary. So I would write\nthis as:\n\n if (queue->compare == compare_commits_by_gen_then_commit_date &&\n queue->nr) {\n struct commit *commit = queue->array[0].data;\n return commit->generation >= min_gen;\n }\n for (i = 0 ...\n\nIf you'd rather not perform the comparison to\ncompare_commits_by_gen_then_commit_date every time you invoke\nqueue_has_nonstale(), that's fine with me too, but document somewhere\nthat queue_has_nonstale() only works if this comparison function is\nused.\n\n> +\t\tif (commit->generation > last_gen)\n> +\t\t\tBUG(\"bad generation skip\");\n> +\n> +\t\tlast_gen = commit->generation;\n\nlast_gen seems to only be used to ensure that the priority queue returns\nelements in the correct order - I think we can generally trust the\nqueue, and if we need to test it, we can do it elsewhere.\n"},{"id":"343756","messageId":"20180403120434.6153fb382d549b2521c06796@google.com","threadId":"48212","inReplyTo":"20180403165143.80661-7-dstolee@microsoft.com","subject":"Re: [PATCH 6/6] commit-graph.txt: update future work","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-03T19:04:34Z","receivedAt":"2018-04-03T19:04:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Apr 2018 12:51:43 -0400\nDerrick Stolee wrote:\n\n> We now calculate generation numbers in the commit-graph file and use\n> them in paint_down_to_common().\n\nFor completeness, I'll mention that I don't see any issues with this\npatch, of course.\n\nThanks for this series.\n"},{"id":"343757","messageId":"20180403190500.GA10930@sigill.intra.peff.net","threadId":"48212","inReplyTo":"20180403184727.GC8377@sigill.intra.peff.net","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-03T19:05:00Z","receivedAt":"2018-04-03T19:05:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 03, 2018 at 02:47:27PM -0400, Jeff King wrote:\n\n> On Tue, Apr 03, 2018 at 02:29:01PM -0400, Derrick Stolee wrote:\n> \n> > If we have generic \"can X reach Y?\" queries, then we can also use generation\n> > numbers there to great effect (by not walking commits Z with gen(Z) <=\n> > gen(Y)). Perhaps I should look at that \"git branch --contains\" thread for\n> > ideas.\n> \n> I think the gist of it is the patch below. Which I hastily adapted from\n> the patch we run at GitHub that uses timestamps as a proxy. So it's\n> possible I completely flubbed the logic. I'm assuming unavailable\n> generation numbers are set to 0; the logic is actually a bit simpler if\n> they end up as (uint32_t)-1.\n\nOh indeed, that is already the value of your UNDEF. So the patch is more\nlike this:\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 45fc56216a..b147b1d0ee 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1584,7 +1584,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n */\n static enum contains_result contains_test(struct commit *candidate,\n \t\t\t\t\t const struct commit_list *want,\n-\t\t\t\t\t struct contains_cache *cache)\n+\t\t\t\t\t struct contains_cache *cache,\n+\t\t\t\t\t uint32_t cutoff)\n {\n \tenum contains_result *cached = contains_cache_at(cache, candidate);\n \n@@ -1598,8 +1599,11 @@ static enum contains_result contains_test(struct commit *candidate,\n \t\treturn CONTAINS_YES;\n \t}\n \n-\t/* Otherwise, we don't know; prepare to recurse */\n \tparse_commit_or_die(candidate);\n+\n+\tif (candidate->generation < cutoff)\n+\t\treturn CONTAINS_NO;\n+\n \treturn CONTAINS_UNKNOWN;\n }\n \n@@ -1615,8 +1619,20 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t\t\t\t struct contains_cache *cache)\n {\n \tstruct contains_stack contains_stack = { 0, 0, NULL };\n-\tenum contains_result result = contains_test(candidate, want, cache);\n+\tenum contains_result result;\n+\tuint32_t cutoff = GENERATION_NUMBER_UNDEF;\n+\tconst struct commit_list *p;\n+\n+\tfor (p = want; p; p = p->next) {\n+\t\tstruct commit *c = p->item;\n+\t\tparse_commit_or_die(c);\n+\t\tif (c->generation < cutoff)\n+\t\t\tcutoff = c->generation;\n+\t}\n+\tif (cutoff == GENERATION_NUMBER_UNDEF)\n+\t\tcutoff = GENERATION_NUMBER_NONE;\n \n+\tresult = contains_test(candidate, want, cache, cutoff);\n \tif (result != CONTAINS_UNKNOWN)\n \t\treturn result;\n \n@@ -1634,7 +1650,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t * If we just popped the stack, parents->item has been marked,\n \t\t * therefore contains_test will return a meaningful yes/no.\n \t\t */\n-\t\telse switch (contains_test(parents->item, want, cache)) {\n+\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n \t\tcase CONTAINS_YES:\n \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n \t\t\tcontains_stack.nr--;\n@@ -1648,7 +1664,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t}\n \t}\n \tfree(contains_stack.contains_stack);\n-\treturn contains_test(candidate, want, cache);\n+\treturn contains_test(candidate, want, cache, cutoff);\n }\n \n static int commit_contains(struct ref_filter *filter, struct commit *commit,\n"},{"id":"343775","messageId":"08d3c9dd-3589-5c1c-4cea-982e8ed28f3e@ramsayjones.plus.com","threadId":"48212","inReplyTo":"20180403182800.GA8377@sigill.intra.peff.net","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-04-03T23:17:06Z","receivedAt":"2018-04-03T23:17:15Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 03/04/18 19:28, Jeff King wrote:\n> On Tue, Apr 03, 2018 at 11:05:36AM -0700, Brandon Williams wrote:\n> \n>> On 04/03, Derrick Stolee wrote:\n>>> The generation number of a commit is defined recursively as follows:\n>>>\n>>> * If a commit A has no parents, then the generation number of A is one.\n>>> * If a commit A has parents, then the generation number of A is one\n>>> more than the maximum generation number among the parents of A.\n>>>\n>>> Add a uint32_t generation field to struct commit so we can pass this\n>>\n>> Is there any reason to believe this would be too small of a value in the\n>> future? Or is a 32 bit unsigned good enough?\n> \n> The linux kernel took ~10 years to produce 500k commits. Even assuming\n> those were all linear (and they're not), that gives us ~80,000 years of\n> leeway. So even if the pace of development speeds up or we have a\n> quicker project, it still seems we have a pretty reasonable safety\n> margin.\n\nI didn't read the patches closely, but isn't it ~20,000 years?\n\nGiven that '#define GENERATION_NUMBER_MAX 0x3FFFFFFF', that is. ;-)\n\nATB,\nRamsay Jones\n\n\n"},{"id":"343776","messageId":"20180403231903.GA9107@sigill.intra.peff.net","threadId":"48212","inReplyTo":"08d3c9dd-3589-5c1c-4cea-982e8ed28f3e@ramsayjones.plus.com","subject":"Re: [PATCH 2/6] commit: add generation number to struct commmit","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-03T23:19:03Z","receivedAt":"2018-04-03T23:19:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 04, 2018 at 12:17:06AM +0100, Ramsay Jones wrote:\n\n> >> Is there any reason to believe this would be too small of a value in the\n> >> future? Or is a 32 bit unsigned good enough?\n> > \n> > The linux kernel took ~10 years to produce 500k commits. Even assuming\n> > those were all linear (and they're not), that gives us ~80,000 years of\n> > leeway. So even if the pace of development speeds up or we have a\n> > quicker project, it still seems we have a pretty reasonable safety\n> > margin.\n> \n> I didn't read the patches closely, but isn't it ~20,000 years?\n> \n> Given that '#define GENERATION_NUMBER_MAX 0x3FFFFFFF', that is. ;-)\n\nWhat, I'm supposed to read the patches before responding? Heresy.\n\n-Peff\n"},{"id":"343786","messageId":"20180404154554.140716-1-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403190500.GA10930@sigill.intra.peff.net","subject":"[PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-04T15:45:53Z","receivedAt":"2018-04-04T15:46:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"A commit A can reach a commit B only if the generation number of A\nis strictly larger than the generation number of B. This condition\nallows significantly short-circuiting commit-graph walks.\n\nUse generation number for '--contains' type queries.\n\nOn a copy of the Linux repository where HEAD is containd in v4.13\nbut no earlier tag, the command 'git tag --contains HEAD' had the\nfollowing peformance improvement:\n\nBefore: 0.81s\nAfter: 0.04s\nRel %: -95%\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n ref-filter.c | 26 +++++++++++++++++++++-----\n 1 file changed, 21 insertions(+), 5 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 45fc56216a..b147b1d0ee 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1584,7 +1584,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n */\n static enum contains_result contains_test(struct commit *candidate,\n \t\t\t\t\t const struct commit_list *want,\n-\t\t\t\t\t struct contains_cache *cache)\n+\t\t\t\t\t struct contains_cache *cache,\n+\t\t\t\t\t uint32_t cutoff)\n {\n \tenum contains_result *cached = contains_cache_at(cache, candidate);\n \n@@ -1598,8 +1599,11 @@ static enum contains_result contains_test(struct commit *candidate,\n \t\treturn CONTAINS_YES;\n \t}\n \n-\t/* Otherwise, we don't know; prepare to recurse */\n \tparse_commit_or_die(candidate);\n+\n+\tif (candidate->generation < cutoff)\n+\t\treturn CONTAINS_NO;\n+\n \treturn CONTAINS_UNKNOWN;\n }\n \n@@ -1615,8 +1619,20 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t\t\t\t struct contains_cache *cache)\n {\n \tstruct contains_stack contains_stack = { 0, 0, NULL };\n-\tenum contains_result result = contains_test(candidate, want, cache);\n+\tenum contains_result result;\n+\tuint32_t cutoff = GENERATION_NUMBER_UNDEF;\n+\tconst struct commit_list *p;\n+\n+\tfor (p = want; p; p = p->next) {\n+\t\tstruct commit *c = p->item;\n+\t\tparse_commit_or_die(c);\n+\t\tif (c->generation < cutoff)\n+\t\t\tcutoff = c->generation;\n+\t}\n+\tif (cutoff == GENERATION_NUMBER_UNDEF)\n+\t\tcutoff = GENERATION_NUMBER_NONE;\n \n+\tresult = contains_test(candidate, want, cache, cutoff);\n \tif (result != CONTAINS_UNKNOWN)\n \t\treturn result;\n \n@@ -1634,7 +1650,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t * If we just popped the stack, parents->item has been marked,\n \t\t * therefore contains_test will return a meaningful yes/no.\n \t\t */\n-\t\telse switch (contains_test(parents->item, want, cache)) {\n+\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n \t\tcase CONTAINS_YES:\n \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n \t\t\tcontains_stack.nr--;\n@@ -1648,7 +1664,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t}\n \t}\n \tfree(contains_stack.contains_stack);\n-\treturn contains_test(candidate, want, cache);\n+\treturn contains_test(candidate, want, cache, cutoff);\n }\n \n static int commit_contains(struct ref_filter *filter, struct commit *commit,\n-- \n2.17.0.rc0\n\n"},{"id":"343787","messageId":"20180404154554.140716-2-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180404154554.140716-1-dstolee@microsoft.com","subject":"[PATCH 8/6] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-04T15:45:54Z","receivedAt":"2018-04-04T15:46:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The containment algorithm for 'git branch --contains' is different\nfrom that for 'git tag --contains' in that it uses is_descendant_of()\ninstead of contains_tag_algo(). The expensive portion of the branch\nalgorithm is computing merge bases.\n\nWhen a commit-graph file exists with generation numbers computed,\nwe can avoid this merge-base calculation when the target commit has\na larger generation number than the target commits.\n\nPerformance tests were run on a copy of the Linux repository where\nHEAD is contained in v4.13 but no earlier tag. Also, all tags were\ncopied to branches and 'git branch --contains' was tested:\n\nBefore: 60.0s\nAfter: 0.4s\nRel %: -99.3%\n\nReported-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 858f4fdbc9..2566cba79f 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1059,12 +1059,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n {\n \tstruct commit_list *bases;\n \tint ret = 0, i;\n+\tuint32_t min_generation = GENERATION_NUMBER_UNDEF;\n \n \tif (parse_commit(commit))\n \t\treturn ret;\n-\tfor (i = 0; i < nr_reference; i++)\n+\tfor (i = 0; i < nr_reference; i++) {\n \t\tif (parse_commit(reference[i]))\n \t\t\treturn ret;\n+\t\tif (min_generation > reference[i]->generation)\n+\t\t\tmin_generation = reference[i]->generation;\n+\t}\n+\n+\tif (commit->generation > min_generation)\n+\t\treturn 0;\n \n \tbases = paint_down_to_common(commit, nr_reference, reference);\n \tif (commit->object.flags & PARENT2)\n-- \n2.17.0.rc0\n\n"},{"id":"343788","messageId":"14e4e1d3-5c77-dc7c-10d5-e9406b992f0e@gmail.com","threadId":"48212","inReplyTo":"20180404154554.140716-2-dstolee@microsoft.com","subject":"Re: [PATCH 8/6] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-04T15:48:42Z","receivedAt":"2018-04-04T15:48:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/4/2018 11:45 AM, Derrick Stolee wrote:\n> The containment algorithm for 'git branch --contains' is different\n> from that for 'git tag --contains' in that it uses is_descendant_of()\n> instead of contains_tag_algo(). The expensive portion of the branch\n> algorithm is computing merge bases.\n>\n> When a commit-graph file exists with generation numbers computed,\n> we can avoid this merge-base calculation when the target commit has\n> a larger generation number than the target commits.\n>\n> Performance tests were run on a copy of the Linux repository where\n> HEAD is contained in v4.13 but no earlier tag. Also, all tags were\n> copied to branches and 'git branch --contains' was tested:\n>\n> Before: 60.0s\n> After: 0.4s\n> Rel %: -99.3%\n>\n> Reported-by: Jeff King \n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 9 ++++++++-\n> 1 file changed, 8 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit.c b/commit.c\n> index 858f4fdbc9..2566cba79f 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -1059,12 +1059,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n> {\n> \tstruct commit_list *bases;\n> \tint ret = 0, i;\n> +\tuint32_t min_generation = GENERATION_NUMBER_UNDEF;\n> \n> \tif (parse_commit(commit))\n> \t\treturn ret;\n> -\tfor (i = 0; i < nr_reference; i++)\n> +\tfor (i = 0; i < nr_reference; i++) {\n> \t\tif (parse_commit(reference[i]))\n> \t\t\treturn ret;\n> +\t\tif (min_generation > reference[i]->generation)\n> +\t\t\tmin_generation = reference[i]->generation;\n> +\t}\n> +\n> +\tif (commit->generation > min_generation)\n> +\t\treturn 0;\n> \n> \tbases = paint_down_to_common(commit, nr_reference, reference);\n> \tif (commit->object.flags & PARENT2)\n\nThis patch may suffice to speed up 'git branch --contains' instead of \nneeding to always use the 'git tag --contains' algorithm as considered \nin [1].\n\nThanks,\n-Stolee\n\n[1] \nhttps://public-inbox.org/git/20180303051516.GE27689@sigill.intra.peff.net/\n     Re: [PATCH 0/4] Speed up git tag --contains\n"},{"id":"343792","messageId":"20180404170131.GD100220@google.com","threadId":"48212","inReplyTo":"14e4e1d3-5c77-dc7c-10d5-e9406b992f0e@gmail.com","subject":"Re: [PATCH 8/6] commit: use generation numbers for in_merge_bases()","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-04-04T17:01:31Z","receivedAt":"2018-04-04T17:01:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 04/04, Derrick Stolee wrote:\n> On 4/4/2018 11:45 AM, Derrick Stolee wrote:\n> > The containment algorithm for 'git branch --contains' is different\n> > from that for 'git tag --contains' in that it uses is_descendant_of()\n> > instead of contains_tag_algo(). The expensive portion of the branch\n> > algorithm is computing merge bases.\n> > \n> > When a commit-graph file exists with generation numbers computed,\n> > we can avoid this merge-base calculation when the target commit has\n> > a larger generation number than the target commits.\n> > \n> > Performance tests were run on a copy of the Linux repository where\n> > HEAD is contained in v4.13 but no earlier tag. Also, all tags were\n> > copied to branches and 'git branch --contains' was tested:\n> > \n> > Before: 60.0s\n> > After: 0.4s\n> > Rel %: -99.3%\n\nNow that is an impressive speedup.\n\n-- \nBrandon Williams\n"},{"id":"343798","messageId":"20180404182242.GA9055@sigill.intra.peff.net","threadId":"48212","inReplyTo":"20180404154554.140716-1-dstolee@microsoft.com","subject":"Re: [PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-04T18:22:42Z","receivedAt":"2018-04-04T18:22:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 04, 2018 at 11:45:53AM -0400, Derrick Stolee wrote:\n\n> @@ -1615,8 +1619,20 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> \t\t\t\t\t struct contains_cache *cache)\n> {\n> \tstruct contains_stack contains_stack = { 0, 0, NULL };\n> -\tenum contains_result result = contains_test(candidate, want, cache);\n> +\tenum contains_result result;\n> +\tuint32_t cutoff = GENERATION_NUMBER_UNDEF;\n> +\tconst struct commit_list *p;\n> +\n> +\tfor (p = want; p; p = p->next) {\n> +\t\tstruct commit *c = p->item;\n> +\t\tparse_commit_or_die(c);\n> +\t\tif (c->generation < cutoff)\n> +\t\t\tcutoff = c->generation;\n> +\t}\n> +\tif (cutoff == GENERATION_NUMBER_UNDEF)\n> +\t\tcutoff = GENERATION_NUMBER_NONE;\n\nHmm, on reflection, I'm not sure if this is right in the face of\nmultiple \"want\" commits, only some of which have generation numbers. We\nprobably want to disable the cutoff if _any_ \"want\" commit doesn't have\na number.\n\nThere's also an obvious corner case where this won't kick in, and you'd\nreally like it to: recently added commits. E.g,. if I do this:\n\n git gc ;# imagine this writes generation numbers\n git pull\n git tag --contains HEAD\n\nthen HEAD isn't going to have a generation number. But this is the case\nwhere we have the most to gain, since we could throw away all of the\nancient tags immediately upon seeing that their generation numbers are\nway less than that of HEAD.\n\nI wonder to what degree it's worth traversing to come up with a\ngeneration number for the \"want\" commits. If we walked, say, 50 commits\nto do it, you'd probably save a lot of work (since the alternative is\nwalking thousands of commits until you realize that some ancient \"v1.0\"\ntag is not useful).\n\nI'd actually go so far as to say that any amount of traversal is\ngenerally going to be worth it to come up with the correct generation\ncutoff here. You can come up with pathological cases where you only have\none really recent tag or something, but in practice every repository\nwhere performance is a concern is going to end up with refs much further\nback than it would take to reach the cutoff condition.\n\n-Peff\n"},{"id":"343799","messageId":"20180404182432.GB9055@sigill.intra.peff.net","threadId":"48212","inReplyTo":"14e4e1d3-5c77-dc7c-10d5-e9406b992f0e@gmail.com","subject":"Re: [PATCH 8/6] commit: use generation numbers for in_merge_bases()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-04T18:24:32Z","receivedAt":"2018-04-04T18:24:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 04, 2018 at 11:48:42AM -0400, Derrick Stolee wrote:\n\n> > diff --git a/commit.c b/commit.c\n> > index 858f4fdbc9..2566cba79f 100644\n> > --- a/commit.c\n> > +++ b/commit.c\n> > @@ -1059,12 +1059,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n> > {\n> > \tstruct commit_list *bases;\n> > \tint ret = 0, i;\n> > +\tuint32_t min_generation = GENERATION_NUMBER_UNDEF;\n> > \tif (parse_commit(commit))\n> > \t\treturn ret;\n> > -\tfor (i = 0; i < nr_reference; i++)\n> > +\tfor (i = 0; i < nr_reference; i++) {\n> > \t\tif (parse_commit(reference[i]))\n> > \t\t\treturn ret;\n> > +\t\tif (min_generation > reference[i]->generation)\n> > +\t\t\tmin_generation = reference[i]->generation;\n> > +\t}\n> > +\n> > +\tif (commit->generation > min_generation)\n> > +\t\treturn 0;\n> > \tbases = paint_down_to_common(commit, nr_reference, reference);\n> > \tif (commit->object.flags & PARENT2)\n> \n> This patch may suffice to speed up 'git branch --contains' instead of\n> needing to always use the 'git tag --contains' algorithm as considered in\n> [1].\n\nI'd have to do some timings, but I suspect we may want to switch to the\n\"tag --contains\" algorithm anyway. This still does N independent\nmerge-base operations, one per ref. So with enough refs, you're still\nbetter off throwing it all into one big traversal.\n\n-Peff\n"},{"id":"343802","messageId":"aa28c3e5-7645-ffe0-78a3-e3cbd2ea8cec@gmail.com","threadId":"48212","inReplyTo":"20180404182432.GB9055@sigill.intra.peff.net","subject":"Re: [PATCH 8/6] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-04T18:53:45Z","receivedAt":"2018-04-04T18:54:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/4/2018 2:24 PM, Jeff King wrote:\n> On Wed, Apr 04, 2018 at 11:48:42AM -0400, Derrick Stolee wrote:\n>\n>>> diff --git a/commit.c b/commit.c\n>>> index 858f4fdbc9..2566cba79f 100644\n>>> --- a/commit.c\n>>> +++ b/commit.c\n>>> @@ -1059,12 +1059,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n>>> {\n>>> \tstruct commit_list *bases;\n>>> \tint ret = 0, i;\n>>> +\tuint32_t min_generation = GENERATION_NUMBER_UNDEF;\n>>> \tif (parse_commit(commit))\n>>> \t\treturn ret;\n>>> -\tfor (i = 0; i < nr_reference; i++)\n>>> +\tfor (i = 0; i < nr_reference; i++) {\n>>> \t\tif (parse_commit(reference[i]))\n>>> \t\t\treturn ret;\n>>> +\t\tif (min_generation > reference[i]->generation)\n>>> +\t\t\tmin_generation = reference[i]->generation;\n>>> +\t}\n>>> +\n>>> +\tif (commit->generation > min_generation)\n>>> +\t\treturn 0;\n>>> \tbases = paint_down_to_common(commit, nr_reference, reference);\n>>> \tif (commit->object.flags & PARENT2)\n>> This patch may suffice to speed up 'git branch --contains' instead of\n>> needing to always use the 'git tag --contains' algorithm as considered in\n>> [1].\n\nI guess I want to specify: the only reason to NOT switch to the tags \nalgorithm is because it _may_ hurt existing cases in certain data shapes...\n\n> I'd have to do some timings, but I suspect we may want to switch to the\n> \"tag --contains\" algorithm anyway. This still does N independent\n> merge-base operations, one per ref. So with enough refs, you're still\n> better off throwing it all into one big traversal.\n\n...and I suppose your timings are to find out if there are data shapes \nwhere the branch algorithm is faster. Perhaps that is impossible now \nthat we have the generation number cutoff for the tag algorithm.\n\nSince the branch algorithm checks generation numbers before triggering \npain_down_to_common(), we will do N independent merge-base calculations, \nwhere N is the number of branches with large enough generation numbers \n(which is why my test does so well: most are below the target generation \nnumber). This doesn't help at all if none of the refs are in the graph.\n\nThe other thing to do is add a minimum generation for the walk in \npaint_down_to_common() so even if commit->generation <= min_generation \nwe still only walk down to commit->generation instead of all merge \nbases. This is something we could change in a later patch.\n\nPatches 7 and 8 seem to me like simple changes with no downside UNLESS \nwe are deciding instead to delete the code I'm changing.\n\nThanks,\n-Stolee\n"},{"id":"343805","messageId":"20180404185950.GF9055@sigill.intra.peff.net","threadId":"48212","inReplyTo":"aa28c3e5-7645-ffe0-78a3-e3cbd2ea8cec@gmail.com","subject":"Re: [PATCH 8/6] commit: use generation numbers for in_merge_bases()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-04T18:59:50Z","receivedAt":"2018-04-04T19:00:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 04, 2018 at 02:53:45PM -0400, Derrick Stolee wrote:\n\n> > I'd have to do some timings, but I suspect we may want to switch to the\n> > \"tag --contains\" algorithm anyway. This still does N independent\n> > merge-base operations, one per ref. So with enough refs, you're still\n> > better off throwing it all into one big traversal.\n> \n> ...and I suppose your timings are to find out if there are data shapes where\n> the branch algorithm is faster. Perhaps that is impossible now that we have\n> the generation number cutoff for the tag algorithm.\n\nWell, I wanted to show the opposite: that the branch algorithm can still\nperform quite poorly. :)\n\nI think with generation numbers that the tag algorithm should always\nperform better, since you can't walk past a merge base when using a\ncutoff. But it could definitely perform worse in a case where you don't\nhave generation numbers.\n\n> Patches 7 and 8 seem to me like simple changes with no downside UNLESS we\n> are deciding instead to delete the code I'm changing.\n\nYeah, I think they are strict improvements modulo the inverted UNDEF\nlogic I mentioned.\n\n-Peff\n"},{"id":"343806","messageId":"1cc07683-8d2e-e9fa-ffee-2663057704be@gmail.com","threadId":"48212","inReplyTo":"20180404182242.GA9055@sigill.intra.peff.net","subject":"Re: [PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-04T19:06:26Z","receivedAt":"2018-04-04T19:06:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/4/2018 2:22 PM, Jeff King wrote:\n> On Wed, Apr 04, 2018 at 11:45:53AM -0400, Derrick Stolee wrote:\n>\n>> @@ -1615,8 +1619,20 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n>> \t\t\t\t\t struct contains_cache *cache)\n>> {\n>> \tstruct contains_stack contains_stack = { 0, 0, NULL };\n>> -\tenum contains_result result = contains_test(candidate, want, cache);\n>> +\tenum contains_result result;\n>> +\tuint32_t cutoff = GENERATION_NUMBER_UNDEF;\n>> +\tconst struct commit_list *p;\n>> +\n>> +\tfor (p = want; p; p = p->next) {\n>> +\t\tstruct commit *c = p->item;\n>> +\t\tparse_commit_or_die(c);\n>> +\t\tif (c->generation < cutoff)\n>> +\t\t\tcutoff = c->generation;\n>> +\t}\n\nNow that you mention it, let me split out the portion you are probably \ntalking about as incorrect:\n\n>> +\tif (cutoff == GENERATION_NUMBER_UNDEF)\n>> +\t\tcutoff = GENERATION_NUMBER_NONE;\n\nYou're right, we don't want this. Since GENERATION_NUMBER_NONE == 0, we \nget no benefit from this. If we keep it GENERATION_NUMBER_UNDEF, then \nour walk will be limited to commits NOT in the commit-graph (which we \nhope is small if proper hygiene is followed).\n\n> Hmm, on reflection, I'm not sure if this is right in the face of\n> multiple \"want\" commits, only some of which have generation numbers. We\n> probably want to disable the cutoff if _any_ \"want\" commit doesn't have\n> a number.\n>\n> There's also an obvious corner case where this won't kick in, and you'd\n> really like it to: recently added commits. E.g,. if I do this:\n>\n> git gc ;# imagine this writes generation numbers\n> git pull\n> git tag --contains HEAD\n>\n> then HEAD isn't going to have a generation number. But this is the case\n> where we have the most to gain, since we could throw away all of the\n> ancient tags immediately upon seeing that their generation numbers are\n> way less than that of HEAD.\n>\n> I wonder to what degree it's worth traversing to come up with a\n> generation number for the \"want\" commits. If we walked, say, 50 commits\n> to do it, you'd probably save a lot of work (since the alternative is\n> walking thousands of commits until you realize that some ancient \"v1.0\"\n> tag is not useful).\n>\n> I'd actually go so far as to say that any amount of traversal is\n> generally going to be worth it to come up with the correct generation\n> cutoff here. You can come up with pathological cases where you only have\n> one really recent tag or something, but in practice every repository\n> where performance is a concern is going to end up with refs much further\n> back than it would take to reach the cutoff condition.\n\nPerhaps there is some value in walking to find the correct cutoff value, \nbut it is difficult to determine how far we are from commits with \ncorrect generation numbers _a priori_. I'd rather rely on the \ncommit-graph being in a good state, not too far behind the refs. An \nadded complexity of computing generation numbers dynamically is that we \nwould need to add a dependence on the commit-graph file's existence at all.\n\nThanks,\n-Stolee\n"},{"id":"343807","messageId":"20180404191657.GA13543@sigill.intra.peff.net","threadId":"48212","inReplyTo":"1cc07683-8d2e-e9fa-ffee-2663057704be@gmail.com","subject":"Re: [PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-04T19:16:57Z","receivedAt":"2018-04-04T19:17:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 04, 2018 at 03:06:26PM -0400, Derrick Stolee wrote:\n\n> > > @@ -1615,8 +1619,20 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> > > \t\t\t\t\t struct contains_cache *cache)\n> > > {\n> > > \tstruct contains_stack contains_stack = { 0, 0, NULL };\n> > > -\tenum contains_result result = contains_test(candidate, want, cache);\n> > > +\tenum contains_result result;\n> > > +\tuint32_t cutoff = GENERATION_NUMBER_UNDEF;\n> > > +\tconst struct commit_list *p;\n> > > +\n> > > +\tfor (p = want; p; p = p->next) {\n> > > +\t\tstruct commit *c = p->item;\n> > > +\t\tparse_commit_or_die(c);\n> > > +\t\tif (c->generation < cutoff)\n> > > +\t\t\tcutoff = c->generation;\n> > > +\t}\n> \n> Now that you mention it, let me split out the portion you are probably\n> talking about as incorrect:\n> \n> > > +\tif (cutoff == GENERATION_NUMBER_UNDEF)\n> > > +\t\tcutoff = GENERATION_NUMBER_NONE;\n> \n> You're right, we don't want this. Since GENERATION_NUMBER_NONE == 0, we get\n> no benefit from this. If we keep it GENERATION_NUMBER_UNDEF, then our walk\n> will be limited to commits NOT in the commit-graph (which we hope is small\n> if proper hygiene is followed).\n\nI think it's more than that. If we leave it at UNDEF, that's wrong,\nbecause contains_test() compares:\n\n candidate->generation < cutoff\n\nwhich would _always_ be true. In other words, we're saying that our\n\"want\" has an insanely high generation number, and traversing can never\nfind it. Which is clearly wrong.\n\nSo we have to put it at \"0\", to say \"you should always traverse, we\ncan't tell you that this is a dead end\". So that part of the logic is\ncurrently correct.\n\nBut what I was getting at is that the loop behavior can't just pick the\nmin cutoff. The min is effectively \"0\" if there's even a single ref for\nwhich we don't have a generation number, because we cannot ever stop\ntraversing (we might get to that commit if we kept going).\n\n(It's also possible I'm confused about how UNDEF and NONE are used; I'm\nassuming commits for which we don't have a generation number available\nwould get UNDEF in their commit->generation field).\n\nIf you could make the assumption that when we have a generation for\ncommit X, then we have a generation for all of its ancestors, things get\neasier. Because then if you hit commit X with a generation number and\nwant to compare it to a cutoff, you know that either:\n\n 1. The cutoff is defined, in which case you can stop traversing if\n we've gone past the cutoff.\n\n 2. The cutoff is undefined, in which case we cannot possibly reach\n our \"want\" by traversing. Even if it has a smaller generation\n number than us, it's on an unrelated line of development.\n\nI don't know that the reachability property is explicitly promised by\nyour work, but it seems like it would be a natural fallout (after all,\nyou have to know the generation of each ancestor in order to compute the\nlater ones, so you're really just promising that you've actually stored\nall the ones you've computed).\n\n> > I wonder to what degree it's worth traversing to come up with a\n> > generation number for the \"want\" commits. If we walked, say, 50 commits\n> > to do it, you'd probably save a lot of work (since the alternative is\n> > walking thousands of commits until you realize that some ancient \"v1.0\"\n> > tag is not useful).\n> > \n> > I'd actually go so far as to say that any amount of traversal is\n> > generally going to be worth it to come up with the correct generation\n> > cutoff here. You can come up with pathological cases where you only have\n> > one really recent tag or something, but in practice every repository\n> > where performance is a concern is going to end up with refs much further\n> > back than it would take to reach the cutoff condition.\n> \n> Perhaps there is some value in walking to find the correct cutoff value, but\n> it is difficult to determine how far we are from commits with correct\n> generation numbers _a priori_. I'd rather rely on the commit-graph being in\n> a good state, not too far behind the refs. An added complexity of computing\n> generation numbers dynamically is that we would need to add a dependence on\n> the commit-graph file's existence at all.\n\nIf you could make the reachability assumption, I think this question\njust goes away. As soon as you hit a commit with _any_ generation\nnumber, you could quit traversing down that path.\n\n-Peff\n"},{"id":"343808","messageId":"fcded90f-cb12-dafd-f21a-18ea67e99406@gmail.com","threadId":"48212","inReplyTo":"20180404191657.GA13543@sigill.intra.peff.net","subject":"Re: [PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-04T19:22:01Z","receivedAt":"2018-04-04T19:23:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/4/2018 3:16 PM, Jeff King wrote:\n> On Wed, Apr 04, 2018 at 03:06:26PM -0400, Derrick Stolee wrote:\n>\n>>>> @@ -1615,8 +1619,20 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n>>>> \t\t\t\t\t struct contains_cache *cache)\n>>>> {\n>>>> \tstruct contains_stack contains_stack = { 0, 0, NULL };\n>>>> -\tenum contains_result result = contains_test(candidate, want, cache);\n>>>> +\tenum contains_result result;\n>>>> +\tuint32_t cutoff = GENERATION_NUMBER_UNDEF;\n>>>> +\tconst struct commit_list *p;\n>>>> +\n>>>> +\tfor (p = want; p; p = p->next) {\n>>>> +\t\tstruct commit *c = p->item;\n>>>> +\t\tparse_commit_or_die(c);\n>>>> +\t\tif (c->generation < cutoff)\n>>>> +\t\t\tcutoff = c->generation;\n>>>> +\t}\n>> Now that you mention it, let me split out the portion you are probably\n>> talking about as incorrect:\n>>\n>>>> +\tif (cutoff == GENERATION_NUMBER_UNDEF)\n>>>> +\t\tcutoff = GENERATION_NUMBER_NONE;\n>> You're right, we don't want this. Since GENERATION_NUMBER_NONE == 0, we get\n>> no benefit from this. If we keep it GENERATION_NUMBER_UNDEF, then our walk\n>> will be limited to commits NOT in the commit-graph (which we hope is small\n>> if proper hygiene is followed).\n> I think it's more than that. If we leave it at UNDEF, that's wrong,\n> because contains_test() compares:\n>\n> candidate->generation < cutoff\n>\n> which would _always_ be true. In other words, we're saying that our\n> \"want\" has an insanely high generation number, and traversing can never\n> find it. Which is clearly wrong.\n\nThat condition is not always true (which is why we use strict comparison \ninstead of <=). If a commit is not in the commit-graph file, then its \ngeneration is equal to GENERATION_NUMBER_UNDEF, as shown in alloc.c:\n\nvoid *alloc_commit_node(void)\n{\n         struct commit *c = alloc_node(&commit_state, sizeof(struct \ncommit));\n         c->object.type = OBJ_COMMIT;\n         c->index = alloc_commit_index();\n         c->graph_pos = COMMIT_NOT_FROM_GRAPH;\n         c->generation = GENERATION_NUMBER_UNDEF;\n         return c;\n}\n\n\n> So we have to put it at \"0\", to say \"you should always traverse, we\n> can't tell you that this is a dead end\". So that part of the logic is\n> currently correct.\n>\n> But what I was getting at is that the loop behavior can't just pick the\n> min cutoff. The min is effectively \"0\" if there's even a single ref for\n> which we don't have a generation number, because we cannot ever stop\n> traversing (we might get to that commit if we kept going).\n>\n> (It's also possible I'm confused about how UNDEF and NONE are used; I'm\n> assuming commits for which we don't have a generation number available\n> would get UNDEF in their commit->generation field).\n\nI think it is this case.\n\n> If you could make the assumption that when we have a generation for\n> commit X, then we have a generation for all of its ancestors, things get\n> easier. Because then if you hit commit X with a generation number and\n> want to compare it to a cutoff, you know that either:\n>\n> 1. The cutoff is defined, in which case you can stop traversing if\n> we've gone past the cutoff.\n>\n> 2. The cutoff is undefined, in which case we cannot possibly reach\n> our \"want\" by traversing. Even if it has a smaller generation\n> number than us, it's on an unrelated line of development.\n>\n> I don't know that the reachability property is explicitly promised by\n> your work, but it seems like it would be a natural fallout (after all,\n> you have to know the generation of each ancestor in order to compute the\n> later ones, so you're really just promising that you've actually stored\n> all the ones you've computed).\n\nThe commit-graph is closed under reachability, so if a commit has a \ngeneration number then it is in the graph and so are all its ancestors.\n\nThe reason for GENERATION_NUMBER_NONE is that the commit-graph file \nstores \"0\" for generation number until this patch. It still satisfies \nthe condition that gen(A) < gen(B) if B can reach A, but also gives us a \ncondition for \"this commit still needs its generation number computed\".\n\n>\n>>> I wonder to what degree it's worth traversing to come up with a\n>>> generation number for the \"want\" commits. If we walked, say, 50 commits\n>>> to do it, you'd probably save a lot of work (since the alternative is\n>>> walking thousands of commits until you realize that some ancient \"v1.0\"\n>>> tag is not useful).\n>>>\n>>> I'd actually go so far as to say that any amount of traversal is\n>>> generally going to be worth it to come up with the correct generation\n>>> cutoff here. You can come up with pathological cases where you only have\n>>> one really recent tag or something, but in practice every repository\n>>> where performance is a concern is going to end up with refs much further\n>>> back than it would take to reach the cutoff condition.\n>> Perhaps there is some value in walking to find the correct cutoff value, but\n>> it is difficult to determine how far we are from commits with correct\n>> generation numbers _a priori_. I'd rather rely on the commit-graph being in\n>> a good state, not too far behind the refs. An added complexity of computing\n>> generation numbers dynamically is that we would need to add a dependence on\n>> the commit-graph file's existence at all.\n> If you could make the reachability assumption, I think this question\n> just goes away. As soon as you hit a commit with _any_ generation\n> number, you could quit traversing down that path.\nThat is the idea. I should make this clearer in all of my commit messages.\n\nThanks,\n-Stolee\n"},{"id":"343809","messageId":"20180404194248.GA14055@sigill.intra.peff.net","threadId":"48212","inReplyTo":"fcded90f-cb12-dafd-f21a-18ea67e99406@gmail.com","subject":"Re: [PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-04T19:42:49Z","receivedAt":"2018-04-04T19:42:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 04, 2018 at 03:22:01PM -0400, Derrick Stolee wrote:\n\n> > I don't know that the reachability property is explicitly promised by\n> > your work, but it seems like it would be a natural fallout (after all,\n> > you have to know the generation of each ancestor in order to compute the\n> > later ones, so you're really just promising that you've actually stored\n> > all the ones you've computed).\n> \n> The commit-graph is closed under reachability, so if a commit has a\n> generation number then it is in the graph and so are all its ancestors.\n\nOK, if we assume that it's closed, then I think we can effectively\nignore the UNDEF cases. They'll just work out. And then yes I'd agree\nthat the:\n\n if (cutoff == UNDEF)\n cutoff = NONE;\n\ncode is wrong. We'd want to keep it at UNDEF so we stop traversing at\nany generation number.\n\n> The reason for GENERATION_NUMBER_NONE is that the commit-graph file stores\n> \"0\" for generation number until this patch. It still satisfies the condition\n> that gen(A) < gen(B) if B can reach A, but also gives us a condition for\n> \"this commit still needs its generation number computed\".\n\nOK. I thought at first that would yield wrong results when comparing\nUNDEF to NONE, but I think for this kind of --contains traversal, it's\nstill OK (NONE is less than UNDEF, but we know that the UNDEF thing\ncannot be found by traversing from a NONE).\n\n> > If you could make the reachability assumption, I think this question\n> > just goes away. As soon as you hit a commit with _any_ generation\n> > number, you could quit traversing down that path.\n> That is the idea. I should make this clearer in all of my commit messages.\n\nYes, please. :) And maybe in the documentation of the file format, if\nit's not there (I didn't check). It's a very useful property, and we\nwant to make sure people making use of the graph know they can depend on\nit.\n\n-Peff\n"},{"id":"343810","messageId":"8a830161-ece7-44ba-e9f2-11073bd23802@gmail.com","threadId":"48212","inReplyTo":"20180404194248.GA14055@sigill.intra.peff.net","subject":"Re: [PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-04T19:45:30Z","receivedAt":"2018-04-04T19:45:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/4/2018 3:42 PM, Jeff King wrote:\n> On Wed, Apr 04, 2018 at 03:22:01PM -0400, Derrick Stolee wrote:\n>\n>> That is the idea. I should make this clearer in all of my commit messages.\n> Yes, please. :) And maybe in the documentation of the file format, if\n> it's not there (I didn't check). It's a very useful property, and we\n> want to make sure people making use of the graph know they can depend on\n> it.\n\nFor v2, I'll expand on the roles of _UNDEF and _NONE in the discussion \nof generation numbers in Documentation/technical/commit-graph.txt (the \ndesign doc instead of the file format).\n\nThanks,\n-Stolee\n"},{"id":"343811","messageId":"20180404194607.GA14395@sigill.intra.peff.net","threadId":"48212","inReplyTo":"8a830161-ece7-44ba-e9f2-11073bd23802@gmail.com","subject":"Re: [PATCH 7/6] ref-filter: use generation number for --contains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-04-04T19:46:07Z","receivedAt":"2018-04-04T19:46:21Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 04, 2018 at 03:45:30PM -0400, Derrick Stolee wrote:\n\n> On 4/4/2018 3:42 PM, Jeff King wrote:\n> > On Wed, Apr 04, 2018 at 03:22:01PM -0400, Derrick Stolee wrote:\n> > \n> > > That is the idea. I should make this clearer in all of my commit messages.\n> > Yes, please. :) And maybe in the documentation of the file format, if\n> > it's not there (I didn't check). It's a very useful property, and we\n> > want to make sure people making use of the graph know they can depend on\n> > it.\n> \n> For v2, I'll expand on the roles of _UNDEF and _NONE in the discussion of\n> generation numbers in Documentation/technical/commit-graph.txt (the design\n> doc instead of the file format).\n\nYeah, that makes sense. Thanks, and thanks for a thoughtful discussion.\nThe performance numbers are very exciting.\n\n-Peff\n"},{"id":"344070","messageId":"867epjez1n.fsf@gmail.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-07T16:55:48Z","receivedAt":"2018-04-07T16:56:02Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Hello,\n\nDerrick Stolee writes:\n\n> This is the first of several \"small\" patches that follow the serialized\n> Git commit graph patch (ds/commit-graph).\n>\n> As described in Documentation/technical/commit-graph.txt, the generation\n> number of a commit is one more than the maximum generation number among\n> its parents (trivially, a commit with no parents has generation number\n> one).\n>\n> This series makes the computation of generation numbers part of the\n> commit-graph write process.\n>\n> Finally, generation numbers are used [...].\n>\n> This does not have a significant performance benefit in repositories\n> of normal size, but in the Windows repository, some merge-base\n> calculations improve from 3.1s to 2.9s. A modest speedup, but provides\n> an actual consumer of generation numbers as a starting point.\n>\n> A more substantial refactoring of revision.c is required before making\n> 'git log --graph' use generation numbers effectively.\n\nI have started working on Jupyter Notebook on Google Colaboratory to\nfind out how much speedup we can get using generation numbers (level\nnegative-cut filter), FELINE index (negative-cut filter) and min-post\nintervals in some spanning tree (positive-cut filter, if I understand it\ncorrectly the base of GRAIL method) in commit graphs.\n\nCurrently I am at the stage of reproducing results in FELINE paper:\n\"Reachability Queries in Very Large Graphs: A Fast Refined Online Search\nApproach\" by Renê R. Veloso, Loïc Cerf, Wagner Meira Jr and Mohammed\nJ. Zaki (2014). This paper is available in the PDF form at\nhttps://openproceedings.org/EDBT/2014/paper_166.pdf\n\nThe Jupyter Notebook (which runs on Google cloud, but can be also run\nlocally) uses Python kernel, NetworkX librabry for graph manipulation,\nand matplotlib (via NetworkX) for display.\n\nAvailable at:\nhttps://colab.research.google.com/drive/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg\nhttps://drive.google.com/file/d/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg/view?usp=sharing\n\nI hope that could be of help, or at least interesting\n--\nJakub Narębski\n"},{"id":"344072","messageId":"863707eye2.fsf@gmail.com","threadId":"48212","inReplyTo":"f8e171f9-0936-1412-81f9-1149fd2c5a23@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-07T17:09:57Z","receivedAt":"2018-04-07T17:10:08Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 4/3/2018 2:03 PM, Brandon Williams wrote:\n>> On 04/03, Derrick Stolee wrote:\n>>> This is the first of several \"small\" patches that follow the serialized\n>>> Git commit graph patch (ds/commit-graph).\n>>>\n>>> As described in Documentation/technical/commit-graph.txt, the generation\n>>> number of a commit is one more than the maximum generation number among\n>>> its parents (trivially, a commit with no parents has generation number\n>>> one).\n[...]\n>>> A more substantial refactoring of revision.c is required before making\n>>> 'git log --graph' use generation numbers effectively.\n>>\n>> log --graph should benefit a lot more from this correct? I know we've\n>> talked a bit about negotiation and I wonder if these generation numbers\n>> should be able to help out a little bit with that some day.\n>\n> 'log --graph' should be a HUGE speedup, when it is refactored. Since\n> the topo-order can \"stream\" commits to the pager, it can be very\n> responsive to return the graph in almost all conditions. (The case\n> where generation numbers are not enough is when filters reduce the set\n> of displayed commits to be very sparse, so many commits are walked\n> anyway.)\n\nI wonder if next big speedup would be to store [some] topological\nordering of commits in the commit graph... It could be done for example\nin two chunks: a mapping to position in topological order, and list of\ncommits sorted in topological order.\n\nNote also that FELINE index uses (or can use -- but it is supposedly the\noptimal choice) position of vertex/node in topological order as one of\nthe two values in the pair that composes FELINE index.\n\n> If we have generic \"can X reach Y?\" queries, then we can also use\n> generation numbers there to great effect (by not walking commits Z\n> with gen(Z) <= gen(Y)). Perhaps I should look at that \"git branch\n> --contains\" thread for ideas.\n\nThis is something that is shown in the Google Colab [Jupyter] Notebook\nI have mentioned:\n\n https://colab.research.google.com/drive/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg\n https://drive.google.com/file/d/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg/view?usp=sharing\n\n> For negotiation, there are some things we can do here. VSTS uses\n> generation numbers as a heuristic for determining \"all wants connected\n> to haves\" which is a condition for halting negotiation. The idea is\n> very simple, and I'd be happy to discuss it on a separate thread.\n\nNice. How much speedup it gives?\n\nBest regards,\n--\nJakub Narębski\n"},{"id":"344082","messageId":"07abd2ce-e042-fe3c-b2db-3d2f2aa44de8@gmail.com","threadId":"48212","inReplyTo":"867epjez1n.fsf@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-08T01:06:24Z","receivedAt":"2018-04-08T01:06:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/7/2018 12:55 PM, Jakub Narebski wrote:\n> Currently I am at the stage of reproducing results in FELINE paper:\n> \"Reachability Queries in Very Large Graphs: A Fast Refined Online Search\n> Approach\" by Renê R. Veloso, Loïc Cerf, Wagner Meira Jr and Mohammed\n> J. Zaki (2014). This paper is available in the PDF form at\n> https://openproceedings.org/EDBT/2014/paper_166.pdf\n>\n> The Jupyter Notebook (which runs on Google cloud, but can be also run\n> locally) uses Python kernel, NetworkX librabry for graph manipulation,\n> and matplotlib (via NetworkX) for display.\n>\n> Available at:\n> https://colab.research.google.com/drive/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg\n> https://drive.google.com/file/d/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg/view?usp=sharing\n>\n> I hope that could be of help, or at least interesting\n\nLet me know when you can give numbers (either raw performance or # of \ncommits walked) for real-world Git commit graphs. The Linux repo is a \ngood example to use for benchmarking, but I also use the Kotlin repo \nsometimes as it has over a million objects and over 250K commits.\n\nOf course, the only important statistic at the end of the day is the \nend-to-end time of a 'git ...' command. Your investigations should \ninform whether it is worth prototyping the feature in the git codebase.\n\nThanks,\n\n-Stolee\n\n"},{"id":"344202","messageId":"20180409164131.37312-1-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180403165143.80661-1-dstolee@microsoft.com","subject":"[PATCH v2 00/10] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:41:48Z","receivedAt":"2018-04-09T16:41:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks for the lively discussion of this patch series in v1!\n\nI've incorporated the feedback from the previous round, added patches\n[7/6] and [8/6], expanded the discussion of generation numbers in the\ndesign document, and added another speedup for 'git branch --contains'.\n\nOne major difference: I renamed the macros from _UNDEF to _INFINITY and\n_NONE to _ZERO. This communicates their value more clearly, since the\nprevious names were unclear about which was larger than the \"real\"\ngeneration numbers.\n\nPatch 2 includes a change to builtin/merge.c and a new test in\nt5318-commit-graph.sh that exposes a problem I found when testing the\nprevious patch series on my box. The \"BUG: bad generation skip\" message\nfrom \"commit.c: use generation to halt paint walk\" would halt a fast-\nforward merge since the HEAD commit was loaded before the core.commitGraph\nconfig setting was loaded. It is crucial that all commits that exist\nin the commit-graph file are loaded from that file or else we will\nlose our expected inequalities of generation numbers.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nThis is the one of several \"small\" patches that follow the serialized\nGit commit graph patch (ds/commit-graph).\n\nAs described in Documentation/technical/commit-graph.txt, the generation\nnumber of a commit is one more than the maximum generation number among\nits parents (trivially, a commit with no parents has generation number\none). This section is expanded to describe the interaction with special\ngeneration numbers GENERATION_NUMBER_INFINITY (commits not in the commit-graph\nfile) and *_ZERO (commits in a commit-graph file written before generation\nnumbers were implemented).\n\nThis series makes the computation of generation numbers part of the\ncommit-graph write process.\n\nFinally, generation numbers are used to order commits in the priority\nqueue in paint_down_to_common(). This allows a constant-time check in\nqueue_has_nonstale() instead of the previous linear-time check.\n\nFurther, use generation numbers for '--contains' queries in 'git tag'\nand 'git branch', providing a significant speedup (at least 95% for\nsome cases).\n\nA more substantial refactoring of revision.c is required before making\n'git log --graph' use generation numbers effectively.\n\nThis patch series depends on v7 of ds/commit-graph.\n\nDerrick Stolee (10):\n object.c: parse commit in graph first\n merge: check config before loading commits\n commit: add generation number to struct commmit\n commit-graph: compute generation numbers\n commit: use generations in paint_down_to_common()\n commit.c: use generation to halt paint walk\n commit-graph.txt: update future work\n ref-filter: use generation number for --contains\n commit: use generation numbers for in_merge_bases()\n commit: add short-circuit to paint_down_to_common()\n\n Documentation/technical/commit-graph.txt | 50 +++++++++++++--\n alloc.c | 1 +\n builtin/merge.c | 5 +-\n commit-graph.c | 48 +++++++++++++++\n commit.c | 78 ++++++++++++++++++++----\n commit.h | 5 ++\n object.c | 4 +-\n ref-filter.c | 24 ++++++--\n t/t5318-commit-graph.sh | 9 +++\n 9 files changed, 197 insertions(+), 27 deletions(-)\n\n-- \n2.17.0\n\n"},{"id":"344203","messageId":"20180409164131.37312-2-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 01/10] object.c: parse commit in graph first","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:41:51Z","receivedAt":"2018-04-09T16:41:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Most code paths load commits using lookup_commit() and then\nparse_commit(). In some cases, including some branch lookups, the commit\nis parsed using parse_object_buffer() which side-steps parse_commit() in\nfavor of parse_commit_buffer().\n\nBefore adding generation numbers to the commit-graph, we need to ensure\nthat any commit that exists in the graph is loaded from the graph, so\ncheck parse_commit_in_graph() before calling parse_commit_buffer().\n\nSigned-off-by: Derrick Stolee \n---\n object.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/object.c b/object.c\nindex e6ad3f61f0..4cd3e98e04 100644\n--- a/object.c\n+++ b/object.c\n@@ -3,6 +3,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"commit.h\"\n+#include \"commit-graph.h\"\n #include \"tag.h\"\n \n static struct object **obj_hash;\n@@ -207,7 +208,8 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n \t} else if (type == OBJ_COMMIT) {\n \t\tstruct commit *commit = lookup_commit(oid);\n \t\tif (commit) {\n-\t\t\tif (parse_commit_buffer(commit, buffer, size))\n+\t\t\tif (!parse_commit_in_graph(commit) &&\n+\t\t\t parse_commit_buffer(commit, buffer, size))\n \t\t\t\treturn NULL;\n \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n \t\t\t\tset_commit_buffer(commit, buffer, size);\n-- \n2.17.0\n\n"},{"id":"344204","messageId":"20180409164131.37312-3-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 02/10] merge: check config before loading commits","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:41:57Z","receivedAt":"2018-04-09T16:42:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of using generation numbers from the commit-graph,\nwe must ensure that all commits that exist in the commit-graph are\nloaded from that file instead of from the object database. Since\nthe commit-graph file is only checked if core.commitGraph is true,\nwe must check the default config before we load any commits.\n\nIn the merge builtin, the config was checked after loading the HEAD\ncommit. This was due to the use of the global 'branch' when checking\nmerge-specific config settings.\n\nMove the config load to be between the initialization of 'branch'\nand the commit lookup. Also add a test to t5318-commit-graph.sh\nthat exercises this code path to prevent a regression.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/merge.c | 5 +++--\n t/t5318-commit-graph.sh | 9 +++++++++\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex ee050a47f3..20897f8223 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -1183,13 +1183,14 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n \tif (branch)\n \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n+\tinit_diff_ui_defaults();\n+\tgit_config(git_merge_config, NULL);\n+\n \tif (!branch || is_null_oid(&head_oid))\n \t\thead_commit = NULL;\n \telse\n \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n \n-\tinit_diff_ui_defaults();\n-\tgit_config(git_merge_config, NULL);\n \n \tif (branch_mergeoptions)\n \t\tparse_branch_merge_options(branch_mergeoptions);\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a380419b65..77d85aefe7 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n \n+test_expect_success 'perform fast-forward merge in full repo' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit checkout -b merge-5-to-8 commits/5 &&\n+\tgit merge commits/8 &&\n+\tgit show-ref -s merge-5-to-8 >output &&\n+\tgit show-ref -s commits/8 >expect &&\n+\ttest_cmp expect output\n+'\n+\n test_done\n-- \n2.17.0\n\n"},{"id":"344205","messageId":"20180409164131.37312-4-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 03/10] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:00Z","receivedAt":"2018-04-09T16:42:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The generation number of a commit is defined recursively as follows:\n\n* If a commit A has no parents, then the generation number of A is one.\n* If a commit A has parents, then the generation number of A is one\n more than the maximum generation number among the parents of A.\n\nAdd a uint32_t generation field to struct commit so we can pass this\ninformation to revision walks. We use two special values to signal\nthe generation number is invalid:\n\nGENERATION_NUMBER_ININITY 0xFFFFFFFF\nGENERATION_NUMBER_ZERO 0\n\nThe first (_INFINITY) means the generation number has not been loaded or\ncomputed. The second (_ZERO) means the generation number was loaded\nfrom a commit graph file that was stored before generation numbers\nwere computed.\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 2 ++\n commit.h | 4 ++++\n 3 files changed, 7 insertions(+)\n\ndiff --git a/alloc.c b/alloc.c\nindex cf4f8b61e1..e8ab14f4a1 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n+\tc->generation = GENERATION_NUMBER_INFINITY;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1fc63d541b..d24b947525 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -264,6 +264,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \tdate_low = get_be32(commit_data + g->hash_len + 12);\n \titem->date = (timestamp_t)((date_high << 32) | date_low);\n \n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+\n \tpptr = &item->parents;\n \n \tedge_value = get_be32(commit_data + g->hash_len);\ndiff --git a/commit.h b/commit.h\nindex e57ae4b583..b91df315c5 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -10,6 +10,9 @@\n #include \"pretty.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+#define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n+#define GENERATION_NUMBER_MAX 0x3FFFFFFF\n+#define GENERATION_NUMBER_ZERO 0\n \n struct commit_list {\n \tstruct commit *item;\n@@ -24,6 +27,7 @@ struct commit {\n \tstruct commit_list *parents;\n \tstruct tree *tree;\n \tuint32_t graph_pos;\n+\tuint32_t generation;\n };\n \n extern int save_commit_buffer;\n-- \n2.17.0\n\n"},{"id":"344206","messageId":"20180409164131.37312-5-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 04/10] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:01Z","receivedAt":"2018-04-09T16:42:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While preparing commits to be written into a commit-graph file, compute\nthe generation numbers using a depth-first strategy.\n\nThe only commits that are walked in this depth-first search are those\nwithout a precomputed generation number. Thus, computation time will be\nrelative to the number of new commits to the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 46 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d24b947525..5fd63acc31 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -419,6 +419,13 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\telse\n \t\t\tpackedDate[0] = 0;\n \n+\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY) {\n+\t\t\tif ((*list)->generation > GENERATION_NUMBER_MAX)\n+\t\t\t\tdie(\"generation number %u is too large to store in commit-graph\",\n+\t\t\t\t (*list)->generation);\n+\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n+\t\t}\n+\n \t\tpackedDate[1] = htonl((*list)->date);\n \t\thashwrite(f, packedDate, 8);\n \n@@ -551,6 +558,43 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n }\n \n+static void compute_generation_numbers(struct commit** commits,\n+\t\t\t\t int nr_commits)\n+{\n+\tint i;\n+\tstruct commit_list *list = NULL;\n+\n+\tfor (i = 0; i < nr_commits; i++) {\n+\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n+\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n+\t\t\tcontinue;\n+\n+\t\tcommit_list_insert(commits[i], &list);\n+\t\twhile (list) {\n+\t\t\tstruct commit *current = list->item;\n+\t\t\tstruct commit_list *parent;\n+\t\t\tint all_parents_computed = 1;\n+\t\t\tuint32_t max_generation = 0;\n+\n+\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n+\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n+\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n+\t\t\t\t\tall_parents_computed = 0;\n+\t\t\t\t\tcommit_list_insert(parent->item, &list);\n+\t\t\t\t\tbreak;\n+\t\t\t\t} else if (parent->item->generation > max_generation) {\n+\t\t\t\t\tmax_generation = parent->item->generation;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (all_parents_computed) {\n+\t\t\t\tcurrent->generation = max_generation + 1;\n+\t\t\t\tpop_commit(&list);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tconst char **pack_indexes,\n \t\t\tint nr_packs,\n@@ -674,6 +718,8 @@ void write_commit_graph(const char *obj_dir,\n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n \n+\tcompute_generation_numbers(commits.list, commits.nr);\n+\n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n \n-- \n2.17.0\n\n"},{"id":"344207","messageId":"20180409164131.37312-6-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 05/10] commit: use generations in paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:03Z","receivedAt":"2018-04-09T16:42:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Define compare_commits_by_gen_then_commit_date(), which uses generation\nnumbers as a primary comparison and commit date to break ties (or as a\ncomparison when both commits do not have computed generation numbers).\n\nSince the commit-graph file is closed under reachability, we know that\nall commits in the file have generation at most GENERATION_NUMBER_MAX\nwhich is less than GENERATION_NUMBER_INFINITY.\n\nThis change does not affect the number of commits that are walked during\nthe execution of paint_down_to_common(), only the order that those\ncommits are inspected. In the case that commit dates violate topological\norder (i.e. a parent is \"newer\" than a child), the previous code could\nwalk a commit twice: if a commit is reached with the PARENT1 bit, but\nlater is re-visited with the PARENT2 bit, then that PARENT2 bit must be\npropagated to its parents. Using generation numbers avoids this extra\neffort, even if it is somewhat rare.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 19 ++++++++++++++++++-\n commit.h | 1 +\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 3e39c86abf..95ae7e13a3 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -624,6 +624,23 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n \treturn 0;\n }\n \n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n+{\n+\tconst struct commit *a = a_, *b = b_;\n+\n+\tif (a->generation < b->generation)\n+\t\treturn 1;\n+\telse if (a->generation > b->generation)\n+\t\treturn -1;\n+\n+\t/* newer commits with larger date first */\n+\tif (a->date < b->date)\n+\t\treturn 1;\n+\telse if (a->date > b->date)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n {\n \tconst struct commit *a = a_, *b = b_;\n@@ -773,7 +790,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n /* all input commits in one and twos[] must have been parsed! */\n static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n {\n-\tstruct prio_queue queue = { compare_commits_by_commit_date };\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n \ndiff --git a/commit.h b/commit.h\nindex b91df315c5..c440f56bf9 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -332,6 +332,7 @@ extern int remove_signature(struct strbuf *buf);\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \n LAST_ARG_MUST_BE_NULL\n extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n-- \n2.17.0\n\n"},{"id":"344208","messageId":"20180409164131.37312-7-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 06/10] commit.c: use generation to halt paint walk","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:05Z","receivedAt":"2018-04-09T16:42:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In paint_down_to_common(), the walk is halted when the queue contains\nonly stale commits. The queue_has_nonstale() method iterates over the\nentire queue looking for a nonstale commit. In a wide commit graph where\nthe two sides share many commits in common, but have deep sets of\ndifferent commits, this method may inspect many elements before finding\na nonstale commit. In the worst case, this can give quadratic\nperformance in paint_down_to_common().\n\nConvert queue_has_nonstale() to use generation numbers for an O(1)\ntermination condition. To properly take advantage of this condition,\ntrack the minimum generation number of a commit that enters the queue\nwith nonstale status.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 95ae7e13a3..00bdc2ab21 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -776,14 +776,22 @@ void sort_in_topological_order(struct commit_list **list, enum rev_sort_order so\n \n static const unsigned all_flags = (PARENT1 | PARENT2 | STALE | RESULT);\n \n-static int queue_has_nonstale(struct prio_queue *queue)\n+static int queue_has_nonstale(struct prio_queue *queue, uint32_t min_gen)\n {\n-\tint i;\n-\tfor (i = 0; i < queue->nr; i++) {\n-\t\tstruct commit *commit = queue->array[i].data;\n-\t\tif (!(commit->object.flags & STALE))\n-\t\t\treturn 1;\n+\tif (min_gen != GENERATION_NUMBER_INFINITY) {\n+\t\tif (queue->nr > 0) {\n+\t\t\tstruct commit *commit = queue->array[0].data;\n+\t\t\treturn commit->generation >= min_gen;\n+\t\t}\n+\t} else {\n+\t\tint i;\n+\t\tfor (i = 0; i < queue->nr; i++) {\n+\t\t\tstruct commit *commit = queue->array[i].data;\n+\t\t\tif (!(commit->object.flags & STALE))\n+\t\t\t\treturn 1;\n+\t\t}\n \t}\n+\n \treturn 0;\n }\n \n@@ -793,6 +801,8 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n+\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n+\tuint32_t min_nonstale_gen = GENERATION_NUMBER_INFINITY;\n \n \tone->object.flags |= PARENT1;\n \tif (!n) {\n@@ -800,17 +810,26 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \t\treturn result;\n \t}\n \tprio_queue_put(&queue, one);\n+\tif (one->generation < min_nonstale_gen)\n+\t\tmin_nonstale_gen = one->generation;\n \n \tfor (i = 0; i < n; i++) {\n \t\ttwos[i]->object.flags |= PARENT2;\n \t\tprio_queue_put(&queue, twos[i]);\n+\t\tif (twos[i]->generation < min_nonstale_gen)\n+\t\t\tmin_nonstale_gen = twos[i]->generation;\n \t}\n \n-\twhile (queue_has_nonstale(&queue)) {\n+\twhile (queue_has_nonstale(&queue, min_nonstale_gen)) {\n \t\tstruct commit *commit = prio_queue_get(&queue);\n \t\tstruct commit_list *parents;\n \t\tint flags;\n \n+\t\tif (commit->generation > last_gen)\n+\t\t\tBUG(\"bad generation skip\");\n+\n+\t\tlast_gen = commit->generation;\n+\n \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n \t\tif (flags == (PARENT1 | PARENT2)) {\n \t\t\tif (!(commit->object.flags & RESULT)) {\n@@ -830,6 +849,10 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \t\t\t\treturn NULL;\n \t\t\tp->object.flags |= flags;\n \t\t\tprio_queue_put(&queue, p);\n+\n+\t\t\tif (!(flags & STALE) &&\n+\t\t\t p->generation < min_nonstale_gen)\n+\t\t\t\tmin_nonstale_gen = p->generation;\n \t\t}\n \t}\n \n-- \n2.17.0\n\n"},{"id":"344209","messageId":"20180409164131.37312-8-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 07/10] commit-graph.txt: update future work","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:07Z","receivedAt":"2018-04-09T16:42:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We now calculate generation numbers in the commit-graph file and use\nthem in paint_down_to_common().\n\nExpand the section on generation numbers to discuss how the two\n\"special\" generation numbers GENERATION_NUMBER_INFINITY and *_ZERO\ninteract with other generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 50 +++++++++++++++++++++---\n 1 file changed, 44 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 0550c6d0dc..a8df0ae9db 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -77,6 +77,49 @@ in the commit graph. We can treat these commits as having \"infinite\"\n generation number and walk until reaching commits with known generation\n number.\n \n+We use the macro GENERATION_NUMBER_INFINITY = 0xFFFFFFFF to mark commits not\n+in the commit-graph file. If a commit-graph file was written by a version\n+of Git that did not compute generation numbers, then those commits will\n+have generation number represented by the macro GENERATION_NUMBER_ZERO = 0.\n+\n+Since the commit-graph file is closed under reachability, we can guarantee\n+the following weaker condition on all commits:\n+\n+ If A and B are commits with generation numbers N amd M, respectively,\n+ and N < M, then A cannot reach B.\n+\n+Note how the strict inequality differs from the inequality when we have\n+fully-computed generation numbers. Using strict inequality may result in\n+walking a few extra commits, but the simplicity in dealing with commits\n+with generation number *_INFINITY or *_ZERO is valuable.\n+\n+Here is a diagram to visualize the shape of the full commit graph, and\n+how different generation numbers relate:\n+\n+ +-----------------------------------------+\n+ | GENERATION_NUMBER_INFINITY = 0xFFFFFFFF |\n+ +-----------------------------------------+\n+\t | | ^\n+\t | | |\n+\t | +------+\n+\t | [gen(A) = gen(B)]\n+\t V\n+ +-------------------------------------+\n+ | 0 < commit->generation < 0x40000000 |\n+ +-------------------------------------+\n+\t | | ^\n+\t | | |\n+\t | +------+\n+\t | [gen(A) > gen(B)]\n+\t V\n+ +-------------------------------------+\n+ | GENERATION_NUMBER_ZERO = 0 |\n+ +-------------------------------------+\n+\t\t\t | ^\n+\t\t\t | |\n+\t\t\t +------+\n+\t\t [gen(A) = gen(B)]\n+\n Design Details\n --------------\n \n@@ -98,17 +141,12 @@ Future Work\n - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n necessary for integration with fsck.\n \n-- The file format includes room for precomputed generation numbers. These\n- are not currently computed, so all generation numbers will be marked as\n- 0 (or \"uncomputed\"). A later patch will include this calculation.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n priority queue with one ordered by generation number. The following\n- operations are important candidates:\n+ operation is an important candidate:\n \n- - paint_down_to_common()\n - 'log --topo-order'\n \n - Currently, parse_commit_gently() requires filling in the root tree\n-- \n2.17.0\n\n"},{"id":"344210","messageId":"20180409164131.37312-11-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 10/10] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:12Z","receivedAt":"2018-04-09T16:42:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git branch --contains', the in_merge_bases_many()\nmethod calls paint_down_to_common() to discover if a specific\ncommit is reachable from a set of branches. Commits with lower\ngeneration number are not needed to correctly answer the\ncontainment query of in_merge_bases_many().\n\nAdd a new parameter, min_generation, to paint_down_to_common() that\nprevents walking commits with generation number strictly less than\nmin_generation. If 0 is given, then there is no functional change.\n\nFor in_merge_bases_many(), we can pass commit->generation as the\ncutoff, and this saves time during 'git branch --contains' queries\nthat would otherwise walk \"around\" the commit we are inspecting.\n\nFor a copy of the Linux repository, where HEAD is checked out at\nv4.13~100, we get the following performance improvement for\n'git branch --contains' over the previous commit:\n\nBefore: 0.21s\nAfter: 0.13s\nRel %: -38%\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 13 +++++++++----\n 1 file changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 0b155dece8..7348075e38 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -796,7 +796,9 @@ static int queue_has_nonstale(struct prio_queue *queue, uint32_t min_gen)\n }\n \n /* all input commits in one and twos[] must have been parsed! */\n-static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n+static struct commit_list *paint_down_to_common(struct commit *one, int n,\n+\t\t\t\t\t\tstruct commit **twos,\n+\t\t\t\t\t\tint min_generation)\n {\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n@@ -830,6 +832,9 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \n \t\tlast_gen = commit->generation;\n \n+\t\tif (commit->generation < min_generation)\n+\t\t\tbreak;\n+\n \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n \t\tif (flags == (PARENT1 | PARENT2)) {\n \t\t\tif (!(commit->object.flags & RESULT)) {\n@@ -882,7 +887,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n \t\t\treturn NULL;\n \t}\n \n-\tlist = paint_down_to_common(one, n, twos);\n+\tlist = paint_down_to_common(one, n, twos, 0);\n \n \twhile (list) {\n \t\tstruct commit *commit = pop_commit(&list);\n@@ -949,7 +954,7 @@ static int remove_redundant(struct commit **array, int cnt)\n \t\t\tfilled_index[filled] = j;\n \t\t\twork[filled++] = array[j];\n \t\t}\n-\t\tcommon = paint_down_to_common(array[i], filled, work);\n+\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n \t\tif (array[i]->object.flags & PARENT2)\n \t\t\tredundant[i] = 1;\n \t\tfor (j = 0; j < filled; j++)\n@@ -1073,7 +1078,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n \tif (commit->generation > min_generation)\n \t\treturn 0;\n \n-\tbases = paint_down_to_common(commit, nr_reference, reference);\n+\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n \tif (commit->object.flags & PARENT2)\n \t\tret = 1;\n \tclear_commit_marks(commit, all_flags);\n-- \n2.17.0\n\n"},{"id":"344211","messageId":"20180409164131.37312-10-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 09/10] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:10Z","receivedAt":"2018-04-09T16:43:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The containment algorithm for 'git branch --contains' is different\nfrom that for 'git tag --contains' in that it uses is_descendant_of()\ninstead of contains_tag_algo(). The expensive portion of the branch\nalgorithm is computing merge bases.\n\nWhen a commit-graph file exists with generation numbers computed,\nwe can avoid this merge-base calculation when the target commit has\na larger generation number than the target commits.\n\nPerformance tests were run on a copy of the Linux repository where\nHEAD is contained in v4.13 but no earlier tag. Also, all tags were\ncopied to branches and 'git branch --contains' was tested:\n\nBefore: 60.0s\nAfter: 0.4s\nRel %: -99.3%\n\nReported-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 00bdc2ab21..0b155dece8 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1059,12 +1059,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n {\n \tstruct commit_list *bases;\n \tint ret = 0, i;\n+\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n \n \tif (parse_commit(commit))\n \t\treturn ret;\n-\tfor (i = 0; i < nr_reference; i++)\n+\tfor (i = 0; i < nr_reference; i++) {\n \t\tif (parse_commit(reference[i]))\n \t\t\treturn ret;\n+\t\tif (min_generation > reference[i]->generation)\n+\t\t\tmin_generation = reference[i]->generation;\n+\t}\n+\n+\tif (commit->generation > min_generation)\n+\t\treturn 0;\n \n \tbases = paint_down_to_common(commit, nr_reference, reference);\n \tif (commit->object.flags & PARENT2)\n-- \n2.17.0\n\n"},{"id":"344212","messageId":"20180409164131.37312-9-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v2 08/10] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-09T16:42:08Z","receivedAt":"2018-04-09T16:43:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"A commit A can reach a commit B only if the generation number of A\nis strictly larger than the generation number of B. This condition\nallows significantly short-circuiting commit-graph walks.\n\nUse generation number for '--contains' type queries.\n\nOn a copy of the Linux repository where HEAD is containd in v4.13\nbut no earlier tag, the command 'git tag --contains HEAD' had the\nfollowing peformance improvement:\n\nBefore: 0.81s\nAfter: 0.04s\nRel %: -95%\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n ref-filter.c | 24 +++++++++++++++++++-----\n 1 file changed, 19 insertions(+), 5 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 45fc56216a..2f5e79b5de 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1584,7 +1584,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n */\n static enum contains_result contains_test(struct commit *candidate,\n \t\t\t\t\t const struct commit_list *want,\n-\t\t\t\t\t struct contains_cache *cache)\n+\t\t\t\t\t struct contains_cache *cache,\n+\t\t\t\t\t uint32_t cutoff)\n {\n \tenum contains_result *cached = contains_cache_at(cache, candidate);\n \n@@ -1598,8 +1599,11 @@ static enum contains_result contains_test(struct commit *candidate,\n \t\treturn CONTAINS_YES;\n \t}\n \n-\t/* Otherwise, we don't know; prepare to recurse */\n \tparse_commit_or_die(candidate);\n+\n+\tif (candidate->generation < cutoff)\n+\t\treturn CONTAINS_NO;\n+\n \treturn CONTAINS_UNKNOWN;\n }\n \n@@ -1615,8 +1619,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t\t\t\t struct contains_cache *cache)\n {\n \tstruct contains_stack contains_stack = { 0, 0, NULL };\n-\tenum contains_result result = contains_test(candidate, want, cache);\n+\tenum contains_result result;\n+\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n+\tconst struct commit_list *p;\n+\n+\tfor (p = want; p; p = p->next) {\n+\t\tstruct commit *c = p->item;\n+\t\tparse_commit_or_die(c);\n+\t\tif (c->generation < cutoff)\n+\t\t\tcutoff = c->generation;\n+\t}\n \n+\tresult = contains_test(candidate, want, cache, cutoff);\n \tif (result != CONTAINS_UNKNOWN)\n \t\treturn result;\n \n@@ -1634,7 +1648,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t * If we just popped the stack, parents->item has been marked,\n \t\t * therefore contains_test will return a meaningful yes/no.\n \t\t */\n-\t\telse switch (contains_test(parents->item, want, cache)) {\n+\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n \t\tcase CONTAINS_YES:\n \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n \t\t\tcontains_stack.nr--;\n@@ -1648,7 +1662,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t}\n \t}\n \tfree(contains_stack.contains_stack);\n-\treturn contains_test(candidate, want, cache);\n+\treturn contains_test(candidate, want, cache, cutoff);\n }\n \n static int commit_contains(struct ref_filter *filter, struct commit *commit,\n-- \n2.17.0\n\n"},{"id":"344220","messageId":"CAGZ79kZ74g7Q9ovX==8WTn-B6gZJyHvRnNVgTGJd3qQD2fdvRA@mail.gmail.com","threadId":"48212","inReplyTo":"20180409164131.37312-4-dstolee@microsoft.com","subject":"Re: [PATCH v2 03/10] commit: add generation number to struct commmit","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-09T17:59:52Z","receivedAt":"2018-04-09T17:59:58Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Apr 9, 2018 at 9:42 AM, Derrick Stolee wrote:\n> The generation number of a commit is defined recursively as follows:\n>\n> * If a commit A has no parents, then the generation number of A is one.\n> * If a commit A has parents, then the generation number of A is one\n> more than the maximum generation number among the parents of A.\n>\n> Add a uint32_t generation field to struct commit so we can pass this\n> information to revision walks. We use two special values to signal\n> the generation number is invalid:\n>\n> GENERATION_NUMBER_ININITY 0xFFFFFFFF\n\nGENERATION_NUMBER_INFINITY\n\nOn disk we currently only store up to 2^30-1,\n(2 bits fewer than MAX_UINT_32), but here we just take the maximum\nvalue of what a uint32_t can store. That miss match should not be a\nproblem albeit aesthetically.\n\nOnce we run into scaling problems, we can just up to uint64_t in the code,\nand defer the solution on disk to a new file format.\n\nWith both ZERO and _INFINITY we are at the border of uint\nwrap-around, so we have to be very careful to not add/subtract\none and then compare. Just to watch out for when reviewing.\n"},{"id":"344432","messageId":"xmqqo9iqze1g.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"20180409164131.37312-3-dstolee@microsoft.com","subject":"Re: [PATCH v2 02/10] merge: check config before loading commits","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-11T02:12:59Z","receivedAt":"2018-04-11T02:13:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> diff --git a/builtin/merge.c b/builtin/merge.c\n> index ee050a47f3..20897f8223 100644\n> --- a/builtin/merge.c\n> +++ b/builtin/merge.c\n> @@ -1183,13 +1183,14 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n> \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n> \tif (branch)\n> \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n> +\tinit_diff_ui_defaults();\n> +\tgit_config(git_merge_config, NULL);\n> +\n> \tif (!branch || is_null_oid(&head_oid))\n> \t\thead_commit = NULL;\n> \telse\n> \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n> \n> -\tinit_diff_ui_defaults();\n> -\tgit_config(git_merge_config, NULL);\n\nWow, that's tricky. git_merge_config() wants to know which \"branch\"\nwe are on, and this place is as early as we can move the call to\nwithout breaking things. Is this to allow parse_object() called\nin lookup_commit_reference_gently() to know if we can rely on the\ndata cached in the commit-graph data?\n\n> Move the config load to be between the initialization of 'branch'\n> and the commit lookup. Also add a test to t5318-commit-graph.sh\n> that exercises this code path to prevent a regression.\n\nIt is not clear to me how a successful merge of commits/8\ndemonstrates that reading the config earlier than before is\nregression free.\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index a380419b65..77d85aefe7 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n> graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n> graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n> \n> +test_expect_success 'perform fast-forward merge in full repo' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit checkout -b merge-5-to-8 commits/5 &&\n> +\tgit merge commits/8 &&\n> +\tgit show-ref -s merge-5-to-8 >output &&\n> +\tgit show-ref -s commits/8 >expect &&\n> +\ttest_cmp expect output\n> +'\n> +\n> test_done\n"},{"id":"344433","messageId":"xmqqefjmzd67.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"20180409164131.37312-4-dstolee@microsoft.com","subject":"Re: [PATCH v2 03/10] commit: add generation number to struct commmit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-11T02:31:44Z","receivedAt":"2018-04-11T02:31:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> The generation number of a commit is defined recursively as follows:\n>\n> * If a commit A has no parents, then the generation number of A is one.\n> * If a commit A has parents, then the generation number of A is one\n> more than the maximum generation number among the parents of A.\n>\n> Add a uint32_t generation field to struct commit so we can pass this\n> information to revision walks. We use two special values to signal\n> the generation number is invalid:\n>\n> GENERATION_NUMBER_ININITY 0xFFFFFFFF\n> GENERATION_NUMBER_ZERO 0\n>\n> The first (_INFINITY) means the generation number has not been loaded or\n> computed. The second (_ZERO) means the generation number was loaded\n> from a commit graph file that was stored before generation numbers\n> were computed.\n\nShould it also be possible for a caller to tell if a given commit\nhas too deep a history, i.e. we do not know its generation number\nexactly, but we know it is larger than 1<<30?\n\nIt seems that we only have a 30-bit field in the file, so wouldn't\nwe need a special value defined in (e.g. \"0\") so that we can tell\nthat the commit has such a large generation number? E.g.\n\n> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n\n\tif (!item->generation)\n\t\titem->generation = GENERATION_NUMBER_OVERFLOW;\n\nwhen we read it from the file?\n\nWe obviously need to do something similar when assigning a\ngeneration number to a child commit, perhaps like\n\n\t#define GENERATION_NUMBER_OVERFLOW (GENERATION_NUMBER_MAX + 1)\n\n\tcommit->generation = 1; /* assume no parent */\n\tfor (p = commit->parents; p; p++) {\n\t\tuint32_t gen = p->item->generation + 1;\n\n\t\tif (gen >= GENERATION_NUMBER_OVERFLOW) {\n\t\t\tcommit->generation = GENERATION_NUMBER_OVERFLOW;\n\t\t\tbreak;\n\t\t} else if (commit->generation < gen)\n\t\t\tcommit->generation = gen;\n\t}\n \nor something? And then on the writing side you'd encode too large a\ngeneration as '0'.\n"},{"id":"344434","messageId":"xmqqa7uazc9e.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"20180409164131.37312-5-dstolee@microsoft.com","subject":"Re: [PATCH v2 04/10] commit-graph: compute generation numbers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-11T02:51:25Z","receivedAt":"2018-04-11T02:51:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY) {\n> +\t\t\tif ((*list)->generation > GENERATION_NUMBER_MAX)\n> +\t\t\t\tdie(\"generation number %u is too large to store in commit-graph\",\n> +\t\t\t\t (*list)->generation);\n> +\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n> +\t\t}\n\n\nHow serious do we want this feature to be? On one extreme, we could\nbe irresponsible and say it will be a problem for our descendants in\nthe future if their repositories have more than billion pearls on a\nsingle strand, and the above certainly is a reasonable way to punt.\nThose who actually encounter the problem will notice by Git dying\nsomewhere rather deep in the callchain.\n\nOr we could say Git actually does support a history that is\narbitrarily long, even though such a deep portion of history will\nnot benefit from having generation numbers in commit-graph.\n\nI've been assuming that our stance is the latter and that is why I\nmade noises about overflowing 30-bit generation field in my review\nof the previous step.\n\nIn case we want to do the \"we know this is very large, but we do not\nknow the exact value\", we may actually want a mode where we can\npretend that GENERATION_NUMBER_MAX is set to quite low (say 256) and\nmake sure that the code to handle overflow behaves sensibly.\n\n> +\tfor (i = 0; i < nr_commits; i++) {\n> +\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n> +\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n> +\t\t\tcontinue;\n> +\n> +\t\tcommit_list_insert(commits[i], &list);\n> +\t\twhile (list) {\n> +...\n> +\t\t}\n> +\t}\n\nSo we go over the list of commits just _once_ and make sure each of\nthem gets the generation assigned correctly by (conceptually\nrecursively but iteratively in implementation by using a commit\nlist) making sure that all its parents have generation assigned and\ncompute the generation for the commit, before moving to the next\none. Which sounds correct.\n\n\n"},{"id":"344435","messageId":"xmqq604yzbqe.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"20180409164131.37312-7-dstolee@microsoft.com","subject":"Re: [PATCH v2 06/10] commit.c: use generation to halt paint walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-11T03:02:49Z","receivedAt":"2018-04-11T03:02:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> @@ -800,17 +810,26 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n> \t\treturn result;\n> \t}\n> \tprio_queue_put(&queue, one);\n> +\tif (one->generation < min_nonstale_gen)\n> +\t\tmin_nonstale_gen = one->generation;\n> \n> \tfor (i = 0; i < n; i++) {\n> \t\ttwos[i]->object.flags |= PARENT2;\n> \t\tprio_queue_put(&queue, twos[i]);\n> +\t\tif (twos[i]->generation < min_nonstale_gen)\n> +\t\t\tmin_nonstale_gen = twos[i]->generation;\n> \t}\n> \n> -\twhile (queue_has_nonstale(&queue)) {\n> +\twhile (queue_has_nonstale(&queue, min_nonstale_gen)) {\n> \t\tstruct commit *commit = prio_queue_get(&queue);\n> \t\tstruct commit_list *parents;\n> \t\tint flags;\n> \n> +\t\tif (commit->generation > last_gen)\n> +\t\t\tBUG(\"bad generation skip\");\n> +\n> +\t\tlast_gen = commit->generation;\n> +\n> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n> \t\tif (flags == (PARENT1 | PARENT2)) {\n> \t\t\tif (!(commit->object.flags & RESULT)) {\n> @@ -830,6 +849,10 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n> \t\t\t\treturn NULL;\n> \t\t\tp->object.flags |= flags;\n\nHmph. Can a commit that used to be not stale (and contributed to\nthe current value of min_nonstale_gen) become stale here by getting\nvisited twice, invalidating the value in min_nonstale_gen?\n\n> \t\t\tprio_queue_put(&queue, p);\n> +\n> +\t\t\tif (!(flags & STALE) &&\n> +\t\t\t p->generation < min_nonstale_gen)\n> +\t\t\t\tmin_nonstale_gen = p->generation;\n> \t\t}\n> \t}\n"},{"id":"344455","messageId":"197f4c77-ab2a-cf5b-463f-471bd5f8966f@gmail.com","threadId":"48212","inReplyTo":"xmqqo9iqze1g.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 02/10] merge: check config before loading commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-11T12:49:44Z","receivedAt":"2018-04-11T12:49:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/10/2018 10:12 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> diff --git a/builtin/merge.c b/builtin/merge.c\n>> index ee050a47f3..20897f8223 100644\n>> --- a/builtin/merge.c\n>> +++ b/builtin/merge.c\n>> @@ -1183,13 +1183,14 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n>> \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n>> \tif (branch)\n>> \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n>> +\tinit_diff_ui_defaults();\n>> +\tgit_config(git_merge_config, NULL);\n>> +\n>> \tif (!branch || is_null_oid(&head_oid))\n>> \t\thead_commit = NULL;\n>> \telse\n>> \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n>> \n>> -\tinit_diff_ui_defaults();\n>> -\tgit_config(git_merge_config, NULL);\n> Wow, that's tricky. git_merge_config() wants to know which \"branch\"\n> we are on, and this place is as early as we can move the call to\n> without breaking things. Is this to allow parse_object() called\n> in lookup_commit_reference_gently() to know if we can rely on the\n> data cached in the commit-graph data?\n\nWhen I saw the bug on my machine, I tracked the issue down to a call to \nparse_commit_in_graph() that skipped the graph check since \ncore_commit_graph was not set. The call stack from this call is as follows:\n\n* lookup_commit_or_die()\n* lookup_commit_reference()\n* lookup_commit_reference_gently()\n* parse_object()\n* parse_object_buffer()\n* parse_commit_in_graph() [as introduced in PATCH 01/10]\n\n>\n>> Move the config load to be between the initialization of 'branch'\n>> and the commit lookup. Also add a test to t5318-commit-graph.sh\n>> that exercises this code path to prevent a regression.\n> It is not clear to me how a successful merge of commits/8\n> demonstrates that reading the config earlier than before is\n> regression free.\n\nI didn't want to introduce commits in an order that led to a commit \nfailing tests, but if you drop the change to builtin/merge.c from this \nseries, the tip commit will fail this test with \"BUG: bad generation skip\".\n\nThe reason for this failure is that commits/5 is loaded from HEAD from \nthe object database, so its generation is marked as \nGENERATION_NUMBER_INFINITY, and the commit is marked as parsed. Later, \nthe commit at merges/3 is loaded from the graph with generation 4. This \ntriggers the BUG statement in paint_down_to_common(). That is why it is \nimportant to check a fast-forward merge.\n\nIn the 'graph_git_behavior' steps of t5318-commit-graph.sh, we were \nalready testing 'git merge-base' to check the commit walk logic.\n\nThanks,\n-Stolee\n"},{"id":"344456","messageId":"01c8d95b-e444-4b69-f083-9832023978a0@gmail.com","threadId":"48212","inReplyTo":"xmqqefjmzd67.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 03/10] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-11T12:57:05Z","receivedAt":"2018-04-11T12:57:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/10/2018 10:31 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> The generation number of a commit is defined recursively as follows:\n>>\n>> * If a commit A has no parents, then the generation number of A is one.\n>> * If a commit A has parents, then the generation number of A is one\n>> more than the maximum generation number among the parents of A.\n>>\n>> Add a uint32_t generation field to struct commit so we can pass this\n>> information to revision walks. We use two special values to signal\n>> the generation number is invalid:\n>>\n>> GENERATION_NUMBER_ININITY 0xFFFFFFFF\n>> GENERATION_NUMBER_ZERO 0\n>>\n>> The first (_INFINITY) means the generation number has not been loaded or\n>> computed. The second (_ZERO) means the generation number was loaded\n>> from a commit graph file that was stored before generation numbers\n>> were computed.\n> Should it also be possible for a caller to tell if a given commit\n> has too deep a history, i.e. we do not know its generation number\n> exactly, but we know it is larger than 1<<30?\n>\n> It seems that we only have a 30-bit field in the file, so wouldn't\n> we need a special value defined in (e.g. \"0\") so that we can tell\n> that the commit has such a large generation number? E.g.\n>\n>> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n> \tif (!item->generation)\n> \t\titem->generation = GENERATION_NUMBER_OVERFLOW;\n>\n> when we read it from the file?\n>\n> We obviously need to do something similar when assigning a\n> generation number to a child commit, perhaps like\n>\n> \t#define GENERATION_NUMBER_OVERFLOW (GENERATION_NUMBER_MAX + 1)\n>\n> \tcommit->generation = 1; /* assume no parent */\n> \tfor (p = commit->parents; p; p++) {\n> \t\tuint32_t gen = p->item->generation + 1;\n>\n> \t\tif (gen >= GENERATION_NUMBER_OVERFLOW) {\n> \t\t\tcommit->generation = GENERATION_NUMBER_OVERFLOW;\n> \t\t\tbreak;\n> \t\t} else if (commit->generation < gen)\n> \t\t\tcommit->generation = gen;\n> \t}\n> \n> or something? And then on the writing side you'd encode too large a\n> generation as '0'.\n\nYou raise a very good point. How about we do a slightly different \narrangement for these overflow commits?\n\nInstead of storing the commits in the commit-graph file as \"0\" (which \ncurrently means \"written by a version of git that did not compute \ngeneration numbers\") we could let GENERATION_NUMBER_MAX be the maximum \ngeneration of a commit in the commit-graph, and if a commit would have \nlarger generation, we collapse it down to that value.\n\nIt slightly complicates the diagram I made in \nDocumentation/technical/commit-graph.txt, but it was already a bit of a \nsimplification. Here is an updated diagram, but likely we will want to \nlimit discussion of the special-case GENERATION_NUMBER_MAX to the prose, \nsince it is not a practical situation at the moment.\n\n     +-----------------------------------------+\n     | GENERATION_NUMBER_INFINITY = 0xFFFFFFFF |\n     +-----------------------------------------+\n       |    |            |      ^\n       |    |            |      |\n       |    |            +------+\n       |    |         [gen(A) = gen(B)]\n       |    V\n       |  +------------------------------------+\n       |  | GENERATION_NUMBER_MAX = 0x3FFFFFFF |\n       |  +------------------------------------+\n       |    |            |      ^\n       |    |            |      |\n       |    |            +------+\n       |    |         [gen(A) = gen(B)]\n       V    V\n     +-------------------------------------+\n     | 0 < commit->generation < 0x3FFFFFFF |\n     +-------------------------------------+\n         |            |      ^\n         |            |      |\n         |            +------+\n         |        [gen(A) > gen(B)]\n         V\n     +-------------------------------------+\n     | GENERATION_NUMBER_ZERO = 0          |\n     +-------------------------------------+\n              |      ^\n              |      |\n              +------+\n              [gen(A) = gen(B)]\n\nThanks,\n-Stolee\n"},{"id":"344457","messageId":"9ba7d27f-ce65-7f2c-601a-1bea3274104e@gmail.com","threadId":"48212","inReplyTo":"xmqqa7uazc9e.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 04/10] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-11T13:02:09Z","receivedAt":"2018-04-11T13:02:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/10/2018 10:51 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> +\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY) {\n>> +\t\t\tif ((*list)->generation > GENERATION_NUMBER_MAX)\n>> +\t\t\t\tdie(\"generation number %u is too large to store in commit-graph\",\n>> +\t\t\t\t (*list)->generation);\n>> +\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n>> +\t\t}\n>\n> How serious do we want this feature to be? On one extreme, we could\n> be irresponsible and say it will be a problem for our descendants in\n> the future if their repositories have more than billion pearls on a\n> single strand, and the above certainly is a reasonable way to punt.\n> Those who actually encounter the problem will notice by Git dying\n> somewhere rather deep in the callchain.\n>\n> Or we could say Git actually does support a history that is\n> arbitrarily long, even though such a deep portion of history will\n> not benefit from having generation numbers in commit-graph.\n>\n> I've been assuming that our stance is the latter and that is why I\n> made noises about overflowing 30-bit generation field in my review\n> of the previous step.\n>\n> In case we want to do the \"we know this is very large, but we do not\n> know the exact value\", we may actually want a mode where we can\n> pretend that GENERATION_NUMBER_MAX is set to quite low (say 256) and\n> make sure that the code to handle overflow behaves sensibly.\n\nI agree. I wonder how we can effectively expose this value into a test. \nIt's probably not sufficient to manually test using compiler flags (\"-D \nGENERATION_NUMBER_MAX=8\").\n\n>\n>> +\tfor (i = 0; i < nr_commits; i++) {\n>> +\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n>> +\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tcommit_list_insert(commits[i], &list);\n>> +\t\twhile (list) {\n>> +...\n>> +\t\t}\n>> +\t}\n> So we go over the list of commits just _once_ and make sure each of\n> them gets the generation assigned correctly by (conceptually\n> recursively but iteratively in implementation by using a commit\n> list) making sure that all its parents have generation assigned and\n> compute the generation for the commit, before moving to the next\n> one. Which sounds correct.\n\nYes, we compute the generation number of a commit exactly once. We use \nthe list as a stack so we do not have recursion limits during our \ndepth-first search (DFS). We rely on the object cache to ensure we store \nthe computed generation numbers, and computed generation numbers provide \ntermination conditions to the DFS.\n\nThanks,\n-Stolee\n"},{"id":"344459","messageId":"cc58d8c1-f8de-4841-934f-56e4ad7729aa@gmail.com","threadId":"48212","inReplyTo":"xmqq604yzbqe.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 06/10] commit.c: use generation to halt paint walk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-11T13:24:01Z","receivedAt":"2018-04-11T13:24:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/10/2018 11:02 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> @@ -800,17 +810,26 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n>> \t\treturn result;\n>> \t}\n>> \tprio_queue_put(&queue, one);\n>> +\tif (one->generation < min_nonstale_gen)\n>> +\t\tmin_nonstale_gen = one->generation;\n>> \n>> \tfor (i = 0; i < n; i++) {\n>> \t\ttwos[i]->object.flags |= PARENT2;\n>> \t\tprio_queue_put(&queue, twos[i]);\n>> +\t\tif (twos[i]->generation < min_nonstale_gen)\n>> +\t\t\tmin_nonstale_gen = twos[i]->generation;\n>> \t}\n>> \n>> -\twhile (queue_has_nonstale(&queue)) {\n>> +\twhile (queue_has_nonstale(&queue, min_nonstale_gen)) {\n>> \t\tstruct commit *commit = prio_queue_get(&queue);\n>> \t\tstruct commit_list *parents;\n>> \t\tint flags;\n>> \n>> +\t\tif (commit->generation > last_gen)\n>> +\t\t\tBUG(\"bad generation skip\");\n>> +\n>> +\t\tlast_gen = commit->generation;\n>> +\n>> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n>> \t\tif (flags == (PARENT1 | PARENT2)) {\n>> \t\t\tif (!(commit->object.flags & RESULT)) {\n>> @@ -830,6 +849,10 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n>> \t\t\t\treturn NULL;\n>> \t\t\tp->object.flags |= flags;\n> Hmph. Can a commit that used to be not stale (and contributed to\n> the current value of min_nonstale_gen) become stale here by getting\n> visited twice, invalidating the value in min_nonstale_gen?\n\nmin_nonstale_gen can be \"wrong\" in the way you say, but fits the \ndefinition from the commit message:\n\n\"To properly take advantage of this condition, track the minimum \ngeneration number of a commit that **enters the queue** with nonstale \nstatus.\" (Emphasis added)\n\nYou make an excellent point about how this can be problematic. I was \nconfused by the lack of clear performance benefits here, but I think \nthat whatever benefits making queue_has_nonstale() be O(1) were removed \nby walking more commits than necessary.\n\nConsider the following commit graph, where M is a parent of both A and \nB, S is a parent of M and B, and there is a large set of commits \nreachable from M with generation number larger than gen(S).\n\nA    B\n| __/|\n|/   |\nM    |\n|\\   |\n. |  |\n. |  |\n. |_/\n|/\nS\n\nBetween A and B, the true merge base is M. Anything reachable from M is \nmarked as stale. When S is added to the queue, it is only reachable from \nB, so it is non-stale. However, it is marked stale after M is walked. \nThe old code would detect this as a termination condition, but the new \ncode would not.\n\nI think this data shape is actually common (not exactly, as it may be \nthat some ancestor of M provides a second path to S) especially in the \nworld of pull requests and users merging master into their topic branches.\n\nI'll remove this commit in the next version, but use the new prototype \nfor queue_has_nonstale() in \"commit: add short-circuit to \npaint_down_to_common()\" using the given 'min_generation' instead of \n'min_nonstale_gen'.\n\nThanks,\n-Stolee\n"},{"id":"344473","messageId":"CAPig+cRHV0+iWS_+QdbJyuWpoXxknVbwngOS-BV4XFcyGRWGaw@mail.gmail.com","threadId":"48212","inReplyTo":"9ba7d27f-ce65-7f2c-601a-1bea3274104e@gmail.com","subject":"Re: [PATCH v2 04/10] commit-graph: compute generation numbers","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-04-11T19:26:07Z","receivedAt":"2018-04-11T19:26:14Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Apr 11, 2018 at 9:02 AM, Derrick Stolee wrote:\n> On 4/10/2018 10:51 PM, Junio C Hamano wrote:\n>> In case we want to do the \"we know this is very large, but we do not\n>> know the exact value\", we may actually want a mode where we can\n>> pretend that GENERATION_NUMBER_MAX is set to quite low (say 256) and\n>> make sure that the code to handle overflow behaves sensibly.\n>\n> I agree. I wonder how we can effectively expose this value into a test. It's\n> probably not sufficient to manually test using compiler flags (\"-D\n> GENERATION_NUMBER_MAX=8\").\n\nA few similar cases of tests needing to tweak some behavior do so by\nenvironment variable. See, for instance, GIT_GETTEXT_POISON and\nGIT_FSMONITOR_TEST.\n"},{"id":"344475","messageId":"868t9t5yjz.fsf@gmail.com","threadId":"48212","inReplyTo":"07abd2ce-e042-fe3c-b2db-3d2f2aa44de8@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-11T19:32:32Z","receivedAt":"2018-04-11T19:32:48Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 4/7/2018 12:55 PM, Jakub Narebski wrote:\n>> Currently I am at the stage of reproducing results in FELINE paper:\n>> \"Reachability Queries in Very Large Graphs: A Fast Refined Online Search\n>> Approach\" by Renê R. Veloso, Loïc Cerf, Wagner Meira Jr and Mohammed\n>> J. Zaki (2014). This paper is available in the PDF form at\n>> https://openproceedings.org/EDBT/2014/paper_166.pdf\n>>\n>> The Jupyter Notebook (which runs on Google cloud, but can be also run\n>> locally) uses Python kernel, NetworkX librabry for graph manipulation,\n>> and matplotlib (via NetworkX) for display.\n>>\n>> Available at:\n>> https://colab.research.google.com/drive/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg\n>> https://drive.google.com/file/d/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg/view?usp=sharing\n>>\n>> I hope that could be of help, or at least interesting\n>\n> Let me know when you can give numbers (either raw performance or # of\n> commits walked) for real-world Git commit graphs. The Linux repo is a\n> good example to use for benchmarking, but I also use the Kotlin repo\n> sometimes as it has over a million objects and over 250K commits.\n\nAs I am curently converting git repository into commit graph, number of\nobjects doesn't matter.\n\nThough Kotlin is nicely in largish size set, not as large as Linux\nkernel which has 750K commits, but mich larger than git.git with 65K\ncommits.\n\n> Of course, the only important statistic at the end of the day is the\n> end-to-end time of a 'git ...' command. Your investigations should\n> inform whether it is worth prototyping the feature in the git\n> codebase.\n\nWhat would you suggest as a good test that could imply performance? The\nGoogle Colab notebook linked to above includes a function to count\nnumber of commits (nodes / vertices in the commit graph) walked,\ncurrently in the worst case scenario.\n\n\nI have tried finding number of false positives for level (generation\nnumber) filter and for FELINE index, and number of false negatives for\nmin-post intervals in the spanning tree (for DFS tree) for 10000\nrandomly selected pairs of commits... but I don't think this is a good\nbenchmark.\n\nI Linux kernel sources (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git)\nthat has 750832 nodes and 811733 edges, and 563747941392 possible\ndirected pairs, we have for 10000 randomly selected pairs of commits:\n\n level-filter has 91 = 0.91% [all] false positives\n FELINE index has 78 = 0.78% [all] false positives\n FELINE index has 1.16667 less false positives than level filter\n\n min-post spanning-tree intervals has 3641 = 36.41% [all] false\n negatives\n\nFor git.git repository (https://github.com/git/git.git) that has 52950\nnodes and 65887 edges the numbers are slighly more in FELINE index\nfavor (also out of 10000 random pairs):\n\n level-filter has 504 = 9.11% false positives\n FELINE index has 125 = 2.26% false positives\n FELINE index has 4.032 less false positives than level filter\n\nThis is for FELINE which does not use level / generatio-numbers filter.\n\nRegards,\n--\nJakub Narębski\n"},{"id":"344477","messageId":"45320265-9fec-cee1-e82c-3ff719bb0435@gmail.com","threadId":"48212","inReplyTo":"868t9t5yjz.fsf@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-11T19:58:27Z","receivedAt":"2018-04-11T19:58:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/11/2018 3:32 PM, Jakub Narebski wrote:\n> What would you suggest as a good test that could imply performance? The\n> Google Colab notebook linked to above includes a function to count\n> number of commits (nodes / vertices in the commit graph) walked,\n> currently in the worst case scenario.\n\nThe two main questions to consider are:\n\n1. Can X reach Y?\n2. What is the set of merge-bases between X and Y?\n\nAnd the thing to measure is a commit count. If possible, it would be \ngood to count commits walked (commits whose parent list is enumerated) \nand commits inspected (commits that were listed as a parent of some \nwalked commit). Walked commits require a commit parse -- albeit from the \ncommit-graph instead of the ODB now -- while inspected commits only \ncheck the in-memory cache.\n\nFor git.git and Linux, I like to use the release tags as tests. They \nprovide a realistic view of the linear history, and maintenance releases \nhave their own history from the major releases.\n\n> I have tried finding number of false positives for level (generation\n> number) filter and for FELINE index, and number of false negatives for\n> min-post intervals in the spanning tree (for DFS tree) for 10000\n> randomly selected pairs of commits... but I don't think this is a good\n> benchmark.\n\nWhat is a false-positive? A case where gen(X) < gen(Y) but Y cannot \nreach X? I do not think that is a great benchmark, but I guess it is \nsomething to measure.\n\n> I Linux kernel sources (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git)\n> that has 750832 nodes and 811733 edges, and 563747941392 possible\n> directed pairs, we have for 10000 randomly selected pairs of commits:\n>\n> level-filter has 91 = 0.91% [all] false positives\n> FELINE index has 78 = 0.78% [all] false positives\n> FELINE index has 1.16667 less false positives than level filter\n>\n> min-post spanning-tree intervals has 3641 = 36.41% [all] false\n> negatives\n\nPerhaps something you can do instead of sampling from N^2 commits in \ntotal is to select a pair of generations (say, G = 20000, G' = 20100) or \nregions of generations ( 20000 <= G <= 20050, 20100 <= G' <= 20150) and \nsee how many false positives you see by testing all pairs (one from each \nlevel). The delta between the generations may need to be smaller to \nactually have a large proportion of unreachable pairs. Try different \nlevels, since major version releases tend to \"pinch\" the commit graph to \na common history.\n\n> For git.git repository (https://github.com/git/git.git) that has 52950\n> nodes and 65887 edges the numbers are slighly more in FELINE index\n> favor (also out of 10000 random pairs):\n>\n> level-filter has 504 = 9.11% false positives\n> FELINE index has 125 = 2.26% false positives\n> FELINE index has 4.032 less false positives than level filter\n>\n> This is for FELINE which does not use level / generatio-numbers filter.\n\nThanks,\n-Stolee\n\n"},{"id":"344484","messageId":"CAGZ79kaqyEvfS2aND-pBkgjN0=GCmU8MiXvoeNJ8RAjpbqyn_w@mail.gmail.com","threadId":"48212","inReplyTo":"9ba7d27f-ce65-7f2c-601a-1bea3274104e@gmail.com","subject":"Re: [PATCH v2 04/10] commit-graph: compute generation numbers","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-11T18:49:31Z","receivedAt":"2018-04-11T20:32:08Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Apr 11, 2018 at 6:02 AM, Derrick Stolee wrote:\n> On 4/10/2018 10:51 PM, Junio C Hamano wrote:\n>>\n>> Derrick Stolee writes:\n>>\n>>> + if ((*list)->generation != GENERATION_NUMBER_INFINITY) {\n>>> + if ((*list)->generation > GENERATION_NUMBER_MAX)\n>>> + die(\"generation number %u is too large to\n>>> store in commit-graph\",\n>>> + (*list)->generation);\n>>> + packedDate[0] |= htonl((*list)->generation << 2);\n>>> + }\n>>\n>>\n>> How serious do we want this feature to be? On one extreme, we could\n>> be irresponsible and say it will be a problem for our descendants in\n>> the future if their repositories have more than billion pearls on a\n>> single strand, and the above certainly is a reasonable way to punt.\n>> Those who actually encounter the problem will notice by Git dying\n>> somewhere rather deep in the callchain.\n>>\n>> Or we could say Git actually does support a history that is\n>> arbitrarily long, even though such a deep portion of history will\n>> not benefit from having generation numbers in commit-graph.\n>>\n>> I've been assuming that our stance is the latter and that is why I\n>> made noises about overflowing 30-bit generation field in my review\n>> of the previous step.\n>>\n>> In case we want to do the \"we know this is very large, but we do not\n>> know the exact value\", we may actually want a mode where we can\n>> pretend that GENERATION_NUMBER_MAX is set to quite low (say 256) and\n>> make sure that the code to handle overflow behaves sensibly.\n>\n>\n> I agree. I wonder how we can effectively expose this value into a test. It's\n> probably not sufficient to manually test using compiler flags (\"-D\n> GENERATION_NUMBER_MAX=8\").\n\nWould using an environment variable for this testing purpose be a good idea?\n\nIf we allow a user to pass in an arbitrary maximum, then we'd have to care about\ngeneration numbers that are stored in the commit graph file larger than that\nuser specific maximum, though.\n\nLooking through the output of \"git grep getenv\" we only have two instances\nwith _DEBUG, both in transport.\n"},{"id":"344507","messageId":"xmqqy3ht2uh6.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"01c8d95b-e444-4b69-f083-9832023978a0@gmail.com","subject":"Re: [PATCH v2 03/10] commit: add generation number to struct commmit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-11T23:28:53Z","receivedAt":"2018-04-11T23:29:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> How about we do a slightly different\n> arrangement for these overflow commits?\n>\n> Instead of storing the commits in the commit-graph file as \"0\" (which\n> currently means \"written by a version of git that did not compute\n> generation numbers\") we could let GENERATION_NUMBER_MAX be the maximum\n> generation of a commit in the commit-graph, and if a commit would have\n> larger generation, we collapse it down to that value.\n\nSure. Any value we can tell that it is special is fine. Thanks.\n"},{"id":"344535","messageId":"xmqqfu403i0p.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"20180409164131.37312-8-dstolee@microsoft.com","subject":"Re: [PATCH v2 07/10] commit-graph.txt: update future work","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-12T09:12:38Z","receivedAt":"2018-04-12T09:12:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +Here is a diagram to visualize the shape of the full commit graph, and\n> +how different generation numbers relate:\n> +\n> + +-----------------------------------------+\n> + | GENERATION_NUMBER_INFINITY = 0xFFFFFFFF |\n> + +-----------------------------------------+\n> +\t | | ^\n> +\t | | |\n> +\t | +------+\n> +\t | [gen(A) = gen(B)]\n> +\t V\n> + +-------------------------------------+\n> + | 0 < commit->generation < 0x40000000 |\n> + +-------------------------------------+\n> +\t | | ^\n> +\t | | |\n> +\t | +------+\n> +\t | [gen(A) > gen(B)]\n> +\t V\n> + +-------------------------------------+\n> + | GENERATION_NUMBER_ZERO = 0 |\n> + +-------------------------------------+\n> +\t\t\t | ^\n> +\t\t\t | |\n> +\t\t\t +------+\n> +\t\t [gen(A) = gen(B)]\n\nIt may be just me but all I can read out of the above is that\ncommit->generation may store 0xFFFFFFFF, a value between 0 and\n0x40000000, or 0. I cannot quite tell what the notation [gen(A)\n gen(B)] is trying to say. I am guessing \"Two generation\nnumbers within the 'valid' range can be compared\" is what the second\none is trying to say, but it is much less interesting to know that\ntwo infinities compare equal than how generation numbers from\ndifferent classes compare, which cannot be depicted in the above\nnotation, I am afraid. For example, don't we want to say that a\ncommit with INF can never be reached by a commit with a valid\ngeneration number, or something like that?\n\n> Design Details\n> --------------\n> \n> @@ -98,17 +141,12 @@ Future Work\n> - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n> necessary for integration with fsck.\n> \n> -- The file format includes room for precomputed generation numbers. These\n> - are not currently computed, so all generation numbers will be marked as\n> - 0 (or \"uncomputed\"). A later patch will include this calculation.\n> -\n> - After computing and storing generation numbers, we must make graph\n> walks aware of generation numbers to gain the performance benefits they\n> enable. This will mostly be accomplished by swapping a commit-date-ordered\n> priority queue with one ordered by generation number. The following\n> - operations are important candidates:\n> + operation is an important candidate:\n\nGood.\n"},{"id":"344542","messageId":"f557058d-7836-4b66-8fda-7fdfc7484219@gmail.com","threadId":"48212","inReplyTo":"xmqqfu403i0p.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 07/10] commit-graph.txt: update future work","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-12T11:35:36Z","receivedAt":"2018-04-12T11:35:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/12/2018 5:12 AM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> +Here is a diagram to visualize the shape of the full commit graph, and\n>> +how different generation numbers relate:\n>> +\n>> + +-----------------------------------------+\n>> + | GENERATION_NUMBER_INFINITY = 0xFFFFFFFF |\n>> + +-----------------------------------------+\n>> +\t | | ^\n>> +\t | | |\n>> +\t | +------+\n>> +\t | [gen(A) = gen(B)]\n>> +\t V\n>> + +-------------------------------------+\n>> + | 0 < commit->generation < 0x40000000 |\n>> + +-------------------------------------+\n>> +\t | | ^\n>> +\t | | |\n>> +\t | +------+\n>> +\t | [gen(A) > gen(B)]\n>> +\t V\n>> + +-------------------------------------+\n>> + | GENERATION_NUMBER_ZERO = 0 |\n>> + +-------------------------------------+\n>> +\t\t\t | ^\n>> +\t\t\t | |\n>> +\t\t\t +------+\n>> +\t\t [gen(A) = gen(B)]\n> It may be just me but all I can read out of the above is that\n> commit->generation may store 0xFFFFFFFF, a value between 0 and\n> 0x40000000, or 0. I cannot quite tell what the notation [gen(A)\n> gen(B)] is trying to say. I am guessing \"Two generation\n> numbers within the 'valid' range can be compared\" is what the second\n> one is trying to say, but it is much less interesting to know that\n> two infinities compare equal than how generation numbers from\n> different classes compare, which cannot be depicted in the above\n> notation, I am afraid. For example, don't we want to say that a\n> commit with INF can never be reached by a commit with a valid\n> generation number, or something like that?\n\nMy intention with the arrows was to demonstrate where parent \nrelationships can go, and the generation-number relation between a \ncommit A with parent B. Clearly, this diagram is less than helpful.\n\n>\n>> Design Details\n>> --------------\n>> \n>> @@ -98,17 +141,12 @@ Future Work\n>> - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n>> necessary for integration with fsck.\n>> \n>> -- The file format includes room for precomputed generation numbers. These\n>> - are not currently computed, so all generation numbers will be marked as\n>> - 0 (or \"uncomputed\"). A later patch will include this calculation.\n>> -\n>> - After computing and storing generation numbers, we must make graph\n>> walks aware of generation numbers to gain the performance benefits they\n>> enable. This will mostly be accomplished by swapping a commit-date-ordered\n>> priority queue with one ordered by generation number. The following\n>> - operations are important candidates:\n>> + operation is an important candidate:\n> Good.\n\n"},{"id":"344596","messageId":"86a7u7mnzi.fsf@gmail.com","threadId":"48212","inReplyTo":"f557058d-7836-4b66-8fda-7fdfc7484219@gmail.com","subject":"Re: [PATCH v2 07/10] commit-graph.txt: update future work","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-13T09:53:21Z","receivedAt":"2018-04-13T09:53:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 4/12/2018 5:12 AM, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>>\n>>> +Here is a diagram to visualize the shape of the full commit graph, and\n>>> +how different generation numbers relate:\n>>> +\n>>> + +-----------------------------------------+\n>>> + | GENERATION_NUMBER_INFINITY = 0xFFFFFFFF |\n>>> + +-----------------------------------------+\n>>> +\t | | ^\n>>> +\t | | |\n>>> +\t | +------+\n>>> +\t | [gen(A) = gen(B)]\n>>> +\t V\n>>> + +-------------------------------------+\n>>> + | 0 < commit->generation < 0x40000000 |\n>>> + +-------------------------------------+\n>>> +\t | | ^\n>>> +\t | | |\n>>> +\t | +------+\n>>> +\t | [gen(A) > gen(B)]\n>>> +\t V\n>>> + +-------------------------------------+\n>>> + | GENERATION_NUMBER_ZERO = 0 |\n>>> + +-------------------------------------+\n>>> +\t\t\t | ^\n>>> +\t\t\t | |\n>>> +\t\t\t +------+\n>>> +\t\t [gen(A) = gen(B)]\n>>\n>> It may be just me but all I can read out of the above is that\n\nIt's not just you.\n\n>> commit->generation may store 0xFFFFFFFF, a value between 0 and\n>> 0x40000000, or 0. I cannot quite tell what the notation [gen(A)\n>> gen(B)] is trying to say. I am guessing \"Two generation\n>> numbers within the 'valid' range can be compared\" is what the second\n>> one is trying to say, but it is much less interesting to know that\n>> two infinities compare equal than how generation numbers from\n>> different classes compare, which cannot be depicted in the above\n>> notation, I am afraid. For example, don't we want to say that a\n>> commit with INF can never be reached by a commit with a valid\n>> generation number, or something like that?\n>\n> My intention with the arrows was to demonstrate where parent\n> relationships can go, and the generation-number relation between a\n> commit A with parent B. Clearly, this diagram is less than helpful.\n\nPerhaps the following table would make the information clearer (perhaps\nin addition to the above graph, but without \"gen(A) {cmp} gen(B)\"\narrows).\n\nI assume that it is possible to have both GENERATION_NUMBER_ZERO and non\nzero generation numbers in one repo, perhaps via alternates. I also\nassume that A != B, and that generation numbers (both set, and 0s) are\ntransitivelu closed under reachability.\n\ngen(A) \\ commit B -> | gen(B)\n \\-----\\ |\ncommit A \\ | 0xFFFFFFFF | larger | smaller | 0x00000000\n----------------\\--------+------------+----------+---------+------------\n0xFFFFFFFF | = > > >\n0 < larger < 0x40000000 | < N = n > >\n0 < smaller < 0x40000000 | < N < N = n >\n0x00000000 | < N < N < N =\n\nThe \"<\", \"=\", \">\" denotes result of comparison between gen(A) and gen(B).\n\nGeneration numbers create a negative-cut filter: \"N\" and \"n\" denote\nsituation where we know from gen(A) and gen(B) that B is not reachable\nfrom A.\n\nAs can be seen if we use gen(A) < gen(B) as cutoff, we don't need to\ntreat \"infinity\" and \"zero\" in a special way.\n\n\nBest,\n-- \nJakub Narębski\n"},{"id":"344690","messageId":"86lgdplogn.fsf@gmail.com","threadId":"48212","inReplyTo":"45320265-9fec-cee1-e82c-3ff719bb0435@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-14T16:52:56Z","receivedAt":"2018-04-14T16:53:08Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 4/11/2018 3:32 PM, Jakub Narebski wrote:\n\n>> What would you suggest as a good test that could imply performance? The\n>> Google Colab notebook linked to above includes a function to count\n>> number of commits (nodes / vertices in the commit graph) walked,\n>> currently in the worst case scenario.\n>\n> The two main questions to consider are:\n>\n> 1. Can X reach Y?\n\nThat is easy to do. The function generic_is_reachable() does\nthat... though using direct translation of the pseudocode for\n\"Algorithm 3: Reachable\" from FELINE paper, which is recursive and\ndoesn't check if vertex was already visited was not good idea for large\ngraphs such as Linux kernel commit graph, oops. That is why\ngeneric_is_reachable_large() was created.\n\n> 2. What is the set of merge-bases between X and Y?\n\nI don't have an algorithm for that in the Google Colaboratory notebook.\nThough I see that there exist algorithms for calculating lowest common\nancestors in DAGs...\n\nI'll have to take a look how Git does that.\n\n>\n> And the thing to measure is a commit count. If possible, it would be\n> good to count commits walked (commits whose parent list is enumerated)\n> and commits inspected (commits that were listed as a parent of some\n> walked commit). Walked commits require a commit parse -- albeit from\n> the commit-graph instead of the ODB now -- while inspected commits\n> only check the in-memory cache.\n\nI don't quite see the distinction. Whether we access generation number\nof a commit (information about level of vertex in graph), or a parent\nlist (vertex successors / neighbours), it both needs accessing\ncommit-graph; well, accessing parents may be more costly for octopus\nmerges (due to having to go through EDGE chunk).\n\nI can easily return the set of visited commits (vertices), or just size\nof said set.\n\n>\n> For git.git and Linux, I like to use the release tags as tests. They\n> provide a realistic view of the linear history, and maintenance\n> releases have their own history from the major releases.\n\nHmmm... testing for v4.9-rc5..v4.9 in Linux kernel commit graphs, the\nFELINE index does not bring any improvements over using just level\n(generation number) filter. But that may be caused by narrowing od\ncommit DAG around releases.\n\nI try do do the same between commits in wide part, with many commits\nwith the same level (same generation number) both for source and for\ntarget commit. Though this may be unfair to level filter, though...\n\n\nNote however that FELINE index is not unabiguous, like generation\nnumbers are (modulo decision whether to start at 0 or at 1); it depends\non the topological ordering chosen for the X elements.\n\n>> I have tried finding number of false positives for level (generation\n>> number) filter and for FELINE index, and number of false negatives for\n>> min-post intervals in the spanning tree (for DFS tree) for 10000\n>> randomly selected pairs of commits... but I don't think this is a good\n>> benchmark.\n>\n> What is a false-positive? A case where gen(X) < gen(Y) but Y cannot\n> reach X?\n\nYes. (And equivalent for FELINE index, which is a pair of integers).\n\n> I do not think that is a great benchmark, but I guess it is\n> something to measure.\n\nI have simply used it to have something to compare.\n\n>> I Linux kernel sources (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git)\n>> that has 750832 nodes and 811733 edges, and 563747941392 possible\n>> directed pairs, we have for 10000 randomly selected pairs of commits:\n>>\n>> level-filter has 91 = 0.91% [all] false positives\n>> FELINE index has 78 = 0.78% [all] false positives\n>> FELINE index has 1.16667 less false positives than level filter\n>>\n>> min-post spanning-tree intervals has 3641 = 36.41% [all] false\n>> negatives\n>\n> Perhaps something you can do instead of sampling from N^2 commits in\n> total is to select a pair of generations (say, G = 20000, G' = 20100)\n> or regions of generations ( 20000 <= G <= 20050, 20100 <= G' <= 20150)\n> and see how many false positives you see by testing all pairs (one\n> from each level). The delta between the generations may need to be\n> smaller to actually have a large proportion of unreachable pairs. Try\n> different levels, since major version releases tend to \"pinch\" the\n> commit graph to a common history.\n\nThat's a good idea.\n\n>> For git.git repository (https://github.com/git/git.git) that has 52950\n>> nodes and 65887 edges the numbers are slighly more in FELINE index\n>> favor (also out of 10000 random pairs):\n>>\n>> level-filter has 504 = 9.11% false positives\n>> FELINE index has 125 = 2.26% false positives\n>> FELINE index has 4.032 less false positives than level filter\n>>\n>> This is for FELINE which does not use level / generatio-numbers filter.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"344883","messageId":"20180417170001.138464-1-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180409164131.37312-1-dstolee@microsoft.com","subject":"[PATCH v3 0/9] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:13Z","receivedAt":"2018-04-17T17:00:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks for all the help on v2. Here are a few changes between versions:\n\n* Removed the constant-time check in queue_has_nonstale() due to the\n possibility of a performance hit and no evidence of a performance\n benefit in typical cases.\n\n* Reordered the commits about loading commits from the commit-graph.\n This way it is easier to demonstrate the incorrect checks. On my\n machine, every commit compiles and the test suite passes, but patches\n 6-8 have the bug that is fixed in patch 9 \"merge: check config before\n loading commits\".\n\n* The interaction with parse_commit_in_graph() from parse_object() is\n replaced with a new 'check_graph' parameter in parse_commit_buffer().\n This allows us to fill in the graph_pos and generation values for\n commits that are parsed directly from a buffer. This keeps the existing\n behavior that a commit parsed this way should match its buffer.\n\n* There was discussion about making GENERATION_NUMBER_MAX assignable by\n an environment variable so we could add tests that exercise the behavior\n of capping a generation at that value. Perhaps the code around this is\n simple enough that we do not need to add that complexity.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nThis is the one of several \"small\" patches that follow the serialized\nGit commit graph patch (ds/commit-graph) and lazy-loading trees\n(ds/lazy-load-trees).\n\nAs described in Documentation/technical/commit-graph.txt, the generation\nnumber of a commit is one more than the maximum generation number among\nits parents (trivially, a commit with no parents has generation number\none). This section is expanded to describe the interaction with special\ngeneration numbers GENERATION_NUMBER_INFINITY (commits not in the commit-graph\nfile) and *_ZERO (commits in a commit-graph file written before generation\nnumbers were implemented).\n\nThis series makes the computation of generation numbers part of the\ncommit-graph write process.\n\nFinally, generation numbers are used to order commits in the priority\nqueue in paint_down_to_common(). This allows a short-circuit mechanism\nto improve performance of `git branch --contains`.\n\nFurther, use generation numbers for 'git tag --contains), providing a\nsignificant speedup (at least 95% for some cases).\n\nA more substantial refactoring of revision.c is required before making\n'git log --graph' use generation numbers effectively.\n\nThis patch series is build on ds/lazy-load-trees.\n\nDerrick Stolee (9):\n commit: add generation number to struct commmit\n commit-graph: compute generation numbers\n commit: use generations in paint_down_to_common()\n commit-graph.txt: update design document\n ref-filter: use generation number for --contains\n commit: use generation numbers for in_merge_bases()\n commit: add short-circuit to paint_down_to_common()\n commit-graph: always load commit-graph information\n merge: check config before loading commits\n\n Documentation/technical/commit-graph.txt | 30 +++++--\n alloc.c | 1 +\n builtin/merge.c | 5 +-\n commit-graph.c | 99 +++++++++++++++++++-----\n commit-graph.h | 8 ++\n commit.c | 54 +++++++++++--\n commit.h | 7 +-\n object.c | 2 +-\n ref-filter.c | 23 +++++-\n sha1_file.c | 2 +-\n t/t5318-commit-graph.sh | 9 +++\n 11 files changed, 199 insertions(+), 41 deletions(-)\n\n\nbase-commit: 7b8a21dba1bce44d64bd86427d3d92437adc4707\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344884","messageId":"20180417170001.138464-2-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 1/9] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:15Z","receivedAt":"2018-04-17T17:00:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The generation number of a commit is defined recursively as follows:\n\n* If a commit A has no parents, then the generation number of A is one.\n* If a commit A has parents, then the generation number of A is one\n more than the maximum generation number among the parents of A.\n\nAdd a uint32_t generation field to struct commit so we can pass this\ninformation to revision walks. We use three special values to signal\nthe generation number is invalid:\n\nGENERATION_NUMBER_INFINITY 0xFFFFFFFF\nGENERATION_NUMBER_MAX 0x3FFFFFFF\nGENERATION_NUMBER_ZERO 0\n\nThe first (_INFINITY) means the generation number has not been loaded or\ncomputed. The second (_MAX) means the generation number is too large to\nstore in the commit-graph file. The third (_ZERO) means the generation\nnumber was loaded from a commit graph file that was written by a version\nof git that did not support generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 2 ++\n commit.h | 4 ++++\n 3 files changed, 7 insertions(+)\n\ndiff --git a/alloc.c b/alloc.c\nindex cf4f8b61e1..e8ab14f4a1 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n+\tc->generation = GENERATION_NUMBER_INFINITY;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 70fa1b25fd..9ad21c3ffb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -262,6 +262,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \tdate_low = get_be32(commit_data + g->hash_len + 12);\n \titem->date = (timestamp_t)((date_high << 32) | date_low);\n \n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+\n \tpptr = &item->parents;\n \n \tedge_value = get_be32(commit_data + g->hash_len);\ndiff --git a/commit.h b/commit.h\nindex 23a3f364ed..aac3b8c56f 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -10,6 +10,9 @@\n #include \"pretty.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+#define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n+#define GENERATION_NUMBER_MAX 0x3FFFFFFF\n+#define GENERATION_NUMBER_ZERO 0\n \n struct commit_list {\n \tstruct commit *item;\n@@ -30,6 +33,7 @@ struct commit {\n \t */\n \tstruct tree *maybe_tree;\n \tuint32_t graph_pos;\n+\tuint32_t generation;\n };\n \n extern int save_commit_buffer;\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344885","messageId":"20180417170001.138464-4-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 3/9] commit: use generations in paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:21Z","receivedAt":"2018-04-17T17:00:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Define compare_commits_by_gen_then_commit_date(), which uses generation\nnumbers as a primary comparison and commit date to break ties (or as a\ncomparison when both commits do not have computed generation numbers).\n\nSince the commit-graph file is closed under reachability, we know that\nall commits in the file have generation at most GENERATION_NUMBER_MAX\nwhich is less than GENERATION_NUMBER_INFINITY.\n\nThis change does not affect the number of commits that are walked during\nthe execution of paint_down_to_common(), only the order that those\ncommits are inspected. In the case that commit dates violate topological\norder (i.e. a parent is \"newer\" than a child), the previous code could\nwalk a commit twice: if a commit is reached with the PARENT1 bit, but\nlater is re-visited with the PARENT2 bit, then that PARENT2 bit must be\npropagated to its parents. Using generation numbers avoids this extra\neffort, even if it is somewhat rare.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 20 +++++++++++++++++++-\n commit.h | 1 +\n 2 files changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 711f674c18..a44899c733 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -640,6 +640,24 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n \treturn 0;\n }\n \n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n+{\n+\tconst struct commit *a = a_, *b = b_;\n+\n+\t/* newer commits first */\n+\tif (a->generation < b->generation)\n+\t\treturn 1;\n+\telse if (a->generation > b->generation)\n+\t\treturn -1;\n+\n+\t/* use date as a heuristic when generataions are equal */\n+\tif (a->date < b->date)\n+\t\treturn 1;\n+\telse if (a->date > b->date)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n {\n \tconst struct commit *a = a_, *b = b_;\n@@ -789,7 +807,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n /* all input commits in one and twos[] must have been parsed! */\n static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n {\n-\tstruct prio_queue queue = { compare_commits_by_commit_date };\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n \ndiff --git a/commit.h b/commit.h\nindex aac3b8c56f..64436ff44e 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -341,6 +341,7 @@ extern int remove_signature(struct strbuf *buf);\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \n LAST_ARG_MUST_BE_NULL\n extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344886","messageId":"20180417170001.138464-3-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 2/9] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:18Z","receivedAt":"2018-04-17T17:00:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While preparing commits to be written into a commit-graph file, compute\nthe generation numbers using a depth-first strategy.\n\nThe only commits that are walked in this depth-first search are those\nwithout a precomputed generation number. Thus, computation time will be\nrelative to the number of new commits to the commit-graph file.\n\nIf a computed generation number would exceed GENERATION_NUMBER_MAX, then\nuse GENERATION_NUMBER_MAX instead.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 46 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9ad21c3ffb..688d5b1801 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -439,6 +439,10 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\telse\n \t\t\tpackedDate[0] = 0;\n \n+\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY) {\n+\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n+\t\t}\n+\n \t\tpackedDate[1] = htonl((*list)->date);\n \t\thashwrite(f, packedDate, 8);\n \n@@ -571,6 +575,46 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n }\n \n+static void compute_generation_numbers(struct commit** commits,\n+\t\t\t\t int nr_commits)\n+{\n+\tint i;\n+\tstruct commit_list *list = NULL;\n+\n+\tfor (i = 0; i < nr_commits; i++) {\n+\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n+\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n+\t\t\tcontinue;\n+\n+\t\tcommit_list_insert(commits[i], &list);\n+\t\twhile (list) {\n+\t\t\tstruct commit *current = list->item;\n+\t\t\tstruct commit_list *parent;\n+\t\t\tint all_parents_computed = 1;\n+\t\t\tuint32_t max_generation = 0;\n+\n+\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n+\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n+\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n+\t\t\t\t\tall_parents_computed = 0;\n+\t\t\t\t\tcommit_list_insert(parent->item, &list);\n+\t\t\t\t\tbreak;\n+\t\t\t\t} else if (parent->item->generation > max_generation) {\n+\t\t\t\t\tmax_generation = parent->item->generation;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (all_parents_computed) {\n+\t\t\t\tcurrent->generation = max_generation + 1;\n+\t\t\t\tpop_commit(&list);\n+\t\t\t}\n+\n+\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n+\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n+\t\t}\n+\t}\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tconst char **pack_indexes,\n \t\t\tint nr_packs,\n@@ -694,6 +738,8 @@ void write_commit_graph(const char *obj_dir,\n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n \n+\tcompute_generation_numbers(commits.list, commits.nr);\n+\n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344887","messageId":"20180417170001.138464-5-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 4/9] commit-graph.txt: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:24Z","receivedAt":"2018-04-17T17:00:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We now calculate generation numbers in the commit-graph file and use\nthem in paint_down_to_common().\n\nExpand the section on generation numbers to discuss how the three\nspecial generation numbers GENERATION_NUMBER_INFINITY, _ZERO, and\n_MAX interact with other generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 30 +++++++++++++++++++-----\n 1 file changed, 24 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 0550c6d0dc..d9f2713efa 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -77,6 +77,29 @@ in the commit graph. We can treat these commits as having \"infinite\"\n generation number and walk until reaching commits with known generation\n number.\n \n+We use the macro GENERATION_NUMBER_INFINITY = 0xFFFFFFFF to mark commits not\n+in the commit-graph file. If a commit-graph file was written by a version\n+of Git that did not compute generation numbers, then those commits will\n+have generation number represented by the macro GENERATION_NUMBER_ZERO = 0.\n+\n+Since the commit-graph file is closed under reachability, we can guarantee\n+the following weaker condition on all commits:\n+\n+ If A and B are commits with generation numbers N amd M, respectively,\n+ and N < M, then A cannot reach B.\n+\n+Note how the strict inequality differs from the inequality when we have\n+fully-computed generation numbers. Using strict inequality may result in\n+walking a few extra commits, but the simplicity in dealing with commits\n+with generation number *_INFINITY or *_ZERO is valuable.\n+\n+We use the macro GENERATION_NUMBER_MAX = 0x3FFFFFFF to for commits whose\n+generation numbers are computed to be at least this value. We limit at\n+this value since it is the largest value that can be stored in the\n+commit-graph file using the 30 bits available to generation numbers. This\n+presents another case where a commit can have generation number equal to\n+that of a parent.\n+\n Design Details\n --------------\n \n@@ -98,17 +121,12 @@ Future Work\n - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n necessary for integration with fsck.\n \n-- The file format includes room for precomputed generation numbers. These\n- are not currently computed, so all generation numbers will be marked as\n- 0 (or \"uncomputed\"). A later patch will include this calculation.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n priority queue with one ordered by generation number. The following\n- operations are important candidates:\n+ operation is an important candidate:\n \n- - paint_down_to_common()\n - 'log --topo-order'\n \n - Currently, parse_commit_gently() requires filling in the root tree\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344888","messageId":"20180417170001.138464-7-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 6/9] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:30Z","receivedAt":"2018-04-17T17:00:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The containment algorithm for 'git branch --contains' is different\nfrom that for 'git tag --contains' in that it uses is_descendant_of()\ninstead of contains_tag_algo(). The expensive portion of the branch\nalgorithm is computing merge bases.\n\nWhen a commit-graph file exists with generation numbers computed,\nwe can avoid this merge-base calculation when the target commit has\na larger generation number than the target commits.\n\nPerformance tests were run on a copy of the Linux repository where\nHEAD is contained in v4.13 but no earlier tag. Also, all tags were\ncopied to branches and 'git branch --contains' was tested:\n\nBefore: 60.0s\nAfter: 0.4s\nRel %: -99.3%\n\nReported-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex a44899c733..bceb79c419 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1053,12 +1053,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n {\n \tstruct commit_list *bases;\n \tint ret = 0, i;\n+\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n \n \tif (parse_commit(commit))\n \t\treturn ret;\n-\tfor (i = 0; i < nr_reference; i++)\n+\tfor (i = 0; i < nr_reference; i++) {\n \t\tif (parse_commit(reference[i]))\n \t\t\treturn ret;\n+\t\tif (min_generation > reference[i]->generation)\n+\t\t\tmin_generation = reference[i]->generation;\n+\t}\n+\n+\tif (commit->generation > min_generation)\n+\t\treturn 0;\n \n \tbases = paint_down_to_common(commit, nr_reference, reference);\n \tif (commit->object.flags & PARENT2)\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344889","messageId":"20180417170001.138464-8-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 7/9] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:32Z","receivedAt":"2018-04-17T17:00:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git branch --contains', the in_merge_bases_many()\nmethod calls paint_down_to_common() to discover if a specific\ncommit is reachable from a set of branches. Commits with lower\ngeneration number are not needed to correctly answer the\ncontainment query of in_merge_bases_many().\n\nAdd a new parameter, min_generation, to paint_down_to_common() that\nprevents walking commits with generation number strictly less than\nmin_generation. If 0 is given, then there is no functional change.\n\nFor in_merge_bases_many(), we can pass commit->generation as the\ncutoff, and this saves time during 'git branch --contains' queries\nthat would otherwise walk \"around\" the commit we are inspecting.\n\nFor a copy of the Linux repository, where HEAD is checked out at\nv4.13~100, we get the following performance improvement for\n'git branch --contains' over the previous commit:\n\nBefore: 0.21s\nAfter: 0.13s\nRel %: -38%\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex bceb79c419..a70f120878 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -805,11 +805,14 @@ static int queue_has_nonstale(struct prio_queue *queue)\n }\n \n /* all input commits in one and twos[] must have been parsed! */\n-static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n+static struct commit_list *paint_down_to_common(struct commit *one, int n,\n+\t\t\t\t\t\tstruct commit **twos,\n+\t\t\t\t\t\tint min_generation)\n {\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n+\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n \n \tone->object.flags |= PARENT1;\n \tif (!n) {\n@@ -828,6 +831,13 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \t\tstruct commit_list *parents;\n \t\tint flags;\n \n+\t\tif (commit->generation > last_gen)\n+\t\t\tBUG(\"bad generation skip\");\n+\t\tlast_gen = commit->generation;\n+\n+\t\tif (commit->generation < min_generation)\n+\t\t\tbreak;\n+\n \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n \t\tif (flags == (PARENT1 | PARENT2)) {\n \t\t\tif (!(commit->object.flags & RESULT)) {\n@@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n \t\t\treturn NULL;\n \t}\n \n-\tlist = paint_down_to_common(one, n, twos);\n+\tlist = paint_down_to_common(one, n, twos, 0);\n \n \twhile (list) {\n \t\tstruct commit *commit = pop_commit(&list);\n@@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n \t\t\tfilled_index[filled] = j;\n \t\t\twork[filled++] = array[j];\n \t\t}\n-\t\tcommon = paint_down_to_common(array[i], filled, work);\n+\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n \t\tif (array[i]->object.flags & PARENT2)\n \t\t\tredundant[i] = 1;\n \t\tfor (j = 0; j < filled; j++)\n@@ -1067,7 +1077,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n \tif (commit->generation > min_generation)\n \t\treturn 0;\n \n-\tbases = paint_down_to_common(commit, nr_reference, reference);\n+\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n \tif (commit->object.flags & PARENT2)\n \t\tret = 1;\n \tclear_commit_marks(commit, all_flags);\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344890","messageId":"20180417170001.138464-9-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 8/9] commit-graph: always load commit-graph information","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:35Z","receivedAt":"2018-04-17T17:00:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Most code paths load commits using lookup_commit() and then\nparse_commit(). In some cases, including some branch lookups, the commit\nis parsed using parse_object_buffer() which side-steps parse_commit() in\nfavor of parse_commit_buffer().\n\nWith generation numbers in the commit-graph, we need to ensure that any\ncommit that exists in the commit-graph file has its generation number\nloaded.\n\nCreate new load_commit_graph_info() method to fill in the information\nfor a commit that exists only in the commit-graph file. Call it from\nparse_commit_buffer() after loading the other commit information from\nthe given buffer. Only fill this information when specified by the\n'check_graph' parameter. This avoids duplicate work when we already\nchecked the graph in parse_commit_gently() or when simply checking the\nbuffer contents in check_commit().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 51 ++++++++++++++++++++++++++++++++------------------\n commit-graph.h | 8 ++++++++\n commit.c | 7 +++++--\n commit.h | 2 +-\n object.c | 2 +-\n sha1_file.c | 2 +-\n 6 files changed, 49 insertions(+), 23 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 688d5b1801..21e853c21a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -245,13 +245,19 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n \treturn &commit_list_insert(c, pptr)->next;\n }\n \n+static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n+{\n+\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+}\n+\n static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n \tuint32_t edge_value;\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n-\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n \n \titem->object.parsed = 1;\n \titem->graph_pos = pos;\n@@ -292,31 +298,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \treturn 1;\n }\n \n+static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n+{\n+\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n+\t\t*pos = item->graph_pos;\n+\t\treturn 1;\n+\t} else {\n+\t\treturn bsearch_graph(commit_graph, &(item->object.oid), pos);\n+\t}\n+}\n+\n int parse_commit_in_graph(struct commit *item)\n {\n+\tuint32_t pos;\n+\n+\tif (item->object.parsed)\n+\t\treturn 0;\n \tif (!core_commit_graph)\n \t\treturn 0;\n-\tif (item->object.parsed)\n-\t\treturn 1;\n-\n \tprepare_commit_graph();\n-\tif (commit_graph) {\n-\t\tuint32_t pos;\n-\t\tint found;\n-\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n-\t\t\tpos = item->graph_pos;\n-\t\t\tfound = 1;\n-\t\t} else {\n-\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n-\t\t}\n-\n-\t\tif (found)\n-\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n-\t}\n-\n+\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n+\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n \treturn 0;\n }\n \n+void load_commit_graph_info(struct commit *item)\n+{\n+\tuint32_t pos;\n+\tif (!core_commit_graph)\n+\t\treturn;\n+\tprepare_commit_graph();\n+\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n+\t\tfill_commit_graph_info(item, commit_graph, pos);\n+}\n+\n static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n {\n \tstruct object_id oid;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 260a468e73..96cccb10f3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n */\n int parse_commit_in_graph(struct commit *item);\n \n+/*\n+ * It is possible that we loaded commit contents from the commit buffer,\n+ * but we also want to ensure the commit-graph content is correctly\n+ * checked and filled. Fill the graph_pos and generation members of\n+ * the given commit.\n+ */\n+void load_commit_graph_info(struct commit *item);\n+\n struct tree *get_commit_tree_in_graph(const struct commit *c);\n \n struct commit_graph {\ndiff --git a/commit.c b/commit.c\nindex a70f120878..9ef6f699bd 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n \treturn ret;\n }\n \n-int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n+int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n {\n \tconst char *tail = buffer;\n \tconst char *bufptr = buffer;\n@@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \t}\n \titem->date = parse_commit_date(bufptr, tail);\n \n+\tif (check_graph)\n+\t\tload_commit_graph_info(item);\n+\n \treturn 0;\n }\n \n@@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn error(\"Object %s not a commit\",\n \t\t\t oid_to_hex(&item->object.oid));\n \t}\n-\tret = parse_commit_buffer(item, buffer, size);\n+\tret = parse_commit_buffer(item, buffer, size, 0);\n \tif (save_commit_buffer && !ret) {\n \t\tset_commit_buffer(item, buffer, size);\n \t\treturn 0;\ndiff --git a/commit.h b/commit.h\nindex 64436ff44e..b5afde1ae9 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n */\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n-int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n+int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\ndiff --git a/object.c b/object.c\nindex e6ad3f61f0..efe4871325 100644\n--- a/object.c\n+++ b/object.c\n@@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n \t} else if (type == OBJ_COMMIT) {\n \t\tstruct commit *commit = lookup_commit(oid);\n \t\tif (commit) {\n-\t\t\tif (parse_commit_buffer(commit, buffer, size))\n+\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n \t\t\t\treturn NULL;\n \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n \t\t\t\tset_commit_buffer(commit, buffer, size);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1b94f39c4c..0fd4f0b8b6 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n {\n \tstruct commit c;\n \tmemset(&c, 0, sizeof(c));\n-\tif (parse_commit_buffer(&c, buf, size))\n+\tif (parse_commit_buffer(&c, buf, size, 0))\n \t\tdie(\"corrupt commit\");\n }\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344891","messageId":"20180417170001.138464-10-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 9/9] merge: check config before loading commits","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:38Z","receivedAt":"2018-04-17T17:00:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Now that we use generation numbers from the commit-graph, we must\nensure that all commits that exist in the commit-graph are loaded\nfrom that file instead of from the object database. Since the\ncommit-graph file is only checked if core.commitGraph is true, we\nmust check the default config before we load any commits.\n\nIn the merge builtin, the config was checked after loading the HEAD\ncommit. This was due to the use of the global 'branch' when checking\nmerge-specific config settings.\n\nMove the config load to be between the initialization of 'branch' and\nthe commit lookup.\n\nWithout this change, a fast-forward merge would hit a BUG(\"bad\ngeneration skip\") statement in commit.c during paint_down_to_common().\nThis is because the HEAD commit would be loaded with \"infinite\"\ngeneration but then reached by commits with \"finite\" generation\nnumbers.\n\nAdd a test to t5318-commit-graph.sh that exercises this code path to\nprevent a regression.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/merge.c | 5 +++--\n t/t5318-commit-graph.sh | 9 +++++++++\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 5e5e4497e3..7e1da6c6ea 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -1148,13 +1148,14 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n \tif (branch)\n \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n+\tinit_diff_ui_defaults();\n+\tgit_config(git_merge_config, NULL);\n+\n \tif (!branch || is_null_oid(&head_oid))\n \t\thead_commit = NULL;\n \telse\n \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n \n-\tinit_diff_ui_defaults();\n-\tgit_config(git_merge_config, NULL);\n \n \tif (branch_mergeoptions)\n \t\tparse_branch_merge_options(branch_mergeoptions);\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a380419b65..77d85aefe7 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n \n+test_expect_success 'perform fast-forward merge in full repo' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit checkout -b merge-5-to-8 commits/5 &&\n+\tgit merge commits/8 &&\n+\tgit show-ref -s merge-5-to-8 >output &&\n+\tgit show-ref -s commits/8 >expect &&\n+\ttest_cmp expect output\n+'\n+\n test_done\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344892","messageId":"20180417170001.138464-6-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v3 5/9] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T17:00:27Z","receivedAt":"2018-04-17T17:01:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"A commit A can reach a commit B only if the generation number of A\nis larger than the generation number of B. This condition allows\nsignificantly short-circuiting commit-graph walks.\n\nUse generation number for 'git tag --contains' queries.\n\nOn a copy of the Linux repository where HEAD is containd in v4.13\nbut no earlier tag, the command 'git tag --contains HEAD' had the\nfollowing peformance improvement:\n\nBefore: 0.81s\nAfter: 0.04s\nRel %: -95%\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n ref-filter.c | 23 +++++++++++++++++++----\n 1 file changed, 19 insertions(+), 4 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex cffd8bf3ce..e2fea6d635 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1587,7 +1587,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n */\n static enum contains_result contains_test(struct commit *candidate,\n \t\t\t\t\t const struct commit_list *want,\n-\t\t\t\t\t struct contains_cache *cache)\n+\t\t\t\t\t struct contains_cache *cache,\n+\t\t\t\t\t uint32_t cutoff)\n {\n \tenum contains_result *cached = contains_cache_at(cache, candidate);\n \n@@ -1603,6 +1604,10 @@ static enum contains_result contains_test(struct commit *candidate,\n \n \t/* Otherwise, we don't know; prepare to recurse */\n \tparse_commit_or_die(candidate);\n+\n+\tif (candidate->generation < cutoff)\n+\t\treturn CONTAINS_NO;\n+\n \treturn CONTAINS_UNKNOWN;\n }\n \n@@ -1618,8 +1623,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t\t\t\t struct contains_cache *cache)\n {\n \tstruct contains_stack contains_stack = { 0, 0, NULL };\n-\tenum contains_result result = contains_test(candidate, want, cache);\n+\tenum contains_result result;\n+\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n+\tconst struct commit_list *p;\n+\n+\tfor (p = want; p; p = p->next) {\n+\t\tstruct commit *c = p->item;\n+\t\tparse_commit_or_die(c);\n+\t\tif (c->generation < cutoff)\n+\t\t\tcutoff = c->generation;\n+\t}\n \n+\tresult = contains_test(candidate, want, cache, cutoff);\n \tif (result != CONTAINS_UNKNOWN)\n \t\treturn result;\n \n@@ -1637,7 +1652,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t * If we just popped the stack, parents->item has been marked,\n \t\t * therefore contains_test will return a meaningful yes/no.\n \t\t */\n-\t\telse switch (contains_test(parents->item, want, cache)) {\n+\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n \t\tcase CONTAINS_YES:\n \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n \t\t\tcontains_stack.nr--;\n@@ -1651,7 +1666,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t}\n \t}\n \tfree(contains_stack.contains_stack);\n-\treturn contains_test(candidate, want, cache);\n+\treturn contains_test(candidate, want, cache, cutoff);\n }\n \n static int commit_contains(struct ref_filter *filter, struct commit *commit,\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344900","messageId":"8500bf59-1910-83db-cdd9-9ccc7d2fe39e@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-9-dstolee@microsoft.com","subject":"Re: [PATCH v3 8/9] commit-graph: always load commit-graph information","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-17T17:50:51Z","receivedAt":"2018-04-17T17:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/17/2018 1:00 PM, Derrick Stolee wrote:\n> Most code paths load commits using lookup_commit() and then\n> parse_commit(). In some cases, including some branch lookups, the commit\n> is parsed using parse_object_buffer() which side-steps parse_commit() in\n> favor of parse_commit_buffer().\n>\n> With generation numbers in the commit-graph, we need to ensure that any\n> commit that exists in the commit-graph file has its generation number\n> loaded.\n>\n> Create new load_commit_graph_info() method to fill in the information\n> for a commit that exists only in the commit-graph file. Call it from\n> parse_commit_buffer() after loading the other commit information from\n> the given buffer. Only fill this information when specified by the\n> 'check_graph' parameter. This avoids duplicate work when we already\n> checked the graph in parse_commit_gently() or when simply checking the\n> buffer contents in check_commit().\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 51 ++++++++++++++++++++++++++++++++------------------\n> commit-graph.h | 8 ++++++++\n> commit.c | 7 +++++--\n> commit.h | 2 +-\n> object.c | 2 +-\n> sha1_file.c | 2 +-\n> 6 files changed, 49 insertions(+), 23 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 688d5b1801..21e853c21a 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -245,13 +245,19 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n> \treturn &commit_list_insert(c, pptr)->next;\n> }\n> \n> +static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n> +{\n> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n> +}\n> +\n> static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n> {\n> \tuint32_t edge_value;\n> \tuint32_t *parent_data_ptr;\n> \tuint64_t date_low, date_high;\n> \tstruct commit_list **pptr;\n> -\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n> \n> \titem->object.parsed = 1;\n> \titem->graph_pos = pos;\n> @@ -292,31 +298,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \treturn 1;\n> }\n> \n> +static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n> +{\n> +\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n> +\t\t*pos = item->graph_pos;\n> +\t\treturn 1;\n> +\t} else {\n> +\t\treturn bsearch_graph(commit_graph, &(item->object.oid), pos);\n\nThe reference to 'commit_graph' in the above line should be 'g'. Sorry!\n\n> +\t}\n> +}\n> +\n> int parse_commit_in_graph(struct commit *item)\n> {\n> +\tuint32_t pos;\n> +\n> +\tif (item->object.parsed)\n> +\t\treturn 0;\n> \tif (!core_commit_graph)\n> \t\treturn 0;\n> -\tif (item->object.parsed)\n> -\t\treturn 1;\n> -\n> \tprepare_commit_graph();\n> -\tif (commit_graph) {\n> -\t\tuint32_t pos;\n> -\t\tint found;\n> -\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n> -\t\t\tpos = item->graph_pos;\n> -\t\t\tfound = 1;\n> -\t\t} else {\n> -\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n> -\t\t}\n> -\n> -\t\tif (found)\n> -\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> -\t}\n> -\n> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> +\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> \treturn 0;\n> }\n> \n> +void load_commit_graph_info(struct commit *item)\n> +{\n> +\tuint32_t pos;\n> +\tif (!core_commit_graph)\n> +\t\treturn;\n> +\tprepare_commit_graph();\n> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> +\t\tfill_commit_graph_info(item, commit_graph, pos);\n> +}\n> +\n> static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n> {\n> \tstruct object_id oid;\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 260a468e73..96cccb10f3 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n> */\n> int parse_commit_in_graph(struct commit *item);\n> \n> +/*\n> + * It is possible that we loaded commit contents from the commit buffer,\n> + * but we also want to ensure the commit-graph content is correctly\n> + * checked and filled. Fill the graph_pos and generation members of\n> + * the given commit.\n> + */\n> +void load_commit_graph_info(struct commit *item);\n> +\n> struct tree *get_commit_tree_in_graph(const struct commit *c);\n> \n> struct commit_graph {\n> diff --git a/commit.c b/commit.c\n> index a70f120878..9ef6f699bd 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n> \treturn ret;\n> }\n> \n> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n> {\n> \tconst char *tail = buffer;\n> \tconst char *bufptr = buffer;\n> @@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n> \t}\n> \titem->date = parse_commit_date(bufptr, tail);\n> \n> +\tif (check_graph)\n> +\t\tload_commit_graph_info(item);\n> +\n> \treturn 0;\n> }\n> \n> @@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \t\treturn error(\"Object %s not a commit\",\n> \t\t\t oid_to_hex(&item->object.oid));\n> \t}\n> -\tret = parse_commit_buffer(item, buffer, size);\n> +\tret = parse_commit_buffer(item, buffer, size, 0);\n> \tif (save_commit_buffer && !ret) {\n> \t\tset_commit_buffer(item, buffer, size);\n> \t\treturn 0;\n> diff --git a/commit.h b/commit.h\n> index 64436ff44e..b5afde1ae9 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n> */\n> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n> \n> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n> int parse_commit_gently(struct commit *item, int quiet_on_missing);\n> static inline int parse_commit(struct commit *item)\n> {\n> diff --git a/object.c b/object.c\n> index e6ad3f61f0..efe4871325 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n> \t} else if (type == OBJ_COMMIT) {\n> \t\tstruct commit *commit = lookup_commit(oid);\n> \t\tif (commit) {\n> -\t\t\tif (parse_commit_buffer(commit, buffer, size))\n> +\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n> \t\t\t\treturn NULL;\n> \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n> \t\t\t\tset_commit_buffer(commit, buffer, size);\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 1b94f39c4c..0fd4f0b8b6 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n> {\n> \tstruct commit c;\n> \tmemset(&c, 0, sizeof(c));\n> -\tif (parse_commit_buffer(&c, buf, size))\n> +\tif (parse_commit_buffer(&c, buf, size, 0))\n> \t\tdie(\"corrupt commit\");\n> }\n> \n\n"},{"id":"344984","messageId":"864lk8io21.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-4-dstolee@microsoft.com","subject":"Re: [PATCH v3 3/9] commit: use generations in paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-18T14:31:18Z","receivedAt":"2018-04-18T14:31:28Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Define compare_commits_by_gen_then_commit_date(), which uses generation\n> numbers as a primary comparison and commit date to break ties (or as a\n> comparison when both commits do not have computed generation numbers).\n>\n> Since the commit-graph file is closed under reachability, we know that\n> all commits in the file have generation at most GENERATION_NUMBER_MAX\n> which is less than GENERATION_NUMBER_INFINITY.\n>\n> This change does not affect the number of commits that are walked during\n> the execution of paint_down_to_common(), only the order that those\n> commits are inspected. In the case that commit dates violate topological\n> order (i.e. a parent is \"newer\" than a child), the previous code could\n> walk a commit twice: if a commit is reached with the PARENT1 bit, but\n> later is re-visited with the PARENT2 bit, then that PARENT2 bit must be\n> propagated to its parents. Using generation numbers avoids this extra\n> effort, even if it is somewhat rare.\n\nDoes it mean that it gives no measureable performance improvements for\ntypical test cases?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 20 +++++++++++++++++++-\n> commit.h | 1 +\n> 2 files changed, 20 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit.c b/commit.c\n> index 711f674c18..a44899c733 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -640,6 +640,24 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n> \treturn 0;\n> }\n> \n> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n> +{\n> +\tconst struct commit *a = a_, *b = b_;\n> +\n> +\t/* newer commits first */\n> +\tif (a->generation < b->generation)\n> +\t\treturn 1;\n> +\telse if (a->generation > b->generation)\n> +\t\treturn -1;\n> +\n> +\t/* use date as a heuristic when generataions are equal */\n\nVery minor typo in above comment:\n\ns/generataions/generations/\n\n> +\tif (a->date < b->date)\n> +\t\treturn 1;\n> +\telse if (a->date > b->date)\n> +\t\treturn -1;\n> +\treturn 0;\n> +}\n> +\n> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n> {\n> \tconst struct commit *a = a_, *b = b_;\n> @@ -789,7 +807,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n> /* all input commits in one and twos[] must have been parsed! */\n> static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n> {\n> -\tstruct prio_queue queue = { compare_commits_by_commit_date };\n> +\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n> \tstruct commit_list *result = NULL;\n> \tint i;\n> \n> diff --git a/commit.h b/commit.h\n> index aac3b8c56f..64436ff44e 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -341,6 +341,7 @@ extern int remove_signature(struct strbuf *buf);\n> extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n> \n> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n> \n> LAST_ARG_MUST_BE_NULL\n> extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n"},{"id":"344986","messageId":"d4095ab6-fca2-8ed5-0185-1f961c5d1168@gmail.com","threadId":"48212","inReplyTo":"864lk8io21.fsf@gmail.com","subject":"Re: [PATCH v3 3/9] commit: use generations in paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-18T14:46:27Z","receivedAt":"2018-04-18T14:46:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/18/2018 10:31 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> Define compare_commits_by_gen_then_commit_date(), which uses generation\n>> numbers as a primary comparison and commit date to break ties (or as a\n>> comparison when both commits do not have computed generation numbers).\n>>\n>> Since the commit-graph file is closed under reachability, we know that\n>> all commits in the file have generation at most GENERATION_NUMBER_MAX\n>> which is less than GENERATION_NUMBER_INFINITY.\n>>\n>> This change does not affect the number of commits that are walked during\n>> the execution of paint_down_to_common(), only the order that those\n>> commits are inspected. In the case that commit dates violate topological\n>> order (i.e. a parent is \"newer\" than a child), the previous code could\n>> walk a commit twice: if a commit is reached with the PARENT1 bit, but\n>> later is re-visited with the PARENT2 bit, then that PARENT2 bit must be\n>> propagated to its parents. Using generation numbers avoids this extra\n>> effort, even if it is somewhat rare.\n> Does it mean that it gives no measureable performance improvements for\n> typical test cases?\n\nNot in this commit. When we add the `min_generation` parameter in a \nlater commit, we do get a significant performance boost (when we can \nsupply a non-zero value to `min_generation`).\n\nThis step of using generation numbers for the priority is important for \nthat commit, but on its own has limited value outside of the clock-skew \ncase mentioned above.\n\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit.c | 20 +++++++++++++++++++-\n>> commit.h | 1 +\n>> 2 files changed, 20 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/commit.c b/commit.c\n>> index 711f674c18..a44899c733 100644\n>> --- a/commit.c\n>> +++ b/commit.c\n>> @@ -640,6 +640,24 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n>> \treturn 0;\n>> }\n>> \n>> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n>> +{\n>> +\tconst struct commit *a = a_, *b = b_;\n>> +\n>> +\t/* newer commits first */\n>> +\tif (a->generation < b->generation)\n>> +\t\treturn 1;\n>> +\telse if (a->generation > b->generation)\n>> +\t\treturn -1;\n>> +\n>> +\t/* use date as a heuristic when generataions are equal */\n> Very minor typo in above comment:\n>\n> s/generataions/generations/\n\nGood catch!\n\n>\n>> +\tif (a->date < b->date)\n>> +\t\treturn 1;\n>> +\telse if (a->date > b->date)\n>> +\t\treturn -1;\n>> +\treturn 0;\n>> +}\n>> +\n>> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n>> {\n>> \tconst struct commit *a = a_, *b = b_;\n>> @@ -789,7 +807,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n>> /* all input commits in one and twos[] must have been parsed! */\n>> static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n>> {\n>> -\tstruct prio_queue queue = { compare_commits_by_commit_date };\n>> +\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n>> \tstruct commit_list *result = NULL;\n>> \tint i;\n>> \n>> diff --git a/commit.h b/commit.h\n>> index aac3b8c56f..64436ff44e 100644\n>> --- a/commit.h\n>> +++ b/commit.h\n>> @@ -341,6 +341,7 @@ extern int remove_signature(struct strbuf *buf);\n>> extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n>> \n>> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n>> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n>> \n>> LAST_ARG_MUST_BE_NULL\n>> extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n\n"},{"id":"345005","messageId":"86y3hkguu3.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-5-dstolee@microsoft.com","subject":"Re: [PATCH v3 4/9] commit-graph.txt: update design document","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-18T19:47:48Z","receivedAt":"2018-04-18T19:47:57Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> We now calculate generation numbers in the commit-graph file and use\n> them in paint_down_to_common().\n\nAll right.\n\n>\n> Expand the section on generation numbers to discuss how the three\n> special generation numbers GENERATION_NUMBER_INFINITY, _ZERO, and\n> _MAX interact with other generation numbers.\n\nVery good.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 30 +++++++++++++++++++-----\n> 1 file changed, 24 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index 0550c6d0dc..d9f2713efa 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -77,6 +77,29 @@ in the commit graph. We can treat these commits as having \"infinite\"\n> generation number and walk until reaching commits with known generation\n> number.\n>\n> +We use the macro GENERATION_NUMBER_INFINITY = 0xFFFFFFFF to mark commits not\n> +in the commit-graph file. If a commit-graph file was written by a version\n> +of Git that did not compute generation numbers, then those commits will\n> +have generation number represented by the macro GENERATION_NUMBER_ZERO = 0.\n\nI have to wonder if there would be any relesed Git that do not compute\ngeneration numbers...\n\nOn the other hand in case the user-visible view of the project history\nchanges, be it because shallow clone is shortened or deepened, or grafts\nfile is edited, or a commit object is replaced with another with\ndifferent parents - we can still use \"commit-graph\" data, just pretend\nthat generation numbers (which are invalid in altered history) are all\nzero. (I'll write about this idea in comments to later series.)\n\nOn the other hand with GENERATION_NUMBER_ZERO these series of patches\nare self-contained and bisectable.\n\n> +\n> +Since the commit-graph file is closed under reachability, we can guarantee\n> +the following weaker condition on all commits:\n\nI have had to look up the contents of the whole file, but it turns out\nthat it is all right: \"weaker condition\" refers to earlier \"N <= M\".\n\nMinor sidenote: if one would be extremly pedantic, one could say that\nprevious condition is incorrect, because it doesn't state explicitely\nthat commit A != commit B. ;-)\n\n> +\n> + If A and B are commits with generation numbers N amd M, respectively,\n> + and N < M, then A cannot reach B.\n> +\n> +Note how the strict inequality differs from the inequality when we have\n> +fully-computed generation numbers. Using strict inequality may result in\n> +walking a few extra commits, but the simplicity in dealing with commits\n> +with generation number *_INFINITY or *_ZERO is valuable.\n> +\n> +We use the macro GENERATION_NUMBER_MAX = 0x3FFFFFFF to for commits whose\n> +generation numbers are computed to be at least this value. We limit at\n> +this value since it is the largest value that can be stored in the\n> +commit-graph file using the 30 bits available to generation numbers. This\n> +presents another case where a commit can have generation number equal to\n> +that of a parent.\n\nI wonder if something like the table I have proposed in v2 version of\nthis patch [1] would make it easier or harder to understand.\n\n[1]: https://public-inbox.org/git/86a7u7mnzi.fsf@gmail.com/\n\nSomething like the following:\n\n | gen(B)\n |\ngen(A) | _INFINITY | _MAX | larger | smaller | _ZERO\n-------------+-----------+----------+----------+----------+--------\n_INFINITY | = | > | > | > | >\n_MAX | < N | = | > | > | >\nlarger | < N | < N | = n | > | >\nsmaller | < N | < N | < N | = n | >\n_ZERO | < N | < N | < N | < N | =\n\nHere \"n\" and \"N\" denotes stronger condition, and \"N\" denotes weaker\ncondition. We have _INFINITY > _MAX > larger > smaller > _ZERO.\n\n> +\n> Design Details\n> --------------\n>\n> @@ -98,17 +121,12 @@ Future Work\n> - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n> necessary for integration with fsck.\n>\n> -- The file format includes room for precomputed generation numbers. These\n> - are not currently computed, so all generation numbers will be marked as\n> - 0 (or \"uncomputed\"). A later patch will include this calculation.\n> -\n> - After computing and storing generation numbers, we must make graph\n> walks aware of generation numbers to gain the performance benefits they\n> enable. This will mostly be accomplished by swapping a commit-date-ordered\n> priority queue with one ordered by generation number. The following\n> - operations are important candidates:\n> + operation is an important candidate:\n>\n> - - paint_down_to_common()\n> - 'log --topo-order'\n>\n> - Currently, parse_commit_gently() requires filling in the root tree\n\nLooks good.\n"},{"id":"345014","messageId":"86r2ncgrcs.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-6-dstolee@microsoft.com","subject":"Re: [PATCH v3 5/9] ref-filter: use generation number for --contains","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-18T21:02:59Z","receivedAt":"2018-04-18T21:03:07Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Here I can offer only the cursory examination, as I don't know this area\nof code in question.\n\nDerrick Stolee writes:\n\n> A commit A can reach a commit B only if the generation number of A\n> is larger than the generation number of B. This condition allows\n> significantly short-circuiting commit-graph walks.\n>\n> Use generation number for 'git tag --contains' queries.\n>\n> On a copy of the Linux repository where HEAD is containd in v4.13\n> but no earlier tag, the command 'git tag --contains HEAD' had the\n> following peformance improvement:\n>\n> Before: 0.81s\n> After: 0.04s\n> Rel %: -95%\n\nA question: what is the performance after if the \"commit-graph\" feature\nis disabled, or there is no commit-graph file? Is there performance\nregression in this case, or is the difference negligible?\n\n>\n> Helped-by: Jeff King \n> Signed-off-by: Derrick Stolee \n> ---\n> ref-filter.c | 23 +++++++++++++++++++----\n> 1 file changed, 19 insertions(+), 4 deletions(-)\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index cffd8bf3ce..e2fea6d635 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -1587,7 +1587,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n> /*\n> * Test whether the candidate or one of its parents is contained in the list.\n ^^^^^^^^^^^^^^^^^^^^^\n\nSidenote: when examining the code after the change, I have noticed that\nthe above part of commit header for the comtains_test() function is no\nlonger entirely correct, as the function only checks the candidate\ncommit, and in no place it access its parents.\n\nBut that is not your problem.\n\n> * Do not recurse to find out, though, but return -1 if inconclusive.\n> */\n> static enum contains_result contains_test(struct commit *candidate,\n> \t\t\t\t\t const struct commit_list *want,\n> -\t\t\t\t\t struct contains_cache *cache)\n> +\t\t\t\t\t struct contains_cache *cache,\n> +\t\t\t\t\t uint32_t cutoff)\n> {\n> \tenum contains_result *cached = contains_cache_at(cache, candidate);\n> \n> @@ -1603,6 +1604,10 @@ static enum contains_result contains_test(struct commit *candidate,\n> \n> \t/* Otherwise, we don't know; prepare to recurse */\n> \tparse_commit_or_die(candidate);\n> +\n> +\tif (candidate->generation < cutoff)\n> +\t\treturn CONTAINS_NO;\n> +\n\nLooks good to me.\n\nThe only [minor] question may be whether to define separate type for\ngeneration numbers, and whether to future proof the tests - though the\nlatter would be almost certainly overengineering, and the former\nprobablt too.\n\n> \treturn CONTAINS_UNKNOWN;\n> }\n> \n> @@ -1618,8 +1623,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> \t\t\t\t\t struct contains_cache *cache)\n> {\n> \tstruct contains_stack contains_stack = { 0, 0, NULL };\n> -\tenum contains_result result = contains_test(candidate, want, cache);\n> +\tenum contains_result result;\n> +\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n> +\tconst struct commit_list *p;\n> +\n> +\tfor (p = want; p; p = p->next) {\n> +\t\tstruct commit *c = p->item;\n> +\t\tparse_commit_or_die(c);\n> +\t\tif (c->generation < cutoff)\n> +\t\t\tcutoff = c->generation;\n> +\t}\n\nSholdn't the above be made conditional on the ability to get generation\nnumbers from the commit-graph file (feature is turned on and file\nexists)? Otherwise here after the change contains_tag_algo() now parses\neach commit in 'want', which I think was not done previously.\n\nWith commit-graph file parsing is [probably] cheap. Without it, not\nnecessary.\n\nBut I might be worrying about nothing.\n\n> \n> +\tresult = contains_test(candidate, want, cache, cutoff);\n\nOther than the question about possible performace regression if\ncommit-graph data is not available, it looks good to me.\n\n> \tif (result != CONTAINS_UNKNOWN)\n> \t\treturn result;\n> \n> @@ -1637,7 +1652,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> \t\t * If we just popped the stack, parents->item has been marked,\n> \t\t * therefore contains_test will return a meaningful yes/no.\n> \t\t */\n> -\t\telse switch (contains_test(parents->item, want, cache)) {\n> +\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n> \t\tcase CONTAINS_YES:\n> \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n> \t\t\tcontains_stack.nr--;\n> @@ -1651,7 +1666,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> \t\t}\n> \t}\n> \tfree(contains_stack.contains_stack);\n> -\treturn contains_test(candidate, want, cache);\n> +\treturn contains_test(candidate, want, cache, cutoff);\n\nSimple change. It looks good to me.\n\n> }\n> \n> static int commit_contains(struct ref_filter *filter, struct commit *commit,\n"},{"id":"345030","messageId":"86k1t4go0l.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-7-dstolee@microsoft.com","subject":"Re: [PATCH v3 6/9] commit: use generation numbers for in_merge_bases()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-18T22:15:06Z","receivedAt":"2018-04-18T22:15:15Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The containment algorithm for 'git branch --contains' is different\n> from that for 'git tag --contains' in that it uses is_descendant_of()\n> instead of contains_tag_algo(). The expensive portion of the branch\n> algorithm is computing merge bases.\n>\n> When a commit-graph file exists with generation numbers computed,\n> we can avoid this merge-base calculation when the target commit has\n> a larger generation number than the target commits.\n\nYou have \"target\" twice in above paragraph; one of those should probably\nbe something else.\n\n>\n> Performance tests were run on a copy of the Linux repository where\n> HEAD is contained in v4.13 but no earlier tag. Also, all tags were\n> copied to branches and 'git branch --contains' was tested:\n>\n> Before: 60.0s\n> After: 0.4s\n> Rel %: -99.3%\n\nNice...\n\n>\n> Reported-by: Jeff King \n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 9 ++++++++-\n> 1 file changed, 8 insertions(+), 1 deletion(-)\n\n...especially for so small changes.\n\n>\n> diff --git a/commit.c b/commit.c\n> index a44899c733..bceb79c419 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -1053,12 +1053,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n> {\n> \tstruct commit_list *bases;\n> \tint ret = 0, i;\n> +\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n> \n> \tif (parse_commit(commit))\n> \t\treturn ret;\n> -\tfor (i = 0; i < nr_reference; i++)\n> +\tfor (i = 0; i < nr_reference; i++) {\n> \t\tif (parse_commit(reference[i]))\n> \t\t\treturn ret;\n> +\t\tif (min_generation > reference[i]->generation)\n> +\t\t\tmin_generation = reference[i]->generation;\n> +\t}\n> +\n> +\tif (commit->generation > min_generation)\n> +\t\treturn 0;\n\nWhy not use \"return ret;\" instead of \"return 0;\", like the rest of the\ncode [cryptically] does, that is:\n\n +\tif (commit->generation > min_generation)\n +\t\treturn ret;\n\n> \n> \tbases = paint_down_to_common(commit, nr_reference, reference);\n> \tif (commit->object.flags & PARENT2)\n\nOtherwise, it looks good to me.\n"},{"id":"345041","messageId":"86bmeggl1m.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-8-dstolee@microsoft.com","subject":"Re: [PATCH v3 7/9] commit: add short-circuit to paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-18T23:19:17Z","receivedAt":"2018-04-18T23:19:25Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> When running 'git branch --contains', the in_merge_bases_many()\n> method calls paint_down_to_common() to discover if a specific\n> commit is reachable from a set of branches. Commits with lower\n> generation number are not needed to correctly answer the\n> containment query of in_merge_bases_many().\n\nRight. This description is not entirely clear to me, but I don't have a\nbetter proposal. Good enough, I guess.\n\n>\n> Add a new parameter, min_generation, to paint_down_to_common() that\n> prevents walking commits with generation number strictly less than\n> min_generation. If 0 is given, then there is no functional change.\n\nIs it new parameter really needed, i.e. do you really need to change the\nsignature of this function? See below for details.\n\n>\n> For in_merge_bases_many(), we can pass commit->generation as the\n> cutoff,...\n\nThis is the only callsite that uses min_generation with non-zero value,\nand it uses commit->generation to fill it... while commit itself is one\nof exiting parameters.\n\n> [...], and this saves time during 'git branch --contains' queries\n> that would otherwise walk \"around\" the commit we are inspecting.\n\nIf I understand the code properly, what happens is that we can now\nshort-circuit if all commits that are left are lower than the target\ncommit.\n\nThis is because max-order priority queue is used: if the commit with\nmaximum generation number is below generation number of target commit,\nthen target commit is not reachable from any commit in the priority\nqueue (all of which has generation number less or equal than the commit\nat head of queue, i.e. all are same level or deeper); compare what I\nhave written in [1]\n\n[1]: https://public-inbox.org/git/866052dkju.fsf@gmail.com/\n\nDo I have that right? If so, it looks all right to me.\n\n>\n> For a copy of the Linux repository, where HEAD is checked out at\n> v4.13~100, we get the following performance improvement for\n> 'git branch --contains' over the previous commit:\n>\n> Before: 0.21s\n> After: 0.13s\n> Rel %: -38%\n\nNice.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 18 ++++++++++++++----\n> 1 file changed, 14 insertions(+), 4 deletions(-)\n>\n> diff --git a/commit.c b/commit.c\n> index bceb79c419..a70f120878 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -805,11 +805,14 @@ static int queue_has_nonstale(struct prio_queue *queue)\n> }\n> \n> /* all input commits in one and twos[] must have been parsed! */\n> -static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n> +static struct commit_list *paint_down_to_common(struct commit *one, int n,\n> +\t\t\t\t\t\tstruct commit **twos,\n> +\t\t\t\t\t\tint min_generation)\n> {\n> \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n> \tstruct commit_list *result = NULL;\n> \tint i;\n> +\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n\nDo we really need to change the signature of paint_down_to_common(), or\nwould it be enough to create a local variable min_generation set\ninitially to one->generation.\n\n + uint32_t min_generation = one->generation;\n +\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n\n> \n> \tone->object.flags |= PARENT1;\n> \tif (!n) {\n> @@ -828,6 +831,13 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n> \t\tstruct commit_list *parents;\n> \t\tint flags;\n> \n> +\t\tif (commit->generation > last_gen)\n> +\t\t\tBUG(\"bad generation skip\");\n> +\t\tlast_gen = commit->generation;\n> +\n> +\t\tif (commit->generation < min_generation)\n> +\t\t\tbreak;\n> +\n\nI think, after looking at the whole post-image code, that it is all\nright.\n\n> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n> \t\tif (flags == (PARENT1 | PARENT2)) {\n> \t\t\tif (!(commit->object.flags & RESULT)) {\n> @@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n> \t\t\treturn NULL;\n> \t}\n> \n> -\tlist = paint_down_to_common(one, n, twos);\n> +\tlist = paint_down_to_common(one, n, twos, 0);\n> \n> \twhile (list) {\n> \t\tstruct commit *commit = pop_commit(&list);\n> @@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\t\tfilled_index[filled] = j;\n> \t\t\twork[filled++] = array[j];\n> \t\t}\n> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n> \t\tif (array[i]->object.flags & PARENT2)\n> \t\t\tredundant[i] = 1;\n> \t\tfor (j = 0; j < filled; j++)\n> @@ -1067,7 +1077,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n> \tif (commit->generation > min_generation)\n> \t\treturn 0;\n> \n> -\tbases = paint_down_to_common(commit, nr_reference, reference);\n> +\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n\nIs it the only case where we would call paint_down_to_common() with\nnon-zero last parameter? Would we always use commit->generation where\ncommit is the first parameter of paint_down_to_common()?\n\nIf both are true and will remain true, then in my humble opinion it is\nnot necessary to change the signature of this function.\n\n> \tif (commit->object.flags & PARENT2)\n> \t\tret = 1;\n> \tclear_commit_marks(commit, all_flags);\n"},{"id":"345044","messageId":"8636zsgj1g.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-9-dstolee@microsoft.com","subject":"Re: [PATCH v3 8/9] commit-graph: always load commit-graph information","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-19T00:02:35Z","receivedAt":"2018-04-19T00:59:25Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Most code paths load commits using lookup_commit() and then\n> parse_commit(). In some cases, including some branch lookups, the commit\n> is parsed using parse_object_buffer() which side-steps parse_commit() in\n> favor of parse_commit_buffer().\n>\n> With generation numbers in the commit-graph, we need to ensure that any\n> commit that exists in the commit-graph file has its generation number\n> loaded.\n\nAll right, that is nice explanation of the why behind this change.\n\n>\n> Create new load_commit_graph_info() method to fill in the information\n> for a commit that exists only in the commit-graph file. Call it from\n> parse_commit_buffer() after loading the other commit information from\n> the given buffer. Only fill this information when specified by the\n> 'check_graph' parameter. This avoids duplicate work when we already\n> checked the graph in parse_commit_gently() or when simply checking the\n> buffer contents in check_commit().\n\nCouldn't this 'check_graph' parameter be a global variable similar to\nthe 'commit_graph' variable? Maybe I am not understanding it.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 51 ++++++++++++++++++++++++++++++++------------------\n> commit-graph.h | 8 ++++++++\n> commit.c | 7 +++++--\n> commit.h | 2 +-\n> object.c | 2 +-\n> sha1_file.c | 2 +-\n> 6 files changed, 49 insertions(+), 23 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 688d5b1801..21e853c21a 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -245,13 +245,19 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n> \treturn &commit_list_insert(c, pptr)->next;\n> }\n> \n> +static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n> +{\n> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n> +}\n> +\n> static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n> {\n> \tuint32_t edge_value;\n> \tuint32_t *parent_data_ptr;\n> \tuint64_t date_low, date_high;\n> \tstruct commit_list **pptr;\n> -\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n\nI'm probably wrong, but isn't it unrelated change?\n\n> \n> \titem->object.parsed = 1;\n> \titem->graph_pos = pos;\n> @@ -292,31 +298,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \treturn 1;\n> }\n> \n> +static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n> +{\n> +\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n> +\t\t*pos = item->graph_pos;\n> +\t\treturn 1;\n> +\t} else {\n> +\t\treturn bsearch_graph(commit_graph, &(item->object.oid), pos);\n> +\t}\n> +}\n\nAll right (after the fix).\n\n> +\n> int parse_commit_in_graph(struct commit *item)\n> {\n> +\tuint32_t pos;\n> +\n> +\tif (item->object.parsed)\n> +\t\treturn 0;\n> \tif (!core_commit_graph)\n> \t\treturn 0;\n> -\tif (item->object.parsed)\n> -\t\treturn 1;\n\nHmmm... previously the function returned 1 if item->object.parsed, now\nit returns 0 for this situation. I don't understand this change.\n\n> -\n> \tprepare_commit_graph();\n> -\tif (commit_graph) {\n> -\t\tuint32_t pos;\n> -\t\tint found;\n> -\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n> -\t\t\tpos = item->graph_pos;\n> -\t\t\tfound = 1;\n> -\t\t} else {\n> -\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n> -\t\t}\n> -\n> -\t\tif (found)\n> -\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> -\t}\n> -\n> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> +\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n\nNice refactoring.\n\n> \treturn 0;\n> }\n> \n> +void load_commit_graph_info(struct commit *item)\n> +{\n> +\tuint32_t pos;\n> +\tif (!core_commit_graph)\n> +\t\treturn;\n> +\tprepare_commit_graph();\n> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> +\t\tfill_commit_graph_info(item, commit_graph, pos);\n> +}\n\nAnd the reason for the refactoring.\n\n> +\n> static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n> {\n> \tstruct object_id oid;\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 260a468e73..96cccb10f3 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n> */\n> int parse_commit_in_graph(struct commit *item);\n> \n> +/*\n> + * It is possible that we loaded commit contents from the commit buffer,\n> + * but we also want to ensure the commit-graph content is correctly\n> + * checked and filled. Fill the graph_pos and generation members of\n> + * the given commit.\n> + */\n> +void load_commit_graph_info(struct commit *item);\n> +\n> struct tree *get_commit_tree_in_graph(const struct commit *c);\n> \n> struct commit_graph {\n> diff --git a/commit.c b/commit.c\n> index a70f120878..9ef6f699bd 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n> \treturn ret;\n> }\n> \n> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n> {\n> \tconst char *tail = buffer;\n> \tconst char *bufptr = buffer;\n> @@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n> \t}\n> \titem->date = parse_commit_date(bufptr, tail);\n> \n> +\tif (check_graph)\n> +\t\tload_commit_graph_info(item);\n> +\n> \treturn 0;\n> }\n> \n> @@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \t\treturn error(\"Object %s not a commit\",\n> \t\t\t oid_to_hex(&item->object.oid));\n> \t}\n> -\tret = parse_commit_buffer(item, buffer, size);\n> +\tret = parse_commit_buffer(item, buffer, size, 0);\n> \tif (save_commit_buffer && !ret) {\n> \t\tset_commit_buffer(item, buffer, size);\n> \t\treturn 0;\n> diff --git a/commit.h b/commit.h\n> index 64436ff44e..b5afde1ae9 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n> */\n> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n> \n> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n> int parse_commit_gently(struct commit *item, int quiet_on_missing);\n> static inline int parse_commit(struct commit *item)\n> {\n> diff --git a/object.c b/object.c\n> index e6ad3f61f0..efe4871325 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n> \t} else if (type == OBJ_COMMIT) {\n> \t\tstruct commit *commit = lookup_commit(oid);\n> \t\tif (commit) {\n> -\t\t\tif (parse_commit_buffer(commit, buffer, size))\n> +\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n> \t\t\t\treturn NULL;\n> \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n> \t\t\t\tset_commit_buffer(commit, buffer, size);\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 1b94f39c4c..0fd4f0b8b6 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n> {\n> \tstruct commit c;\n> \tmemset(&c, 0, sizeof(c));\n> -\tif (parse_commit_buffer(&c, buf, size))\n> +\tif (parse_commit_buffer(&c, buf, size, 0))\n> \t\tdie(\"corrupt commit\");\n> }\n"},{"id":"345045","messageId":"86y3hkf4d9.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"Re: [PATCH v3 0/9] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-19T00:04:50Z","receivedAt":"2018-04-19T00:59:26Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> -- >8 --\n>\n> This is the one of several \"small\" patches that follow the serialized\n> Git commit graph patch (ds/commit-graph) and lazy-loading trees\n> (ds/lazy-load-trees).\n>\n> As described in Documentation/technical/commit-graph.txt, the generation\n> number of a commit is one more than the maximum generation number among\n> its parents (trivially, a commit with no parents has generation number\n> one). This section is expanded to describe the interaction with special\n> generation numbers GENERATION_NUMBER_INFINITY (commits not in the commit-graph\n> file) and *_ZERO (commits in a commit-graph file written before generation\n> numbers were implemented).\n>\n> This series makes the computation of generation numbers part of the\n> commit-graph write process.\n>\n> Finally, generation numbers are used to order commits in the priority\n> queue in paint_down_to_common(). This allows a short-circuit mechanism\n> to improve performance of `git branch --contains`.\n>\n> Further, use generation numbers for 'git tag --contains', providing a\n> significant speedup (at least 95% for some cases).\n>\n> A more substantial refactoring of revision.c is required before making\n> 'git log --graph' use generation numbers effectively.\n>\n> This patch series is build on ds/lazy-load-trees.\n>\n> Derrick Stolee (9):\n> commit: add generation number to struct commmit\n\nNice and short patch. Looks good to me.\n\n> commit-graph: compute generation numbers\n\nAnother quite easy to understand patch. LGTM.\n\n> commit: use generations in paint_down_to_common()\n\nNice and short patch; minor typo in comment in code.\nOtherwise it looks good to me.\n\n> commit-graph.txt: update design document\n\nI see that diagram got removed in this version; maybe it could be\nreplaced with relationship table?\n\nAnyway, it looks good to me.\n\n> ref-filter: use generation number for --contains\n\nA question: how performance looks like after the change if commit-graph\nis not available?\n\n> commit: use generation numbers for in_merge_bases()\n\nPossible typo in the commit message, and stylistic inconsistence in\nin_merge_bases() - though actually more clear than existing code.\n\nShort, simple, and gives good performance improvenebts.\n\n> commit: add short-circuit to paint_down_to_common()\n\nLooks good to me; ignore [mostly] what I have written in response to the\npatch in question.\n\n> commit-graph: always load commit-graph information\n\nLooks all right; question: parameter or one more global variable.\n\n> merge: check config before loading commits\n\nThis looks good to me.\n\n>\n> Documentation/technical/commit-graph.txt | 30 +++++--\n> alloc.c | 1 +\n> builtin/merge.c | 5 +-\n> commit-graph.c | 99 +++++++++++++++++++-----\n> commit-graph.h | 8 ++\n> commit.c | 54 +++++++++++--\n> commit.h | 7 +-\n> object.c | 2 +-\n> ref-filter.c | 23 +++++-\n> sha1_file.c | 2 +-\n> t/t5318-commit-graph.sh | 9 +++\n> 11 files changed, 199 insertions(+), 41 deletions(-)\n>\n>\n> base-commit: 7b8a21dba1bce44d64bd86427d3d92437adc4707\n"},{"id":"345080","messageId":"86sh7rfvga.fsf@gmail.com","threadId":"48212","inReplyTo":"20180417170001.138464-8-dstolee@microsoft.com","subject":"Re: [PATCH v3 7/9] commit: add short-circuit to paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-19T08:32:05Z","receivedAt":"2018-04-19T08:32:14Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> @@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n> \t\t\treturn NULL;\n> \t}\n> \n> -\tlist = paint_down_to_common(one, n, twos);\n> +\tlist = paint_down_to_common(one, n, twos, 0);\n> \n> \twhile (list) {\n> \t\tstruct commit *commit = pop_commit(&list);\n> @@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\t\tfilled_index[filled] = j;\n> \t\t\twork[filled++] = array[j];\n> \t\t}\n> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n> \t\tif (array[i]->object.flags & PARENT2)\n> \t\t\tredundant[i] = 1;\n> \t\tfor (j = 0; j < filled; j++)\n\nWouldn't it be better and more readable to create a symbolic name for\nthis 0, for example:\n\n -\tlist = paint_down_to_common(one, n, twos);\n +\tlist = paint_down_to_common(one, n, twos, GENERATION_NO_CUTOFF);\n\nBest,\n-- \nJakub Narębski\n"},{"id":"345379","messageId":"86d0yscmra.fsf@gmail.com","threadId":"48212","inReplyTo":"86lgdplogn.fsf@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-21T20:44:57Z","receivedAt":"2018-04-21T20:45:04Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski writes:\n> Derrick Stolee writes:\n>> On 4/11/2018 3:32 PM, Jakub Narebski wrote:\n>\n>>> What would you suggest as a good test that could imply performance? The\n>>> Google Colab notebook linked to above includes a function to count\n>>> number of commits (nodes / vertices in the commit graph) walked,\n>>> currently in the worst case scenario.\n>>\n>> The two main questions to consider are:\n>>\n>> 1. Can X reach Y?\n>\n> That is easy to do. The function generic_is_reachable() does\n> that... though using direct translation of the pseudocode for\n> \"Algorithm 3: Reachable\" from FELINE paper, which is recursive and\n> doesn't check if vertex was already visited was not good idea for large\n> graphs such as Linux kernel commit graph, oops. That is why\n> generic_is_reachable_large() was created.\n[...]\n\n>> And the thing to measure is a commit count. If possible, it would be\n>> good to count commits walked (commits whose parent list is enumerated)\n>> and commits inspected (commits that were listed as a parent of some\n>> walked commit). Walked commits require a commit parse -- albeit from\n>> the commit-graph instead of the ODB now -- while inspected commits\n>> only check the in-memory cache.\n[...]\n>>\n>> For git.git and Linux, I like to use the release tags as tests. They\n>> provide a realistic view of the linear history, and maintenance\n>> releases have their own history from the major releases.\n>\n> Hmmm... testing for v4.9-rc5..v4.9 in Linux kernel commit graphs, the\n> FELINE index does not bring any improvements over using just level\n> (generation number) filter. But that may be caused by narrowing od\n> commit DAG around releases.\n>\n> I try do do the same between commits in wide part, with many commits\n> with the same level (same generation number) both for source and for\n> target commit. Though this may be unfair to level filter, though...\n>\n>\n> Note however that FELINE index is not unabiguous, like generation\n> numbers are (modulo decision whether to start at 0 or at 1); it depends\n> on the topological ordering chosen for the X elements.\n\nOne can now test reachability on git.git repository; there is a form\nwhere one can plug source and destination revisions at\nhttps://colab.research.google.com/drive/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg#scrollTo=svNUnSA9O_NK&line=2&uniqifier=1\n\nI have tried the case that is quite unfair to the generation numbers\nfilter, namely the check between one of recent tags, and one commit that\nshares generation number among largest number of other commits.\n\nHere level = generation number-1 (as it starts at 0 for root commit, not\n1).\n\nThe results are:\n * src = 468165c1d = v2.17.0\n * dst = 66d2e04ec = v2.0.5-5-g66d2e04ec\n\n * 468165c1d has level 18418 which it shares with 6 commits\n * 66d2e04ec has level 14776 which it shares with 93 commits\n * gen(468165c1d) - gen(66d2e04ec) = 3642\n\n algorithm | access | walk | maxdepth | visited | level-f | FELINE-f |\n -----------+---------+--------+----------+---------+----------+-----------+\n naive | 48865 | 39599 | 244 | 9200 | | |\n level | 3086 | 2492 | 113 | 528 | 285 | |\n FELINE | 283 | 216 | 68 | 0 | | 25 |\n lev+FELINE | 282 | 215 | 68 | 0 | 5 | 24 |\n -----------+---------+--------+----------+---------+----------+-----------+\n lev+FEL+mpi| 79 | 59 | 21 | 0 | 0 | 0 |\n\nHere we have:\n* 'naive' implementation means simple DFS walk, without any filters (cut-offs)\n* 'level' means using levels / generation numbers based negative-cut filter\n* 'FELINE' means using FELINE index based negative-cut filter\n* 'lev+FELINE' means combining generation numbers filter with FELINE filter\n* 'mpi' means min-post [smanning-tree] intervals for positive-cut filter;\n note that the code does not walk the path after cut, but it is easy to do\n\nThe stats have the following meaning:\n* 'access' means accessing the node\n* 'walk' is actual walking the node\n* 'maxdepth' is maximum depth of the stack used for DFS\n* 'level-f' and 'FELINE-f' is number of times levels filter or FELINE filter\n were used for negative-cut; note that those are not disjoint; node can\n be rejected by both level filter and FELINE filter\n\nFor v2.17.0 and v2.17.0-rc2 the numbers are much less in FELINE favor:\nthe results are the same, with 5 commits accessed and 6 walked compared\nto 61574 accessed in naive algorithm.\n\nThe git.git commit graph has 53128 nodes and 66124 edges, 4 tips / heads\n(different child-less commits) and 9 roots, and has average clustering\ncoefficient 0.000409217.\n\nP.S. Would it be better to move the discussion about possible extensions\nto the commit-graph in the form of new chunks (topological order, FELINE\nindex, min-post intervals, bloom filter for changed files, etc.) be\nmoved into separate thread?\n-- \nJakub Narębski\n"},{"id":"345463","messageId":"68cadcfc-5fed-9c2c-8f5c-034ec64a347e@gmail.com","threadId":"48212","inReplyTo":"86d0yscmra.fsf@gmail.com","subject":"Re: [PATCH 0/6] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-23T13:54:56Z","receivedAt":"2018-04-23T13:55:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/21/2018 4:44 PM, Jakub Narebski wrote:\n> Jakub Narebski writes:\n>> Derrick Stolee writes:\n>>> On 4/11/2018 3:32 PM, Jakub Narebski wrote:\n>>>> What would you suggest as a good test that could imply performance? The\n>>>> Google Colab notebook linked to above includes a function to count\n>>>> number of commits (nodes / vertices in the commit graph) walked,\n>>>> currently in the worst case scenario.\n>>> The two main questions to consider are:\n>>>\n>>> 1. Can X reach Y?\n>> That is easy to do. The function generic_is_reachable() does\n>> that... though using direct translation of the pseudocode for\n>> \"Algorithm 3: Reachable\" from FELINE paper, which is recursive and\n>> doesn't check if vertex was already visited was not good idea for large\n>> graphs such as Linux kernel commit graph, oops. That is why\n>> generic_is_reachable_large() was created.\n> [...]\n>\n>>> And the thing to measure is a commit count. If possible, it would be\n>>> good to count commits walked (commits whose parent list is enumerated)\n>>> and commits inspected (commits that were listed as a parent of some\n>>> walked commit). Walked commits require a commit parse -- albeit from\n>>> the commit-graph instead of the ODB now -- while inspected commits\n>>> only check the in-memory cache.\n> [...]\n>>> For git.git and Linux, I like to use the release tags as tests. They\n>>> provide a realistic view of the linear history, and maintenance\n>>> releases have their own history from the major releases.\n>> Hmmm... testing for v4.9-rc5..v4.9 in Linux kernel commit graphs, the\n>> FELINE index does not bring any improvements over using just level\n>> (generation number) filter. But that may be caused by narrowing od\n>> commit DAG around releases.\n>>\n>> I try do do the same between commits in wide part, with many commits\n>> with the same level (same generation number) both for source and for\n>> target commit. Though this may be unfair to level filter, though...\n>>\n>>\n>> Note however that FELINE index is not unabiguous, like generation\n>> numbers are (modulo decision whether to start at 0 or at 1); it depends\n>> on the topological ordering chosen for the X elements.\n> One can now test reachability on git.git repository; there is a form\n> where one can plug source and destination revisions at\n> https://colab.research.google.com/drive/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg#scrollTo=svNUnSA9O_NK&line=2&uniqifier=1\n>\n> I have tried the case that is quite unfair to the generation numbers\n> filter, namely the check between one of recent tags, and one commit that\n> shares generation number among largest number of other commits.\n>\n> Here level = generation number-1 (as it starts at 0 for root commit, not\n> 1).\n>\n> The results are:\n> * src = 468165c1d = v2.17.0\n> * dst = 66d2e04ec = v2.0.5-5-g66d2e04ec\n>\n> * 468165c1d has level 18418 which it shares with 6 commits\n> * 66d2e04ec has level 14776 which it shares with 93 commits\n> * gen(468165c1d) - gen(66d2e04ec) = 3642\n>\n> algorithm | access | walk | maxdepth | visited | level-f | FELINE-f |\n> -----------+---------+--------+----------+---------+----------+-----------+\n> naive | 48865 | 39599 | 244 | 9200 | | |\n> level | 3086 | 2492 | 113 | 528 | 285 | |\n> FELINE | 283 | 216 | 68 | 0 | | 25 |\n> lev+FELINE | 282 | 215 | 68 | 0 | 5 | 24 |\n> -----------+---------+--------+----------+---------+----------+-----------+\n> lev+FEL+mpi| 79 | 59 | 21 | 0 | 0 | 0 |\n>\n> Here we have:\n> * 'naive' implementation means simple DFS walk, without any filters (cut-offs)\n> * 'level' means using levels / generation numbers based negative-cut filter\n> * 'FELINE' means using FELINE index based negative-cut filter\n> * 'lev+FELINE' means combining generation numbers filter with FELINE filter\n> * 'mpi' means min-post [smanning-tree] intervals for positive-cut filter;\n> note that the code does not walk the path after cut, but it is easy to do\n>\n> The stats have the following meaning:\n> * 'access' means accessing the node\n> * 'walk' is actual walking the node\n> * 'maxdepth' is maximum depth of the stack used for DFS\n> * 'level-f' and 'FELINE-f' is number of times levels filter or FELINE filter\n> were used for negative-cut; note that those are not disjoint; node can\n> be rejected by both level filter and FELINE filter\n>\n> For v2.17.0 and v2.17.0-rc2 the numbers are much less in FELINE favor:\n> the results are the same, with 5 commits accessed and 6 walked compared\n> to 61574 accessed in naive algorithm.\n>\n> The git.git commit graph has 53128 nodes and 66124 edges, 4 tips / heads\n> (different child-less commits) and 9 roots, and has average clustering\n> coefficient 0.000409217.\n\nThanks for these results. Now, write a patch. I'm sticking to generation \nnumbers for my patch because of the simplified computation, but you can \ncontribute a FELINE implementation.\n\n> P.S. Would it be better to move the discussion about possible extensions\n> to the commit-graph in the form of new chunks (topological order, FELINE\n> index, min-post intervals, bloom filter for changed files, etc.) be\n> moved into separate thread?\n\nYes. I think we've exhausted this thought experiment and future \ndiscussion should revolve around actual implementations in Git with \nend-to-end performance times. The computation time for computing the \nFELINE index should be included in that discussion.\n\nThanks,\n-Stolee\n"},{"id":"345466","messageId":"48f910b6-8762-f34b-b644-d6c7ff99499f@gmail.com","threadId":"48212","inReplyTo":"86r2ncgrcs.fsf@gmail.com","subject":"Re: [PATCH v3 5/9] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-23T14:22:22Z","receivedAt":"2018-04-23T14:22:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/18/2018 5:02 PM, Jakub Narebski wrote:\n> Here I can offer only the cursory examination, as I don't know this area\n> of code in question.\n>\n> Derrick Stolee writes:\n>\n>> A commit A can reach a commit B only if the generation number of A\n>> is larger than the generation number of B. This condition allows\n>> significantly short-circuiting commit-graph walks.\n>>\n>> Use generation number for 'git tag --contains' queries.\n>>\n>> On a copy of the Linux repository where HEAD is containd in v4.13\n>> but no earlier tag, the command 'git tag --contains HEAD' had the\n>> following peformance improvement:\n>>\n>> Before: 0.81s\n>> After: 0.04s\n>> Rel %: -95%\n> A question: what is the performance after if the \"commit-graph\" feature\n> is disabled, or there is no commit-graph file? Is there performance\n> regression in this case, or is the difference negligible?\n\nNegligible, since we are adding a small number of integer comparisons \nand the main cost is in commit parsing. More on commit parsing in \nresponse to your comments below.\n\n>\n>> Helped-by: Jeff King \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> ref-filter.c | 23 +++++++++++++++++++----\n>> 1 file changed, 19 insertions(+), 4 deletions(-)\n>>\n>> diff --git a/ref-filter.c b/ref-filter.c\n>> index cffd8bf3ce..e2fea6d635 100644\n>> --- a/ref-filter.c\n>> +++ b/ref-filter.c\n>> @@ -1587,7 +1587,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n>> /*\n>> * Test whether the candidate or one of its parents is contained in the list.\n> ^^^^^^^^^^^^^^^^^^^^^\n>\n> Sidenote: when examining the code after the change, I have noticed that\n> the above part of commit header for the comtains_test() function is no\n> longer entirely correct, as the function only checks the candidate\n> commit, and in no place it access its parents.\n>\n> But that is not your problem.\n\nI'll add a commit in the next version that fixes this comment before I \nmake any changes to the method.\n\n>\n>> * Do not recurse to find out, though, but return -1 if inconclusive.\n>> */\n>> static enum contains_result contains_test(struct commit *candidate,\n>> \t\t\t\t\t const struct commit_list *want,\n>> -\t\t\t\t\t struct contains_cache *cache)\n>> +\t\t\t\t\t struct contains_cache *cache,\n>> +\t\t\t\t\t uint32_t cutoff)\n>> {\n>> \tenum contains_result *cached = contains_cache_at(cache, candidate);\n>> \n>> @@ -1603,6 +1604,10 @@ static enum contains_result contains_test(struct commit *candidate,\n>> \n>> \t/* Otherwise, we don't know; prepare to recurse */\n>> \tparse_commit_or_die(candidate);\n>> +\n>> +\tif (candidate->generation < cutoff)\n>> +\t\treturn CONTAINS_NO;\n>> +\n> Looks good to me.\n>\n> The only [minor] question may be whether to define separate type for\n> generation numbers, and whether to future proof the tests - though the\n> latter would be almost certainly overengineering, and the former\n> probablt too.\n\nIf we have multiple notions of generation, then we can refactor all \nreferences to the \"generation\" member.\n\n>\n>> \treturn CONTAINS_UNKNOWN;\n>> }\n>> \n>> @@ -1618,8 +1623,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n>> \t\t\t\t\t struct contains_cache *cache)\n>> {\n>> \tstruct contains_stack contains_stack = { 0, 0, NULL };\n>> -\tenum contains_result result = contains_test(candidate, want, cache);\n>> +\tenum contains_result result;\n>> +\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n>> +\tconst struct commit_list *p;\n>> +\n>> +\tfor (p = want; p; p = p->next) {\n>> +\t\tstruct commit *c = p->item;\n>> +\t\tparse_commit_or_die(c);\n>> +\t\tif (c->generation < cutoff)\n>> +\t\t\tcutoff = c->generation;\n>> +\t}\n> Sholdn't the above be made conditional on the ability to get generation\n> numbers from the commit-graph file (feature is turned on and file\n> exists)? Otherwise here after the change contains_tag_algo() now parses\n> each commit in 'want', which I think was not done previously.\n>\n> With commit-graph file parsing is [probably] cheap. Without it, not\n> necessary.\n>\n> But I might be worrying about nothing.\n\nNot nothing. This parses the \"wants\" when we previously did not parse \nthe wants. Further: this parsing happens before we do the simple check \nof comparing the OID of the candidate against the wants.\n\nThe question is: are these parsed commits significant compared to the \nwalk that will parse many more commits? It is certainly possible.\n\nOne way to fix this is to call 'prepare_commit_graph()' directly and \nthen test that 'commit_graph' is non-null before performing any parses. \nI'm not thrilled with how that couples the commit-graph implementation \nto this feature, but that may be necessary to avoid regressions in the \nnon-commit-graph case.\n\n>\n>> \n>> +\tresult = contains_test(candidate, want, cache, cutoff);\n> Other than the question about possible performace regression if\n> commit-graph data is not available, it looks good to me.\n>\n>> \tif (result != CONTAINS_UNKNOWN)\n>> \t\treturn result;\n>> \n>> @@ -1637,7 +1652,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n>> \t\t * If we just popped the stack, parents->item has been marked,\n>> \t\t * therefore contains_test will return a meaningful yes/no.\n>> \t\t */\n>> -\t\telse switch (contains_test(parents->item, want, cache)) {\n>> +\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n>> \t\tcase CONTAINS_YES:\n>> \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n>> \t\t\tcontains_stack.nr--;\n>> @@ -1651,7 +1666,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n>> \t\t}\n>> \t}\n>> \tfree(contains_stack.contains_stack);\n>> -\treturn contains_test(candidate, want, cache);\n>> +\treturn contains_test(candidate, want, cache, cutoff);\n> Simple change. It looks good to me.\n>\n>> }\n>> \n>> static int commit_contains(struct ref_filter *filter, struct commit *commit,\n\n"},{"id":"345467","messageId":"258ad61b-c262-5144-12e7-5068851e34d4@gmail.com","threadId":"48212","inReplyTo":"86k1t4go0l.fsf@gmail.com","subject":"Re: [PATCH v3 6/9] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-23T14:31:19Z","receivedAt":"2018-04-23T14:31:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/18/2018 6:15 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> The containment algorithm for 'git branch --contains' is different\n>> from that for 'git tag --contains' in that it uses is_descendant_of()\n>> instead of contains_tag_algo(). The expensive portion of the branch\n>> algorithm is computing merge bases.\n>>\n>> When a commit-graph file exists with generation numbers computed,\n>> we can avoid this merge-base calculation when the target commit has\n>> a larger generation number than the target commits.\n> You have \"target\" twice in above paragraph; one of those should probably\n> be something else.\n\nThanks. Second \"target\" should be \"initial\".\n\n> [...]\n>> +\n>> +\tif (commit->generation > min_generation)\n>> +\t\treturn 0;\n> Why not use \"return ret;\" instead of \"return 0;\", like the rest of the\n> code [cryptically] does, that is:\n>\n> +\tif (commit->generation > min_generation)\n> +\t\treturn ret;\n\nSure.\n\nThanks,\n-Stolee\n"},{"id":"345468","messageId":"bd2011f0-41d4-4550-5392-65a3816ed4d3@gmail.com","threadId":"48212","inReplyTo":"86bmeggl1m.fsf@gmail.com","subject":"Re: [PATCH v3 7/9] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-23T14:40:54Z","receivedAt":"2018-04-23T14:41:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/18/2018 7:19 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n[...]\n>> [...], and this saves time during 'git branch --contains' queries\n>> that would otherwise walk \"around\" the commit we are inspecting.\n> If I understand the code properly, what happens is that we can now\n> short-circuit if all commits that are left are lower than the target\n> commit.\n>\n> This is because max-order priority queue is used: if the commit with\n> maximum generation number is below generation number of target commit,\n> then target commit is not reachable from any commit in the priority\n> queue (all of which has generation number less or equal than the commit\n> at head of queue, i.e. all are same level or deeper); compare what I\n> have written in [1]\n>\n> [1]: https://public-inbox.org/git/866052dkju.fsf@gmail.com/\n>\n> Do I have that right? If so, it looks all right to me.\n\nYes, the priority queue needs to compare via generation number first or \nthere will be errors. This is why we could not use commit time before.\n\n>\n>> For a copy of the Linux repository, where HEAD is checked out at\n>> v4.13~100, we get the following performance improvement for\n>> 'git branch --contains' over the previous commit:\n>>\n>> Before: 0.21s\n>> After: 0.13s\n>> Rel %: -38%\n> [...]\n>> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n>> \t\tif (flags == (PARENT1 | PARENT2)) {\n>> \t\t\tif (!(commit->object.flags & RESULT)) {\n>> @@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n>> \t\t\treturn NULL;\n>> \t}\n>> \n>> -\tlist = paint_down_to_common(one, n, twos);\n>> +\tlist = paint_down_to_common(one, n, twos, 0);\n>> \n>> \twhile (list) {\n>> \t\tstruct commit *commit = pop_commit(&list);\n>> @@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n>> \t\t\tfilled_index[filled] = j;\n>> \t\t\twork[filled++] = array[j];\n>> \t\t}\n>> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n>> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n>> \t\tif (array[i]->object.flags & PARENT2)\n>> \t\t\tredundant[i] = 1;\n>> \t\tfor (j = 0; j < filled; j++)\n>> @@ -1067,7 +1077,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n>> \tif (commit->generation > min_generation)\n>> \t\treturn 0;\n>> \n>> -\tbases = paint_down_to_common(commit, nr_reference, reference);\n>> +\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n> Is it the only case where we would call paint_down_to_common() with\n> non-zero last parameter? Would we always use commit->generation where\n> commit is the first parameter of paint_down_to_common()?\n>\n> If both are true and will remain true, then in my humble opinion it is\n> not necessary to change the signature of this function.\n\nWe need to change the signature some way, but maybe the way I chose is \nnot the best.\n\nTo elaborate: paint_down_to_common() is used for multiple purposes. The \ncaller here that supplies 'commit->generation' is used only to compute \nreachability (by testing if the flag PARENT2 exists on the commit, then \nclears all flags). The other callers expect the full walk down to the \ncommon commits, and keeps those PARENT1, PARENT2, and STALE flags for \nfuture use (such as reporting merge bases). Usually the call to \npaint_down_to_common() is followed by a revision walk that only halts \nwhen reaching root commits or commits with both PARENT1 and PARENT2 \nflags on, so always short-circuiting on generations would break the \nfunctionality; this is confirmed by the t5318-commit-graph.sh.\n\nAn alternative to the signature change is to add a boolean parameter \n\"use_cutoff\" or something, that specifies \"don't walk beyond the \ncommit\". This may give a more of a clear description of what it will do \nwith the generation value, but since we are already performing \ngeneration comparisons before calling paint_down_to_common() I find this \nsimple enough.\n\nThanks,\n-Stolee\n\n"},{"id":"345470","messageId":"88b0fb38-57d5-f654-27c8-dd1807436d93@gmail.com","threadId":"48212","inReplyTo":"8636zsgj1g.fsf@gmail.com","subject":"Re: [PATCH v3 8/9] commit-graph: always load commit-graph information","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-23T14:49:48Z","receivedAt":"2018-04-23T14:49:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/18/2018 8:02 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> Most code paths load commits using lookup_commit() and then\n>> parse_commit(). In some cases, including some branch lookups, the commit\n>> is parsed using parse_object_buffer() which side-steps parse_commit() in\n>> favor of parse_commit_buffer().\n>>\n>> With generation numbers in the commit-graph, we need to ensure that any\n>> commit that exists in the commit-graph file has its generation number\n>> loaded.\n> All right, that is nice explanation of the why behind this change.\n>\n>> Create new load_commit_graph_info() method to fill in the information\n>> for a commit that exists only in the commit-graph file. Call it from\n>> parse_commit_buffer() after loading the other commit information from\n>> the given buffer. Only fill this information when specified by the\n>> 'check_graph' parameter. This avoids duplicate work when we already\n>> checked the graph in parse_commit_gently() or when simply checking the\n>> buffer contents in check_commit().\n> Couldn't this 'check_graph' parameter be a global variable similar to\n> the 'commit_graph' variable? Maybe I am not understanding it.\n\nSee the two callers at the bottom of the patch. They have different \npurposes: one needs to fill in a valid commit struct, the other needs to \ncheck the commit buffer is valid (then throws away the struct). They \nhave different values for 'check_graph'. Also, in parse_commit_gently() \nwe check parse_commit_in_graph() before we call parse_commit_buffer, so \nwe do not want to repeat work; in the case of a valid commit-graph file, \nbut the commit is not in the commit-graph, we would repeat our binary \nsearch for the same commit.\n\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 51 ++++++++++++++++++++++++++++++++------------------\n>> commit-graph.h | 8 ++++++++\n>> commit.c | 7 +++++--\n>> commit.h | 2 +-\n>> object.c | 2 +-\n>> sha1_file.c | 2 +-\n>> 6 files changed, 49 insertions(+), 23 deletions(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 688d5b1801..21e853c21a 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -245,13 +245,19 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n>> \treturn &commit_list_insert(c, pptr)->next;\n>> }\n>> \n>> +static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n>> +{\n>> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n>> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n>> +}\n>> +\n>> static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n>> {\n>> \tuint32_t edge_value;\n>> \tuint32_t *parent_data_ptr;\n>> \tuint64_t date_low, date_high;\n>> \tstruct commit_list **pptr;\n>> -\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n>> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n> I'm probably wrong, but isn't it unrelated change?\n\nYou're right. I saw this while I was in here, and there was a similar \ncomment on this change in a different patch. Probably best to keep these \ncleanup things in a separate commit.\n\n>> \titem->object.parsed = 1;\n>> \titem->graph_pos = pos;\n>> @@ -292,31 +298,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n>> \treturn 1;\n>> }\n>> \n>> +static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n>> +{\n>> +\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n>> +\t\t*pos = item->graph_pos;\n>> +\t\treturn 1;\n>> +\t} else {\n>> +\t\treturn bsearch_graph(commit_graph, &(item->object.oid), pos);\n>> +\t}\n>> +}\n> All right (after the fix).\n>\n>> +\n>> int parse_commit_in_graph(struct commit *item)\n>> {\n>> +\tuint32_t pos;\n>> +\n>> +\tif (item->object.parsed)\n>> +\t\treturn 0;\n>> \tif (!core_commit_graph)\n>> \t\treturn 0;\n>> -\tif (item->object.parsed)\n>> -\t\treturn 1;\n> Hmmm... previously the function returned 1 if item->object.parsed, now\n> it returns 0 for this situation. I don't understand this change.\n\nThe good news is that this change is unimportant (the only caller is \nparse_commit_gently() which checks item->object.parsed before calling \nparse_commit_in_graph()). I wonder why I reordered those things, anyway. \nI'll revert to simplify the patch.\n\n>\n>> -\n>> \tprepare_commit_graph();\n>> -\tif (commit_graph) {\n>> -\t\tuint32_t pos;\n>> -\t\tint found;\n>> -\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n>> -\t\t\tpos = item->graph_pos;\n>> -\t\t\tfound = 1;\n>> -\t\t} else {\n>> -\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n>> -\t\t}\n>> -\n>> -\t\tif (found)\n>> -\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n>> -\t}\n>> -\n>> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n>> +\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> Nice refactoring.\n>\n>> \treturn 0;\n>> }\n>> \n>> +void load_commit_graph_info(struct commit *item)\n>> +{\n>> +\tuint32_t pos;\n>> +\tif (!core_commit_graph)\n>> +\t\treturn;\n>> +\tprepare_commit_graph();\n>> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n>> +\t\tfill_commit_graph_info(item, commit_graph, pos);\n>> +}\n> And the reason for the refactoring.\n>\n>> +\n>> static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n>> {\n>> \tstruct object_id oid;\n>> diff --git a/commit-graph.h b/commit-graph.h\n>> index 260a468e73..96cccb10f3 100644\n>> --- a/commit-graph.h\n>> +++ b/commit-graph.h\n>> @@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n>> */\n>> int parse_commit_in_graph(struct commit *item);\n>> \n>> +/*\n>> + * It is possible that we loaded commit contents from the commit buffer,\n>> + * but we also want to ensure the commit-graph content is correctly\n>> + * checked and filled. Fill the graph_pos and generation members of\n>> + * the given commit.\n>> + */\n>> +void load_commit_graph_info(struct commit *item);\n>> +\n>> struct tree *get_commit_tree_in_graph(const struct commit *c);\n>> \n>> struct commit_graph {\n>> diff --git a/commit.c b/commit.c\n>> index a70f120878..9ef6f699bd 100644\n>> --- a/commit.c\n>> +++ b/commit.c\n>> @@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n>> \treturn ret;\n>> }\n>> \n>> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n>> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n>> {\n>> \tconst char *tail = buffer;\n>> \tconst char *bufptr = buffer;\n>> @@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n>> \t}\n>> \titem->date = parse_commit_date(bufptr, tail);\n>> \n>> +\tif (check_graph)\n>> +\t\tload_commit_graph_info(item);\n>> +\n>> \treturn 0;\n>> }\n>> \n>> @@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n>> \t\treturn error(\"Object %s not a commit\",\n>> \t\t\t oid_to_hex(&item->object.oid));\n>> \t}\n>> -\tret = parse_commit_buffer(item, buffer, size);\n>> +\tret = parse_commit_buffer(item, buffer, size, 0);\n>> \tif (save_commit_buffer && !ret) {\n>> \t\tset_commit_buffer(item, buffer, size);\n>> \t\treturn 0;\n>> diff --git a/commit.h b/commit.h\n>> index 64436ff44e..b5afde1ae9 100644\n>> --- a/commit.h\n>> +++ b/commit.h\n>> @@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n>> */\n>> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n>> \n>> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n>> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n>> int parse_commit_gently(struct commit *item, int quiet_on_missing);\n>> static inline int parse_commit(struct commit *item)\n>> {\n>> diff --git a/object.c b/object.c\n>> index e6ad3f61f0..efe4871325 100644\n>> --- a/object.c\n>> +++ b/object.c\n>> @@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n>> \t} else if (type == OBJ_COMMIT) {\n>> \t\tstruct commit *commit = lookup_commit(oid);\n>> \t\tif (commit) {\n>> -\t\t\tif (parse_commit_buffer(commit, buffer, size))\n>> +\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n>> \t\t\t\treturn NULL;\n>> \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n>> \t\t\t\tset_commit_buffer(commit, buffer, size);\n>> diff --git a/sha1_file.c b/sha1_file.c\n>> index 1b94f39c4c..0fd4f0b8b6 100644\n>> --- a/sha1_file.c\n>> +++ b/sha1_file.c\n>> @@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n>> {\n>> \tstruct commit c;\n>> \tmemset(&c, 0, sizeof(c));\n>> -\tif (parse_commit_buffer(&c, buf, size))\n>> +\tif (parse_commit_buffer(&c, buf, size, 0))\n>> \t\tdie(\"corrupt commit\");\n>> }\n\n"},{"id":"345471","messageId":"f4d3bfcc-514e-f4ab-922a-c5214b327c0d@gmail.com","threadId":"48212","inReplyTo":"86y3hkf4d9.fsf@gmail.com","subject":"Re: [PATCH v3 0/9] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-23T14:54:07Z","receivedAt":"2018-04-23T14:54:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/18/2018 8:04 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> -- >8 --\n>>\n>> This is the one of several \"small\" patches that follow the serialized\n>> Git commit graph patch (ds/commit-graph) and lazy-loading trees\n>> (ds/lazy-load-trees).\n>>\n>> As described in Documentation/technical/commit-graph.txt, the generation\n>> number of a commit is one more than the maximum generation number among\n>> its parents (trivially, a commit with no parents has generation number\n>> one). This section is expanded to describe the interaction with special\n>> generation numbers GENERATION_NUMBER_INFINITY (commits not in the commit-graph\n>> file) and *_ZERO (commits in a commit-graph file written before generation\n>> numbers were implemented).\n>>\n>> This series makes the computation of generation numbers part of the\n>> commit-graph write process.\n>>\n>> Finally, generation numbers are used to order commits in the priority\n>> queue in paint_down_to_common(). This allows a short-circuit mechanism\n>> to improve performance of `git branch --contains`.\n>>\n>> Further, use generation numbers for 'git tag --contains', providing a\n>> significant speedup (at least 95% for some cases).\n>>\n>> A more substantial refactoring of revision.c is required before making\n>> 'git log --graph' use generation numbers effectively.\n>>\n>> This patch series is build on ds/lazy-load-trees.\n>>\n>> Derrick Stolee (9):\n>> commit: add generation number to struct commmit\n> Nice and short patch. Looks good to me.\n>\n>> commit-graph: compute generation numbers\n> Another quite easy to understand patch. LGTM.\n>\n>> commit: use generations in paint_down_to_common()\n> Nice and short patch; minor typo in comment in code.\n> Otherwise it looks good to me.\n>\n>> commit-graph.txt: update design document\n> I see that diagram got removed in this version; maybe it could be\n> replaced with relationship table?\n>\n> Anyway, it looks good to me.\n\nThe diagrams and tables seemed to cause more confusion than clarity. I \nthink the reader should create their own mental model from the \ndefinitions and description and we should avoid trying to make a summary.\n\n>\n>> ref-filter: use generation number for --contains\n> A question: how performance looks like after the change if commit-graph\n> is not available?\n\nThe performance issue is minor, but will be fixed in v4.\n\n>\n>> commit: use generation numbers for in_merge_bases()\n> Possible typo in the commit message, and stylistic inconsistence in\n> in_merge_bases() - though actually more clear than existing code.\n>\n> Short, simple, and gives good performance improvenebts.\n>\n>> commit: add short-circuit to paint_down_to_common()\n> Looks good to me; ignore [mostly] what I have written in response to the\n> patch in question.\n>\n>> commit-graph: always load commit-graph information\n> Looks all right; question: parameter or one more global variable.\n\nI responded to say that the global variable approach is incorrect. \nParameter is important to functionality and performance.\n\n>\n>> merge: check config before loading commits\n> This looks good to me.\n>\n>> Documentation/technical/commit-graph.txt | 30 +++++--\n>> alloc.c | 1 +\n>> builtin/merge.c | 5 +-\n>> commit-graph.c | 99 +++++++++++++++++++-----\n>> commit-graph.h | 8 ++\n>> commit.c | 54 +++++++++++--\n>> commit.h | 7 +-\n>> object.c | 2 +-\n>> ref-filter.c | 23 +++++-\n>> sha1_file.c | 2 +-\n>> t/t5318-commit-graph.sh | 9 +++\n>> 11 files changed, 199 insertions(+), 41 deletions(-)\n>>\n>>\n>> base-commit: 7b8a21dba1bce44d64bd86427d3d92437adc4707\n\n"},{"id":"345514","messageId":"86efj5fvrz.fsf@gmail.com","threadId":"48212","inReplyTo":"bd2011f0-41d4-4550-5392-65a3816ed4d3@gmail.com","subject":"Re: [PATCH v3 7/9] commit: add short-circuit to paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-23T21:38:40Z","receivedAt":"2018-04-23T21:40:07Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 4/18/2018 7:19 PM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n>>\n> [...]\n>>> [...], and this saves time during 'git branch --contains' queries\n>>> that would otherwise walk \"around\" the commit we are inspecting.\n>>>\n>> If I understand the code properly, what happens is that we can now\n>> short-circuit if all commits that are left are lower than the target\n>> commit.\n>>\n>> This is because max-order priority queue is used: if the commit with\n>> maximum generation number is below generation number of target commit,\n>> then target commit is not reachable from any commit in the priority\n>> queue (all of which has generation number less or equal than the commit\n>> at head of queue, i.e. all are same level or deeper); compare what I\n>> have written in [1]\n>>\n>> [1]: https://public-inbox.org/git/866052dkju.fsf@gmail.com/\n>>\n>> Do I have that right? If so, it looks all right to me.\n>\n> Yes, the priority queue needs to compare via generation number first\n> or there will be errors. This is why we could not use commit time\n> before.\n\nI was more concerned about getting right the order in the priority queue\n(does it return minimal or maximal generation number).\n\nI understand that the cutoff could not be used without generation\nnumbers because of the possibility of clock skew - using cutoff on dates\ncould lead to wrong results.\n\n>>> For a copy of the Linux repository, where HEAD is checked out at\n>>> v4.13~100, we get the following performance improvement for\n>>> 'git branch --contains' over the previous commit:\n>>>\n>>> Before: 0.21s\n>>> After: 0.13s\n>>> Rel %: -38%\n>> [...]\n>>> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n>>> \t\tif (flags == (PARENT1 | PARENT2)) {\n>>> \t\t\tif (!(commit->object.flags & RESULT)) {\n>>> @@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n>>> \t\t\treturn NULL;\n>>> \t}\n>>> -\tlist = paint_down_to_common(one, n, twos);\n>>> +\tlist = paint_down_to_common(one, n, twos, 0);\n>>> \twhile (list) {\n>>> \t\tstruct commit *commit = pop_commit(&list);\n>>> @@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n>>> \t\t\tfilled_index[filled] = j;\n>>> \t\t\twork[filled++] = array[j];\n>>> \t\t}\n>>> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n>>> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n>>> \t\tif (array[i]->object.flags & PARENT2)\n>>> \t\t\tredundant[i] = 1;\n>>> \t\tfor (j = 0; j < filled; j++)\n>>> @@ -1067,7 +1077,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n>>> \tif (commit->generation > min_generation)\n>>> \t\treturn 0;\n>>> -\tbases = paint_down_to_common(commit, nr_reference, reference);\n>>> +\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n>>\n>> Is it the only case where we would call paint_down_to_common() with\n>> non-zero last parameter? Would we always use commit->generation where\n>> commit is the first parameter of paint_down_to_common()?\n>>\n>> If both are true and will remain true, then in my humble opinion it is\n>> not necessary to change the signature of this function.\n>\n> We need to change the signature some way, but maybe the way I chose is\n> not the best.\n\nNo, after taking longer I think the new signature is a good choice.\n\n> To elaborate: paint_down_to_common() is used for multiple\n> purposes. The caller here that supplies 'commit->generation' is used\n> only to compute reachability (by testing if the flag PARENT2 exists on\n> the commit, then clears all flags). The other callers expect the full\n> walk down to the common commits, and keeps those PARENT1, PARENT2, and\n> STALE flags for future use (such as reporting merge bases). Usually\n> the call to paint_down_to_common() is followed by a revision walk that\n> only halts when reaching root commits or commits with both PARENT1 and\n> PARENT2 flags on, so always short-circuiting on generations would\n> break the functionality; this is confirmed by the\n> t5318-commit-graph.sh.\n\nRight.\n\nI have realized that just after sending the email. I'm sorry about this.\n\n>\n> An alternative to the signature change is to add a boolean parameter\n> \"use_cutoff\" or something, that specifies \"don't walk beyond the\n> commit\". This may give a more of a clear description of what it will\n> do with the generation value, but since we are already performing\n> generation comparisons before calling paint_down_to_common() I find\n> this simple enough.\n\nTwo things:\n\n1. The signature proposed in the patch is more generic. The cutoff does\n not need to be equal to the generation number of the commit, though\n currently it always (all of one time the new mechanism is used) is.\n\n So now I think the new signature of paint_down_to_common() is all\n right as it is proposed here.\n\n2. The way generation numbers are defined (with 0 being a special case,\n and generation numbers starting from 1 for parent-less commits), and\n the way they are compared (using strict comparison, to avoid having\n to special-case _ZERO, _MAX and _INFINITY generation numbers) the\n cutoff of 0 means no cutoff.\n\n On the other hand cutoff of 0 can be understood as meaning no cutoff\n as a special case.\n\n It could be made more clear to use (as I proposed elsewhere in this\n thread) symbolic name for this no-cutoff case via preprocessor\n constants or enums, e.g. GENERATION_NO_CUTOFF:\n\n @@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n \t\t\treturn NULL;\n \t}\n -\tlist = paint_down_to_common(one, n, twos);\n +\tlist = paint_down_to_common(one, n, twos, GENERATION_NO_CUTOFF);\n \twhile (list) {\n \t\tstruct commit *commit = pop_commit(&list);\n @@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n \t\t\tfilled_index[filled] = j;\n \t\t\twork[filled++] = array[j];\n \t\t}\n -\t\tcommon = paint_down_to_common(array[i], filled, work);\n +\t\tcommon = paint_down_to_common(array[i], filled, work, GENERATION_NO_CUTOFF);\n \t\tif (array[i]->object.flags & PARENT2)\n \t\t\tredundant[i] = 1;\n \t\tfor (j = 0; j < filled; j++)\n\n\n But whether it makes code more readable, or less readable, is a\n matter of opinion and taste.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"345643","messageId":"b32b7548-6b58-9619-b53e-aa0bad7a81eb@gmail.com","threadId":"48212","inReplyTo":"86efj5fvrz.fsf@gmail.com","subject":"Re: [PATCH v3 7/9] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-24T12:31:07Z","receivedAt":"2018-04-24T12:31:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/23/2018 5:38 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> On 4/18/2018 7:19 PM, Jakub Narebski wrote:\n>>> Derrick Stolee writes:\n>>>\n>> [...]\n>>>> [...], and this saves time during 'git branch --contains' queries\n>>>> that would otherwise walk \"around\" the commit we are inspecting.\n>>>>\n>>> If I understand the code properly, what happens is that we can now\n>>> short-circuit if all commits that are left are lower than the target\n>>> commit.\n>>>\n>>> This is because max-order priority queue is used: if the commit with\n>>> maximum generation number is below generation number of target commit,\n>>> then target commit is not reachable from any commit in the priority\n>>> queue (all of which has generation number less or equal than the commit\n>>> at head of queue, i.e. all are same level or deeper); compare what I\n>>> have written in [1]\n>>>\n>>> [1]: https://public-inbox.org/git/866052dkju.fsf@gmail.com/\n>>>\n>>> Do I have that right? If so, it looks all right to me.\n>> Yes, the priority queue needs to compare via generation number first\n>> or there will be errors. This is why we could not use commit time\n>> before.\n> I was more concerned about getting right the order in the priority queue\n> (does it return minimal or maximal generation number).\n>\n> I understand that the cutoff could not be used without generation\n> numbers because of the possibility of clock skew - using cutoff on dates\n> could lead to wrong results.\n\nMaximal generation number is important so we do not visit commits \nmultiple times (say, once with PARENT1 set, and a second time when \nPARENT2 is set). A minimal generation number order would create a DFS \norder and walk until the cutoff every time.\n\nIn cases without clock skew, maximal generation number order will walk \nthe same set of commits as maximal commit time; the order may differ, \nbut only between incomparable commits.\n\n>>>> For a copy of the Linux repository, where HEAD is checked out at\n>>>> v4.13~100, we get the following performance improvement for\n>>>> 'git branch --contains' over the previous commit:\n>>>>\n>>>> Before: 0.21s\n>>>> After: 0.13s\n>>>> Rel %: -38%\n>>> [...]\n>>>> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n>>>> \t\tif (flags == (PARENT1 | PARENT2)) {\n>>>> \t\t\tif (!(commit->object.flags & RESULT)) {\n>>>> @@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n>>>> \t\t\treturn NULL;\n>>>> \t}\n>>>> -\tlist = paint_down_to_common(one, n, twos);\n>>>> +\tlist = paint_down_to_common(one, n, twos, 0);\n>>>> \twhile (list) {\n>>>> \t\tstruct commit *commit = pop_commit(&list);\n>>>> @@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n>>>> \t\t\tfilled_index[filled] = j;\n>>>> \t\t\twork[filled++] = array[j];\n>>>> \t\t}\n>>>> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n>>>> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n>>>> \t\tif (array[i]->object.flags & PARENT2)\n>>>> \t\t\tredundant[i] = 1;\n>>>> \t\tfor (j = 0; j < filled; j++)\n>>>> @@ -1067,7 +1077,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n>>>> \tif (commit->generation > min_generation)\n>>>> \t\treturn 0;\n>>>> -\tbases = paint_down_to_common(commit, nr_reference, reference);\n>>>> +\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n>>> Is it the only case where we would call paint_down_to_common() with\n>>> non-zero last parameter? Would we always use commit->generation where\n>>> commit is the first parameter of paint_down_to_common()?\n>>>\n>>> If both are true and will remain true, then in my humble opinion it is\n>>> not necessary to change the signature of this function.\n>> We need to change the signature some way, but maybe the way I chose is\n>> not the best.\n> No, after taking longer I think the new signature is a good choice.\n>\n>> To elaborate: paint_down_to_common() is used for multiple\n>> purposes. The caller here that supplies 'commit->generation' is used\n>> only to compute reachability (by testing if the flag PARENT2 exists on\n>> the commit, then clears all flags). The other callers expect the full\n>> walk down to the common commits, and keeps those PARENT1, PARENT2, and\n>> STALE flags for future use (such as reporting merge bases). Usually\n>> the call to paint_down_to_common() is followed by a revision walk that\n>> only halts when reaching root commits or commits with both PARENT1 and\n>> PARENT2 flags on, so always short-circuiting on generations would\n>> break the functionality; this is confirmed by the\n>> t5318-commit-graph.sh.\n> Right.\n>\n> I have realized that just after sending the email. I'm sorry about this.\n>\n>> An alternative to the signature change is to add a boolean parameter\n>> \"use_cutoff\" or something, that specifies \"don't walk beyond the\n>> commit\". This may give a more of a clear description of what it will\n>> do with the generation value, but since we are already performing\n>> generation comparisons before calling paint_down_to_common() I find\n>> this simple enough.\n> Two things:\n>\n> 1. The signature proposed in the patch is more generic. The cutoff does\n> not need to be equal to the generation number of the commit, though\n> currently it always (all of one time the new mechanism is used) is.\n>\n> So now I think the new signature of paint_down_to_common() is all\n> right as it is proposed here.\n>\n> 2. The way generation numbers are defined (with 0 being a special case,\n> and generation numbers starting from 1 for parent-less commits), and\n> the way they are compared (using strict comparison, to avoid having\n> to special-case _ZERO, _MAX and _INFINITY generation numbers) the\n> cutoff of 0 means no cutoff.\n>\n> On the other hand cutoff of 0 can be understood as meaning no cutoff\n> as a special case.\n>\n> It could be made more clear to use (as I proposed elsewhere in this\n> thread) symbolic name for this no-cutoff case via preprocessor\n> constants or enums, e.g. GENERATION_NO_CUTOFF:\n>\n> @@ -876,7 +886,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n> \t\t\treturn NULL;\n> \t}\n> -\tlist = paint_down_to_common(one, n, twos);\n> +\tlist = paint_down_to_common(one, n, twos, GENERATION_NO_CUTOFF);\n> \twhile (list) {\n> \t\tstruct commit *commit = pop_commit(&list);\n> @@ -943,7 +953,7 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\t\tfilled_index[filled] = j;\n> \t\t\twork[filled++] = array[j];\n> \t\t}\n> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n> +\t\tcommon = paint_down_to_common(array[i], filled, work, GENERATION_NO_CUTOFF);\n> \t\tif (array[i]->object.flags & PARENT2)\n> \t\t\tredundant[i] = 1;\n> \t\tfor (j = 0; j < filled; j++)\n>\n>\n> But whether it makes code more readable, or less readable, is a\n> matter of opinion and taste.\n>\n\nSince paint_down_to_common() is static to this file, I think 0 is \ncleaner. If the method was external and used by other .c files, then I \nwould use this macro trick to clarify \"what does this zero parameter mean?\".\n\nThanks,\n-Stolee\n"},{"id":"345667","messageId":"868t9cfn6f.fsf@gmail.com","threadId":"48212","inReplyTo":"48f910b6-8762-f34b-b644-d6c7ff99499f@gmail.com","subject":"Re: [PATCH v3 5/9] ref-filter: use generation number for --contains","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-24T18:56:40Z","receivedAt":"2018-04-24T18:56:48Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 4/18/2018 5:02 PM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n>>\n>>> A commit A can reach a commit B only if the generation number of A\n>>> is larger than the generation number of B. This condition allows\n>>> significantly short-circuiting commit-graph walks.\n>>>\n>>> Use generation number for 'git tag --contains' queries.\n>>>\n>>> On a copy of the Linux repository where HEAD is containd in v4.13\n>>> but no earlier tag, the command 'git tag --contains HEAD' had the\n>>> following peformance improvement:\n>>>\n>>> Before: 0.81s\n>>> After: 0.04s\n>>> Rel %: -95%\n>>\n>> A question: what is the performance after if the \"commit-graph\" feature\n>> is disabled, or there is no commit-graph file? Is there performance\n>> regression in this case, or is the difference negligible?\n>\n> Negligible, since we are adding a small number of integer comparisons\n> and the main cost is in commit parsing. More on commit parsing in\n> response to your comments below.\n\nIf it is proven to be always negligible, then its all right. If it is\nunlikely to be non-negligible, well, still O.K. But I wonder if maybe\nthere is some situation where the cost of extra parsing is non-negligble.\n\n[...]\n>>> @@ -1618,8 +1623,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n>>> \t\t\t\t\t struct contains_cache *cache)\n>>> {\n>>> \tstruct contains_stack contains_stack = { 0, 0, NULL };\n>>> -\tenum contains_result result = contains_test(candidate, want, cache);\n>>> +\tenum contains_result result;\n>>> +\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n>>> +\tconst struct commit_list *p;\n>>> +\n>>> +\tfor (p = want; p; p = p->next) {\n>>> +\t\tstruct commit *c = p->item;\n>>> +\t\tparse_commit_or_die(c);\n>>> +\t\tif (c->generation < cutoff)\n>>> +\t\t\tcutoff = c->generation;\n>>> +\t}\n>> Sholdn't the above be made conditional on the ability to get generation\n>> numbers from the commit-graph file (feature is turned on and file\n>> exists)? Otherwise here after the change contains_tag_algo() now parses\n>> each commit in 'want', which I think was not done previously.\n>>\n>> With commit-graph file parsing is [probably] cheap. Without it, not\n>> necessary.\n>>\n>> But I might be worrying about nothing.\n>\n> Not nothing. This parses the \"wants\" when we previously did not parse\n> the wants. Further: this parsing happens before we do the simple check\n> of comparing the OID of the candidate against the wants.\n>\n> The question is: are these parsed commits significant compared to the\n> walk that will parse many more commits? It is certainly possible.\n>\n> One way to fix this is to call 'prepare_commit_graph()' directly and\n> then test that 'commit_graph' is non-null before performing any\n> parses. I'm not thrilled with how that couples the commit-graph\n> implementation to this feature, but that may be necessary to avoid\n> regressions in the non-commit-graph case.\n\nAnother possible solution (not sure if better or worse) would be to\nchange the signature of contains_tag_algo() function to take parameter\nor flag that would decide whether to parse \"wants\".\n\nBest,\n-- \nJakub Narębski\n"},{"id":"345786","messageId":"1c63c945-aea0-4d15-9d0c-35092a00c30d@gmail.com","threadId":"48212","inReplyTo":"868t9cfn6f.fsf@gmail.com","subject":"Re: [PATCH v3 5/9] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-25T14:11:41Z","receivedAt":"2018-04-25T14:11:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/24/2018 2:56 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>> One way to fix this is to call 'prepare_commit_graph()' directly and\n>> then test that 'commit_graph' is non-null before performing any\n>> parses. I'm not thrilled with how that couples the commit-graph\n>> implementation to this feature, but that may be necessary to avoid\n>> regressions in the non-commit-graph case.\n> Another possible solution (not sure if better or worse) would be to\n> change the signature of contains_tag_algo() function to take parameter\n> or flag that would decide whether to parse \"wants\".\n\nIf I reorder commits so \"commit-graph:always load commit-graph \ninformation\" is before this one, then we can call \nload_commit_graph_info() which just fills the generation and graph_pos \ninformation. This will keep the coupling very light, instead of needing \nto call prepare_commit_graph() or checking the config setting.\n\nThanks,\n-Stolee\n"},{"id":"345787","messageId":"20180425143735.240183-1-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180417170001.138464-1-dstolee@microsoft.com","subject":"[PATCH v4 00/10] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:53Z","receivedAt":"2018-04-25T14:37:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks for the feedback on the previous version. I think this series is\nstabilizing nicely. I'll reply to this message with an inter-diff as it\nis not too large to share but would clutter this cover letter.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nThis is the one of several \"small\" patches that follow the serialized\nGit commit graph patch (ds/commit-graph) and lazy-loading trees\n(ds/lazy-load-trees).\n\nAs described in Documentation/technical/commit-graph.txt, the generation\nnumber of a commit is one more than the maximum generation number among\nits parents (trivially, a commit with no parents has generation number\none). This section is expanded to describe the interaction with special\ngeneration numbers GENERATION_NUMBER_INFINITY (commits not in the commit-graph\nfile) and *_ZERO (commits in a commit-graph file written before generation\nnumbers were implemented).\n\nThis series makes the computation of generation numbers part of the\ncommit-graph write process.\n\nFinally, generation numbers are used to order commits in the priority\nqueue in paint_down_to_common(). This allows a short-circuit mechanism\nto improve performance of `git branch --contains`.\n\nFurther, use generation numbers for 'git tag --contains), providing a\nsignificant speedup (at least 95% for some cases).\n\nA more substantial refactoring of revision.c is required before making\n'git log --graph' use generation numbers effectively.\n\nThis patch series is built on ds/lazy-load-trees.\n\nDerrick Stolee (10):\n ref-filter: fix outdated comment on in_commit_list\n commit: add generation number to struct commmit\n commit-graph: compute generation numbers\n commit: use generations in paint_down_to_common()\n commit-graph: always load commit-graph information\n ref-filter: use generation number for --contains\n commit: use generation numbers for in_merge_bases()\n commit: add short-circuit to paint_down_to_common()\n merge: check config before loading commits\n commit-graph.txt: update design document\n\n Documentation/technical/commit-graph.txt | 30 ++++++--\n alloc.c | 1 +\n builtin/merge.c | 7 +-\n commit-graph.c | 92 ++++++++++++++++++++----\n commit-graph.h | 8 +++\n commit.c | 54 +++++++++++---\n commit.h | 7 +-\n object.c | 2 +-\n ref-filter.c | 26 +++++--\n sha1_file.c | 2 +-\n t/t5318-commit-graph.sh | 9 +++\n 11 files changed, 198 insertions(+), 40 deletions(-)\n\n\nbase-commit: 7b8a21dba1bce44d64bd86427d3d92437adc4707\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345788","messageId":"20180425143735.240183-2-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 01/10] ref-filter: fix outdated comment on in_commit_list","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:54Z","receivedAt":"2018-04-25T14:38:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The in_commit_list() method does not check the parents of\nthe candidate for containment in the list. Fix the comment\nthat incorrectly states that it does.\n\nReported-by: Jakub Narebski \nSigned-off-by: Derrick Stolee \n---\n ref-filter.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex cffd8bf3ce..aff24d93be 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1582,7 +1582,7 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n }\n \n /*\n- * Test whether the candidate or one of its parents is contained in the list.\n+ * Test whether the candidate is contained in the list.\n * Do not recurse to find out, though, but return -1 if inconclusive.\n */\n static enum contains_result contains_test(struct commit *candidate,\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345789","messageId":"20180425143735.240183-3-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 02/10] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:55Z","receivedAt":"2018-04-25T14:38:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The generation number of a commit is defined recursively as follows:\n\n* If a commit A has no parents, then the generation number of A is one.\n* If a commit A has parents, then the generation number of A is one\n more than the maximum generation number among the parents of A.\n\nAdd a uint32_t generation field to struct commit so we can pass this\ninformation to revision walks. We use three special values to signal\nthe generation number is invalid:\n\nGENERATION_NUMBER_INFINITY 0xFFFFFFFF\nGENERATION_NUMBER_MAX 0x3FFFFFFF\nGENERATION_NUMBER_ZERO 0\n\nThe first (_INFINITY) means the generation number has not been loaded or\ncomputed. The second (_MAX) means the generation number is too large to\nstore in the commit-graph file. The third (_ZERO) means the generation\nnumber was loaded from a commit graph file that was written by a version\nof git that did not support generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 2 ++\n commit.h | 4 ++++\n 3 files changed, 7 insertions(+)\n\ndiff --git a/alloc.c b/alloc.c\nindex cf4f8b61e1..e8ab14f4a1 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n+\tc->generation = GENERATION_NUMBER_INFINITY;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 70fa1b25fd..9ad21c3ffb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -262,6 +262,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \tdate_low = get_be32(commit_data + g->hash_len + 12);\n \titem->date = (timestamp_t)((date_high << 32) | date_low);\n \n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+\n \tpptr = &item->parents;\n \n \tedge_value = get_be32(commit_data + g->hash_len);\ndiff --git a/commit.h b/commit.h\nindex 23a3f364ed..aac3b8c56f 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -10,6 +10,9 @@\n #include \"pretty.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+#define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n+#define GENERATION_NUMBER_MAX 0x3FFFFFFF\n+#define GENERATION_NUMBER_ZERO 0\n \n struct commit_list {\n \tstruct commit *item;\n@@ -30,6 +33,7 @@ struct commit {\n \t */\n \tstruct tree *maybe_tree;\n \tuint32_t graph_pos;\n+\tuint32_t generation;\n };\n \n extern int save_commit_buffer;\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345790","messageId":"20180425143735.240183-4-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 03/10] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:56Z","receivedAt":"2018-04-25T14:38:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While preparing commits to be written into a commit-graph file, compute\nthe generation numbers using a depth-first strategy.\n\nThe only commits that are walked in this depth-first search are those\nwithout a precomputed generation number. Thus, computation time will be\nrelative to the number of new commits to the commit-graph file.\n\nIf a computed generation number would exceed GENERATION_NUMBER_MAX, then\nuse GENERATION_NUMBER_MAX instead.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9ad21c3ffb..047fa9fca5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -439,6 +439,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\telse\n \t\t\tpackedDate[0] = 0;\n \n+\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY)\n+\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n+\n \t\tpackedDate[1] = htonl((*list)->date);\n \t\thashwrite(f, packedDate, 8);\n \n@@ -571,6 +574,46 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n }\n \n+static void compute_generation_numbers(struct commit** commits,\n+\t\t\t\t int nr_commits)\n+{\n+\tint i;\n+\tstruct commit_list *list = NULL;\n+\n+\tfor (i = 0; i < nr_commits; i++) {\n+\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n+\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n+\t\t\tcontinue;\n+\n+\t\tcommit_list_insert(commits[i], &list);\n+\t\twhile (list) {\n+\t\t\tstruct commit *current = list->item;\n+\t\t\tstruct commit_list *parent;\n+\t\t\tint all_parents_computed = 1;\n+\t\t\tuint32_t max_generation = 0;\n+\n+\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n+\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n+\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n+\t\t\t\t\tall_parents_computed = 0;\n+\t\t\t\t\tcommit_list_insert(parent->item, &list);\n+\t\t\t\t\tbreak;\n+\t\t\t\t} else if (parent->item->generation > max_generation) {\n+\t\t\t\t\tmax_generation = parent->item->generation;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (all_parents_computed) {\n+\t\t\t\tcurrent->generation = max_generation + 1;\n+\t\t\t\tpop_commit(&list);\n+\t\t\t}\n+\n+\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n+\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n+\t\t}\n+\t}\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tconst char **pack_indexes,\n \t\t\tint nr_packs,\n@@ -694,6 +737,8 @@ void write_commit_graph(const char *obj_dir,\n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n \n+\tcompute_generation_numbers(commits.list, commits.nr);\n+\n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345791","messageId":"20180425143735.240183-8-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 07/10] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:59Z","receivedAt":"2018-04-25T14:38:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The containment algorithm for 'git branch --contains' is different\nfrom that for 'git tag --contains' in that it uses is_descendant_of()\ninstead of contains_tag_algo(). The expensive portion of the branch\nalgorithm is computing merge bases.\n\nWhen a commit-graph file exists with generation numbers computed,\nwe can avoid this merge-base calculation when the target commit has\na larger generation number than the initial commits.\n\nPerformance tests were run on a copy of the Linux repository where\nHEAD is contained in v4.13 but no earlier tag. Also, all tags were\ncopied to branches and 'git branch --contains' was tested:\n\nBefore: 60.0s\nAfter: 0.4s\nRel %: -99.3%\n\nReported-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 39a3749abd..7bb007f56a 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1056,12 +1056,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n {\n \tstruct commit_list *bases;\n \tint ret = 0, i;\n+\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n \n \tif (parse_commit(commit))\n \t\treturn ret;\n-\tfor (i = 0; i < nr_reference; i++)\n+\tfor (i = 0; i < nr_reference; i++) {\n \t\tif (parse_commit(reference[i]))\n \t\t\treturn ret;\n+\t\tif (min_generation > reference[i]->generation)\n+\t\t\tmin_generation = reference[i]->generation;\n+\t}\n+\n+\tif (commit->generation > min_generation)\n+\t\treturn ret;\n \n \tbases = paint_down_to_common(commit, nr_reference, reference);\n \tif (commit->object.flags & PARENT2)\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345792","messageId":"20180425143735.240183-7-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 06/10] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:58Z","receivedAt":"2018-04-25T14:38:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"A commit A can reach a commit B only if the generation number of A\nis strictly larger than the generation number of B. This condition\nallows significantly short-circuiting commit-graph walks.\n\nUse generation number for '--contains' type queries.\n\nOn a copy of the Linux repository where HEAD is containd in v4.13\nbut no earlier tag, the command 'git tag --contains HEAD' had the\nfollowing peformance improvement:\n\nBefore: 0.81s\nAfter: 0.04s\nRel %: -95%\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n ref-filter.c | 24 ++++++++++++++++++++----\n 1 file changed, 20 insertions(+), 4 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex aff24d93be..fb35067fc9 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -16,6 +16,7 @@\n #include \"trailer.h\"\n #include \"wt-status.h\"\n #include \"commit-slab.h\"\n+#include \"commit-graph.h\"\n \n static struct ref_msg {\n \tconst char *gone;\n@@ -1587,7 +1588,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n */\n static enum contains_result contains_test(struct commit *candidate,\n \t\t\t\t\t const struct commit_list *want,\n-\t\t\t\t\t struct contains_cache *cache)\n+\t\t\t\t\t struct contains_cache *cache,\n+\t\t\t\t\t uint32_t cutoff)\n {\n \tenum contains_result *cached = contains_cache_at(cache, candidate);\n \n@@ -1603,6 +1605,10 @@ static enum contains_result contains_test(struct commit *candidate,\n \n \t/* Otherwise, we don't know; prepare to recurse */\n \tparse_commit_or_die(candidate);\n+\n+\tif (candidate->generation < cutoff)\n+\t\treturn CONTAINS_NO;\n+\n \treturn CONTAINS_UNKNOWN;\n }\n \n@@ -1618,8 +1624,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t\t\t\t struct contains_cache *cache)\n {\n \tstruct contains_stack contains_stack = { 0, 0, NULL };\n-\tenum contains_result result = contains_test(candidate, want, cache);\n+\tenum contains_result result;\n+\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n+\tconst struct commit_list *p;\n+\n+\tfor (p = want; p; p = p->next) {\n+\t\tstruct commit *c = p->item;\n+\t\tload_commit_graph_info(c);\n+\t\tif (c->generation < cutoff)\n+\t\t\tcutoff = c->generation;\n+\t}\n \n+\tresult = contains_test(candidate, want, cache, cutoff);\n \tif (result != CONTAINS_UNKNOWN)\n \t\treturn result;\n \n@@ -1637,7 +1653,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t * If we just popped the stack, parents->item has been marked,\n \t\t * therefore contains_test will return a meaningful yes/no.\n \t\t */\n-\t\telse switch (contains_test(parents->item, want, cache)) {\n+\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n \t\tcase CONTAINS_YES:\n \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n \t\t\tcontains_stack.nr--;\n@@ -1651,7 +1667,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t}\n \t}\n \tfree(contains_stack.contains_stack);\n-\treturn contains_test(candidate, want, cache);\n+\treturn contains_test(candidate, want, cache, cutoff);\n }\n \n static int commit_contains(struct ref_filter *filter, struct commit *commit,\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345793","messageId":"20180425143735.240183-6-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 05/10] commit-graph: always load commit-graph information","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:58Z","receivedAt":"2018-04-25T14:38:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Most code paths load commits using lookup_commit() and then\nparse_commit(). In some cases, including some branch lookups, the commit\nis parsed using parse_object_buffer() which side-steps parse_commit() in\nfavor of parse_commit_buffer().\n\nWith generation numbers in the commit-graph, we need to ensure that any\ncommit that exists in the commit-graph file has its generation number\nloaded.\n\nCreate new load_commit_graph_info() method to fill in the information\nfor a commit that exists only in the commit-graph file. Call it from\nparse_commit_buffer() after loading the other commit information from\nthe given buffer. Only fill this information when specified by the\n'check_graph' parameter.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 45 ++++++++++++++++++++++++++++++---------------\n commit-graph.h | 8 ++++++++\n commit.c | 7 +++++--\n commit.h | 2 +-\n object.c | 2 +-\n sha1_file.c | 2 +-\n 6 files changed, 46 insertions(+), 20 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 047fa9fca5..aebd242def 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -245,6 +245,12 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n \treturn &commit_list_insert(c, pptr)->next;\n }\n \n+static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n+{\n+\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+}\n+\n static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n \tuint32_t edge_value;\n@@ -292,31 +298,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \treturn 1;\n }\n \n+static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n+{\n+\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n+\t\t*pos = item->graph_pos;\n+\t\treturn 1;\n+\t} else {\n+\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t}\n+}\n+\n int parse_commit_in_graph(struct commit *item)\n {\n+\tuint32_t pos;\n+\n \tif (!core_commit_graph)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n-\n \tprepare_commit_graph();\n-\tif (commit_graph) {\n-\t\tuint32_t pos;\n-\t\tint found;\n-\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n-\t\t\tpos = item->graph_pos;\n-\t\t\tfound = 1;\n-\t\t} else {\n-\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n-\t\t}\n-\n-\t\tif (found)\n-\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n-\t}\n-\n+\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n+\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n \treturn 0;\n }\n \n+void load_commit_graph_info(struct commit *item)\n+{\n+\tuint32_t pos;\n+\tif (!core_commit_graph)\n+\t\treturn;\n+\tprepare_commit_graph();\n+\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n+\t\tfill_commit_graph_info(item, commit_graph, pos);\n+}\n+\n static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n {\n \tstruct object_id oid;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 260a468e73..96cccb10f3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n */\n int parse_commit_in_graph(struct commit *item);\n \n+/*\n+ * It is possible that we loaded commit contents from the commit buffer,\n+ * but we also want to ensure the commit-graph content is correctly\n+ * checked and filled. Fill the graph_pos and generation members of\n+ * the given commit.\n+ */\n+void load_commit_graph_info(struct commit *item);\n+\n struct tree *get_commit_tree_in_graph(const struct commit *c);\n \n struct commit_graph {\ndiff --git a/commit.c b/commit.c\nindex 4d00b0a1d6..39a3749abd 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n \treturn ret;\n }\n \n-int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n+int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n {\n \tconst char *tail = buffer;\n \tconst char *bufptr = buffer;\n@@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \t}\n \titem->date = parse_commit_date(bufptr, tail);\n \n+\tif (check_graph)\n+\t\tload_commit_graph_info(item);\n+\n \treturn 0;\n }\n \n@@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn error(\"Object %s not a commit\",\n \t\t\t oid_to_hex(&item->object.oid));\n \t}\n-\tret = parse_commit_buffer(item, buffer, size);\n+\tret = parse_commit_buffer(item, buffer, size, 0);\n \tif (save_commit_buffer && !ret) {\n \t\tset_commit_buffer(item, buffer, size);\n \t\treturn 0;\ndiff --git a/commit.h b/commit.h\nindex 64436ff44e..b5afde1ae9 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n */\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n-int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n+int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\ndiff --git a/object.c b/object.c\nindex e6ad3f61f0..efe4871325 100644\n--- a/object.c\n+++ b/object.c\n@@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n \t} else if (type == OBJ_COMMIT) {\n \t\tstruct commit *commit = lookup_commit(oid);\n \t\tif (commit) {\n-\t\t\tif (parse_commit_buffer(commit, buffer, size))\n+\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n \t\t\t\treturn NULL;\n \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n \t\t\t\tset_commit_buffer(commit, buffer, size);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1b94f39c4c..0fd4f0b8b6 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n {\n \tstruct commit c;\n \tmemset(&c, 0, sizeof(c));\n-\tif (parse_commit_buffer(&c, buf, size))\n+\tif (parse_commit_buffer(&c, buf, size, 0))\n \t\tdie(\"corrupt commit\");\n }\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345794","messageId":"20180425143735.240183-9-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 08/10] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:38:00Z","receivedAt":"2018-04-25T14:38:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git branch --contains', the in_merge_bases_many()\nmethod calls paint_down_to_common() to discover if a specific\ncommit is reachable from a set of branches. Commits with lower\ngeneration number are not needed to correctly answer the\ncontainment query of in_merge_bases_many().\n\nAdd a new parameter, min_generation, to paint_down_to_common() that\nprevents walking commits with generation number strictly less than\nmin_generation. If 0 is given, then there is no functional change.\n\nFor in_merge_bases_many(), we can pass commit->generation as the\ncutoff, and this saves time during 'git branch --contains' queries\nthat would otherwise walk \"around\" the commit we are inspecting.\n\nFor a copy of the Linux repository, where HEAD is checked out at\nv4.13~100, we get the following performance improvement for\n'git branch --contains' over the previous commit:\n\nBefore: 0.21s\nAfter: 0.13s\nRel %: -38%\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 7bb007f56a..e2e16ea1a7 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -808,11 +808,14 @@ static int queue_has_nonstale(struct prio_queue *queue)\n }\n \n /* all input commits in one and twos[] must have been parsed! */\n-static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n+static struct commit_list *paint_down_to_common(struct commit *one, int n,\n+\t\t\t\t\t\tstruct commit **twos,\n+\t\t\t\t\t\tint min_generation)\n {\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n+\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n \n \tone->object.flags |= PARENT1;\n \tif (!n) {\n@@ -831,6 +834,13 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \t\tstruct commit_list *parents;\n \t\tint flags;\n \n+\t\tif (commit->generation > last_gen)\n+\t\t\tBUG(\"bad generation skip\");\n+\t\tlast_gen = commit->generation;\n+\n+\t\tif (commit->generation < min_generation)\n+\t\t\tbreak;\n+\n \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n \t\tif (flags == (PARENT1 | PARENT2)) {\n \t\t\tif (!(commit->object.flags & RESULT)) {\n@@ -879,7 +889,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n \t\t\treturn NULL;\n \t}\n \n-\tlist = paint_down_to_common(one, n, twos);\n+\tlist = paint_down_to_common(one, n, twos, 0);\n \n \twhile (list) {\n \t\tstruct commit *commit = pop_commit(&list);\n@@ -946,7 +956,7 @@ static int remove_redundant(struct commit **array, int cnt)\n \t\t\tfilled_index[filled] = j;\n \t\t\twork[filled++] = array[j];\n \t\t}\n-\t\tcommon = paint_down_to_common(array[i], filled, work);\n+\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n \t\tif (array[i]->object.flags & PARENT2)\n \t\t\tredundant[i] = 1;\n \t\tfor (j = 0; j < filled; j++)\n@@ -1070,7 +1080,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n \tif (commit->generation > min_generation)\n \t\treturn ret;\n \n-\tbases = paint_down_to_common(commit, nr_reference, reference);\n+\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n \tif (commit->object.flags & PARENT2)\n \t\tret = 1;\n \tclear_commit_marks(commit, all_flags);\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345795","messageId":"20180425143735.240183-5-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 04/10] commit: use generations in paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:37:57Z","receivedAt":"2018-04-25T14:39:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Define compare_commits_by_gen_then_commit_date(), which uses generation\nnumbers as a primary comparison and commit date to break ties (or as a\ncomparison when both commits do not have computed generation numbers).\n\nSince the commit-graph file is closed under reachability, we know that\nall commits in the file have generation at most GENERATION_NUMBER_MAX\nwhich is less than GENERATION_NUMBER_INFINITY.\n\nThis change does not affect the number of commits that are walked during\nthe execution of paint_down_to_common(), only the order that those\ncommits are inspected. In the case that commit dates violate topological\norder (i.e. a parent is \"newer\" than a child), the previous code could\nwalk a commit twice: if a commit is reached with the PARENT1 bit, but\nlater is re-visited with the PARENT2 bit, then that PARENT2 bit must be\npropagated to its parents. Using generation numbers avoids this extra\neffort, even if it is somewhat rare.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 20 +++++++++++++++++++-\n commit.h | 1 +\n 2 files changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 711f674c18..4d00b0a1d6 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -640,6 +640,24 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n \treturn 0;\n }\n \n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n+{\n+\tconst struct commit *a = a_, *b = b_;\n+\n+\t/* newer commits first */\n+\tif (a->generation < b->generation)\n+\t\treturn 1;\n+\telse if (a->generation > b->generation)\n+\t\treturn -1;\n+\n+\t/* use date as a heuristic when generations are equal */\n+\tif (a->date < b->date)\n+\t\treturn 1;\n+\telse if (a->date > b->date)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n {\n \tconst struct commit *a = a_, *b = b_;\n@@ -789,7 +807,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n /* all input commits in one and twos[] must have been parsed! */\n static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n {\n-\tstruct prio_queue queue = { compare_commits_by_commit_date };\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n \ndiff --git a/commit.h b/commit.h\nindex aac3b8c56f..64436ff44e 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -341,6 +341,7 @@ extern int remove_signature(struct strbuf *buf);\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \n LAST_ARG_MUST_BE_NULL\n extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345796","messageId":"20180425143735.240183-11-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 10/10] commit-graph.txt: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:38:03Z","receivedAt":"2018-04-25T14:39:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We now calculate generation numbers in the commit-graph file and use\nthem in paint_down_to_common().\n\nExpand the section on generation numbers to discuss how the three\nspecial generation numbers GENERATION_NUMBER_INFINITY, _ZERO, and\n_MAX interact with other generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 30 +++++++++++++++++++-----\n 1 file changed, 24 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 0550c6d0dc..d9f2713efa 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -77,6 +77,29 @@ in the commit graph. We can treat these commits as having \"infinite\"\n generation number and walk until reaching commits with known generation\n number.\n \n+We use the macro GENERATION_NUMBER_INFINITY = 0xFFFFFFFF to mark commits not\n+in the commit-graph file. If a commit-graph file was written by a version\n+of Git that did not compute generation numbers, then those commits will\n+have generation number represented by the macro GENERATION_NUMBER_ZERO = 0.\n+\n+Since the commit-graph file is closed under reachability, we can guarantee\n+the following weaker condition on all commits:\n+\n+ If A and B are commits with generation numbers N amd M, respectively,\n+ and N < M, then A cannot reach B.\n+\n+Note how the strict inequality differs from the inequality when we have\n+fully-computed generation numbers. Using strict inequality may result in\n+walking a few extra commits, but the simplicity in dealing with commits\n+with generation number *_INFINITY or *_ZERO is valuable.\n+\n+We use the macro GENERATION_NUMBER_MAX = 0x3FFFFFFF to for commits whose\n+generation numbers are computed to be at least this value. We limit at\n+this value since it is the largest value that can be stored in the\n+commit-graph file using the 30 bits available to generation numbers. This\n+presents another case where a commit can have generation number equal to\n+that of a parent.\n+\n Design Details\n --------------\n \n@@ -98,17 +121,12 @@ Future Work\n - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n necessary for integration with fsck.\n \n-- The file format includes room for precomputed generation numbers. These\n- are not currently computed, so all generation numbers will be marked as\n- 0 (or \"uncomputed\"). A later patch will include this calculation.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n priority queue with one ordered by generation number. The following\n- operations are important candidates:\n+ operation is an important candidate:\n \n- - paint_down_to_common()\n - 'log --topo-order'\n \n - Currently, parse_commit_gently() requires filling in the root tree\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345797","messageId":"20180425143735.240183-10-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v4 09/10] merge: check config before loading commits","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-25T14:38:02Z","receivedAt":"2018-04-25T14:39:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Now that we use generation numbers from the commit-graph, we must\nensure that all commits that exist in the commit-graph are loaded\nfrom that file instead of from the object database. Since the\ncommit-graph file is only checked if core.commitGraph is true, we\nmust check the default config before we load any commits.\n\nIn the merge builtin, the config was checked after loading the HEAD\ncommit. This was due to the use of the global 'branch' when checking\nmerge-specific config settings.\n\nMove the config load to be between the initialization of 'branch' and\nthe commit lookup.\n\nWithout this change, a fast-forward merge would hit a BUG(\"bad\ngeneration skip\") statement in commit.c during paint_down_to_common().\nThis is because the HEAD commit would be loaded with \"infinite\"\ngeneration but then reached by commits with \"finite\" generation\nnumbers.\n\nAdd a test to t5318-commit-graph.sh that exercises this code path to\nprevent a regression.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/merge.c | 7 ++++---\n t/t5318-commit-graph.sh | 9 +++++++++\n 2 files changed, 13 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 5e5e4497e3..b819756946 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -1148,14 +1148,15 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n \tif (branch)\n \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n+\n+\tinit_diff_ui_defaults();\n+\tgit_config(git_merge_config, NULL);\n+\n \tif (!branch || is_null_oid(&head_oid))\n \t\thead_commit = NULL;\n \telse\n \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n \n-\tinit_diff_ui_defaults();\n-\tgit_config(git_merge_config, NULL);\n-\n \tif (branch_mergeoptions)\n \t\tparse_branch_merge_options(branch_mergeoptions);\n \targc = parse_options(argc, argv, prefix, builtin_merge_options,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a380419b65..77d85aefe7 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n \n+test_expect_success 'perform fast-forward merge in full repo' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit checkout -b merge-5-to-8 commits/5 &&\n+\tgit merge commits/8 &&\n+\tgit show-ref -s merge-5-to-8 >output &&\n+\tgit show-ref -s commits/8 >expect &&\n+\ttest_cmp expect output\n+'\n+\n test_done\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"345799","messageId":"527a8d47-15d4-78a1-4320-97ddc27bce48@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"Re: [PATCH v4 00/10] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-25T14:40:05Z","receivedAt":"2018-04-25T14:40:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As promised, here is the diff from v3.\n\nThanks,\n-Stolee\n\n-- >8 --\n\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 7e1da6c6ea..b819756946 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -1148,6 +1148,7 @@ int cmd_merge(int argc, const char **argv, const \nchar *prefix)\n         branch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, \nNULL);\n         if (branch)\n                 skip_prefix(branch, \"refs/heads/\", &branch);\n+\n         init_diff_ui_defaults();\n         git_config(git_merge_config, NULL);\n\n@@ -1156,7 +1157,6 @@ int cmd_merge(int argc, const char **argv, const \nchar *prefix)\n         else\n                 head_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n\n-\n         if (branch_mergeoptions)\n                 parse_branch_merge_options(branch_mergeoptions);\n         argc = parse_options(argc, argv, prefix, builtin_merge_options,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 21e853c21a..aebd242def 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -257,7 +257,7 @@ static int fill_commit_in_graph(struct commit *item, \nstruct commit_graph *g, uin\n         uint32_t *parent_data_ptr;\n         uint64_t date_low, date_high;\n         struct commit_list **pptr;\n-       const unsigned char *commit_data = g->chunk_commit_data + \nGRAPH_DATA_WIDTH * pos;\n+       const unsigned char *commit_data = g->chunk_commit_data + \n(g->hash_len + 16) * pos;\n\n         item->object.parsed = 1;\n         item->graph_pos = pos;\n@@ -304,7 +304,7 @@ static int find_commit_in_graph(struct commit *item, \nstruct commit_graph *g, uin\n                 *pos = item->graph_pos;\n                 return 1;\n         } else {\n-               return bsearch_graph(commit_graph, &(item->object.oid), \npos);\n+               return bsearch_graph(g, &(item->object.oid), pos);\n         }\n  }\n\n@@ -312,10 +312,10 @@ int parse_commit_in_graph(struct commit *item)\n  {\n         uint32_t pos;\n\n-       if (item->object.parsed)\n-               return 0;\n         if (!core_commit_graph)\n                 return 0;\n+       if (item->object.parsed)\n+               return 1;\n         prepare_commit_graph();\n         if (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n                 return fill_commit_in_graph(item, commit_graph, pos);\n@@ -454,9 +454,8 @@ static void write_graph_chunk_data(struct hashfile \n*f, int hash_len,\n                 else\n                         packedDate[0] = 0;\n\n-               if ((*list)->generation != GENERATION_NUMBER_INFINITY) {\n+               if ((*list)->generation != GENERATION_NUMBER_INFINITY)\n                         packedDate[0] |= htonl((*list)->generation << 2);\n-               }\n\n                 packedDate[1] = htonl((*list)->date);\n                 hashwrite(f, packedDate, 8);\ndiff --git a/commit.c b/commit.c\nindex 9ef6f699bd..e2e16ea1a7 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -653,7 +653,7 @@ int compare_commits_by_gen_then_commit_date(const \nvoid *a_, const void *b_, void\n         else if (a->generation > b->generation)\n                 return -1;\n\n-       /* use date as a heuristic when generataions are equal */\n+       /* use date as a heuristic when generations are equal */\n         if (a->date < b->date)\n                 return 1;\n         else if (a->date > b->date)\n@@ -1078,7 +1078,7 @@ int in_merge_bases_many(struct commit *commit, int \nnr_reference, struct commit *\n         }\n\n         if (commit->generation > min_generation)\n-               return 0;\n+               return ret;\n\n         bases = paint_down_to_common(commit, nr_reference, reference, \ncommit->generation);\n         if (commit->object.flags & PARENT2)\ndiff --git a/ref-filter.c b/ref-filter.c\nindex e2fea6d635..fb35067fc9 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -16,6 +16,7 @@\n  #include \"trailer.h\"\n  #include \"wt-status.h\"\n  #include \"commit-slab.h\"\n+#include \"commit-graph.h\"\n\n  static struct ref_msg {\n         const char *gone;\n@@ -1582,7 +1583,7 @@ static int in_commit_list(const struct commit_list \n*want, struct commit *c)\n  }\n\n  /*\n- * Test whether the candidate or one of its parents is contained in the \nlist.\n+ * Test whether the candidate is contained in the list.\n   * Do not recurse to find out, though, but return -1 if inconclusive.\n   */\n  static enum contains_result contains_test(struct commit *candidate,\n@@ -1629,7 +1630,7 @@ static enum contains_result \ncontains_tag_algo(struct commit *candidate,\n\n         for (p = want; p; p = p->next) {\n                 struct commit *c = p->item;\n-               parse_commit_or_die(c);\n+               load_commit_graph_info(c);\n                 if (c->generation < cutoff)\n                         cutoff = c->generation;\n         }\n\n\n"},{"id":"345854","messageId":"xmqqvace3d9s.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"20180425143735.240183-4-dstolee@microsoft.com","subject":"Re: [PATCH v4 03/10] commit-graph: compute generation numbers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-26T02:35:59Z","receivedAt":"2018-04-26T02:36:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> @@ -439,6 +439,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \t\telse\n> \t\t\tpackedDate[0] = 0;\n> \n> +\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY)\n> +\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n> +\n> \t\tpackedDate[1] = htonl((*list)->date);\n> \t\thashwrite(f, packedDate, 8);\n\nThe ones that have infinity are written as zero here. The code that\nreads the generation field off of a file in fill_commit_graph_info()\nand fill_commit_in_graph() both leave such a record in file as-is,\nso the reader of what we write out will think it is _ZERO, not _INF.\n\nNot that it matters, as it seems that most of the code being added\nby this series treat _ZERO and _INF more or less interchangeably.\nBut it does raise another question, i.e. do we need both _ZERO and\n_INF, or is it sufficient to have just a single _UNKNOWN?\n\n> @@ -571,6 +574,46 @@ static void close_reachable(struct packed_oid_list *oids)\n> \t}\n> }\n> \n> +static void compute_generation_numbers(struct commit** commits,\n> +\t\t\t\t int nr_commits)\n> +{\n> +\tint i;\n> +\tstruct commit_list *list = NULL;\n> +\n> +\tfor (i = 0; i < nr_commits; i++) {\n> +\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n> +\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n> +\t\t\tcontinue;\n> +\n> +\t\tcommit_list_insert(commits[i], &list);\n> +\t\twhile (list) {\n> +\t\t\tstruct commit *current = list->item;\n> +\t\t\tstruct commit_list *parent;\n> +\t\t\tint all_parents_computed = 1;\n> +\t\t\tuint32_t max_generation = 0;\n> +\n> +\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n> +\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n> +\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n> +\t\t\t\t\tall_parents_computed = 0;\n> +\t\t\t\t\tcommit_list_insert(parent->item, &list);\n> +\t\t\t\t\tbreak;\n> +\t\t\t\t} else if (parent->item->generation > max_generation) {\n> +\t\t\t\t\tmax_generation = parent->item->generation;\n> +\t\t\t\t}\n> +\t\t\t}\n> +\n> +\t\t\tif (all_parents_computed) {\n> +\t\t\t\tcurrent->generation = max_generation + 1;\n> +\t\t\t\tpop_commit(&list);\n> +\t\t\t}\n\nIf we haven't computed all parents' generations yet,\ncurrent->generation is undefined (or at least \"left as\ninitialized\"), so it does not make much sense to attempt to clip it\nat _MAX at this point. At leat not yet.\n\nIOW, shouldn't the following two lines be inside the \"we now know\ngenno of all parents, so we can compute genno for commit\" block\nabove?\n\n> +\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n> +\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n> +\t\t}\n> +\t}\n> +}\n> +\n> void write_commit_graph(const char *obj_dir,\n> \t\t\tconst char **pack_indexes,\n> \t\t\tint nr_packs,\n> @@ -694,6 +737,8 @@ void write_commit_graph(const char *obj_dir,\n> \tif (commits.nr >= GRAPH_PARENT_MISSING)\n> \t\tdie(_(\"too many commits to write graph\"));\n> \n> +\tcompute_generation_numbers(commits.list, commits.nr);\n> +\n> \tgraph_name = get_commit_graph_filename(obj_dir);\n> \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n"},{"id":"345855","messageId":"xmqqr2n23b4o.fsf@gitster-ct.c.googlers.com","threadId":"48212","inReplyTo":"20180425143735.240183-5-dstolee@microsoft.com","subject":"Re: [PATCH v4 04/10] commit: use generations in paint_down_to_common()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-26T03:22:15Z","receivedAt":"2018-04-26T03:22:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Define compare_commits_by_gen_then_commit_date(), which uses generation\n> numbers as a primary comparison and commit date to break ties (or as a\n> comparison when both commits do not have computed generation numbers).\n>\n> Since the commit-graph file is closed under reachability, we know that\n> all commits in the file have generation at most GENERATION_NUMBER_MAX\n> which is less than GENERATION_NUMBER_INFINITY.\n\nI suspect that my puzzlement may be coming from my not \"getting\"\nwhat you meant by \"closed under reachability\", but could you also\nexplain how _INF and _ZERO interact with commits with normal\ngeneration numbers? I've always assumed that genno will be used\nonly when comparing two commits with valid genno and otherwise we'd\nfall back to the traditional date based one, but...\n\n> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n> +{\n> +\tconst struct commit *a = a_, *b = b_;\n> +\n> +\t/* newer commits first */\n> +\tif (a->generation < b->generation)\n> +\t\treturn 1;\n> +\telse if (a->generation > b->generation)\n> +\t\treturn -1;\n\n... this does not check if a->generation is _ZERO or _INF. \n\nBoth being _MAX is OK (the control will fall through and use the\ndates below). One being _MAX and the other being a normal value is\nalso OK (the above comparisons will declare the commit with _MAX is\nfarther than less-than-max one from a root).\n\nOr is the assumption that if one has _ZERO, that must have come from\nan ancient commit-graph file and none of the commits have anything\nbut _ZERO?\n\n> +\t/* use date as a heuristic when generations are equal */\n> +\tif (a->date < b->date)\n> +\t\treturn 1;\n> +\telse if (a->date > b->date)\n> +\t\treturn -1;\n> +\treturn 0;\n> +}\n> +\n> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n> {\n> \tconst struct commit *a = a_, *b = b_;\n> @@ -789,7 +807,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n> /* all input commits in one and twos[] must have been parsed! */\n> static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n> {\n> -\tstruct prio_queue queue = { compare_commits_by_commit_date };\n> +\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n> \tstruct commit_list *result = NULL;\n> \tint i;\n> \n> diff --git a/commit.h b/commit.h\n> index aac3b8c56f..64436ff44e 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -341,6 +341,7 @@ extern int remove_signature(struct strbuf *buf);\n> extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n> \n> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n> \n> LAST_ARG_MUST_BE_NULL\n> extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n"},{"id":"345885","messageId":"86tvryjq7p.fsf@gmail.com","threadId":"48212","inReplyTo":"xmqqr2n23b4o.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 04/10] commit: use generations in paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-26T09:02:02Z","receivedAt":"2018-04-26T09:02:12Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano writes:\n> Derrick Stolee writes:\n>\n>> Define compare_commits_by_gen_then_commit_date(), which uses generation\n>> numbers as a primary comparison and commit date to break ties (or as a\n>> comparison when both commits do not have computed generation numbers).\n>>\n>> Since the commit-graph file is closed under reachability, we know that\n>> all commits in the file have generation at most GENERATION_NUMBER_MAX\n>> which is less than GENERATION_NUMBER_INFINITY.\n>\n> I suspect that my puzzlement may be coming from my not \"getting\"\n> what you meant by \"closed under reachability\",\n\nIt means that if commit A is in the commit graph, then all of its\nancestors (all commits reachable from A) are also in the commit graph.\n\n> but could you also\n> explain how _INF and _ZERO interact with commits with normal\n> generation numbers? I've always assumed that genno will be used\n> only when comparing two commits with valid genno and otherwise we'd\n> fall back to the traditional date based one, but...\n>\n>> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n>> +{\n>> +\tconst struct commit *a = a_, *b = b_;\n>> +\n>> +\t/* newer commits first */\n>> +\tif (a->generation < b->generation)\n>> +\t\treturn 1;\n>> +\telse if (a->generation > b->generation)\n>> +\t\treturn -1;\n>\n> ... this does not check if a->generation is _ZERO or _INF. \n>\n> Both being _MAX is OK (the control will fall through and use the\n> dates below). One being _MAX and the other being a normal value is\n> also OK (the above comparisons will declare the commit with _MAX is\n> farther than less-than-max one from a root).\n>\n> Or is the assumption that if one has _ZERO, that must have come from\n> an ancient commit-graph file and none of the commits have anything\n> but _ZERO?\n\nThere is stronger and weaker version of the negative-cut criteria based\non generation numbers.\n\nThe strong criteria:\n\n if A != B and gen(A) <= gen(B), then A cannot reach B\n\nThe weaker criteria:\n\n if gen(A) < gen(B), then A cannot reach B\n\n\nBecause commit-graph is closed under reachability, this means that\n\n if A is in commit graph, and B is outside of it, then A cannot reach B\n\nIf A is in commit graph, then either _MAX >= gen(A) >= 1,\nor gen(A) == _ZERO. Because _INFINITY > _MAX > _ZERO, then we have\n\n if _MAX >= gen(A) >= 1 || gen(A) == 0, and gen(B) == _INFINITY\n then A cannot reach B\n\nwhich also fullfils the weaker criteria\n\n if gen(A) < gen(B), then A cannot reach B\n\n\nIf both A and B are outside commit-graph, i.e. gen(A) = gen(B) = _INFINITY,\nor if both A and B have gen(A) = gen(B) = _MAX,\nor if both A and B come from old commit graph with gen(A) = gen(B) =_ZERO,\nthen we cannot say anything about reachability... and weak criteria\nalso does not say anything about reachability.\n\n\nMaybe the following ASCII table would make it clear.\n\n | gen(B)\n | ................................ :::::::\ngen(A) | _INFINITY | _MAX | larger | smaller | _ZERO\n-------------+-----------+----------+----------+----------+--------\n_INFINITY | = | > | > | > | >\n_MAX | < Nn | = | > | > | >\nlarger | < Nn | < Nn | = n | > | >\nsmaller | < Nn | < Nn | < Nn | = n | >\n_ZERO | < Nn | < Nn | < Nn | < Nn | =\n\nHere \"n\" denotes stronger condition, and \"N\" denotes weaker condition.\nWe have _INFINITY > _MAX > larger > smaller > _ZERO.\n\n\nNOTE however that it is a *tradeoff*. Using weaker criteria, with\nstrict inequality, means that we don't need to handle _INFINITY, _MAX\nand _ZERO corner-cases in a special way; but it also means that we would\nwalk slightly more commits than if we used stronger criteria, with less\nor equals.\n\nFor Linux kernel public repository commit graph[1] we have maximum of 512\ncommits sharing the same level, 5.43 sharing the same commit on average,\nand 50% of time only 2 commits sharing the same level (median, or 2nd\nquartile, or 50% percentile). This is roughly the amount of commits we\nwalk more with weaker cut-off condition.\n\n[1]: with 750k commits, but which is not largest commit graph any more :-0\n\n>> +\t/* use date as a heuristic when generations are equal */\n>> +\tif (a->date < b->date)\n>> +\t\treturn 1;\n>> +\telse if (a->date > b->date)\n>> +\t\treturn -1;\n>> +\treturn 0;\n>> +}\n\nHTH\n-- \nJakub Narębski\n"},{"id":"345894","messageId":"21272fff-acf9-8641-3aa9-1d026843ff3a@gmail.com","threadId":"48212","inReplyTo":"xmqqvace3d9s.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/10] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-26T12:58:39Z","receivedAt":"2018-04-26T12:59:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"n 4/25/2018 10:35 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> @@ -439,6 +439,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n>> \t\telse\n>> \t\t\tpackedDate[0] = 0;\n>> \n>> +\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY)\n>> +\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n>> +\n>> \t\tpackedDate[1] = htonl((*list)->date);\n>> \t\thashwrite(f, packedDate, 8);\n> The ones that have infinity are written as zero here. The code that\n> reads the generation field off of a file in fill_commit_graph_info()\n> and fill_commit_in_graph() both leave such a record in file as-is,\n> so the reader of what we write out will think it is _ZERO, not _INF.\n>\n> Not that it matters, as it seems that most of the code being added\n> by this series treat _ZERO and _INF more or less interchangeably.\n> But it does raise another question, i.e. do we need both _ZERO and\n> _INF, or is it sufficient to have just a single _UNKNOWN?\n\nThis code is confusing. The 'if' condition is useless, since at this \npoint every commit should be finite (since we computed generation \nnumbers for everyone). We should just write the value always.\n\nFor the sake of discussion, the value _INFINITY means not in the graph \nand _ZERO means in the graph without a computed generation number. It's \na small distinction, but it gives a single boundary to use for \nreachability queries that test generation number.\n\n>\n>> @@ -571,6 +574,46 @@ static void close_reachable(struct packed_oid_list *oids)\n>> \t}\n>> }\n>> \n>> +static void compute_generation_numbers(struct commit** commits,\n>> +\t\t\t\t int nr_commits)\n>> +{\n>> +\tint i;\n>> +\tstruct commit_list *list = NULL;\n>> +\n>> +\tfor (i = 0; i < nr_commits; i++) {\n>> +\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n>> +\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tcommit_list_insert(commits[i], &list);\n>> +\t\twhile (list) {\n>> +\t\t\tstruct commit *current = list->item;\n>> +\t\t\tstruct commit_list *parent;\n>> +\t\t\tint all_parents_computed = 1;\n>> +\t\t\tuint32_t max_generation = 0;\n>> +\n>> +\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n>> +\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n>> +\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n>> +\t\t\t\t\tall_parents_computed = 0;\n>> +\t\t\t\t\tcommit_list_insert(parent->item, &list);\n>> +\t\t\t\t\tbreak;\n>> +\t\t\t\t} else if (parent->item->generation > max_generation) {\n>> +\t\t\t\t\tmax_generation = parent->item->generation;\n>> +\t\t\t\t}\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (all_parents_computed) {\n>> +\t\t\t\tcurrent->generation = max_generation + 1;\n>> +\t\t\t\tpop_commit(&list);\n>> +\t\t\t}\n> If we haven't computed all parents' generations yet,\n> current->generation is undefined (or at least \"left as\n> initialized\"), so it does not make much sense to attempt to clip it\n> at _MAX at this point. At leat not yet.\n>\n> IOW, shouldn't the following two lines be inside the \"we now know\n> genno of all parents, so we can compute genno for commit\" block\n> above?\n\nYou're right! Good catch. This code sets every merge commit to _MAX. It \nshould be in the block above.\n\n>\n>> +\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n>> +\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n>> +\t\t}\n>> +\t}\n>> +}\n>> +\n>> void write_commit_graph(const char *obj_dir,\n>> \t\t\tconst char **pack_indexes,\n>> \t\t\tint nr_packs,\n>> @@ -694,6 +737,8 @@ void write_commit_graph(const char *obj_dir,\n>> \tif (commits.nr >= GRAPH_PARENT_MISSING)\n>> \t\tdie(_(\"too many commits to write graph\"));\n>> \n>> +\tcompute_generation_numbers(commits.list, commits.nr);\n>> +\n>> \tgraph_name = get_commit_graph_filename(obj_dir);\n>> \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n"},{"id":"345895","messageId":"3a7e7f76-834b-2671-6c61-67a42b2eeb77@gmail.com","threadId":"48212","inReplyTo":"21272fff-acf9-8641-3aa9-1d026843ff3a@gmail.com","subject":"Re: [PATCH v4 03/10] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-26T13:49:09Z","receivedAt":"2018-04-26T13:49:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 4/26/2018 8:58 AM, Derrick Stolee wrote:\n> n 4/25/2018 10:35 PM, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>>\n>>> @@ -439,6 +439,9 @@ static void write_graph_chunk_data(struct \n>>> hashfile *f, int hash_len,\n>>>           else\n>>>               packedDate[0] = 0;\n>>>   +        if ((*list)->generation != GENERATION_NUMBER_INFINITY)\n>>> +            packedDate[0] |= htonl((*list)->generation << 2);\n>>> +\n>>>           packedDate[1] = htonl((*list)->date);\n>>>           hashwrite(f, packedDate, 8);\n>> The ones that have infinity are written as zero here.  The code that\n>> reads the generation field off of a file in fill_commit_graph_info()\n>> and fill_commit_in_graph() both leave such a record in file as-is,\n>> so the reader of what we write out will think it is _ZERO, not _INF.\n>>\n>> Not that it matters, as it seems that most of the code being added\n>> by this series treat _ZERO and _INF more or less interchangeably.\n>> But it does raise another question, i.e. do we need both _ZERO and\n>> _INF, or is it sufficient to have just a single _UNKNOWN?\n>\n> This code is confusing. The 'if' condition is useless, since at this \n> point every commit should be finite (since we computed generation \n> numbers for everyone). We should just write the value always.\n>\n> For the sake of discussion, the value _INFINITY means not in the graph \n> and _ZERO means in the graph without a computed generation number. \n> It's a small distinction, but it gives a single boundary to use for \n> reachability queries that test generation number.\n>\n>>\n>>> @@ -571,6 +574,46 @@ static void close_reachable(struct \n>>> packed_oid_list *oids)\n>>>       }\n>>>   }\n>>>   +static void compute_generation_numbers(struct commit** commits,\n>>> +                       int nr_commits)\n>>> +{\n>>> +    int i;\n>>> +    struct commit_list *list = NULL;\n>>> +\n>>> +    for (i = 0; i < nr_commits; i++) {\n>>> +        if (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n>>> +            commits[i]->generation != GENERATION_NUMBER_ZERO)\n>>> +            continue;\n>>> +\n>>> +        commit_list_insert(commits[i], &list);\n>>> +        while (list) {\n>>> +            struct commit *current = list->item;\n>>> +            struct commit_list *parent;\n>>> +            int all_parents_computed = 1;\n>>> +            uint32_t max_generation = 0;\n>>> +\n>>> +            for (parent = current->parents; parent; parent = \n>>> parent->next) {\n>>> +                if (parent->item->generation == \n>>> GENERATION_NUMBER_INFINITY ||\n>>> +                    parent->item->generation == \n>>> GENERATION_NUMBER_ZERO) {\n>>> +                    all_parents_computed = 0;\n>>> +                    commit_list_insert(parent->item, &list);\n>>> +                    break;\n>>> +                } else if (parent->item->generation > \n>>> max_generation) {\n>>> +                    max_generation = parent->item->generation;\n>>> +                }\n>>> +            }\n>>> +\n>>> +            if (all_parents_computed) {\n>>> +                current->generation = max_generation + 1;\n>>> +                pop_commit(&list);\n>>> +            }\n>> If we haven't computed all parents' generations yet,\n>> current->generation is undefined (or at least \"left as\n>> initialized\"), so it does not make much sense to attempt to clip it\n>> at _MAX at this point.  At leat not yet.\n>>\n>> IOW, shouldn't the following two lines be inside the \"we now know\n>> genno of all parents, so we can compute genno for commit\" block\n>> above?\n>\n> You're right! Good catch. This code sets every merge commit to _MAX. \n> It should be in the block above.\n>\n>>\n>>> +            if (current->generation > GENERATION_NUMBER_MAX)\n>>> +                current->generation = GENERATION_NUMBER_MAX;\n>>> +        }\n>>> +    }\n\nThis bothered me: why didn't I catch a bug here? I rebased my \"fsck\" RFC \nonto this branch and it succeeded. Then, I realized that this does not \nactually write incorrect values, since we re-visit this commit again \nafter we pop the stack down to this commit. However, there is time in \nthe middle where we have set the generation (in memory) incorrectly and \nthat could easily turn into a real bug by a later change.\n\nI'll stick the _MAX check in the if above to prevent confusion.\n\nThanks,\n-Stolee\n\n"},{"id":"346040","messageId":"86o9i3jt0a.fsf@gmail.com","threadId":"48212","inReplyTo":"86tvryjq7p.fsf@gmail.com","subject":"Re: [PATCH v4 04/10] commit: use generations in paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-28T14:38:29Z","receivedAt":"2018-04-28T14:38:38Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski writes:\n> Junio C Hamano writes:\n>> Derrick Stolee writes:\n[...]\n>>> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n>>> +{\n>>> +\tconst struct commit *a = a_, *b = b_;\n>>> +\n>>> +\t/* newer commits first */\n>>> +\tif (a->generation < b->generation)\n>>> +\t\treturn 1;\n>>> +\telse if (a->generation > b->generation)\n>>> +\t\treturn -1;\n>>\n>> ... this does not check if a->generation is _ZERO or _INF. \n>>\n>> Both being _MAX is OK (the control will fall through and use the\n>> dates below). One being _MAX and the other being a normal value is\n>> also OK (the above comparisons will declare the commit with _MAX is\n>> farther than less-than-max one from a root).\n>>\n>> Or is the assumption that if one has _ZERO, that must have come from\n>> an ancient commit-graph file and none of the commits have anything\n>> but _ZERO?\n>\n> There is stronger and weaker version of the negative-cut criteria based\n> on generation numbers.\n>\n> The strong criteria:\n>\n> if A != B and gen(A) <= gen(B), then A cannot reach B\n>\n> The weaker criteria:\n>\n> if gen(A) < gen(B), then A cannot reach B\n>\n>\n> Because commit-graph is closed under reachability, this means that\n>\n> if A is in commit graph, and B is outside of it, then A cannot reach B\n>\n> If A is in commit graph, then either _MAX >= gen(A) >= 1,\n> or gen(A) == _ZERO. Because _INFINITY > _MAX > _ZERO, then we have\n>\n> if _MAX >= gen(A) >= 1 || gen(A) == 0, and gen(B) == _INFINITY\n> then A cannot reach B\n>\n> which also fullfils the weaker criteria\n>\n> if gen(A) < gen(B), then A cannot reach B\n>\n>\n> If both A and B are outside commit-graph, i.e. gen(A) = gen(B) = _INFINITY,\n> or if both A and B have gen(A) = gen(B) = _MAX,\n> or if both A and B come from old commit graph with gen(A) = gen(B) =_ZERO,\n> then we cannot say anything about reachability... and weak criteria\n> also does not say anything about reachability.\n>\n>\n> Maybe the following ASCII table would make it clear.\n>\n> | gen(B)\n> | ................................ :::::::\n> gen(A) | _INFINITY | _MAX | larger | smaller | _ZERO\n> -------------+-----------+----------+----------+----------+--------\n> _INFINITY | = | > | > | > | >\n> _MAX | < Nn | = | > | > | >\n> larger | < Nn | < Nn | = n | > | >\n> smaller | < Nn | < Nn | < Nn | = n | >\n> _ZERO | < Nn | < Nn | < Nn | < Nn | =\n>\n> Here \"n\" denotes stronger condition, and \"N\" denotes weaker condition.\n> We have _INFINITY > _MAX > larger > smaller > _ZERO.\n>\n>\n> NOTE however that it is a *tradeoff*. Using weaker criteria, with\n> strict inequality, means that we don't need to handle _INFINITY, _MAX\n> and _ZERO corner-cases in a special way; but it also means that we would\n> walk slightly more commits than if we used stronger criteria, with less\n> or equals.\n\nActually, if we look at the table above, it turns out that we can use\nthe stronger version of negative-cut criteria without special-casing all\nthe possible combinations. Just use stronger criteria on normal range,\nweaker criteria if any of generation numbers is special generation\nnumber.\n\n if _MAX > gen(A) > _ZERO and\n _MAX > gen(B) > _ZERO then\n\n if A != B and gen(A) <= gen(B) then\n A cannot reach B\n else\n A can reach B\n\n else /* at least one special case */\n\n if gen(A) < gen(B) then\n A cannot reach B\n else\n A can reach B\n\n\nNOTE that it specifically does not matter for created here\ncompare_commits_by_gen_then_commit_date(), as it requires strict\ninequality for sorting - which using weak criteria explains why we don't\nneed any special cases in the code here.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346046","messageId":"86in8bjl4j.fsf@gmail.com","threadId":"48212","inReplyTo":"527a8d47-15d4-78a1-4320-97ddc27bce48@gmail.com","subject":"Re: [PATCH v4 00/10] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-28T17:28:44Z","receivedAt":"2018-04-28T17:28:52Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> As promised, here is the diff from v3.\n\nWhat is this strange string \"       \" in place of tabs in the interdiff?\n\" \" here is Unicode Character 'NO-BREAK SPACE' (U+00A0).\n\nThough it doesn't matter for viewing, my newsreader (Gnus from GNU\nEmacs) thinks that it is worth notifying about when replying.\n\nAlso, it looks like at least in one place the diff got line-wrapped.\n\n> Thanks,\n> -Stolee\n>\n> -- >8 --\n>\n> diff --git a/builtin/merge.c b/builtin/merge.c\n> index 7e1da6c6ea..b819756946 100644\n> --- a/builtin/merge.c\n> +++ b/builtin/merge.c\n> @@ -1148,6 +1148,7 @@ int cmd_merge(int argc, const char **argv, const\n> char *prefix)\n>         branch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid,\n> NULL);\n>         if (branch)\n>                 skip_prefix(branch, \"refs/heads/\", &branch);\n> +\n>         init_diff_ui_defaults();\n>         git_config(git_merge_config, NULL);\n>\n> @@ -1156,7 +1157,6 @@ int cmd_merge(int argc, const char **argv, const\n> char *prefix)\n>         else\n>                 head_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n>\n> -\n>         if (branch_mergeoptions)\n>                 parse_branch_merge_options(branch_mergeoptions);\n>         argc = parse_options(argc, argv, prefix, builtin_merge_options,\n\nWhitespace fixes, all right.\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 21e853c21a..aebd242def 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -257,7 +257,7 @@ static int fill_commit_in_graph(struct commit\n> *item, struct commit_graph *g, uin\n>         uint32_t *parent_data_ptr;\n>         uint64_t date_low, date_high;\n>         struct commit_list **pptr;\n> -       const unsigned char *commit_data = g->chunk_commit_data +\n> GRAPH_DATA_WIDTH * pos;\n> +       const unsigned char *commit_data = g->chunk_commit_data +\n> (g->hash_len + 16) * pos;\n>\n>         item->object.parsed = 1;\n>         item->graph_pos = pos;\n\nThis was accidental change in v3 (unrelated to the changes in commit it\nwere in). Though I wonder if the symbolic constant route is not better\n- though as separate standalone commit.\n\n> @@ -304,7 +304,7 @@ static int find_commit_in_graph(struct commit\n> *item, struct commit_graph *g, uin\n>                 *pos = item->graph_pos;\n>                 return 1;\n>         } else {\n> -               return bsearch_graph(commit_graph,\n> &(item->object.oid), pos);\n> +               return bsearch_graph(g, &(item->object.oid), pos);\n>         }\n>  }\n>\n\nFixup for a commit, that was sent in separate fixup email in v3. All\nright.\n\nThough I wonder if it wouldn't be better to call global variable\n'the_commit_graph' to avoid such errors in the future...\n\n> @@ -312,10 +312,10 @@ int parse_commit_in_graph(struct commit *item)\n>  {\n>         uint32_t pos;\n>\n> -       if (item->object.parsed)\n> -               return 0;\n>         if (!core_commit_graph)\n>                 return 0;\n> +       if (item->object.parsed)\n> +               return 1;\n>         prepare_commit_graph();\n>         if (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n>                 return fill_commit_in_graph(item, commit_graph, pos);\n\nFixed accidental flip-flopping about return value when\nitem->object.parsed. I'd have to take a look at actual commits to say\nwhether I think it is all right or not.\n\n> @@ -454,9 +454,8 @@ static void write_graph_chunk_data(struct hashfile\n> *f, int hash_len,\n>                 else\n>                         packedDate[0] = 0;\n>\n> -               if ((*list)->generation != GENERATION_NUMBER_INFINITY) {\n> +               if ((*list)->generation != GENERATION_NUMBER_INFINITY)\n>                         packedDate[0] |= htonl((*list)->generation << 2);\n> -               }\n>\n>                 packedDate[1] = htonl((*list)->date);\n>                 hashwrite(f, packedDate, 8);\n\nCoding style change, to be more in line with CodingGuidelines, namely\nthat we usually do not use block for single-command in conditionals.\n\nAll right.\n\n> diff --git a/commit.c b/commit.c\n> index 9ef6f699bd..e2e16ea1a7 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -653,7 +653,7 @@ int compare_commits_by_gen_then_commit_date(const\n> void *a_, const void *b_, void\n>         else if (a->generation > b->generation)\n>                 return -1;\n>\n> -       /* use date as a heuristic when generataions are equal */\n> +       /* use date as a heuristic when generations are equal */\n>         if (a->date < b->date)\n>                 return 1;\n>         else if (a->date > b->date)\n\nFixed typo in comment. All right.\n\n> @@ -1078,7 +1078,7 @@ int in_merge_bases_many(struct commit *commit,\n> int nr_reference, struct commit *\n>         }\n>\n>         if (commit->generation > min_generation)\n> -               return 0;\n> +               return ret;\n>\n>         bases = paint_down_to_common(commit, nr_reference, reference,\n> commit->generation);\n>         if (commit->object.flags & PARENT2)\n\nUnifying way of returning result (to one that was used before this\ncommit in this fragment of the git code). Looks all right, from what I\nremember.\n\n> diff --git a/ref-filter.c b/ref-filter.c\n> index e2fea6d635..fb35067fc9 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -16,6 +16,7 @@\n>  #include \"trailer.h\"\n>  #include \"wt-status.h\"\n>  #include \"commit-slab.h\"\n> +#include \"commit-graph.h\"\n>\n>  static struct ref_msg {\n>         const char *gone;\n> @@ -1629,7 +1630,7 @@ static enum contains_result\n> contains_tag_algo(struct commit *candidate,\n>\n>         for (p = want; p; p = p->next) {\n>                 struct commit *c = p->item;\n> -               parse_commit_or_die(c);\n> +               load_commit_graph_info(c);\n>                 if (c->generation < cutoff)\n>                         cutoff = c->generation;\n>         }\n\nAvoiding performance penalty when not using commit-graph feature (or\nwhen it is turned off). Looks good on first glance.\n\n> @@ -1582,7 +1583,7 @@ static int in_commit_list(const struct\n> commit_list *want, struct commit *c)\n>  }\n>\n>  /*\n> - * Test whether the candidate or one of its parents is contained in\n> the list.\n> + * Test whether the candidate is contained in the list.\n>   * Do not recurse to find out, though, but return -1 if inconclusive.\n>   */\n>  static enum contains_result contains_test(struct commit *candidate,\n\nBringing comment in line with the function it is about. Good.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346047","messageId":"86d0yjjjx3.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-2-dstolee@microsoft.com","subject":"Re: [PATCH v4 01/10] ref-filter: fix outdated comment on in_commit_list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-28T17:54:48Z","receivedAt":"2018-04-28T17:54:55Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The in_commit_list() method does not check the parents of\n> the candidate for containment in the list. Fix the comment\n> that incorrectly states that it does.\n>\n> Reported-by: Jakub Narebski \n> Signed-off-by: Derrick Stolee \n> ---\n> ref-filter.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index cffd8bf3ce..aff24d93be 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -1582,7 +1582,7 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n> }\n> \n> /*\n> - * Test whether the candidate or one of its parents is contained in the list.\n> + * Test whether the candidate is contained in the list.\n> * Do not recurse to find out, though, but return -1 if inconclusive.\n> */\n> static enum contains_result contains_test(struct commit *candidate,\n\nAll right. Always good to have comment and code match.\n\n\nFYI: the contains_test() function described in this comment only checks\nthe candidate, and never access candidate commit parents. All\nrecursion, which naturally includes checking parents, is in the\ncontains_tag_algo().\n\nI guess that the code was refactored, but comment had not been changed.\n\n-- \nJakub Narębski\n"},{"id":"346055","messageId":"86604bj6xr.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-3-dstolee@microsoft.com","subject":"Re: [PATCH v4 02/10] commit: add generation number to struct commmit","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-28T22:35:12Z","receivedAt":"2018-04-28T22:35:19Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The generation number of a commit is defined recursively as follows:\n>\n> * If a commit A has no parents, then the generation number of A is one.\n> * If a commit A has parents, then the generation number of A is one\n> more than the maximum generation number among the parents of A.\n\nVery minor nitpick: it would be more readable wrapped differently:\n\n * If a commit A has parents, then the generation number of A is\n one more than the maximum generation number among parents of A.\n\nVery minor nitpick: possibly \"parents\", not \"the parents\", but I am\nnot native English speaker.\n\n>\n> Add a uint32_t generation field to struct commit so we can pass this\n> information to revision walks. We use three special values to signal\n> the generation number is invalid:\n>\n> GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n> GENERATION_NUMBER_MAX 0x3FFFFFFF\n> GENERATION_NUMBER_ZERO 0\n>\n> The first (_INFINITY) means the generation number has not been loaded or\n> computed. The second (_MAX) means the generation number is too large to\n> store in the commit-graph file. The third (_ZERO) means the generation\n> number was loaded from a commit graph file that was written by a version\n> of git that did not support generation numbers.\n\nGood explanation; I wonder if we want to have it in some shortened form\nalso in comments, and not only in the commit message.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> alloc.c | 1 +\n> commit-graph.c | 2 ++\n> commit.h | 4 ++++\n> 3 files changed, 7 insertions(+)\n\nI have reordered patches to make it easier to review.\n\n> diff --git a/commit.h b/commit.h\n> index 23a3f364ed..aac3b8c56f 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -10,6 +10,9 @@\n> #include \"pretty.h\"\n> \n> #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n> +#define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n> +#define GENERATION_NUMBER_MAX 0x3FFFFFFF\n> +#define GENERATION_NUMBER_ZERO 0\n\nI wonder if it wouldn't be good to have some short in-line comments\nexplaining those constants, or a block comment above them.\n\n> \n> struct commit_list {\n> \tstruct commit *item;\n> @@ -30,6 +33,7 @@ struct commit {\n> \t */\n> \tstruct tree *maybe_tree;\n> \tuint32_t graph_pos;\n> +\tuint32_t generation;\n> };\n> \n> extern int save_commit_buffer;\n\nAll right, simple addition of the new field. Nothing to go wrong here.\n\nSidenote: With 0x7FFFFFFF being (if I am not wrong) maximum graph_pos\nand maximum number of nodes in commit graph, we won't hit 0x3FFFFFFF\ngeneration number limit for all except very, very linear histories.\n\n>\n> diff --git a/alloc.c b/alloc.c\n> index cf4f8b61e1..e8ab14f4a1 100644\n> --- a/alloc.c\n> +++ b/alloc.c\n> @@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n> \tc->object.type = OBJ_COMMIT;\n> \tc->index = alloc_commit_index();\n> \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n> +\tc->generation = GENERATION_NUMBER_INFINITY;\n> \treturn c;\n> }\n\nAll right, start with initializing it with \"not from commit-graph\" value\nafter allocation.\n\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 70fa1b25fd..9ad21c3ffb 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -262,6 +262,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \tdate_low = get_be32(commit_data + g->hash_len + 12);\n> \titem->date = (timestamp_t)((date_high << 32) | date_low);\n> \n> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n> +\n\nI guess we should not worry about these \"magical constants\" sprinkled\nhere, like \"+ 8\" above.\n\nLet's examine how it goes, taking a look at commit-graph-format.txt\nin Documentation/technical/commit-graph-format.txt\n\n * The first H (g->hash_len) bytes are for the OID of the root tree.\n * The next 8 bytes are for the positions of the first two parents [...]\n\nSo 'commit_data + g->hash_len + 8' is our offset from the start of\ncommit data. All right.\n\n * The next 8 bytes store the generation number of the commit and\n the commit time in seconds since EPOCH. The generation number\n uses the higher 30 bits of the first 4 bytes. [...]\n\nThe higher 30 bits of the 4 bytes, which is 32 bits, means that we need\nto shift 32-bit value 2 bits right, so that we get lower 30 bits of\n32-bit value. All right.\n\n All 4-byte numbers are in network order.\n\nShouldn't it be ntohl() to convert from network order to host order, and\nnot get_be32()? I guess they are the same (network order is big-endian\norder), and get_be32() is what rest of git uses...\n\nLooks all right.\n\n> \tpptr = &item->parents;\n> \n> \tedge_value = get_be32(commit_data + g->hash_len);\n"},{"id":"346070","messageId":"86r2myidmq.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-4-dstolee@microsoft.com","subject":"Re: [PATCH v4 03/10] commit-graph: compute generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-29T09:08:13Z","receivedAt":"2018-04-29T09:08:22Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> While preparing commits to be written into a commit-graph file, compute\n> the generation numbers using a depth-first strategy.\n\nSidenote: for generation numbers it does not matter if we use\ndepth-first or breadth-first strategy, but it is more natural to use\ndepth-first search because generation numbers need post-order processing\n(parents before child).\n\n>\n> The only commits that are walked in this depth-first search are those\n> without a precomputed generation number. Thus, computation time will be\n> relative to the number of new commits to the commit-graph file.\n\nA question: what happens if the existing commit graph is from older\nversion of git and has _ZERO for generation numbers?\n\nAnswer: I see that we treat both _INFINITY (not in commit-graph) and\n_ZERO (in commit graph but not computed) as not computed generation\nnumbers. All right.\n\n>\n> If a computed generation number would exceed GENERATION_NUMBER_MAX, then\n> use GENERATION_NUMBER_MAX instead.\n\nAll right, though I guess this would remain theoretical for a long\nwhile.\n\nWe don't have any way of testing this, at least not without recompiling\nGit with lower value of GENERATION_NUMBER_MAX -- which means not\nautomatically, isn't it?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 45 +++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 45 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 9ad21c3ffb..047fa9fca5 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -439,6 +439,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \t\telse\n> \t\t\tpackedDate[0] = 0;\n> \n> +\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY)\n> +\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n> +\n\nIf we stumble upon commit marked as \"not in commit-graph\" while writing\ncommit graph, it is a BUG(), isn't it?\n\n(Problem noticed by Junio.)\n\nIt is a bit strange to me that the code uses get_be32 for reading, but\nhtonl for writing. Is Git tested on non little-endian machines, like\nbig-endian ppc64 or s390x, or on mixed-endian machines (or\nselectable-endian machines with data endianness set to non\nlittle-endian, like ia64)? If not, could we use for example openSUSE\nBuild Service (https://build.opensuse.org/) for this?\n\n> \t\tpackedDate[1] = htonl((*list)->date);\n> \t\thashwrite(f, packedDate, 8);\n> \n> @@ -571,6 +574,46 @@ static void close_reachable(struct packed_oid_list *oids)\n> \t}\n> }\n> \n> +static void compute_generation_numbers(struct commit** commits,\n> +\t\t\t\t int nr_commits)\n> +{\n> +\tint i;\n> +\tstruct commit_list *list = NULL;\n\nAll right, commit_list will work as stack.\n\n> +\n> +\tfor (i = 0; i < nr_commits; i++) {\n> +\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n> +\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n> +\t\t\tcontinue;\n\nAll right, we consider _INFINITY and _SERO as not computed. If\ngeneration number is computed (by 'recursion' or from commit graph), we\n(re)use it. This means that generation number calculation is\nincremental, as intended -- good.\n\n> +\n> +\t\tcommit_list_insert(commits[i], &list);\n\nStart depth-first walks from commits given.\n\n> +\t\twhile (list) {\n> +\t\t\tstruct commit *current = list->item;\n> +\t\t\tstruct commit_list *parent;\n> +\t\t\tint all_parents_computed = 1;\n\nHere all_parents_computed is a boolean flag. I see that it is easier to\nstart with assumption that all parents will have computed generation\nnumbers.\n\n> +\t\t\tuint32_t max_generation = 0;\n\nThe generation number value of 0 functions as sentinel; generation\nnumbers start from 1. Not that it matters much, as lowest possible\ngeneration number is 1, and we could have started from that value.\n\n> +\n> +\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n> +\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n> +\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n> +\t\t\t\t\tall_parents_computed = 0;\n> +\t\t\t\t\tcommit_list_insert(parent->item, &list);\n> +\t\t\t\t\tbreak;\n\nIf some parent doesn't have generation number calculated, we add it to\nstack (and break out of loop because it is depth-first walk), and mark\nthis situation. All right.\n\n> +\t\t\t\t} else if (parent->item->generation > max_generation) {\n> +\t\t\t\t\tmax_generation = parent->item->generation;\n\nOtherwise, update max_generation. All right.\n\n> +\t\t\t\t}\n> +\t\t\t}\n> +\n> +\t\t\tif (all_parents_computed) {\n> +\t\t\t\tcurrent->generation = max_generation + 1;\n> +\t\t\t\tpop_commit(&list);\n> +\t\t\t}\n> +\n> +\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n> +\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n\nThis conditional should be inside all_parents_computed test, for example\nlike this:\n\n +\t\t\tif (all_parents_computed) {\n +\t\t\t\tcurrent->generation = max_generation + 1;\n +\t\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n +\t\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n +\n +\t\t\t\tpop_commit(&list);\n +\t\t\t}\n\n(Noticed by Junio.)\n\nSidenote: when we revisit the commit, returning from depth-first walk of\none of its parents, we calculate max_generation from scratch again.\nThis does not matter for performance, as it's just data access and\ncalculating maximum - any workaround to not restart those calculations\nwould take more time and memory. And it's simple.\n\n> +\t\t}\n> +\t}\n> +}\n> +\n> void write_commit_graph(const char *obj_dir,\n> \t\t\tconst char **pack_indexes,\n> \t\t\tint nr_packs,\n> @@ -694,6 +737,8 @@ void write_commit_graph(const char *obj_dir,\n> \tif (commits.nr >= GRAPH_PARENT_MISSING)\n> \t\tdie(_(\"too many commits to write graph\"));\n> \n> +\tcompute_generation_numbers(commits.list, commits.nr);\n> +\n\nNice and simple. All right.\n\nI guess that we do not pass \"struct packed_commit_list commits\" as\nargument to compute_generation_numbers instead of \"struct commit**\ncommits.list\" and \"int commits.nr\" to compute_generation_numbers() to\nkeep the latter nice and generic?\n\n> \tgraph_name = get_commit_graph_filename(obj_dir);\n> \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346077","messageId":"86k1sqhvgm.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-5-dstolee@microsoft.com","subject":"Re: [PATCH v4 04/10] commit: use generations in paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-29T15:40:41Z","receivedAt":"2018-04-29T15:40:51Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Define compare_commits_by_gen_then_commit_date(), which uses generation\n> numbers as a primary comparison and commit date to break ties (or as a\n> comparison when both commits do not have computed generation numbers).\n\nAll right, this looks reasonable thing to do when we have access to\ncommit generation numbers..\n\n> Since the commit-graph file is closed under reachability, we know that\n> all commits in the file have generation at most GENERATION_NUMBER_MAX\n> which is less than GENERATION_NUMBER_INFINITY.\n\nThus the condition that if B is reachable from A, then gen(A) >= gen(B),\neven if they have generation numbers _INFINITY, _MAX or _ZERO.\n\nWe use generation numbers, if possible, to choose closest commit; if\nnot, we use dates.\n\n>\n> This change does not affect the number of commits that are walked during\n> the execution of paint_down_to_common(), only the order that those\n> commits are inspected. In the case that commit dates violate topological\n> order (i.e. a parent is \"newer\" than a child), the previous code could\n> walk a commit twice: if a commit is reached with the PARENT1 bit, but\n> later is re-visited with the PARENT2 bit, then that PARENT2 bit must be\n> propagated to its parents. Using generation numbers avoids this extra\n> effort, even if it is somewhat rare.\n\nActually the ordering of commits walked does not affect the correctness\nof the result. Better ordering means that commits do not need to be\nwalked twice; I think it would be possible to craft repository in which\nunlucky clock skew would lead to depth-first walk of commits later part\nof walk would mark STALE.\n\nPedantry aside, I think it is a good description of analysis of change\nresults.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 20 +++++++++++++++++++-\n> commit.h | 1 +\n> 2 files changed, 20 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit.c b/commit.c\n> index 711f674c18..4d00b0a1d6 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -640,6 +640,24 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n> \treturn 0;\n> }\n> \n> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n> +{\n> +\tconst struct commit *a = a_, *b = b_;\n> +\n> +\t/* newer commits first */\n\nTo be pedantic, larger generation number does not necessary mean that\ncommit was created later (is newer), only that it is on longer chain\nsince common ancestor or root commit.\n\n> +\tif (a->generation < b->generation)\n> +\t\treturn 1;\n> +\telse if (a->generation > b->generation)\n> +\t\treturn -1;\n\nIf the commit-graph feature is not available, or is disabled, all\ncommits would have the same generation number (_INFINITY), then this\nblock would be always practically no-op.\n\nThis is not very costly: 2 access to data which should be in cache, and\n1 to 2 comparison operations. But I wonder if we wouldn't want to avoid\nthis nano-cost if possible...\n\n> +\n> +\t/* use date as a heuristic when generations are equal */\n> +\tif (a->date < b->date)\n> +\t\treturn 1;\n> +\telse if (a->date > b->date)\n> +\t\treturn -1;\n> +\treturn 0;\n\nThe above is the same code as in compare_commits_by_commit_date(), but\nthere it is with \"newer commits with larger date first\" as comment\ninstead.\n\nAll right: we need inlining for speed.\n\n> +}\n> +\n> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n> {\n> \tconst struct commit *a = a_, *b = b_;\n> @@ -789,7 +807,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n> /* all input commits in one and twos[] must have been parsed! */\n> static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n> {\n> -\tstruct prio_queue queue = { compare_commits_by_commit_date };\n> +\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n\nI wonder if it would be worth it to avoid comparing by generation\nnumbers without commit graph data:\n\n +\tstruct prio_queue queue;\n [...]\n +\tif (commit_graph)\n +\t\tqueue.compare = compare_commits_by_gen_then_commit_date;\n +\telse\n +\t\tqueue.compare = compare_commits_by_commit_date;\n\nOr something like that. But perhaps this nano-optimization is not worth\nit (it is not that complicated, though).\n\n\nSidenote: when I searched for compare_commits_by_commit_date use, I have\nnoticed that it is used, I think as heuristics, for packfile creation in\nupload-pack.c and fetch-pack.c. Would they similarly improve with\ncompare_commits_by_gen_then_commit_date?\n\nThis is of course not something that this commit, or this patch series,\nneeds to address now.\n\n> \tstruct commit_list *result = NULL;\n> \tint i;\n> \n> diff --git a/commit.h b/commit.h\n> index aac3b8c56f..64436ff44e 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -341,6 +341,7 @@ extern int remove_signature(struct strbuf *buf);\n> extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n> \n> int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n> +int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n\nAll right.\n\n> \n> LAST_ARG_MUST_BE_NULL\n> extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n"},{"id":"346105","messageId":"866049irrx.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-6-dstolee@microsoft.com","subject":"Re: [PATCH v4 05/10] commit-graph: always load commit-graph information","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-29T22:14:58Z","receivedAt":"2018-04-29T22:15:08Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Most code paths load commits using lookup_commit() and then\n> parse_commit().\n\nAnd this automatically loads commit graph if needed, thanks to changes\nin parse_commit_gently(), which parse_commit() uses.\n\n> In some cases, including some branch lookups, the commit\n> is parsed using parse_object_buffer() which side-steps parse_commit() in\n> favor of parse_commit_buffer().\n\nI guess the problem is that we cannot just add parse_commit_in_graph() \nlike we did in parse_commit_gently(), for some reason? Like for example\nthat parse_commit_gently() uses parse_commit_buffer() - which could have\nbeen handled by moving parse_commit_in_graph() down the call chain from\nparse_commit_gently() to parse_commit_buffer()... if not the fact that\ncheck_commit() also uses parse_commit_buffer(), but it does not want to\nload commit graph. Am I right?\n\n>\n> With generation numbers in the commit-graph, we need to ensure that any\n> commit that exists in the commit-graph file has its generation number\n> loaded.\n\nIs it generation number, or generation number and position in commit\ngraph?\n\n>\n> Create new load_commit_graph_info() method to fill in the information\n> for a commit that exists only in the commit-graph file. Call it from\n> parse_commit_buffer() after loading the other commit information from\n> the given buffer. Only fill this information when specified by the\n> 'check_graph' parameter.\n\nI think this commit would be easier to review if it was split into pure\nrefactoring part (extracting fill_commit_graph_info() and\nfind_commit_in_graph()). On the other hand the refactoring was needed\nto reduce code duplication betweem existing parse_commit_in_graph() and\nnew load_commit_graph_info() functions.\n\nI guess that the difference between parse_commit_in_graph() and\nload_commit_graph_info() is that the former cares only about having just\nenough information that is needed for parse_commit_gently() - and does\nnot load graph data if commit is parsed, while the latter is about\nloading commit-graph data like generation numbers.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 45 ++++++++++++++++++++++++++++++---------------\n> commit-graph.h | 8 ++++++++\n> commit.c | 7 +++++--\n> commit.h | 2 +-\n> object.c | 2 +-\n> sha1_file.c | 2 +-\n> 6 files changed, 46 insertions(+), 20 deletions(-)\n\nI wonder if it would be possible to add tests for this feature, for\nexample that commit-graph is read when it should (including those branch\nlookups), and is not read when the feature should be disabled.\n\nBut the only way to test it I can think of is a stupid one: create\ninvalid commit graph, and check that git fails as expected (trying to\nread said malformed file), and does not fail if commit graph feature is\ndisabled.\n\n>\n\nLet me reorder files (BTW, is there a way for Git to put *.h files\nbefore *.c files in diff?) for easier review:\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 260a468e73..96cccb10f3 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n> */\n> int parse_commit_in_graph(struct commit *item);\n> \n> +/*\n> + * It is possible that we loaded commit contents from the commit buffer,\n> + * but we also want to ensure the commit-graph content is correctly\n> + * checked and filled. Fill the graph_pos and generation members of\n> + * the given commit.\n> + */\n> +void load_commit_graph_info(struct commit *item);\n> +\n> struct tree *get_commit_tree_in_graph(const struct commit *c);\n> \n> struct commit_graph {\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 047fa9fca5..aebd242def 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -245,6 +245,12 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n> \treturn &commit_list_insert(c, pptr)->next;\n> }\n> \n> +static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n> +{\n> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n> +}\n\nThe comment in the header file commit-graph.h talks about filling\ngraph_pos and generation members of the given commit, but I don't see\nfilling graph_pos member here.\n\nSidenote: it is a tiny little bit strange to see symbolic constants like\nGRAPH_DATA_WIDTH near using magic values such as 8 and 2.\n\n> +\n> static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n> {\n> \tuint32_t edge_value;\n> @@ -292,31 +298,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \treturn 1;\n> }\n> \n> +static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n> +{\n> +\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n> +\t\t*pos = item->graph_pos;\n> +\t\treturn 1;\n> +\t} else {\n> +\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n> +\t}\n> +}\n\nNice refactoring here.\n\n> +\n> int parse_commit_in_graph(struct commit *item)\n> {\n> +\tuint32_t pos;\n> +\n> \tif (!core_commit_graph)\n> \t\treturn 0;\n> \tif (item->object.parsed)\n> \t\treturn 1;\n> -\n> \tprepare_commit_graph();\n> -\tif (commit_graph) {\n> -\t\tuint32_t pos;\n> -\t\tint found;\n> -\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n> -\t\t\tpos = item->graph_pos;\n> -\t\t\tfound = 1;\n> -\t\t} else {\n> -\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n> -\t\t}\n> -\n> -\t\tif (found)\n> -\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> -\t}\n> -\n> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> +\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> \treturn 0;\n> }\n> \n> +void load_commit_graph_info(struct commit *item)\n> +{\n> +\tuint32_t pos;\n> +\tif (!core_commit_graph)\n> +\t\treturn;\n> +\tprepare_commit_graph();\n> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> +\t\tfill_commit_graph_info(item, commit_graph, pos);\n> +}\n\nSimilar functions, different goals (as the names imply).\n\n> +\n> static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n> {\n> \tstruct object_id oid;\n> diff --git a/commit.c b/commit.c\n> index 4d00b0a1d6..39a3749abd 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n> \treturn ret;\n> }\n> \n> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n> {\n> \tconst char *tail = buffer;\n> \tconst char *bufptr = buffer;\n> @@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n> \t}\n> \titem->date = parse_commit_date(bufptr, tail);\n> \n> +\tif (check_graph)\n> +\t\tload_commit_graph_info(item);\n> +\n\nAll right, read commit-graph specific data after parsing commit itself.\nIt is at the end because commit object needs to be parsed sequentially,\nand it includes more info that is contained in commit-graph CDAT+EDGE\ndata.\n\n> \treturn 0;\n> }\n> \n> @@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \t\treturn error(\"Object %s not a commit\",\n> \t\t\t oid_to_hex(&item->object.oid));\n> \t}\n> -\tret = parse_commit_buffer(item, buffer, size);\n> +\tret = parse_commit_buffer(item, buffer, size, 0);\n\nThe parse_commit_gently() contract is that it provides only bare minimum\nof information, from commit-graph if possible, and does read object from\ndisk and parses it only when it could not avoid it. If it needs to\nparse it, it doesn't need to fill commit-graph specific data again.\n\nAll right.\n\n> \tif (save_commit_buffer && !ret) {\n> \t\tset_commit_buffer(item, buffer, size);\n> \t\treturn 0;\n> diff --git a/commit.h b/commit.h\n> index 64436ff44e..b5afde1ae9 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n> */\n> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n> \n> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n> int parse_commit_gently(struct commit *item, int quiet_on_missing);\n> static inline int parse_commit(struct commit *item)\n> {\n> diff --git a/object.c b/object.c\n> index e6ad3f61f0..efe4871325 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n> \t} else if (type == OBJ_COMMIT) {\n> \t\tstruct commit *commit = lookup_commit(oid);\n> \t\tif (commit) {\n> -\t\t\tif (parse_commit_buffer(commit, buffer, size))\n> +\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n\nAll that rigamarole was needed because of\n\nDS> In some cases, including some branch lookups, the commit\nDS> is parsed using parse_object_buffer() which side-steps parse_commit() in\nDS> favor of parse_commit_buffer().\n\nHere we want parse_object_buffer() to get also commit-graph specific\ndata, if available. All right.\n\n> \t\t\t\treturn NULL;\n> \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n> \t\t\t\tset_commit_buffer(commit, buffer, size);\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 1b94f39c4c..0fd4f0b8b6 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n> {\n> \tstruct commit c;\n> \tmemset(&c, 0, sizeof(c));\n> -\tif (parse_commit_buffer(&c, buf, size))\n> +\tif (parse_commit_buffer(&c, buf, size, 0))\n\nFor check we don't need commit graph data. Looks all right.\n\n> \t\tdie(\"corrupt commit\");\n> }\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346110","messageId":"86y3h5hd1i.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-6-dstolee@microsoft.com","subject":"Re: [PATCH v4 05/10] commit-graph: always load commit-graph information","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-29T22:18:33Z","receivedAt":"2018-04-29T22:18:43Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[Forgot about one thing]\n\nDerrick Stolee writes:\n\n> Create new load_commit_graph_info() method to fill in the information\n> for a commit that exists only in the commit-graph file.\n\nThe above sentence is a bit hard to parse because of ambiguity: is it\n\"the information\" that exists only in the commit-graph file, or \"a\ncommit\" that exists only in the commit-graph file?\n\n> Call it from\n> parse_commit_buffer() after loading the other commit information from\n> the given buffer. Only fill this information when specified by the\n> 'check_graph' parameter.\n>\n> Signed-off-by: Derrick Stolee \n\n-- \nJakub Narębski\n"},{"id":"346142","messageId":"4d6aa9bf-7bb3-8e56-83c7-88962bd64308@gmail.com","threadId":"48212","inReplyTo":"86604bj6xr.fsf@gmail.com","subject":"Re: [PATCH v4 02/10] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-30T12:05:04Z","receivedAt":"2018-04-30T12:05:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/28/2018 6:35 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> The generation number of a commit is defined recursively as follows:\n>>\n>> * If a commit A has no parents, then the generation number of A is one.\n>> * If a commit A has parents, then the generation number of A is one\n>> more than the maximum generation number among the parents of A.\n> Very minor nitpick: it would be more readable wrapped differently:\n>\n> * If a commit A has parents, then the generation number of A is\n> one more than the maximum generation number among parents of A.\n>\n> Very minor nitpick: possibly \"parents\", not \"the parents\", but I am\n> not native English speaker.\n>\n>> Add a uint32_t generation field to struct commit so we can pass this\n>> information to revision walks. We use three special values to signal\n>> the generation number is invalid:\n>>\n>> GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n>> GENERATION_NUMBER_MAX 0x3FFFFFFF\n>> GENERATION_NUMBER_ZERO 0\n>>\n>> The first (_INFINITY) means the generation number has not been loaded or\n>> computed. The second (_MAX) means the generation number is too large to\n>> store in the commit-graph file. The third (_ZERO) means the generation\n>> number was loaded from a commit graph file that was written by a version\n>> of git that did not support generation numbers.\n> Good explanation; I wonder if we want to have it in some shortened form\n> also in comments, and not only in the commit message.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> alloc.c | 1 +\n>> commit-graph.c | 2 ++\n>> commit.h | 4 ++++\n>> 3 files changed, 7 insertions(+)\n> I have reordered patches to make it easier to review.\n>\n>> diff --git a/commit.h b/commit.h\n>> index 23a3f364ed..aac3b8c56f 100644\n>> --- a/commit.h\n>> +++ b/commit.h\n>> @@ -10,6 +10,9 @@\n>> #include \"pretty.h\"\n>> \n>> #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n>> +#define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n>> +#define GENERATION_NUMBER_MAX 0x3FFFFFFF\n>> +#define GENERATION_NUMBER_ZERO 0\n> I wonder if it wouldn't be good to have some short in-line comments\n> explaining those constants, or a block comment above them.\n>\n>> \n>> struct commit_list {\n>> \tstruct commit *item;\n>> @@ -30,6 +33,7 @@ struct commit {\n>> \t */\n>> \tstruct tree *maybe_tree;\n>> \tuint32_t graph_pos;\n>> +\tuint32_t generation;\n>> };\n>> \n>> extern int save_commit_buffer;\n> All right, simple addition of the new field. Nothing to go wrong here.\n>\n> Sidenote: With 0x7FFFFFFF being (if I am not wrong) maximum graph_pos\n> and maximum number of nodes in commit graph, we won't hit 0x3FFFFFFF\n> generation number limit for all except very, very linear histories.\n\nBoth of these limits are far away from being realistic. But we could \nextend the maximum graph_pos independently from the maximum generation \nnumber now that we have the \"capped\" logic.\n\n>\n>> diff --git a/alloc.c b/alloc.c\n>> index cf4f8b61e1..e8ab14f4a1 100644\n>> --- a/alloc.c\n>> +++ b/alloc.c\n>> @@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n>> \tc->object.type = OBJ_COMMIT;\n>> \tc->index = alloc_commit_index();\n>> \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n>> +\tc->generation = GENERATION_NUMBER_INFINITY;\n>> \treturn c;\n>> }\n> All right, start with initializing it with \"not from commit-graph\" value\n> after allocation.\n>\n>> \n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 70fa1b25fd..9ad21c3ffb 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -262,6 +262,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n>> \tdate_low = get_be32(commit_data + g->hash_len + 12);\n>> \titem->date = (timestamp_t)((date_high << 32) | date_low);\n>> \n>> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n>> +\n> I guess we should not worry about these \"magical constants\" sprinkled\n> here, like \"+ 8\" above.\n>\n> Let's examine how it goes, taking a look at commit-graph-format.txt\n> in Documentation/technical/commit-graph-format.txt\n>\n> * The first H (g->hash_len) bytes are for the OID of the root tree.\n> * The next 8 bytes are for the positions of the first two parents [...]\n>\n> So 'commit_data + g->hash_len + 8' is our offset from the start of\n> commit data. All right.\n>\n> * The next 8 bytes store the generation number of the commit and\n> the commit time in seconds since EPOCH. The generation number\n> uses the higher 30 bits of the first 4 bytes. [...]\n>\n> The higher 30 bits of the 4 bytes, which is 32 bits, means that we need\n> to shift 32-bit value 2 bits right, so that we get lower 30 bits of\n> 32-bit value. All right.\n>\n> All 4-byte numbers are in network order.\n>\n> Shouldn't it be ntohl() to convert from network order to host order, and\n> not get_be32()? I guess they are the same (network order is big-endian\n> order), and get_be32() is what rest of git uses...\n\nntohl() takes a 32-bit value, while get_be32() takes a pointer. This \nmakes pulling network-bytes out of streams much cleaner with get_be32(), \nso I try to use that whenever possible.\n\n"},{"id":"346175","messageId":"86po2ghcw2.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-7-dstolee@microsoft.com","subject":"Re: [PATCH v4 06/10] ref-filter: use generation number for --contains","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-30T16:34:05Z","receivedAt":"2018-04-30T16:34:14Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> A commit A can reach a commit B only if the generation number of A\n> is strictly larger than the generation number of B. This condition\n> allows significantly short-circuiting commit-graph walks.\n>\n> Use generation number for '--contains' type queries.\n>\n> On a copy of the Linux repository where HEAD is containd in v4.13\n\nMinor typo: containd -> contained.\n\n> but no earlier tag, the command 'git tag --contains HEAD' had the\n> following peformance improvement:\n>\n> Before: 0.81s\n> After: 0.04s\n> Rel %: -95%\n\nVery nice. I guess that any performance changes for when commit-graph\nfeature is not available are negligible / not measurable.\n\nRel % = (before - after)/before * 100%, isn't it?.\n\nGood.\n\n>\n> Helped-by: Jeff King \n> Signed-off-by: Derrick Stolee \n> ---\n> ref-filter.c | 24 ++++++++++++++++++++----\n> 1 file changed, 20 insertions(+), 4 deletions(-)\n>\n> diff --git a/ref-filter.c b/ref-filter.c\n> index aff24d93be..fb35067fc9 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -16,6 +16,7 @@\n> #include \"trailer.h\"\n> #include \"wt-status.h\"\n> #include \"commit-slab.h\"\n> +#include \"commit-graph.h\"\n> \n> static struct ref_msg {\n> \tconst char *gone;\n> @@ -1587,7 +1588,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n> */\n> static enum contains_result contains_test(struct commit *candidate,\n> \t\t\t\t\t const struct commit_list *want,\n> -\t\t\t\t\t struct contains_cache *cache)\n> +\t\t\t\t\t struct contains_cache *cache,\n> +\t\t\t\t\t uint32_t cutoff)\n> {\n> \tenum contains_result *cached = contains_cache_at(cache, candidate);\n> \n> @@ -1603,6 +1605,10 @@ static enum contains_result contains_test(struct commit *candidate,\n> \n> \t/* Otherwise, we don't know; prepare to recurse */\n> \tparse_commit_or_die(candidate);\n> +\n> +\tif (candidate->generation < cutoff)\n> +\t\treturn CONTAINS_NO;\n> +\n\nWe use here weaker negative-cut criteria, which has the advantage of\nsimply automatic handling of special values: _INFINITY, _MAX, _ZERO.\n\nStronger version:\n\n if A != B and A ---> B, then gen(A) > gen(B)\n\n if gen(A) <= gen(B) and A != B, then A -/-> B\n\nWeaker version:\n\n if gen(A) < gen(B), then A -/-> B\n\nIf commit-graph feature is not available, then all generation numbers\nwould be _INFINITY, and cutoff would also be _INFINITY - which means\nthis operation is practically no-op. One memory access (probably from\ncache) and one comparison is very cheap.\n\nAll right.\n\n> \treturn CONTAINS_UNKNOWN;\n> }\n> \n> @@ -1618,8 +1624,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> \t\t\t\t\t struct contains_cache *cache)\n> {\n> \tstruct contains_stack contains_stack = { 0, 0, NULL };\n> -\tenum contains_result result = contains_test(candidate, want, cache);\n> +\tenum contains_result result;\n> +\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n> +\tconst struct commit_list *p;\n> +\n> +\tfor (p = want; p; p = p->next) {\n> +\t\tstruct commit *c = p->item;\n> +\t\tload_commit_graph_info(c);\n> +\t\tif (c->generation < cutoff)\n> +\t\t\tcutoff = c->generation;\n> +\t}\n\nFor each in wants, load generation numbers if needed and find lowest\none. Anything lower cannot reach any of wants. All right.\n\nIf commit-graph feature is not available, this is practically no-op. It\nis fast, as it only accesses memory - it does not access disk, nor do it\nneeds to do any decompression, un-deltafication or parsing.\n\nAll right.\n\n> \n> +\tresult = contains_test(candidate, want, cache, cutoff);\n> \tif (result != CONTAINS_UNKNOWN)\n> \t\treturn result;\n> \n> @@ -1637,7 +1653,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> \t\t * If we just popped the stack, parents->item has been marked,\n> \t\t * therefore contains_test will return a meaningful yes/no.\n> \t\t */\n> -\t\telse switch (contains_test(parents->item, want, cache)) {\n> +\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n> \t\tcase CONTAINS_YES:\n> \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n> \t\t\tcontains_stack.nr--;\n> @@ -1651,7 +1667,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n> \t\t}\n> \t}\n> \tfree(contains_stack.contains_stack);\n> -\treturn contains_test(candidate, want, cache);\n> +\treturn contains_test(candidate, want, cache, cutoff);\n\nThose two just update callsite to new signatore. All right.\n\n> }\n> \n> static int commit_contains(struct ref_filter *filter, struct commit *commit,\n"},{"id":"346180","messageId":"86in88hbfx.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-8-dstolee@microsoft.com","subject":"Re: [PATCH v4 07/10] commit: use generation numbers for in_merge_bases()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-30T17:05:22Z","receivedAt":"2018-04-30T17:05:31Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The containment algorithm for 'git branch --contains' is different\n> from that for 'git tag --contains' in that it uses is_descendant_of()\n> instead of contains_tag_algo(). The expensive portion of the branch\n> algorithm is computing merge bases.\n>\n> When a commit-graph file exists with generation numbers computed,\n> we can avoid this merge-base calculation when the target commit has\n> a larger generation number than the initial commits.\n\nRight.\n\n>\n> Performance tests were run on a copy of the Linux repository where\n> HEAD is contained in v4.13 but no earlier tag. Also, all tags were\n> copied to branches and 'git branch --contains' was tested:\n\nI guess that it is equivalent of 'git tag --contains' setup from\nprevious commit, just for 'git branch --contains', isn't it?\n\n>\n> Before: 60.0s\n> After: 0.4s\n> Rel %: -99.3%\n\nVery nice.\n\nSidenote: an alternative to using \"Rel %\" of -99.3% (which is calculated\nas (before-after)/before) would be to use \"Speedup\" of 150 x (calculated\nas before/after). One one hand it might be more readable, on the other\nhand it might be a bit misleading.\n\nYet another alternative would be to use a chart like the following:\n\n time Before After\n Before 60.0s -- -99.3%\n After 0.4s +149% --\n\nAnyway, consistency in presentation in patch series is good. So I am\nfor keeping your notation thorough the series.\n\n>\n> Reported-by: Jeff King \n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 9 ++++++++-\n> 1 file changed, 8 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit.c b/commit.c\n> index 39a3749abd..7bb007f56a 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -1056,12 +1056,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n\nLet's give it a bit more context:\n\n /*\n * Is \"commit\" an ancestor of one of the \"references\"?\n */\n int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit **reference)\n> {\n> \tstruct commit_list *bases;\n> \tint ret = 0, i;\n> +\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n> \n> \tif (parse_commit(commit))\n> \t\treturn ret;\n> -\tfor (i = 0; i < nr_reference; i++)\n> +\tfor (i = 0; i < nr_reference; i++) {\n> \t\tif (parse_commit(reference[i]))\n> \t\t\treturn ret;\n\nWe use parse_commit(), so there is no need for calling\nload_commit_graph_info(), like in previous patch.\n\nAll right.\n\n> +\t\tif (min_generation > reference[i]->generation)\n\nAt first glance, I thought it was wrong; but it is the same as the\nfollowing, it is just a matter of taste (which feels more natural):\n\n +\t\tif (reference[i]->generation < min_generation)\n\n> +\t\t\tmin_generation = reference[i]->generation;\n> +\t}\n> +\n> +\tif (commit->generation > min_generation)\n> +\t\treturn ret;\n\nAll right, using weak version of generation numbers based negative-cut\nnicely handles automatically all corner-cases, including the case where\ncommit-graaph feature is turned off.\n\nIf commit-graph feature is not available, it costs only few memory\naccess and few comparisons than before, and performance is dominated by\nsomething else anyway. Negligible and possibly unnoticeable change, I\nguess. Good.\n\n> \n> \tbases = paint_down_to_common(commit, nr_reference, reference);\n> \tif (commit->object.flags & PARENT2)\n"},{"id":"346208","messageId":"864ljsgwx1.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-9-dstolee@microsoft.com","subject":"Re: [PATCH v4 08/10] commit: add short-circuit to paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-30T22:19:06Z","receivedAt":"2018-04-30T22:19:16Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> When running 'git branch --contains', the in_merge_bases_many()\n> method calls paint_down_to_common() to discover if a specific\n> commit is reachable from a set of branches. Commits with lower\n> generation number are not needed to correctly answer the\n> containment query of in_merge_bases_many().\n>\n> Add a new parameter, min_generation, to paint_down_to_common() that\n> prevents walking commits with generation number strictly less than\n> min_generation. If 0 is given, then there is no functional change.\n\nThis is thanks to the fact that generation numbers start at zero (as\nspecial case, though, with _ZERO), and we use strict inequality to avoid\nhandling _ZERO etc. in a special way.\n\nAs you wrote in response in previous version of this series, because\npaint_down_to_common() is file-local, there is no need to come up with\nsymbolic name for GENERATION_NO_CUTOFF case.\n\nAll right.\n\n>\n> For in_merge_bases_many(), we can pass commit->generation as the\n> cutoff, and this saves time during 'git branch --contains' queries\n> that would otherwise walk \"around\" the commit we are inspecting.\n\nAll right, and when using paint_down_to_common() to actually find merge\nbases, and not only check for containment, we cannot use cutoff.\nTherefore at least one call site needs to run it without functional\nchange... which we can do. Good.\n\n>\n> For a copy of the Linux repository, where HEAD is checked out at\n> v4.13~100, we get the following performance improvement for\n> 'git branch --contains' over the previous commit:\n>\n> Before: 0.21s\n> After: 0.13s\n> Rel %: -38%\n\nNice.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 18 ++++++++++++++----\n> 1 file changed, 14 insertions(+), 4 deletions(-)\n\nLet me reorder chunks a bit to make it easier to review.\n\n>\n> diff --git a/commit.c b/commit.c\n> index 7bb007f56a..e2e16ea1a7 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -1070,7 +1080,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n> \tif (commit->generation > min_generation)\n> \t\treturn ret;\n> \n> -\tbases = paint_down_to_common(commit, nr_reference, reference);\n> +\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n> \tif (commit->object.flags & PARENT2)\n> \t\tret = 1;\n> \tclear_commit_marks(commit, all_flags);\n> @@ -808,11 +808,14 @@ static int queue_has_nonstale(struct prio_queue *queue)\n> }\n> \n> /* all input commits in one and twos[] must have been parsed! */\n> -static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n> +static struct commit_list *paint_down_to_common(struct commit *one, int n,\n> +\t\t\t\t\t\tstruct commit **twos,\n> +\t\t\t\t\t\tint min_generation)\n> {\n> \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n> \tstruct commit_list *result = NULL;\n> \tint i;\n> +\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n> \n> \tone->object.flags |= PARENT1;\n> \tif (!n) {\n> @@ -831,6 +834,13 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n> \t\tstruct commit_list *parents;\n> \t\tint flags;\n> \n> +\t\tif (commit->generation > last_gen)\n> +\t\t\tBUG(\"bad generation skip\");\n> +\t\tlast_gen = commit->generation;\n\nShouldn't we provide more information about where the problem is to the\nuser, to make it easier to debug the repository / commit-graph data?\n\nGood to have this sanity check here.\n\n> +\n> +\t\tif (commit->generation < min_generation)\n> +\t\t\tbreak;\n\nSo the reasoning for this, as far as I understand, is the following.\nPlease correct me if I am wrong.\n\nThe callsite with non-zero min_generation, in_merge_bases_many(), tries\nto find out if \"commit\" is an ancestor of one of the \"references\". At\nleast one of \"references\" is above \"commit\", so in_merge_bases_many()\nuses paint_down_to_common() - but is interested only if \"commit\" was\npainted as reachable from one of \"references\".\n\nThus we can interrupt the walk if we know that none of [considered]\ncommits in the queue can reach \"commit\"/\"one\" - as if they were all\nSTALE.\n\nThe search is done using priority queue (a bit like in Dijkstra\nalgorithm), with newer commits - with larger generation numbers -\nconsidered first. Thus if current commit has generation number less\nthan min_generation cutoff, i.e. if it is below \"commit\", then all\nremaining commits in the queue are below cutoff.\n\nGood.\n\n> +\n> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n> \t\tif (flags == (PARENT1 | PARENT2)) {\n> \t\t\tif (!(commit->object.flags & RESULT)) {\n> @@ -879,7 +889,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n> \t\t\treturn NULL;\n> \t}\n> \n> -\tlist = paint_down_to_common(one, n, twos);\n> +\tlist = paint_down_to_common(one, n, twos, 0);\n\nWhen calculating merge bases there is no such possibility of an early\nreturn due to generation number cutoff. All right then.\n\n> \n> \twhile (list) {\n> \t\tstruct commit *commit = pop_commit(&list);\n> @@ -946,7 +956,7 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\t\tfilled_index[filled] = j;\n> \t\t\twork[filled++] = array[j];\n> \t\t}\n> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n\nHere we are interested not only if \"one\"/array[i] is reachable from\n\"twos\"/work, but also if \"twos\" is reachable from \"one\". Simple cutoff\nonly works in one way, though I wonder if we couldn't use cutoff being\nminimum generation number of \"one\" and \"twos\" together.\n\nBut that may be left for a separate commit (after checking that the\nabove is correct).\n\nNot as simple and obvious as paint_down_to_common() used in\nin_merge_bases_any(), so it is all right.\n\n> \t\tif (array[i]->object.flags & PARENT2)\n> \t\t\tredundant[i] = 1;\n> \t\tfor (j = 0; j < filled; j++)\n"},{"id":"346212","messageId":"86po2gfgpp.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-10-dstolee@microsoft.com","subject":"Re: [PATCH v4 09/10] merge: check config before loading commits","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-30T22:54:26Z","receivedAt":"2018-04-30T22:54:36Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Now that we use generation numbers from the commit-graph, we must\n> ensure that all commits that exist in the commit-graph are loaded\n> from that file instead of from the object database. Since the\n> commit-graph file is only checked if core.commitGraph is true, we\n> must check the default config before we load any commits.\n>\n> In the merge builtin, the config was checked after loading the HEAD\n> commit. This was due to the use of the global 'branch' when checking\n> merge-specific config settings.\n>\n> Move the config load to be between the initialization of 'branch' and\n> the commit lookup.\n\nSidenote: I wonder why reading config was postponed to later in the\ncommand lifetime... I guess it was to avoid having to read config if\nHEAD was invalid.\n\n>\n> Without this change, a fast-forward merge would hit a BUG(\"bad\n> generation skip\") statement in commit.c during paint_down_to_common().\n> This is because the HEAD commit would be loaded with \"infinite\"\n> generation but then reached by commits with \"finite\" generation\n> numbers.\n\nI guess this is because we avoid re-parsing objects at all costs; we\nwant to avoid re-reading commit graph too.\n\n>\n> Add a test to t5318-commit-graph.sh that exercises this code path to\n> prevent a regression.\n\nI would prefer if this commit was put earlier in the series, to avoid\nhaving broken Git (and thus a possibility of problems when bisecting) in\nbetween those two commits.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/merge.c | 7 ++++---\n> t/t5318-commit-graph.sh | 9 +++++++++\n> 2 files changed, 13 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/merge.c b/builtin/merge.c\n> index 5e5e4497e3..b819756946 100644\n> --- a/builtin/merge.c\n> +++ b/builtin/merge.c\n> @@ -1148,14 +1148,15 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n> \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n> \tif (branch)\n> \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n> +\n> +\tinit_diff_ui_defaults();\n> +\tgit_config(git_merge_config, NULL);\n> +\n> \tif (!branch || is_null_oid(&head_oid))\n> \t\thead_commit = NULL;\n> \telse\n> \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n> \n> -\tinit_diff_ui_defaults();\n> -\tgit_config(git_merge_config, NULL);\n> -\n\nGood.\n\n> \tif (branch_mergeoptions)\n> \t\tparse_branch_merge_options(branch_mergeoptions);\n> \targc = parse_options(argc, argv, prefix, builtin_merge_options,\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index a380419b65..77d85aefe7 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n> graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n> graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n> \n> +test_expect_success 'perform fast-forward merge in full repo' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit checkout -b merge-5-to-8 commits/5 &&\n> +\tgit merge commits/8 &&\n> +\tgit show-ref -s merge-5-to-8 >output &&\n> +\tgit show-ref -s commits/8 >expect &&\n> +\ttest_cmp expect output\n> +'\n\nAll right. (though I wonder if this tests catches all problems where\nBUG(\"bad generation skip\") could have been encountered.\n\n> +\n> test_done\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346214","messageId":"86d0ygfext.fsf@gmail.com","threadId":"48212","inReplyTo":"20180425143735.240183-11-dstolee@microsoft.com","subject":"Re: [PATCH v4 10/10] commit-graph.txt: update design document","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-30T23:32:46Z","receivedAt":"2018-04-30T23:32:55Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> We now calculate generation numbers in the commit-graph file and use\n> them in paint_down_to_common().\n>\n> Expand the section on generation numbers to discuss how the three\n> special generation numbers GENERATION_NUMBER_INFINITY, _ZERO, and\n> _MAX interact with other generation numbers.\n>\n> Signed-off-by: Derrick Stolee \n\nLooks good.\n\n> ---\n> Documentation/technical/commit-graph.txt | 30 +++++++++++++++++++-----\n> 1 file changed, 24 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index 0550c6d0dc..d9f2713efa 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -77,6 +77,29 @@ in the commit graph. We can treat these commits as having \"infinite\"\n> generation number and walk until reaching commits with known generation\n> number.\n> \n> +We use the macro GENERATION_NUMBER_INFINITY = 0xFFFFFFFF to mark commits not\n> +in the commit-graph file. If a commit-graph file was written by a version\n> +of Git that did not compute generation numbers, then those commits will\n> +have generation number represented by the macro GENERATION_NUMBER_ZERO = 0.\n> +\n> +Since the commit-graph file is closed under reachability, we can guarantee\n> +the following weaker condition on all commits:\n> +\n> + If A and B are commits with generation numbers N amd M, respectively,\n> + and N < M, then A cannot reach B.\n> +\n> +Note how the strict inequality differs from the inequality when we have\n> +fully-computed generation numbers. Using strict inequality may result in\n> +walking a few extra commits,\n\nThe linux kernel commit graph has maximum of 513 commits sharing the\nsame generation number, but is is 5.43 commits sharing the same\ngeneration number on average, with standard deviation 10.70; median is\neven lower: it is 2, with 5.35 median absolute deviation (MAD).\n\nSo on average it would be a few extra commits. Right.\n\n> but the simplicity in dealing with commits\n> +with generation number *_INFINITY or *_ZERO is valuable.\n\nAs I wrote before, handling those corner cases in more complicated, but\nnot that complicated. We could simply use stronger condition if both\ngeneration numbers are ordinary generation numbers, and weaker condition\nwhen at least one generation number has one of those special values.\n\n> +\n> +We use the macro GENERATION_NUMBER_MAX = 0x3FFFFFFF to for commits whose\n> +generation numbers are computed to be at least this value. We limit at\n> +this value since it is the largest value that can be stored in the\n> +commit-graph file using the 30 bits available to generation numbers. This\n> +presents another case where a commit can have generation number equal to\n> +that of a parent.\n\nOrdinary generation numbers, where stronger condition holds, are those\nbetween GENERATION_NUMBER_ZERO < gen(C) < GENERATION_NUMBER_MAX.\n\n> +\n> Design Details\n> --------------\n> \n> @@ -98,17 +121,12 @@ Future Work\n> - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n> necessary for integration with fsck.\n> \n> -- The file format includes room for precomputed generation numbers. These\n> - are not currently computed, so all generation numbers will be marked as\n> - 0 (or \"uncomputed\"). A later patch will include this calculation.\n> -\n\nGood.\n\n> - After computing and storing generation numbers, we must make graph\n> walks aware of generation numbers to gain the performance benefits they\n> enable. This will mostly be accomplished by swapping a commit-date-ordered\n> priority queue with one ordered by generation number. The following\n> - operations are important candidates:\n> + operation is an important candidate:\n> \n> - - paint_down_to_common()\n> - 'log --topo-order'\n\nAnother possible candidates:\n\n - remove_redundant() - see comment in previous patch\n - still_interesting() - where Git uses date slop to stop walking\n too far\n\n> \n> - Currently, parse_commit_gently() requires filling in the root tree\n\nOne important issue left is handling features that change view of\nproject history, and their interaction with commit-graph feature.\n\nWhat would happen, if we turn on commit-graph feature, generate commit\ngraph file, and then:\n\n * use graft file or remove graft entries to cut history, or remove cut\n or join two [independent] histories.\n * use git-replace mechanims to do the same\n * in shallow clone, deepen or shorten the clone\n\nWhat would happen if without re-generating commit-graph file (assuming\ntha Git wouldn't do it for us), we run some feature that makes use of\ncommit-graph data:\n\n - git branch --contains\n - git tag --contains\n - git rev-list A..B\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346247","messageId":"3e3ae7e4-9efd-3046-c0c2-7b3bf12d0d81@gmail.com","threadId":"48212","inReplyTo":"864ljsgwx1.fsf@gmail.com","subject":"Re: [PATCH v4 08/10] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-01T11:47:01Z","receivedAt":"2018-05-01T11:47:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/30/2018 6:19 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> When running 'git branch --contains', the in_merge_bases_many()\n>> method calls paint_down_to_common() to discover if a specific\n>> commit is reachable from a set of branches. Commits with lower\n>> generation number are not needed to correctly answer the\n>> containment query of in_merge_bases_many().\n>>\n>> Add a new parameter, min_generation, to paint_down_to_common() that\n>> prevents walking commits with generation number strictly less than\n>> min_generation. If 0 is given, then there is no functional change.\n> This is thanks to the fact that generation numbers start at zero (as\n> special case, though, with _ZERO), and we use strict inequality to avoid\n> handling _ZERO etc. in a special way.\n>\n> As you wrote in response in previous version of this series, because\n> paint_down_to_common() is file-local, there is no need to come up with\n> symbolic name for GENERATION_NO_CUTOFF case.\n>\n> All right.\n>\n>> For in_merge_bases_many(), we can pass commit->generation as the\n>> cutoff, and this saves time during 'git branch --contains' queries\n>> that would otherwise walk \"around\" the commit we are inspecting.\n> All right, and when using paint_down_to_common() to actually find merge\n> bases, and not only check for containment, we cannot use cutoff.\n> Therefore at least one call site needs to run it without functional\n> change... which we can do. Good.\n>\n>> For a copy of the Linux repository, where HEAD is checked out at\n>> v4.13~100, we get the following performance improvement for\n>> 'git branch --contains' over the previous commit:\n>>\n>> Before: 0.21s\n>> After: 0.13s\n>> Rel %: -38%\n> Nice.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit.c | 18 ++++++++++++++----\n>> 1 file changed, 14 insertions(+), 4 deletions(-)\n> Let me reorder chunks a bit to make it easier to review.\n>\n>> diff --git a/commit.c b/commit.c\n>> index 7bb007f56a..e2e16ea1a7 100644\n>> --- a/commit.c\n>> +++ b/commit.c\n>> @@ -1070,7 +1080,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n>> \tif (commit->generation > min_generation)\n>> \t\treturn ret;\n>> \n>> -\tbases = paint_down_to_common(commit, nr_reference, reference);\n>> +\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n>> \tif (commit->object.flags & PARENT2)\n>> \t\tret = 1;\n>> \tclear_commit_marks(commit, all_flags);\n>> @@ -808,11 +808,14 @@ static int queue_has_nonstale(struct prio_queue *queue)\n>> }\n>> \n>> /* all input commits in one and twos[] must have been parsed! */\n>> -static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n>> +static struct commit_list *paint_down_to_common(struct commit *one, int n,\n>> +\t\t\t\t\t\tstruct commit **twos,\n>> +\t\t\t\t\t\tint min_generation)\n>> {\n>> \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n>> \tstruct commit_list *result = NULL;\n>> \tint i;\n>> +\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n>> \n>> \tone->object.flags |= PARENT1;\n>> \tif (!n) {\n>> @@ -831,6 +834,13 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n>> \t\tstruct commit_list *parents;\n>> \t\tint flags;\n>> \n>> +\t\tif (commit->generation > last_gen)\n>> +\t\t\tBUG(\"bad generation skip\");\n>> +\t\tlast_gen = commit->generation;\n> Shouldn't we provide more information about where the problem is to the\n> user, to make it easier to debug the repository / commit-graph data?\n>\n> Good to have this sanity check here.\n\nThis BUG() _should_ only be seen by developers who add callers which do \nnot load commits from the commit-graph file. There is a chance that \nthere are cases not covered by this patch and the added tests, though. \nHopefully we catch them all by dogfooding the feature before turning it \non by default.\n\nI can add the following to help debug these bad situations:\n\n+\t\t\tBUG(\"bad generation skip %d > %d at %s\",\n+\t\t\t commit->generation, last_gen,\n+\t\t\t oid_to_hex(&commit->object.oid));\n\n\n\n>\n>> +\n>> +\t\tif (commit->generation < min_generation)\n>> +\t\t\tbreak;\n> So the reasoning for this, as far as I understand, is the following.\n> Please correct me if I am wrong.\n>\n> The callsite with non-zero min_generation, in_merge_bases_many(), tries\n> to find out if \"commit\" is an ancestor of one of the \"references\". At\n> least one of \"references\" is above \"commit\", so in_merge_bases_many()\n> uses paint_down_to_common() - but is interested only if \"commit\" was\n> painted as reachable from one of \"references\".\n>\n> Thus we can interrupt the walk if we know that none of [considered]\n> commits in the queue can reach \"commit\"/\"one\" - as if they were all\n> STALE.\n>\n> The search is done using priority queue (a bit like in Dijkstra\n> algorithm), with newer commits - with larger generation numbers -\n> considered first. Thus if current commit has generation number less\n> than min_generation cutoff, i.e. if it is below \"commit\", then all\n> remaining commits in the queue are below cutoff.\n>\n> Good.\n>\n>> +\n>> \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n>> \t\tif (flags == (PARENT1 | PARENT2)) {\n>> \t\t\tif (!(commit->object.flags & RESULT)) {\n>> @@ -879,7 +889,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n>> \t\t\treturn NULL;\n>> \t}\n>> \n>> -\tlist = paint_down_to_common(one, n, twos);\n>> +\tlist = paint_down_to_common(one, n, twos, 0);\n> When calculating merge bases there is no such possibility of an early\n> return due to generation number cutoff. All right then.\n>\n>> \n>> \twhile (list) {\n>> \t\tstruct commit *commit = pop_commit(&list);\n>> @@ -946,7 +956,7 @@ static int remove_redundant(struct commit **array, int cnt)\n>> \t\t\tfilled_index[filled] = j;\n>> \t\t\twork[filled++] = array[j];\n>> \t\t}\n>> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n>> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n> Here we are interested not only if \"one\"/array[i] is reachable from\n> \"twos\"/work, but also if \"twos\" is reachable from \"one\". Simple cutoff\n> only works in one way, though I wonder if we couldn't use cutoff being\n> minimum generation number of \"one\" and \"twos\" together.\n>\n> But that may be left for a separate commit (after checking that the\n> above is correct).\n>\n> Not as simple and obvious as paint_down_to_common() used in\n> in_merge_bases_any(), so it is all right.\n\nThanks for reporting this. Since we are only concerned about \nreachability in this method, it is a good candidate to use \nmin_generation. It is also subtle enough that we should leave it as a \nseparate commit. Also, we can measure performance improvements \nseparately, as I will mention in my commit message (but I'll copy it here):\n\n     For a copy of the Linux repository, we measured the following\n     performance improvements:\n\n     git merge-base v3.3 v4.5\n\n     Before: 234 ms\n      After: 208 ms\n      Rel %: -11%\n\n     git merge-base v4.3 v4.5\n\n     Before: 102 ms\n      After:  83 ms\n      Rel %: -19%\n\n     The experiments above were chosen to demonstrate that we are\n     improving the filtering of the merge-base set. In the first\n     example, more time is spent walking the history to find the\n     set of merge bases before the remove_redundant() call. The\n     starting commits are closer together in the second example,\n     therefore more time is spent in remove_redundant(). The relative\n     change in performance differs as expected.\n\n\n>\n>> \t\tif (array[i]->object.flags & PARENT2)\n>> \t\t\tredundant[i] = 1;\n>> \t\tfor (j = 0; j < filled; j++)\n\n"},{"id":"346248","messageId":"214dea35-a89c-4115-a51a-5110c4e53314@gmail.com","threadId":"48212","inReplyTo":"86po2gfgpp.fsf@gmail.com","subject":"Re: [PATCH v4 09/10] merge: check config before loading commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-01T11:52:26Z","receivedAt":"2018-05-01T11:52:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/30/2018 6:54 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> Now that we use generation numbers from the commit-graph, we must\n>> ensure that all commits that exist in the commit-graph are loaded\n>> from that file instead of from the object database. Since the\n>> commit-graph file is only checked if core.commitGraph is true, we\n>> must check the default config before we load any commits.\n>>\n>> In the merge builtin, the config was checked after loading the HEAD\n>> commit. This was due to the use of the global 'branch' when checking\n>> merge-specific config settings.\n>>\n>> Move the config load to be between the initialization of 'branch' and\n>> the commit lookup.\n> Sidenote: I wonder why reading config was postponed to later in the\n> command lifetime... I guess it was to avoid having to read config if\n> HEAD was invalid.\n\nThe 'branch' does need to be loaded before the call to git_config (as I \nfound out after moving the config call too early), so I suppose it was \nnatural to pair that with resolving head_commit.\n\n>\n>> Without this change, a fast-forward merge would hit a BUG(\"bad\n>> generation skip\") statement in commit.c during paint_down_to_common().\n>> This is because the HEAD commit would be loaded with \"infinite\"\n>> generation but then reached by commits with \"finite\" generation\n>> numbers.\n> I guess this is because we avoid re-parsing objects at all costs; we\n> want to avoid re-reading commit graph too.\n>\n>> Add a test to t5318-commit-graph.sh that exercises this code path to\n>> prevent a regression.\n> I would prefer if this commit was put earlier in the series, to avoid\n> having broken Git (and thus a possibility of problems when bisecting) in\n> between those two commits.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/merge.c | 7 ++++---\n>> t/t5318-commit-graph.sh | 9 +++++++++\n>> 2 files changed, 13 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/builtin/merge.c b/builtin/merge.c\n>> index 5e5e4497e3..b819756946 100644\n>> --- a/builtin/merge.c\n>> +++ b/builtin/merge.c\n>> @@ -1148,14 +1148,15 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n>> \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n>> \tif (branch)\n>> \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n>> +\n>> +\tinit_diff_ui_defaults();\n>> +\tgit_config(git_merge_config, NULL);\n>> +\n>> \tif (!branch || is_null_oid(&head_oid))\n>> \t\thead_commit = NULL;\n>> \telse\n>> \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n>> \n>> -\tinit_diff_ui_defaults();\n>> -\tgit_config(git_merge_config, NULL);\n>> -\n> Good.\n>\n>> \tif (branch_mergeoptions)\n>> \t\tparse_branch_merge_options(branch_mergeoptions);\n>> \targc = parse_options(argc, argv, prefix, builtin_merge_options,\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index a380419b65..77d85aefe7 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n>> graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n>> graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n>> \n>> +test_expect_success 'perform fast-forward merge in full repo' '\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\tgit checkout -b merge-5-to-8 commits/5 &&\n>> +\tgit merge commits/8 &&\n>> +\tgit show-ref -s merge-5-to-8 >output &&\n>> +\tgit show-ref -s commits/8 >expect &&\n>> +\ttest_cmp expect output\n>> +'\n> All right. (though I wonder if this tests catches all problems where\n> BUG(\"bad generation skip\") could have been encountered.\n\nWe will never know until we have this series running in the wild (and \neven then, some features are very obscure) and enough people turn on the \nconfig setting.\n\nOne goal of the \"fsck and gc\" series is to get this feature running \nduring the rest of the test suite as much as possible, so we can get \nadditional coverage. Also to get more experience from the community \ndogfooding the feature.\n\n>\n>> +\n>> test_done\n> Best,\n\n"},{"id":"346250","messageId":"2ea2c1d0-e2ad-2051-ba98-1584fcb51a46@gmail.com","threadId":"48212","inReplyTo":"86d0ygfext.fsf@gmail.com","subject":"Re: [PATCH v4 10/10] commit-graph.txt: update design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-01T12:00:51Z","receivedAt":"2018-05-01T12:00:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/30/2018 7:32 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> We now calculate generation numbers in the commit-graph file and use\n>> them in paint_down_to_common().\n>>\n>> Expand the section on generation numbers to discuss how the three\n>> special generation numbers GENERATION_NUMBER_INFINITY, _ZERO, and\n>> _MAX interact with other generation numbers.\n>>\n>> Signed-off-by: Derrick Stolee \n> Looks good.\n>\n>> ---\n>> Documentation/technical/commit-graph.txt | 30 +++++++++++++++++++-----\n>> 1 file changed, 24 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n>> index 0550c6d0dc..d9f2713efa 100644\n>> --- a/Documentation/technical/commit-graph.txt\n>> +++ b/Documentation/technical/commit-graph.txt\n>> @@ -77,6 +77,29 @@ in the commit graph. We can treat these commits as having \"infinite\"\n>> generation number and walk until reaching commits with known generation\n>> number.\n>> \n>> +We use the macro GENERATION_NUMBER_INFINITY = 0xFFFFFFFF to mark commits not\n>> +in the commit-graph file. If a commit-graph file was written by a version\n>> +of Git that did not compute generation numbers, then those commits will\n>> +have generation number represented by the macro GENERATION_NUMBER_ZERO = 0.\n>> +\n>> +Since the commit-graph file is closed under reachability, we can guarantee\n>> +the following weaker condition on all commits:\n>> +\n>> + If A and B are commits with generation numbers N amd M, respectively,\n>> + and N < M, then A cannot reach B.\n>> +\n>> +Note how the strict inequality differs from the inequality when we have\n>> +fully-computed generation numbers. Using strict inequality may result in\n>> +walking a few extra commits,\n> The linux kernel commit graph has maximum of 513 commits sharing the\n> same generation number, but is is 5.43 commits sharing the same\n> generation number on average, with standard deviation 10.70; median is\n> even lower: it is 2, with 5.35 median absolute deviation (MAD).\n>\n> So on average it would be a few extra commits. Right.\n>\n>> but the simplicity in dealing with commits\n>> +with generation number *_INFINITY or *_ZERO is valuable.\n> As I wrote before, handling those corner cases in more complicated, but\n> not that complicated. We could simply use stronger condition if both\n> generation numbers are ordinary generation numbers, and weaker condition\n> when at least one generation number has one of those special values.\n>\n>> +\n>> +We use the macro GENERATION_NUMBER_MAX = 0x3FFFFFFF to for commits whose\n>> +generation numbers are computed to be at least this value. We limit at\n>> +this value since it is the largest value that can be stored in the\n>> +commit-graph file using the 30 bits available to generation numbers. This\n>> +presents another case where a commit can have generation number equal to\n>> +that of a parent.\n> Ordinary generation numbers, where stronger condition holds, are those\n> between GENERATION_NUMBER_ZERO < gen(C) < GENERATION_NUMBER_MAX.\n>\n>> +\n>> Design Details\n>> --------------\n>> \n>> @@ -98,17 +121,12 @@ Future Work\n>> - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n>> necessary for integration with fsck.\n>> \n>> -- The file format includes room for precomputed generation numbers. These\n>> - are not currently computed, so all generation numbers will be marked as\n>> - 0 (or \"uncomputed\"). A later patch will include this calculation.\n>> -\n> Good.\n>\n>> - After computing and storing generation numbers, we must make graph\n>> walks aware of generation numbers to gain the performance benefits they\n>> enable. This will mostly be accomplished by swapping a commit-date-ordered\n>> priority queue with one ordered by generation number. The following\n>> - operations are important candidates:\n>> + operation is an important candidate:\n>> \n>> - - paint_down_to_common()\n>> - 'log --topo-order'\n> Another possible candidates:\n>\n> - remove_redundant() - see comment in previous patch\n> - still_interesting() - where Git uses date slop to stop walking\n> too far\n\nremove_redundant() will be included in v5, thanks.\n\nInstead of \"still_interesting()\" I'll add \"git tag --merged\" as the \ncandidate to consider, as discussed in [1].\n\n[1] https://public-inbox.org/git/87fu3g67ry.fsf@lant.ki.iif.hu/t/#u\n     \"branch --contains / tag --merged inconsistency\"\n\n>\n>> \n>> - Currently, parse_commit_gently() requires filling in the root tree\n> One important issue left is handling features that change view of\n> project history, and their interaction with commit-graph feature.\n>\n> What would happen, if we turn on commit-graph feature, generate commit\n> graph file, and then:\n>\n> * use graft file or remove graft entries to cut history, or remove cut\n> or join two [independent] histories.\n> * use git-replace mechanims to do the same\n> * in shallow clone, deepen or shorten the clone\n>\n> What would happen if without re-generating commit-graph file (assuming\n> tha Git wouldn't do it for us), we run some feature that makes use of\n> commit-graph data:\n>\n> - git branch --contains\n> - git tag --contains\n> - git rev-list A..B\n>\n\nThe commit-graph is not supported in these scenarios (yet). grafts are \nspecifically mentioned in the future work section.\n\nI'm not particularly interested in supporting these features, so they \nare good venues for other contributors to get involved in the \ncommit-graph feature. Eventually, they will be blockers to making the \ncommit-graph feature a \"default\" feature. That is when I will pay \nattention to these situations. For now, a user must opt-in to having a \ncommit-graph file (and that same user has possibly opted in to these \nhistory modifying features).\n\nThanks,\n-Stolee\n"},{"id":"346264","messageId":"0bd1ffe0-c727-653a-46a3-f9d4ea17bec2@gmail.com","threadId":"48212","inReplyTo":"86r2myidmq.fsf@gmail.com","subject":"Re: [PATCH v4 03/10] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-01T12:10:20Z","receivedAt":"2018-05-01T12:10:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/29/2018 5:08 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> While preparing commits to be written into a commit-graph file, compute\n>> the generation numbers using a depth-first strategy.\n> Sidenote: for generation numbers it does not matter if we use\n> depth-first or breadth-first strategy, but it is more natural to use\n> depth-first search because generation numbers need post-order processing\n> (parents before child).\n>\n>> The only commits that are walked in this depth-first search are those\n>> without a precomputed generation number. Thus, computation time will be\n>> relative to the number of new commits to the commit-graph file.\n> A question: what happens if the existing commit graph is from older\n> version of git and has _ZERO for generation numbers?\n>\n> Answer: I see that we treat both _INFINITY (not in commit-graph) and\n> _ZERO (in commit graph but not computed) as not computed generation\n> numbers. All right.\n>\n>> If a computed generation number would exceed GENERATION_NUMBER_MAX, then\n>> use GENERATION_NUMBER_MAX instead.\n> All right, though I guess this would remain theoretical for a long\n> while.\n>\n> We don't have any way of testing this, at least not without recompiling\n> Git with lower value of GENERATION_NUMBER_MAX -- which means not\n> automatically, isn't it?\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 45 +++++++++++++++++++++++++++++++++++++++++++++\n>> 1 file changed, 45 insertions(+)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 9ad21c3ffb..047fa9fca5 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -439,6 +439,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n>> \t\telse\n>> \t\t\tpackedDate[0] = 0;\n>> \n>> +\t\tif ((*list)->generation != GENERATION_NUMBER_INFINITY)\n>> +\t\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n>> +\n> If we stumble upon commit marked as \"not in commit-graph\" while writing\n> commit graph, it is a BUG(), isn't it?\n>\n> (Problem noticed by Junio.)\n\nSince we are computing the values for all commits in the list, this \ncondition is not important and will be removed.\n\n>\n> It is a bit strange to me that the code uses get_be32 for reading, but\n> htonl for writing. Is Git tested on non little-endian machines, like\n> big-endian ppc64 or s390x, or on mixed-endian machines (or\n> selectable-endian machines with data endianness set to non\n> little-endian, like ia64)? If not, could we use for example openSUSE\n> Build Service (https://build.opensuse.org/) for this?\n\nSince we are packing two values into 64 bits, I am using htonl() here to \narrange the 30-bit generation number alongside the 34-bit commit date \nvalue, then writing with hashwrite(). The other 32-bit integers are \nwritten with hashwrite_be32() to avoid translating this data in-memory.\n\n>\n>> \t\tpackedDate[1] = htonl((*list)->date);\n>> \t\thashwrite(f, packedDate, 8);\n>> \n>> @@ -571,6 +574,46 @@ static void close_reachable(struct packed_oid_list *oids)\n>> \t}\n>> }\n>> \n>> +static void compute_generation_numbers(struct commit** commits,\n>> +\t\t\t\t int nr_commits)\n>> +{\n>> +\tint i;\n>> +\tstruct commit_list *list = NULL;\n> All right, commit_list will work as stack.\n>\n>> +\n>> +\tfor (i = 0; i < nr_commits; i++) {\n>> +\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n>> +\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n>> +\t\t\tcontinue;\n> All right, we consider _INFINITY and _SERO as not computed. If\n> generation number is computed (by 'recursion' or from commit graph), we\n> (re)use it. This means that generation number calculation is\n> incremental, as intended -- good.\n>\n>> +\n>> +\t\tcommit_list_insert(commits[i], &list);\n> Start depth-first walks from commits given.\n>\n>> +\t\twhile (list) {\n>> +\t\t\tstruct commit *current = list->item;\n>> +\t\t\tstruct commit_list *parent;\n>> +\t\t\tint all_parents_computed = 1;\n> Here all_parents_computed is a boolean flag. I see that it is easier to\n> start with assumption that all parents will have computed generation\n> numbers.\n>\n>> +\t\t\tuint32_t max_generation = 0;\n> The generation number value of 0 functions as sentinel; generation\n> numbers start from 1. Not that it matters much, as lowest possible\n> generation number is 1, and we could have started from that value.\n\nExcept that for a commit with no parents, we want it to receive \ngeneration number max_generation + 1 = 1, so this value of 0 is important.\n\n>\n>> +\n>> +\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n>> +\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n>> +\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n>> +\t\t\t\t\tall_parents_computed = 0;\n>> +\t\t\t\t\tcommit_list_insert(parent->item, &list);\n>> +\t\t\t\t\tbreak;\n> If some parent doesn't have generation number calculated, we add it to\n> stack (and break out of loop because it is depth-first walk), and mark\n> this situation. All right.\n>\n>> +\t\t\t\t} else if (parent->item->generation > max_generation) {\n>> +\t\t\t\t\tmax_generation = parent->item->generation;\n> Otherwise, update max_generation. All right.\n>\n>> +\t\t\t\t}\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (all_parents_computed) {\n>> +\t\t\t\tcurrent->generation = max_generation + 1;\n>> +\t\t\t\tpop_commit(&list);\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n>> +\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n> This conditional should be inside all_parents_computed test, for example\n> like this:\n>\n> +\t\t\tif (all_parents_computed) {\n> +\t\t\t\tcurrent->generation = max_generation + 1;\n> +\t\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n> +\t\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n> +\n> +\t\t\t\tpop_commit(&list);\n> +\t\t\t}\n>\n> (Noticed by Junio.)\n>\n> Sidenote: when we revisit the commit, returning from depth-first walk of\n> one of its parents, we calculate max_generation from scratch again.\n> This does not matter for performance, as it's just data access and\n> calculating maximum - any workaround to not restart those calculations\n> would take more time and memory. And it's simple.\n>\n>> +\t\t}\n>> +\t}\n>> +}\n>> +\n>> void write_commit_graph(const char *obj_dir,\n>> \t\t\tconst char **pack_indexes,\n>> \t\t\tint nr_packs,\n>> @@ -694,6 +737,8 @@ void write_commit_graph(const char *obj_dir,\n>> \tif (commits.nr >= GRAPH_PARENT_MISSING)\n>> \t\tdie(_(\"too many commits to write graph\"));\n>> \n>> +\tcompute_generation_numbers(commits.list, commits.nr);\n>> +\n> Nice and simple. All right.\n>\n> I guess that we do not pass \"struct packed_commit_list commits\" as\n> argument to compute_generation_numbers instead of \"struct commit**\n> commits.list\" and \"int commits.nr\" to compute_generation_numbers() to\n> keep the latter nice and generic?\n\nGood catch. There is no reason to not use packed_commit_list here.\n\n>\n>> \tgraph_name = get_commit_graph_filename(obj_dir);\n>> \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n> Best,\n\n"},{"id":"346266","messageId":"4273bd76-94c5-d3c8-7f4e-b4cdb4bc35eb@gmail.com","threadId":"48212","inReplyTo":"866049irrx.fsf@gmail.com","subject":"Re: [PATCH v4 05/10] commit-graph: always load commit-graph information","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-01T12:19:38Z","receivedAt":"2018-05-01T12:19:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/29/2018 6:14 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> Most code paths load commits using lookup_commit() and then\n>> parse_commit().\n> And this automatically loads commit graph if needed, thanks to changes\n> in parse_commit_gently(), which parse_commit() uses.\n>\n>> In some cases, including some branch lookups, the commit\n>> is parsed using parse_object_buffer() which side-steps parse_commit() in\n>> favor of parse_commit_buffer().\n> I guess the problem is that we cannot just add parse_commit_in_graph()\n> like we did in parse_commit_gently(), for some reason? Like for example\n> that parse_commit_gently() uses parse_commit_buffer() - which could have\n> been handled by moving parse_commit_in_graph() down the call chain from\n> parse_commit_gently() to parse_commit_buffer()... if not the fact that\n> check_commit() also uses parse_commit_buffer(), but it does not want to\n> load commit graph. Am I right?\n\nIf a caller uses parse_commit_buffer() directly, then we will guarantee \nthat all values in the struct commit that would be loaded from the \nbuffer are loaded from the buffer. This means we do NOT load the root \ntree id or commit date from the commit-graph file. We do still need to \nload the data that is not available in the buffer, such as graph_pos and \ngeneration.\n\n>\n>> With generation numbers in the commit-graph, we need to ensure that any\n>> commit that exists in the commit-graph file has its generation number\n>> loaded.\n> Is it generation number, or generation number and position in commit\n> graph?\n\nWe don't need to ensure the graph_pos (the commit will never be \nre-parsed, so we will not try to find it in the commit-graph file \nagain), but we DO need to ensure the generation (or our commit walks \nwill be incorrect). We get the graph_pos as a side-effect.\n\n>\n>> Create new load_commit_graph_info() method to fill in the information\n>> for a commit that exists only in the commit-graph file. Call it from\n>> parse_commit_buffer() after loading the other commit information from\n>> the given buffer. Only fill this information when specified by the\n>> 'check_graph' parameter.\n> I think this commit would be easier to review if it was split into pure\n> refactoring part (extracting fill_commit_graph_info() and\n> find_commit_in_graph()). On the other hand the refactoring was needed\n> to reduce code duplication betweem existing parse_commit_in_graph() and\n> new load_commit_graph_info() functions.\n>\n> I guess that the difference between parse_commit_in_graph() and\n> load_commit_graph_info() is that the former cares only about having just\n> enough information that is needed for parse_commit_gently() - and does\n> not load graph data if commit is parsed, while the latter is about\n> loading commit-graph data like generation numbers.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 45 ++++++++++++++++++++++++++++++---------------\n>> commit-graph.h | 8 ++++++++\n>> commit.c | 7 +++++--\n>> commit.h | 2 +-\n>> object.c | 2 +-\n>> sha1_file.c | 2 +-\n>> 6 files changed, 46 insertions(+), 20 deletions(-)\n> I wonder if it would be possible to add tests for this feature, for\n> example that commit-graph is read when it should (including those branch\n> lookups), and is not read when the feature should be disabled.\n>\n> But the only way to test it I can think of is a stupid one: create\n> invalid commit graph, and check that git fails as expected (trying to\n> read said malformed file), and does not fail if commit graph feature is\n> disabled.\n>\n> Let me reorder files (BTW, is there a way for Git to put *.h files\n> before *.c files in diff?) for easier review:\n>\n>> diff --git a/commit-graph.h b/commit-graph.h\n>> index 260a468e73..96cccb10f3 100644\n>> --- a/commit-graph.h\n>> +++ b/commit-graph.h\n>> @@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n>> */\n>> int parse_commit_in_graph(struct commit *item);\n>> \n>> +/*\n>> + * It is possible that we loaded commit contents from the commit buffer,\n>> + * but we also want to ensure the commit-graph content is correctly\n>> + * checked and filled. Fill the graph_pos and generation members of\n>> + * the given commit.\n>> + */\n>> +void load_commit_graph_info(struct commit *item);\n>> +\n>> struct tree *get_commit_tree_in_graph(const struct commit *c);\n>> \n>> struct commit_graph {\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 047fa9fca5..aebd242def 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -245,6 +245,12 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n>> \treturn &commit_list_insert(c, pptr)->next;\n>> }\n>> \n>> +static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n>> +{\n>> +\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n>> +\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n>> +}\n> The comment in the header file commit-graph.h talks about filling\n> graph_pos and generation members of the given commit, but I don't see\n> filling graph_pos member here.\n\nWe are missing the following line:\n\n+    item->graph_pos = pos;\n\nI will add it for v5. The equivalent line exists in fill_commit_in_graph().\n\n>\n> Sidenote: it is a tiny little bit strange to see symbolic constants like\n> GRAPH_DATA_WIDTH near using magic values such as 8 and 2.\n\nThere needs to be some boundary between abstraction and concreteness \nwhen dealing directly with a binary file format. GRAPH_DATA_WIDTH helps \nus navigate to the correct \"row\" in the chunk, while we use the \nconstants 8 and 2 to get the correct \"column\" out of that row.\n\n>\n>> +\n>> static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n>> {\n>> \tuint32_t edge_value;\n>> @@ -292,31 +298,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n>> \treturn 1;\n>> }\n>> \n>> +static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n>> +{\n>> +\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n>> +\t\t*pos = item->graph_pos;\n>> +\t\treturn 1;\n>> +\t} else {\n>> +\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n>> +\t}\n>> +}\n> Nice refactoring here.\n>\n>> +\n>> int parse_commit_in_graph(struct commit *item)\n>> {\n>> +\tuint32_t pos;\n>> +\n>> \tif (!core_commit_graph)\n>> \t\treturn 0;\n>> \tif (item->object.parsed)\n>> \t\treturn 1;\n>> -\n>> \tprepare_commit_graph();\n>> -\tif (commit_graph) {\n>> -\t\tuint32_t pos;\n>> -\t\tint found;\n>> -\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n>> -\t\t\tpos = item->graph_pos;\n>> -\t\t\tfound = 1;\n>> -\t\t} else {\n>> -\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n>> -\t\t}\n>> -\n>> -\t\tif (found)\n>> -\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n>> -\t}\n>> -\n>> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n>> +\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n>> \treturn 0;\n>> }\n>> \n>> +void load_commit_graph_info(struct commit *item)\n>> +{\n>> +\tuint32_t pos;\n>> +\tif (!core_commit_graph)\n>> +\t\treturn;\n>> +\tprepare_commit_graph();\n>> +\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n>> +\t\tfill_commit_graph_info(item, commit_graph, pos);\n>> +}\n> Similar functions, different goals (as the names imply).\n>\n>> +\n>> static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n>> {\n>> \tstruct object_id oid;\n>> diff --git a/commit.c b/commit.c\n>> index 4d00b0a1d6..39a3749abd 100644\n>> --- a/commit.c\n>> +++ b/commit.c\n>> @@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n>> \treturn ret;\n>> }\n>> \n>> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n>> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n>> {\n>> \tconst char *tail = buffer;\n>> \tconst char *bufptr = buffer;\n>> @@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n>> \t}\n>> \titem->date = parse_commit_date(bufptr, tail);\n>> \n>> +\tif (check_graph)\n>> +\t\tload_commit_graph_info(item);\n>> +\n> All right, read commit-graph specific data after parsing commit itself.\n> It is at the end because commit object needs to be parsed sequentially,\n> and it includes more info that is contained in commit-graph CDAT+EDGE\n> data.\n>\n>> \treturn 0;\n>> }\n>> \n>> @@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n>> \t\treturn error(\"Object %s not a commit\",\n>> \t\t\t oid_to_hex(&item->object.oid));\n>> \t}\n>> -\tret = parse_commit_buffer(item, buffer, size);\n>> +\tret = parse_commit_buffer(item, buffer, size, 0);\n> The parse_commit_gently() contract is that it provides only bare minimum\n> of information, from commit-graph if possible, and does read object from\n> disk and parses it only when it could not avoid it. If it needs to\n> parse it, it doesn't need to fill commit-graph specific data again.\n>\n> All right.\n>\n>> \tif (save_commit_buffer && !ret) {\n>> \t\tset_commit_buffer(item, buffer, size);\n>> \t\treturn 0;\n>> diff --git a/commit.h b/commit.h\n>> index 64436ff44e..b5afde1ae9 100644\n>> --- a/commit.h\n>> +++ b/commit.h\n>> @@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n>> */\n>> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n>> \n>> -int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n>> +int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n>> int parse_commit_gently(struct commit *item, int quiet_on_missing);\n>> static inline int parse_commit(struct commit *item)\n>> {\n>> diff --git a/object.c b/object.c\n>> index e6ad3f61f0..efe4871325 100644\n>> --- a/object.c\n>> +++ b/object.c\n>> @@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n>> \t} else if (type == OBJ_COMMIT) {\n>> \t\tstruct commit *commit = lookup_commit(oid);\n>> \t\tif (commit) {\n>> -\t\t\tif (parse_commit_buffer(commit, buffer, size))\n>> +\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n> All that rigamarole was needed because of\n>\n> DS> In some cases, including some branch lookups, the commit\n> DS> is parsed using parse_object_buffer() which side-steps parse_commit() in\n> DS> favor of parse_commit_buffer().\n>\n> Here we want parse_object_buffer() to get also commit-graph specific\n> data, if available. All right.\n>\n>> \t\t\t\treturn NULL;\n>> \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n>> \t\t\t\tset_commit_buffer(commit, buffer, size);\n>> diff --git a/sha1_file.c b/sha1_file.c\n>> index 1b94f39c4c..0fd4f0b8b6 100644\n>> --- a/sha1_file.c\n>> +++ b/sha1_file.c\n>> @@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n>> {\n>> \tstruct commit c;\n>> \tmemset(&c, 0, sizeof(c));\n>> -\tif (parse_commit_buffer(&c, buf, size))\n>> +\tif (parse_commit_buffer(&c, buf, size, 0))\n> For check we don't need commit graph data. Looks all right.\n>\n>> \t\tdie(\"corrupt commit\");\n>> }\n> Best,\n\n"},{"id":"346271","messageId":"20180501124652.155781-1-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180425143735.240183-1-dstolee@microsoft.com","subject":"[PATCH v5 00/11] Compute and consume generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:03Z","receivedAt":"2018-05-01T12:47:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Most of the changes from v4 are cosmetic, but there is one new commit:\n\n\tcommit: use generation number in remove_redundant()\n\nOther changes are non-functional, but do clarify things.\n\nInter-diff from v4:\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex d9f2713efa..e1a883eb46 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -125,9 +125,10 @@ Future Work\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n priority queue with one ordered by generation number. The following\n- operation is an important candidate:\n+ operations are important candidates:\n\n - 'log --topo-order'\n+ - 'tag --merged'\n\n - Currently, parse_commit_gently() requires filling in the root tree\n object for a commit. This passes through lookup_tree() and consequently\ndiff --git a/commit-graph.c b/commit-graph.c\nindex aebd242def..a8c337dd77 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -248,6 +248,7 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n const unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+ item->graph_pos = pos;\n item->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n\n@@ -454,8 +455,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n else\n packedDate[0] = 0;\n\n- if ((*list)->generation != GENERATION_NUMBER_INFINITY)\n- packedDate[0] |= htonl((*list)->generation << 2);\n+ packedDate[0] |= htonl((*list)->generation << 2);\n\n packedDate[1] = htonl((*list)->date);\n hashwrite(f, packedDate, 8);\n@@ -589,18 +589,17 @@ static void close_reachable(struct packed_oid_list *oids)\n }\n }\n\n-static void compute_generation_numbers(struct commit** commits,\n- int nr_commits)\n+static void compute_generation_numbers(struct packed_commit_list* commits)\n {\n int i;\n struct commit_list *list = NULL;\n\n- for (i = 0; i < nr_commits; i++) {\n- if (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n- commits[i]->generation != GENERATION_NUMBER_ZERO)\n+ for (i = 0; i < commits->nr; i++) {\n+ if (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n+ commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n continue;\n\n- commit_list_insert(commits[i], &list);\n+ commit_list_insert(commits->list[i], &list);\n while (list) {\n struct commit *current = list->item;\n struct commit_list *parent;\n@@ -621,10 +620,10 @@ static void compute_generation_numbers(struct commit** commits,\n if (all_parents_computed) {\n current->generation = max_generation + 1;\n pop_commit(&list);\n- }\n\n- if (current->generation > GENERATION_NUMBER_MAX)\n- current->generation = GENERATION_NUMBER_MAX;\n+ if (current->generation > GENERATION_NUMBER_MAX)\n+ current->generation = GENERATION_NUMBER_MAX;\n+ }\n }\n }\n }\n@@ -752,7 +751,7 @@ void write_commit_graph(const char *obj_dir,\n if (commits.nr >= GRAPH_PARENT_MISSING)\n die(_(\"too many commits to write graph\"));\n\n- compute_generation_numbers(commits.list, commits.nr);\n+ compute_generation_numbers(&commits);\n\n graph_name = get_commit_graph_filename(obj_dir);\n fd = hold_lock_file_for_update(&lk, graph_name, 0);\ndiff --git a/commit.c b/commit.c\nindex e2e16ea1a7..5064db4e61 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -835,7 +835,9 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n,\n int flags;\n\n if (commit->generation > last_gen)\n- BUG(\"bad generation skip\");\n+ BUG(\"bad generation skip %8x > %8x at %s\",\n+ commit->generation, last_gen,\n+ oid_to_hex(&commit->object.oid));\n last_gen = commit->generation;\n\n if (commit->generation < min_generation)\n@@ -947,6 +949,7 @@ static int remove_redundant(struct commit **array, int cnt)\n parse_commit(array[i]);\n for (i = 0; i < cnt; i++) {\n struct commit_list *common;\n+ uint32_t min_generation = GENERATION_NUMBER_INFINITY;\n\n if (redundant[i])\n continue;\n@@ -955,8 +958,12 @@ static int remove_redundant(struct commit **array, int cnt)\n continue;\n filled_index[filled] = j;\n work[filled++] = array[j];\n+\n+ if (array[j]->generation < min_generation)\n+ min_generation = array[j]->generation;\n }\n- common = paint_down_to_common(array[i], filled, work, 0);\n+ common = paint_down_to_common(array[i], filled, work,\n+ min_generation);\n if (array[i]->object.flags & PARENT2)\n redundant[i] = 1;\n for (j = 0; j < filled; j++)\n@@ -1073,7 +1080,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n for (i = 0; i < nr_reference; i++) {\n if (parse_commit(reference[i]))\n return ret;\n- if (min_generation > reference[i]->generation)\n+ if (reference[i]->generation < min_generation)\n min_generation = reference[i]->generation;\n }\n\n\n-- >8 --\n\nDerrick Stolee (11):\n ref-filter: fix outdated comment on in_commit_list\n commit: add generation number to struct commmit\n commit-graph: compute generation numbers\n commit: use generations in paint_down_to_common()\n commit-graph: always load commit-graph information\n ref-filter: use generation number for --contains\n commit: use generation numbers for in_merge_bases()\n commit: add short-circuit to paint_down_to_common()\n commit: use generation number in remove_redundant()\n merge: check config before loading commits\n commit-graph.txt: update design document\n\n Documentation/technical/commit-graph.txt | 30 ++++++--\n alloc.c | 1 +\n builtin/merge.c | 7 +-\n commit-graph.c | 91 ++++++++++++++++++++----\n commit-graph.h | 8 +++\n commit.c | 61 +++++++++++++---\n commit.h | 7 +-\n object.c | 2 +-\n ref-filter.c | 26 +++++--\n sha1_file.c | 2 +-\n t/t5318-commit-graph.sh | 9 +++\n 11 files changed, 204 insertions(+), 40 deletions(-)\n\n\nbase-commit: 7b8a21dba1bce44d64bd86427d3d92437adc4707\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346272","messageId":"20180501124652.155781-2-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 01/11] ref-filter: fix outdated comment on in_commit_list","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:05Z","receivedAt":"2018-05-01T12:47:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The in_commit_list() method does not check the parents of\nthe candidate for containment in the list. Fix the comment\nthat incorrectly states that it does.\n\nReported-by: Jakub Narebski \nSigned-off-by: Derrick Stolee \n---\n ref-filter.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex cffd8bf3ce..aff24d93be 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -1582,7 +1582,7 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n }\n \n /*\n- * Test whether the candidate or one of its parents is contained in the list.\n+ * Test whether the candidate is contained in the list.\n * Do not recurse to find out, though, but return -1 if inconclusive.\n */\n static enum contains_result contains_test(struct commit *candidate,\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346273","messageId":"20180501124652.155781-4-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 03/11] commit-graph: compute generation numbers","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:09Z","receivedAt":"2018-05-01T12:47:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While preparing commits to be written into a commit-graph file, compute\nthe generation numbers using a depth-first strategy.\n\nThe only commits that are walked in this depth-first search are those\nwithout a precomputed generation number. Thus, computation time will be\nrelative to the number of new commits to the commit-graph file.\n\nIf a computed generation number would exceed GENERATION_NUMBER_MAX, then\nuse GENERATION_NUMBER_MAX instead.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9ad21c3ffb..36d765e10a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -439,6 +439,8 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\telse\n \t\t\tpackedDate[0] = 0;\n \n+\t\tpackedDate[0] |= htonl((*list)->generation << 2);\n+\n \t\tpackedDate[1] = htonl((*list)->date);\n \t\thashwrite(f, packedDate, 8);\n \n@@ -571,6 +573,45 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n }\n \n+static void compute_generation_numbers(struct packed_commit_list* commits)\n+{\n+\tint i;\n+\tstruct commit_list *list = NULL;\n+\n+\tfor (i = 0; i < commits->nr; i++) {\n+\t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n+\t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n+\t\t\tcontinue;\n+\n+\t\tcommit_list_insert(commits->list[i], &list);\n+\t\twhile (list) {\n+\t\t\tstruct commit *current = list->item;\n+\t\t\tstruct commit_list *parent;\n+\t\t\tint all_parents_computed = 1;\n+\t\t\tuint32_t max_generation = 0;\n+\n+\t\t\tfor (parent = current->parents; parent; parent = parent->next) {\n+\t\t\t\tif (parent->item->generation == GENERATION_NUMBER_INFINITY ||\n+\t\t\t\t parent->item->generation == GENERATION_NUMBER_ZERO) {\n+\t\t\t\t\tall_parents_computed = 0;\n+\t\t\t\t\tcommit_list_insert(parent->item, &list);\n+\t\t\t\t\tbreak;\n+\t\t\t\t} else if (parent->item->generation > max_generation) {\n+\t\t\t\t\tmax_generation = parent->item->generation;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (all_parents_computed) {\n+\t\t\t\tcurrent->generation = max_generation + 1;\n+\t\t\t\tpop_commit(&list);\n+\n+\t\t\t\tif (current->generation > GENERATION_NUMBER_MAX)\n+\t\t\t\t\tcurrent->generation = GENERATION_NUMBER_MAX;\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tconst char **pack_indexes,\n \t\t\tint nr_packs,\n@@ -694,6 +735,8 @@ void write_commit_graph(const char *obj_dir,\n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n \n+\tcompute_generation_numbers(&commits);\n+\n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tfd = hold_lock_file_for_update(&lk, graph_name, 0);\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346274","messageId":"20180501124652.155781-3-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 02/11] commit: add generation number to struct commmit","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:07Z","receivedAt":"2018-05-01T12:47:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The generation number of a commit is defined recursively as follows:\n\n* If a commit A has no parents, then the generation number of A is one.\n* If a commit A has parents, then the generation number of A is one\n more than the maximum generation number among the parents of A.\n\nAdd a uint32_t generation field to struct commit so we can pass this\ninformation to revision walks. We use three special values to signal\nthe generation number is invalid:\n\nGENERATION_NUMBER_INFINITY 0xFFFFFFFF\nGENERATION_NUMBER_MAX 0x3FFFFFFF\nGENERATION_NUMBER_ZERO 0\n\nThe first (_INFINITY) means the generation number has not been loaded or\ncomputed. The second (_MAX) means the generation number is too large to\nstore in the commit-graph file. The third (_ZERO) means the generation\nnumber was loaded from a commit graph file that was written by a version\nof git that did not support generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 2 ++\n commit.h | 4 ++++\n 3 files changed, 7 insertions(+)\n\ndiff --git a/alloc.c b/alloc.c\nindex cf4f8b61e1..e8ab14f4a1 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -94,6 +94,7 @@ void *alloc_commit_node(void)\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n \tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n+\tc->generation = GENERATION_NUMBER_INFINITY;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 70fa1b25fd..9ad21c3ffb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -262,6 +262,8 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \tdate_low = get_be32(commit_data + g->hash_len + 12);\n \titem->date = (timestamp_t)((date_high << 32) | date_low);\n \n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+\n \tpptr = &item->parents;\n \n \tedge_value = get_be32(commit_data + g->hash_len);\ndiff --git a/commit.h b/commit.h\nindex 23a3f364ed..aac3b8c56f 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -10,6 +10,9 @@\n #include \"pretty.h\"\n \n #define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+#define GENERATION_NUMBER_INFINITY 0xFFFFFFFF\n+#define GENERATION_NUMBER_MAX 0x3FFFFFFF\n+#define GENERATION_NUMBER_ZERO 0\n \n struct commit_list {\n \tstruct commit *item;\n@@ -30,6 +33,7 @@ struct commit {\n \t */\n \tstruct tree *maybe_tree;\n \tuint32_t graph_pos;\n+\tuint32_t generation;\n };\n \n extern int save_commit_buffer;\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346275","messageId":"20180501124652.155781-5-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 04/11] commit: use generations in paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:11Z","receivedAt":"2018-05-01T12:47:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Define compare_commits_by_gen_then_commit_date(), which uses generation\nnumbers as a primary comparison and commit date to break ties (or as a\ncomparison when both commits do not have computed generation numbers).\n\nSince the commit-graph file is closed under reachability, we know that\nall commits in the file have generation at most GENERATION_NUMBER_MAX\nwhich is less than GENERATION_NUMBER_INFINITY.\n\nThis change does not affect the number of commits that are walked during\nthe execution of paint_down_to_common(), only the order that those\ncommits are inspected. In the case that commit dates violate topological\norder (i.e. a parent is \"newer\" than a child), the previous code could\nwalk a commit twice: if a commit is reached with the PARENT1 bit, but\nlater is re-visited with the PARENT2 bit, then that PARENT2 bit must be\npropagated to its parents. Using generation numbers avoids this extra\neffort, even if it is somewhat rare.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 20 +++++++++++++++++++-\n commit.h | 1 +\n 2 files changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 711f674c18..4d00b0a1d6 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -640,6 +640,24 @@ static int compare_commits_by_author_date(const void *a_, const void *b_,\n \treturn 0;\n }\n \n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused)\n+{\n+\tconst struct commit *a = a_, *b = b_;\n+\n+\t/* newer commits first */\n+\tif (a->generation < b->generation)\n+\t\treturn 1;\n+\telse if (a->generation > b->generation)\n+\t\treturn -1;\n+\n+\t/* use date as a heuristic when generations are equal */\n+\tif (a->date < b->date)\n+\t\treturn 1;\n+\telse if (a->date > b->date)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused)\n {\n \tconst struct commit *a = a_, *b = b_;\n@@ -789,7 +807,7 @@ static int queue_has_nonstale(struct prio_queue *queue)\n /* all input commits in one and twos[] must have been parsed! */\n static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n {\n-\tstruct prio_queue queue = { compare_commits_by_commit_date };\n+\tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n \ndiff --git a/commit.h b/commit.h\nindex aac3b8c56f..64436ff44e 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -341,6 +341,7 @@ extern int remove_signature(struct strbuf *buf);\n extern int check_commit_signature(const struct commit *commit, struct signature_check *sigc);\n \n int compare_commits_by_commit_date(const void *a_, const void *b_, void *unused);\n+int compare_commits_by_gen_then_commit_date(const void *a_, const void *b_, void *unused);\n \n LAST_ARG_MUST_BE_NULL\n extern int run_commit_hook(int editor_is_used, const char *index_file, const char *name, ...);\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346276","messageId":"20180501124652.155781-6-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 05/11] commit-graph: always load commit-graph information","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:13Z","receivedAt":"2018-05-01T12:47:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Most code paths load commits using lookup_commit() and then\nparse_commit(). In some cases, including some branch lookups, the commit\nis parsed using parse_object_buffer() which side-steps parse_commit() in\nfavor of parse_commit_buffer().\n\nWith generation numbers in the commit-graph, we need to ensure that any\ncommit that exists in the commit-graph file has its generation number\nloaded.\n\nCreate new load_commit_graph_info() method to fill in the information\nfor a commit that exists only in the commit-graph file. Call it from\nparse_commit_buffer() after loading the other commit information from\nthe given buffer. Only fill this information when specified by the\n'check_graph' parameter.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 46 +++++++++++++++++++++++++++++++---------------\n commit-graph.h | 8 ++++++++\n commit.c | 7 +++++--\n commit.h | 2 +-\n object.c | 2 +-\n sha1_file.c | 2 +-\n 6 files changed, 47 insertions(+), 20 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 36d765e10a..a8c337dd77 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -245,6 +245,13 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n \treturn &commit_list_insert(c, pptr)->next;\n }\n \n+static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n+{\n+\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\titem->graph_pos = pos;\n+\titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n+}\n+\n static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n \tuint32_t edge_value;\n@@ -292,31 +299,40 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \treturn 1;\n }\n \n+static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n+{\n+\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n+\t\t*pos = item->graph_pos;\n+\t\treturn 1;\n+\t} else {\n+\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t}\n+}\n+\n int parse_commit_in_graph(struct commit *item)\n {\n+\tuint32_t pos;\n+\n \tif (!core_commit_graph)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n-\n \tprepare_commit_graph();\n-\tif (commit_graph) {\n-\t\tuint32_t pos;\n-\t\tint found;\n-\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n-\t\t\tpos = item->graph_pos;\n-\t\t\tfound = 1;\n-\t\t} else {\n-\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n-\t\t}\n-\n-\t\tif (found)\n-\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n-\t}\n-\n+\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n+\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n \treturn 0;\n }\n \n+void load_commit_graph_info(struct commit *item)\n+{\n+\tuint32_t pos;\n+\tif (!core_commit_graph)\n+\t\treturn;\n+\tprepare_commit_graph();\n+\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n+\t\tfill_commit_graph_info(item, commit_graph, pos);\n+}\n+\n static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *c)\n {\n \tstruct object_id oid;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 260a468e73..96cccb10f3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -17,6 +17,14 @@ char *get_commit_graph_filename(const char *obj_dir);\n */\n int parse_commit_in_graph(struct commit *item);\n \n+/*\n+ * It is possible that we loaded commit contents from the commit buffer,\n+ * but we also want to ensure the commit-graph content is correctly\n+ * checked and filled. Fill the graph_pos and generation members of\n+ * the given commit.\n+ */\n+void load_commit_graph_info(struct commit *item);\n+\n struct tree *get_commit_tree_in_graph(const struct commit *c);\n \n struct commit_graph {\ndiff --git a/commit.c b/commit.c\nindex 4d00b0a1d6..39a3749abd 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -331,7 +331,7 @@ const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n \treturn ret;\n }\n \n-int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size)\n+int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph)\n {\n \tconst char *tail = buffer;\n \tconst char *bufptr = buffer;\n@@ -386,6 +386,9 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \t}\n \titem->date = parse_commit_date(bufptr, tail);\n \n+\tif (check_graph)\n+\t\tload_commit_graph_info(item);\n+\n \treturn 0;\n }\n \n@@ -412,7 +415,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn error(\"Object %s not a commit\",\n \t\t\t oid_to_hex(&item->object.oid));\n \t}\n-\tret = parse_commit_buffer(item, buffer, size);\n+\tret = parse_commit_buffer(item, buffer, size, 0);\n \tif (save_commit_buffer && !ret) {\n \t\tset_commit_buffer(item, buffer, size);\n \t\treturn 0;\ndiff --git a/commit.h b/commit.h\nindex 64436ff44e..b5afde1ae9 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -72,7 +72,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n */\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n-int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n+int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\ndiff --git a/object.c b/object.c\nindex e6ad3f61f0..efe4871325 100644\n--- a/object.c\n+++ b/object.c\n@@ -207,7 +207,7 @@ struct object *parse_object_buffer(const struct object_id *oid, enum object_type\n \t} else if (type == OBJ_COMMIT) {\n \t\tstruct commit *commit = lookup_commit(oid);\n \t\tif (commit) {\n-\t\t\tif (parse_commit_buffer(commit, buffer, size))\n+\t\t\tif (parse_commit_buffer(commit, buffer, size, 1))\n \t\t\t\treturn NULL;\n \t\t\tif (!get_cached_commit_buffer(commit, NULL)) {\n \t\t\t\tset_commit_buffer(commit, buffer, size);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1b94f39c4c..0fd4f0b8b6 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1755,7 +1755,7 @@ static void check_commit(const void *buf, size_t size)\n {\n \tstruct commit c;\n \tmemset(&c, 0, sizeof(c));\n-\tif (parse_commit_buffer(&c, buf, size))\n+\tif (parse_commit_buffer(&c, buf, size, 0))\n \t\tdie(\"corrupt commit\");\n }\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346277","messageId":"20180501124652.155781-8-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 07/11] commit: use generation numbers for in_merge_bases()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:17Z","receivedAt":"2018-05-01T12:47:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The containment algorithm for 'git branch --contains' is different\nfrom that for 'git tag --contains' in that it uses is_descendant_of()\ninstead of contains_tag_algo(). The expensive portion of the branch\nalgorithm is computing merge bases.\n\nWhen a commit-graph file exists with generation numbers computed,\nwe can avoid this merge-base calculation when the target commit has\na larger generation number than the initial commits.\n\nPerformance tests were run on a copy of the Linux repository where\nHEAD is contained in v4.13 but no earlier tag. Also, all tags were\ncopied to branches and 'git branch --contains' was tested:\n\nBefore: 60.0s\nAfter: 0.4s\nRel %: -99.3%\n\nReported-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 39a3749abd..3ecdc13356 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1056,12 +1056,19 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n {\n \tstruct commit_list *bases;\n \tint ret = 0, i;\n+\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n \n \tif (parse_commit(commit))\n \t\treturn ret;\n-\tfor (i = 0; i < nr_reference; i++)\n+\tfor (i = 0; i < nr_reference; i++) {\n \t\tif (parse_commit(reference[i]))\n \t\t\treturn ret;\n+\t\tif (reference[i]->generation < min_generation)\n+\t\t\tmin_generation = reference[i]->generation;\n+\t}\n+\n+\tif (commit->generation > min_generation)\n+\t\treturn ret;\n \n \tbases = paint_down_to_common(commit, nr_reference, reference);\n \tif (commit->object.flags & PARENT2)\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346278","messageId":"20180501124652.155781-9-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 08/11] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:19Z","receivedAt":"2018-05-01T12:47:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git branch --contains', the in_merge_bases_many()\nmethod calls paint_down_to_common() to discover if a specific\ncommit is reachable from a set of branches. Commits with lower\ngeneration number are not needed to correctly answer the\ncontainment query of in_merge_bases_many().\n\nAdd a new parameter, min_generation, to paint_down_to_common() that\nprevents walking commits with generation number strictly less than\nmin_generation. If 0 is given, then there is no functional change.\n\nFor in_merge_bases_many(), we can pass commit->generation as the\ncutoff, and this saves time during 'git branch --contains' queries\nthat would otherwise walk \"around\" the commit we are inspecting.\n\nFor a copy of the Linux repository, where HEAD is checked out at\nv4.13~100, we get the following performance improvement for\n'git branch --contains' over the previous commit:\n\nBefore: 0.21s\nAfter: 0.13s\nRel %: -38%\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 3ecdc13356..9875feec01 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -808,11 +808,14 @@ static int queue_has_nonstale(struct prio_queue *queue)\n }\n \n /* all input commits in one and twos[] must have been parsed! */\n-static struct commit_list *paint_down_to_common(struct commit *one, int n, struct commit **twos)\n+static struct commit_list *paint_down_to_common(struct commit *one, int n,\n+\t\t\t\t\t\tstruct commit **twos,\n+\t\t\t\t\t\tint min_generation)\n {\n \tstruct prio_queue queue = { compare_commits_by_gen_then_commit_date };\n \tstruct commit_list *result = NULL;\n \tint i;\n+\tuint32_t last_gen = GENERATION_NUMBER_INFINITY;\n \n \tone->object.flags |= PARENT1;\n \tif (!n) {\n@@ -831,6 +834,15 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n \t\tstruct commit_list *parents;\n \t\tint flags;\n \n+\t\tif (commit->generation > last_gen)\n+\t\t\tBUG(\"bad generation skip %8x > %8x at %s\",\n+\t\t\t commit->generation, last_gen,\n+\t\t\t oid_to_hex(&commit->object.oid));\n+\t\tlast_gen = commit->generation;\n+\n+\t\tif (commit->generation < min_generation)\n+\t\t\tbreak;\n+\n \t\tflags = commit->object.flags & (PARENT1 | PARENT2 | STALE);\n \t\tif (flags == (PARENT1 | PARENT2)) {\n \t\t\tif (!(commit->object.flags & RESULT)) {\n@@ -879,7 +891,7 @@ static struct commit_list *merge_bases_many(struct commit *one, int n, struct co\n \t\t\treturn NULL;\n \t}\n \n-\tlist = paint_down_to_common(one, n, twos);\n+\tlist = paint_down_to_common(one, n, twos, 0);\n \n \twhile (list) {\n \t\tstruct commit *commit = pop_commit(&list);\n@@ -946,7 +958,7 @@ static int remove_redundant(struct commit **array, int cnt)\n \t\t\tfilled_index[filled] = j;\n \t\t\twork[filled++] = array[j];\n \t\t}\n-\t\tcommon = paint_down_to_common(array[i], filled, work);\n+\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n \t\tif (array[i]->object.flags & PARENT2)\n \t\t\tredundant[i] = 1;\n \t\tfor (j = 0; j < filled; j++)\n@@ -1070,7 +1082,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n \tif (commit->generation > min_generation)\n \t\treturn ret;\n \n-\tbases = paint_down_to_common(commit, nr_reference, reference);\n+\tbases = paint_down_to_common(commit, nr_reference, reference, commit->generation);\n \tif (commit->object.flags & PARENT2)\n \t\tret = 1;\n \tclear_commit_marks(commit, all_flags);\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346279","messageId":"20180501124652.155781-10-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 09/11] commit: use generation number in remove_redundant()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:21Z","receivedAt":"2018-05-01T12:47:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The static remove_redundant() method is used to filter a list\nof commits by removing those that are reachable from another\ncommit in the list. This is used to remove all possible merge-\nbases except a maximal, mutually independent set.\n\nTo determine these commits are independent, we use a number of\npaint_down_to_common() walks and use the PARENT1, PARENT2 flags\nto determine reachability. Since we only care about reachability\nand not the full set of merge-bases between 'one' and 'twos', we\ncan use the 'min_generation' parameter to short-circuit the walk.\n\nWhen no commit-graph exists, there is no change in behavior.\n\nFor a copy of the Linux repository, we measured the following\nperformance improvements:\n\ngit merge-base v3.3 v4.5\n\nBefore: 234 ms\n After: 208 ms\n Rel %: -11%\n\ngit merge-base v4.3 v4.5\n\nBefore: 102 ms\n After: 83 ms\n Rel %: -19%\n\nThe experiments above were chosen to demonstrate that we are\nimproving the filtering of the merge-base set. In the first\nexample, more time is spent walking the history to find the\nset of merge bases before the remove_redundant() call. The\nstarting commits are closer together in the second example,\ntherefore more time is spent in remove_redundant(). The relative\nchange in performance differs as expected.\n\nReported-by: Jakub Narebski \nSigned-off-by: Derrick Stolee \n---\n commit.c | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex 9875feec01..5064db4e61 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -949,6 +949,7 @@ static int remove_redundant(struct commit **array, int cnt)\n \t\tparse_commit(array[i]);\n \tfor (i = 0; i < cnt; i++) {\n \t\tstruct commit_list *common;\n+\t\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n \n \t\tif (redundant[i])\n \t\t\tcontinue;\n@@ -957,8 +958,12 @@ static int remove_redundant(struct commit **array, int cnt)\n \t\t\t\tcontinue;\n \t\t\tfilled_index[filled] = j;\n \t\t\twork[filled++] = array[j];\n+\n+\t\t\tif (array[j]->generation < min_generation)\n+\t\t\t\tmin_generation = array[j]->generation;\n \t\t}\n-\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n+\t\tcommon = paint_down_to_common(array[i], filled, work,\n+\t\t\t\t\t min_generation);\n \t\tif (array[i]->object.flags & PARENT2)\n \t\t\tredundant[i] = 1;\n \t\tfor (j = 0; j < filled; j++)\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346280","messageId":"20180501124652.155781-11-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 10/11] merge: check config before loading commits","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:23Z","receivedAt":"2018-05-01T12:47:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Now that we use generation numbers from the commit-graph, we must\nensure that all commits that exist in the commit-graph are loaded\nfrom that file instead of from the object database. Since the\ncommit-graph file is only checked if core.commitGraph is true, we\nmust check the default config before we load any commits.\n\nIn the merge builtin, the config was checked after loading the HEAD\ncommit. This was due to the use of the global 'branch' when checking\nmerge-specific config settings.\n\nMove the config load to be between the initialization of 'branch' and\nthe commit lookup.\n\nWithout this change, a fast-forward merge would hit a BUG(\"bad\ngeneration skip\") statement in commit.c during paint_down_to_common().\nThis is because the HEAD commit would be loaded with \"infinite\"\ngeneration but then reached by commits with \"finite\" generation\nnumbers.\n\nAdd a test to t5318-commit-graph.sh that exercises this code path to\nprevent a regression.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/merge.c | 7 ++++---\n t/t5318-commit-graph.sh | 9 +++++++++\n 2 files changed, 13 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 5e5e4497e3..b819756946 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -1148,14 +1148,15 @@ int cmd_merge(int argc, const char **argv, const char *prefix)\n \tbranch = branch_to_free = resolve_refdup(\"HEAD\", 0, &head_oid, NULL);\n \tif (branch)\n \t\tskip_prefix(branch, \"refs/heads/\", &branch);\n+\n+\tinit_diff_ui_defaults();\n+\tgit_config(git_merge_config, NULL);\n+\n \tif (!branch || is_null_oid(&head_oid))\n \t\thead_commit = NULL;\n \telse\n \t\thead_commit = lookup_commit_or_die(&head_oid, \"HEAD\");\n \n-\tinit_diff_ui_defaults();\n-\tgit_config(git_merge_config, NULL);\n-\n \tif (branch_mergeoptions)\n \t\tparse_branch_merge_options(branch_mergeoptions);\n \targc = parse_options(argc, argv, prefix, builtin_merge_options,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a380419b65..77d85aefe7 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n \n+test_expect_success 'perform fast-forward merge in full repo' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit checkout -b merge-5-to-8 commits/5 &&\n+\tgit merge commits/8 &&\n+\tgit show-ref -s merge-5-to-8 >output &&\n+\tgit show-ref -s commits/8 >expect &&\n+\ttest_cmp expect output\n+'\n+\n test_done\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346281","messageId":"20180501124652.155781-12-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 11/11] commit-graph.txt: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:25Z","receivedAt":"2018-05-01T12:47:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We now calculate generation numbers in the commit-graph file and use\nthem in paint_down_to_common().\n\nExpand the section on generation numbers to discuss how the three\nspecial generation numbers GENERATION_NUMBER_INFINITY, _ZERO, and\n_MAX interact with other generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 29 ++++++++++++++++++++----\n 1 file changed, 24 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 0550c6d0dc..e1a883eb46 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -77,6 +77,29 @@ in the commit graph. We can treat these commits as having \"infinite\"\n generation number and walk until reaching commits with known generation\n number.\n \n+We use the macro GENERATION_NUMBER_INFINITY = 0xFFFFFFFF to mark commits not\n+in the commit-graph file. If a commit-graph file was written by a version\n+of Git that did not compute generation numbers, then those commits will\n+have generation number represented by the macro GENERATION_NUMBER_ZERO = 0.\n+\n+Since the commit-graph file is closed under reachability, we can guarantee\n+the following weaker condition on all commits:\n+\n+ If A and B are commits with generation numbers N amd M, respectively,\n+ and N < M, then A cannot reach B.\n+\n+Note how the strict inequality differs from the inequality when we have\n+fully-computed generation numbers. Using strict inequality may result in\n+walking a few extra commits, but the simplicity in dealing with commits\n+with generation number *_INFINITY or *_ZERO is valuable.\n+\n+We use the macro GENERATION_NUMBER_MAX = 0x3FFFFFFF to for commits whose\n+generation numbers are computed to be at least this value. We limit at\n+this value since it is the largest value that can be stored in the\n+commit-graph file using the 30 bits available to generation numbers. This\n+presents another case where a commit can have generation number equal to\n+that of a parent.\n+\n Design Details\n --------------\n \n@@ -98,18 +121,14 @@ Future Work\n - The 'commit-graph' subcommand does not have a \"verify\" mode that is\n necessary for integration with fsck.\n \n-- The file format includes room for precomputed generation numbers. These\n- are not currently computed, so all generation numbers will be marked as\n- 0 (or \"uncomputed\"). A later patch will include this calculation.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n priority queue with one ordered by generation number. The following\n operations are important candidates:\n \n- - paint_down_to_common()\n - 'log --topo-order'\n+ - 'tag --merged'\n \n - Currently, parse_commit_gently() requires filling in the root tree\n object for a commit. This passes through lookup_tree() and consequently\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346282","messageId":"20180501124652.155781-7-dstolee@microsoft.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"[PATCH v5 06/11] ref-filter: use generation number for --contains","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-01T12:47:15Z","receivedAt":"2018-05-01T12:47:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"A commit A can reach a commit B only if the generation number of A\nis strictly larger than the generation number of B. This condition\nallows significantly short-circuiting commit-graph walks.\n\nUse generation number for '--contains' type queries.\n\nOn a copy of the Linux repository where HEAD is contained in v4.13\nbut no earlier tag, the command 'git tag --contains HEAD' had the\nfollowing peformance improvement:\n\nBefore: 0.81s\nAfter: 0.04s\nRel %: -95%\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n ref-filter.c | 24 ++++++++++++++++++++----\n 1 file changed, 20 insertions(+), 4 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex aff24d93be..fb35067fc9 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -16,6 +16,7 @@\n #include \"trailer.h\"\n #include \"wt-status.h\"\n #include \"commit-slab.h\"\n+#include \"commit-graph.h\"\n \n static struct ref_msg {\n \tconst char *gone;\n@@ -1587,7 +1588,8 @@ static int in_commit_list(const struct commit_list *want, struct commit *c)\n */\n static enum contains_result contains_test(struct commit *candidate,\n \t\t\t\t\t const struct commit_list *want,\n-\t\t\t\t\t struct contains_cache *cache)\n+\t\t\t\t\t struct contains_cache *cache,\n+\t\t\t\t\t uint32_t cutoff)\n {\n \tenum contains_result *cached = contains_cache_at(cache, candidate);\n \n@@ -1603,6 +1605,10 @@ static enum contains_result contains_test(struct commit *candidate,\n \n \t/* Otherwise, we don't know; prepare to recurse */\n \tparse_commit_or_die(candidate);\n+\n+\tif (candidate->generation < cutoff)\n+\t\treturn CONTAINS_NO;\n+\n \treturn CONTAINS_UNKNOWN;\n }\n \n@@ -1618,8 +1624,18 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t\t\t\t struct contains_cache *cache)\n {\n \tstruct contains_stack contains_stack = { 0, 0, NULL };\n-\tenum contains_result result = contains_test(candidate, want, cache);\n+\tenum contains_result result;\n+\tuint32_t cutoff = GENERATION_NUMBER_INFINITY;\n+\tconst struct commit_list *p;\n+\n+\tfor (p = want; p; p = p->next) {\n+\t\tstruct commit *c = p->item;\n+\t\tload_commit_graph_info(c);\n+\t\tif (c->generation < cutoff)\n+\t\t\tcutoff = c->generation;\n+\t}\n \n+\tresult = contains_test(candidate, want, cache, cutoff);\n \tif (result != CONTAINS_UNKNOWN)\n \t\treturn result;\n \n@@ -1637,7 +1653,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t * If we just popped the stack, parents->item has been marked,\n \t\t * therefore contains_test will return a meaningful yes/no.\n \t\t */\n-\t\telse switch (contains_test(parents->item, want, cache)) {\n+\t\telse switch (contains_test(parents->item, want, cache, cutoff)) {\n \t\tcase CONTAINS_YES:\n \t\t\t*contains_cache_at(cache, commit) = CONTAINS_YES;\n \t\t\tcontains_stack.nr--;\n@@ -1651,7 +1667,7 @@ static enum contains_result contains_tag_algo(struct commit *candidate,\n \t\t}\n \t}\n \tfree(contains_stack.contains_stack);\n-\treturn contains_test(candidate, want, cache);\n+\treturn contains_test(candidate, want, cache, cutoff);\n }\n \n static int commit_contains(struct ref_filter *filter, struct commit *commit,\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"346293","messageId":"1cfe38f6-925b-d36b-53ae-6b586eed199c@gmail.com","threadId":"48212","inReplyTo":"20180501124652.155781-10-dstolee@microsoft.com","subject":"Re: [PATCH v5 09/11] commit: use generation number in remove_redundant()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-01T15:37:43Z","receivedAt":"2018-05-01T15:37:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 5/1/2018 8:47 AM, Derrick Stolee wrote:\n> The static remove_redundant() method is used to filter a list\n> of commits by removing those that are reachable from another\n> commit in the list. This is used to remove all possible merge-\n> bases except a maximal, mutually independent set.\n>\n> To determine these commits are independent, we use a number of\n> paint_down_to_common() walks and use the PARENT1, PARENT2 flags\n> to determine reachability. Since we only care about reachability\n> and not the full set of merge-bases between 'one' and 'twos', we\n> can use the 'min_generation' parameter to short-circuit the walk.\n>\n> When no commit-graph exists, there is no change in behavior.\n>\n> For a copy of the Linux repository, we measured the following\n> performance improvements:\n>\n> git merge-base v3.3 v4.5\n>\n> Before: 234 ms\n> After: 208 ms\n> Rel %: -11%\n>\n> git merge-base v4.3 v4.5\n>\n> Before: 102 ms\n> After: 83 ms\n> Rel %: -19%\n>\n> The experiments above were chosen to demonstrate that we are\n> improving the filtering of the merge-base set. In the first\n> example, more time is spent walking the history to find the\n> set of merge bases before the remove_redundant() call. The\n> starting commits are closer together in the second example,\n> therefore more time is spent in remove_redundant(). The relative\n> change in performance differs as expected.\n>\n> Reported-by: Jakub Narebski \n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 7 ++++++-\n> 1 file changed, 6 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit.c b/commit.c\n> index 9875feec01..5064db4e61 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -949,6 +949,7 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\tparse_commit(array[i]);\n> \tfor (i = 0; i < cnt; i++) {\n> \t\tstruct commit_list *common;\n> +\t\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n\nThis initialization should be\n\n     uint32_t min_generation = array[i]->generation;\n\nsince the assignment (using j) below skips the ith commit.\n\n> \n> \t\tif (redundant[i])\n> \t\t\tcontinue;\n> @@ -957,8 +958,12 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\t\t\tcontinue;\n> \t\t\tfilled_index[filled] = j;\n> \t\t\twork[filled++] = array[j];\n> +\n> +\t\t\tif (array[j]->generation < min_generation)\n> +\t\t\t\tmin_generation = array[j]->generation;\n> \t\t}\n> -\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n> +\t\tcommon = paint_down_to_common(array[i], filled, work,\n> +\t\t\t\t\t min_generation);\n> \t\tif (array[i]->object.flags & PARENT2)\n> \t\t\tredundant[i] = 1;\n> \t\tfor (j = 0; j < filled; j++)\n\n"},{"id":"346435","messageId":"86o9hy1ocm.fsf@gmail.com","threadId":"48212","inReplyTo":"2ea2c1d0-e2ad-2051-ba98-1584fcb51a46@gmail.com","subject":"Re: [PATCH v4 10/10] commit-graph.txt: update design document","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-02T07:57:45Z","receivedAt":"2018-05-02T07:57:55Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 4/30/2018 7:32 PM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n[...]\n>>> - After computing and storing generation numbers, we must make graph\n>>> walks aware of generation numbers to gain the performance benefits they\n>>> enable. This will mostly be accomplished by swapping a commit-date-ordered\n>>> priority queue with one ordered by generation number. The following\n>>> - operations are important candidates:\n>>> + operation is an important candidate:\n>>> - - paint_down_to_common()\n>>> - 'log --topo-order'\n>>\n>> Another possible candidates:\n>>\n>> - remove_redundant() - see comment in previous patch\n>> - still_interesting() - where Git uses date slop to stop walking\n>> too far\n>\n> remove_redundant() will be included in v5, thanks.\n\nOh. Nice.\n\nI'll try to review the new patch in detail soon.\n\n> Instead of \"still_interesting()\" I'll add \"git tag --merged\" as the\n> candidate to consider, as discussed in [1].\n>\n> [1] https://public-inbox.org/git/87fu3g67ry.fsf@lant.ki.iif.hu/t/#u\n>     \"branch --contains / tag --merged inconsistency\"\n\nAll right. I have mentioned still_interesting() as a hint where\npossible additional generation numbers based optimization may lurk\n(because that's where heuristic based on dates is used - similarly to\nhow it was done in this series with paint_down_to_common()).\n\n[...]\n>> One important issue left is handling features that change view of\n>> project history, and their interaction with commit-graph feature.\n>>\n>> What would happen, if we turn on commit-graph feature, generate commit\n>> graph file, and then:\n>>\n>> * use graft file or remove graft entries to cut history, or remove cut\n>> or join two [independent] histories.\n>> * use git-replace mechanims to do the same\n>> * in shallow clone, deepen or shorten the clone\n>>\n>> What would happen if without re-generating commit-graph file (assuming\n>> tha Git wouldn't do it for us), we run some feature that makes use of\n>> commit-graph data:\n>>\n>> - git branch --contains\n>> - git tag --contains\n>> - git rev-list A..B\n>>\n>\n> The commit-graph is not supported in these scenarios (yet). grafts are\n> specifically mentioned in the future work section.\n>\n> I'm not particularly interested in supporting these features, so they\n> are good venues for other contributors to get involved in the\n> commit-graph feature. Eventually, they will be blockers to making the\n> commit-graph feature a \"default\" feature. That is when I will pay\n> attention to these situations. For now, a user must opt-in to having a\n> commit-graph file (and that same user has possibly opted in to these\n> history modifying features).\n\nWell, that is sensible approach. Get commit-graph features in working\ncondition, and worry about beng able to make it on by default later.\n\nNice to have it clarified. I'll stop nagging about that, then ;-P\n\nOne issue: 'grafts' are mentioned in the future work section of the\ntechnical documentation, but we don't have *any* warning about\ncommit-graph limitations in user-facing documentation, that is\ngit-commit-graph(1) manpage.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346446","messageId":"86h8nq1dzh.fsf@gmail.com","threadId":"48212","inReplyTo":"214dea35-a89c-4115-a51a-5110c4e53314@gmail.com","subject":"Re: [PATCH v4 09/10] merge: check config before loading commits","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-02T11:41:38Z","receivedAt":"2018-05-02T11:41:48Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 4/30/2018 6:54 PM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n>>\n>>> Now that we use generation numbers from the commit-graph, we must\n>>> ensure that all commits that exist in the commit-graph are loaded\n>>> from that file instead of from the object database. Since the\n>>> commit-graph file is only checked if core.commitGraph is true, we\n>>> must check the default config before we load any commits.\n>>>\n>>> In the merge builtin, the config was checked after loading the HEAD\n>>> commit. This was due to the use of the global 'branch' when checking\n>>> merge-specific config settings.\n>>>\n>>> Move the config load to be between the initialization of 'branch' and\n>>> the commit lookup.\n>>\n>> Sidenote: I wonder why reading config was postponed to later in the\n>> command lifetime... I guess it was to avoid having to read config if\n>> HEAD was invalid.\n>\n> The 'branch' does need to be loaded before the call to git_config (as\n> I found out after moving the config call too early), so I suppose it\n> was natural to pair that with resolving head_commit.\n\nRight, so there was only a limited number of places where call to\ngit_config could be put correctly. Now I wonder no more.\n\n[...]\n>>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>>> index a380419b65..77d85aefe7 100755\n>>> --- a/t/t5318-commit-graph.sh\n>>> +++ b/t/t5318-commit-graph.sh\n>>> @@ -221,4 +221,13 @@ test_expect_success 'write graph in bare repo' '\n>>> graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n>>> graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' bare commits/8 merge/2\n>>> +test_expect_success 'perform fast-forward merge in full repo' '\n>>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>>> +\tgit checkout -b merge-5-to-8 commits/5 &&\n>>> +\tgit merge commits/8 &&\n>>> +\tgit show-ref -s merge-5-to-8 >output &&\n>>> +\tgit show-ref -s commits/8 >expect &&\n>>> +\ttest_cmp expect output\n>>> +'\n>> All right. (though I wonder if this tests catches all problems where\n>> BUG(\"bad generation skip\") could have been encountered.\n>\n> We will never know until we have this series running in the wild (and\n> even then, some features are very obscure) and enough people turn on\n> the config setting.\n>\n> One goal of the \"fsck and gc\" series is to get this feature running\n> during the rest of the test suite as much as possible, so we can get\n> additional coverage. Also to get more experience from the community\n> dogfooding the feature.\n\nSidenote: for two out of three features that change the view of history\nwe could also update commit-graph automatically:\n* the shortening or deepening of shallow clone could also re-calculate\n the commit graph (or invalidate it)\n* git-replace could check if the replacement modifies history, and if\n so, recalculate the commit graph (or invalidate it/check its validity)\n* there is no such possibility for grafts, but they are deprecated anyway\n\n-- \nJakub Narębski\n"},{"id":"346449","messageId":"86a7ti1a4h.fsf@gmail.com","threadId":"48212","inReplyTo":"3e3ae7e4-9efd-3046-c0c2-7b3bf12d0d81@gmail.com","subject":"Re: [PATCH v4 08/10] commit: add short-circuit to paint_down_to_common()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-02T13:05:02Z","receivedAt":"2018-05-02T13:05:12Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 4/30/2018 6:19 PM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n[...]\n>>> @@ -831,6 +834,13 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n, struc\n>>> \t\tstruct commit_list *parents;\n>>> \t\tint flags;\n>>> +\t\tif (commit->generation > last_gen)\n>>> +\t\t\tBUG(\"bad generation skip\");\n>>> +\t\tlast_gen = commit->generation;\n>> Shouldn't we provide more information about where the problem is to the\n>> user, to make it easier to debug the repository / commit-graph data?\n>>\n>> Good to have this sanity check here.\n>\n> This BUG() _should_ only be seen by developers who add callers which\n> do not load commits from the commit-graph file. There is a chance that\n> there are cases not covered by this patch and the added tests,\n> though. Hopefully we catch them all by dogfooding the feature before\n> turning it on by default.\n>\n> I can add the following to help debug these bad situations:\n>\n> +\t\t\tBUG(\"bad generation skip %d > %d at %s\",\n> +\t\t\t commit->generation, last_gen,\n> +\t\t\t oid_to_hex(&commit->object.oid));\n\nOn one hand, after thiking about this a bit, I agree that this BUG() is\nmore about catching the errors in Git code, rather than in repository.\n\nOn the other hand, the more detailed information could help determining\nwhat the problems is (e.g. that \"at \" is at HEAD).\n\nHopefully we won't see which is which, as it would mean bugs in Git ;))\n\n[...]\n>>> @@ -946,7 +956,7 @@ static int remove_redundant(struct commit **array, int cnt)\n>>> \t\t\tfilled_index[filled] = j;\n>>> \t\t\twork[filled++] = array[j];\n>>> \t\t}\n>>> -\t\tcommon = paint_down_to_common(array[i], filled, work);\n>>> +\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n>>\n>> Here we are interested not only if \"one\"/array[i] is reachable from\n>> \"twos\"/work, but also if \"twos\" is reachable from \"one\". Simple cutoff\n>> only works in one way, though I wonder if we couldn't use cutoff being\n>> minimum generation number of \"one\" and \"twos\" together.\n>>\n>> But that may be left for a separate commit (after checking that the\n>> above is correct).\n>>\n>> Not as simple and obvious as paint_down_to_common() used in\n>> in_merge_bases_any(), so it is all right.\n>\n> Thanks for reporting this. Since we are only concerned about\n> reachability in this method, it is a good candidate to use\n> min_generation. It is also subtle enough that we should leave it as a\n> separate commit.\n\nThanks for checking this, and for the followup.\n\n> Also, we can measure performance improvements\n> separately, as I will mention in my commit message (but I'll copy it\n> here):\n>\n>     For a copy of the Linux repository, we measured the following\n>     performance improvements:\n>\n>     git merge-base v3.3 v4.5\n>\n>     Before: 234 ms\n>      After: 208 ms\n>      Rel %: -11%\n>\n>     git merge-base v4.3 v4.5\n>\n>     Before: 102 ms\n>      After:  83 ms\n>      Rel %: -19%\n>\n>     The experiments above were chosen to demonstrate that we are\n>     improving the filtering of the merge-base set. In the first\n>     example, more time is spent walking the history to find the\n>     set of merge bases before the remove_redundant() call. The\n>     starting commits are closer together in the second example,\n>     therefore more time is spent in remove_redundant(). The relative\n>     change in performance differs as expected.\n\nNice.\n\nI was not expecting as much performance improvements as we got for\n--contains tests because remove_redundant() is a final step in longer\nprocess, dominated by man calculations. Still, nothing to sneeze about.\n\nBest regards,\n-- \nJakub Narębski\n\n"},{"id":"346450","messageId":"9a440cce-76ab-3d62-5a06-cf411fe35b48@gmail.com","threadId":"48212","inReplyTo":"86a7ti1a4h.fsf@gmail.com","subject":"Re: [PATCH v4 08/10] commit: add short-circuit to paint_down_to_common()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-02T13:42:42Z","receivedAt":"2018-05-02T13:42:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/2/2018 9:05 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>>     For a copy of the Linux repository, we measured the following\n>>     performance improvements:\n>>\n>>     git merge-base v3.3 v4.5\n>>\n>>     Before: 234 ms\n>>      After: 208 ms\n>>      Rel %: -11%\n>>\n>>     git merge-base v4.3 v4.5\n>>\n>>     Before: 102 ms\n>>      After:  83 ms\n>>      Rel %: -19%\n>>\n>>     The experiments above were chosen to demonstrate that we are\n>>     improving the filtering of the merge-base set. In the first\n>>     example, more time is spent walking the history to find the\n>>     set of merge bases before the remove_redundant() call. The\n>>     starting commits are closer together in the second example,\n>>     therefore more time is spent in remove_redundant(). The relative\n>>     change in performance differs as expected.\n> Nice.\n>\n> I was not expecting as much performance improvements as we got for\n> --contains tests because remove_redundant() is a final step in longer\n> process, dominated by man calculations. Still, nothing to sneeze about.\n\nOne reason these numbers are not too surprising is that \nremove_redundant() can demonstrate quadratic behavior. It is calculating \npair-wise reachability by starting a walk at each of the candidates (in \nthe worst case). In typical cases, the first walk marks many of the \nother candidates as redundant and we don't need to start walks from \nthose commits.\n\nA possible optimization could be to sort the candidates by descending \ngeneration so we find the first walk is likely to mark the rest as \nredundant. But this may already be the case if the candidates are added \nto the list in order of \"discovery\" which is already simulating this \nbehavior.\n\nThanks,\n-Stolee\n"},{"id":"346465","messageId":"861seu11at.fsf@gmail.com","threadId":"48212","inReplyTo":"0bd1ffe0-c727-653a-46a3-f9d4ea17bec2@gmail.com","subject":"Re: [PATCH v4 03/10] commit-graph: compute generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-02T16:15:38Z","receivedAt":"2018-05-02T16:15:49Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 4/29/2018 5:08 AM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n\n[...]\n>> It is a bit strange to me that the code uses get_be32 for reading, but\n>> htonl for writing. Is Git tested on non little-endian machines, like\n>> big-endian ppc64 or s390x, or on mixed-endian machines (or\n>> selectable-endian machines with data endianness set to non\n>> little-endian, like ia64)? If not, could we use for example openSUSE\n>> Build Service (https://build.opensuse.org/) for this?\n>\n> Since we are packing two values into 64 bits, I am using htonl() here\n> to arrange the 30-bit generation number alongside the 34-bit commit\n> date value, then writing with hashwrite(). The other 32-bit integers\n> are written with hashwrite_be32() to avoid translating this data\n> in-memory.\n\nO.K., so you are using what is more effective and easier to use.\nNice to know, thanks for the information.\n\n[...]\n>>> +static void compute_generation_numbers(struct commit** commits,\n>>> +\t\t\t\t int nr_commits)\n>>> +{\n[...]\n>>> +\tfor (i = 0; i < nr_commits; i++) {\n>>> +\t\tif (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n>>> +\t\t commits[i]->generation != GENERATION_NUMBER_ZERO)\n>>> +\t\t\tcontinue;\n\n[...]\n\n>>> +\tcompute_generation_numbers(commits.list, commits.nr);\n>>> +\n>> Nice and simple. All right.\n>>\n>> I guess that we do not pass \"struct packed_commit_list commits\" as\n>> argument to compute_generation_numbers instead of \"struct commit**\n>> commits.list\" and \"int commits.nr\" to compute_generation_numbers() to\n>> keep the latter nice and generic?\n>\n> Good catch. There is no reason to not use packed_commit_list here.\n\nActually, now that v5 shows how using packed_commit_list looks like, in\nmy opinion it looks uglier. And it might be easier to make mistake.\n\nAlso, depending on how compiler is able to optimize it, the version\npassing packed_commit_list as an argument has one more indirection\n(following two pointers) in the loop.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346511","messageId":"86sh79yolb.fsf@gmail.com","threadId":"48212","inReplyTo":"20180501124652.155781-1-dstolee@microsoft.com","subject":"Re: [PATCH v5 00/11] Compute and consume generation numbers","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-03T11:18:24Z","receivedAt":"2018-05-03T11:18:34Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Most of the changes from v4 are cosmetic, but there is one new commit:\n>\n> \tcommit: use generation number in remove_redundant()\n>\n> Other changes are non-functional, but do clarify things.\n\nI wonder if out perf framework in t/perf could help here to show\nperformance gains for the whole series. Though it may not include\noperations that are most helped by this one.\n\nFor commit-graph feature if would be nice, if feasible, to see changes\nin performance from before version, checking both state where feature is\nenabled to see the gains, and state where feature is disabled to see if\nthere are no performance regressions.\n\n>\n> Inter-diff from v4:\n\nO.K., now to commenting on inter-changes.\n\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index d9f2713efa..e1a883eb46 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -125,9 +125,10 @@ Future Work\n> walks aware of generation numbers to gain the performance benefits they\n> enable. This will mostly be accomplished by swapping a commit-date-ordered\n> priority queue with one ordered by generation number. The following\n> - operation is an important candidate:\n> + operations are important candidates:\n>\n> - 'log --topo-order'\n> + - 'tag --merged'\n>\n> - Currently, parse_commit_gently() requires filling in the root tree\n> object for a commit. This passes through lookup_tree() and consequently\n\nO.K., this is about discussion in \"branch --contains / tag --merged\ninconsistency\" thread:\n\n https://public-inbox.org/git/87fu3g67ry.fsf@lant.ki.iif.hu/t/#u\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index aebd242def..a8c337dd77 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -248,6 +248,7 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n> static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n> {\n> const unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n> + item->graph_pos = pos;\n> item->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n> }\n>\n\nMinor bugfix.\n\n> @@ -454,8 +455,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> else\n> packedDate[0] = 0;\n>\n> - if ((*list)->generation != GENERATION_NUMBER_INFINITY)\n> - packedDate[0] |= htonl((*list)->generation << 2);\n> + packedDate[0] |= htonl((*list)->generation << 2);\n>\n> packedDate[1] = htonl((*list)->date);\n> hashwrite(f, packedDate, 8);\n\nMinor bugfix.\n\n> @@ -589,18 +589,17 @@ static void close_reachable(struct packed_oid_list *oids)\n> }\n> }\n>\n> -static void compute_generation_numbers(struct commit** commits,\n> - int nr_commits)\n> +static void compute_generation_numbers(struct packed_commit_list* commits)\n> {\n> int i;\n> struct commit_list *list = NULL;\n>\n> - for (i = 0; i < nr_commits; i++) {\n> - if (commits[i]->generation != GENERATION_NUMBER_INFINITY &&\n> - commits[i]->generation != GENERATION_NUMBER_ZERO)\n> + for (i = 0; i < commits->nr; i++) {\n> + if (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n> + commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n> continue;\n>\n> - commit_list_insert(commits[i], &list);\n> + commit_list_insert(commits->list[i], &list);\n> while (list) {\n> struct commit *current = list->item;\n> struct commit_list *parent;\n\nRefactoring: signature change from pair of struct commit** + int to\nstruct packed_commit_list*.\n\nI think that it makes code a bit uglier for no gain, but that is just my\npersonal opinion; it is a matter of taste.\n\n> @@ -621,10 +620,10 @@ static void compute_generation_numbers(struct commit** commits,\n> if (all_parents_computed) {\n> current->generation = max_generation + 1;\n> pop_commit(&list);\n> - }\n>\n> - if (current->generation > GENERATION_NUMBER_MAX)\n> - current->generation = GENERATION_NUMBER_MAX;\n> + if (current->generation > GENERATION_NUMBER_MAX)\n> + current->generation = GENERATION_NUMBER_MAX;\n> + }\n> }\n> }\n> }\n\nBugfix (though it didn't result in wrong information writen out, just in\ninconsistent state in the middle of computation).\n\n> @@ -752,7 +751,7 @@ void write_commit_graph(const char *obj_dir,\n> if (commits.nr >= GRAPH_PARENT_MISSING)\n> die(_(\"too many commits to write graph\"));\n>\n> - compute_generation_numbers(commits.list, commits.nr);\n> + compute_generation_numbers(&commits);\n>\n> graph_name = get_commit_graph_filename(obj_dir);\n> fd = hold_lock_file_for_update(&lk, graph_name, 0);\n\nThe other side of signature change.\n\n> diff --git a/commit.c b/commit.c\n> index e2e16ea1a7..5064db4e61 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -835,7 +835,9 @@ static struct commit_list *paint_down_to_common(struct commit *one, int n,\n> int flags;\n>\n> if (commit->generation > last_gen)\n> - BUG(\"bad generation skip\");\n> + BUG(\"bad generation skip %8x > %8x at %s\",\n> + commit->generation, last_gen,\n> + oid_to_hex(&commit->object.oid));\n> last_gen = commit->generation;\n>\n> if (commit->generation < min_generation)\n\nMore detailed BUG() message, always nice to have.\n\n> @@ -947,6 +949,7 @@ static int remove_redundant(struct commit **array, int cnt)\n> parse_commit(array[i]);\n> for (i = 0; i < cnt; i++) {\n> struct commit_list *common;\n> + uint32_t min_generation = GENERATION_NUMBER_INFINITY;\n>\n> if (redundant[i])\n> continue;\n> @@ -955,8 +958,12 @@ static int remove_redundant(struct commit **array, int cnt)\n> continue;\n> filled_index[filled] = j;\n> work[filled++] = array[j];\n> +\n> + if (array[j]->generation < min_generation)\n> + min_generation = array[j]->generation;\n> }\n> - common = paint_down_to_common(array[i], filled, work, 0);\n> + common = paint_down_to_common(array[i], filled, work,\n> + min_generation);\n> if (array[i]->object.flags & PARENT2)\n> redundant[i] = 1;\n> for (j = 0; j < filled; j++)\n\nNew commit in series. Change looks quite short, gives measurable\nperformance gains (in appropriate case).\n\n> @@ -1073,7 +1080,7 @@ int in_merge_bases_many(struct commit *commit, int nr_reference, struct commit *\n> for (i = 0; i < nr_reference; i++) {\n> if (parse_commit(reference[i]))\n> return ret;\n> - if (min_generation > reference[i]->generation)\n> + if (reference[i]->generation < min_generation)\n> min_generation = reference[i]->generation;\n> }\n>\n>\n\nStyle change.\n\n> -- >8 --\n>\n> Derrick Stolee (11):\n> ref-filter: fix outdated comment on in_commit_list\n> commit: add generation number to struct commmit\n> commit-graph: compute generation numbers\n> commit: use generations in paint_down_to_common()\n> commit-graph: always load commit-graph information\n> ref-filter: use generation number for --contains\n> commit: use generation numbers for in_merge_bases()\n> commit: add short-circuit to paint_down_to_common()\n> commit: use generation number in remove_redundant()\n> merge: check config before loading commits\n> commit-graph.txt: update design document\n\nIt looks like the series is maturing nicely.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346560","messageId":"867eokzigo.fsf@gmail.com","threadId":"48212","inReplyTo":"20180501124652.155781-10-dstolee@microsoft.com","subject":"Re: [PATCH v5 09/11] commit: use generation number in remove_redundant()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-03T18:45:27Z","receivedAt":"2018-05-03T18:45:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The static remove_redundant() method is used to filter a list\n> of commits by removing those that are reachable from another\n> commit in the list. This is used to remove all possible merge-\n> bases except a maximal, mutually independent set.\n>\n> To determine these commits are independent, we use a number of\n> paint_down_to_common() walks and use the PARENT1, PARENT2 flags\n> to determine reachability. Since we only care about reachability\n> and not the full set of merge-bases between 'one' and 'twos', we\n> can use the 'min_generation' parameter to short-circuit the walk.\n>\n> When no commit-graph exists, there is no change in behavior.\n>\n> For a copy of the Linux repository, we measured the following\n> performance improvements:\n>\n> git merge-base v3.3 v4.5\n>\n> Before: 234 ms\n> After: 208 ms\n> Rel %: -11%\n>\n> git merge-base v4.3 v4.5\n>\n> Before: 102 ms\n> After: 83 ms\n> Rel %: -19%\n>\n> The experiments above were chosen to demonstrate that we are\n> improving the filtering of the merge-base set. In the first\n> example, more time is spent walking the history to find the\n> set of merge bases before the remove_redundant() call. The\n> starting commits are closer together in the second example,\n> therefore more time is spent in remove_redundant(). The relative\n> change in performance differs as expected.\n>\n> Reported-by: Jakub Narebski \n> Signed-off-by: Derrick Stolee \n\nGood description.\n\n> ---\n> commit.c | 7 ++++++-\n> 1 file changed, 6 insertions(+), 1 deletion(-)\n>\n\nLet me extend context a bit to make it easier to review.\n\n> diff --git a/commit.c b/commit.c\n> index 9875feec01..5064db4e61 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -949,6 +949,7 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\tparse_commit(array[i]);\n> \tfor (i = 0; i < cnt; i++) {\n> \t\tstruct commit_list *common;\n> +\t\tuint32_t min_generation = GENERATION_NUMBER_INFINITY;\n\nAs you have noticed, and how it is already fixed in 'pu' it should be\n\n +\t\tuint32_t min_generation = array[i]->generation;\n\n> \n> \t\tif (redundant[i])\n> \t\t\tcontinue;\n> @@ -957,8 +958,12 @@ static int remove_redundant(struct commit **array, int cnt)\n> \t\t\t\tcontinue;\n> \t\t\tfilled_index[filled] = j;\n> \t\t\twork[filled++] = array[j];\n> +\n> +\t\t\tif (array[j]->generation < min_generation)\n> +\t\t\t\tmin_generation = array[j]->generation;\n\nremove_redundant() checks if i-th commit is reachable from commits\ni+1..cnt, and vice versa - via checking PARENT1 and PARENT2 flag,\nrespectively.\n\nAs you have noticed this means that the min_generation cutoff should be\nminimum of array[i]->generation, and all of array[j]->generation for\nj=i+1..cnt. There is no reason going further down if we are interested\nonly in reachability, and not actually in merge bases.\n\n> \t\t}\n> -\t\tcommon = paint_down_to_common(array[i], filled, work, 0);\n> +\t\tcommon = paint_down_to_common(array[i], filled, work,\n> +\t\t\t\t\t min_generation);\n> \t\tif (array[i]->object.flags & PARENT2)\n> \t\t\tredundant[i] = 1;\n> \t\tfor (j = 0; j < filled; j++)\n \t\t\tif (work[j]->object.flags & PARENT1)\n \t\t\t\tredundant[filled_index[j]] = 1;\n\nBeside this issue, nice and simple speedup. Good.\n\nBest,\n-- \nJakub Narębski\n"}]}