{"thread":{"id":"56035","subject":"[PATCH 2/7] merge-ort: add some more explanations in collect_merge_info_callback()","startedAt":"2021-07-01T03:46:28Z","lastAt":"2021-07-21T04:25:18Z","messageCount":52,"participants":["Elijah Newren via GitGitGadget","Ævar Arnfjörð Bjarmason","Elijah Newren","Bagas Sanjaya","Derrick Stolee","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"428867","messageId":"8aea3713902b7d006f527ccd76faf6f944529bdb.1625111177.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH 2/7] merge-ort: add some more explanations in collect_merge_info_callback()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:12Z","receivedAt":"2021-07-01T03:46:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe previous patch possibly raises some questions about whether\nadditional cases in collect_merge_info_callback() can be handled early.\nAdd some explanations in the form of comments to help explain these\nbetter. While we're at it, add a few comments to denote what a few\nboolean '0' or '1' values stand for.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 20 +++++++++++++++-----\n 1 file changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 6299b4f9413..843fa693145 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1018,8 +1018,8 @@ static int collect_merge_info_callback(int n,\n \tif (side1_matches_mbase && side2_matches_mbase) {\n \t\t/* mbase, side1, & side2 all match; use mbase as resolution */\n \t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n-\t\t\t\tnames, names+0, mbase_null, 0,\n-\t\t\t\tfilemask, dirmask, 1);\n+\t\t\t\tnames, names+0, mbase_null, 0 /* df_conflict */,\n+\t\t\t\tfilemask, dirmask, 1 /* resolved */);\n \t\treturn mask;\n \t}\n \n@@ -1061,14 +1061,24 @@ static int collect_merge_info_callback(int n,\n \t}\n \n \t/*\n-\t * Gather additional information used in rename detection.\n+\t * Sometimes we can tell that a source path need not be included in\n+\t * rename detection -- namely, whenever either\n+\t * side1_matches_mbase && side2_null\n+\t * or\n+\t * side2_matches_mbase && side1_null\n+\t * However, we call collect_rename_info() even in those cases,\n+\t * because exact renames are cheap and would let us remove both a\n+\t * source and destination path. We'll cull the unneeded sources\n+\t * later.\n \t */\n \tcollect_rename_info(opt, names, dirname, fullpath,\n \t\t\t filemask, dirmask, match_mask);\n \n \t/*\n-\t * Record information about the path so we can resolve later in\n-\t * process_entries.\n+\t * None of the special cases above matched, so we have a\n+\t * provisional conflict. (Rename detection might allow us to\n+\t * unconflict some more cases, but that comes later so all we can\n+\t * do now is record the different non-null file hashes.)\n \t */\n \tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n \t\t\tnames, NULL, 0, df_conflict, filemask, dirmask, 0);\n-- \ngitgitgadget\n\n"},{"id":"428868","messageId":"pull.988.git.1625111177.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":null,"subject":"[PATCH 0/7] Optimization batch 14: trivial directory resolution","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:10Z","receivedAt":"2021-07-01T03:46:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends textually on ort-perf-batch-12, but is semantically\nindependent. (It is both semantically and textually independent of\nort-perf-batch-13.)\n\nMost of my previous series dramatically accelerated cases with lots of\nrenames, while providing comparatively minor benefits for cases with few or\nno renames. This series is the opposite; it provides huge benefits when\nthere are few or no renames, and comparatively smaller (though still quite\ndecent) benefits for cases with many uncached renames.\n\n=== Basic Optimization idea ===\n\nunpack_trees has had a concept of trivial merges for individual files (see\nDocumentation/technical/trivial-merge.txt). The same idea can be applied in\nmerge-ort. It'd be really nice to extend that idea to trees as well, as it\ncould provide a huge performance boost; sadly however, applying it in\ngeneral would wreck both regular rename detection (the unmatched side can\nhave new files that serve as potential destinations in rename detection) and\ndirectory rename detection (the unmatched side could have a new directory\nthat was moved into it).\n\nIf we somehow knew rename detection wasn't needed, we could do trivial\ndirectory resolution. In the past, this wasn't possible. However...\n\nWith recent optimizations we have created a possibility to do trivial\ndirectory resolutions in some cases. These came from the addition of the\n\"skipping irrelevant renames\" optimizations (from ort-perf-batch-9 and\nort-perf-batch-10), and in particular noting that we added an ability to\nentirely skip rename detection in commit f89b4f2bee (\"merge-ort: skip rename\ndetection entirely if possible\", 2021-03-11) when there are no relevant\nsources. We can detect if there are no relevant sources without recursing\ninto the directories in question.\n\nAs a cherry on top, the caching of renames (from ort-perf-batch-11) allows\nus to cover additional cases.\n\nThis series is all about adding all the special checks needed to safely\nperform trival directory resolutions.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 5.235 s ± 0.042 s 204.2 ms ± 3.0 ms\nmega-renames: 9.419 s ± 0.107 s 1.076 s ± 0.015 s\njust-one-mega: 480.1 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with were:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\nElijah Newren (7):\n merge-ort: resolve paths early when we have sufficient information\n merge-ort: add some more explanations in collect_merge_info_callback()\n merge-ort: add data structures for allowable trivial directory\n resolves\n merge-ort: add a handle_deferred_entries() helper function\n merge-ort: defer recursing into directories when merge base is matched\n merge-ort: avoid recursing into directories when we don't need to\n merge-ort: restart merge with cached renames to reduce process entry\n cost\n\n merge-ort.c | 403 +++++++++++++++++++++++++++-\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 393 insertions(+), 12 deletions(-)\n\n\nbase-commit: 2eeee12b02e441ac05054a5a5ecbcea6964a1e6b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-988%2Fnewren%2Fort-perf-batch-14-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-988/newren/ort-perf-batch-14-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/988\n-- \ngitgitgadget\n"},{"id":"428869","messageId":"5dca982c0b061ae8d6335d4f22b78811dced054a.1625111177.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH 1/7] merge-ort: resolve paths early when we have sufficient information","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:11Z","receivedAt":"2021-07-01T03:46:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen there are no directories involved at a given path, and all three\nsides have a file at that path, and two of the three sides of history\nmatch, we can immediately resolve the merge of that path in\ncollect_merge_info() and do not need to wait until process_entries().\n\nThis is actually a very minor improvement: half the time when I run it,\nI see an improvement; the other half a slowdown. It seems to be in the\nrange of noise. However, this idea serves as the beginning of some\nbigger optimizations coming in the following patches.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e3a5dfc7b31..6299b4f9413 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1023,6 +1023,43 @@ static int collect_merge_info_callback(int n,\n \t\treturn mask;\n \t}\n \n+\t/*\n+\t * If the sides match, and all three paths are present and are\n+\t * files, then we can take either as the resolution. We can't do\n+\t * this with trees, because there may be rename sources from the\n+\t * merge_base.\n+\t */\n+\tif (sides_match && filemask == 0x07) {\n+\t\t/* use side1 (== side2) version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/*\n+\t * If side1 matches mbase and all three paths are present and are\n+\t * files, then we can use side2 as the resolution. We cannot\n+\t * necessarily do so this for trees, because there may be rename\n+\t * destinations within side2.\n+\t */\n+\tif (side1_matches_mbase && filemask == 0x07) {\n+\t\t/* use side2 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+2, side2_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/* Similar to above but swapping sides 1 and 2 */\n+\tif (side2_matches_mbase && filemask == 0x07) {\n+\t\t/* use side1 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n \t/*\n \t * Gather additional information used in rename detection.\n \t */\n-- \ngitgitgadget\n\n"},{"id":"428870","messageId":"317553eadb68ce162b5ebea24045a9ca75342e91.1625111177.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH 5/7] merge-ort: defer recursing into directories when merge base is matched","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:15Z","receivedAt":"2021-07-01T03:46:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen one side of history matches the merge base (including when the\nmerge base has no entry for the given directory), have\ncollect_merge_info_callback() defer recursing into the directory. To\nensure those entries are eventually handled, add a call to\nhandled_deferred_entries() in collect_merge_info() after\ntraverse_trees() returns.\n\nNote that the condition in collect_merge_info_callback() may look more\ncomplicated than necessary at first glance;\nrenames->trivial_merges_okay[side] is always true until\nhandle_deferred_entries() is called, and possible_trivial_merges[side]\nis always empty right now (and in the future won't be filled until\nhandle_deferred_entries() is called). However, when\nhandle_deferred_entries() calls traverse_trees() for the relevant\ndeferred directories, those traverse_trees() calls will once again end\nup in collect_merge_info_callback() for all the entries under those\nsubdirectories. The extra conditions are there for such deferred cases\nand will be used more as we do more with those variables.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 32 ++++++++++++++++++++++++++++++--\n 1 file changed, 30 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex eb0e18d7546..bf0712d18a0 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1141,8 +1141,35 @@ static int collect_merge_info_callback(int n,\n \t\tstruct tree_desc t[3];\n \t\tvoid *buf[3] = {NULL, NULL, NULL};\n \t\tconst char *original_dir_name;\n-\t\tint i, ret;\n+\t\tint i, ret, side;\n \n+\t\t/*\n+\t\t * Check for whether we can avoid recursing due to one side\n+\t\t * matching the merge base. The side that does NOT match is\n+\t\t * the one that might have a rename destination we need.\n+\t\t */\n+\t\tassert(!side1_matches_mbase || !side2_matches_mbase);\n+\t\tside = side1_matches_mbase ? MERGE_SIDE2 :\n+\t\t\tside2_matches_mbase ? MERGE_SIDE1 : MERGE_BASE;\n+\t\tif (filemask == 0 && (dirmask == 2 || dirmask == 4)) {\n+\t\t\t/*\n+\t\t\t * Also defer recursing into new directories; set up a\n+\t\t\t * few variables to let us do so.\n+\t\t\t */\n+\t\t\tci->match_mask = (7 - dirmask);\n+\t\t\tside = dirmask / 2;\n+\t\t}\n+\t\tif (renames->dir_rename_mask != 0x07 &&\n+\t\t (side != MERGE_BASE) &&\n+\t\t renames->trivial_merges_okay[side] &&\n+\t\t !strset_contains(&renames->target_dirs[side], pi.string)) {\n+\t\t\tstrintmap_set(&renames->possible_trivial_merges[side],\n+\t\t\t\t pi.string, renames->dir_rename_mask);\n+\t\t\trenames->dir_rename_mask = prev_dir_rename_mask;\n+\t\t\treturn mask;\n+\t\t}\n+\n+\t\t/* We need to recurse */\n \t\tci->match_mask &= filemask;\n \t\tnewinfo = *info;\n \t\tnewinfo.prev = info;\n@@ -1196,7 +1223,6 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n-MAYBE_UNUSED\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1285,6 +1311,8 @@ static int collect_merge_info(struct merge_options *opt,\n \n \ttrace2_region_enter(\"merge\", \"traverse_trees\", opt->repo);\n \tret = traverse_trees(NULL, 3, t, &info);\n+\tif (ret == 0)\n+\t\tret = handle_deferred_entries(opt, &info);\n \ttrace2_region_leave(\"merge\", \"traverse_trees\", opt->repo);\n \n \treturn ret;\n-- \ngitgitgadget\n\n"},{"id":"428871","messageId":"7e28323b624ad2d8d12123783f00f5a8fbb248e8.1625111177.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH 4/7] merge-ort: add a handle_deferred_entries() helper function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:14Z","receivedAt":"2021-07-01T03:46:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn order to allow trivial directory resolution, we first need to be able\nto gather more information to determine if the optimization is safe. To\nenable that, we need a way of deferring the recursion into the directory\nuntil a later time. Naturally, deferring the entry into a subtree means\nthat we need some function that will later recurse into the subdirectory\nexactly the same way that collect_merge_info_callback() would have done.\n\nAdd a helper function that does this. For now this function is not used\nbut a subsequent commit will change that. Future commits will also make\nthe function sometimes resolve directories instead of traversing inside.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 64 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 3d3f00b3b45..eb0e18d7546 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1196,6 +1196,70 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+MAYBE_UNUSED\n+static int handle_deferred_entries(struct merge_options *opt,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tstruct rename_info *renames = &opt->priv->renames;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\tint side, ret = 0;\n+\n+\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n+\t\trenames->trivial_merges_okay[side] = 0;\n+\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n+\t\t\tstruct conflict_info *ci;\n+\t\t\tunsigned dirmask;\n+\t\t\tstruct tree_desc t[3];\n+\t\t\tvoid *buf[3] = {NULL,};\n+\t\t\tint i;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\t\t\tdirmask = ci->dirmask;\n+\n+\t\t\tinfo->name = path;\n+\t\t\tinfo->namelen = strlen(path);\n+\t\t\tinfo->pathlen = info->namelen + 1;\n+\n+\t\t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n+\t\t\t\tif (i == 1 && ci->match_mask == 3)\n+\t\t\t\t\tt[1] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 5)\n+\t\t\t\t\tt[2] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 6)\n+\t\t\t\t\tt[2] = t[1];\n+\t\t\t\telse {\n+\t\t\t\t\tconst struct object_id *oid = NULL;\n+\t\t\t\t\tif (dirmask & 1)\n+\t\t\t\t\t\toid = &ci->stages[i].oid;\n+\t\t\t\t\tbuf[i] = fill_tree_descriptor(opt->repo,\n+\t\t\t\t\t\t\t\t t+i, oid);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tci->match_mask &= ci->filemask;\n+\t\t\topt->priv->current_dir_name = path;\n+\t\t\trenames->dir_rename_mask = dir_rename_mask;\n+\t\t\tif (renames->dir_rename_mask == 0 ||\n+\t\t\t renames->dir_rename_mask == 0x07)\n+\t\t\t\tret = traverse_trees(NULL, 3, t, info);\n+\t\t\telse\n+\t\t\t\tret = traverse_trees_wrapper(NULL, 3, t, info);\n+\n+\t\t\tfor (i = MERGE_BASE; i <= MERGE_SIDE2; i++)\n+\t\t\t\tfree(buf[i]);\n+\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n static int collect_merge_info(struct merge_options *opt,\n \t\t\t struct tree *merge_base,\n \t\t\t struct tree *side1,\n-- \ngitgitgadget\n\n"},{"id":"428872","messageId":"f7ac01055d9d2e9e2dfdfd780ff7f10fbfd05d5b.1625111177.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH 3/7] merge-ort: add data structures for allowable trivial directory resolves","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:13Z","receivedAt":"2021-07-01T03:46:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs noted a few commits ago, we can resolve individual files early if all\nthree sides of the merge have a file at the path and two of the three\nsides match. We would really like to do the same thing with\ndirectories, because being able to do a trivial directory resolve means\nwe don't have to recurse into the directory, potentially saving us a\nhuge amount of time in both collect_merge_info() and process_entries().\nUnfortunately, resolving directories early would mean missing any\nrenames whose source or destination is underneath that directory.\n\nIf we somehow knew there weren't any renames under the directory in\nquestion, then we could resolve it early. Sadly, it is impossible to\ndetermine whether there are renames under the directory in question\nwithout recursing into it, and this has traditionally kept us from ever\nimplementing such an optimization.\n\nIn commit f89b4f2bee (\"merge-ort: skip rename detection entirely if\npossible\", 2021-03-11), we added an additional reason that rename\ndetection could be skipped entirely -- namely, if no *relevant* sources\nwere present. Without completing collect_merge_info_callback(), we do\nnot yet know if there are no relevant sources. However, we do know that\nif the current directory on one side matches the merge base, then every\nsource file within that directory will not be RELEVANT_CONTENT, and a\nfew simple checks can often let us rule out RELEVANT_LOCATION as well.\nThis suggests we can just defer recursing into such directories until\nthe end of collect_merge_info.\n\nSince the deferred directories are known to not add any relevant sources\ndue to the above properties, then if there are no relevant sources after\nwe've traversed all paths other than the deferred ones, then we know\nthere are not any relevant sources. Under those conditions, rename\ndetection is unnecessary, and that means we can resolve the deferred\ndirectories without recursing into them.\n\nNote that the logic for skipping rename detection was also modified\nfurther in commit 76e253793c (\"merge-ort, diffcore-rename: employ cached\nrenames when possible\", 2021-01-30); in particular rename detection can\nbe skipped if we already have cached renames for each relevant source.\nWe can take advantage of this information as well with our deferral of\nrecursing into directories where one side matches the merge base.\n\nAdd some data structures that we will use to do these deferrals, with\nsome lengthy comments explaining their purpose.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 843fa693145..3d3f00b3b45 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -119,6 +119,51 @@ struct rename_info {\n \t */\n \tstruct strintmap relevant_sources[3];\n \n+\t/*\n+\t * possible_trivial_merges: directories we defer recursing into\n+\t *\n+\t * possible_trivial_merges is a map of directory names to\n+\t * dir_rename_mask. When we detect that a directory is unchanged on\n+\t * one side, we can sometimes resolve the directory without recursing\n+\t * into it. Renames are the only things that can prevent such an\n+\t * optimization. However, for rename sources:\n+\t * - If no parent directory needed directory rename detection, then\n+\t * no path under such a directory can be a relevant_source.\n+\t * and for rename destinations:\n+\t * - If no cached rename has a target path under the directory AND\n+\t * - If there are no unpaired relevant_sources elsewhere in the\n+\t * repository\n+\t * then we don't need any path under this directory for a rename\n+\t * destination. The only way to know the last item above is to defer\n+\t * handling such directories until the end of collect_merge_info(),\n+\t * in handle_deferred_entries().\n+\t *\n+\t * For each we store dir_rename_mask, since that's the only bit of\n+\t * information we need, other than the path, to resume the recursive\n+\t * traversal.\n+\t */\n+\tstruct strintmap possible_trivial_merges[3];\n+\n+\t/*\n+\t * trivial_merges_okay: if trivial directory merges are okay\n+\t *\n+\t * See possible_trivial_merges above. The \"no unpaired\n+\t * relevant_sources elsewhere in the repository\" is a single boolean\n+\t * per merge side, which we store here. Note that while 0 means no,\n+\t * 1 only means \"maybe\" rather than \"yes\"; we optimistically set it\n+\t * to 1 initially and only clear when we determine it is unsafe to\n+\t * do trivial directory merges.\n+\t */\n+\tunsigned trivial_merges_okay[3];\n+\n+\t/*\n+\t * target_dirs: ancestor directories of rename targets\n+\t *\n+\t * target_dirs contains all directory names that are an ancestor of\n+\t * any rename destination.\n+\t */\n+\tstruct strset target_dirs[3];\n+\n \t/*\n \t * dir_rename_mask:\n \t * 0: optimization removing unmodified potential rename source okay\n@@ -490,6 +535,9 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrintmap_func(&renames->dirs_removed[i]);\n \t\tstrmap_func(&renames->dir_renames[i], 0);\n \t\tstrintmap_func(&renames->relevant_sources[i]);\n+\t\tstrintmap_func(&renames->possible_trivial_merges[i]);\n+\t\tstrset_func(&renames->target_dirs[i]);\n+\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n \t\tif (i != renames->cached_pairs_valid_side) {\n@@ -4045,12 +4093,17 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n \t\t\t\t\t -1 /* explicitly invalid */,\n \t\t\t\t\t NULL, 0);\n+\t\tstrintmap_init_with_options(&renames->possible_trivial_merges[i],\n+\t\t\t\t\t 0, NULL, 0);\n+\t\tstrset_init_with_options(&renames->target_dirs[i],\n+\t\t\t\t\t NULL, 1);\n \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_target_names[i],\n \t\t\t\t\t NULL, 0);\n+\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n \t}\n \n \t/*\n-- \ngitgitgadget\n\n"},{"id":"428873","messageId":"3409a6cd631deb361d3ecb94491c0c297c52fb53.1625111177.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH 6/7] merge-ort: avoid recursing into directories when we don't need to","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:16Z","receivedAt":"2021-07-01T03:46:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis combines the work of the last several patches, and implements the\nconditions when we don't need to recurse into directories. It's perhaps\neasiest to see the logic by separating the fact that a directory might\nhave both rename sources and rename destinations:\n\n * rename sources: only files present in the merge base can serve as\n rename sources, and only when one side deletes that file. When the\n tree on one side matches the merge base, that means every file\n within the subtree matches the merge base. This means that the\n skip-irrelevant-rename-detection optimization from before kicks in\n and we don't need any of these files as rename sources.\n\n * rename destinations: the tree that does not match the merge base\n might have newly added and hence unmatched destination files.\n This is what usually prevents us from doing trivial directory\n resolutions in the merge machinery. However, the fact that we have\n deferred recursing into this directory until the end means we know\n whether there are any unmatched relevant potential rename sources\n elsewhere in this merge. If there are no unmatched such relevant\n sources anywhere, then there is no need to look for unmatched\n potential rename destinations to match them with.\n\nThis informs our algorithm:\n * Search through relevant_sources; if we have entries, they better all\n be reflected in cached_pairs or cached_irrelevant, otherwise they\n represent an unmatched potential rename source (causing the\n optimization to be disallowed).\n * For any relevant_source represented in cached_pairs, we do need to\n to make sure to get the destination for each source, meaning we need\n to recurse into any ancestor directories of those destinations.\n * Once we've recursed into all the rename destinations for any\n relevant_sources in cached_pairs, we can then do the trivial\n directory resolution for the remaining directories.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 5.235 s ± 0.042 s 205.1 ms ± 3.8 ms\n mega-renames: 9.419 s ± 0.107 s 1.564 s ± 0.010 s\n just-one-mega: 480.1 ms ± 3.9 ms 479.5 ms ± 3.9 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 98 insertions(+), 3 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex bf0712d18a0..c9cf7a158c8 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1223,6 +1223,18 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+static void resolve_trivial_directory_merge(struct conflict_info *ci, int side)\n+{\n+\tVERIFY_CI(ci);\n+\tassert((side == 1 && ci->match_mask == 5) ||\n+\t (side == 2 && ci->match_mask == 3));\n+\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n+\tci->merged.result.mode = ci->stages[side].mode;\n+\tci->merged.is_null = is_null_oid(&ci->stages[side].oid);\n+\tci->match_mask = 0;\n+\tci->merged.clean = 1; /* (ci->filemask == 0); */\n+}\n+\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1232,9 +1244,71 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tint side, ret = 0;\n \n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n-\t\trenames->trivial_merges_okay[side] = 0;\n-\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n-\t\t\t\t\t &iter, entry) {\n+\t\tunsigned optimization_okay = 1;\n+\t\tstruct strintmap copy;\n+\n+\t\t/* Loop over the set of paths we need to know rename info for */\n+\t\tstrset_for_each_entry(&renames->relevant_sources[side],\n+\t\t\t\t &iter, entry) {\n+\t\t\tchar *rename_target, *dir, *dir_marker;\n+\t\t\tstruct strmap_entry *e;\n+\n+\t\t\t/*\n+\t\t\t * if we don't know delete/rename info for this path,\n+\t\t\t * then we need to recurse into all trees to get all\n+\t\t\t * adds to make sure we have it.\n+\t\t\t */\n+\t\t\tif (strset_contains(&renames->cached_irrelevant[side],\n+\t\t\t\t\t entry->key))\n+\t\t\t\tcontinue;\n+\t\t\te = strmap_get_entry(&renames->cached_pairs[side],\n+\t\t\t\t\t entry->key);\n+\t\t\tif (!e) {\n+\t\t\t\toptimization_okay = 0;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/* If this is a delete, we have enough info already */\n+\t\t\trename_target = e->value;\n+\t\t\tif (!rename_target)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* If we already walked the rename target, we're good */\n+\t\t\tif (strmap_contains(&opt->priv->paths, rename_target))\n+\t\t\t\tcontinue;\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, we need to get a list of directories that\n+\t\t\t * will need to be recursed into to get this\n+\t\t\t * rename_target.\n+\t\t\t */\n+\t\t\tdir = xstrdup(rename_target);\n+\t\t\twhile ((dir_marker = strrchr(dir, '/'))) {\n+\t\t\t\t*dir_marker = '\\0';\n+\t\t\t\tif (strset_contains(&renames->target_dirs[side],\n+\t\t\t\t\t\t dir))\n+\t\t\t\t\tbreak;\n+\t\t\t\tstrset_add(&renames->target_dirs[side], dir);\n+\t\t\t}\n+\t\t\tfree(dir);\n+\t\t}\n+\t\trenames->trivial_merges_okay[side] = optimization_okay;\n+\t\t/*\n+\t\t * We need to recurse into any directories in\n+\t\t * possible_trivial_merges[side] found in target_dirs[side].\n+\t\t * But when we recurse, we may need to queue up some of the\n+\t\t * subdirectories for possible_trivial_merges[side]. Since\n+\t\t * we can't safely iterate through a hashmap while also adding\n+\t\t * entries, move the entries into 'copy', iterate over 'copy',\n+\t\t * and then we'll also iterate anything added into\n+\t\t * possible_trivial_merges[side] once this loop is done.\n+\t\t */\n+\t\tcopy = renames->possible_trivial_merges[side];\n+\t\tstrintmap_init_with_options(&renames->possible_trivial_merges[side],\n+\t\t\t\t\t 0,\n+\t\t\t\t\t NULL,\n+\t\t\t\t\t 0);\n+\t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n \t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n \t\t\tstruct conflict_info *ci;\n@@ -1247,6 +1321,13 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tVERIFY_CI(ci);\n \t\t\tdirmask = ci->dirmask;\n \n+\t\t\tif (optimization_okay &&\n+\t\t\t !strset_contains(&renames->target_dirs[side],\n+\t\t\t\t\t path)) {\n+\t\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tinfo->name = path;\n \t\t\tinfo->namelen = strlen(path);\n \t\t\tinfo->pathlen = info->namelen + 1;\n@@ -1282,6 +1363,20 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tif (ret < 0)\n \t\t\t\treturn ret;\n \t\t}\n+\t\tstrintmap_clear(©);\n+\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tstruct conflict_info *ci;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\n+\t\t\tassert(renames->trivial_merges_okay[side] &&\n+\t\t\t !strset_contains(&renames->target_dirs[side],\n+\t\t\t\t\t\tpath));\n+\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t}\n \t}\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"428874","messageId":"76bc73262c4d01db12b2c0fa401b7ccad2988e1c.1625111177.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T03:46:17Z","receivedAt":"2021-07-01T03:46:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe merge algorithm mostly consists of the following three functions:\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\nPrior to the trivial directory resolution optimization of the last half\ndozen commits, process_entries() was consistently the slowest, followed\nby collect_merge_info(), then detect_and_process_renames(). When the\ntrivial directory resolution applies, it often dramatically decreases\nthe amount of time spent in the two slower functions.\n\nLooking at the performance results in the previous commit, the trivial\ndirectory resolution optimization helps amazingly well when there are no\nrelevant renames. It also helps really well when reapplying a long\nseries of linear commits (such as in a rebase or cherry-pick), since the\nrelevant renames may well be cached from the first reapplied commit.\nBut when there are any relevant renames that are not cached (represented\nby the just-one-mega testcase), then the optimization does not help at\nall.\n\nOften, I noticed that when the optimization does not apply, it is\nbecause there are a handful of relevant sources -- maybe even only one.\nIt felt frustrating to need to recurse into potentially hundreds or even\nthousands of directories just for a single rename, but it was needed for\ncorrectness.\n\nHowever, staring at this list of functions and noticing that\nprocess_entries() is the most expensive and knowing I could avoid it if\nI had cached renames suggested a simple idea: change\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\ninto\n collect_merge_info()\n detect_and_process_renames()\n \n collect_merge_info()\n detect_and_process_renames()\n process_entries()\n\nThis may seem odd and look like more work. However, note that although\nwe run collect_merge_info() twice, the second time we get to employ\ntrivial directory resolves, which makes it much faster, so the increased\ntime in collect_merge_info() is small. While we run\ndetect_and_process_renames() again, all renames are cached so it's\nnearly a no-op (we don't call into diffcore_rename_extended() but we do\nhave a little bit of data structure checking and fixing up). And the\nbig payoff comes from the fact that process_entries(), will be much\nfaster due to having far fewer entries to process.\n\nThis restarting only makes sense if we can save recursing into enough\ndirectories to make it worth our while. Introduce a simple heuristic to\nguide this. Note that this heuristic uses a \"wanted_factor\" that I have\nvirtually no actual real world data for, just some back-of-the-envelope\nquasi-scientific calculations that I included in some comments and then\nplucked a simple round number out of thin air. It could be that\ntweaking this number to make it either higher or lower improves the\noptimization. (There's slightly more here; when I first introduced this\noptimization, I used a factor of 10, because I was completely confident\nit was big enough to not cause slowdowns in special cases. I was\ncertain it was higher than needed. Several months later, I added the\nrough calculations which make me think the optimal number is close to 2;\nbut instead of pushing to the limit, I just bumped it to 3 to reduce the\nrisk that there are special cases where this optimization can result in\nslowing down the code a little. If the ratio of path counts is below 3,\nwe probably will only see minor performance improvements at best\nanyway.)\n\nAlso, note that while the diffstat looks kind of long (nearly 100\nlines), more than half of it is in two comments explaining how things\nwork.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 205.1 ms ± 3.8 ms 204.2 ms ± 3.0 ms\n mega-renames: 1.564 s ± 0.010 s 1.076 s ± 0.015 s\n just-one-mega: 479.5 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 106 ++++++++++++++++++++++++++--\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 101 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex c9cf7a158c8..51274f40059 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -209,6 +209,7 @@ struct rename_info {\n \t * MERGE_SIDE2: cached data from side2 can be reused\n \t * MERGE_SIDE1: cached data from side1 can be reused\n \t * 0: no cached data can be reused\n+\t * -1: See redo_after_renames; both sides can be reused.\n \t */\n \tint cached_pairs_valid_side;\n \n@@ -254,6 +255,28 @@ struct rename_info {\n \t */\n \tstruct strset cached_irrelevant[3];\n \n+\t/*\n+\t * redo_after_renames: optimization flag for \"restarting\" the merge\n+\t *\n+\t * Sometimes it pays to detect renames, cache them, and then\n+\t * restart the merge operation from the beginning. The reason for\n+\t * this is that when we know where all the renames are, we know\n+\t * whether a certain directory has any paths under it affected --\n+\t * and if a directory is not affected then it permits us to do\n+\t * trivial tree merging in more cases. Doing trivial tree merging\n+\t * prevents the need to run process_entry() on every path\n+\t * underneath trees that can be trivially merged, and\n+\t * process_entry() is more expensive than collect_merge_info() --\n+\t * plus, the second collect_merge_info() will be much faster since\n+\t * it doesn't have to recurse into the relevant trees.\n+\t *\n+\t * Values for this flag:\n+\t * 0 = don't bother, not worth it (or conditions not yet checked)\n+\t * 1 = conditions for optimization met, optimization worthwhile\n+\t * 2 = we already did it (don't restart merge yet again)\n+\t */\n+\tunsigned redo_after_renames;\n+\n \t/*\n \t * needed_limit: value needed for inexact rename detection to run\n \t *\n@@ -540,7 +563,8 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n-\t\tif (i != renames->cached_pairs_valid_side) {\n+\t\tif (i != renames->cached_pairs_valid_side &&\n+\t\t -1 != renames->cached_pairs_valid_side) {\n \t\t\tstrset_func(&renames->cached_target_names[i]);\n \t\t\tstrmap_func(&renames->cached_pairs[i], 1);\n \t\t\tstrset_func(&renames->cached_irrelevant[i]);\n@@ -1242,7 +1266,9 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n \tint side, ret = 0;\n+\tint path_count_before, path_count_after = 0;\n \n+\tpath_count_before = strmap_get_size(&opt->priv->paths);\n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n \t\tunsigned optimization_okay = 1;\n \t\tstruct strintmap copy;\n@@ -1377,7 +1403,51 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t\t\tpath));\n \t\t\tresolve_trivial_directory_merge(ci, side);\n \t\t}\n+\t\tif (!optimization_okay || path_count_after)\n+\t\t\tpath_count_after = strmap_get_size(&opt->priv->paths);\n \t}\n+\tif (path_count_after) {\n+\t\t/*\n+\t\t * Not sure were the right cut-off is for the optimization\n+\t\t * to redo collect_merge_info after we've cached the\n+\t\t * regular renames is. Basically, collect_merge_info(),\n+\t\t * detect_regular_renames(), and process_entries() are\n+\t\t * similar costs and all big tent poles. Caching the\n+\t\t * result of detect_regular_renames() means that redoing\n+\t\t * that one function will cost us virtually 0 extra, so it\n+\t\t * depends on the other two functions, which are both O(N)\n+\t\t * cost in the number of paths. Thus, it makes sense that\n+\t\t * if we can cut the number of paths in half, then redoing\n+\t\t * collect_merge_info() at half cost in order to get\n+\t\t * process_entries() at half cost should be about equal\n+\t\t * cost. If we can cut by more than half, then we would\n+\t\t * win. The fact that process_entries() is about 10%-20%\n+\t\t * more expensive than collect_merge_info() suggests we\n+\t\t * could make the factor be less than two. The fact that\n+\t\t * even when we have renames cached, we still have to\n+\t\t * traverse down to the individual (relevant) renames,\n+\t\t * which suggests we should perhaps use a bigger factor.\n+\t\t *\n+\t\t * The exact number isn't critical, since the code will\n+\t\t * work even if we get the factor wrong -- it just might be\n+\t\t * slightly slower if we're a bit off. For now, just error\n+\t\t * on the side of a bigger fudge. For the linux kernel\n+\t\t * testcases I was looking at with massive renames, the\n+\t\t * ratio came in around 50 to 250, which clearly would\n+\t\t * trigger this optimization and provided some *very* nice\n+\t\t * speedups.\n+\t\t */\n+\t\tint wanted_factor = 3;\n+\n+\t\t/* We should only redo collect_merge_info one time */\n+\t\tassert(renames->redo_after_renames == 0);\n+\n+\t\tif (path_count_after / path_count_before > wanted_factor) {\n+\t\t\trenames->redo_after_renames = 1;\n+\t\t\trenames->cached_pairs_valid_side = -1;\n+\t\t}\n+\t} else if (renames->redo_after_renames == 2)\n+\t\trenames->redo_after_renames = 0;\n \treturn ret;\n }\n \n@@ -2820,8 +2890,8 @@ static int compare_pairs(const void *a_, const void *b_)\n }\n \n /* Call diffcore_rename() to update deleted/added pairs into rename pairs */\n-static void detect_regular_renames(struct merge_options *opt,\n-\t\t\t\t unsigned side_index)\n+static int detect_regular_renames(struct merge_options *opt,\n+\t\t\t\t unsigned side_index)\n {\n \tstruct diff_options diff_opts;\n \tstruct rename_info *renames = &opt->priv->renames;\n@@ -2834,7 +2904,7 @@ static void detect_regular_renames(struct merge_options *opt,\n \t\t * side had directory renames.\n \t\t */\n \t\tresolve_diffpair_statuses(&renames->pairs[side_index]);\n-\t\treturn;\n+\t\treturn 0;\n \t}\n \n \tpartial_clear_dir_rename_count(&renames->dir_rename_count[side_index]);\n@@ -2869,6 +2939,18 @@ static void detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff.nr = 0;\n \tdiff_queued_diff.queue = NULL;\n \tdiff_flush(&diff_opts);\n+\n+\tif (renames->redo_after_renames) {\n+\t\tint i;\n+\t\tstruct diff_filepair *p;\n+\n+\t\trenames->redo_after_renames = 2;\n+\t\tfor (i = 0; i < renames->pairs[side_index].nr; ++i) {\n+\t\t\tp = renames->pairs[side_index].queue[i];\n+\t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n+\t\t}\n+\t}\n+\treturn 1;\n }\n \n /*\n@@ -2958,14 +3040,19 @@ static int detect_and_process_renames(struct merge_options *opt,\n \tstruct diff_queue_struct combined;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint need_dir_renames, s, clean = 1;\n+\tunsigned detection_run = 0;\n \n \tmemset(&combined, 0, sizeof(combined));\n \tif (!possible_renames(renames))\n \t\tgoto cleanup;\n \n \ttrace2_region_enter(\"merge\", \"regular renames\", opt->repo);\n-\tdetect_regular_renames(opt, MERGE_SIDE1);\n-\tdetect_regular_renames(opt, MERGE_SIDE2);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE1);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE2);\n+\tif (renames->redo_after_renames && detection_run) {\n+\t\ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n+\t\tgoto cleanup;\n+\t}\n \tuse_cached_pairs(opt, &renames->cached_pairs[1], &renames->pairs[1]);\n \tuse_cached_pairs(opt, &renames->cached_pairs[2], &renames->pairs[2]);\n \ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n@@ -4380,6 +4467,7 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \t\t\t\t\t opt->subtree_shift);\n \t}\n \n+redo:\n \ttrace2_region_enter(\"merge\", \"collect_merge_info\", opt->repo);\n \tif (collect_merge_info(opt, merge_base, side1, side2) != 0) {\n \t\t/*\n@@ -4399,6 +4487,12 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \tresult->clean = detect_and_process_renames(opt, merge_base,\n \t\t\t\t\t\t side1, side2);\n \ttrace2_region_leave(\"merge\", \"renames\", opt->repo);\n+\tif (opt->priv->renames.redo_after_renames == 2) {\n+\t\ttrace2_region_enter(\"merge\", \"reset_maps\", opt->repo);\n+\t\tclear_or_reinit_internal_opts(opt->priv, 1);\n+\t\ttrace2_region_leave(\"merge\", \"reset_maps\", opt->repo);\n+\t\tgoto redo;\n+\t}\n \n \ttrace2_region_enter(\"merge\", \"process_entries\", opt->repo);\n \tprocess_entries(opt, &working_tree_oid);\ndiff --git a/t/t6423-merge-rename-directories.sh b/t/t6423-merge-rename-directories.sh\nindex e834b7e6efe..d8919d276a1 100755\n--- a/t/t6423-merge-rename-directories.sh\n+++ b/t/t6423-merge-rename-directories.sh\n@@ -4797,7 +4797,7 @@ test_setup_12f () {\n \t)\n }\n \n-test_expect_merge_algorithm failure failure '12f: Trivial directory resolve, caching, all kinds of fun' '\n+test_expect_merge_algorithm failure success '12f: Trivial directory resolve, caching, all kinds of fun' '\n \ttest_setup_12f &&\n \t(\n \t\tcd 12f &&\n-- \ngitgitgadget\n"},{"id":"428896","messageId":"87wnqacg9i.fsf@evledraar.gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] Optimization batch 14: trivial directory resolution","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T13:21:14Z","receivedAt":"2021-07-01T13:31:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Elijah Newren via GitGitGadget wrote:\n\n> This series depends textually on ort-perf-batch-12, but is semantically\n> independent. (It is both semantically and textually independent of\n> ort-perf-batch-13.)\n\nFor others following along, that ort-perf-batch-12 is at\nhttps://lore.kernel.org/git/pull.962.v4.git.1623168703.gitgitgadget@gmail.com/#t\n& currently marked as 'will merge to next' in what's cooking.\n\n> Most of my previous series dramatically accelerated cases with lots of\n> renames, while providing comparatively minor benefits for cases with few or\n> no renames. This series is the opposite; it provides huge benefits when\n> there are few or no renames, and comparatively smaller (though still quite\n> decent) benefits for cases with many uncached renames.\n\nSounds good, one thing I haven't seen at a glance is how these\nperformance numbers compare to the merge-recursive backend. Are we in a\nstate of reaching parity with it, or pulling ahead?\n\n> [...]\n> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> changes in just this series improves the performance as follows:\n>\n> Before Series After Series\n> no-renames: 5.235 s ± 0.042 s 204.2 ms ± 3.0 ms\n> mega-renames: 9.419 s ± 0.107 s 1.076 s ± 0.015 s\n> just-one-mega: 480.1 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n>\n>\n> As a reminder, before any merge-ort/diffcore-rename performance work, the\n> performance results we started with were:\n>\n> no-renames-am: 6.940 s ± 0.485 s\n> no-renames: 18.912 s ± 0.174 s\n> mega-renames: 5964.031 s ± 10.459 s\n> just-one-mega: 149.583 s ± 0.751 s\n\nI haven't given any of this a detailed look, just a note/question that\n(depending on the answer to the \"v.s. merge-recursive above\") we may\nwant to consider bumping the default for the diff.renamelimit at some\npoint along with any major optimizations.\n\n\n\nAs an aside that we have diff.renamelimit is one of the most \"dangerous\"\nlandmines/fork-in-eye/shotgun-to-foot edge cases we have in using diff\nas plumbing IMO.\n\nE.g. I somewhat recently had to deal with some 3rd party Go-language\nlint plugin that can be configured to enforce lints \"as of a commit\".\nI.e. it does a diff from that commit, sees in any introduced \"issues\"\nare \"new\", and complains accordingly. The idea is that it allows you to\nenforce lints on \"only new code\", say ignoring the return value of\nos.Write(), without insisting that all existing code must be\nwhitelisted/fixed first.\n\nThe problem being two-fold, one that the thing will get slower over time\nas we grow history (can't be avoided), but the more subtle one that at\nsome point we'll bump into the diff.renamelimit, and whatever unlucky\nsob does so will find that the lint is now complaining about ALL THE\nTHINGS, since \"old\" code is now ending up as \"new\" to a naïve diff\nparser relying on not bumping into the diff.renamelimit.\n\nArguably bumping the diff.renamelimit would make that sort of problem\nworse for plumbing consumers, since they'd have more rope with which to\nhang themselves, maybe it's better to step on that landmine early.\n\nSorry about the digression somewhat pointless but perhaps amusing\ndigression in the last 4 paragraphs :)\n\nP.S.: I ended up dealing with the Go plugin by not using the \"diff\"\n feature, but just a one-off giant whitelist of all existing\n instances of stuff it would complain about.\n"},{"id":"428940","messageId":"CABPp-BGRgL-SnHDgefq887e51z9t_TMXcc4DLrazggC-1iQWng@mail.gmail.com","threadId":"56035","inReplyTo":"87wnqacg9i.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/7] Optimization batch 14: trivial directory resolution","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-01T15:04:19Z","receivedAt":"2021-07-01T15:04:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 1, 2021 at 6:31 AM Ævar Arnfjörð Bjarmason wrote:\n>\n> On Thu, Jul 01 2021, Elijah Newren via GitGitGadget wrote:\n>\n> > This series depends textually on ort-perf-batch-12, but is semantically\n> > independent. (It is both semantically and textually independent of\n> > ort-perf-batch-13.)\n>\n> For others following along, that ort-perf-batch-12 is at\n> https://lore.kernel.org/git/pull.962.v4.git.1623168703.gitgitgadget@gmail.com/#t\n> & currently marked as 'will merge to next' in what's cooking.\n>\n> > Most of my previous series dramatically accelerated cases with lots of\n> > renames, while providing comparatively minor benefits for cases with few or\n> > no renames. This series is the opposite; it provides huge benefits when\n> > there are few or no renames, and comparatively smaller (though still quite\n> > decent) benefits for cases with many uncached renames.\n>\n> Sounds good, one thing I haven't seen at a glance is how these\n> performance numbers compare to the merge-recursive backend. Are we in a\n> state of reaching parity with it, or pulling ahead?\n\nOh, wow, I really need to improve my wording if that's a question.\nWhen I referred to \"the performance results we started with\" at the\nend, that's referring to the numbers I got with merge-recursive before\nI started this journey. So these were the merge-recursive numbers\n(with git-2.29.0 or git-2.30.0):\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\nAs per commit 557ac0350d (\"merge-ort: begin performance work;\ninstrument with trace2_region_* calls\", 2021-01-23), merge-ort was\nfaster than merge-recursive at the beginning. But I wasn't content\nwith that....\n\nAlso in the cover letter, I showed the merge-ort numbers before and\nafter this series:\n\nno-renames: 5.235 s ± 0.042 s 204.2 ms ± 3.0 ms\nmega-renames: 9.419 s ± 0.107 s 1.076 s ± 0.015 s\njust-one-mega: 480.1 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\nSo yeah, we've pulled a little ahead. (Like my understatement there?)\n\nGranted, merge-recursive doesn't take quite as long as it used to; it\nalso benefited from some of my optimizations[1]. Nowhere near as much\nas merge-ort benefited, but it still would be about 20x faster on the\ncases with \"mega\" in their name. So, although today's merge-ort is\n~5542.7x faster than git-2.29.0's merge-recursive for a massive set of\nrenames in a really long rebase sequence, it is probably only a mere\n277x faster than today's merge-recursive on the same case.\n\n\nIf you'd like to catch up on the various optimizations, you can find\nmost of them with this:\n\ngit log --reverse --grep=mega-renames --author=newren\n\n\n[1] In particular, merge-recursive would benefit from these ones:\n12bd17521c (\"Merge branch 'en/diffcore-rename'\", 2021-03-01),\nd3a035b055 (\"Merge branch 'en/merge-ort-perf'\", 2021-02-11), and\na5ac31b5b1 (\"Merge branch 'en/diffcore-rename'\", 2021-01-25)\n\n> > [...]\n> > For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> > performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> > changes in just this series improves the performance as follows:\n> >\n> > Before Series After Series\n> > no-renames: 5.235 s ± 0.042 s 204.2 ms ± 3.0 ms\n> > mega-renames: 9.419 s ± 0.107 s 1.076 s ± 0.015 s\n> > just-one-mega: 480.1 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n> >\n> >\n> > As a reminder, before any merge-ort/diffcore-rename performance work, the\n> > performance results we started with were:\n> >\n> > no-renames-am: 6.940 s ± 0.485 s\n> > no-renames: 18.912 s ± 0.174 s\n> > mega-renames: 5964.031 s ± 10.459 s\n> > just-one-mega: 149.583 s ± 0.751 s\n>\n> I haven't given any of this a detailed look, just a note/question that\n> (depending on the answer to the \"v.s. merge-recursive above\") we may\n> want to consider bumping the default for the diff.renamelimit at some\n> point along with any major optimizations.\n\nI do think we should bump the diff.renamelimit, but not for these\nreasons. We should bump them for the same reasons we bumped them last\ntime at commit 92c57e5c1d (\"bump rename limit defaults (again)\",\n2011-02-19).\n\nIn particular, none of my optimizations made the quadratic portion of\nrename detection any faster. It just added multiple ways for things\nto either skip rename detection or be handled by a linear algorithm\ninstead and not be included in the quadratic loop. Since\ndiff.renamelimit only applies to the stuff that goes into the\nquadratic portion, most folks will notice that newer versions of git\ndetect renames with the same limits that older git would have given up\nwith.\n\nIt perhaps also is worth bearing that some of my rename detection\noptimizations were specific to three-way merges (and thus help\nmerge/cherry-pick/rebase/etc. but not diff/log), some of my\noptimizations were specific to reapplying long sequences of linear\ncommits (and thus help cherry-pick or rebase but not diff, log, or\neven merge), while others help out all uses of the diff machinery\n(diff/log/merge/cherry-pick/rebase/revert/etc.).\n\n> \n>\n> As an aside that we have diff.renamelimit is one of the most \"dangerous\"\n> landmines/fork-in-eye/shotgun-to-foot edge cases we have in using diff\n> as plumbing IMO.\n>\n> E.g. I somewhat recently had to deal with some 3rd party Go-language\n> lint plugin that can be configured to enforce lints \"as of a commit\".\n> I.e. it does a diff from that commit, sees in any introduced \"issues\"\n> are \"new\", and complains accordingly. The idea is that it allows you to\n> enforce lints on \"only new code\", say ignoring the return value of\n> os.Write(), without insisting that all existing code must be\n> whitelisted/fixed first.\n>\n> The problem being two-fold, one that the thing will get slower over time\n> as we grow history (can't be avoided), but the more subtle one that at\n> some point we'll bump into the diff.renamelimit, and whatever unlucky\n> sob does so will find that the lint is now complaining about ALL THE\n> THINGS, since \"old\" code is now ending up as \"new\" to a naïve diff\n> parser relying on not bumping into the diff.renamelimit.\n>\n> Arguably bumping the diff.renamelimit would make that sort of problem\n> worse for plumbing consumers, since they'd have more rope with which to\n> hang themselves, maybe it's better to step on that landmine early.\n>\n> Sorry about the digression somewhat pointless but perhaps amusing\n> digression in the last 4 paragraphs :)\n\nCertainly a digression, but yes it's amusing. :-)\n\nLet me add a digression of my own: I got started on this because\npeople complained they couldn't cherry-pick patches to maintenance\nbranches, git just messed it all up when it gave up on renames. I\ntold them to set diff.renameLimit higher, and they told me that didn't\nwork.\n\nI didn't believe them.\n\nSo, I started on a bit of a journey, somewhere around commit\n9f7e4bfa3b (\"diff: remove silent clamp of renameLimit\", 2017-11-13).\nThen after that commit, git could detect renames but took forever\ndoing so (we needed a renameLimit of 48941 for the very first testcase\nI was pointed at). I dug into that, trying to figure out why\ncherry-picking a simple patch that modified 7 files (and not with\nparticularly big edits) would take something approaching 10 minutes to\ncomplete. Then I went and found optimizations and rewrote the entire\nmerge machinery. Now that same cherry-pick can complete without\nbumping diff.renameLimit (1000 is more than enough), and completes in\nless than a second (well, with all my optimizations, including the\nseries not yet submitted).\n\nSo, as I said, I didn't believe them when they reported git couldn't\ndetect renames for their case. Look at all the work I've done in\norder to continue to not believe them. ;-)\n"},{"id":"428979","messageId":"CABPp-BH3H20wAoD4CJmQjj2iVhnXN+M=XvQdUWUgvY0VxPHKZw@mail.gmail.com","threadId":"56035","inReplyTo":"CABPp-BGRgL-SnHDgefq887e51z9t_TMXcc4DLrazggC-1iQWng@mail.gmail.com","subject":"Re: [PATCH 0/7] Optimization batch 14: trivial directory resolution","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-01T19:22:31Z","receivedAt":"2021-07-01T19:22:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"A small correction/clarification...\n\nOn Thu, Jul 1, 2021 at 8:04 AM Elijah Newren wrote:\n>\n> On Thu, Jul 1, 2021 at 6:31 AM Ævar Arnfjörð Bjarmason wrote:\n> >\n> > On Thu, Jul 01 2021, Elijah Newren via GitGitGadget wrote:\n> >\n> > > This series depends textually on ort-perf-batch-12, but is semantically\n> > > independent. (It is both semantically and textually independent of\n> > > ort-perf-batch-13.)\n> >\n> > For others following along, that ort-perf-batch-12 is at\n> > https://lore.kernel.org/git/pull.962.v4.git.1623168703.gitgitgadget@gmail.com/#t\n> > & currently marked as 'will merge to next' in what's cooking.\n\nYeah, I should have referred to it as en/ort-perf-batch-12, the branch\nname Junio used.\n\n> Granted, merge-recursive doesn't take quite as long as it used to; it\n> also benefited from some of my optimizations[1]. Nowhere near as much\n> as merge-ort benefited, but it still would be about 20x faster on the\n> cases with \"mega\" in their name. So, although today's merge-ort is\n> ~5542.7x faster than git-2.29.0's merge-recursive for a massive set of\n> renames in a really long rebase sequence, it is probably only a mere\n> 277x faster than today's merge-recursive on the same case.\n\nThe 20x number here I was just spit-balling, whereas the other numbers\nwere measured. I think 20x may have been a bit too high. Regardless,\nthough, to me the important takeaway is not the performance difference\nbetween merge-ort and merge-recursive now (though that's also huge in\nmerge-ort's favor), but between merge-ort now and merge-recursive when\nI started. If someone really wants me to get the difference between\nthe two now, I can dig in, but it's annoyingly painful waiting for\nmerge-recursive to finish tests several times.\n"},{"id":"429921","messageId":"5dca982c0b061ae8d6335d4f22b78811dced054a.1626204784.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] merge-ort: resolve paths early when we have sufficient information","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:32:57Z","receivedAt":"2021-07-13T19:33:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen there are no directories involved at a given path, and all three\nsides have a file at that path, and two of the three sides of history\nmatch, we can immediately resolve the merge of that path in\ncollect_merge_info() and do not need to wait until process_entries().\n\nThis is actually a very minor improvement: half the time when I run it,\nI see an improvement; the other half a slowdown. It seems to be in the\nrange of noise. However, this idea serves as the beginning of some\nbigger optimizations coming in the following patches.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e3a5dfc7b31..6299b4f9413 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1023,6 +1023,43 @@ static int collect_merge_info_callback(int n,\n \t\treturn mask;\n \t}\n \n+\t/*\n+\t * If the sides match, and all three paths are present and are\n+\t * files, then we can take either as the resolution. We can't do\n+\t * this with trees, because there may be rename sources from the\n+\t * merge_base.\n+\t */\n+\tif (sides_match && filemask == 0x07) {\n+\t\t/* use side1 (== side2) version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/*\n+\t * If side1 matches mbase and all three paths are present and are\n+\t * files, then we can use side2 as the resolution. We cannot\n+\t * necessarily do so this for trees, because there may be rename\n+\t * destinations within side2.\n+\t */\n+\tif (side1_matches_mbase && filemask == 0x07) {\n+\t\t/* use side2 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+2, side2_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/* Similar to above but swapping sides 1 and 2 */\n+\tif (side2_matches_mbase && filemask == 0x07) {\n+\t\t/* use side1 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n \t/*\n \t * Gather additional information used in rename detection.\n \t */\n-- \ngitgitgadget\n\n"},{"id":"429922","messageId":"8aea3713902b7d006f527ccd76faf6f944529bdb.1626204784.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] merge-ort: add some more explanations in collect_merge_info_callback()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:32:58Z","receivedAt":"2021-07-13T19:33:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe previous patch possibly raises some questions about whether\nadditional cases in collect_merge_info_callback() can be handled early.\nAdd some explanations in the form of comments to help explain these\nbetter. While we're at it, add a few comments to denote what a few\nboolean '0' or '1' values stand for.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 20 +++++++++++++++-----\n 1 file changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 6299b4f9413..843fa693145 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1018,8 +1018,8 @@ static int collect_merge_info_callback(int n,\n \tif (side1_matches_mbase && side2_matches_mbase) {\n \t\t/* mbase, side1, & side2 all match; use mbase as resolution */\n \t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n-\t\t\t\tnames, names+0, mbase_null, 0,\n-\t\t\t\tfilemask, dirmask, 1);\n+\t\t\t\tnames, names+0, mbase_null, 0 /* df_conflict */,\n+\t\t\t\tfilemask, dirmask, 1 /* resolved */);\n \t\treturn mask;\n \t}\n \n@@ -1061,14 +1061,24 @@ static int collect_merge_info_callback(int n,\n \t}\n \n \t/*\n-\t * Gather additional information used in rename detection.\n+\t * Sometimes we can tell that a source path need not be included in\n+\t * rename detection -- namely, whenever either\n+\t * side1_matches_mbase && side2_null\n+\t * or\n+\t * side2_matches_mbase && side1_null\n+\t * However, we call collect_rename_info() even in those cases,\n+\t * because exact renames are cheap and would let us remove both a\n+\t * source and destination path. We'll cull the unneeded sources\n+\t * later.\n \t */\n \tcollect_rename_info(opt, names, dirname, fullpath,\n \t\t\t filemask, dirmask, match_mask);\n \n \t/*\n-\t * Record information about the path so we can resolve later in\n-\t * process_entries.\n+\t * None of the special cases above matched, so we have a\n+\t * provisional conflict. (Rename detection might allow us to\n+\t * unconflict some more cases, but that comes later so all we can\n+\t * do now is record the different non-null file hashes.)\n \t */\n \tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n \t\t\tnames, NULL, 0, df_conflict, filemask, dirmask, 0);\n-- \ngitgitgadget\n\n"},{"id":"429923","messageId":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.git.1625111177.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] Optimization batch 14: trivial directory resolution","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:32:56Z","receivedAt":"2021-07-13T19:33:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"[This is mostly unchanged since v1; I'm primarily resending since it's been\ntwo weeks and I want to bump it so folks have a chance to notice and review\nit. But there was one change, noted below.]\n\nThis series depends textually on ort-perf-batch-12, but is semantically\nindependent. (It is both semantically and textually independent of\nort-perf-batch-13.)\n\nMost of my previous series dramatically accelerated cases with lots of\nrenames, while providing comparatively minor benefits for cases with few or\nno renames. This series is the opposite; it provides huge benefits when\nthere are few or no renames, and comparatively smaller (though still quite\ndecent) benefits for cases with many uncached renames.\n\nChanges since v1:\n\n * Minor tweak to the final patch to correct implicit assumption that rename\n detection running implies all renames were found (rename limits could\n have been exceeded and prevented finding renames)\n\n=== Basic Optimization idea ===\n\nunpack_trees has had a concept of trivial merges for individual files (see\nDocumentation/technical/trivial-merge.txt). The same idea can be applied in\nmerge-ort. It'd be really nice to extend that idea to trees as well, as it\ncould provide a huge performance boost; sadly however, applying it in\ngeneral would wreck both regular rename detection (the unmatched side can\nhave new files that serve as potential destinations in rename detection) and\ndirectory rename detection (the unmatched side could have a new directory\nthat was moved into it).\n\nIf we somehow knew rename detection wasn't needed, we could do trivial\ndirectory resolution. In the past, this wasn't possible. However...\n\nWith recent optimizations we have created a possibility to do trivial\ndirectory resolutions in some cases. These came from the addition of the\n\"skipping irrelevant renames\" optimizations (from ort-perf-batch-9 and\nort-perf-batch-10), and in particular noting that we added an ability to\nentirely skip rename detection in commit f89b4f2bee (\"merge-ort: skip rename\ndetection entirely if possible\", 2021-03-11) when there are no relevant\nsources. We can detect if there are no relevant sources without recursing\ninto the directories in question.\n\nAs a cherry on top, the caching of renames (from ort-perf-batch-11) allows\nus to cover additional cases.\n\nThis series is all about adding all the special checks needed to safely\nperform trival directory resolutions.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 5.235 s ± 0.042 s 204.2 ms ± 3.0 ms\nmega-renames: 9.419 s ± 0.107 s 1.076 s ± 0.015 s\njust-one-mega: 480.1 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with (for merge-recursive as of git-2.30.0)\nwere:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\nElijah Newren (7):\n merge-ort: resolve paths early when we have sufficient information\n merge-ort: add some more explanations in collect_merge_info_callback()\n merge-ort: add data structures for allowable trivial directory\n resolves\n merge-ort: add a handle_deferred_entries() helper function\n merge-ort: defer recursing into directories when merge base is matched\n merge-ort: avoid recursing into directories when we don't need to\n merge-ort: restart merge with cached renames to reduce process entry\n cost\n\n merge-ort.c | 408 +++++++++++++++++++++++++++-\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 398 insertions(+), 12 deletions(-)\n\n\nbase-commit: 2eeee12b02e441ac05054a5a5ecbcea6964a1e6b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-988%2Fnewren%2Fort-perf-batch-14-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-988/newren/ort-perf-batch-14-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/988\n\nRange-diff vs v1:\n\n 1: 5dca982c0b = 1: 5dca982c0b merge-ort: resolve paths early when we have sufficient information\n 2: 8aea371390 = 2: 8aea371390 merge-ort: add some more explanations in collect_merge_info_callback()\n 3: f7ac01055d = 3: f7ac01055d merge-ort: add data structures for allowable trivial directory resolves\n 4: 7e28323b62 = 4: 7e28323b62 merge-ort: add a handle_deferred_entries() helper function\n 5: 317553eadb = 5: 317553eadb merge-ort: defer recursing into directories when merge base is matched\n 6: 3409a6cd63 = 6: 3409a6cd63 merge-ort: avoid recursing into directories when we don't need to\n 7: 76bc73262c ! 7: 7133f0efa5 merge-ort: restart merge with cached renames to reduce process entry cost\n @@ merge-ort.c: static void detect_regular_renames(struct merge_options *opt,\n \t}\n \n \tpartial_clear_dir_rename_count(&renames->dir_rename_count[side_index]);\n +@@ merge-ort.c: static void detect_regular_renames(struct merge_options *opt,\n + \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n + \tresolve_diffpair_statuses(&diff_queued_diff);\n + \n ++\tif (diff_opts.needed_rename_limit > 0)\n ++\t\trenames->redo_after_renames = 0;\n + \tif (diff_opts.needed_rename_limit > renames->needed_limit)\n + \t\trenames->needed_limit = diff_opts.needed_rename_limit;\n + \n @@ merge-ort.c: static void detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff.nr = 0;\n \tdiff_queued_diff.queue = NULL;\n \tdiff_flush(&diff_opts);\n +\n -+\tif (renames->redo_after_renames) {\n -+\t\tint i;\n -+\t\tstruct diff_filepair *p;\n -+\n -+\t\trenames->redo_after_renames = 2;\n -+\t\tfor (i = 0; i < renames->pairs[side_index].nr; ++i) {\n -+\t\t\tp = renames->pairs[side_index].queue[i];\n -+\t\t\tpossibly_cache_new_pair(renames, p, side_index, NULL);\n -+\t\t}\n -+\t}\n +\treturn 1;\n }\n \n @@ merge-ort.c: static int detect_and_process_renames(struct merge_options *opt,\n +\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE1);\n +\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE2);\n +\tif (renames->redo_after_renames && detection_run) {\n ++\t\tint i, side;\n ++\t\tstruct diff_filepair *p;\n ++\n ++\t\t/* Cache the renames, we found */\n ++\t\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n ++\t\t\tfor (i = 0; i < renames->pairs[side].nr; ++i) {\n ++\t\t\t\tp = renames->pairs[side].queue[i];\n ++\t\t\t\tpossibly_cache_new_pair(renames, p, side, NULL);\n ++\t\t\t}\n ++\t\t}\n ++\n ++\t\t/* Restart the merge with the cached renames */\n ++\t\trenames->redo_after_renames = 2;\n +\t\ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n +\t\tgoto cleanup;\n +\t}\n\n-- \ngitgitgadget\n"},{"id":"429924","messageId":"f7ac01055d9d2e9e2dfdfd780ff7f10fbfd05d5b.1626204784.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v2 3/7] merge-ort: add data structures for allowable trivial directory resolves","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:32:59Z","receivedAt":"2021-07-13T19:33:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs noted a few commits ago, we can resolve individual files early if all\nthree sides of the merge have a file at the path and two of the three\nsides match. We would really like to do the same thing with\ndirectories, because being able to do a trivial directory resolve means\nwe don't have to recurse into the directory, potentially saving us a\nhuge amount of time in both collect_merge_info() and process_entries().\nUnfortunately, resolving directories early would mean missing any\nrenames whose source or destination is underneath that directory.\n\nIf we somehow knew there weren't any renames under the directory in\nquestion, then we could resolve it early. Sadly, it is impossible to\ndetermine whether there are renames under the directory in question\nwithout recursing into it, and this has traditionally kept us from ever\nimplementing such an optimization.\n\nIn commit f89b4f2bee (\"merge-ort: skip rename detection entirely if\npossible\", 2021-03-11), we added an additional reason that rename\ndetection could be skipped entirely -- namely, if no *relevant* sources\nwere present. Without completing collect_merge_info_callback(), we do\nnot yet know if there are no relevant sources. However, we do know that\nif the current directory on one side matches the merge base, then every\nsource file within that directory will not be RELEVANT_CONTENT, and a\nfew simple checks can often let us rule out RELEVANT_LOCATION as well.\nThis suggests we can just defer recursing into such directories until\nthe end of collect_merge_info.\n\nSince the deferred directories are known to not add any relevant sources\ndue to the above properties, then if there are no relevant sources after\nwe've traversed all paths other than the deferred ones, then we know\nthere are not any relevant sources. Under those conditions, rename\ndetection is unnecessary, and that means we can resolve the deferred\ndirectories without recursing into them.\n\nNote that the logic for skipping rename detection was also modified\nfurther in commit 76e253793c (\"merge-ort, diffcore-rename: employ cached\nrenames when possible\", 2021-01-30); in particular rename detection can\nbe skipped if we already have cached renames for each relevant source.\nWe can take advantage of this information as well with our deferral of\nrecursing into directories where one side matches the merge base.\n\nAdd some data structures that we will use to do these deferrals, with\nsome lengthy comments explaining their purpose.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 843fa693145..3d3f00b3b45 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -119,6 +119,51 @@ struct rename_info {\n \t */\n \tstruct strintmap relevant_sources[3];\n \n+\t/*\n+\t * possible_trivial_merges: directories we defer recursing into\n+\t *\n+\t * possible_trivial_merges is a map of directory names to\n+\t * dir_rename_mask. When we detect that a directory is unchanged on\n+\t * one side, we can sometimes resolve the directory without recursing\n+\t * into it. Renames are the only things that can prevent such an\n+\t * optimization. However, for rename sources:\n+\t * - If no parent directory needed directory rename detection, then\n+\t * no path under such a directory can be a relevant_source.\n+\t * and for rename destinations:\n+\t * - If no cached rename has a target path under the directory AND\n+\t * - If there are no unpaired relevant_sources elsewhere in the\n+\t * repository\n+\t * then we don't need any path under this directory for a rename\n+\t * destination. The only way to know the last item above is to defer\n+\t * handling such directories until the end of collect_merge_info(),\n+\t * in handle_deferred_entries().\n+\t *\n+\t * For each we store dir_rename_mask, since that's the only bit of\n+\t * information we need, other than the path, to resume the recursive\n+\t * traversal.\n+\t */\n+\tstruct strintmap possible_trivial_merges[3];\n+\n+\t/*\n+\t * trivial_merges_okay: if trivial directory merges are okay\n+\t *\n+\t * See possible_trivial_merges above. The \"no unpaired\n+\t * relevant_sources elsewhere in the repository\" is a single boolean\n+\t * per merge side, which we store here. Note that while 0 means no,\n+\t * 1 only means \"maybe\" rather than \"yes\"; we optimistically set it\n+\t * to 1 initially and only clear when we determine it is unsafe to\n+\t * do trivial directory merges.\n+\t */\n+\tunsigned trivial_merges_okay[3];\n+\n+\t/*\n+\t * target_dirs: ancestor directories of rename targets\n+\t *\n+\t * target_dirs contains all directory names that are an ancestor of\n+\t * any rename destination.\n+\t */\n+\tstruct strset target_dirs[3];\n+\n \t/*\n \t * dir_rename_mask:\n \t * 0: optimization removing unmodified potential rename source okay\n@@ -490,6 +535,9 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrintmap_func(&renames->dirs_removed[i]);\n \t\tstrmap_func(&renames->dir_renames[i], 0);\n \t\tstrintmap_func(&renames->relevant_sources[i]);\n+\t\tstrintmap_func(&renames->possible_trivial_merges[i]);\n+\t\tstrset_func(&renames->target_dirs[i]);\n+\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n \t\tif (i != renames->cached_pairs_valid_side) {\n@@ -4045,12 +4093,17 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n \t\t\t\t\t -1 /* explicitly invalid */,\n \t\t\t\t\t NULL, 0);\n+\t\tstrintmap_init_with_options(&renames->possible_trivial_merges[i],\n+\t\t\t\t\t 0, NULL, 0);\n+\t\tstrset_init_with_options(&renames->target_dirs[i],\n+\t\t\t\t\t NULL, 1);\n \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_target_names[i],\n \t\t\t\t\t NULL, 0);\n+\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n \t}\n \n \t/*\n-- \ngitgitgadget\n\n"},{"id":"429926","messageId":"7133f0efa520b3d0cadb059151daa12484fdb003.1626204784.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:33:03Z","receivedAt":"2021-07-13T19:33:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe merge algorithm mostly consists of the following three functions:\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\nPrior to the trivial directory resolution optimization of the last half\ndozen commits, process_entries() was consistently the slowest, followed\nby collect_merge_info(), then detect_and_process_renames(). When the\ntrivial directory resolution applies, it often dramatically decreases\nthe amount of time spent in the two slower functions.\n\nLooking at the performance results in the previous commit, the trivial\ndirectory resolution optimization helps amazingly well when there are no\nrelevant renames. It also helps really well when reapplying a long\nseries of linear commits (such as in a rebase or cherry-pick), since the\nrelevant renames may well be cached from the first reapplied commit.\nBut when there are any relevant renames that are not cached (represented\nby the just-one-mega testcase), then the optimization does not help at\nall.\n\nOften, I noticed that when the optimization does not apply, it is\nbecause there are a handful of relevant sources -- maybe even only one.\nIt felt frustrating to need to recurse into potentially hundreds or even\nthousands of directories just for a single rename, but it was needed for\ncorrectness.\n\nHowever, staring at this list of functions and noticing that\nprocess_entries() is the most expensive and knowing I could avoid it if\nI had cached renames suggested a simple idea: change\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\ninto\n collect_merge_info()\n detect_and_process_renames()\n \n collect_merge_info()\n detect_and_process_renames()\n process_entries()\n\nThis may seem odd and look like more work. However, note that although\nwe run collect_merge_info() twice, the second time we get to employ\ntrivial directory resolves, which makes it much faster, so the increased\ntime in collect_merge_info() is small. While we run\ndetect_and_process_renames() again, all renames are cached so it's\nnearly a no-op (we don't call into diffcore_rename_extended() but we do\nhave a little bit of data structure checking and fixing up). And the\nbig payoff comes from the fact that process_entries(), will be much\nfaster due to having far fewer entries to process.\n\nThis restarting only makes sense if we can save recursing into enough\ndirectories to make it worth our while. Introduce a simple heuristic to\nguide this. Note that this heuristic uses a \"wanted_factor\" that I have\nvirtually no actual real world data for, just some back-of-the-envelope\nquasi-scientific calculations that I included in some comments and then\nplucked a simple round number out of thin air. It could be that\ntweaking this number to make it either higher or lower improves the\noptimization. (There's slightly more here; when I first introduced this\noptimization, I used a factor of 10, because I was completely confident\nit was big enough to not cause slowdowns in special cases. I was\ncertain it was higher than needed. Several months later, I added the\nrough calculations which make me think the optimal number is close to 2;\nbut instead of pushing to the limit, I just bumped it to 3 to reduce the\nrisk that there are special cases where this optimization can result in\nslowing down the code a little. If the ratio of path counts is below 3,\nwe probably will only see minor performance improvements at best\nanyway.)\n\nAlso, note that while the diffstat looks kind of long (nearly 100\nlines), more than half of it is in two comments explaining how things\nwork.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 205.1 ms ± 3.8 ms 204.2 ms ± 3.0 ms\n mega-renames: 1.564 s ± 0.010 s 1.076 s ± 0.015 s\n just-one-mega: 479.5 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 111 ++++++++++++++++++++++++++--\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 106 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex c9cf7a158c8..99af64fce7d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -209,6 +209,7 @@ struct rename_info {\n \t * MERGE_SIDE2: cached data from side2 can be reused\n \t * MERGE_SIDE1: cached data from side1 can be reused\n \t * 0: no cached data can be reused\n+\t * -1: See redo_after_renames; both sides can be reused.\n \t */\n \tint cached_pairs_valid_side;\n \n@@ -254,6 +255,28 @@ struct rename_info {\n \t */\n \tstruct strset cached_irrelevant[3];\n \n+\t/*\n+\t * redo_after_renames: optimization flag for \"restarting\" the merge\n+\t *\n+\t * Sometimes it pays to detect renames, cache them, and then\n+\t * restart the merge operation from the beginning. The reason for\n+\t * this is that when we know where all the renames are, we know\n+\t * whether a certain directory has any paths under it affected --\n+\t * and if a directory is not affected then it permits us to do\n+\t * trivial tree merging in more cases. Doing trivial tree merging\n+\t * prevents the need to run process_entry() on every path\n+\t * underneath trees that can be trivially merged, and\n+\t * process_entry() is more expensive than collect_merge_info() --\n+\t * plus, the second collect_merge_info() will be much faster since\n+\t * it doesn't have to recurse into the relevant trees.\n+\t *\n+\t * Values for this flag:\n+\t * 0 = don't bother, not worth it (or conditions not yet checked)\n+\t * 1 = conditions for optimization met, optimization worthwhile\n+\t * 2 = we already did it (don't restart merge yet again)\n+\t */\n+\tunsigned redo_after_renames;\n+\n \t/*\n \t * needed_limit: value needed for inexact rename detection to run\n \t *\n@@ -540,7 +563,8 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n-\t\tif (i != renames->cached_pairs_valid_side) {\n+\t\tif (i != renames->cached_pairs_valid_side &&\n+\t\t -1 != renames->cached_pairs_valid_side) {\n \t\t\tstrset_func(&renames->cached_target_names[i]);\n \t\t\tstrmap_func(&renames->cached_pairs[i], 1);\n \t\t\tstrset_func(&renames->cached_irrelevant[i]);\n@@ -1242,7 +1266,9 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n \tint side, ret = 0;\n+\tint path_count_before, path_count_after = 0;\n \n+\tpath_count_before = strmap_get_size(&opt->priv->paths);\n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n \t\tunsigned optimization_okay = 1;\n \t\tstruct strintmap copy;\n@@ -1377,7 +1403,51 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t\t\tpath));\n \t\t\tresolve_trivial_directory_merge(ci, side);\n \t\t}\n+\t\tif (!optimization_okay || path_count_after)\n+\t\t\tpath_count_after = strmap_get_size(&opt->priv->paths);\n \t}\n+\tif (path_count_after) {\n+\t\t/*\n+\t\t * Not sure were the right cut-off is for the optimization\n+\t\t * to redo collect_merge_info after we've cached the\n+\t\t * regular renames is. Basically, collect_merge_info(),\n+\t\t * detect_regular_renames(), and process_entries() are\n+\t\t * similar costs and all big tent poles. Caching the\n+\t\t * result of detect_regular_renames() means that redoing\n+\t\t * that one function will cost us virtually 0 extra, so it\n+\t\t * depends on the other two functions, which are both O(N)\n+\t\t * cost in the number of paths. Thus, it makes sense that\n+\t\t * if we can cut the number of paths in half, then redoing\n+\t\t * collect_merge_info() at half cost in order to get\n+\t\t * process_entries() at half cost should be about equal\n+\t\t * cost. If we can cut by more than half, then we would\n+\t\t * win. The fact that process_entries() is about 10%-20%\n+\t\t * more expensive than collect_merge_info() suggests we\n+\t\t * could make the factor be less than two. The fact that\n+\t\t * even when we have renames cached, we still have to\n+\t\t * traverse down to the individual (relevant) renames,\n+\t\t * which suggests we should perhaps use a bigger factor.\n+\t\t *\n+\t\t * The exact number isn't critical, since the code will\n+\t\t * work even if we get the factor wrong -- it just might be\n+\t\t * slightly slower if we're a bit off. For now, just error\n+\t\t * on the side of a bigger fudge. For the linux kernel\n+\t\t * testcases I was looking at with massive renames, the\n+\t\t * ratio came in around 50 to 250, which clearly would\n+\t\t * trigger this optimization and provided some *very* nice\n+\t\t * speedups.\n+\t\t */\n+\t\tint wanted_factor = 3;\n+\n+\t\t/* We should only redo collect_merge_info one time */\n+\t\tassert(renames->redo_after_renames == 0);\n+\n+\t\tif (path_count_after / path_count_before > wanted_factor) {\n+\t\t\trenames->redo_after_renames = 1;\n+\t\t\trenames->cached_pairs_valid_side = -1;\n+\t\t}\n+\t} else if (renames->redo_after_renames == 2)\n+\t\trenames->redo_after_renames = 0;\n \treturn ret;\n }\n \n@@ -2820,8 +2890,8 @@ static int compare_pairs(const void *a_, const void *b_)\n }\n \n /* Call diffcore_rename() to update deleted/added pairs into rename pairs */\n-static void detect_regular_renames(struct merge_options *opt,\n-\t\t\t\t unsigned side_index)\n+static int detect_regular_renames(struct merge_options *opt,\n+\t\t\t\t unsigned side_index)\n {\n \tstruct diff_options diff_opts;\n \tstruct rename_info *renames = &opt->priv->renames;\n@@ -2834,7 +2904,7 @@ static void detect_regular_renames(struct merge_options *opt,\n \t\t * side had directory renames.\n \t\t */\n \t\tresolve_diffpair_statuses(&renames->pairs[side_index]);\n-\t\treturn;\n+\t\treturn 0;\n \t}\n \n \tpartial_clear_dir_rename_count(&renames->dir_rename_count[side_index]);\n@@ -2860,6 +2930,8 @@ static void detect_regular_renames(struct merge_options *opt,\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n \tresolve_diffpair_statuses(&diff_queued_diff);\n \n+\tif (diff_opts.needed_rename_limit > 0)\n+\t\trenames->redo_after_renames = 0;\n \tif (diff_opts.needed_rename_limit > renames->needed_limit)\n \t\trenames->needed_limit = diff_opts.needed_rename_limit;\n \n@@ -2869,6 +2941,8 @@ static void detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff.nr = 0;\n \tdiff_queued_diff.queue = NULL;\n \tdiff_flush(&diff_opts);\n+\n+\treturn 1;\n }\n \n /*\n@@ -2958,14 +3032,32 @@ static int detect_and_process_renames(struct merge_options *opt,\n \tstruct diff_queue_struct combined;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint need_dir_renames, s, clean = 1;\n+\tunsigned detection_run = 0;\n \n \tmemset(&combined, 0, sizeof(combined));\n \tif (!possible_renames(renames))\n \t\tgoto cleanup;\n \n \ttrace2_region_enter(\"merge\", \"regular renames\", opt->repo);\n-\tdetect_regular_renames(opt, MERGE_SIDE1);\n-\tdetect_regular_renames(opt, MERGE_SIDE2);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE1);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE2);\n+\tif (renames->redo_after_renames && detection_run) {\n+\t\tint i, side;\n+\t\tstruct diff_filepair *p;\n+\n+\t\t/* Cache the renames, we found */\n+\t\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n+\t\t\tfor (i = 0; i < renames->pairs[side].nr; ++i) {\n+\t\t\t\tp = renames->pairs[side].queue[i];\n+\t\t\t\tpossibly_cache_new_pair(renames, p, side, NULL);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Restart the merge with the cached renames */\n+\t\trenames->redo_after_renames = 2;\n+\t\ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n+\t\tgoto cleanup;\n+\t}\n \tuse_cached_pairs(opt, &renames->cached_pairs[1], &renames->pairs[1]);\n \tuse_cached_pairs(opt, &renames->cached_pairs[2], &renames->pairs[2]);\n \ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n@@ -4380,6 +4472,7 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \t\t\t\t\t opt->subtree_shift);\n \t}\n \n+redo:\n \ttrace2_region_enter(\"merge\", \"collect_merge_info\", opt->repo);\n \tif (collect_merge_info(opt, merge_base, side1, side2) != 0) {\n \t\t/*\n@@ -4399,6 +4492,12 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \tresult->clean = detect_and_process_renames(opt, merge_base,\n \t\t\t\t\t\t side1, side2);\n \ttrace2_region_leave(\"merge\", \"renames\", opt->repo);\n+\tif (opt->priv->renames.redo_after_renames == 2) {\n+\t\ttrace2_region_enter(\"merge\", \"reset_maps\", opt->repo);\n+\t\tclear_or_reinit_internal_opts(opt->priv, 1);\n+\t\ttrace2_region_leave(\"merge\", \"reset_maps\", opt->repo);\n+\t\tgoto redo;\n+\t}\n \n \ttrace2_region_enter(\"merge\", \"process_entries\", opt->repo);\n \tprocess_entries(opt, &working_tree_oid);\ndiff --git a/t/t6423-merge-rename-directories.sh b/t/t6423-merge-rename-directories.sh\nindex e834b7e6efe..d8919d276a1 100755\n--- a/t/t6423-merge-rename-directories.sh\n+++ b/t/t6423-merge-rename-directories.sh\n@@ -4797,7 +4797,7 @@ test_setup_12f () {\n \t)\n }\n \n-test_expect_merge_algorithm failure failure '12f: Trivial directory resolve, caching, all kinds of fun' '\n+test_expect_merge_algorithm failure success '12f: Trivial directory resolve, caching, all kinds of fun' '\n \ttest_setup_12f &&\n \t(\n \t\tcd 12f &&\n-- \ngitgitgadget\n"},{"id":"429925","messageId":"7e28323b624ad2d8d12123783f00f5a8fbb248e8.1626204784.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] merge-ort: add a handle_deferred_entries() helper function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:33:00Z","receivedAt":"2021-07-13T19:33:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn order to allow trivial directory resolution, we first need to be able\nto gather more information to determine if the optimization is safe. To\nenable that, we need a way of deferring the recursion into the directory\nuntil a later time. Naturally, deferring the entry into a subtree means\nthat we need some function that will later recurse into the subdirectory\nexactly the same way that collect_merge_info_callback() would have done.\n\nAdd a helper function that does this. For now this function is not used\nbut a subsequent commit will change that. Future commits will also make\nthe function sometimes resolve directories instead of traversing inside.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 64 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 3d3f00b3b45..eb0e18d7546 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1196,6 +1196,70 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+MAYBE_UNUSED\n+static int handle_deferred_entries(struct merge_options *opt,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tstruct rename_info *renames = &opt->priv->renames;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\tint side, ret = 0;\n+\n+\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n+\t\trenames->trivial_merges_okay[side] = 0;\n+\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n+\t\t\tstruct conflict_info *ci;\n+\t\t\tunsigned dirmask;\n+\t\t\tstruct tree_desc t[3];\n+\t\t\tvoid *buf[3] = {NULL,};\n+\t\t\tint i;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\t\t\tdirmask = ci->dirmask;\n+\n+\t\t\tinfo->name = path;\n+\t\t\tinfo->namelen = strlen(path);\n+\t\t\tinfo->pathlen = info->namelen + 1;\n+\n+\t\t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n+\t\t\t\tif (i == 1 && ci->match_mask == 3)\n+\t\t\t\t\tt[1] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 5)\n+\t\t\t\t\tt[2] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 6)\n+\t\t\t\t\tt[2] = t[1];\n+\t\t\t\telse {\n+\t\t\t\t\tconst struct object_id *oid = NULL;\n+\t\t\t\t\tif (dirmask & 1)\n+\t\t\t\t\t\toid = &ci->stages[i].oid;\n+\t\t\t\t\tbuf[i] = fill_tree_descriptor(opt->repo,\n+\t\t\t\t\t\t\t\t t+i, oid);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tci->match_mask &= ci->filemask;\n+\t\t\topt->priv->current_dir_name = path;\n+\t\t\trenames->dir_rename_mask = dir_rename_mask;\n+\t\t\tif (renames->dir_rename_mask == 0 ||\n+\t\t\t renames->dir_rename_mask == 0x07)\n+\t\t\t\tret = traverse_trees(NULL, 3, t, info);\n+\t\t\telse\n+\t\t\t\tret = traverse_trees_wrapper(NULL, 3, t, info);\n+\n+\t\t\tfor (i = MERGE_BASE; i <= MERGE_SIDE2; i++)\n+\t\t\t\tfree(buf[i]);\n+\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n static int collect_merge_info(struct merge_options *opt,\n \t\t\t struct tree *merge_base,\n \t\t\t struct tree *side1,\n-- \ngitgitgadget\n\n"},{"id":"429927","messageId":"317553eadb68ce162b5ebea24045a9ca75342e91.1626204784.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] merge-ort: defer recursing into directories when merge base is matched","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:33:01Z","receivedAt":"2021-07-13T19:33:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen one side of history matches the merge base (including when the\nmerge base has no entry for the given directory), have\ncollect_merge_info_callback() defer recursing into the directory. To\nensure those entries are eventually handled, add a call to\nhandled_deferred_entries() in collect_merge_info() after\ntraverse_trees() returns.\n\nNote that the condition in collect_merge_info_callback() may look more\ncomplicated than necessary at first glance;\nrenames->trivial_merges_okay[side] is always true until\nhandle_deferred_entries() is called, and possible_trivial_merges[side]\nis always empty right now (and in the future won't be filled until\nhandle_deferred_entries() is called). However, when\nhandle_deferred_entries() calls traverse_trees() for the relevant\ndeferred directories, those traverse_trees() calls will once again end\nup in collect_merge_info_callback() for all the entries under those\nsubdirectories. The extra conditions are there for such deferred cases\nand will be used more as we do more with those variables.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 32 ++++++++++++++++++++++++++++++--\n 1 file changed, 30 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex eb0e18d7546..bf0712d18a0 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1141,8 +1141,35 @@ static int collect_merge_info_callback(int n,\n \t\tstruct tree_desc t[3];\n \t\tvoid *buf[3] = {NULL, NULL, NULL};\n \t\tconst char *original_dir_name;\n-\t\tint i, ret;\n+\t\tint i, ret, side;\n \n+\t\t/*\n+\t\t * Check for whether we can avoid recursing due to one side\n+\t\t * matching the merge base. The side that does NOT match is\n+\t\t * the one that might have a rename destination we need.\n+\t\t */\n+\t\tassert(!side1_matches_mbase || !side2_matches_mbase);\n+\t\tside = side1_matches_mbase ? MERGE_SIDE2 :\n+\t\t\tside2_matches_mbase ? MERGE_SIDE1 : MERGE_BASE;\n+\t\tif (filemask == 0 && (dirmask == 2 || dirmask == 4)) {\n+\t\t\t/*\n+\t\t\t * Also defer recursing into new directories; set up a\n+\t\t\t * few variables to let us do so.\n+\t\t\t */\n+\t\t\tci->match_mask = (7 - dirmask);\n+\t\t\tside = dirmask / 2;\n+\t\t}\n+\t\tif (renames->dir_rename_mask != 0x07 &&\n+\t\t (side != MERGE_BASE) &&\n+\t\t renames->trivial_merges_okay[side] &&\n+\t\t !strset_contains(&renames->target_dirs[side], pi.string)) {\n+\t\t\tstrintmap_set(&renames->possible_trivial_merges[side],\n+\t\t\t\t pi.string, renames->dir_rename_mask);\n+\t\t\trenames->dir_rename_mask = prev_dir_rename_mask;\n+\t\t\treturn mask;\n+\t\t}\n+\n+\t\t/* We need to recurse */\n \t\tci->match_mask &= filemask;\n \t\tnewinfo = *info;\n \t\tnewinfo.prev = info;\n@@ -1196,7 +1223,6 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n-MAYBE_UNUSED\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1285,6 +1311,8 @@ static int collect_merge_info(struct merge_options *opt,\n \n \ttrace2_region_enter(\"merge\", \"traverse_trees\", opt->repo);\n \tret = traverse_trees(NULL, 3, t, &info);\n+\tif (ret == 0)\n+\t\tret = handle_deferred_entries(opt, &info);\n \ttrace2_region_leave(\"merge\", \"traverse_trees\", opt->repo);\n \n \treturn ret;\n-- \ngitgitgadget\n\n"},{"id":"429928","messageId":"3409a6cd631deb361d3ecb94491c0c297c52fb53.1626204784.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] merge-ort: avoid recursing into directories when we don't need to","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-13T19:33:02Z","receivedAt":"2021-07-13T19:33:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis combines the work of the last several patches, and implements the\nconditions when we don't need to recurse into directories. It's perhaps\neasiest to see the logic by separating the fact that a directory might\nhave both rename sources and rename destinations:\n\n * rename sources: only files present in the merge base can serve as\n rename sources, and only when one side deletes that file. When the\n tree on one side matches the merge base, that means every file\n within the subtree matches the merge base. This means that the\n skip-irrelevant-rename-detection optimization from before kicks in\n and we don't need any of these files as rename sources.\n\n * rename destinations: the tree that does not match the merge base\n might have newly added and hence unmatched destination files.\n This is what usually prevents us from doing trivial directory\n resolutions in the merge machinery. However, the fact that we have\n deferred recursing into this directory until the end means we know\n whether there are any unmatched relevant potential rename sources\n elsewhere in this merge. If there are no unmatched such relevant\n sources anywhere, then there is no need to look for unmatched\n potential rename destinations to match them with.\n\nThis informs our algorithm:\n * Search through relevant_sources; if we have entries, they better all\n be reflected in cached_pairs or cached_irrelevant, otherwise they\n represent an unmatched potential rename source (causing the\n optimization to be disallowed).\n * For any relevant_source represented in cached_pairs, we do need to\n to make sure to get the destination for each source, meaning we need\n to recurse into any ancestor directories of those destinations.\n * Once we've recursed into all the rename destinations for any\n relevant_sources in cached_pairs, we can then do the trivial\n directory resolution for the remaining directories.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 5.235 s ± 0.042 s 205.1 ms ± 3.8 ms\n mega-renames: 9.419 s ± 0.107 s 1.564 s ± 0.010 s\n just-one-mega: 480.1 ms ± 3.9 ms 479.5 ms ± 3.9 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 98 insertions(+), 3 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex bf0712d18a0..c9cf7a158c8 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1223,6 +1223,18 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+static void resolve_trivial_directory_merge(struct conflict_info *ci, int side)\n+{\n+\tVERIFY_CI(ci);\n+\tassert((side == 1 && ci->match_mask == 5) ||\n+\t (side == 2 && ci->match_mask == 3));\n+\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n+\tci->merged.result.mode = ci->stages[side].mode;\n+\tci->merged.is_null = is_null_oid(&ci->stages[side].oid);\n+\tci->match_mask = 0;\n+\tci->merged.clean = 1; /* (ci->filemask == 0); */\n+}\n+\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1232,9 +1244,71 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tint side, ret = 0;\n \n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n-\t\trenames->trivial_merges_okay[side] = 0;\n-\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n-\t\t\t\t\t &iter, entry) {\n+\t\tunsigned optimization_okay = 1;\n+\t\tstruct strintmap copy;\n+\n+\t\t/* Loop over the set of paths we need to know rename info for */\n+\t\tstrset_for_each_entry(&renames->relevant_sources[side],\n+\t\t\t\t &iter, entry) {\n+\t\t\tchar *rename_target, *dir, *dir_marker;\n+\t\t\tstruct strmap_entry *e;\n+\n+\t\t\t/*\n+\t\t\t * if we don't know delete/rename info for this path,\n+\t\t\t * then we need to recurse into all trees to get all\n+\t\t\t * adds to make sure we have it.\n+\t\t\t */\n+\t\t\tif (strset_contains(&renames->cached_irrelevant[side],\n+\t\t\t\t\t entry->key))\n+\t\t\t\tcontinue;\n+\t\t\te = strmap_get_entry(&renames->cached_pairs[side],\n+\t\t\t\t\t entry->key);\n+\t\t\tif (!e) {\n+\t\t\t\toptimization_okay = 0;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/* If this is a delete, we have enough info already */\n+\t\t\trename_target = e->value;\n+\t\t\tif (!rename_target)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* If we already walked the rename target, we're good */\n+\t\t\tif (strmap_contains(&opt->priv->paths, rename_target))\n+\t\t\t\tcontinue;\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, we need to get a list of directories that\n+\t\t\t * will need to be recursed into to get this\n+\t\t\t * rename_target.\n+\t\t\t */\n+\t\t\tdir = xstrdup(rename_target);\n+\t\t\twhile ((dir_marker = strrchr(dir, '/'))) {\n+\t\t\t\t*dir_marker = '\\0';\n+\t\t\t\tif (strset_contains(&renames->target_dirs[side],\n+\t\t\t\t\t\t dir))\n+\t\t\t\t\tbreak;\n+\t\t\t\tstrset_add(&renames->target_dirs[side], dir);\n+\t\t\t}\n+\t\t\tfree(dir);\n+\t\t}\n+\t\trenames->trivial_merges_okay[side] = optimization_okay;\n+\t\t/*\n+\t\t * We need to recurse into any directories in\n+\t\t * possible_trivial_merges[side] found in target_dirs[side].\n+\t\t * But when we recurse, we may need to queue up some of the\n+\t\t * subdirectories for possible_trivial_merges[side]. Since\n+\t\t * we can't safely iterate through a hashmap while also adding\n+\t\t * entries, move the entries into 'copy', iterate over 'copy',\n+\t\t * and then we'll also iterate anything added into\n+\t\t * possible_trivial_merges[side] once this loop is done.\n+\t\t */\n+\t\tcopy = renames->possible_trivial_merges[side];\n+\t\tstrintmap_init_with_options(&renames->possible_trivial_merges[side],\n+\t\t\t\t\t 0,\n+\t\t\t\t\t NULL,\n+\t\t\t\t\t 0);\n+\t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n \t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n \t\t\tstruct conflict_info *ci;\n@@ -1247,6 +1321,13 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tVERIFY_CI(ci);\n \t\t\tdirmask = ci->dirmask;\n \n+\t\t\tif (optimization_okay &&\n+\t\t\t !strset_contains(&renames->target_dirs[side],\n+\t\t\t\t\t path)) {\n+\t\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tinfo->name = path;\n \t\t\tinfo->namelen = strlen(path);\n \t\t\tinfo->pathlen = info->namelen + 1;\n@@ -1282,6 +1363,20 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tif (ret < 0)\n \t\t\t\treturn ret;\n \t\t}\n+\t\tstrintmap_clear(©);\n+\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tstruct conflict_info *ci;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\n+\t\t\tassert(renames->trivial_merges_okay[side] &&\n+\t\t\t !strset_contains(&renames->target_dirs[side],\n+\t\t\t\t\t\tpath));\n+\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t}\n \t}\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"429951","messageId":"7f4fc9f6-9b46-6e39-1a66-b72e0c583f21@gmail.com","threadId":"56035","inReplyTo":"8aea3713902b7d006f527ccd76faf6f944529bdb.1626204784.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/7] merge-ort: add some more explanations in collect_merge_info_callback()","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-07-13T23:34:19Z","receivedAt":"2021-07-13T23:34:25Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 14/07/21 02.32, Elijah Newren via GitGitGadget wrote:\n> @@ -1018,8 +1018,8 @@ static int collect_merge_info_callback(int n,\n> \tif (side1_matches_mbase && side2_matches_mbase) {\n> \t\t/* mbase, side1, & side2 all match; use mbase as resolution */\n> \t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> -\t\t\t\tnames, names+0, mbase_null, 0,\n> -\t\t\t\tfilemask, dirmask, 1);\n> +\t\t\t\tnames, names+0, mbase_null, 0 /* df_conflict */,\n> +\t\t\t\tfilemask, dirmask, 1 /* resolved */);\n> \t\treturn mask;\n> \t}\n>\n\n\nIs df_conflict stands for directory-file conflict?\n\n> \t/*\n> -\t * Record information about the path so we can resolve later in\n> -\t * process_entries.\n> +\t * None of the special cases above matched, so we have a\n> +\t * provisional conflict. (Rename detection might allow us to\n> +\t * unconflict some more cases, but that comes later so all we can\n> +\t * do now is record the different non-null file hashes.)\n> \t */\n> \tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> \t\t\tnames, NULL, 0, df_conflict, filemask, dirmask, 0);\n> \n\nSo when none of special cases matched, we assumed there's conflict \n(although provisional), right?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"429968","messageId":"CABPp-BFBgCHAq=zWVoPPsDx04agDunAb3vVSgfJs4B2xVpasLg@mail.gmail.com","threadId":"56035","inReplyTo":"7f4fc9f6-9b46-6e39-1a66-b72e0c583f21@gmail.com","subject":"Re: [PATCH v2 2/7] merge-ort: add some more explanations in collect_merge_info_callback()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-14T00:19:11Z","receivedAt":"2021-07-14T00:19:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jul 13, 2021 at 4:34 PM Bagas Sanjaya wrote:\n>\n> On 14/07/21 02.32, Elijah Newren via GitGitGadget wrote:\n> > @@ -1018,8 +1018,8 @@ static int collect_merge_info_callback(int n,\n> > if (side1_matches_mbase && side2_matches_mbase) {\n> > /* mbase, side1, & side2 all match; use mbase as resolution */\n> > setup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> > - names, names+0, mbase_null, 0,\n> > - filemask, dirmask, 1);\n> > + names, names+0, mbase_null, 0 /* df_conflict */,\n> > + filemask, dirmask, 1 /* resolved */);\n> > return mask;\n> > }\n> >\n>\n>\n> Is df_conflict stands for directory-file conflict?\n\nYes, eventually propagating up to conflict_info->df_conflict, defined as:\n\n /* Whether this path is/was involved in a directory/file conflict */\n unsigned df_conflict:1;\n\n> > /*\n> > - * Record information about the path so we can resolve later in\n> > - * process_entries.\n> > + * None of the special cases above matched, so we have a\n> > + * provisional conflict. (Rename detection might allow us to\n> > + * unconflict some more cases, but that comes later so all we can\n> > + * do now is record the different non-null file hashes.)\n> > */\n> > setup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n> > names, NULL, 0, df_conflict, filemask, dirmask, 0);\n> >\n>\n> So when none of special cases matched, we assumed there's conflict\n> (although provisional), right?\n\nYes, the \"provisional\" adjective in particular is there because we\nrevisit each case in process_entries(), and resolve those that can be\nvia content merges, renormalization, etc. The content merges need to\nwait until after rename detection allows us to pair up different\nfiles. If the different file versions are still not resolved after\nprocess_entries() then it becomes an actual conflict rather than just\na provisional conflict.\n"},{"id":"430201","messageId":"f8d4eef7-9768-571f-2fb9-76284d69108f@gmail.com","threadId":"56035","inReplyTo":"f7ac01055d9d2e9e2dfdfd780ff7f10fbfd05d5b.1626204784.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/7] merge-ort: add data structures for allowable trivial directory resolves","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-15T13:54:26Z","receivedAt":"2021-07-15T13:54:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/13/2021 3:32 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n\nThe first two patches were easy to follow. This one requires a more\ncareful read, so here is my attempt to double-check that I am\nreading it right.\n\n> In commit f89b4f2bee (\"merge-ort: skip rename detection entirely if\n> possible\", 2021-03-11), we added an additional reason that rename\n> detection could be skipped entirely -- namely, if no *relevant* sources\n> were present. Without completing collect_merge_info_callback(), we do\n> not yet know if there are no relevant sources. However, we do know that\n> if the current directory on one side matches the merge base, then every\n> source file within that directory will not be RELEVANT_CONTENT, and a\n> few simple checks can often let us rule out RELEVANT_LOCATION as well.\n> This suggests we can just defer recursing into such directories until\n> the end of collect_merge_info.\n\nWe are not making early decisions, but collecting a list of directories\nthat cannot be relevant sources. These directories could still be\nrelevant locations (or content), but we don't want to pay the cost of\nrecursing right away.\n\n> Since the deferred directories are known to not add any relevant sources\n> due to the above properties, then if there are no relevant sources after\n> we've traversed all paths other than the deferred ones, then we know\n> there are not any relevant sources. Under those conditions, rename\n> detection is unnecessary, and that means we can resolve the deferred\n> directories without recursing into them.\n\nOnly in the case where there are no relevant sources at the end of\nscanning all directories (possibly non-recursively), then we can skip\nrecursing into these directories because finding relevant locations\nis wasted effort now.\n\nThis means the performance benefit will be limited to cases where\nrename detection isn't needed at all, but that is a frequent enough\ncase to merit the overhead of tracking this information differently.\n\n> Note that the logic for skipping rename detection was also modified\n> further in commit 76e253793c (\"merge-ort, diffcore-rename: employ cached\n> renames when possible\", 2021-01-30); in particular rename detection can\n> be skipped if we already have cached renames for each relevant source.\n> We can take advantage of this information as well with our deferral of\n> recursing into directories where one side matches the merge base.\n> \n> Add some data structures that we will use to do these deferrals, with\n> some lengthy comments explaining their purpose.\n\nAnd now that the idea is understood, we are just laying the groundwork\nfor the process by creating the tracking data structures, not actually\nthe algorithm to use them. Makes sense to split them up.\n\n> +\t/*\n> +\t * possible_trivial_merges: directories we defer recursing into\n\nNot only are we deferring the recursion, we might be able to avoid\nit entirely. Perhaps \"directories to be explored only when needed\"\nwould emphasize this point differently?\n\n> +\t *\n> +\t * possible_trivial_merges is a map of directory names to\n> +\t * dir_rename_mask. When we detect that a directory is unchanged on\n> +\t * one side, we can sometimes resolve the directory without recursing\n> +\t * into it. Renames are the only things that can prevent such an\n> +\t * optimization. However, for rename sources:\n> +\t * - If no parent directory needed directory rename detection, then\n> +\t * no path under such a directory can be a relevant_source.\n> +\t * and for rename destinations:\n> +\t * - If no cached rename has a target path under the directory AND\n> +\t * - If there are no unpaired relevant_sources elsewhere in the\n> +\t * repository\n> +\t * then we don't need any path under this directory for a rename\n> +\t * destination. The only way to know the last item above is to defer\n> +\t * handling such directories until the end of collect_merge_info(),\n> +\t * in handle_deferred_entries().\n> +\t *\n> +\t * For each we store dir_rename_mask, since that's the only bit of\n> +\t * information we need, other than the path, to resume the recursive\n> +\t * traversal.\n> +\t */\n> +\tstruct strintmap possible_trivial_merges[3];\n> +\n> +\t/*\n> +\t * trivial_merges_okay: if trivial directory merges are okay\n> +\t *\n> +\t * See possible_trivial_merges above. The \"no unpaired\n> +\t * relevant_sources elsewhere in the repository\" is a single boolean\n> +\t * per merge side, which we store here. Note that while 0 means no,\n> +\t * 1 only means \"maybe\" rather than \"yes\"; we optimistically set it\n> +\t * to 1 initially and only clear when we determine it is unsafe to\n> +\t * do trivial directory merges.\n> +\t */\n> +\tunsigned trivial_merges_okay[3];\n> +\n> +\t/*\n> +\t * target_dirs: ancestor directories of rename targets\n> +\t *\n> +\t * target_dirs contains all directory names that are an ancestor of\n> +\t * any rename destination.\n> +\t */\n> +\tstruct strset target_dirs[3];\n\nThese three members seem tightly coupled. Would it be beneficial to\ngroup the data into a 'struct trivial_merge_data' that could then\nhave an array of three here?\n\n> \t/*\n> \t * dir_rename_mask:\n> \t * 0: optimization removing unmodified potential rename source okay\n> @@ -490,6 +535,9 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> \t\tstrintmap_func(&renames->dirs_removed[i]);\n> \t\tstrmap_func(&renames->dir_renames[i], 0);\n> \t\tstrintmap_func(&renames->relevant_sources[i]);\n> +\t\tstrintmap_func(&renames->possible_trivial_merges[i]);\n> +\t\tstrset_func(&renames->target_dirs[i]);\n> +\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n\nThis could be collected into an initializer method for the data\nrequired by this optimization instead of three initializations\namong many.\n\n> \t\tif (!reinitialize)\n> \t\t\tassert(renames->cached_pairs_valid_side == 0);\n> \t\tif (i != renames->cached_pairs_valid_side) {\n> @@ -4045,12 +4093,17 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n> \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n> \t\t\t\t\t -1 /* explicitly invalid */,\n> \t\t\t\t\t NULL, 0);\n> +\t\tstrintmap_init_with_options(&renames->possible_trivial_merges[i],\n> +\t\t\t\t\t 0, NULL, 0);\n> +\t\tstrset_init_with_options(&renames->target_dirs[i],\n> +\t\t\t\t\t NULL, 1);\n> \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n> \t\t\t\t\t NULL, 1);\n> \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n> \t\t\t\t\t NULL, 1);\n> \t\tstrset_init_with_options(&renames->cached_target_names[i],\n> \t\t\t\t\t NULL, 0);\n> +\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n\nHere, there is even a separation between the grouped data. If you\ndon't like the idea of creating helper methods, then at least these\nnew lines could be grouped together. Bonus points for adding\nwhitespace between these lines and the others.\n\nThanks,\n-Stolee\n"},{"id":"430202","messageId":"4aa2f487-ad4b-b7fc-2ee0-b78c3703d955@gmail.com","threadId":"56035","inReplyTo":"7e28323b624ad2d8d12123783f00f5a8fbb248e8.1626204784.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/7] merge-ort: add a handle_deferred_entries() helper function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-15T14:32:05Z","receivedAt":"2021-07-15T14:32:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> In order to allow trivial directory resolution, we first need to be able\n> to gather more information to determine if the optimization is safe. To\n> enable that, we need a way of deferring the recursion into the directory\n> until a later time. Naturally, deferring the entry into a subtree means\n> that we need some function that will later recurse into the subdirectory\n> exactly the same way that collect_merge_info_callback() would have done.\n> \n> Add a helper function that does this. For now this function is not used\n> but a subsequent commit will change that. Future commits will also make\n> the function sometimes resolve directories instead of traversing inside.\n...> +MAYBE_UNUSED\n> +static int handle_deferred_entries(struct merge_options *opt,\n> +\t\t\t\t struct traverse_info *info)\n> +{\n> +\tstruct rename_info *renames = &opt->priv->renames;\n> +\tstruct hashmap_iter iter;\n> +\tstruct strmap_entry *entry;\n> +\tint side, ret = 0;\n> +\n> +\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n> +\t\trenames->trivial_merges_okay[side] = 0;\n\nWe are unconditionally setting this to zero? I will await the\nuse of the method to see how that affects things.\n\n> +\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n> +\t\t\t\t\t &iter, entry) {\n\nThe rest of this loop seems like standard logic for handling\nthese tree walks.\n\nThanks,\n-Stolee\n"},{"id":"430205","messageId":"24a7bea5-6325-289d-8d49-619b5e7ec65b@gmail.com","threadId":"56035","inReplyTo":"317553eadb68ce162b5ebea24045a9ca75342e91.1626204784.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/7] merge-ort: defer recursing into directories when merge base is matched","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-15T14:43:00Z","receivedAt":"2021-07-15T14:43:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n...\n> +\t\t/*\n> +\t\t * Check for whether we can avoid recursing due to one side\n> +\t\t * matching the merge base. The side that does NOT match is\n> +\t\t * the one that might have a rename destination we need.\n> +\t\t */\n> +\t\tassert(!side1_matches_mbase || !side2_matches_mbase);\n> +\t\tside = side1_matches_mbase ? MERGE_SIDE2 :\n> +\t\t\tside2_matches_mbase ? MERGE_SIDE1 : MERGE_BASE;\n> +\t\tif (filemask == 0 && (dirmask == 2 || dirmask == 4)) {\n> +\t\t\t/*\n> +\t\t\t * Also defer recursing into new directories; set up a\n> +\t\t\t * few variables to let us do so.\n> +\t\t\t */\n> +\t\t\tci->match_mask = (7 - dirmask);\n> +\t\t\tside = dirmask / 2;\n\nClever.\n\n> +\t\t}\n> +\t\tif (renames->dir_rename_mask != 0x07 &&\n> +\t\t (side != MERGE_BASE) &&\n\nnit: these parens are not necessary?\n\n> +\t\t renames->trivial_merges_okay[side] &&\n> +\t\t !strset_contains(&renames->target_dirs[side], pi.string)) {\n\nDoes this last condition mean \"this directory is not already a parent of a\nchosen rename target\"?\n\n> +\t\t\tstrintmap_set(&renames->possible_trivial_merges[side],\n> +\t\t\t\t pi.string, renames->dir_rename_mask);\n> +\t\t\trenames->dir_rename_mask = prev_dir_rename_mask;\n> +\t\t\treturn mask;\n> +\t\t}\n> +\n> +\t\t/* We need to recurse */\n> \t\tci->match_mask &= filemask;\nThanks,\n-Stolee\n"},{"id":"430207","messageId":"24c5ef26-f5d3-fb0e-b2da-24cdac90450d@gmail.com","threadId":"56035","inReplyTo":"3409a6cd631deb361d3ecb94491c0c297c52fb53.1626204784.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/7] merge-ort: avoid recursing into directories when we don't need to","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-15T14:55:26Z","receivedAt":"2021-07-15T14:55:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n...\n> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> performance work; instrument with trace2_region_* calls\", 2020-10-28),\n> this change improves the performance as follows:\n> \n> Before After\n> no-renames: 5.235 s ± 0.042 s 205.1 ms ± 3.8 ms\n\nWow! This is quite the savings, and reinforces that when no renames\nexist we are likely to hit this optimization.\n\n> mega-renames: 9.419 s ± 0.107 s 1.564 s ± 0.010 s\n\nI'm surprised that this one works so well, too. Clearly, there must\nbe a lot of directories that can be skipped despite many renames\nexisting.\n\n> just-one-mega: 480.1 ms ± 3.9 ms 479.5 ms ± 3.9 ms\n\nAnd no overhead added to this case. Good.\n\n> +\t\t/* Loop over the set of paths we need to know rename info for */\n> +\t\tstrset_for_each_entry(&renames->relevant_sources[side],\n> +\t\t\t\t &iter, entry) {\n> +\t\t\tchar *rename_target, *dir, *dir_marker;\n> +\t\t\tstruct strmap_entry *e;\n> +\n> +\t\t\t/*\n> +\t\t\t * if we don't know delete/rename info for this path,\n> +\t\t\t * then we need to recurse into all trees to get all\n> +\t\t\t * adds to make sure we have it.\n> +\t\t\t */\n\nsuper-nit: s/if/If/\n\nOtherwise, I can't speak to the code being 100% correct because\nthis area is so dense. I find the code to be well organized,\nwhich will help finding and fixing any potential bugs that might\nshow up in strange corner cases later.\n\nThanks,\n-Stolee\n"},{"id":"430208","messageId":"d91ed8a0-b37b-7dfa-10bf-e068f30e9691@gmail.com","threadId":"56035","inReplyTo":"7133f0efa520b3d0cadb059151daa12484fdb003.1626204784.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-15T15:09:59Z","receivedAt":"2021-07-15T15:10:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n...\n> Often, I noticed that when the optimization does not apply, it is\n> because there are a handful of relevant sources -- maybe even only one.\n> It felt frustrating to need to recurse into potentially hundreds or even\n> thousands of directories just for a single rename, but it was needed for\n> correctness.\n> \n> However, staring at this list of functions and noticing that\n> process_entries() is the most expensive and knowing I could avoid it if\n> I had cached renames suggested a simple idea: change\n> collect_merge_info()\n> detect_and_process_renames()\n> process_entries()\n> into\n> collect_merge_info()\n> detect_and_process_renames()\n> \n> collect_merge_info()\n> detect_and_process_renames()\n> process_entries()\n> \n> This may seem odd and look like more work. However, note that although\n> we run collect_merge_info() twice, the second time we get to employ\n> trivial directory resolves, which makes it much faster, so the increased\n> time in collect_merge_info() is small. While we run\n> detect_and_process_renames() again, all renames are cached so it's\n> nearly a no-op (we don't call into diffcore_rename_extended() but we do\n> have a little bit of data structure checking and fixing up). And the\n> big payoff comes from the fact that process_entries(), will be much\n> faster due to having far fewer entries to process.\n\nI enjoyed the story you tell here.\n\n> +\tif (path_count_after) {\n> +\t\t/*\n> +\t\t * Not sure were the right cut-off is for the optimization\n> +\t\t * to redo collect_merge_info after we've cached the\n> +\t\t * regular renames is. Basically, collect_merge_info(),\n> +\t\t * detect_regular_renames(), and process_entries() are\n> +\t\t * similar costs and all big tent poles. Caching the\n> +\t\t * result of detect_regular_renames() means that redoing\n> +\t\t * that one function will cost us virtually 0 extra, so it\n> +\t\t * depends on the other two functions, which are both O(N)\n> +\t\t * cost in the number of paths. Thus, it makes sense that\n> +\t\t * if we can cut the number of paths in half, then redoing\n> +\t\t * collect_merge_info() at half cost in order to get\n> +\t\t * process_entries() at half cost should be about equal\n> +\t\t * cost. If we can cut by more than half, then we would\n> +\t\t * win. The fact that process_entries() is about 10%-20%\n> +\t\t * more expensive than collect_merge_info() suggests we\n> +\t\t * could make the factor be less than two. The fact that\n> +\t\t * even when we have renames cached, we still have to\n> +\t\t * traverse down to the individual (relevant) renames,\n> +\t\t * which suggests we should perhaps use a bigger factor.\n> +\t\t *\n> +\t\t * The exact number isn't critical, since the code will\n> +\t\t * work even if we get the factor wrong -- it just might be\n> +\t\t * slightly slower if we're a bit off. For now, just error\n> +\t\t * on the side of a bigger fudge. For the linux kernel\n\nsuper-nit: s/linux/Linux/\n\n> +\t\t * testcases I was looking at with massive renames, the\n> +\t\t * ratio came in around 50 to 250, which clearly would\n> +\t\t * trigger this optimization and provided some *very* nice\n> +\t\t * speedups.\n\nThis bit of your testing might be more appropriate for your commit\nmessage. This discussion of a test made at a certain point in time\nis more likely to go stale than the description of how this factor\ndoes not change correctness, only performance.\n\n> +\t\t */\n> +\t\tint wanted_factor = 3;\n\nand perhaps make it 'const'?\n\n> +\n> +\t\t/* We should only redo collect_merge_info one time */\n> +\t\tassert(renames->redo_after_renames == 0);\n> +\n> +\t\tif (path_count_after / path_count_before > wanted_factor) {\n\nWith truncation from integer division, this condition is equivalent* to\n\n\tpath_count_after >= 4 * path_count_before\n\nOr, do you want to change this to a \">=\" so that the factor of 3 seems\nmore accurate?\n\n*I understand the intention of using division to avoid (unlikely)\noverflow via multiplication. The truncation is causing some confusion.\n \n> -test_expect_merge_algorithm failure failure '12f: Trivial directory resolve, caching, all kinds of fun' '\n> +test_expect_merge_algorithm failure success '12f: Trivial directory resolve, caching, all kinds of fun' '\n> \ttest_setup_12f &&\n> \t(\n> \t\tcd 12f &&\n> \n\nOh, and a bonus test success! Excellent.\n\nThanks,\n-Stolee\n"},{"id":"430227","messageId":"CABPp-BEeZuzngVfeuykmTN8R1+fX=UG4ckW=eVcrYkO-8MPO3Q@mail.gmail.com","threadId":"56035","inReplyTo":"f8d4eef7-9768-571f-2fb9-76284d69108f@gmail.com","subject":"Re: [PATCH v2 3/7] merge-ort: add data structures for allowable trivial directory resolves","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-15T15:54:36Z","receivedAt":"2021-07-15T15:54:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 15, 2021 at 6:54 AM Derrick Stolee wrote:\n>\n> On 7/13/2021 3:32 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n>\n> The first two patches were easy to follow. This one requires a more\n> careful read, so here is my attempt to double-check that I am\n> reading it right.\n\nYep, you seem to be understanding all of it just fine (does that mean\nI did a good job explaining?), and made some good suggestions to\nimprove it further.\n\n> > In commit f89b4f2bee (\"merge-ort: skip rename detection entirely if\n> > possible\", 2021-03-11), we added an additional reason that rename\n> > detection could be skipped entirely -- namely, if no *relevant* sources\n> > were present. Without completing collect_merge_info_callback(), we do\n> > not yet know if there are no relevant sources. However, we do know that\n> > if the current directory on one side matches the merge base, then every\n> > source file within that directory will not be RELEVANT_CONTENT, and a\n> > few simple checks can often let us rule out RELEVANT_LOCATION as well.\n> > This suggests we can just defer recursing into such directories until\n> > the end of collect_merge_info.\n>\n> We are not making early decisions, but collecting a list of directories\n> that cannot be relevant sources. These directories could still be\n> relevant locations (or content), but we don't want to pay the cost of\n> recursing right away.\n>\n> > Since the deferred directories are known to not add any relevant sources\n> > due to the above properties, then if there are no relevant sources after\n> > we've traversed all paths other than the deferred ones, then we know\n> > there are not any relevant sources. Under those conditions, rename\n> > detection is unnecessary, and that means we can resolve the deferred\n> > directories without recursing into them.\n>\n> Only in the case where there are no relevant sources at the end of\n> scanning all directories (possibly non-recursively), then we can skip\n> recursing into these directories because finding relevant locations\n> is wasted effort now.\n>\n> This means the performance benefit will be limited to cases where\n> rename detection isn't needed at all, but that is a frequent enough\n> case to merit the overhead of tracking this information differently.\n>\n> > Note that the logic for skipping rename detection was also modified\n> > further in commit 76e253793c (\"merge-ort, diffcore-rename: employ cached\n> > renames when possible\", 2021-01-30); in particular rename detection can\n> > be skipped if we already have cached renames for each relevant source.\n> > We can take advantage of this information as well with our deferral of\n> > recursing into directories where one side matches the merge base.\n> >\n> > Add some data structures that we will use to do these deferrals, with\n> > some lengthy comments explaining their purpose.\n>\n> And now that the idea is understood, we are just laying the groundwork\n> for the process by creating the tracking data structures, not actually\n> the algorithm to use them. Makes sense to split them up.\n>\n> > + /*\n> > + * possible_trivial_merges: directories we defer recursing into\n>\n> Not only are we deferring the recursion, we might be able to avoid\n> it entirely. Perhaps \"directories to be explored only when needed\"\n> would emphasize this point differently?\n\nI like the sound of that; seems clearer. Will update.\n\n> > + *\n> > + * possible_trivial_merges is a map of directory names to\n> > + * dir_rename_mask. When we detect that a directory is unchanged on\n> > + * one side, we can sometimes resolve the directory without recursing\n> > + * into it. Renames are the only things that can prevent such an\n> > + * optimization. However, for rename sources:\n> > + * - If no parent directory needed directory rename detection, then\n> > + * no path under such a directory can be a relevant_source.\n> > + * and for rename destinations:\n> > + * - If no cached rename has a target path under the directory AND\n> > + * - If there are no unpaired relevant_sources elsewhere in the\n> > + * repository\n> > + * then we don't need any path under this directory for a rename\n> > + * destination. The only way to know the last item above is to defer\n> > + * handling such directories until the end of collect_merge_info(),\n> > + * in handle_deferred_entries().\n> > + *\n> > + * For each we store dir_rename_mask, since that's the only bit of\n> > + * information we need, other than the path, to resume the recursive\n> > + * traversal.\n> > + */\n> > + struct strintmap possible_trivial_merges[3];\n> > +\n> > + /*\n> > + * trivial_merges_okay: if trivial directory merges are okay\n> > + *\n> > + * See possible_trivial_merges above. The \"no unpaired\n> > + * relevant_sources elsewhere in the repository\" is a single boolean\n> > + * per merge side, which we store here. Note that while 0 means no,\n> > + * 1 only means \"maybe\" rather than \"yes\"; we optimistically set it\n> > + * to 1 initially and only clear when we determine it is unsafe to\n> > + * do trivial directory merges.\n> > + */\n> > + unsigned trivial_merges_okay[3];\n> > +\n> > + /*\n> > + * target_dirs: ancestor directories of rename targets\n> > + *\n> > + * target_dirs contains all directory names that are an ancestor of\n> > + * any rename destination.\n> > + */\n> > + struct strset target_dirs[3];\n>\n> These three members seem tightly coupled. Would it be beneficial to\n> group the data into a 'struct trivial_merge_data' that could then\n> have an array of three here?\n\nOoh, I like that idea; makes sense. That'll naturally flow with your\nlast two comments too.\n\n> > /*\n> > * dir_rename_mask:\n> > * 0: optimization removing unmodified potential rename source okay\n> > @@ -490,6 +535,9 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n> > strintmap_func(&renames->dirs_removed[i]);\n> > strmap_func(&renames->dir_renames[i], 0);\n> > strintmap_func(&renames->relevant_sources[i]);\n> > + strintmap_func(&renames->possible_trivial_merges[i]);\n> > + strset_func(&renames->target_dirs[i]);\n> > + renames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n>\n> This could be collected into an initializer method for the data\n> required by this optimization instead of three initializations\n> among many.\n>\n> > if (!reinitialize)\n> > assert(renames->cached_pairs_valid_side == 0);\n> > if (i != renames->cached_pairs_valid_side) {\n> > @@ -4045,12 +4093,17 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n> > strintmap_init_with_options(&renames->relevant_sources[i],\n> > -1 /* explicitly invalid */,\n> > NULL, 0);\n> > + strintmap_init_with_options(&renames->possible_trivial_merges[i],\n> > + 0, NULL, 0);\n> > + strset_init_with_options(&renames->target_dirs[i],\n> > + NULL, 1);\n> > strmap_init_with_options(&renames->cached_pairs[i],\n> > NULL, 1);\n> > strset_init_with_options(&renames->cached_irrelevant[i],\n> > NULL, 1);\n> > strset_init_with_options(&renames->cached_target_names[i],\n> > NULL, 0);\n> > + renames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n>\n> Here, there is even a separation between the grouped data. If you\n> don't like the idea of creating helper methods, then at least these\n> new lines could be grouped together. Bonus points for adding\n> whitespace between these lines and the others.\n"},{"id":"430228","messageId":"CABPp-BGmNOGjSkJTGE7EFLv0oemzBA8kgSw_4EDjBjT+8V6oeA@mail.gmail.com","threadId":"56035","inReplyTo":"4aa2f487-ad4b-b7fc-2ee0-b78c3703d955@gmail.com","subject":"Re: [PATCH v2 4/7] merge-ort: add a handle_deferred_entries() helper function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-15T15:59:14Z","receivedAt":"2021-07-15T15:59:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 15, 2021 at 7:32 AM Derrick Stolee wrote:\n>\n> On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > In order to allow trivial directory resolution, we first need to be able\n> > to gather more information to determine if the optimization is safe. To\n> > enable that, we need a way of deferring the recursion into the directory\n> > until a later time. Naturally, deferring the entry into a subtree means\n> > that we need some function that will later recurse into the subdirectory\n> > exactly the same way that collect_merge_info_callback() would have done.\n> >\n> > Add a helper function that does this. For now this function is not used\n> > but a subsequent commit will change that. Future commits will also make\n> > the function sometimes resolve directories instead of traversing inside.\n> ...> +MAYBE_UNUSED\n> > +static int handle_deferred_entries(struct merge_options *opt,\n> > + struct traverse_info *info)\n> > +{\n> > + struct rename_info *renames = &opt->priv->renames;\n> > + struct hashmap_iter iter;\n> > + struct strmap_entry *entry;\n> > + int side, ret = 0;\n> > +\n> > + for (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n> > + renames->trivial_merges_okay[side] = 0;\n>\n> We are unconditionally setting this to zero? I will await the\n> use of the method to see how that affects things.\n\nUnconditionally set to 0 because at this point in the series we don't\nyet have the logic to handle it. This line will be replaced later\nonce we add the necessary logic.\n\n>\n> > + strintmap_for_each_entry(&renames->possible_trivial_merges[side],\n> > + &iter, entry) {\n>\n> The rest of this loop seems like standard logic for handling\n> these tree walks.\n>\n> Thanks,\n> -Stolee\n"},{"id":"430229","messageId":"CABPp-BGX-02M2Aw+Fw-bArNd5tOdp_sgcE_oRGiTDicNsgxdgA@mail.gmail.com","threadId":"56035","inReplyTo":"24a7bea5-6325-289d-8d49-619b5e7ec65b@gmail.com","subject":"Re: [PATCH v2 5/7] merge-ort: defer recursing into directories when merge base is matched","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-15T16:03:29Z","receivedAt":"2021-07-15T16:03:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 15, 2021 at 7:43 AM Derrick Stolee wrote:\n>\n> On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> ...\n> > + /*\n> > + * Check for whether we can avoid recursing due to one side\n> > + * matching the merge base. The side that does NOT match is\n> > + * the one that might have a rename destination we need.\n> > + */\n> > + assert(!side1_matches_mbase || !side2_matches_mbase);\n> > + side = side1_matches_mbase ? MERGE_SIDE2 :\n> > + side2_matches_mbase ? MERGE_SIDE1 : MERGE_BASE;\n> > + if (filemask == 0 && (dirmask == 2 || dirmask == 4)) {\n> > + /*\n> > + * Also defer recursing into new directories; set up a\n> > + * few variables to let us do so.\n> > + */\n> > + ci->match_mask = (7 - dirmask);\n> > + side = dirmask / 2;\n>\n> Clever.\n>\n> > + }\n> > + if (renames->dir_rename_mask != 0x07 &&\n> > + (side != MERGE_BASE) &&\n>\n> nit: these parens are not necessary?\n\nIndeed; I'll remove them.\n\n> > + renames->trivial_merges_okay[side] &&\n> > + !strset_contains(&renames->target_dirs[side], pi.string)) {\n>\n> Does this last condition mean \"this directory is not already a parent of a\n> chosen rename target\"?\n\nI'm not sure what you mean by \"chosen\" here. If by \"chosen\" you mean\n\"cached\" (which comes from ort-perf-batch-11's caching of upstream\nrenames from previously cherry-picked commits), then yes.\n\nPerhaps it's worth noting that rename detection has not yet been run\nfor this merge by the time we hit this logic, so the only renames we\ncan know are the cached ones from a previous pick.\n\n> > + strintmap_set(&renames->possible_trivial_merges[side],\n> > + pi.string, renames->dir_rename_mask);\n> > + renames->dir_rename_mask = prev_dir_rename_mask;\n> > + return mask;\n> > + }\n> > +\n> > + /* We need to recurse */\n> > ci->match_mask &= filemask;\n> Thanks,\n> -Stolee\n"},{"id":"430231","messageId":"CABPp-BHk6=WR-T-tFJ8v_xpqp0emW8Z29VkCkxePfR_tuUcBJw@mail.gmail.com","threadId":"56035","inReplyTo":"24c5ef26-f5d3-fb0e-b2da-24cdac90450d@gmail.com","subject":"Re: [PATCH v2 6/7] merge-ort: avoid recursing into directories when we don't need to","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-15T16:28:43Z","receivedAt":"2021-07-15T16:28:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 15, 2021 at 7:55 AM Derrick Stolee wrote:\n>\n> On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> ...\n> > For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> > performance work; instrument with trace2_region_* calls\", 2020-10-28),\n> > this change improves the performance as follows:\n> >\n> > Before After\n> > no-renames: 5.235 s ± 0.042 s 205.1 ms ± 3.8 ms\n>\n> Wow! This is quite the savings, and reinforces that when no renames\n> exist we are likely to hit this optimization.\n\nJust to clarify, \"no-renames\" was an unfortunate misnomer. I should\nhave called it \"few-renames\".\n\nBut yeah, I was pretty amazed a year ago (when I initially implemented\nthis optimization) seeing the speedups on this testcase -- a testcase\nthat I hadn't even been paying much attention to beyond trying to\navoid regressing it.\n\n> > mega-renames: 9.419 s ± 0.107 s 1.564 s ± 0.010 s\n>\n> I'm surprised that this one works so well, too. Clearly, there must\n> be a lot of directories that can be skipped despite many renames\n> existing.\n\nOne small reason it works is that some of the commits don't touch any\npaths involved in the big directory rename. That means that for those\ncommits, all the renames under that big directory are irrelevant.\nThis fact makes the optimization work for 4 of the 35 patches. (Of\ncourse, I did pick that particular series of commits for rebasing\nbecause it touched the directory being renamed a lot, including adding\nfiles to it, which made it a better stress test case for my\noptimizations. But that does make it less likely for the\nirrelevant-renames criteria to help us out on this particular\noptimization.)\n\nThe bigger reason this optimization works so well on this testcase, is\nthe mere fact that this rebase involved a sequence of 35 commits. The\noptimization doesn't work for the first commit (as noted by the\njust-one-mega testcase below). However, once the first commit is\npicked, renames on the upstream side that were relevant in the first\ncommit are cached, so the optimization does trigger for many of the\nsubsequent commits. And whenever the optimization doesn't work on a\nsubsequent commit -- which happens because the new commit modifies\ndifferent files that make other renames become relevant -- then after\ndetecting the new renames those also get cached. More cached renames\nmeans a higher likelihood that we can skip recursing into directories\nwhen rebasing subsequent commits.\n\n> > just-one-mega: 480.1 ms ± 3.9 ms 479.5 ms ± 3.9 ms\n>\n> And no overhead added to this case. Good.\n>\n> > + /* Loop over the set of paths we need to know rename info for */\n> > + strset_for_each_entry(&renames->relevant_sources[side],\n> > + &iter, entry) {\n> > + char *rename_target, *dir, *dir_marker;\n> > + struct strmap_entry *e;\n> > +\n> > + /*\n> > + * if we don't know delete/rename info for this path,\n> > + * then we need to recurse into all trees to get all\n> > + * adds to make sure we have it.\n> > + */\n>\n> super-nit: s/if/If/\n\nWill fix.\n\n> Otherwise, I can't speak to the code being 100% correct because\n> this area is so dense. I find the code to be well organized,\n> which will help finding and fixing any potential bugs that might\n> show up in strange corner cases later.\n>\n> Thanks,\n> -Stolee\n"},{"id":"430238","messageId":"CABPp-BF+gR8WtpWt_DVDoWe16R4B65h-59zGOZ5j4vUJKp_Nuw@mail.gmail.com","threadId":"56035","inReplyTo":"d91ed8a0-b37b-7dfa-10bf-e068f30e9691@gmail.com","subject":"Re: [PATCH v2 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-15T16:53:49Z","receivedAt":"2021-07-15T16:54:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 15, 2021 at 8:10 AM Derrick Stolee wrote:\n>\n> On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> ...\n> > Often, I noticed that when the optimization does not apply, it is\n> > because there are a handful of relevant sources -- maybe even only one.\n> > It felt frustrating to need to recurse into potentially hundreds or even\n> > thousands of directories just for a single rename, but it was needed for\n> > correctness.\n> >\n> > However, staring at this list of functions and noticing that\n> > process_entries() is the most expensive and knowing I could avoid it if\n> > I had cached renames suggested a simple idea: change\n> > collect_merge_info()\n> > detect_and_process_renames()\n> > process_entries()\n> > into\n> > collect_merge_info()\n> > detect_and_process_renames()\n> > \n> > collect_merge_info()\n> > detect_and_process_renames()\n> > process_entries()\n> >\n> > This may seem odd and look like more work. However, note that although\n> > we run collect_merge_info() twice, the second time we get to employ\n> > trivial directory resolves, which makes it much faster, so the increased\n> > time in collect_merge_info() is small. While we run\n> > detect_and_process_renames() again, all renames are cached so it's\n> > nearly a no-op (we don't call into diffcore_rename_extended() but we do\n> > have a little bit of data structure checking and fixing up). And the\n> > big payoff comes from the fact that process_entries(), will be much\n> > faster due to having far fewer entries to process.\n>\n> I enjoyed the story you tell here.\n\n:-)\n\n> > + if (path_count_after) {\n> > + /*\n> > + * Not sure were the right cut-off is for the optimization\n> > + * to redo collect_merge_info after we've cached the\n> > + * regular renames is. Basically, collect_merge_info(),\n> > + * detect_regular_renames(), and process_entries() are\n> > + * similar costs and all big tent poles. Caching the\n> > + * result of detect_regular_renames() means that redoing\n> > + * that one function will cost us virtually 0 extra, so it\n> > + * depends on the other two functions, which are both O(N)\n> > + * cost in the number of paths. Thus, it makes sense that\n> > + * if we can cut the number of paths in half, then redoing\n> > + * collect_merge_info() at half cost in order to get\n> > + * process_entries() at half cost should be about equal\n> > + * cost. If we can cut by more than half, then we would\n> > + * win. The fact that process_entries() is about 10%-20%\n> > + * more expensive than collect_merge_info() suggests we\n> > + * could make the factor be less than two. The fact that\n> > + * even when we have renames cached, we still have to\n> > + * traverse down to the individual (relevant) renames,\n> > + * which suggests we should perhaps use a bigger factor.\n> > + *\n> > + * The exact number isn't critical, since the code will\n> > + * work even if we get the factor wrong -- it just might be\n> > + * slightly slower if we're a bit off. For now, just error\n> > + * on the side of a bigger fudge. For the linux kernel\n>\n> super-nit: s/linux/Linux/\n\nYeah, I tend to refer to projects by the name of their repository\ninstead of their proper name. (I do it with git too.) Bad habit.\nWill fix. That is, I will fix this instance. Not sure I can fix the\nhabit.\n\n> > + * testcases I was looking at with massive renames, the\n> > + * ratio came in around 50 to 250, which clearly would\n> > + * trigger this optimization and provided some *very* nice\n> > + * speedups.\n>\n> This bit of your testing might be more appropriate for your commit\n> message. This discussion of a test made at a certain point in time\n> is more likely to go stale than the description of how this factor\n> does not change correctness, only performance.\n\nThe commit message does include discussion on how this factor only\nchanges performance, not correctness. I left this comment in the code\nbecause I figured it looked weird and magic and deserved an\nexplanation without resorting to git-log or git-blame. Granted, I\nwrote this comment and the commit message at much different times (I\nwrote the comment first, then the commit message many months later)\nand thus summarized a bit differently. But I thought I had the same\nrelevant content in both places.\n\nAre there pieces you feel are missing from the commit message? Should\nI trim this comment down in the code and just let people look for the\ncommit message for more details?\n\n> > + */\n> > + int wanted_factor = 3;\n>\n> and perhaps make it 'const'?\n\nSure, will do.\n\n> > +\n> > + /* We should only redo collect_merge_info one time */\n> > + assert(renames->redo_after_renames == 0);\n> > +\n> > + if (path_count_after / path_count_before > wanted_factor) {\n>\n> With truncation from integer division, this condition is equivalent* to\n>\n> path_count_after >= 4 * path_count_before\n>\n> Or, do you want to change this to a \">=\" so that the factor of 3 seems\n> more accurate?\n>\n> *I understand the intention of using division to avoid (unlikely)\n> overflow via multiplication. The truncation is causing some confusion.\n\nGood catch; I'll fix it up to use \">=\".\n\n> > -test_expect_merge_algorithm failure failure '12f: Trivial directory resolve, caching, all kinds of fun' '\n> > +test_expect_merge_algorithm failure success '12f: Trivial directory resolve, caching, all kinds of fun' '\n> > test_setup_12f &&\n> > (\n> > cd 12f &&\n> >\n>\n> Oh, and a bonus test success! Excellent.\n\nYeah, this testcase was slightly weird in the order I sent it\nupstream. 12f was written specifically with this optimization in mind\nas a way of ensuring code coverage of the restart logic. I would have\nwaited to submit the 12f testcase with this series, but the testcase\nalso demonstrated an important directory rename detection bug that I\nfound existed in both merge-recursive and merge-ort at the time. See\ncommit 902c521a35 (\"t6423: more involved directory rename test\",\n2020-10-15)\n\nThe merge-ort bug was fixed with commit 203c872c4f (\"merge-ort: fix a\ndirectory rename detection bug\", 2021-01-19). The merge-recursive bug\nstill exists.\n\nSo, this series just fixed up the final thing that 12f was testing for\n-- namely that it included two collect_merge_info() region_enter\ntrace2 calls per commit instead of just one.\n\nPerhaps I could have split the test, but both things require a\nrelatively big set of files which makes the test a bit more expensive\nso I didn't want to duplicate it. Besides, having both factors\ninvolved makes it a better stress test of the restart logic.\n"},{"id":"430242","messageId":"9dedcf3b-5484-89d6-6800-77e18dfa6ea3@gmail.com","threadId":"56035","inReplyTo":"CABPp-BGX-02M2Aw+Fw-bArNd5tOdp_sgcE_oRGiTDicNsgxdgA@mail.gmail.com","subject":"Re: [PATCH v2 5/7] merge-ort: defer recursing into directories when merge base is matched","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-15T17:14:17Z","receivedAt":"2021-07-15T17:14:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/15/2021 12:03 PM, Elijah Newren wrote:\n> On Thu, Jul 15, 2021 at 7:43 AM Derrick Stolee wrote:\n>>\n>> On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n...\n>>> + renames->trivial_merges_okay[side] &&\n>>> + !strset_contains(&renames->target_dirs[side], pi.string)) {\n>>\n>> Does this last condition mean \"this directory is not already a parent of a\n>> chosen rename target\"?\n> \n> I'm not sure what you mean by \"chosen\" here. If by \"chosen\" you mean\n> \"cached\" (which comes from ort-perf-batch-11's caching of upstream\n> renames from previously cherry-picked commits), then yes.\n> \n> Perhaps it's worth noting that rename detection has not yet been run\n> for this merge by the time we hit this logic, so the only renames we\n> can know are the cached ones from a previous pick.\n\nI was missing the interaction with the cached results, which now makes\nmuch more sense here.\n\nThanks,\n-Stolee\n"},{"id":"430245","messageId":"9260bd3d-2325-fde7-bfa2-081700f6a731@gmail.com","threadId":"56035","inReplyTo":"CABPp-BF+gR8WtpWt_DVDoWe16R4B65h-59zGOZ5j4vUJKp_Nuw@mail.gmail.com","subject":"Re: [PATCH v2 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-15T17:19:16Z","receivedAt":"2021-07-15T17:19:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/15/2021 12:53 PM, Elijah Newren wrote:\n> On Thu, Jul 15, 2021 at 8:10 AM Derrick Stolee wrote:\n>>\n>> On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n...\n>>> + * The exact number isn't critical, since the code will\n>>> + * work even if we get the factor wrong -- it just might be\n>>> + * slightly slower if we're a bit off. For now, just error\n>>> + * on the side of a bigger fudge. For the linux kernel\n>>\n>> super-nit: s/linux/Linux/\n> \n> Yeah, I tend to refer to projects by the name of their repository\n> instead of their proper name. (I do it with git too.) Bad habit.\n> Will fix. That is, I will fix this instance. Not sure I can fix the\n> habit.\n\nIf you had written it as torvalds/linux, then I wouldn't have batted\nan eye, because that is clearly a repo name (at least, clear to me).\n\n>>> + * testcases I was looking at with massive renames, the\n>>> + * ratio came in around 50 to 250, which clearly would\n>>> + * trigger this optimization and provided some *very* nice\n>>> + * speedups.\n>>\n>> This bit of your testing might be more appropriate for your commit\n>> message. This discussion of a test made at a certain point in time\n>> is more likely to go stale than the description of how this factor\n>> does not change correctness, only performance.\n> \n> The commit message does include discussion on how this factor only\n> changes performance, not correctness. I left this comment in the code\n> because I figured it looked weird and magic and deserved an\n> explanation without resorting to git-log or git-blame. Granted, I\n> wrote this comment and the commit message at much different times (I\n> wrote the comment first, then the commit message many months later)\n> and thus summarized a bit differently. But I thought I had the same\n> relevant content in both places.\n> \n> Are there pieces you feel are missing from the commit message? Should\n> I trim this comment down in the code and just let people look for the\n> commit message for more details?\n\nI meant to say \"these kinds of details are better for the commit\nmessage instead of comments\" and not say \"your commit message\ndoesn't have enough.\"\n\nI don't feel strongly about this being present in the comment or\nnot, but it seems like something that could be dropped from the\ncomment without loss of information.\n\nThanks,\n-Stolee\n"},{"id":"430248","messageId":"CABPp-BEHC8dbGmMq9M-Jn899c+cVN4AhOD8Bio52WupEywEa2g@mail.gmail.com","threadId":"56035","inReplyTo":"9260bd3d-2325-fde7-bfa2-081700f6a731@gmail.com","subject":"Re: [PATCH v2 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-15T17:32:20Z","receivedAt":"2021-07-15T17:32:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 15, 2021 at 10:19 AM Derrick Stolee wrote:\n>\n> On 7/15/2021 12:53 PM, Elijah Newren wrote:\n> > On Thu, Jul 15, 2021 at 8:10 AM Derrick Stolee wrote:\n> >>\n> >> On 7/13/2021 3:33 PM, Elijah Newren via GitGitGadget wrote:\n> >>> From: Elijah Newren \n> ...\n> >>> + * The exact number isn't critical, since the code will\n> >>> + * work even if we get the factor wrong -- it just might be\n> >>> + * slightly slower if we're a bit off. For now, just error\n> >>> + * on the side of a bigger fudge. For the linux kernel\n> >>\n> >> super-nit: s/linux/Linux/\n> >\n> > Yeah, I tend to refer to projects by the name of their repository\n> > instead of their proper name. (I do it with git too.) Bad habit.\n> > Will fix. That is, I will fix this instance. Not sure I can fix the\n> > habit.\n>\n> If you had written it as torvalds/linux, then I wouldn't have batted\n> an eye, because that is clearly a repo name (at least, clear to me).\n\nYeah, the thing is I use the repo name even in cases where it's not\nambiguous whether the project or the repo is meant. For example, I\nwould often write something like \"I'm part of the git project.\"\nChristian has been trying to fix my capitalization. Folks in another\nproject tried to \"fix\" my capitalization too. Perhaps by demanding\nfive letters of all caps, they were able to get one out of me[1]. I\nsuspect, though, that if they had just a single repo instead of having\nhundreds of repositories, they might not have even gotten that one\nletter of capitalization from me. ;-)\n\n[1] https://blogs.gnome.org/newren/2006/04/22/enlightening-mankind-about-the-correct-spelling-of-gnome/\n\n> >>> + * testcases I was looking at with massive renames, the\n> >>> + * ratio came in around 50 to 250, which clearly would\n> >>> + * trigger this optimization and provided some *very* nice\n> >>> + * speedups.\n> >>\n> >> This bit of your testing might be more appropriate for your commit\n> >> message. This discussion of a test made at a certain point in time\n> >> is more likely to go stale than the description of how this factor\n> >> does not change correctness, only performance.\n> >\n> > The commit message does include discussion on how this factor only\n> > changes performance, not correctness. I left this comment in the code\n> > because I figured it looked weird and magic and deserved an\n> > explanation without resorting to git-log or git-blame. Granted, I\n> > wrote this comment and the commit message at much different times (I\n> > wrote the comment first, then the commit message many months later)\n> > and thus summarized a bit differently. But I thought I had the same\n> > relevant content in both places.\n> >\n> > Are there pieces you feel are missing from the commit message? Should\n> > I trim this comment down in the code and just let people look for the\n> > commit message for more details?\n>\n> I meant to say \"these kinds of details are better for the commit\n> message instead of comments\" and not say \"your commit message\n> doesn't have enough.\"\n>\n> I don't feel strongly about this being present in the comment or\n> not, but it seems like something that could be dropped from the\n> comment without loss of information.\n\nAh, makes sense. I'll trim it down.\n"},{"id":"430294","messageId":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v2.git.1626204784.gitgitgadget@gmail.com","subject":"[PATCH v3 0/7] Optimization batch 14: trivial directory resolution","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:30Z","receivedAt":"2021-07-16T05:22:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends textually on ort-perf-batch-12, but is semantically\nindependent. (It is both semantically and textually independent of\nort-perf-batch-13.)\n\nMost of my previous series dramatically accelerated cases with lots of\nrenames, while providing comparatively minor benefits for cases with few or\nno renames. This series is the opposite; it provides huge benefits when\nthere are few or no renames, and comparatively smaller (though still quite\ndecent) benefits for cases with many uncached renames.\n\nChanges since v2, addressing feedback from Stolee:\n\n * Created a separate struct for three related variables to hint they are\n related\n * Simplified a lengthy comment that was duplicated by the commit message\n * Various other minor cleanups\n\nChanges since v1:\n\n * Minor tweak to the final patch to correct implicit assumption that rename\n detection running implies all renames were found (rename limits could\n have been exceeded and prevented finding renames)\n\n=== Basic Optimization idea ===\n\nunpack_trees has had a concept of trivial merges for individual files (see\nDocumentation/technical/trivial-merge.txt). The same idea can be applied in\nmerge-ort. It'd be really nice to extend that idea to trees as well, as it\ncould provide a huge performance boost; sadly however, applying it in\ngeneral would wreck both regular rename detection (the unmatched side can\nhave new files that serve as potential destinations in rename detection) and\ndirectory rename detection (the unmatched side could have a new directory\nthat was moved into it).\n\nIf we somehow knew rename detection wasn't needed, we could do trivial\ndirectory resolution. In the past, this wasn't possible. However...\n\nWith recent optimizations we have created a possibility to do trivial\ndirectory resolutions in some cases. These came from the addition of the\n\"skipping irrelevant renames\" optimizations (from ort-perf-batch-9 and\nort-perf-batch-10), and in particular noting that we added an ability to\nentirely skip rename detection in commit f89b4f2bee (\"merge-ort: skip rename\ndetection entirely if possible\", 2021-03-11) when there are no relevant\nsources. We can detect if there are no relevant sources without recursing\ninto the directories in question.\n\nAs a cherry on top, the caching of renames (from ort-perf-batch-11) allows\nus to cover additional cases.\n\nThis series is all about adding all the special checks needed to safely\nperform trival directory resolutions.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 5.235 s ± 0.042 s 204.2 ms ± 3.0 ms\nmega-renames: 9.419 s ± 0.107 s 1.076 s ± 0.015 s\njust-one-mega: 480.1 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with (for merge-recursive as of git-2.30.0)\nwere:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\nElijah Newren (7):\n merge-ort: resolve paths early when we have sufficient information\n merge-ort: add some more explanations in collect_merge_info_callback()\n merge-ort: add data structures for allowable trivial directory\n resolves\n merge-ort: add a handle_deferred_entries() helper function\n merge-ort: defer recursing into directories when merge base is matched\n merge-ort: avoid recursing into directories when we don't need to\n merge-ort: restart merge with cached renames to reduce process entry\n cost\n\n merge-ort.c | 399 +++++++++++++++++++++++++++-\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 389 insertions(+), 12 deletions(-)\n\n\nbase-commit: 2eeee12b02e441ac05054a5a5ecbcea6964a1e6b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-988%2Fnewren%2Fort-perf-batch-14-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-988/newren/ort-perf-batch-14-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/988\n\nRange-diff vs v2:\n\n 1: 5dca982c0b0 = 1: 5dca982c0b0 merge-ort: resolve paths early when we have sufficient information\n 2: 8aea3713902 = 2: 8aea3713902 merge-ort: add some more explanations in collect_merge_info_callback()\n 3: f7ac01055d9 ! 3: c2b45fef1d7 merge-ort: add data structures for allowable trivial directory resolves\n @@ Commit message\n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n -@@ merge-ort.c: struct rename_info {\n - \t */\n - \tstruct strintmap relevant_sources[3];\n +@@ merge-ort.c: struct traversal_callback_data {\n + \tstruct name_entry names[3];\n + };\n \n ++struct deferred_traversal_data {\n +\t/*\n -+\t * possible_trivial_merges: directories we defer recursing into\n ++\t * possible_trivial_merges: directories to be explored only when needed\n +\t *\n +\t * possible_trivial_merges is a map of directory names to\n +\t * dir_rename_mask. When we detect that a directory is unchanged on\n @@ merge-ort.c: struct rename_info {\n +\t * information we need, other than the path, to resume the recursive\n +\t * traversal.\n +\t */\n -+\tstruct strintmap possible_trivial_merges[3];\n ++\tstruct strintmap possible_trivial_merges;\n +\n +\t/*\n +\t * trivial_merges_okay: if trivial directory merges are okay\n @@ merge-ort.c: struct rename_info {\n +\t * to 1 initially and only clear when we determine it is unsafe to\n +\t * do trivial directory merges.\n +\t */\n -+\tunsigned trivial_merges_okay[3];\n ++\tunsigned trivial_merges_okay;\n +\n +\t/*\n +\t * target_dirs: ancestor directories of rename targets\n @@ merge-ort.c: struct rename_info {\n +\t * target_dirs contains all directory names that are an ancestor of\n +\t * any rename destination.\n +\t */\n -+\tstruct strset target_dirs[3];\n ++\tstruct strset target_dirs;\n ++};\n ++\n + struct rename_info {\n + \t/*\n + \t * All variables that are arrays of size 3 correspond to data tracked\n +@@ merge-ort.c: struct rename_info {\n + \t */\n + \tstruct strintmap relevant_sources[3];\n + \n ++\tstruct deferred_traversal_data deferred[3];\n +\n \t/*\n \t * dir_rename_mask:\n \t * 0: optimization removing unmodified potential rename source okay\n @@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n - \t\tstrintmap_func(&renames->dirs_removed[i]);\n - \t\tstrmap_func(&renames->dir_renames[i], 0);\n - \t\tstrintmap_func(&renames->relevant_sources[i]);\n -+\t\tstrintmap_func(&renames->possible_trivial_merges[i]);\n -+\t\tstrset_func(&renames->target_dirs[i]);\n -+\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n - \t\tif (!reinitialize)\n - \t\t\tassert(renames->cached_pairs_valid_side == 0);\n - \t\tif (i != renames->cached_pairs_valid_side) {\n + \t\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n + \t\t}\n + \t}\n ++\tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n ++\t\tstrintmap_func(&renames->deferred[i].possible_trivial_merges);\n ++\t\tstrset_func(&renames->deferred[i].target_dirs);\n ++\t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n ++\t}\n + \trenames->cached_pairs_valid_side = 0;\n + \trenames->dir_rename_mask = 0;\n + \n @@ merge-ort.c: static void merge_start(struct merge_options *opt, struct merge_result *result)\n - \t\tstrintmap_init_with_options(&renames->relevant_sources[i],\n - \t\t\t\t\t -1 /* explicitly invalid */,\n - \t\t\t\t\t NULL, 0);\n -+\t\tstrintmap_init_with_options(&renames->possible_trivial_merges[i],\n -+\t\t\t\t\t 0, NULL, 0);\n -+\t\tstrset_init_with_options(&renames->target_dirs[i],\n -+\t\t\t\t\t NULL, 1);\n - \t\tstrmap_init_with_options(&renames->cached_pairs[i],\n - \t\t\t\t\t NULL, 1);\n - \t\tstrset_init_with_options(&renames->cached_irrelevant[i],\n - \t\t\t\t\t NULL, 1);\n \t\tstrset_init_with_options(&renames->cached_target_names[i],\n \t\t\t\t\t NULL, 0);\n -+\t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n \t}\n ++\tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n ++\t\tstrintmap_init_with_options(&renames->deferred[i].possible_trivial_merges,\n ++\t\t\t\t\t 0, NULL, 0);\n ++\t\tstrset_init_with_options(&renames->deferred[i].target_dirs,\n ++\t\t\t\t\t NULL, 1);\n ++\t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n ++\t}\n \n \t/*\n + \t * Although we initialize opt->priv->paths with strdup_strings=0,\n 4: 7e28323b624 ! 4: 1cf4a47562a merge-ort: add a handle_deferred_entries() helper function\n @@ merge-ort.c: static int collect_merge_info_callback(int n,\n +\tint side, ret = 0;\n +\n +\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n -+\t\trenames->trivial_merges_okay[side] = 0;\n -+\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n ++\t\trenames->deferred[side].trivial_merges_okay = 0;\n ++\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n +\t\t\t\t\t &iter, entry) {\n +\t\t\tconst char *path = entry->key;\n +\t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n 5: 317553eadb6 ! 5: 79c51536829 merge-ort: defer recursing into directories when merge base is matched\n @@ merge-ort.c: static int collect_merge_info_callback(int n,\n +\t\t\tside = dirmask / 2;\n +\t\t}\n +\t\tif (renames->dir_rename_mask != 0x07 &&\n -+\t\t (side != MERGE_BASE) &&\n -+\t\t renames->trivial_merges_okay[side] &&\n -+\t\t !strset_contains(&renames->target_dirs[side], pi.string)) {\n -+\t\t\tstrintmap_set(&renames->possible_trivial_merges[side],\n ++\t\t side != MERGE_BASE &&\n ++\t\t renames->deferred[side].trivial_merges_okay &&\n ++\t\t !strset_contains(&renames->deferred[side].target_dirs,\n ++\t\t\t\t pi.string)) {\n ++\t\t\tstrintmap_set(&renames->deferred[side].possible_trivial_merges,\n +\t\t\t\t pi.string, renames->dir_rename_mask);\n +\t\t\trenames->dir_rename_mask = prev_dir_rename_mask;\n +\t\t\treturn mask;\n 6: 3409a6cd631 ! 6: 572cc5e94d2 merge-ort: avoid recursing into directories when we don't need to\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n \tint side, ret = 0;\n \n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n --\t\trenames->trivial_merges_okay[side] = 0;\n --\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n +-\t\trenames->deferred[side].trivial_merges_okay = 0;\n +-\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n -\t\t\t\t\t &iter, entry) {\n +\t\tunsigned optimization_okay = 1;\n +\t\tstruct strintmap copy;\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n +\t\t\tstruct strmap_entry *e;\n +\n +\t\t\t/*\n -+\t\t\t * if we don't know delete/rename info for this path,\n ++\t\t\t * If we don't know delete/rename info for this path,\n +\t\t\t * then we need to recurse into all trees to get all\n +\t\t\t * adds to make sure we have it.\n +\t\t\t */\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n +\t\t\tdir = xstrdup(rename_target);\n +\t\t\twhile ((dir_marker = strrchr(dir, '/'))) {\n +\t\t\t\t*dir_marker = '\\0';\n -+\t\t\t\tif (strset_contains(&renames->target_dirs[side],\n ++\t\t\t\tif (strset_contains(&renames->deferred[side].target_dirs,\n +\t\t\t\t\t\t dir))\n +\t\t\t\t\tbreak;\n -+\t\t\t\tstrset_add(&renames->target_dirs[side], dir);\n ++\t\t\t\tstrset_add(&renames->deferred[side].target_dirs,\n ++\t\t\t\t\t dir);\n +\t\t\t}\n +\t\t\tfree(dir);\n +\t\t}\n -+\t\trenames->trivial_merges_okay[side] = optimization_okay;\n ++\t\trenames->deferred[side].trivial_merges_okay = optimization_okay;\n +\t\t/*\n +\t\t * We need to recurse into any directories in\n +\t\t * possible_trivial_merges[side] found in target_dirs[side].\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n +\t\t * and then we'll also iterate anything added into\n +\t\t * possible_trivial_merges[side] once this loop is done.\n +\t\t */\n -+\t\tcopy = renames->possible_trivial_merges[side];\n -+\t\tstrintmap_init_with_options(&renames->possible_trivial_merges[side],\n ++\t\tcopy = renames->deferred[side].possible_trivial_merges;\n ++\t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n +\t\t\t\t\t 0,\n +\t\t\t\t\t NULL,\n +\t\t\t\t\t 0);\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tdirmask = ci->dirmask;\n \n +\t\t\tif (optimization_okay &&\n -+\t\t\t !strset_contains(&renames->target_dirs[side],\n ++\t\t\t !strset_contains(&renames->deferred[side].target_dirs,\n +\t\t\t\t\t path)) {\n +\t\t\t\tresolve_trivial_directory_merge(ci, side);\n +\t\t\t\tcontinue;\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\treturn ret;\n \t\t}\n +\t\tstrintmap_clear(©);\n -+\t\tstrintmap_for_each_entry(&renames->possible_trivial_merges[side],\n ++\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n +\t\t\t\t\t &iter, entry) {\n +\t\t\tconst char *path = entry->key;\n +\t\t\tstruct conflict_info *ci;\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n +\t\t\tci = strmap_get(&opt->priv->paths, path);\n +\t\t\tVERIFY_CI(ci);\n +\n -+\t\t\tassert(renames->trivial_merges_okay[side] &&\n -+\t\t\t !strset_contains(&renames->target_dirs[side],\n ++\t\t\tassert(renames->deferred[side].trivial_merges_okay &&\n ++\t\t\t !strset_contains(&renames->deferred[side].target_dirs,\n +\t\t\t\t\t\tpath));\n +\t\t\tresolve_trivial_directory_merge(ci, side);\n +\t\t}\n 7: 7133f0efa52 ! 7: a9cbc1d4f18 merge-ort: restart merge with cached renames to reduce process entry cost\n @@ merge-ort.c: struct rename_info {\n \t * needed_limit: value needed for inexact rename detection to run\n \t *\n @@ merge-ort.c: static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n - \t\trenames->trivial_merges_okay[i] = 1; /* 1 == maybe */\n + \t\tstrintmap_func(&renames->relevant_sources[i]);\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n -\t\tif (i != renames->cached_pairs_valid_side) {\n @@ merge-ort.c: static int handle_deferred_entries(struct merge_options *opt,\n \t}\n +\tif (path_count_after) {\n +\t\t/*\n -+\t\t * Not sure were the right cut-off is for the optimization\n -+\t\t * to redo collect_merge_info after we've cached the\n -+\t\t * regular renames is. Basically, collect_merge_info(),\n -+\t\t * detect_regular_renames(), and process_entries() are\n -+\t\t * similar costs and all big tent poles. Caching the\n -+\t\t * result of detect_regular_renames() means that redoing\n -+\t\t * that one function will cost us virtually 0 extra, so it\n -+\t\t * depends on the other two functions, which are both O(N)\n -+\t\t * cost in the number of paths. Thus, it makes sense that\n -+\t\t * if we can cut the number of paths in half, then redoing\n -+\t\t * collect_merge_info() at half cost in order to get\n -+\t\t * process_entries() at half cost should be about equal\n -+\t\t * cost. If we can cut by more than half, then we would\n -+\t\t * win. The fact that process_entries() is about 10%-20%\n -+\t\t * more expensive than collect_merge_info() suggests we\n -+\t\t * could make the factor be less than two. The fact that\n -+\t\t * even when we have renames cached, we still have to\n -+\t\t * traverse down to the individual (relevant) renames,\n -+\t\t * which suggests we should perhaps use a bigger factor.\n -+\t\t *\n -+\t\t * The exact number isn't critical, since the code will\n -+\t\t * work even if we get the factor wrong -- it just might be\n -+\t\t * slightly slower if we're a bit off. For now, just error\n -+\t\t * on the side of a bigger fudge. For the linux kernel\n -+\t\t * testcases I was looking at with massive renames, the\n -+\t\t * ratio came in around 50 to 250, which clearly would\n -+\t\t * trigger this optimization and provided some *very* nice\n -+\t\t * speedups.\n ++\t\t * The choice of wanted_factor here does not affect\n ++\t\t * correctness, only performance. When the\n ++\t\t * path_count_after / path_count_before\n ++\t\t * ratio is high, redoing after renames is a big\n ++\t\t * performance boost. I suspect that redoing is a wash\n ++\t\t * somewhere near a value of 2, and below that redoing will\n ++\t\t * slow things down. I applied a fudge factor and picked\n ++\t\t * 3; see the commit message when this was introduced for\n ++\t\t * back of the envelope calculations for this ratio.\n +\t\t */\n -+\t\tint wanted_factor = 3;\n ++\t\tconst int wanted_factor = 3;\n +\n +\t\t/* We should only redo collect_merge_info one time */\n +\t\tassert(renames->redo_after_renames == 0);\n +\n -+\t\tif (path_count_after / path_count_before > wanted_factor) {\n ++\t\tif (path_count_after / path_count_before >= wanted_factor) {\n +\t\t\trenames->redo_after_renames = 1;\n +\t\t\trenames->cached_pairs_valid_side = -1;\n +\t\t}\n\n-- \ngitgitgadget\n"},{"id":"430295","messageId":"5dca982c0b061ae8d6335d4f22b78811dced054a.1626412958.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v3 1/7] merge-ort: resolve paths early when we have sufficient information","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:31Z","receivedAt":"2021-07-16T05:22:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen there are no directories involved at a given path, and all three\nsides have a file at that path, and two of the three sides of history\nmatch, we can immediately resolve the merge of that path in\ncollect_merge_info() and do not need to wait until process_entries().\n\nThis is actually a very minor improvement: half the time when I run it,\nI see an improvement; the other half a slowdown. It seems to be in the\nrange of noise. However, this idea serves as the beginning of some\nbigger optimizations coming in the following patches.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e3a5dfc7b31..6299b4f9413 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1023,6 +1023,43 @@ static int collect_merge_info_callback(int n,\n \t\treturn mask;\n \t}\n \n+\t/*\n+\t * If the sides match, and all three paths are present and are\n+\t * files, then we can take either as the resolution. We can't do\n+\t * this with trees, because there may be rename sources from the\n+\t * merge_base.\n+\t */\n+\tif (sides_match && filemask == 0x07) {\n+\t\t/* use side1 (== side2) version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/*\n+\t * If side1 matches mbase and all three paths are present and are\n+\t * files, then we can use side2 as the resolution. We cannot\n+\t * necessarily do so this for trees, because there may be rename\n+\t * destinations within side2.\n+\t */\n+\tif (side1_matches_mbase && filemask == 0x07) {\n+\t\t/* use side2 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+2, side2_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/* Similar to above but swapping sides 1 and 2 */\n+\tif (side2_matches_mbase && filemask == 0x07) {\n+\t\t/* use side1 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n \t/*\n \t * Gather additional information used in rename detection.\n \t */\n-- \ngitgitgadget\n\n"},{"id":"430296","messageId":"8aea3713902b7d006f527ccd76faf6f944529bdb.1626412958.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v3 2/7] merge-ort: add some more explanations in collect_merge_info_callback()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:32Z","receivedAt":"2021-07-16T05:22:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe previous patch possibly raises some questions about whether\nadditional cases in collect_merge_info_callback() can be handled early.\nAdd some explanations in the form of comments to help explain these\nbetter. While we're at it, add a few comments to denote what a few\nboolean '0' or '1' values stand for.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 20 +++++++++++++++-----\n 1 file changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 6299b4f9413..843fa693145 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1018,8 +1018,8 @@ static int collect_merge_info_callback(int n,\n \tif (side1_matches_mbase && side2_matches_mbase) {\n \t\t/* mbase, side1, & side2 all match; use mbase as resolution */\n \t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n-\t\t\t\tnames, names+0, mbase_null, 0,\n-\t\t\t\tfilemask, dirmask, 1);\n+\t\t\t\tnames, names+0, mbase_null, 0 /* df_conflict */,\n+\t\t\t\tfilemask, dirmask, 1 /* resolved */);\n \t\treturn mask;\n \t}\n \n@@ -1061,14 +1061,24 @@ static int collect_merge_info_callback(int n,\n \t}\n \n \t/*\n-\t * Gather additional information used in rename detection.\n+\t * Sometimes we can tell that a source path need not be included in\n+\t * rename detection -- namely, whenever either\n+\t * side1_matches_mbase && side2_null\n+\t * or\n+\t * side2_matches_mbase && side1_null\n+\t * However, we call collect_rename_info() even in those cases,\n+\t * because exact renames are cheap and would let us remove both a\n+\t * source and destination path. We'll cull the unneeded sources\n+\t * later.\n \t */\n \tcollect_rename_info(opt, names, dirname, fullpath,\n \t\t\t filemask, dirmask, match_mask);\n \n \t/*\n-\t * Record information about the path so we can resolve later in\n-\t * process_entries.\n+\t * None of the special cases above matched, so we have a\n+\t * provisional conflict. (Rename detection might allow us to\n+\t * unconflict some more cases, but that comes later so all we can\n+\t * do now is record the different non-null file hashes.)\n \t */\n \tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n \t\t\tnames, NULL, 0, df_conflict, filemask, dirmask, 0);\n-- \ngitgitgadget\n\n"},{"id":"430297","messageId":"1cf4a47562a891ff5fd009c6c5ba836bf798da87.1626412958.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v3 4/7] merge-ort: add a handle_deferred_entries() helper function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:34Z","receivedAt":"2021-07-16T05:22:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn order to allow trivial directory resolution, we first need to be able\nto gather more information to determine if the optimization is safe. To\nenable that, we need a way of deferring the recursion into the directory\nuntil a later time. Naturally, deferring the entry into a subtree means\nthat we need some function that will later recurse into the subdirectory\nexactly the same way that collect_merge_info_callback() would have done.\n\nAdd a helper function that does this. For now this function is not used\nbut a subsequent commit will change that. Future commits will also make\nthe function sometimes resolve directories instead of traversing inside.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 64 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex d9263ec5aca..f0a07684df6 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1202,6 +1202,70 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+MAYBE_UNUSED\n+static int handle_deferred_entries(struct merge_options *opt,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tstruct rename_info *renames = &opt->priv->renames;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\tint side, ret = 0;\n+\n+\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n+\t\trenames->deferred[side].trivial_merges_okay = 0;\n+\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n+\t\t\tstruct conflict_info *ci;\n+\t\t\tunsigned dirmask;\n+\t\t\tstruct tree_desc t[3];\n+\t\t\tvoid *buf[3] = {NULL,};\n+\t\t\tint i;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\t\t\tdirmask = ci->dirmask;\n+\n+\t\t\tinfo->name = path;\n+\t\t\tinfo->namelen = strlen(path);\n+\t\t\tinfo->pathlen = info->namelen + 1;\n+\n+\t\t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n+\t\t\t\tif (i == 1 && ci->match_mask == 3)\n+\t\t\t\t\tt[1] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 5)\n+\t\t\t\t\tt[2] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 6)\n+\t\t\t\t\tt[2] = t[1];\n+\t\t\t\telse {\n+\t\t\t\t\tconst struct object_id *oid = NULL;\n+\t\t\t\t\tif (dirmask & 1)\n+\t\t\t\t\t\toid = &ci->stages[i].oid;\n+\t\t\t\t\tbuf[i] = fill_tree_descriptor(opt->repo,\n+\t\t\t\t\t\t\t\t t+i, oid);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tci->match_mask &= ci->filemask;\n+\t\t\topt->priv->current_dir_name = path;\n+\t\t\trenames->dir_rename_mask = dir_rename_mask;\n+\t\t\tif (renames->dir_rename_mask == 0 ||\n+\t\t\t renames->dir_rename_mask == 0x07)\n+\t\t\t\tret = traverse_trees(NULL, 3, t, info);\n+\t\t\telse\n+\t\t\t\tret = traverse_trees_wrapper(NULL, 3, t, info);\n+\n+\t\t\tfor (i = MERGE_BASE; i <= MERGE_SIDE2; i++)\n+\t\t\t\tfree(buf[i]);\n+\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n static int collect_merge_info(struct merge_options *opt,\n \t\t\t struct tree *merge_base,\n \t\t\t struct tree *side1,\n-- \ngitgitgadget\n\n"},{"id":"430298","messageId":"c2b45fef1d75046f8cf3c7ab04e044df33795770.1626412958.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v3 3/7] merge-ort: add data structures for allowable trivial directory resolves","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:33Z","receivedAt":"2021-07-16T05:22:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs noted a few commits ago, we can resolve individual files early if all\nthree sides of the merge have a file at the path and two of the three\nsides match. We would really like to do the same thing with\ndirectories, because being able to do a trivial directory resolve means\nwe don't have to recurse into the directory, potentially saving us a\nhuge amount of time in both collect_merge_info() and process_entries().\nUnfortunately, resolving directories early would mean missing any\nrenames whose source or destination is underneath that directory.\n\nIf we somehow knew there weren't any renames under the directory in\nquestion, then we could resolve it early. Sadly, it is impossible to\ndetermine whether there are renames under the directory in question\nwithout recursing into it, and this has traditionally kept us from ever\nimplementing such an optimization.\n\nIn commit f89b4f2bee (\"merge-ort: skip rename detection entirely if\npossible\", 2021-03-11), we added an additional reason that rename\ndetection could be skipped entirely -- namely, if no *relevant* sources\nwere present. Without completing collect_merge_info_callback(), we do\nnot yet know if there are no relevant sources. However, we do know that\nif the current directory on one side matches the merge base, then every\nsource file within that directory will not be RELEVANT_CONTENT, and a\nfew simple checks can often let us rule out RELEVANT_LOCATION as well.\nThis suggests we can just defer recursing into such directories until\nthe end of collect_merge_info.\n\nSince the deferred directories are known to not add any relevant sources\ndue to the above properties, then if there are no relevant sources after\nwe've traversed all paths other than the deferred ones, then we know\nthere are not any relevant sources. Under those conditions, rename\ndetection is unnecessary, and that means we can resolve the deferred\ndirectories without recursing into them.\n\nNote that the logic for skipping rename detection was also modified\nfurther in commit 76e253793c (\"merge-ort, diffcore-rename: employ cached\nrenames when possible\", 2021-01-30); in particular rename detection can\nbe skipped if we already have cached renames for each relevant source.\nWe can take advantage of this information as well with our deferral of\nrecursing into directories where one side matches the merge base.\n\nAdd some data structures that we will use to do these deferrals, with\nsome lengthy comments explaining their purpose.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 843fa693145..d9263ec5aca 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -62,6 +62,53 @@ struct traversal_callback_data {\n \tstruct name_entry names[3];\n };\n \n+struct deferred_traversal_data {\n+\t/*\n+\t * possible_trivial_merges: directories to be explored only when needed\n+\t *\n+\t * possible_trivial_merges is a map of directory names to\n+\t * dir_rename_mask. When we detect that a directory is unchanged on\n+\t * one side, we can sometimes resolve the directory without recursing\n+\t * into it. Renames are the only things that can prevent such an\n+\t * optimization. However, for rename sources:\n+\t * - If no parent directory needed directory rename detection, then\n+\t * no path under such a directory can be a relevant_source.\n+\t * and for rename destinations:\n+\t * - If no cached rename has a target path under the directory AND\n+\t * - If there are no unpaired relevant_sources elsewhere in the\n+\t * repository\n+\t * then we don't need any path under this directory for a rename\n+\t * destination. The only way to know the last item above is to defer\n+\t * handling such directories until the end of collect_merge_info(),\n+\t * in handle_deferred_entries().\n+\t *\n+\t * For each we store dir_rename_mask, since that's the only bit of\n+\t * information we need, other than the path, to resume the recursive\n+\t * traversal.\n+\t */\n+\tstruct strintmap possible_trivial_merges;\n+\n+\t/*\n+\t * trivial_merges_okay: if trivial directory merges are okay\n+\t *\n+\t * See possible_trivial_merges above. The \"no unpaired\n+\t * relevant_sources elsewhere in the repository\" is a single boolean\n+\t * per merge side, which we store here. Note that while 0 means no,\n+\t * 1 only means \"maybe\" rather than \"yes\"; we optimistically set it\n+\t * to 1 initially and only clear when we determine it is unsafe to\n+\t * do trivial directory merges.\n+\t */\n+\tunsigned trivial_merges_okay;\n+\n+\t/*\n+\t * target_dirs: ancestor directories of rename targets\n+\t *\n+\t * target_dirs contains all directory names that are an ancestor of\n+\t * any rename destination.\n+\t */\n+\tstruct strset target_dirs;\n+};\n+\n struct rename_info {\n \t/*\n \t * All variables that are arrays of size 3 correspond to data tracked\n@@ -119,6 +166,8 @@ struct rename_info {\n \t */\n \tstruct strintmap relevant_sources[3];\n \n+\tstruct deferred_traversal_data deferred[3];\n+\n \t/*\n \t * dir_rename_mask:\n \t * 0: optimization removing unmodified potential rename source okay\n@@ -501,6 +550,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \t\t}\n \t}\n+\tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n+\t\tstrintmap_func(&renames->deferred[i].possible_trivial_merges);\n+\t\tstrset_func(&renames->deferred[i].target_dirs);\n+\t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n+\t}\n \trenames->cached_pairs_valid_side = 0;\n \trenames->dir_rename_mask = 0;\n \n@@ -4052,6 +4106,13 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\tstrset_init_with_options(&renames->cached_target_names[i],\n \t\t\t\t\t NULL, 0);\n \t}\n+\tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n+\t\tstrintmap_init_with_options(&renames->deferred[i].possible_trivial_merges,\n+\t\t\t\t\t 0, NULL, 0);\n+\t\tstrset_init_with_options(&renames->deferred[i].target_dirs,\n+\t\t\t\t\t NULL, 1);\n+\t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n+\t}\n \n \t/*\n \t * Although we initialize opt->priv->paths with strdup_strings=0,\n-- \ngitgitgadget\n\n"},{"id":"430299","messageId":"79c5153682980f8657b3fdc7bb85cbe5ea816398.1626412958.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v3 5/7] merge-ort: defer recursing into directories when merge base is matched","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:35Z","receivedAt":"2021-07-16T05:22:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen one side of history matches the merge base (including when the\nmerge base has no entry for the given directory), have\ncollect_merge_info_callback() defer recursing into the directory. To\nensure those entries are eventually handled, add a call to\nhandled_deferred_entries() in collect_merge_info() after\ntraverse_trees() returns.\n\nNote that the condition in collect_merge_info_callback() may look more\ncomplicated than necessary at first glance;\nrenames->trivial_merges_okay[side] is always true until\nhandle_deferred_entries() is called, and possible_trivial_merges[side]\nis always empty right now (and in the future won't be filled until\nhandle_deferred_entries() is called). However, when\nhandle_deferred_entries() calls traverse_trees() for the relevant\ndeferred directories, those traverse_trees() calls will once again end\nup in collect_merge_info_callback() for all the entries under those\nsubdirectories. The extra conditions are there for such deferred cases\nand will be used more as we do more with those variables.\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 33 +++++++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex f0a07684df6..dbccf8c62e2 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1147,8 +1147,36 @@ static int collect_merge_info_callback(int n,\n \t\tstruct tree_desc t[3];\n \t\tvoid *buf[3] = {NULL, NULL, NULL};\n \t\tconst char *original_dir_name;\n-\t\tint i, ret;\n+\t\tint i, ret, side;\n \n+\t\t/*\n+\t\t * Check for whether we can avoid recursing due to one side\n+\t\t * matching the merge base. The side that does NOT match is\n+\t\t * the one that might have a rename destination we need.\n+\t\t */\n+\t\tassert(!side1_matches_mbase || !side2_matches_mbase);\n+\t\tside = side1_matches_mbase ? MERGE_SIDE2 :\n+\t\t\tside2_matches_mbase ? MERGE_SIDE1 : MERGE_BASE;\n+\t\tif (filemask == 0 && (dirmask == 2 || dirmask == 4)) {\n+\t\t\t/*\n+\t\t\t * Also defer recursing into new directories; set up a\n+\t\t\t * few variables to let us do so.\n+\t\t\t */\n+\t\t\tci->match_mask = (7 - dirmask);\n+\t\t\tside = dirmask / 2;\n+\t\t}\n+\t\tif (renames->dir_rename_mask != 0x07 &&\n+\t\t side != MERGE_BASE &&\n+\t\t renames->deferred[side].trivial_merges_okay &&\n+\t\t !strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t pi.string)) {\n+\t\t\tstrintmap_set(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t pi.string, renames->dir_rename_mask);\n+\t\t\trenames->dir_rename_mask = prev_dir_rename_mask;\n+\t\t\treturn mask;\n+\t\t}\n+\n+\t\t/* We need to recurse */\n \t\tci->match_mask &= filemask;\n \t\tnewinfo = *info;\n \t\tnewinfo.prev = info;\n@@ -1202,7 +1230,6 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n-MAYBE_UNUSED\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1291,6 +1318,8 @@ static int collect_merge_info(struct merge_options *opt,\n \n \ttrace2_region_enter(\"merge\", \"traverse_trees\", opt->repo);\n \tret = traverse_trees(NULL, 3, t, &info);\n+\tif (ret == 0)\n+\t\tret = handle_deferred_entries(opt, &info);\n \ttrace2_region_leave(\"merge\", \"traverse_trees\", opt->repo);\n \n \treturn ret;\n-- \ngitgitgadget\n\n"},{"id":"430300","messageId":"572cc5e94d2fc3b4ae848ec6e48f0ef577a39fa6.1626412958.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v3 6/7] merge-ort: avoid recursing into directories when we don't need to","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:36Z","receivedAt":"2021-07-16T05:22:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis combines the work of the last several patches, and implements the\nconditions when we don't need to recurse into directories. It's perhaps\neasiest to see the logic by separating the fact that a directory might\nhave both rename sources and rename destinations:\n\n * rename sources: only files present in the merge base can serve as\n rename sources, and only when one side deletes that file. When the\n tree on one side matches the merge base, that means every file\n within the subtree matches the merge base. This means that the\n skip-irrelevant-rename-detection optimization from before kicks in\n and we don't need any of these files as rename sources.\n\n * rename destinations: the tree that does not match the merge base\n might have newly added and hence unmatched destination files.\n This is what usually prevents us from doing trivial directory\n resolutions in the merge machinery. However, the fact that we have\n deferred recursing into this directory until the end means we know\n whether there are any unmatched relevant potential rename sources\n elsewhere in this merge. If there are no unmatched such relevant\n sources anywhere, then there is no need to look for unmatched\n potential rename destinations to match them with.\n\nThis informs our algorithm:\n * Search through relevant_sources; if we have entries, they better all\n be reflected in cached_pairs or cached_irrelevant, otherwise they\n represent an unmatched potential rename source (causing the\n optimization to be disallowed).\n * For any relevant_source represented in cached_pairs, we do need to\n to make sure to get the destination for each source, meaning we need\n to recurse into any ancestor directories of those destinations.\n * Once we've recursed into all the rename destinations for any\n relevant_sources in cached_pairs, we can then do the trivial\n directory resolution for the remaining directories.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 5.235 s ± 0.042 s 205.1 ms ± 3.8 ms\n mega-renames: 9.419 s ± 0.107 s 1.564 s ± 0.010 s\n just-one-mega: 480.1 ms ± 3.9 ms 479.5 ms ± 3.9 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 99 insertions(+), 3 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex dbccf8c62e2..a013708fa79 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1230,6 +1230,18 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+static void resolve_trivial_directory_merge(struct conflict_info *ci, int side)\n+{\n+\tVERIFY_CI(ci);\n+\tassert((side == 1 && ci->match_mask == 5) ||\n+\t (side == 2 && ci->match_mask == 3));\n+\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n+\tci->merged.result.mode = ci->stages[side].mode;\n+\tci->merged.is_null = is_null_oid(&ci->stages[side].oid);\n+\tci->match_mask = 0;\n+\tci->merged.clean = 1; /* (ci->filemask == 0); */\n+}\n+\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1239,9 +1251,72 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tint side, ret = 0;\n \n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n-\t\trenames->deferred[side].trivial_merges_okay = 0;\n-\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n-\t\t\t\t\t &iter, entry) {\n+\t\tunsigned optimization_okay = 1;\n+\t\tstruct strintmap copy;\n+\n+\t\t/* Loop over the set of paths we need to know rename info for */\n+\t\tstrset_for_each_entry(&renames->relevant_sources[side],\n+\t\t\t\t &iter, entry) {\n+\t\t\tchar *rename_target, *dir, *dir_marker;\n+\t\t\tstruct strmap_entry *e;\n+\n+\t\t\t/*\n+\t\t\t * If we don't know delete/rename info for this path,\n+\t\t\t * then we need to recurse into all trees to get all\n+\t\t\t * adds to make sure we have it.\n+\t\t\t */\n+\t\t\tif (strset_contains(&renames->cached_irrelevant[side],\n+\t\t\t\t\t entry->key))\n+\t\t\t\tcontinue;\n+\t\t\te = strmap_get_entry(&renames->cached_pairs[side],\n+\t\t\t\t\t entry->key);\n+\t\t\tif (!e) {\n+\t\t\t\toptimization_okay = 0;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/* If this is a delete, we have enough info already */\n+\t\t\trename_target = e->value;\n+\t\t\tif (!rename_target)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* If we already walked the rename target, we're good */\n+\t\t\tif (strmap_contains(&opt->priv->paths, rename_target))\n+\t\t\t\tcontinue;\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, we need to get a list of directories that\n+\t\t\t * will need to be recursed into to get this\n+\t\t\t * rename_target.\n+\t\t\t */\n+\t\t\tdir = xstrdup(rename_target);\n+\t\t\twhile ((dir_marker = strrchr(dir, '/'))) {\n+\t\t\t\t*dir_marker = '\\0';\n+\t\t\t\tif (strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t\t dir))\n+\t\t\t\t\tbreak;\n+\t\t\t\tstrset_add(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t dir);\n+\t\t\t}\n+\t\t\tfree(dir);\n+\t\t}\n+\t\trenames->deferred[side].trivial_merges_okay = optimization_okay;\n+\t\t/*\n+\t\t * We need to recurse into any directories in\n+\t\t * possible_trivial_merges[side] found in target_dirs[side].\n+\t\t * But when we recurse, we may need to queue up some of the\n+\t\t * subdirectories for possible_trivial_merges[side]. Since\n+\t\t * we can't safely iterate through a hashmap while also adding\n+\t\t * entries, move the entries into 'copy', iterate over 'copy',\n+\t\t * and then we'll also iterate anything added into\n+\t\t * possible_trivial_merges[side] once this loop is done.\n+\t\t */\n+\t\tcopy = renames->deferred[side].possible_trivial_merges;\n+\t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t\t 0,\n+\t\t\t\t\t NULL,\n+\t\t\t\t\t 0);\n+\t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n \t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n \t\t\tstruct conflict_info *ci;\n@@ -1254,6 +1329,13 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tVERIFY_CI(ci);\n \t\t\tdirmask = ci->dirmask;\n \n+\t\t\tif (optimization_okay &&\n+\t\t\t !strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t path)) {\n+\t\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tinfo->name = path;\n \t\t\tinfo->namelen = strlen(path);\n \t\t\tinfo->pathlen = info->namelen + 1;\n@@ -1289,6 +1371,20 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tif (ret < 0)\n \t\t\t\treturn ret;\n \t\t}\n+\t\tstrintmap_clear(©);\n+\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tstruct conflict_info *ci;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\n+\t\t\tassert(renames->deferred[side].trivial_merges_okay &&\n+\t\t\t !strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t\tpath));\n+\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t}\n \t}\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"430301","messageId":"a9cbc1d4f181e9ce943bab47e51d6fdaea14b9ac.1626412958.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v3 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-16T05:22:37Z","receivedAt":"2021-07-16T05:22:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe merge algorithm mostly consists of the following three functions:\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\nPrior to the trivial directory resolution optimization of the last half\ndozen commits, process_entries() was consistently the slowest, followed\nby collect_merge_info(), then detect_and_process_renames(). When the\ntrivial directory resolution applies, it often dramatically decreases\nthe amount of time spent in the two slower functions.\n\nLooking at the performance results in the previous commit, the trivial\ndirectory resolution optimization helps amazingly well when there are no\nrelevant renames. It also helps really well when reapplying a long\nseries of linear commits (such as in a rebase or cherry-pick), since the\nrelevant renames may well be cached from the first reapplied commit.\nBut when there are any relevant renames that are not cached (represented\nby the just-one-mega testcase), then the optimization does not help at\nall.\n\nOften, I noticed that when the optimization does not apply, it is\nbecause there are a handful of relevant sources -- maybe even only one.\nIt felt frustrating to need to recurse into potentially hundreds or even\nthousands of directories just for a single rename, but it was needed for\ncorrectness.\n\nHowever, staring at this list of functions and noticing that\nprocess_entries() is the most expensive and knowing I could avoid it if\nI had cached renames suggested a simple idea: change\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\ninto\n collect_merge_info()\n detect_and_process_renames()\n \n collect_merge_info()\n detect_and_process_renames()\n process_entries()\n\nThis may seem odd and look like more work. However, note that although\nwe run collect_merge_info() twice, the second time we get to employ\ntrivial directory resolves, which makes it much faster, so the increased\ntime in collect_merge_info() is small. While we run\ndetect_and_process_renames() again, all renames are cached so it's\nnearly a no-op (we don't call into diffcore_rename_extended() but we do\nhave a little bit of data structure checking and fixing up). And the\nbig payoff comes from the fact that process_entries(), will be much\nfaster due to having far fewer entries to process.\n\nThis restarting only makes sense if we can save recursing into enough\ndirectories to make it worth our while. Introduce a simple heuristic to\nguide this. Note that this heuristic uses a \"wanted_factor\" that I have\nvirtually no actual real world data for, just some back-of-the-envelope\nquasi-scientific calculations that I included in some comments and then\nplucked a simple round number out of thin air. It could be that\ntweaking this number to make it either higher or lower improves the\noptimization. (There's slightly more here; when I first introduced this\noptimization, I used a factor of 10, because I was completely confident\nit was big enough to not cause slowdowns in special cases. I was\ncertain it was higher than needed. Several months later, I added the\nrough calculations which make me think the optimal number is close to 2;\nbut instead of pushing to the limit, I just bumped it to 3 to reduce the\nrisk that there are special cases where this optimization can result in\nslowing down the code a little. If the ratio of path counts is below 3,\nwe probably will only see minor performance improvements at best\nanyway.)\n\nAlso, note that while the diffstat looks kind of long (nearly 100\nlines), more than half of it is in two comments explaining how things\nwork.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 205.1 ms ± 3.8 ms 204.2 ms ± 3.0 ms\n mega-renames: 1.564 s ± 0.010 s 1.076 s ± 0.015 s\n just-one-mega: 479.5 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 92 +++++++++++++++++++++++++++--\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 87 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex a013708fa79..e361443087a 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -213,6 +213,7 @@ struct rename_info {\n \t * MERGE_SIDE2: cached data from side2 can be reused\n \t * MERGE_SIDE1: cached data from side1 can be reused\n \t * 0: no cached data can be reused\n+\t * -1: See redo_after_renames; both sides can be reused.\n \t */\n \tint cached_pairs_valid_side;\n \n@@ -258,6 +259,28 @@ struct rename_info {\n \t */\n \tstruct strset cached_irrelevant[3];\n \n+\t/*\n+\t * redo_after_renames: optimization flag for \"restarting\" the merge\n+\t *\n+\t * Sometimes it pays to detect renames, cache them, and then\n+\t * restart the merge operation from the beginning. The reason for\n+\t * this is that when we know where all the renames are, we know\n+\t * whether a certain directory has any paths under it affected --\n+\t * and if a directory is not affected then it permits us to do\n+\t * trivial tree merging in more cases. Doing trivial tree merging\n+\t * prevents the need to run process_entry() on every path\n+\t * underneath trees that can be trivially merged, and\n+\t * process_entry() is more expensive than collect_merge_info() --\n+\t * plus, the second collect_merge_info() will be much faster since\n+\t * it doesn't have to recurse into the relevant trees.\n+\t *\n+\t * Values for this flag:\n+\t * 0 = don't bother, not worth it (or conditions not yet checked)\n+\t * 1 = conditions for optimization met, optimization worthwhile\n+\t * 2 = we already did it (don't restart merge yet again)\n+\t */\n+\tunsigned redo_after_renames;\n+\n \t/*\n \t * needed_limit: value needed for inexact rename detection to run\n \t *\n@@ -541,7 +564,8 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrintmap_func(&renames->relevant_sources[i]);\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n-\t\tif (i != renames->cached_pairs_valid_side) {\n+\t\tif (i != renames->cached_pairs_valid_side &&\n+\t\t -1 != renames->cached_pairs_valid_side) {\n \t\t\tstrset_func(&renames->cached_target_names[i]);\n \t\t\tstrmap_func(&renames->cached_pairs[i], 1);\n \t\t\tstrset_func(&renames->cached_irrelevant[i]);\n@@ -1249,7 +1273,9 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n \tint side, ret = 0;\n+\tint path_count_before, path_count_after = 0;\n \n+\tpath_count_before = strmap_get_size(&opt->priv->paths);\n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n \t\tunsigned optimization_okay = 1;\n \t\tstruct strintmap copy;\n@@ -1385,7 +1411,32 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t\t\tpath));\n \t\t\tresolve_trivial_directory_merge(ci, side);\n \t\t}\n+\t\tif (!optimization_okay || path_count_after)\n+\t\t\tpath_count_after = strmap_get_size(&opt->priv->paths);\n \t}\n+\tif (path_count_after) {\n+\t\t/*\n+\t\t * The choice of wanted_factor here does not affect\n+\t\t * correctness, only performance. When the\n+\t\t * path_count_after / path_count_before\n+\t\t * ratio is high, redoing after renames is a big\n+\t\t * performance boost. I suspect that redoing is a wash\n+\t\t * somewhere near a value of 2, and below that redoing will\n+\t\t * slow things down. I applied a fudge factor and picked\n+\t\t * 3; see the commit message when this was introduced for\n+\t\t * back of the envelope calculations for this ratio.\n+\t\t */\n+\t\tconst int wanted_factor = 3;\n+\n+\t\t/* We should only redo collect_merge_info one time */\n+\t\tassert(renames->redo_after_renames == 0);\n+\n+\t\tif (path_count_after / path_count_before >= wanted_factor) {\n+\t\t\trenames->redo_after_renames = 1;\n+\t\t\trenames->cached_pairs_valid_side = -1;\n+\t\t}\n+\t} else if (renames->redo_after_renames == 2)\n+\t\trenames->redo_after_renames = 0;\n \treturn ret;\n }\n \n@@ -2828,8 +2879,8 @@ static int compare_pairs(const void *a_, const void *b_)\n }\n \n /* Call diffcore_rename() to update deleted/added pairs into rename pairs */\n-static void detect_regular_renames(struct merge_options *opt,\n-\t\t\t\t unsigned side_index)\n+static int detect_regular_renames(struct merge_options *opt,\n+\t\t\t\t unsigned side_index)\n {\n \tstruct diff_options diff_opts;\n \tstruct rename_info *renames = &opt->priv->renames;\n@@ -2842,7 +2893,7 @@ static void detect_regular_renames(struct merge_options *opt,\n \t\t * side had directory renames.\n \t\t */\n \t\tresolve_diffpair_statuses(&renames->pairs[side_index]);\n-\t\treturn;\n+\t\treturn 0;\n \t}\n \n \tpartial_clear_dir_rename_count(&renames->dir_rename_count[side_index]);\n@@ -2868,6 +2919,8 @@ static void detect_regular_renames(struct merge_options *opt,\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n \tresolve_diffpair_statuses(&diff_queued_diff);\n \n+\tif (diff_opts.needed_rename_limit > 0)\n+\t\trenames->redo_after_renames = 0;\n \tif (diff_opts.needed_rename_limit > renames->needed_limit)\n \t\trenames->needed_limit = diff_opts.needed_rename_limit;\n \n@@ -2877,6 +2930,8 @@ static void detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff.nr = 0;\n \tdiff_queued_diff.queue = NULL;\n \tdiff_flush(&diff_opts);\n+\n+\treturn 1;\n }\n \n /*\n@@ -2966,14 +3021,32 @@ static int detect_and_process_renames(struct merge_options *opt,\n \tstruct diff_queue_struct combined;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint need_dir_renames, s, clean = 1;\n+\tunsigned detection_run = 0;\n \n \tmemset(&combined, 0, sizeof(combined));\n \tif (!possible_renames(renames))\n \t\tgoto cleanup;\n \n \ttrace2_region_enter(\"merge\", \"regular renames\", opt->repo);\n-\tdetect_regular_renames(opt, MERGE_SIDE1);\n-\tdetect_regular_renames(opt, MERGE_SIDE2);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE1);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE2);\n+\tif (renames->redo_after_renames && detection_run) {\n+\t\tint i, side;\n+\t\tstruct diff_filepair *p;\n+\n+\t\t/* Cache the renames, we found */\n+\t\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n+\t\t\tfor (i = 0; i < renames->pairs[side].nr; ++i) {\n+\t\t\t\tp = renames->pairs[side].queue[i];\n+\t\t\t\tpossibly_cache_new_pair(renames, p, side, NULL);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Restart the merge with the cached renames */\n+\t\trenames->redo_after_renames = 2;\n+\t\ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n+\t\tgoto cleanup;\n+\t}\n \tuse_cached_pairs(opt, &renames->cached_pairs[1], &renames->pairs[1]);\n \tuse_cached_pairs(opt, &renames->cached_pairs[2], &renames->pairs[2]);\n \ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n@@ -4390,6 +4463,7 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \t\t\t\t\t opt->subtree_shift);\n \t}\n \n+redo:\n \ttrace2_region_enter(\"merge\", \"collect_merge_info\", opt->repo);\n \tif (collect_merge_info(opt, merge_base, side1, side2) != 0) {\n \t\t/*\n@@ -4409,6 +4483,12 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \tresult->clean = detect_and_process_renames(opt, merge_base,\n \t\t\t\t\t\t side1, side2);\n \ttrace2_region_leave(\"merge\", \"renames\", opt->repo);\n+\tif (opt->priv->renames.redo_after_renames == 2) {\n+\t\ttrace2_region_enter(\"merge\", \"reset_maps\", opt->repo);\n+\t\tclear_or_reinit_internal_opts(opt->priv, 1);\n+\t\ttrace2_region_leave(\"merge\", \"reset_maps\", opt->repo);\n+\t\tgoto redo;\n+\t}\n \n \ttrace2_region_enter(\"merge\", \"process_entries\", opt->repo);\n \tprocess_entries(opt, &working_tree_oid);\ndiff --git a/t/t6423-merge-rename-directories.sh b/t/t6423-merge-rename-directories.sh\nindex e834b7e6efe..d8919d276a1 100755\n--- a/t/t6423-merge-rename-directories.sh\n+++ b/t/t6423-merge-rename-directories.sh\n@@ -4797,7 +4797,7 @@ test_setup_12f () {\n \t)\n }\n \n-test_expect_merge_algorithm failure failure '12f: Trivial directory resolve, caching, all kinds of fun' '\n+test_expect_merge_algorithm failure success '12f: Trivial directory resolve, caching, all kinds of fun' '\n \ttest_setup_12f &&\n \t(\n \t\tcd 12f &&\n-- \ngitgitgadget\n"},{"id":"430621","messageId":"1eeec718-a315-c038-d54a-4503a5ca88d1@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/7] Optimization batch 14: trivial directory resolution","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-20T13:00:48Z","receivedAt":"2021-07-20T13:00:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/16/2021 1:22 AM, Elijah Newren via GitGitGadget wrote:\n> This series depends textually on ort-perf-batch-12, but is semantically\n> independent. (It is both semantically and textually independent of\n> ort-perf-batch-13.)\n...\n> Range-diff vs v2:\n\nThank you for making these edits.\n\n> ++struct deferred_traversal_data {\n\nIn particular, I think this new struct works rather well.\n\nThis version LGTM.\n\nThanks,\n-Stolee\n"},{"id":"430706","messageId":"xmqqk0lkhdal.fsf@gitster.g","threadId":"56035","inReplyTo":"1eeec718-a315-c038-d54a-4503a5ca88d1@gmail.com","subject":"Re: [PATCH v3 0/7] Optimization batch 14: trivial directory resolution","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-20T21:43:14Z","receivedAt":"2021-07-20T21:43:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 7/16/2021 1:22 AM, Elijah Newren via GitGitGadget wrote:\n>> This series depends textually on ort-perf-batch-12, but is semantically\n>> independent. (It is both semantically and textually independent of\n>> ort-perf-batch-13.)\n> ...\n>> Range-diff vs v2:\n>\n> Thank you for making these edits.\n>\n>> ++struct deferred_traversal_data {\n>\n> In particular, I think this new struct works rather well.\n>\n> This version LGTM.\n>\n> Thanks,\n> -Stolee\n\nThanks, both.\n"},{"id":"430733","messageId":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v3.git.1626412958.gitgitgadget@gmail.com","subject":"[PATCH v4 0/7] Optimization batch 14: trivial directory resolution","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:23:57Z","receivedAt":"2021-07-21T04:24:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends textually on ort-perf-batch-12, but is semantically\nindependent. (It is both semantically and textually independent of\nort-perf-batch-13.)\n\nMost of my previous series dramatically accelerated cases with lots of\nrenames, while providing comparatively minor benefits for cases with few or\nno renames. This series is the opposite; it provides huge benefits when\nthere are few or no renames, and comparatively smaller (though still quite\ndecent) benefits for cases with many uncached renames.\n\nChanges since v3:\n\n * Add Stolee's Acked-by.\n\nChanges since v2, addressing feedback from Stolee:\n\n * Created a separate struct for three related variables to hint they are\n related\n * Simplified a lengthy comment that was duplicated by the commit message\n * Various other minor cleanups\n\nChanges since v1:\n\n * Minor tweak to the final patch to correct implicit assumption that rename\n detection running implies all renames were found (rename limits could\n have been exceeded and prevented finding renames)\n\n=== Basic Optimization idea ===\n\nunpack_trees has had a concept of trivial merges for individual files (see\nDocumentation/technical/trivial-merge.txt). The same idea can be applied in\nmerge-ort. It'd be really nice to extend that idea to trees as well, as it\ncould provide a huge performance boost; sadly however, applying it in\ngeneral would wreck both regular rename detection (the unmatched side can\nhave new files that serve as potential destinations in rename detection) and\ndirectory rename detection (the unmatched side could have a new directory\nthat was moved into it).\n\nIf we somehow knew rename detection wasn't needed, we could do trivial\ndirectory resolution. In the past, this wasn't possible. However...\n\nWith recent optimizations we have created a possibility to do trivial\ndirectory resolutions in some cases. These came from the addition of the\n\"skipping irrelevant renames\" optimizations (from ort-perf-batch-9 and\nort-perf-batch-10), and in particular noting that we added an ability to\nentirely skip rename detection in commit f89b4f2bee (\"merge-ort: skip rename\ndetection entirely if possible\", 2021-03-11) when there are no relevant\nsources. We can detect if there are no relevant sources without recursing\ninto the directories in question.\n\nAs a cherry on top, the caching of renames (from ort-perf-batch-11) allows\nus to cover additional cases.\n\nThis series is all about adding all the special checks needed to safely\nperform trival directory resolutions.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 5.235 s ± 0.042 s 204.2 ms ± 3.0 ms\nmega-renames: 9.419 s ± 0.107 s 1.076 s ± 0.015 s\njust-one-mega: 480.1 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with (for merge-recursive as of git-2.30.0)\nwere:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\nElijah Newren (7):\n merge-ort: resolve paths early when we have sufficient information\n merge-ort: add some more explanations in collect_merge_info_callback()\n merge-ort: add data structures for allowable trivial directory\n resolves\n merge-ort: add a handle_deferred_entries() helper function\n merge-ort: defer recursing into directories when merge base is matched\n merge-ort: avoid recursing into directories when we don't need to\n merge-ort: restart merge with cached renames to reduce process entry\n cost\n\n merge-ort.c | 399 +++++++++++++++++++++++++++-\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 389 insertions(+), 12 deletions(-)\n\n\nbase-commit: 2eeee12b02e441ac05054a5a5ecbcea6964a1e6b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-988%2Fnewren%2Fort-perf-batch-14-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-988/newren/ort-perf-batch-14-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/988\n\nRange-diff vs v3:\n\n 1: 5dca982c0b0 ! 1: 7fdfeb159d0 merge-ort: resolve paths early when we have sufficient information\n @@ Commit message\n range of noise. However, this idea serves as the beginning of some\n bigger optimizations coming in the following patches.\n \n + Acked-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n 2: 8aea3713902 ! 2: 7a0085f2da9 merge-ort: add some more explanations in collect_merge_info_callback()\n @@ Commit message\n better. While we're at it, add a few comments to denote what a few\n boolean '0' or '1' values stand for.\n \n + Acked-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n 3: c2b45fef1d7 ! 3: d8165740316 merge-ort: add data structures for allowable trivial directory resolves\n @@ Commit message\n Add some data structures that we will use to do these deferrals, with\n some lengthy comments explaining their purpose.\n \n + Acked-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n 4: 1cf4a47562a ! 4: ff568a612f5 merge-ort: add a handle_deferred_entries() helper function\n @@ Commit message\n but a subsequent commit will change that. Future commits will also make\n the function sometimes resolve directories instead of traversing inside.\n \n + Acked-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n 5: 79c51536829 ! 5: 5b01c118f10 merge-ort: defer recursing into directories when merge base is matched\n @@ Commit message\n subdirectories. The extra conditions are there for such deferred cases\n and will be used more as we do more with those variables.\n \n + Acked-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n 6: 572cc5e94d2 ! 6: 7b211271815 merge-ort: avoid recursing into directories when we don't need to\n @@ Commit message\n mega-renames: 9.419 s ± 0.107 s 1.564 s ± 0.010 s\n just-one-mega: 480.1 ms ± 3.9 ms 479.5 ms ± 3.9 ms\n \n + Acked-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n 7: a9cbc1d4f18 ! 7: c9ada8369e6 merge-ort: restart merge with cached renames to reduce process entry cost\n @@ Commit message\n mega-renames: 1.564 s ± 0.010 s 1.076 s ± 0.015 s\n just-one-mega: 479.5 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n \n + Acked-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## merge-ort.c ##\n\n-- \ngitgitgadget\n"},{"id":"430734","messageId":"7fdfeb159d0125cce980e63bcc1415144b91e486.1626841444.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","subject":"[PATCH v4 1/7] merge-ort: resolve paths early when we have sufficient information","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:23:58Z","receivedAt":"2021-07-21T04:24:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen there are no directories involved at a given path, and all three\nsides have a file at that path, and two of the three sides of history\nmatch, we can immediately resolve the merge of that path in\ncollect_merge_info() and do not need to wait until process_entries().\n\nThis is actually a very minor improvement: half the time when I run it,\nI see an improvement; the other half a slowdown. It seems to be in the\nrange of noise. However, this idea serves as the beginning of some\nbigger optimizations coming in the following patches.\n\nAcked-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex e3a5dfc7b31..6299b4f9413 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1023,6 +1023,43 @@ static int collect_merge_info_callback(int n,\n \t\treturn mask;\n \t}\n \n+\t/*\n+\t * If the sides match, and all three paths are present and are\n+\t * files, then we can take either as the resolution. We can't do\n+\t * this with trees, because there may be rename sources from the\n+\t * merge_base.\n+\t */\n+\tif (sides_match && filemask == 0x07) {\n+\t\t/* use side1 (== side2) version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/*\n+\t * If side1 matches mbase and all three paths are present and are\n+\t * files, then we can use side2 as the resolution. We cannot\n+\t * necessarily do so this for trees, because there may be rename\n+\t * destinations within side2.\n+\t */\n+\tif (side1_matches_mbase && filemask == 0x07) {\n+\t\t/* use side2 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+2, side2_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n+\t/* Similar to above but swapping sides 1 and 2 */\n+\tif (side2_matches_mbase && filemask == 0x07) {\n+\t\t/* use side1 version as resolution */\n+\t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n+\t\t\t\tnames, names+1, side1_null, 0,\n+\t\t\t\tfilemask, dirmask, 1);\n+\t\treturn mask;\n+\t}\n+\n \t/*\n \t * Gather additional information used in rename detection.\n \t */\n-- \ngitgitgadget\n\n"},{"id":"430735","messageId":"7a0085f2da9d9517b20745dcb8be4c15852bd56f.1626841444.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","subject":"[PATCH v4 2/7] merge-ort: add some more explanations in collect_merge_info_callback()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:23:59Z","receivedAt":"2021-07-21T04:24:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe previous patch possibly raises some questions about whether\nadditional cases in collect_merge_info_callback() can be handled early.\nAdd some explanations in the form of comments to help explain these\nbetter. While we're at it, add a few comments to denote what a few\nboolean '0' or '1' values stand for.\n\nAcked-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 20 +++++++++++++++-----\n 1 file changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 6299b4f9413..843fa693145 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1018,8 +1018,8 @@ static int collect_merge_info_callback(int n,\n \tif (side1_matches_mbase && side2_matches_mbase) {\n \t\t/* mbase, side1, & side2 all match; use mbase as resolution */\n \t\tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n-\t\t\t\tnames, names+0, mbase_null, 0,\n-\t\t\t\tfilemask, dirmask, 1);\n+\t\t\t\tnames, names+0, mbase_null, 0 /* df_conflict */,\n+\t\t\t\tfilemask, dirmask, 1 /* resolved */);\n \t\treturn mask;\n \t}\n \n@@ -1061,14 +1061,24 @@ static int collect_merge_info_callback(int n,\n \t}\n \n \t/*\n-\t * Gather additional information used in rename detection.\n+\t * Sometimes we can tell that a source path need not be included in\n+\t * rename detection -- namely, whenever either\n+\t * side1_matches_mbase && side2_null\n+\t * or\n+\t * side2_matches_mbase && side1_null\n+\t * However, we call collect_rename_info() even in those cases,\n+\t * because exact renames are cheap and would let us remove both a\n+\t * source and destination path. We'll cull the unneeded sources\n+\t * later.\n \t */\n \tcollect_rename_info(opt, names, dirname, fullpath,\n \t\t\t filemask, dirmask, match_mask);\n \n \t/*\n-\t * Record information about the path so we can resolve later in\n-\t * process_entries.\n+\t * None of the special cases above matched, so we have a\n+\t * provisional conflict. (Rename detection might allow us to\n+\t * unconflict some more cases, but that comes later so all we can\n+\t * do now is record the different non-null file hashes.)\n \t */\n \tsetup_path_info(opt, &pi, dirname, info->pathlen, fullpath,\n \t\t\tnames, NULL, 0, df_conflict, filemask, dirmask, 0);\n-- \ngitgitgadget\n\n"},{"id":"430736","messageId":"d81657403167ba3cddb4c6f8d55dc0385cc13cd3.1626841444.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","subject":"[PATCH v4 3/7] merge-ort: add data structures for allowable trivial directory resolves","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:24:00Z","receivedAt":"2021-07-21T04:24:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs noted a few commits ago, we can resolve individual files early if all\nthree sides of the merge have a file at the path and two of the three\nsides match. We would really like to do the same thing with\ndirectories, because being able to do a trivial directory resolve means\nwe don't have to recurse into the directory, potentially saving us a\nhuge amount of time in both collect_merge_info() and process_entries().\nUnfortunately, resolving directories early would mean missing any\nrenames whose source or destination is underneath that directory.\n\nIf we somehow knew there weren't any renames under the directory in\nquestion, then we could resolve it early. Sadly, it is impossible to\ndetermine whether there are renames under the directory in question\nwithout recursing into it, and this has traditionally kept us from ever\nimplementing such an optimization.\n\nIn commit f89b4f2bee (\"merge-ort: skip rename detection entirely if\npossible\", 2021-03-11), we added an additional reason that rename\ndetection could be skipped entirely -- namely, if no *relevant* sources\nwere present. Without completing collect_merge_info_callback(), we do\nnot yet know if there are no relevant sources. However, we do know that\nif the current directory on one side matches the merge base, then every\nsource file within that directory will not be RELEVANT_CONTENT, and a\nfew simple checks can often let us rule out RELEVANT_LOCATION as well.\nThis suggests we can just defer recursing into such directories until\nthe end of collect_merge_info.\n\nSince the deferred directories are known to not add any relevant sources\ndue to the above properties, then if there are no relevant sources after\nwe've traversed all paths other than the deferred ones, then we know\nthere are not any relevant sources. Under those conditions, rename\ndetection is unnecessary, and that means we can resolve the deferred\ndirectories without recursing into them.\n\nNote that the logic for skipping rename detection was also modified\nfurther in commit 76e253793c (\"merge-ort, diffcore-rename: employ cached\nrenames when possible\", 2021-01-30); in particular rename detection can\nbe skipped if we already have cached renames for each relevant source.\nWe can take advantage of this information as well with our deferral of\nrecursing into directories where one side matches the merge base.\n\nAdd some data structures that we will use to do these deferrals, with\nsome lengthy comments explaining their purpose.\n\nAcked-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 843fa693145..d9263ec5aca 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -62,6 +62,53 @@ struct traversal_callback_data {\n \tstruct name_entry names[3];\n };\n \n+struct deferred_traversal_data {\n+\t/*\n+\t * possible_trivial_merges: directories to be explored only when needed\n+\t *\n+\t * possible_trivial_merges is a map of directory names to\n+\t * dir_rename_mask. When we detect that a directory is unchanged on\n+\t * one side, we can sometimes resolve the directory without recursing\n+\t * into it. Renames are the only things that can prevent such an\n+\t * optimization. However, for rename sources:\n+\t * - If no parent directory needed directory rename detection, then\n+\t * no path under such a directory can be a relevant_source.\n+\t * and for rename destinations:\n+\t * - If no cached rename has a target path under the directory AND\n+\t * - If there are no unpaired relevant_sources elsewhere in the\n+\t * repository\n+\t * then we don't need any path under this directory for a rename\n+\t * destination. The only way to know the last item above is to defer\n+\t * handling such directories until the end of collect_merge_info(),\n+\t * in handle_deferred_entries().\n+\t *\n+\t * For each we store dir_rename_mask, since that's the only bit of\n+\t * information we need, other than the path, to resume the recursive\n+\t * traversal.\n+\t */\n+\tstruct strintmap possible_trivial_merges;\n+\n+\t/*\n+\t * trivial_merges_okay: if trivial directory merges are okay\n+\t *\n+\t * See possible_trivial_merges above. The \"no unpaired\n+\t * relevant_sources elsewhere in the repository\" is a single boolean\n+\t * per merge side, which we store here. Note that while 0 means no,\n+\t * 1 only means \"maybe\" rather than \"yes\"; we optimistically set it\n+\t * to 1 initially and only clear when we determine it is unsafe to\n+\t * do trivial directory merges.\n+\t */\n+\tunsigned trivial_merges_okay;\n+\n+\t/*\n+\t * target_dirs: ancestor directories of rename targets\n+\t *\n+\t * target_dirs contains all directory names that are an ancestor of\n+\t * any rename destination.\n+\t */\n+\tstruct strset target_dirs;\n+};\n+\n struct rename_info {\n \t/*\n \t * All variables that are arrays of size 3 correspond to data tracked\n@@ -119,6 +166,8 @@ struct rename_info {\n \t */\n \tstruct strintmap relevant_sources[3];\n \n+\tstruct deferred_traversal_data deferred[3];\n+\n \t/*\n \t * dir_rename_mask:\n \t * 0: optimization removing unmodified potential rename source okay\n@@ -501,6 +550,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \t\t}\n \t}\n+\tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n+\t\tstrintmap_func(&renames->deferred[i].possible_trivial_merges);\n+\t\tstrset_func(&renames->deferred[i].target_dirs);\n+\t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n+\t}\n \trenames->cached_pairs_valid_side = 0;\n \trenames->dir_rename_mask = 0;\n \n@@ -4052,6 +4106,13 @@ static void merge_start(struct merge_options *opt, struct merge_result *result)\n \t\tstrset_init_with_options(&renames->cached_target_names[i],\n \t\t\t\t\t NULL, 0);\n \t}\n+\tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) {\n+\t\tstrintmap_init_with_options(&renames->deferred[i].possible_trivial_merges,\n+\t\t\t\t\t 0, NULL, 0);\n+\t\tstrset_init_with_options(&renames->deferred[i].target_dirs,\n+\t\t\t\t\t NULL, 1);\n+\t\trenames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */\n+\t}\n \n \t/*\n \t * Although we initialize opt->priv->paths with strdup_strings=0,\n-- \ngitgitgadget\n\n"},{"id":"430737","messageId":"ff568a612f5559789326108d5fb6d7090a6a56b6.1626841444.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","subject":"[PATCH v4 4/7] merge-ort: add a handle_deferred_entries() helper function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:24:01Z","receivedAt":"2021-07-21T04:24:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn order to allow trivial directory resolution, we first need to be able\nto gather more information to determine if the optimization is safe. To\nenable that, we need a way of deferring the recursion into the directory\nuntil a later time. Naturally, deferring the entry into a subtree means\nthat we need some function that will later recurse into the subdirectory\nexactly the same way that collect_merge_info_callback() would have done.\n\nAdd a helper function that does this. For now this function is not used\nbut a subsequent commit will change that. Future commits will also make\nthe function sometimes resolve directories instead of traversing inside.\n\nAcked-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 64 insertions(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex d9263ec5aca..f0a07684df6 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1202,6 +1202,70 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+MAYBE_UNUSED\n+static int handle_deferred_entries(struct merge_options *opt,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tstruct rename_info *renames = &opt->priv->renames;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\tint side, ret = 0;\n+\n+\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n+\t\trenames->deferred[side].trivial_merges_okay = 0;\n+\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n+\t\t\tstruct conflict_info *ci;\n+\t\t\tunsigned dirmask;\n+\t\t\tstruct tree_desc t[3];\n+\t\t\tvoid *buf[3] = {NULL,};\n+\t\t\tint i;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\t\t\tdirmask = ci->dirmask;\n+\n+\t\t\tinfo->name = path;\n+\t\t\tinfo->namelen = strlen(path);\n+\t\t\tinfo->pathlen = info->namelen + 1;\n+\n+\t\t\tfor (i = 0; i < 3; i++, dirmask >>= 1) {\n+\t\t\t\tif (i == 1 && ci->match_mask == 3)\n+\t\t\t\t\tt[1] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 5)\n+\t\t\t\t\tt[2] = t[0];\n+\t\t\t\telse if (i == 2 && ci->match_mask == 6)\n+\t\t\t\t\tt[2] = t[1];\n+\t\t\t\telse {\n+\t\t\t\t\tconst struct object_id *oid = NULL;\n+\t\t\t\t\tif (dirmask & 1)\n+\t\t\t\t\t\toid = &ci->stages[i].oid;\n+\t\t\t\t\tbuf[i] = fill_tree_descriptor(opt->repo,\n+\t\t\t\t\t\t\t\t t+i, oid);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tci->match_mask &= ci->filemask;\n+\t\t\topt->priv->current_dir_name = path;\n+\t\t\trenames->dir_rename_mask = dir_rename_mask;\n+\t\t\tif (renames->dir_rename_mask == 0 ||\n+\t\t\t renames->dir_rename_mask == 0x07)\n+\t\t\t\tret = traverse_trees(NULL, 3, t, info);\n+\t\t\telse\n+\t\t\t\tret = traverse_trees_wrapper(NULL, 3, t, info);\n+\n+\t\t\tfor (i = MERGE_BASE; i <= MERGE_SIDE2; i++)\n+\t\t\t\tfree(buf[i]);\n+\n+\t\t\tif (ret < 0)\n+\t\t\t\treturn ret;\n+\t\t}\n+\t}\n+\treturn ret;\n+}\n+\n static int collect_merge_info(struct merge_options *opt,\n \t\t\t struct tree *merge_base,\n \t\t\t struct tree *side1,\n-- \ngitgitgadget\n\n"},{"id":"430738","messageId":"5b01c118f10ec78eb77a696fa20a88038dfc28c9.1626841444.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","subject":"[PATCH v4 5/7] merge-ort: defer recursing into directories when merge base is matched","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:24:02Z","receivedAt":"2021-07-21T04:25:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen one side of history matches the merge base (including when the\nmerge base has no entry for the given directory), have\ncollect_merge_info_callback() defer recursing into the directory. To\nensure those entries are eventually handled, add a call to\nhandled_deferred_entries() in collect_merge_info() after\ntraverse_trees() returns.\n\nNote that the condition in collect_merge_info_callback() may look more\ncomplicated than necessary at first glance;\nrenames->trivial_merges_okay[side] is always true until\nhandle_deferred_entries() is called, and possible_trivial_merges[side]\nis always empty right now (and in the future won't be filled until\nhandle_deferred_entries() is called). However, when\nhandle_deferred_entries() calls traverse_trees() for the relevant\ndeferred directories, those traverse_trees() calls will once again end\nup in collect_merge_info_callback() for all the entries under those\nsubdirectories. The extra conditions are there for such deferred cases\nand will be used more as we do more with those variables.\n\nAcked-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 33 +++++++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex f0a07684df6..dbccf8c62e2 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1147,8 +1147,36 @@ static int collect_merge_info_callback(int n,\n \t\tstruct tree_desc t[3];\n \t\tvoid *buf[3] = {NULL, NULL, NULL};\n \t\tconst char *original_dir_name;\n-\t\tint i, ret;\n+\t\tint i, ret, side;\n \n+\t\t/*\n+\t\t * Check for whether we can avoid recursing due to one side\n+\t\t * matching the merge base. The side that does NOT match is\n+\t\t * the one that might have a rename destination we need.\n+\t\t */\n+\t\tassert(!side1_matches_mbase || !side2_matches_mbase);\n+\t\tside = side1_matches_mbase ? MERGE_SIDE2 :\n+\t\t\tside2_matches_mbase ? MERGE_SIDE1 : MERGE_BASE;\n+\t\tif (filemask == 0 && (dirmask == 2 || dirmask == 4)) {\n+\t\t\t/*\n+\t\t\t * Also defer recursing into new directories; set up a\n+\t\t\t * few variables to let us do so.\n+\t\t\t */\n+\t\t\tci->match_mask = (7 - dirmask);\n+\t\t\tside = dirmask / 2;\n+\t\t}\n+\t\tif (renames->dir_rename_mask != 0x07 &&\n+\t\t side != MERGE_BASE &&\n+\t\t renames->deferred[side].trivial_merges_okay &&\n+\t\t !strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t pi.string)) {\n+\t\t\tstrintmap_set(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t pi.string, renames->dir_rename_mask);\n+\t\t\trenames->dir_rename_mask = prev_dir_rename_mask;\n+\t\t\treturn mask;\n+\t\t}\n+\n+\t\t/* We need to recurse */\n \t\tci->match_mask &= filemask;\n \t\tnewinfo = *info;\n \t\tnewinfo.prev = info;\n@@ -1202,7 +1230,6 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n-MAYBE_UNUSED\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1291,6 +1318,8 @@ static int collect_merge_info(struct merge_options *opt,\n \n \ttrace2_region_enter(\"merge\", \"traverse_trees\", opt->repo);\n \tret = traverse_trees(NULL, 3, t, &info);\n+\tif (ret == 0)\n+\t\tret = handle_deferred_entries(opt, &info);\n \ttrace2_region_leave(\"merge\", \"traverse_trees\", opt->repo);\n \n \treturn ret;\n-- \ngitgitgadget\n\n"},{"id":"430739","messageId":"7b2112718157e65ba558842e27521df8c351f596.1626841444.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","subject":"[PATCH v4 6/7] merge-ort: avoid recursing into directories when we don't need to","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:24:03Z","receivedAt":"2021-07-21T04:25:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis combines the work of the last several patches, and implements the\nconditions when we don't need to recurse into directories. It's perhaps\neasiest to see the logic by separating the fact that a directory might\nhave both rename sources and rename destinations:\n\n * rename sources: only files present in the merge base can serve as\n rename sources, and only when one side deletes that file. When the\n tree on one side matches the merge base, that means every file\n within the subtree matches the merge base. This means that the\n skip-irrelevant-rename-detection optimization from before kicks in\n and we don't need any of these files as rename sources.\n\n * rename destinations: the tree that does not match the merge base\n might have newly added and hence unmatched destination files.\n This is what usually prevents us from doing trivial directory\n resolutions in the merge machinery. However, the fact that we have\n deferred recursing into this directory until the end means we know\n whether there are any unmatched relevant potential rename sources\n elsewhere in this merge. If there are no unmatched such relevant\n sources anywhere, then there is no need to look for unmatched\n potential rename destinations to match them with.\n\nThis informs our algorithm:\n * Search through relevant_sources; if we have entries, they better all\n be reflected in cached_pairs or cached_irrelevant, otherwise they\n represent an unmatched potential rename source (causing the\n optimization to be disallowed).\n * For any relevant_source represented in cached_pairs, we do need to\n to make sure to get the destination for each source, meaning we need\n to recurse into any ancestor directories of those destinations.\n * Once we've recursed into all the rename destinations for any\n relevant_sources in cached_pairs, we can then do the trivial\n directory resolution for the remaining directories.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 5.235 s ± 0.042 s 205.1 ms ± 3.8 ms\n mega-renames: 9.419 s ± 0.107 s 1.564 s ± 0.010 s\n just-one-mega: 480.1 ms ± 3.9 ms 479.5 ms ± 3.9 ms\n\nAcked-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 99 insertions(+), 3 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex dbccf8c62e2..a013708fa79 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1230,6 +1230,18 @@ static int collect_merge_info_callback(int n,\n \treturn mask;\n }\n \n+static void resolve_trivial_directory_merge(struct conflict_info *ci, int side)\n+{\n+\tVERIFY_CI(ci);\n+\tassert((side == 1 && ci->match_mask == 5) ||\n+\t (side == 2 && ci->match_mask == 3));\n+\toidcpy(&ci->merged.result.oid, &ci->stages[side].oid);\n+\tci->merged.result.mode = ci->stages[side].mode;\n+\tci->merged.is_null = is_null_oid(&ci->stages[side].oid);\n+\tci->match_mask = 0;\n+\tci->merged.clean = 1; /* (ci->filemask == 0); */\n+}\n+\n static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t struct traverse_info *info)\n {\n@@ -1239,9 +1251,72 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tint side, ret = 0;\n \n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n-\t\trenames->deferred[side].trivial_merges_okay = 0;\n-\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n-\t\t\t\t\t &iter, entry) {\n+\t\tunsigned optimization_okay = 1;\n+\t\tstruct strintmap copy;\n+\n+\t\t/* Loop over the set of paths we need to know rename info for */\n+\t\tstrset_for_each_entry(&renames->relevant_sources[side],\n+\t\t\t\t &iter, entry) {\n+\t\t\tchar *rename_target, *dir, *dir_marker;\n+\t\t\tstruct strmap_entry *e;\n+\n+\t\t\t/*\n+\t\t\t * If we don't know delete/rename info for this path,\n+\t\t\t * then we need to recurse into all trees to get all\n+\t\t\t * adds to make sure we have it.\n+\t\t\t */\n+\t\t\tif (strset_contains(&renames->cached_irrelevant[side],\n+\t\t\t\t\t entry->key))\n+\t\t\t\tcontinue;\n+\t\t\te = strmap_get_entry(&renames->cached_pairs[side],\n+\t\t\t\t\t entry->key);\n+\t\t\tif (!e) {\n+\t\t\t\toptimization_okay = 0;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/* If this is a delete, we have enough info already */\n+\t\t\trename_target = e->value;\n+\t\t\tif (!rename_target)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* If we already walked the rename target, we're good */\n+\t\t\tif (strmap_contains(&opt->priv->paths, rename_target))\n+\t\t\t\tcontinue;\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, we need to get a list of directories that\n+\t\t\t * will need to be recursed into to get this\n+\t\t\t * rename_target.\n+\t\t\t */\n+\t\t\tdir = xstrdup(rename_target);\n+\t\t\twhile ((dir_marker = strrchr(dir, '/'))) {\n+\t\t\t\t*dir_marker = '\\0';\n+\t\t\t\tif (strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t\t dir))\n+\t\t\t\t\tbreak;\n+\t\t\t\tstrset_add(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t dir);\n+\t\t\t}\n+\t\t\tfree(dir);\n+\t\t}\n+\t\trenames->deferred[side].trivial_merges_okay = optimization_okay;\n+\t\t/*\n+\t\t * We need to recurse into any directories in\n+\t\t * possible_trivial_merges[side] found in target_dirs[side].\n+\t\t * But when we recurse, we may need to queue up some of the\n+\t\t * subdirectories for possible_trivial_merges[side]. Since\n+\t\t * we can't safely iterate through a hashmap while also adding\n+\t\t * entries, move the entries into 'copy', iterate over 'copy',\n+\t\t * and then we'll also iterate anything added into\n+\t\t * possible_trivial_merges[side] once this loop is done.\n+\t\t */\n+\t\tcopy = renames->deferred[side].possible_trivial_merges;\n+\t\tstrintmap_init_with_options(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t\t 0,\n+\t\t\t\t\t NULL,\n+\t\t\t\t\t 0);\n+\t\tstrintmap_for_each_entry(©, &iter, entry) {\n \t\t\tconst char *path = entry->key;\n \t\t\tunsigned dir_rename_mask = (intptr_t)entry->value;\n \t\t\tstruct conflict_info *ci;\n@@ -1254,6 +1329,13 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tVERIFY_CI(ci);\n \t\t\tdirmask = ci->dirmask;\n \n+\t\t\tif (optimization_okay &&\n+\t\t\t !strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t path)) {\n+\t\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tinfo->name = path;\n \t\t\tinfo->namelen = strlen(path);\n \t\t\tinfo->pathlen = info->namelen + 1;\n@@ -1289,6 +1371,20 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\tif (ret < 0)\n \t\t\t\treturn ret;\n \t\t}\n+\t\tstrintmap_clear(©);\n+\t\tstrintmap_for_each_entry(&renames->deferred[side].possible_trivial_merges,\n+\t\t\t\t\t &iter, entry) {\n+\t\t\tconst char *path = entry->key;\n+\t\t\tstruct conflict_info *ci;\n+\n+\t\t\tci = strmap_get(&opt->priv->paths, path);\n+\t\t\tVERIFY_CI(ci);\n+\n+\t\t\tassert(renames->deferred[side].trivial_merges_okay &&\n+\t\t\t !strset_contains(&renames->deferred[side].target_dirs,\n+\t\t\t\t\t\tpath));\n+\t\t\tresolve_trivial_directory_merge(ci, side);\n+\t\t}\n \t}\n \treturn ret;\n }\n-- \ngitgitgadget\n\n"},{"id":"430740","messageId":"c9ada8369e6575be488028aae0f654422a9b1410.1626841444.git.gitgitgadget@gmail.com","threadId":"56035","inReplyTo":"pull.988.v4.git.1626841444.gitgitgadget@gmail.com","subject":"[PATCH v4 7/7] merge-ort: restart merge with cached renames to reduce process entry cost","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-21T04:24:04Z","receivedAt":"2021-07-21T04:25:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe merge algorithm mostly consists of the following three functions:\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\nPrior to the trivial directory resolution optimization of the last half\ndozen commits, process_entries() was consistently the slowest, followed\nby collect_merge_info(), then detect_and_process_renames(). When the\ntrivial directory resolution applies, it often dramatically decreases\nthe amount of time spent in the two slower functions.\n\nLooking at the performance results in the previous commit, the trivial\ndirectory resolution optimization helps amazingly well when there are no\nrelevant renames. It also helps really well when reapplying a long\nseries of linear commits (such as in a rebase or cherry-pick), since the\nrelevant renames may well be cached from the first reapplied commit.\nBut when there are any relevant renames that are not cached (represented\nby the just-one-mega testcase), then the optimization does not help at\nall.\n\nOften, I noticed that when the optimization does not apply, it is\nbecause there are a handful of relevant sources -- maybe even only one.\nIt felt frustrating to need to recurse into potentially hundreds or even\nthousands of directories just for a single rename, but it was needed for\ncorrectness.\n\nHowever, staring at this list of functions and noticing that\nprocess_entries() is the most expensive and knowing I could avoid it if\nI had cached renames suggested a simple idea: change\n collect_merge_info()\n detect_and_process_renames()\n process_entries()\ninto\n collect_merge_info()\n detect_and_process_renames()\n \n collect_merge_info()\n detect_and_process_renames()\n process_entries()\n\nThis may seem odd and look like more work. However, note that although\nwe run collect_merge_info() twice, the second time we get to employ\ntrivial directory resolves, which makes it much faster, so the increased\ntime in collect_merge_info() is small. While we run\ndetect_and_process_renames() again, all renames are cached so it's\nnearly a no-op (we don't call into diffcore_rename_extended() but we do\nhave a little bit of data structure checking and fixing up). And the\nbig payoff comes from the fact that process_entries(), will be much\nfaster due to having far fewer entries to process.\n\nThis restarting only makes sense if we can save recursing into enough\ndirectories to make it worth our while. Introduce a simple heuristic to\nguide this. Note that this heuristic uses a \"wanted_factor\" that I have\nvirtually no actual real world data for, just some back-of-the-envelope\nquasi-scientific calculations that I included in some comments and then\nplucked a simple round number out of thin air. It could be that\ntweaking this number to make it either higher or lower improves the\noptimization. (There's slightly more here; when I first introduced this\noptimization, I used a factor of 10, because I was completely confident\nit was big enough to not cause slowdowns in special cases. I was\ncertain it was higher than needed. Several months later, I added the\nrough calculations which make me think the optimal number is close to 2;\nbut instead of pushing to the limit, I just bumped it to 3 to reduce the\nrisk that there are special cases where this optimization can result in\nslowing down the code a little. If the ratio of path counts is below 3,\nwe probably will only see minor performance improvements at best\nanyway.)\n\nAlso, note that while the diffstat looks kind of long (nearly 100\nlines), more than half of it is in two comments explaining how things\nwork.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 205.1 ms ± 3.8 ms 204.2 ms ± 3.0 ms\n mega-renames: 1.564 s ± 0.010 s 1.076 s ± 0.015 s\n just-one-mega: 479.5 ms ± 3.9 ms 364.1 ms ± 7.0 ms\n\nAcked-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 92 +++++++++++++++++++++++++++--\n t/t6423-merge-rename-directories.sh | 2 +-\n 2 files changed, 87 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex a013708fa79..e361443087a 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -213,6 +213,7 @@ struct rename_info {\n \t * MERGE_SIDE2: cached data from side2 can be reused\n \t * MERGE_SIDE1: cached data from side1 can be reused\n \t * 0: no cached data can be reused\n+\t * -1: See redo_after_renames; both sides can be reused.\n \t */\n \tint cached_pairs_valid_side;\n \n@@ -258,6 +259,28 @@ struct rename_info {\n \t */\n \tstruct strset cached_irrelevant[3];\n \n+\t/*\n+\t * redo_after_renames: optimization flag for \"restarting\" the merge\n+\t *\n+\t * Sometimes it pays to detect renames, cache them, and then\n+\t * restart the merge operation from the beginning. The reason for\n+\t * this is that when we know where all the renames are, we know\n+\t * whether a certain directory has any paths under it affected --\n+\t * and if a directory is not affected then it permits us to do\n+\t * trivial tree merging in more cases. Doing trivial tree merging\n+\t * prevents the need to run process_entry() on every path\n+\t * underneath trees that can be trivially merged, and\n+\t * process_entry() is more expensive than collect_merge_info() --\n+\t * plus, the second collect_merge_info() will be much faster since\n+\t * it doesn't have to recurse into the relevant trees.\n+\t *\n+\t * Values for this flag:\n+\t * 0 = don't bother, not worth it (or conditions not yet checked)\n+\t * 1 = conditions for optimization met, optimization worthwhile\n+\t * 2 = we already did it (don't restart merge yet again)\n+\t */\n+\tunsigned redo_after_renames;\n+\n \t/*\n \t * needed_limit: value needed for inexact rename detection to run\n \t *\n@@ -541,7 +564,8 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \t\tstrintmap_func(&renames->relevant_sources[i]);\n \t\tif (!reinitialize)\n \t\t\tassert(renames->cached_pairs_valid_side == 0);\n-\t\tif (i != renames->cached_pairs_valid_side) {\n+\t\tif (i != renames->cached_pairs_valid_side &&\n+\t\t -1 != renames->cached_pairs_valid_side) {\n \t\t\tstrset_func(&renames->cached_target_names[i]);\n \t\t\tstrmap_func(&renames->cached_pairs[i], 1);\n \t\t\tstrset_func(&renames->cached_irrelevant[i]);\n@@ -1249,7 +1273,9 @@ static int handle_deferred_entries(struct merge_options *opt,\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n \tint side, ret = 0;\n+\tint path_count_before, path_count_after = 0;\n \n+\tpath_count_before = strmap_get_size(&opt->priv->paths);\n \tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n \t\tunsigned optimization_okay = 1;\n \t\tstruct strintmap copy;\n@@ -1385,7 +1411,32 @@ static int handle_deferred_entries(struct merge_options *opt,\n \t\t\t\t\t\tpath));\n \t\t\tresolve_trivial_directory_merge(ci, side);\n \t\t}\n+\t\tif (!optimization_okay || path_count_after)\n+\t\t\tpath_count_after = strmap_get_size(&opt->priv->paths);\n \t}\n+\tif (path_count_after) {\n+\t\t/*\n+\t\t * The choice of wanted_factor here does not affect\n+\t\t * correctness, only performance. When the\n+\t\t * path_count_after / path_count_before\n+\t\t * ratio is high, redoing after renames is a big\n+\t\t * performance boost. I suspect that redoing is a wash\n+\t\t * somewhere near a value of 2, and below that redoing will\n+\t\t * slow things down. I applied a fudge factor and picked\n+\t\t * 3; see the commit message when this was introduced for\n+\t\t * back of the envelope calculations for this ratio.\n+\t\t */\n+\t\tconst int wanted_factor = 3;\n+\n+\t\t/* We should only redo collect_merge_info one time */\n+\t\tassert(renames->redo_after_renames == 0);\n+\n+\t\tif (path_count_after / path_count_before >= wanted_factor) {\n+\t\t\trenames->redo_after_renames = 1;\n+\t\t\trenames->cached_pairs_valid_side = -1;\n+\t\t}\n+\t} else if (renames->redo_after_renames == 2)\n+\t\trenames->redo_after_renames = 0;\n \treturn ret;\n }\n \n@@ -2828,8 +2879,8 @@ static int compare_pairs(const void *a_, const void *b_)\n }\n \n /* Call diffcore_rename() to update deleted/added pairs into rename pairs */\n-static void detect_regular_renames(struct merge_options *opt,\n-\t\t\t\t unsigned side_index)\n+static int detect_regular_renames(struct merge_options *opt,\n+\t\t\t\t unsigned side_index)\n {\n \tstruct diff_options diff_opts;\n \tstruct rename_info *renames = &opt->priv->renames;\n@@ -2842,7 +2893,7 @@ static void detect_regular_renames(struct merge_options *opt,\n \t\t * side had directory renames.\n \t\t */\n \t\tresolve_diffpair_statuses(&renames->pairs[side_index]);\n-\t\treturn;\n+\t\treturn 0;\n \t}\n \n \tpartial_clear_dir_rename_count(&renames->dir_rename_count[side_index]);\n@@ -2868,6 +2919,8 @@ static void detect_regular_renames(struct merge_options *opt,\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n \tresolve_diffpair_statuses(&diff_queued_diff);\n \n+\tif (diff_opts.needed_rename_limit > 0)\n+\t\trenames->redo_after_renames = 0;\n \tif (diff_opts.needed_rename_limit > renames->needed_limit)\n \t\trenames->needed_limit = diff_opts.needed_rename_limit;\n \n@@ -2877,6 +2930,8 @@ static void detect_regular_renames(struct merge_options *opt,\n \tdiff_queued_diff.nr = 0;\n \tdiff_queued_diff.queue = NULL;\n \tdiff_flush(&diff_opts);\n+\n+\treturn 1;\n }\n \n /*\n@@ -2966,14 +3021,32 @@ static int detect_and_process_renames(struct merge_options *opt,\n \tstruct diff_queue_struct combined;\n \tstruct rename_info *renames = &opt->priv->renames;\n \tint need_dir_renames, s, clean = 1;\n+\tunsigned detection_run = 0;\n \n \tmemset(&combined, 0, sizeof(combined));\n \tif (!possible_renames(renames))\n \t\tgoto cleanup;\n \n \ttrace2_region_enter(\"merge\", \"regular renames\", opt->repo);\n-\tdetect_regular_renames(opt, MERGE_SIDE1);\n-\tdetect_regular_renames(opt, MERGE_SIDE2);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE1);\n+\tdetection_run |= detect_regular_renames(opt, MERGE_SIDE2);\n+\tif (renames->redo_after_renames && detection_run) {\n+\t\tint i, side;\n+\t\tstruct diff_filepair *p;\n+\n+\t\t/* Cache the renames, we found */\n+\t\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; side++) {\n+\t\t\tfor (i = 0; i < renames->pairs[side].nr; ++i) {\n+\t\t\t\tp = renames->pairs[side].queue[i];\n+\t\t\t\tpossibly_cache_new_pair(renames, p, side, NULL);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/* Restart the merge with the cached renames */\n+\t\trenames->redo_after_renames = 2;\n+\t\ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n+\t\tgoto cleanup;\n+\t}\n \tuse_cached_pairs(opt, &renames->cached_pairs[1], &renames->pairs[1]);\n \tuse_cached_pairs(opt, &renames->cached_pairs[2], &renames->pairs[2]);\n \ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n@@ -4390,6 +4463,7 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \t\t\t\t\t opt->subtree_shift);\n \t}\n \n+redo:\n \ttrace2_region_enter(\"merge\", \"collect_merge_info\", opt->repo);\n \tif (collect_merge_info(opt, merge_base, side1, side2) != 0) {\n \t\t/*\n@@ -4409,6 +4483,12 @@ static void merge_ort_nonrecursive_internal(struct merge_options *opt,\n \tresult->clean = detect_and_process_renames(opt, merge_base,\n \t\t\t\t\t\t side1, side2);\n \ttrace2_region_leave(\"merge\", \"renames\", opt->repo);\n+\tif (opt->priv->renames.redo_after_renames == 2) {\n+\t\ttrace2_region_enter(\"merge\", \"reset_maps\", opt->repo);\n+\t\tclear_or_reinit_internal_opts(opt->priv, 1);\n+\t\ttrace2_region_leave(\"merge\", \"reset_maps\", opt->repo);\n+\t\tgoto redo;\n+\t}\n \n \ttrace2_region_enter(\"merge\", \"process_entries\", opt->repo);\n \tprocess_entries(opt, &working_tree_oid);\ndiff --git a/t/t6423-merge-rename-directories.sh b/t/t6423-merge-rename-directories.sh\nindex e834b7e6efe..d8919d276a1 100755\n--- a/t/t6423-merge-rename-directories.sh\n+++ b/t/t6423-merge-rename-directories.sh\n@@ -4797,7 +4797,7 @@ test_setup_12f () {\n \t)\n }\n \n-test_expect_merge_algorithm failure failure '12f: Trivial directory resolve, caching, all kinds of fun' '\n+test_expect_merge_algorithm failure success '12f: Trivial directory resolve, caching, all kinds of fun' '\n \ttest_setup_12f &&\n \t(\n \t\tcd 12f &&\n-- \ngitgitgadget\n"}]}