{"thread":{"id":"55146","subject":"[PATCH 00/10] Optimization batch 8: use file basenames even more","startedAt":"2021-02-14T07:59:51Z","lastAt":"2021-03-09T21:53:08Z","messageCount":61,"participants":["Elijah Newren via GitGitGadget","Derrick Stolee","Elijah Newren","Ævar Arnfjörð Bjarmason","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"416866","messageId":"pull.844.git.1613289544.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":null,"subject":"[PATCH 00/10] Optimization batch 8: use file basenames even more","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:58:53Z","receivedAt":"2021-02-14T07:59:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on en/diffcore-rename (a concatenation of what I was\ncalling ort-perf-batch-6 and ort-perf-batch-7).\n\nI apologize for submitting this so soon after a re-roll of the other two\nseries, but ideas from it kept coming up in the discussions, so I figured\nit'd help to just have them all be submitted. This series can also be rolled\ninto en/diffcore-rename if it helps simplify.\n\n=== Optimization idea ===\n\nThis series uses file basenames (portions of the path after the last '/',\nincluding file extension) in a more involved fashion to guide rename\ndetection. It's a follow-on improvement to \"Optimization #3\" from my Git\nMerge 2020 talk[1]. The basic idea behind this series is the same as the\nlast series: people frequently move files across directories while keeping\nthe filenames the same, thus files with the same basename are likely rename\ncandidates. However, the previous optimization only applies when basenames\nare unique among remaining adds and deletes after exact rename detection, so\nwe need to do something else to match up the remaining basenames. When there\nare many files with the same basename (e.g. .gitignore, Makefile,\nbuild.gradle, or maybe even setup.c, AbtractFactory.java, etc.), being able\nto \"guess\" which directory a given file likely would have moved to can\nprovide us with a likely rename candidate if there is a file with the same\nbasename in that directory. Since exact rename detection is done first, we\ncan use nearby exact renames to help us guess where any given non-unique\nbasename file may have moved; it just means doing \"directory rename\ndetection\" limited to exact renames.\n\nThere are definitely cases when this strategy still won't help us: (1) We\nonly use this strategy when the directory in which the original file was\nfound has also been removed, (2) a lack of exact renames from the given\ndirectory will prevents us from making a new directory prediction, (3) even\nif we predict a new directory there may be no file with the given basename\nin it, and (4) even if there is an unmatched add with the appropriate\nbasename in the predicted directory, it may not meet the higher\nmin_basename_score similarity threshold.\n\nIt may be worth noting that directory rename detection at most predicts one\nnew directory, which we use to ensure that we only compare any given file\nwith at most one other file. That's important for compatibility with future\noptimizations.\n\nHowever, despite the caveats and limited applicability, this idea provides\nsome nice speedups.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\nmega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\njust-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with (as noted in the same commit message)\nwere:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\n=== Alternative, rejected idea ===\n\nThere was an alternative idea to the series presented here that I also\ntried: instead of using directory rename detection based on exact renames to\npredict where files would be renamed and then comparing to the file with the\nsame basename in the new directory, one could instead take all files with\nthe same basename -- both sources and destinations -- and then do a smaller\nM x N comparison on all those files to find renames. Any non-matches after\nthat step could be combined with all other files for the big inexact rename\ndetection step.\n\nThere are two problems with such a strategy, though.\n\nOne is that in the worst case, you approximately double the cost of rename\ndetection (if most potential rename pairs all have the same basename but\nthey aren't actually matches, you end up comparing twice).\n\nThe second issue isn't clear until trying to combine this idea with later\nperformance optimizations. The next optimization will provide a way to\nfilter out several of the rename sources. If our inexact rename detection\nmatrix is sized 1 x 4000 because we can remove all but one source file, but\nwe have 100 files with the same basename, then a 100 x 100 comparison is\nactually more costly than a 1 x 4000 comparison -- and we don't need most of\nthe renames from the 100 x 100 comparison. The advantage of the directory\nrename detection based idea for finding which basenames to match up, is that\nthe cost for each file is linear (or, said another way, scales proportional\nto doing a diff on that file). As such, the costs for this preliminary\noptimization are nicely controlled and the worst case scenario is it has\nspent a little extra time upfront but still has to do the full inexact\nrename detection.\n\n[1]\nhttps://github.com/newren/presentations/blob/pdfs/merge-performance/merge-performance-slides.pdf\n\nElijah Newren (10):\n Move computation of dir_rename_count from merge-ort to diffcore-rename\n diffcore-rename: add functions for clearing dir_rename_count\n diffcore-rename: move dir_rename_counts into a dir_rename_info struct\n diffcore-rename: extend cleanup_dir_rename_info()\n diffcore-rename: compute dir_rename_counts in stages\n diffcore-rename: add a mapping of destination names to their indices\n diffcore-rename: add a dir_rename_guess field to dir_rename_info\n diffcore-rename: add a new idx_possible_rename function\n diffcore-rename: limit dir_rename_counts computation to relevant dirs\n diffcore-rename: use directory rename guided basename comparisons\n\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 439 ++++++++++++++++++++++++++++++++--\n diffcore.h | 7 +\n merge-ort.c | 144 +----------\n 4 files changed, 439 insertions(+), 153 deletions(-)\n\n\nbase-commit: aeca14f748afc7fb5b65bca56ea2ebd970729814\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-844%2Fnewren%2Fort-perf-batch-8-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-844/newren/ort-perf-batch-8-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/844\n-- \ngitgitgadget\n"},{"id":"416867","messageId":"612da82f049cbe877cf924f743a3e4059483ce51.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 02/10] diffcore-rename: add functions for clearing dir_rename_count","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:58:55Z","receivedAt":"2021-02-14T08:00:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs we adjust the usage of dir_rename_count we want to have functions for\nclearing, or partially clearing it out. Add such functions.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 19 +++++++++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 12 +++---------\n 3 files changed, 24 insertions(+), 9 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 33cfc5848611..614a8d63012d 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -486,6 +486,25 @@ static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n \t}\n }\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrmap_for_each_entry(dir_rename_count, &iter, entry) {\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tstrintmap_clear(counts);\n+\t}\n+\tstrmap_partial_clear(dir_rename_count, 1);\n+}\n+\n+MAYBE_UNUSED\n+static void clear_dir_rename_count(struct strmap *dir_rename_count)\n+{\n+\tpartial_clear_dir_rename_count(dir_rename_count);\n+\tstrmap_clear(dir_rename_count, 1);\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\ndiff --git a/diffcore.h b/diffcore.h\nindex db55d3853071..c6ba64abd198 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -161,6 +161,8 @@ struct diff_filepair *diff_queue(struct diff_queue_struct *,\n \t\t\t\t struct diff_filespec *);\n void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n+\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex c4467e073b45..467404cc0a35 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -351,17 +351,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \n \t/* Free memory used by various renames maps */\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstruct hashmap_iter iter;\n-\t\tstruct strmap_entry *entry;\n-\n \t\tstrset_func(&renames->dirs_removed[i]);\n \n-\t\tstrmap_for_each_entry(&renames->dir_rename_count[i],\n-\t\t\t\t &iter, entry) {\n-\t\t\tstruct strintmap *counts = entry->value;\n-\t\t\tstrintmap_clear(counts);\n-\t\t}\n-\t\tstrmap_func(&renames->dir_rename_count[i], 1);\n+\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n+\t\tif (!reinitialize)\n+\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \n \t\tstrmap_func(&renames->dir_renames[i], 0);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"416868","messageId":"fec4f1d44c0694cf92c9b98f2b6519a15fb78188.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 01/10] Move computation of dir_rename_count from merge-ort to diffcore-rename","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:58:54Z","receivedAt":"2021-02-14T08:00:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA previous commit noted that it is very common for people to move files\nacross directories while keeping their filename the same. The last few\ncommits took advantage of this and showed that we can accelerate rename\ndetection significantly using basenames; since files with the same\nbasename serve as likely rename candidates, we can check those first and\nremove them from the rename candidate pool if they are sufficiently\nsimilar.\n\nUnfortunately, the previous optimization was limited by the fact that\nthe remaining basenames after exact rename detection are not always\nunique. Many repositories have hundreds of build files with the same\nname (e.g. Makefile, .gitignore, build.gradle, etc.), and may even have\nhundreds of source files with the same name. (For example, the linux\nkernel has 100 setup.c, 87 irq.c, and 112 core.c files. A repository at\n$DAYJOB has a lot of ObjectFactory.java and Plugin.java files).\n\nFor these files with non-unique basenames, we are faced with the task of\nattempting to determine or guess which directory they may have been\nrelocated to. Such a task is precisely the job of directory rename\ndetection. However, there are two catches: (1) the directory rename\ndetection code has traditionally been part of the merge machinery rather\nthan diffcore-rename.c, and (2) directory rename detection currently\nruns after regular rename detection is complete. The 1st catch is just\nan implementation issue that can be overcome by some code shuffling.\nThe 2nd requires us to add a further approximation: we only have access\nto exact renames at this point, so we need to do directory rename\ndetection based on just exact renames. In some cases we won't have\nexact renames, in which case this extra optimization won't apply. We\nalso choose to not apply the optimization unless we know that the\nunderlying directory was removed, which will require extra data to be\npassed in to diffcore_rename_extended(). Also, even if we get a\nprediction about which directory a file may have relocated to, we will\nstill need to check to see if there is a file in the predicted\ndirectory, and then compare the two files to see if they meet the higher\nmin_basename_score threshold required for marking the two files as\nrenames.\n\nThis commit and the next few will set up the necessary infrastructure to\ndo such computations. This commit merely moves the computation of\ndir_rename_count from merge-ort.c to diffcore-rename.c, making slight\nadjustments to the data structures based on the move. While the\ndiffstat looks large, viewing this commit with --color-moved makes it\nclear that only about 20 lines changed. With this patch, the\ncomputation of dir_rename_count is still only done after inexact rename\ndetection, but subsequent commits will add a preliminary computation of\ndir_rename_count after exact rename detection, followed by some updates\nafter inexact rename detection.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 134 +++++++++++++++++++++++++++++++++++++++++++++-\n diffcore.h | 5 ++\n merge-ort.c | 132 ++-------------------------------------------\n 3 files changed, 141 insertions(+), 130 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 41558185ae1d..33cfc5848611 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,125 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+static void dirname_munge(char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\tif (!slash)\n+\t\tslash = filename;\n+\t*slash = '\\0';\n+}\n+\n+static void increment_count(struct strmap *dir_rename_count,\n+\t\t\t char *old_dir,\n+\t\t\t char *new_dir)\n+{\n+\tstruct strintmap *counts;\n+\tstruct strmap_entry *e;\n+\n+\t/* Get the {new_dirs -> counts} mapping using old_dir */\n+\te = strmap_get_entry(dir_rename_count, old_dir);\n+\tif (e) {\n+\t\tcounts = e->value;\n+\t} else {\n+\t\tcounts = xmalloc(sizeof(*counts));\n+\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n+\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t}\n+\n+\t/* Increment the count for new_dir */\n+\tstrintmap_incr(counts, new_dir, 1);\n+}\n+\n+static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t const char *oldname,\n+\t\t\t\t const char *newname)\n+{\n+\tchar *old_dir = xstrdup(oldname);\n+\tchar *new_dir = xstrdup(newname);\n+\tchar new_dir_first_char = new_dir[0];\n+\tint first_time_in_loop = 1;\n+\n+\twhile (1) {\n+\t\tdirname_munge(old_dir);\n+\t\tdirname_munge(new_dir);\n+\n+\t\t/*\n+\t\t * When renaming\n+\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n+\t\t * then this suggests that both\n+\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n+\t\t * a/b/c/d/ => a/b/some/thing/else/\n+\t\t * so we want to increment counters for both. We do NOT,\n+\t\t * however, also want to suggest that there was the following\n+\t\t * rename:\n+\t\t * a/b/c/ => a/b/some/thing/\n+\t\t * so we need to quit at that point.\n+\t\t *\n+\t\t * Note the when first_time_in_loop, we only strip off the\n+\t\t * basename, and we don't care if that's different.\n+\t\t */\n+\t\tif (!first_time_in_loop) {\n+\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n+\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n+\t\t\tif (!*new_dir) {\n+\t\t\t\t/*\n+\t\t\t\t * Special case when renaming to root directory,\n+\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n+\t\t\t\t * something like\n+\t\t\t\t * a/b/subdir => subdir\n+\t\t\t\t * and so dirname_munge() sets things up so that\n+\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n+\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n+\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n+\t\t\t\t * in new_dir, so we have to compare differently.\n+\t\t\t\t */\n+\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n+\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t} else {\n+\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strset_contains(dirs_removed, old_dir))\n+\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\telse\n+\t\t\tbreak;\n+\n+\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n+\t\tif (!*old_dir || !*new_dir)\n+\t\t\tbreak;\n+\n+\t\tfirst_time_in_loop = 0;\n+\t}\n+\n+\t/* Free resources we don't need anymore */\n+\tfree(old_dir);\n+\tfree(new_dir);\n+}\n+\n+static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed)\n+{\n+\tint i;\n+\n+\t/* Set up dir_rename_count */\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\t/*\n+\t\t * Make dir_rename_count contain a map of a map:\n+\t\t * old_directory -> {new_directory -> count}\n+\t\t * In other words, for every pair look at the directories for\n+\t\t * the old filename and the new filename and count how many\n+\t\t * times that pairing occurs.\n+\t\t */\n+\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\t\t\t\t rename_dst[i].p->one->path,\n+\t\t\t\t\t rename_dst[i].p->two->path);\n+\t}\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\n@@ -640,7 +759,9 @@ static void remove_unneeded_paths_from_src(int detecting_copies)\n \trename_src_nr = new_num_src;\n }\n \n-void diffcore_rename(struct diff_options *options)\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count)\n {\n \tint detect_rename = options->detect_rename;\n \tint minimum_score = options->rename_score;\n@@ -653,6 +774,7 @@ void diffcore_rename(struct diff_options *options)\n \tstruct progress *progress = NULL;\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n+\tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n@@ -841,6 +963,11 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n+\t/*\n+\t * Now that renames have been computed, compute dir_rename_count */\n+\tif (dirs_removed && dir_rename_count)\n+\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n@@ -923,3 +1050,8 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"write back to queue\", options->repo);\n \treturn;\n }\n+\n+void diffcore_rename(struct diff_options *options)\n+{\n+\tdiffcore_rename_extended(options, NULL, NULL);\n+}\ndiff --git a/diffcore.h b/diffcore.h\nindex d2a63c5c71f4..db55d3853071 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -8,6 +8,8 @@\n \n struct diff_options;\n struct repository;\n+struct strmap;\n+struct strset;\n struct userdiff_driver;\n \n /* This header file is internal between diff.c and its diff transformers\n@@ -161,6 +163,9 @@ void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count);\n void diffcore_merge_broken(void);\n void diffcore_pickaxe(struct diff_options *);\n void diffcore_order(const char *orderfile);\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 603d30c52170..c4467e073b45 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1302,131 +1302,6 @@ static char *handle_path_level_conflicts(struct merge_options *opt,\n \treturn new_path;\n }\n \n-static void dirname_munge(char *filename)\n-{\n-\tchar *slash = strrchr(filename, '/');\n-\tif (!slash)\n-\t\tslash = filename;\n-\t*slash = '\\0';\n-}\n-\n-static void increment_count(struct strmap *dir_rename_count,\n-\t\t\t char *old_dir,\n-\t\t\t char *new_dir)\n-{\n-\tstruct strintmap *counts;\n-\tstruct strmap_entry *e;\n-\n-\t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n-\tif (e) {\n-\t\tcounts = e->value;\n-\t} else {\n-\t\tcounts = xmalloc(sizeof(*counts));\n-\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n-\t}\n-\n-\t/* Increment the count for new_dir */\n-\tstrintmap_incr(counts, new_dir, 1);\n-}\n-\n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t const char *oldname,\n-\t\t\t\t const char *newname)\n-{\n-\tchar *old_dir = xstrdup(oldname);\n-\tchar *new_dir = xstrdup(newname);\n-\tchar new_dir_first_char = new_dir[0];\n-\tint first_time_in_loop = 1;\n-\n-\twhile (1) {\n-\t\tdirname_munge(old_dir);\n-\t\tdirname_munge(new_dir);\n-\n-\t\t/*\n-\t\t * When renaming\n-\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n-\t\t * then this suggests that both\n-\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n-\t\t * a/b/c/d/ => a/b/some/thing/else/\n-\t\t * so we want to increment counters for both. We do NOT,\n-\t\t * however, also want to suggest that there was the following\n-\t\t * rename:\n-\t\t * a/b/c/ => a/b/some/thing/\n-\t\t * so we need to quit at that point.\n-\t\t *\n-\t\t * Note the when first_time_in_loop, we only strip off the\n-\t\t * basename, and we don't care if that's different.\n-\t\t */\n-\t\tif (!first_time_in_loop) {\n-\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n-\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n-\t\t\tif (!*new_dir) {\n-\t\t\t\t/*\n-\t\t\t\t * Special case when renaming to root directory,\n-\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n-\t\t\t\t * something like\n-\t\t\t\t * a/b/subdir => subdir\n-\t\t\t\t * and so dirname_munge() sets things up so that\n-\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n-\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n-\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n-\t\t\t\t * in new_dir, so we have to compare differently.\n-\t\t\t\t */\n-\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n-\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t} else {\n-\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n-\t\telse\n-\t\t\tbreak;\n-\n-\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n-\t\tif (!*old_dir || !*new_dir)\n-\t\t\tbreak;\n-\n-\t\tfirst_time_in_loop = 0;\n-\t}\n-\n-\t/* Free resources we don't need anymore */\n-\tfree(old_dir);\n-\tfree(new_dir);\n-}\n-\n-static void compute_rename_counts(struct diff_queue_struct *pairs,\n-\t\t\t\t struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pairs->nr; ++i) {\n-\t\tstruct diff_filepair *pair = pairs->queue[i];\n-\n-\t\t/* File not part of directory rename if it wasn't renamed */\n-\t\tif (pair->status != 'R')\n-\t\t\tcontinue;\n-\n-\t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n-\t\t * old_directory -> {new_directory -> count}\n-\t\t * In other words, for every pair look at the directories for\n-\t\t * the old filename and the new filename and count how many\n-\t\t * times that pairing occurs.\n-\t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n-\t\t\t\t\t pair->one->path,\n-\t\t\t\t\t pair->two->path);\n-\t}\n-}\n-\n static void get_provisional_directory_renames(struct merge_options *opt,\n \t\t\t\t\t unsigned side,\n \t\t\t\t\t int *clean)\n@@ -1435,9 +1310,6 @@ static void get_provisional_directory_renames(struct merge_options *opt,\n \tstruct strmap_entry *entry;\n \tstruct rename_info *renames = &opt->priv->renames;\n \n-\tcompute_rename_counts(&renames->pairs[side],\n-\t\t\t &renames->dir_rename_count[side],\n-\t\t\t &renames->dirs_removed[side]);\n \t/*\n \t * Collapse\n \t * dir_rename_count: old_directory -> {new_directory -> count}\n@@ -2162,7 +2034,9 @@ static void detect_regular_renames(struct merge_options *opt,\n \n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n-\tdiffcore_rename(&diff_opts);\n+\tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t &renames->dirs_removed[side_index],\n+\t\t\t\t &renames->dir_rename_count[side_index]);\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n \tresolve_diffpair_statuses(&diff_queued_diff);\n \n-- \ngitgitgadget\n\n"},{"id":"416869","messageId":"93f98fc0b2644aab9e98b7a32e88561618c7d4c0.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 03/10] diffcore-rename: move dir_rename_counts into a dir_rename_info struct","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:58:56Z","receivedAt":"2021-02-14T08:00:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis is a purely cosmetic change for now, but we will be adding\nadditional information to the struct and changing where and how it is\nsetup and used in subsequent patches.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 39 ++++++++++++++++++++++++++-------------\n 1 file changed, 26 insertions(+), 13 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 614a8d63012d..7759c9a3a2ed 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,11 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+struct dir_rename_info {\n+\tstruct strmap *dir_rename_count;\n+\tunsigned setup;\n+};\n+\n static void dirname_munge(char *filename)\n {\n \tchar *slash = strrchr(filename, '/');\n@@ -375,7 +380,7 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n-static void increment_count(struct strmap *dir_rename_count,\n+static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n {\n@@ -383,20 +388,20 @@ static void increment_count(struct strmap *dir_rename_count,\n \tstruct strmap_entry *e;\n \n \t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n+\te = strmap_get_entry(info->dir_rename_count, old_dir);\n \tif (e) {\n \t\tcounts = e->value;\n \t} else {\n \t\tcounts = xmalloc(sizeof(*counts));\n \t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t\tstrmap_put(info->dir_rename_count, old_dir, counts);\n \t}\n \n \t/* Increment the count for new_dir */\n \tstrintmap_incr(counts, new_dir, 1);\n }\n \n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t const char *oldname,\n \t\t\t\t const char *newname)\n@@ -450,7 +455,7 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t}\n \n \t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\t\tincrement_count(info, old_dir, new_dir);\n \t\telse\n \t\t\tbreak;\n \n@@ -466,12 +471,15 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n+static void compute_dir_rename_counts(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n-\t/* Set up dir_rename_count */\n+\tinfo->setup = 1;\n+\tinfo->dir_rename_count = dir_rename_count;\n+\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n \t\t * Make dir_rename_count contain a map of a map:\n@@ -480,7 +488,7 @@ static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t * the old filename and the new filename and count how many\n \t\t * times that pairing occurs.\n \t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n@@ -499,10 +507,13 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n }\n \n MAYBE_UNUSED\n-static void clear_dir_rename_count(struct strmap *dir_rename_count)\n+static void cleanup_dir_rename_info(struct dir_rename_info *info)\n {\n-\tpartial_clear_dir_rename_count(dir_rename_count);\n-\tstrmap_clear(dir_rename_count, 1);\n+\tif (!info->setup)\n+\t\treturn;\n+\n+\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\tstrmap_clear(info->dir_rename_count, 1);\n }\n \n static const char *get_basename(const char *filename)\n@@ -791,8 +802,10 @@ void diffcore_rename_extended(struct diff_options *options,\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct dir_rename_info info;\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n+\tinfo.setup = 0;\n \tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n@@ -985,7 +998,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/*\n \t * Now that renames have been computed, compute dir_rename_count */\n \tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n \n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n-- \ngitgitgadget\n\n"},{"id":"416870","messageId":"f7bdad78219de6819d0403f8957e9a0c8b4218bc.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 04/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:58:57Z","receivedAt":"2021-02-14T08:00:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen diffcore_rename_extended() is passed a NULL dir_rename_count, we\nwill still want to create a temporary one for use by\nfind_basename_matches(), but have it fully deallocated before\ndiffcore_rename_extended() returns. However, when\ndiffcore_rename_extended() is passed a dir_rename_count, we want to fill\nthat strmap with appropriate values and return it. However, for our\ninterim purposes we may also add entries corresponding to directories\nthat cannot have been renamed due to still existing on both sides.\n\nExtend cleanup_dir_rename_info() to handle these two different cases,\ncleaning up the relevant bits of information for each case.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 38 +++++++++++++++++++++++++++++++++++---\n 1 file changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 7759c9a3a2ed..aa21d4e7175c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -507,13 +507,45 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n }\n \n MAYBE_UNUSED\n-static void cleanup_dir_rename_info(struct dir_rename_info *info)\n+static void cleanup_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t int keep_dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n \tif (!info->setup)\n \t\treturn;\n \n-\tpartial_clear_dir_rename_count(info->dir_rename_count);\n-\tstrmap_clear(info->dir_rename_count, 1);\n+\tif (!keep_dir_rename_count) {\n+\t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\t\tstrmap_clear(info->dir_rename_count, 1);\n+\t\tFREE_AND_NULL(info->dir_rename_count);\n+\t} else {\n+\t\t/*\n+\t\t * Although dir_rename_count was passed in\n+\t\t * diffcore_rename_extended() and we want to keep it around and\n+\t\t * return it to that caller, we first want to remove any data\n+\t\t * associated with directories that weren't renamed.\n+\t\t */\n+\t\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n+\t\tint i;\n+\n+\t\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\t\tconst char *source_dir = entry->key;\n+\t\t\tstruct strintmap *counts = entry->value;\n+\n+\t\t\tif (!strset_contains(dirs_removed, source_dir)) {\n+\t\t\t\tstring_list_append(&to_remove, source_dir);\n+\t\t\t\tstrintmap_clear(counts);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tfor (i=0; idir_rename_count,\n+\t\t\t\t to_remove.items[i].string, 1);\n+\t\tstring_list_clear(&to_remove, 0);\n+\t}\n }\n \n static const char *get_basename(const char *filename)\n-- \ngitgitgadget\n\n"},{"id":"416871","messageId":"3a29cf9e526fba0227a7eec92c0c6bd58a7850f0.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 05/10] diffcore-rename: compute dir_rename_counts in stages","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:58:58Z","receivedAt":"2021-02-14T08:01:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to first compute dir_rename_counts based just on exact renames\nto start, as that can provide us useful information in\nfind_basename_matches(). That will give us an incomplete result, which\nwe can then later augment as basename and inexact rename matches are\nfound.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 76 ++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 62 insertions(+), 14 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex aa21d4e7175c..489e9cb0871e 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -411,6 +411,28 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tchar new_dir_first_char = new_dir[0];\n \tint first_time_in_loop = 1;\n \n+\tif (!info->setup)\n+\t\t/*\n+\t\t * info->setup is 0 here in two cases: (1) all auxiliary\n+\t\t * vars (like dirs_removed) were NULL so\n+\t\t * initialize_dir_rename_info() returned early, or (2)\n+\t\t * either break detection or copy detection are active so\n+\t\t * that we never called initialize_dir_rename_info(). In\n+\t\t * the former case, we don't have enough info to know if\n+\t\t * directories were renamed (because dirs_removed lets us\n+\t\t * know about a necessary prerequisite, namely if they were\n+\t\t * removed), and in the latter, we don't care about\n+\t\t * directory renames or find_basename_matches.\n+\t\t *\n+\t\t * This matters because both basename and inexact matching\n+\t\t * will also call update_dir_rename_counts(). In either of\n+\t\t * the above two cases info->dir_rename_counts will not\n+\t\t * have been properly initialized which prevents us from\n+\t\t * updating it, but in these two cases we don't care about\n+\t\t * dir_rename_counts anyway, so we can just exit early.\n+\t\t */\n+\t\treturn;\n+\n \twhile (1) {\n \t\tdirname_munge(old_dir);\n \t\tdirname_munge(new_dir);\n@@ -471,14 +493,22 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct dir_rename_info *info,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t struct strmap *dir_rename_count)\n+static void initialize_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n+\tinfo->setup = 0;\n+\tif (!dirs_removed)\n+\t\treturn;\n \tinfo->setup = 1;\n+\n \tinfo->dir_rename_count = dir_rename_count;\n+\tif (!info->dir_rename_count) {\n+\t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n+\t\tstrmap_init(info->dir_rename_count);\n+\t}\n \n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n@@ -506,7 +536,6 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n \tstrmap_partial_clear(dir_rename_count, 1);\n }\n \n-MAYBE_UNUSED\n static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t int keep_dir_rename_count)\n@@ -561,7 +590,9 @@ static const char *get_basename(const char *filename)\n }\n \n static int find_basename_matches(struct diff_options *options,\n-\t\t\t\t int minimum_score)\n+\t\t\t\t int minimum_score,\n+\t\t\t\t struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed)\n {\n \t/*\n \t * When I checked in early 2020, over 76% of file renames in linux\n@@ -669,6 +700,8 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\tcontinue;\n \t\t\trecord_rename_pair(dst_index, src_index, score);\n \t\t\trenames++;\n+\t\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t\t one->path, two->path);\n \n \t\t\t/*\n \t\t\t * Found a rename so don't need text anymore; if we\n@@ -752,7 +785,12 @@ static int too_many_rename_candidates(int num_destinations, int num_sources,\n \treturn 1;\n }\n \n-static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, int copies)\n+static int find_renames(struct diff_score *mx,\n+\t\t\tint dst_cnt,\n+\t\t\tint minimum_score,\n+\t\t\tint copies,\n+\t\t\tstruct dir_rename_info *info,\n+\t\t\tstruct strset *dirs_removed)\n {\n \tint count = 0, i;\n \n@@ -769,6 +807,9 @@ static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, i\n \t\t\tcontinue;\n \t\trecord_rename_pair(mx[i].dst, mx[i].src, mx[i].score);\n \t\tcount++;\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t rename_src[mx[i].src].p->one->path,\n+\t\t\t\t\t rename_dst[mx[i].dst].p->two->path);\n \t}\n \treturn count;\n }\n@@ -840,6 +881,8 @@ void diffcore_rename_extended(struct diff_options *options,\n \tinfo.setup = 0;\n \tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n+\tif (dirs_removed && (break_idx || want_copies))\n+\t\tBUG(\"dirs_removed incompatible with break/copy detection\");\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n \n@@ -931,10 +974,17 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tremove_unneeded_paths_from_src(want_copies);\n \t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n \n+\t\t/* Preparation for basename-driven matching. */\n+\t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n+\t\tinitialize_dir_rename_info(&info,\n+\t\t\t\t\t dirs_removed, dir_rename_count);\n+\t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n+\n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n-\t\t\t\t\t\t min_basename_score);\n+\t\t\t\t\t\t min_basename_score,\n+\t\t\t\t\t\t &info, dirs_removed);\n \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n \n \t\t/*\n@@ -1020,18 +1070,15 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/* cost matrix sorted by most to least similar pair */\n \tSTABLE_QSORT(mx, dst_cnt * NUM_CANDIDATE_PER_DST, score_compare);\n \n-\trename_count += find_renames(mx, dst_cnt, minimum_score, 0);\n+\trename_count += find_renames(mx, dst_cnt, minimum_score, 0,\n+\t\t\t\t &info, dirs_removed);\n \tif (want_copies)\n-\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1);\n+\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1,\n+\t\t\t\t\t &info, dirs_removed);\n \tfree(mx);\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n-\t/*\n-\t * Now that renames have been computed, compute dir_rename_count */\n-\tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n-\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n@@ -1103,6 +1150,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tif (rename_dst[i].filespec_to_free)\n \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n \n+\tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n \trename_dst_nr = rename_dst_alloc = 0;\n \tFREE_AND_NULL(rename_src);\n-- \ngitgitgadget\n\n"},{"id":"416872","messageId":"dffecc064dd36d9ba3885d947e544a2ce010a01d.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 06/10] diffcore-rename: add a mapping of destination names to their indices","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:58:59Z","receivedAt":"2021-02-14T08:01:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd an idx_map member to struct dir_rename_info, which tracks a mapping\nof the full filename to the index within rename_dst where that filename\nis found. We will later use this for quickly finding an array entry in\nrename_dst given the pathname.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 22 +++++++++++++++++++++-\n 1 file changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 489e9cb0871e..db569e4a0b0a 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -368,6 +368,7 @@ static int find_exact_renames(struct diff_options *options)\n }\n \n struct dir_rename_info {\n+\tstruct strintmap idx_map;\n \tstruct strmap *dir_rename_count;\n \tunsigned setup;\n };\n@@ -509,10 +510,26 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n \t\tstrmap_init(info->dir_rename_count);\n \t}\n+\tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \n+\t/*\n+\t * Loop setting up both info->idx_map, and doing setup of\n+\t * info->dir_rename_count.\n+\t */\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n+\t\t * For non-renamed files, make idx_map contain mapping of\n+\t\t * filename -> index (index within rename_dst, that is)\n+\t\t */\n+\t\tif (!rename_dst[i].is_rename) {\n+\t\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\t\tstrintmap_set(&info->idx_map, filename, i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * For everything else (i.e. renamed files), make\n+\t\t * dir_rename_count contain a map of a map:\n \t\t * old_directory -> {new_directory -> count}\n \t\t * In other words, for every pair look at the directories for\n \t\t * the old filename and the new filename and count how many\n@@ -546,6 +563,9 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \tif (!info->setup)\n \t\treturn;\n \n+\t/* idx_map */\n+\tstrintmap_clear(&info->idx_map);\n+\n \tif (!keep_dir_rename_count) {\n \t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n \t\tstrmap_clear(info->dir_rename_count, 1);\n-- \ngitgitgadget\n\n"},{"id":"416873","messageId":"cbd055ab33998962cb7712906cdad5dff3390660.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 08/10] diffcore-rename: add a new idx_possible_rename function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:59:01Z","receivedAt":"2021-02-14T08:01:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nfind_basename_matches() is great when both the remaining set of possible\nrename sources and the remaining set of possible rename destinations\nhave exactly one file each with a given basename. It allows us to match\nup files that have been moved to different directories without changing\nfilenames.\n\nWhen basenames are not unique, though, we want to be able to guess which\ndirectories the source files have been moved to. Since this is the job\nof directory rename detection, we employ it. However, since it is a\ndirectory rename detection idea, we also limit it to cases where we know\nthere could have been a directory rename, i.e. where the source\ndirectory has been removed. This has to be signalled by dirs_removed\nbeing non-NULL and containing an entry for the relevant directory.\nSince merge-ort.c is the only caller that currently does so, this\noptimization is only effective for merge-ort right now. In the future,\nthis condition could be reconsidered or we could modify other callers to\npass the necessary strset.\n\nAnyway, that's a lot of background so that we can actually describe the\nnew function. Add an idx_possible_rename() function which combines the\nrecently added dir_rename_guess and idx_map fields to provide the index\nwithin rename_dst of a potential match for a given file.\n\nFuture commits will add checks after calling this function to compare\nthe resulting 'likely rename' candidates to see if the two files meet\nthe elevated min_basename_score threshold for marking them as actual\nrenames.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 81 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex d24f104aa81c..1e4a56adde2c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -374,6 +374,12 @@ struct dir_rename_info {\n \tunsigned setup;\n };\n \n+static char *get_dirname(const char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\treturn slash ? xstrndup(filename, slash-filename) : xstrdup(\"\");\n+}\n+\n static void dirname_munge(char *filename)\n {\n \tchar *slash = strrchr(filename, '/');\n@@ -651,6 +657,81 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n+MAYBE_UNUSED\n+static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n+{\n+\t/*\n+\t * Our comparison of files with the same basename (see\n+\t * find_basename_matches() below), is only helpful when after exact\n+\t * rename detection we have exactly one file with a given basename\n+\t * among the rename sources and also only exactly one file with\n+\t * that basename among the rename destinations. When we have\n+\t * multiple files with the same basename in either set, we do not\n+\t * know which to compare against. However, there are some\n+\t * filenames that occur in large numbers (particularly\n+\t * build-related filenames such as 'Makefile', '.gitignore', or\n+\t * 'build.gradle' that potentially exist within every single\n+\t * subdirectory), and for performance we want to be able to quickly\n+\t * find renames for these files too.\n+\t *\n+\t * The reason basename comparisons are a useful heuristic was that it\n+\t * is common for people to move files across directories while keeping\n+\t * their filename the same. If we had a way of determining or even\n+\t * making a good educated guess about which directory these non-unique\n+\t * basename files had moved the file to, we could check it.\n+\t * Luckily...\n+\t *\n+\t * When an entire directory is in fact renamed, we have two factors\n+\t * helping us out:\n+\t * (a) the original directory disappeared giving us a hint\n+\t * about when we can apply an extra heuristic.\n+\t * (a) we often have several files within that directory and\n+\t * subdirectories that are renamed without changes\n+\t * So, rules for a heuristic:\n+\t * (0) If there basename matches are non-unique (the condition under\n+\t * which this function is called) AND\n+\t * (1) the directory in which the file was found has disappeared\n+\t * (i.e. dirs_removed is non-NULL and has a relevant entry) THEN\n+\t * (2) use exact renames of files within the directory to determine\n+\t * where the directory is likely to have been renamed to. IF\n+\t * there is at least one exact rename from within that\n+\t * directory, we can proceed.\n+\t * (3) If there are multiple places the directory could have been\n+\t * renamed to based on exact renames, ignore all but one of them.\n+\t * Just use the destination with the most renames going to it.\n+\t * (4) Check if applying that directory rename to the original file\n+\t * would result in a destination filename that is in the\n+\t * potential rename set. If so, return the index of the\n+\t * destination file (the index within rename_dst).\n+\t * (5) Compare the original file and returned destination for\n+\t * similarity, and if they are sufficiently similar, record the\n+\t * rename.\n+\t *\n+\t * This function, idx_possible_rename(), is only responsible for (4).\n+\t * The conditions/steps in (1)-(3) are handled via setting up\n+\t * dir_rename_count and dir_rename_guess in\n+\t * initialize_dir_rename_info(). Steps (0) and (5) are handled by\n+\t * the caller of this function.\n+\t */\n+\tchar *old_dir, *new_dir, *new_path;\n+\tint idx;\n+\n+\tif (!info->setup)\n+\t\treturn -1;\n+\n+\told_dir = get_dirname(filename);\n+\tnew_dir = strmap_get(&info->dir_rename_guess, old_dir);\n+\tfree(old_dir);\n+\tif (!new_dir)\n+\t\treturn -1;\n+\n+\tnew_path = xstrfmt(\"%s/%s\", new_dir, get_basename(filename));\n+\n+\tidx = strintmap_get(&info->idx_map, new_path);\n+\tfree(new_path);\n+\treturn idx;\n+}\n+\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t struct dir_rename_info *info,\n-- \ngitgitgadget\n\n"},{"id":"416874","messageId":"4983a1c2f908f02bc8a47f883c31652723ffde51.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 07/10] diffcore-rename: add a dir_rename_guess field to dir_rename_info","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:59:00Z","receivedAt":"2021-02-14T08:01:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndir_rename_counts has a mapping of a mapping, in particular, it has\n old_dir => { new_dir => count }\nWe want a simple mapping of\n old_dir => new_dir\nbased on which new_dir had the highest count for a given old_dir.\nIntroduce dir_rename_guess for this purpose.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 42 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex db569e4a0b0a..d24f104aa81c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -369,6 +369,7 @@ static int find_exact_renames(struct diff_options *options)\n \n struct dir_rename_info {\n \tstruct strintmap idx_map;\n+\tstruct strmap dir_rename_guess;\n \tstruct strmap *dir_rename_count;\n \tunsigned setup;\n };\n@@ -381,6 +382,24 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n+static const char *get_highest_rename_path(struct strintmap *counts)\n+{\n+\tint highest_count = 0;\n+\tconst char *highest_destination_dir = NULL;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrintmap_for_each_entry(counts, &iter, entry) {\n+\t\tconst char *destination_dir = entry->key;\n+\t\tintptr_t count = (intptr_t)entry->value;\n+\t\tif (count > highest_count) {\n+\t\t\thighest_count = count;\n+\t\t\thighest_destination_dir = destination_dir;\n+\t\t}\n+\t}\n+\treturn highest_destination_dir;\n+}\n+\n static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n@@ -498,6 +517,8 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t struct strmap *dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n \tint i;\n \n \tinfo->setup = 0;\n@@ -511,6 +532,7 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\tstrmap_init(info->dir_rename_count);\n \t}\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n+\tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n \n \t/*\n \t * Loop setting up both info->idx_map, and doing setup of\n@@ -539,6 +561,23 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n+\n+\t/*\n+\t * Now we collapse\n+\t * dir_rename_count: old_directory -> {new_directory -> count}\n+\t * down to\n+\t * dir_rename_guess: old_directory -> best_new_directory\n+\t * where best_new_directory is the one with the highest count.\n+\t */\n+\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\t/* entry->key is source_dir */\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tchar *best_newdir;\n+\n+\t\tbest_newdir = xstrdup(get_highest_rename_path(counts));\n+\t\tstrmap_put(&info->dir_rename_guess, entry->key,\n+\t\t\t best_newdir);\n+\t}\n }\n \n void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n@@ -566,6 +605,9 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \t/* idx_map */\n \tstrintmap_clear(&info->idx_map);\n \n+\t/* dir_rename_guess */\n+\tstrmap_clear(&info->dir_rename_guess, 1);\n+\n \tif (!keep_dir_rename_count) {\n \t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n \t\tstrmap_clear(info->dir_rename_count, 1);\n-- \ngitgitgadget\n\n"},{"id":"416875","messageId":"1df498b3a2f00cdcbe93acda1dc637e3fca0ee0b.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 10/10] diffcore-rename: use directory rename guided basename comparisons","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:59:03Z","receivedAt":"2021-02-14T08:02:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nHook the work from the last several patches together so that when\nbasenames in the sets of possible remaining rename sources or\ndestinations aren't unique, we can guess which directory source files\nwere renamed into. When that guess gives us a pairing of files, and\nthose files are sufficiently similar, we record the two files as a\nrename and remove them from the large matrix of comparisons for inexact\nrename detection.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n\nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 32 +++++++++++++++++++++++---------\n 2 files changed, 24 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex 80fcf9542441..8673a5c5b2f2 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -186,7 +186,7 @@ mark a file pair as a rename and stop considering other candidates for\n better matches. At most, one comparison is done per file in this\n preliminary pass; so if there are several remaining ext.txt files\n throughout the directory hierarchy after exact rename detection, this\n-preliminary step will be skipped for those files.\n+preliminary step may be skipped for those files.\n \n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 5de4497e04fa..70a484b9b63e 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -667,7 +667,6 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n-MAYBE_UNUSED\n static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n {\n \t/*\n@@ -780,8 +779,6 @@ static int find_basename_matches(struct diff_options *options,\n \tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n-\tstruct hashmap_iter iter;\n-\tstruct strmap_entry *entry;\n \n \t/*\n \t * The prefeteching stuff wants to know if it can skip prefetching\n@@ -831,17 +828,34 @@ static int find_basename_matches(struct diff_options *options,\n \t}\n \n \t/* Now look for basename matchups and do similarity estimation */\n-\tstrintmap_for_each_entry(&sources, &iter, entry) {\n-\t\tconst char *base = entry->key;\n-\t\tintptr_t src_index = (intptr_t)entry->value;\n+\tfor (i = 0; i < rename_src_nr; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base = NULL;\n+\t\tintptr_t src_index;\n \t\tintptr_t dst_index;\n-\t\tif (src_index == -1)\n-\t\t\tcontinue;\n \n-\t\tif (0 <= (dst_index = strintmap_get(&dests, base))) {\n+\t\t/* Is this basename unique among remaining sources? */\n+\t\tbase = get_basename(filename);\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tassert(src_index == -1 || src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n \t\t\tstruct diff_filespec *one, *two;\n \t\t\tint score;\n \n+\t\t\t/* Find a matching destination, if possible */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (src_index == -1 || dst_index == -1) {\n+\t\t\t\tsrc_index = i;\n+\t\t\t\tdst_index = idx_possible_rename(filename, info);\n+\t\t\t}\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n \t\t\t/* Estimate the similarity */\n \t\t\tone = rename_src[src_index].p->one;\n \t\t\ttwo = rename_dst[dst_index].p->two;\n-- \ngitgitgadget\n"},{"id":"416876","messageId":"4e095ea7c4390cb47828bbba50af876249983870.1613289544.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH 09/10] diffcore-rename: limit dir_rename_counts computation to relevant dirs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:59:02Z","receivedAt":"2021-02-14T08:02:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe are using dir_rename_counts to count the number of other directories\nthat files within a directory moved to. We only need this information\nfor directories that disappeared, though, so we can return early from\nupdate_dir_rename_counts() for other paths.\n\nWhile dirs_removed provides the relevant information for us right now,\nwe introduce a new info->relevant_source_dirs parameter because future\noptimizations will want to change how things are called somewhat.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 1e4a56adde2c..5de4497e04fa 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -371,6 +371,7 @@ struct dir_rename_info {\n \tstruct strintmap idx_map;\n \tstruct strmap dir_rename_guess;\n \tstruct strmap *dir_rename_count;\n+\tstruct strset *relevant_source_dirs;\n \tunsigned setup;\n };\n \n@@ -460,7 +461,13 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\treturn;\n \n \twhile (1) {\n+\t\t/* Get old_dir, skip if its directory isn't relevant. */\n \t\tdirname_munge(old_dir);\n+\t\tif (info->relevant_source_dirs &&\n+\t\t !strset_contains(info->relevant_source_dirs, old_dir))\n+\t\t\tbreak;\n+\n+\t\t/* Get new_dir */\n \t\tdirname_munge(new_dir);\n \n \t\t/*\n@@ -540,6 +547,9 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n \n+\t/* Setup info->relevant_source_dirs */\n+\tinfo->relevant_source_dirs = dirs_removed;\n+\n \t/*\n \t * Loop setting up both info->idx_map, and doing setup of\n \t * info->dir_rename_count.\n-- \ngitgitgadget\n\n"},{"id":"417630","messageId":"93f98fc0b2644aab9e98b7a32e88561618c7d4c0.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 03/10] diffcore-rename: move dir_rename_counts into a dir_rename_info struct","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:00Z","receivedAt":"2021-02-23T23:50:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis is a purely cosmetic change for now, but we will be adding\nadditional information to the struct and changing where and how it is\nsetup and used in subsequent patches.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 39 ++++++++++++++++++++++++++-------------\n 1 file changed, 26 insertions(+), 13 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 614a8d63012d..7759c9a3a2ed 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,11 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+struct dir_rename_info {\n+\tstruct strmap *dir_rename_count;\n+\tunsigned setup;\n+};\n+\n static void dirname_munge(char *filename)\n {\n \tchar *slash = strrchr(filename, '/');\n@@ -375,7 +380,7 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n-static void increment_count(struct strmap *dir_rename_count,\n+static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n {\n@@ -383,20 +388,20 @@ static void increment_count(struct strmap *dir_rename_count,\n \tstruct strmap_entry *e;\n \n \t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n+\te = strmap_get_entry(info->dir_rename_count, old_dir);\n \tif (e) {\n \t\tcounts = e->value;\n \t} else {\n \t\tcounts = xmalloc(sizeof(*counts));\n \t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t\tstrmap_put(info->dir_rename_count, old_dir, counts);\n \t}\n \n \t/* Increment the count for new_dir */\n \tstrintmap_incr(counts, new_dir, 1);\n }\n \n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t const char *oldname,\n \t\t\t\t const char *newname)\n@@ -450,7 +455,7 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t}\n \n \t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\t\tincrement_count(info, old_dir, new_dir);\n \t\telse\n \t\t\tbreak;\n \n@@ -466,12 +471,15 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n+static void compute_dir_rename_counts(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n-\t/* Set up dir_rename_count */\n+\tinfo->setup = 1;\n+\tinfo->dir_rename_count = dir_rename_count;\n+\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n \t\t * Make dir_rename_count contain a map of a map:\n@@ -480,7 +488,7 @@ static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t * the old filename and the new filename and count how many\n \t\t * times that pairing occurs.\n \t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n@@ -499,10 +507,13 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n }\n \n MAYBE_UNUSED\n-static void clear_dir_rename_count(struct strmap *dir_rename_count)\n+static void cleanup_dir_rename_info(struct dir_rename_info *info)\n {\n-\tpartial_clear_dir_rename_count(dir_rename_count);\n-\tstrmap_clear(dir_rename_count, 1);\n+\tif (!info->setup)\n+\t\treturn;\n+\n+\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\tstrmap_clear(info->dir_rename_count, 1);\n }\n \n static const char *get_basename(const char *filename)\n@@ -791,8 +802,10 @@ void diffcore_rename_extended(struct diff_options *options,\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct dir_rename_info info;\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n+\tinfo.setup = 0;\n \tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n@@ -985,7 +998,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/*\n \t * Now that renames have been computed, compute dir_rename_count */\n \tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n \n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n-- \ngitgitgadget\n\n"},{"id":"417631","messageId":"612da82f049cbe877cf924f743a3e4059483ce51.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 02/10] diffcore-rename: add functions for clearing dir_rename_count","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:43:59Z","receivedAt":"2021-02-23T23:52:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs we adjust the usage of dir_rename_count we want to have functions for\nclearing, or partially clearing it out. Add such functions.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 19 +++++++++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 12 +++---------\n 3 files changed, 24 insertions(+), 9 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 33cfc5848611..614a8d63012d 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -486,6 +486,25 @@ static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n \t}\n }\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrmap_for_each_entry(dir_rename_count, &iter, entry) {\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tstrintmap_clear(counts);\n+\t}\n+\tstrmap_partial_clear(dir_rename_count, 1);\n+}\n+\n+MAYBE_UNUSED\n+static void clear_dir_rename_count(struct strmap *dir_rename_count)\n+{\n+\tpartial_clear_dir_rename_count(dir_rename_count);\n+\tstrmap_clear(dir_rename_count, 1);\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\ndiff --git a/diffcore.h b/diffcore.h\nindex db55d3853071..c6ba64abd198 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -161,6 +161,8 @@ struct diff_filepair *diff_queue(struct diff_queue_struct *,\n \t\t\t\t struct diff_filespec *);\n void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n+\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex c4467e073b45..467404cc0a35 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -351,17 +351,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \n \t/* Free memory used by various renames maps */\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstruct hashmap_iter iter;\n-\t\tstruct strmap_entry *entry;\n-\n \t\tstrset_func(&renames->dirs_removed[i]);\n \n-\t\tstrmap_for_each_entry(&renames->dir_rename_count[i],\n-\t\t\t\t &iter, entry) {\n-\t\t\tstruct strintmap *counts = entry->value;\n-\t\t\tstrintmap_clear(counts);\n-\t\t}\n-\t\tstrmap_func(&renames->dir_rename_count[i], 1);\n+\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n+\t\tif (!reinitialize)\n+\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \n \t\tstrmap_func(&renames->dir_renames[i], 0);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"417632","messageId":"fec4f1d44c0694cf92c9b98f2b6519a15fb78188.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 01/10] Move computation of dir_rename_count from merge-ort to diffcore-rename","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:43:58Z","receivedAt":"2021-02-23T23:52:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA previous commit noted that it is very common for people to move files\nacross directories while keeping their filename the same. The last few\ncommits took advantage of this and showed that we can accelerate rename\ndetection significantly using basenames; since files with the same\nbasename serve as likely rename candidates, we can check those first and\nremove them from the rename candidate pool if they are sufficiently\nsimilar.\n\nUnfortunately, the previous optimization was limited by the fact that\nthe remaining basenames after exact rename detection are not always\nunique. Many repositories have hundreds of build files with the same\nname (e.g. Makefile, .gitignore, build.gradle, etc.), and may even have\nhundreds of source files with the same name. (For example, the linux\nkernel has 100 setup.c, 87 irq.c, and 112 core.c files. A repository at\n$DAYJOB has a lot of ObjectFactory.java and Plugin.java files).\n\nFor these files with non-unique basenames, we are faced with the task of\nattempting to determine or guess which directory they may have been\nrelocated to. Such a task is precisely the job of directory rename\ndetection. However, there are two catches: (1) the directory rename\ndetection code has traditionally been part of the merge machinery rather\nthan diffcore-rename.c, and (2) directory rename detection currently\nruns after regular rename detection is complete. The 1st catch is just\nan implementation issue that can be overcome by some code shuffling.\nThe 2nd requires us to add a further approximation: we only have access\nto exact renames at this point, so we need to do directory rename\ndetection based on just exact renames. In some cases we won't have\nexact renames, in which case this extra optimization won't apply. We\nalso choose to not apply the optimization unless we know that the\nunderlying directory was removed, which will require extra data to be\npassed in to diffcore_rename_extended(). Also, even if we get a\nprediction about which directory a file may have relocated to, we will\nstill need to check to see if there is a file in the predicted\ndirectory, and then compare the two files to see if they meet the higher\nmin_basename_score threshold required for marking the two files as\nrenames.\n\nThis commit and the next few will set up the necessary infrastructure to\ndo such computations. This commit merely moves the computation of\ndir_rename_count from merge-ort.c to diffcore-rename.c, making slight\nadjustments to the data structures based on the move. While the\ndiffstat looks large, viewing this commit with --color-moved makes it\nclear that only about 20 lines changed. With this patch, the\ncomputation of dir_rename_count is still only done after inexact rename\ndetection, but subsequent commits will add a preliminary computation of\ndir_rename_count after exact rename detection, followed by some updates\nafter inexact rename detection.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 134 +++++++++++++++++++++++++++++++++++++++++++++-\n diffcore.h | 5 ++\n merge-ort.c | 132 ++-------------------------------------------\n 3 files changed, 141 insertions(+), 130 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 41558185ae1d..33cfc5848611 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,125 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+static void dirname_munge(char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\tif (!slash)\n+\t\tslash = filename;\n+\t*slash = '\\0';\n+}\n+\n+static void increment_count(struct strmap *dir_rename_count,\n+\t\t\t char *old_dir,\n+\t\t\t char *new_dir)\n+{\n+\tstruct strintmap *counts;\n+\tstruct strmap_entry *e;\n+\n+\t/* Get the {new_dirs -> counts} mapping using old_dir */\n+\te = strmap_get_entry(dir_rename_count, old_dir);\n+\tif (e) {\n+\t\tcounts = e->value;\n+\t} else {\n+\t\tcounts = xmalloc(sizeof(*counts));\n+\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n+\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t}\n+\n+\t/* Increment the count for new_dir */\n+\tstrintmap_incr(counts, new_dir, 1);\n+}\n+\n+static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t const char *oldname,\n+\t\t\t\t const char *newname)\n+{\n+\tchar *old_dir = xstrdup(oldname);\n+\tchar *new_dir = xstrdup(newname);\n+\tchar new_dir_first_char = new_dir[0];\n+\tint first_time_in_loop = 1;\n+\n+\twhile (1) {\n+\t\tdirname_munge(old_dir);\n+\t\tdirname_munge(new_dir);\n+\n+\t\t/*\n+\t\t * When renaming\n+\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n+\t\t * then this suggests that both\n+\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n+\t\t * a/b/c/d/ => a/b/some/thing/else/\n+\t\t * so we want to increment counters for both. We do NOT,\n+\t\t * however, also want to suggest that there was the following\n+\t\t * rename:\n+\t\t * a/b/c/ => a/b/some/thing/\n+\t\t * so we need to quit at that point.\n+\t\t *\n+\t\t * Note the when first_time_in_loop, we only strip off the\n+\t\t * basename, and we don't care if that's different.\n+\t\t */\n+\t\tif (!first_time_in_loop) {\n+\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n+\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n+\t\t\tif (!*new_dir) {\n+\t\t\t\t/*\n+\t\t\t\t * Special case when renaming to root directory,\n+\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n+\t\t\t\t * something like\n+\t\t\t\t * a/b/subdir => subdir\n+\t\t\t\t * and so dirname_munge() sets things up so that\n+\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n+\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n+\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n+\t\t\t\t * in new_dir, so we have to compare differently.\n+\t\t\t\t */\n+\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n+\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t} else {\n+\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strset_contains(dirs_removed, old_dir))\n+\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\telse\n+\t\t\tbreak;\n+\n+\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n+\t\tif (!*old_dir || !*new_dir)\n+\t\t\tbreak;\n+\n+\t\tfirst_time_in_loop = 0;\n+\t}\n+\n+\t/* Free resources we don't need anymore */\n+\tfree(old_dir);\n+\tfree(new_dir);\n+}\n+\n+static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed)\n+{\n+\tint i;\n+\n+\t/* Set up dir_rename_count */\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\t/*\n+\t\t * Make dir_rename_count contain a map of a map:\n+\t\t * old_directory -> {new_directory -> count}\n+\t\t * In other words, for every pair look at the directories for\n+\t\t * the old filename and the new filename and count how many\n+\t\t * times that pairing occurs.\n+\t\t */\n+\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\t\t\t\t rename_dst[i].p->one->path,\n+\t\t\t\t\t rename_dst[i].p->two->path);\n+\t}\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\n@@ -640,7 +759,9 @@ static void remove_unneeded_paths_from_src(int detecting_copies)\n \trename_src_nr = new_num_src;\n }\n \n-void diffcore_rename(struct diff_options *options)\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count)\n {\n \tint detect_rename = options->detect_rename;\n \tint minimum_score = options->rename_score;\n@@ -653,6 +774,7 @@ void diffcore_rename(struct diff_options *options)\n \tstruct progress *progress = NULL;\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n+\tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n@@ -841,6 +963,11 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n+\t/*\n+\t * Now that renames have been computed, compute dir_rename_count */\n+\tif (dirs_removed && dir_rename_count)\n+\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n@@ -923,3 +1050,8 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"write back to queue\", options->repo);\n \treturn;\n }\n+\n+void diffcore_rename(struct diff_options *options)\n+{\n+\tdiffcore_rename_extended(options, NULL, NULL);\n+}\ndiff --git a/diffcore.h b/diffcore.h\nindex d2a63c5c71f4..db55d3853071 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -8,6 +8,8 @@\n \n struct diff_options;\n struct repository;\n+struct strmap;\n+struct strset;\n struct userdiff_driver;\n \n /* This header file is internal between diff.c and its diff transformers\n@@ -161,6 +163,9 @@ void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count);\n void diffcore_merge_broken(void);\n void diffcore_pickaxe(struct diff_options *);\n void diffcore_order(const char *orderfile);\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 603d30c52170..c4467e073b45 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1302,131 +1302,6 @@ static char *handle_path_level_conflicts(struct merge_options *opt,\n \treturn new_path;\n }\n \n-static void dirname_munge(char *filename)\n-{\n-\tchar *slash = strrchr(filename, '/');\n-\tif (!slash)\n-\t\tslash = filename;\n-\t*slash = '\\0';\n-}\n-\n-static void increment_count(struct strmap *dir_rename_count,\n-\t\t\t char *old_dir,\n-\t\t\t char *new_dir)\n-{\n-\tstruct strintmap *counts;\n-\tstruct strmap_entry *e;\n-\n-\t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n-\tif (e) {\n-\t\tcounts = e->value;\n-\t} else {\n-\t\tcounts = xmalloc(sizeof(*counts));\n-\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n-\t}\n-\n-\t/* Increment the count for new_dir */\n-\tstrintmap_incr(counts, new_dir, 1);\n-}\n-\n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t const char *oldname,\n-\t\t\t\t const char *newname)\n-{\n-\tchar *old_dir = xstrdup(oldname);\n-\tchar *new_dir = xstrdup(newname);\n-\tchar new_dir_first_char = new_dir[0];\n-\tint first_time_in_loop = 1;\n-\n-\twhile (1) {\n-\t\tdirname_munge(old_dir);\n-\t\tdirname_munge(new_dir);\n-\n-\t\t/*\n-\t\t * When renaming\n-\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n-\t\t * then this suggests that both\n-\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n-\t\t * a/b/c/d/ => a/b/some/thing/else/\n-\t\t * so we want to increment counters for both. We do NOT,\n-\t\t * however, also want to suggest that there was the following\n-\t\t * rename:\n-\t\t * a/b/c/ => a/b/some/thing/\n-\t\t * so we need to quit at that point.\n-\t\t *\n-\t\t * Note the when first_time_in_loop, we only strip off the\n-\t\t * basename, and we don't care if that's different.\n-\t\t */\n-\t\tif (!first_time_in_loop) {\n-\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n-\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n-\t\t\tif (!*new_dir) {\n-\t\t\t\t/*\n-\t\t\t\t * Special case when renaming to root directory,\n-\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n-\t\t\t\t * something like\n-\t\t\t\t * a/b/subdir => subdir\n-\t\t\t\t * and so dirname_munge() sets things up so that\n-\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n-\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n-\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n-\t\t\t\t * in new_dir, so we have to compare differently.\n-\t\t\t\t */\n-\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n-\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t} else {\n-\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n-\t\telse\n-\t\t\tbreak;\n-\n-\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n-\t\tif (!*old_dir || !*new_dir)\n-\t\t\tbreak;\n-\n-\t\tfirst_time_in_loop = 0;\n-\t}\n-\n-\t/* Free resources we don't need anymore */\n-\tfree(old_dir);\n-\tfree(new_dir);\n-}\n-\n-static void compute_rename_counts(struct diff_queue_struct *pairs,\n-\t\t\t\t struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pairs->nr; ++i) {\n-\t\tstruct diff_filepair *pair = pairs->queue[i];\n-\n-\t\t/* File not part of directory rename if it wasn't renamed */\n-\t\tif (pair->status != 'R')\n-\t\t\tcontinue;\n-\n-\t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n-\t\t * old_directory -> {new_directory -> count}\n-\t\t * In other words, for every pair look at the directories for\n-\t\t * the old filename and the new filename and count how many\n-\t\t * times that pairing occurs.\n-\t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n-\t\t\t\t\t pair->one->path,\n-\t\t\t\t\t pair->two->path);\n-\t}\n-}\n-\n static void get_provisional_directory_renames(struct merge_options *opt,\n \t\t\t\t\t unsigned side,\n \t\t\t\t\t int *clean)\n@@ -1435,9 +1310,6 @@ static void get_provisional_directory_renames(struct merge_options *opt,\n \tstruct strmap_entry *entry;\n \tstruct rename_info *renames = &opt->priv->renames;\n \n-\tcompute_rename_counts(&renames->pairs[side],\n-\t\t\t &renames->dir_rename_count[side],\n-\t\t\t &renames->dirs_removed[side]);\n \t/*\n \t * Collapse\n \t * dir_rename_count: old_directory -> {new_directory -> count}\n@@ -2162,7 +2034,9 @@ static void detect_regular_renames(struct merge_options *opt,\n \n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n-\tdiffcore_rename(&diff_opts);\n+\tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t &renames->dirs_removed[side_index],\n+\t\t\t\t &renames->dir_rename_count[side_index]);\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n \tresolve_diffpair_statuses(&diff_queued_diff);\n \n-- \ngitgitgadget\n\n"},{"id":"417633","messageId":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.git.1613289544.gitgitgadget@gmail.com","subject":"[PATCH v2 00/10] Optimization batch 8: use file basenames even more","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:43:57Z","receivedAt":"2021-02-23T23:56:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on en/diffcore-rename (a concatenation of what I was\ncalling ort-perf-batch-6 and ort-perf-batch-7).\n\nThere are no changes since v1; it's just a resend a week and a half later to\nbump it so it isn't lost.\n\n=== Optimization idea ===\n\nThis series uses file basenames (portions of the path after the last '/',\nincluding file extension) in a more involved fashion to guide rename\ndetection. It's a follow-on improvement to \"Optimization #3\" from my Git\nMerge 2020 talk[1]. The basic idea behind this series is the same as the\nlast series: people frequently move files across directories while keeping\nthe filenames the same, thus files with the same basename are likely rename\ncandidates. However, the previous optimization only applies when basenames\nare unique among remaining adds and deletes after exact rename detection, so\nwe need to do something else to match up the remaining basenames. When there\nare many files with the same basename (e.g. .gitignore, Makefile,\nbuild.gradle, or maybe even setup.c, AbtractFactory.java, etc.), being able\nto \"guess\" which directory a given file likely would have moved to can\nprovide us with a likely rename candidate if there is a file with the same\nbasename in that directory. Since exact rename detection is done first, we\ncan use nearby exact renames to help us guess where any given non-unique\nbasename file may have moved; it just means doing \"directory rename\ndetection\" limited to exact renames.\n\nThere are definitely cases when this strategy still won't help us: (1) We\nonly use this strategy when the directory in which the original file was\nfound has also been removed, (2) a lack of exact renames from the given\ndirectory will prevents us from making a new directory prediction, (3) even\nif we predict a new directory there may be no file with the given basename\nin it, and (4) even if there is an unmatched add with the appropriate\nbasename in the predicted directory, it may not meet the higher\nmin_basename_score similarity threshold.\n\nIt may be worth noting that directory rename detection at most predicts one\nnew directory, which we use to ensure that we only compare any given file\nwith at most one other file. That's important for compatibility with future\noptimizations.\n\nHowever, despite the caveats and limited applicability, this idea provides\nsome nice speedups.\n\n=== Results ===\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28), the\nchanges in just this series improves the performance as follows:\n\n Before Series After Series\nno-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\nmega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\njust-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with (as noted in the same commit message)\nwere:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\n=== Alternative, rejected idea ===\n\nThere was an alternative idea to the series presented here that I also\ntried: instead of using directory rename detection based on exact renames to\npredict where files would be renamed and then comparing to the file with the\nsame basename in the new directory, one could instead take all files with\nthe same basename -- both sources and destinations -- and then do a smaller\nM x N comparison on all those files to find renames. Any non-matches after\nthat step could be combined with all other files for the big inexact rename\ndetection step.\n\nThere are two problems with such a strategy, though.\n\nOne is that in the worst case, you approximately double the cost of rename\ndetection (if most potential rename pairs all have the same basename but\nthey aren't actually matches, you end up comparing twice).\n\nThe second issue isn't clear until trying to combine this idea with later\nperformance optimizations. The next optimization will provide a way to\nfilter out several of the rename sources. If our inexact rename detection\nmatrix is sized 1 x 4000 because we can remove all but one source file, but\nwe have 100 files with the same basename, then a 100 x 100 comparison is\nactually more costly than a 1 x 4000 comparison -- and we don't need most of\nthe renames from the 100 x 100 comparison. The advantage of the directory\nrename detection based idea for finding which basenames to match up, is that\nthe cost for each file is linear (or, said another way, scales proportional\nto doing a diff on that file). As such, the costs for this preliminary\noptimization are nicely controlled and the worst case scenario is it has\nspent a little extra time upfront but still has to do the full inexact\nrename detection.\n\n[1]\nhttps://github.com/newren/presentations/blob/pdfs/merge-performance/merge-performance-slides.pdf\n\nElijah Newren (10):\n Move computation of dir_rename_count from merge-ort to diffcore-rename\n diffcore-rename: add functions for clearing dir_rename_count\n diffcore-rename: move dir_rename_counts into a dir_rename_info struct\n diffcore-rename: extend cleanup_dir_rename_info()\n diffcore-rename: compute dir_rename_counts in stages\n diffcore-rename: add a mapping of destination names to their indices\n diffcore-rename: add a dir_rename_guess field to dir_rename_info\n diffcore-rename: add a new idx_possible_rename function\n diffcore-rename: limit dir_rename_counts computation to relevant dirs\n diffcore-rename: use directory rename guided basename comparisons\n\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 439 ++++++++++++++++++++++++++++++++--\n diffcore.h | 7 +\n merge-ort.c | 144 +----------\n 4 files changed, 439 insertions(+), 153 deletions(-)\n\n\nbase-commit: aeca14f748afc7fb5b65bca56ea2ebd970729814\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-844%2Fnewren%2Fort-perf-batch-8-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-844/newren/ort-perf-batch-8-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/844\n\nRange-diff vs v1:\n\n 1: fec4f1d44c06 = 1: fec4f1d44c06 Move computation of dir_rename_count from merge-ort to diffcore-rename\n 2: 612da82f049c = 2: 612da82f049c diffcore-rename: add functions for clearing dir_rename_count\n 3: 93f98fc0b264 = 3: 93f98fc0b264 diffcore-rename: move dir_rename_counts into a dir_rename_info struct\n 4: f7bdad78219d = 4: f7bdad78219d diffcore-rename: extend cleanup_dir_rename_info()\n 5: 3a29cf9e526f = 5: 3a29cf9e526f diffcore-rename: compute dir_rename_counts in stages\n 6: dffecc064dd3 = 6: dffecc064dd3 diffcore-rename: add a mapping of destination names to their indices\n 7: 4983a1c2f908 = 7: 4983a1c2f908 diffcore-rename: add a dir_rename_guess field to dir_rename_info\n 8: cbd055ab3399 = 8: cbd055ab3399 diffcore-rename: add a new idx_possible_rename function\n 9: 4e095ea7c439 = 9: 4e095ea7c439 diffcore-rename: limit dir_rename_counts computation to relevant dirs\n 10: 1df498b3a2f0 = 10: 805c101cfd84 diffcore-rename: use directory rename guided basename comparisons\n\n-- \ngitgitgadget\n"},{"id":"417634","messageId":"f7bdad78219de6819d0403f8957e9a0c8b4218bc.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 04/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:01Z","receivedAt":"2021-02-23T23:56:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen diffcore_rename_extended() is passed a NULL dir_rename_count, we\nwill still want to create a temporary one for use by\nfind_basename_matches(), but have it fully deallocated before\ndiffcore_rename_extended() returns. However, when\ndiffcore_rename_extended() is passed a dir_rename_count, we want to fill\nthat strmap with appropriate values and return it. However, for our\ninterim purposes we may also add entries corresponding to directories\nthat cannot have been renamed due to still existing on both sides.\n\nExtend cleanup_dir_rename_info() to handle these two different cases,\ncleaning up the relevant bits of information for each case.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 38 +++++++++++++++++++++++++++++++++++---\n 1 file changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 7759c9a3a2ed..aa21d4e7175c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -507,13 +507,45 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n }\n \n MAYBE_UNUSED\n-static void cleanup_dir_rename_info(struct dir_rename_info *info)\n+static void cleanup_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t int keep_dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n \tif (!info->setup)\n \t\treturn;\n \n-\tpartial_clear_dir_rename_count(info->dir_rename_count);\n-\tstrmap_clear(info->dir_rename_count, 1);\n+\tif (!keep_dir_rename_count) {\n+\t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\t\tstrmap_clear(info->dir_rename_count, 1);\n+\t\tFREE_AND_NULL(info->dir_rename_count);\n+\t} else {\n+\t\t/*\n+\t\t * Although dir_rename_count was passed in\n+\t\t * diffcore_rename_extended() and we want to keep it around and\n+\t\t * return it to that caller, we first want to remove any data\n+\t\t * associated with directories that weren't renamed.\n+\t\t */\n+\t\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n+\t\tint i;\n+\n+\t\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\t\tconst char *source_dir = entry->key;\n+\t\t\tstruct strintmap *counts = entry->value;\n+\n+\t\t\tif (!strset_contains(dirs_removed, source_dir)) {\n+\t\t\t\tstring_list_append(&to_remove, source_dir);\n+\t\t\t\tstrintmap_clear(counts);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tfor (i=0; idir_rename_count,\n+\t\t\t\t to_remove.items[i].string, 1);\n+\t\tstring_list_clear(&to_remove, 0);\n+\t}\n }\n \n static const char *get_basename(const char *filename)\n-- \ngitgitgadget\n\n"},{"id":"417636","messageId":"cbd055ab33998962cb7712906cdad5dff3390660.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 08/10] diffcore-rename: add a new idx_possible_rename function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:05Z","receivedAt":"2021-02-23T23:56:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nfind_basename_matches() is great when both the remaining set of possible\nrename sources and the remaining set of possible rename destinations\nhave exactly one file each with a given basename. It allows us to match\nup files that have been moved to different directories without changing\nfilenames.\n\nWhen basenames are not unique, though, we want to be able to guess which\ndirectories the source files have been moved to. Since this is the job\nof directory rename detection, we employ it. However, since it is a\ndirectory rename detection idea, we also limit it to cases where we know\nthere could have been a directory rename, i.e. where the source\ndirectory has been removed. This has to be signalled by dirs_removed\nbeing non-NULL and containing an entry for the relevant directory.\nSince merge-ort.c is the only caller that currently does so, this\noptimization is only effective for merge-ort right now. In the future,\nthis condition could be reconsidered or we could modify other callers to\npass the necessary strset.\n\nAnyway, that's a lot of background so that we can actually describe the\nnew function. Add an idx_possible_rename() function which combines the\nrecently added dir_rename_guess and idx_map fields to provide the index\nwithin rename_dst of a potential match for a given file.\n\nFuture commits will add checks after calling this function to compare\nthe resulting 'likely rename' candidates to see if the two files meet\nthe elevated min_basename_score threshold for marking them as actual\nrenames.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 81 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex d24f104aa81c..1e4a56adde2c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -374,6 +374,12 @@ struct dir_rename_info {\n \tunsigned setup;\n };\n \n+static char *get_dirname(const char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\treturn slash ? xstrndup(filename, slash-filename) : xstrdup(\"\");\n+}\n+\n static void dirname_munge(char *filename)\n {\n \tchar *slash = strrchr(filename, '/');\n@@ -651,6 +657,81 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n+MAYBE_UNUSED\n+static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n+{\n+\t/*\n+\t * Our comparison of files with the same basename (see\n+\t * find_basename_matches() below), is only helpful when after exact\n+\t * rename detection we have exactly one file with a given basename\n+\t * among the rename sources and also only exactly one file with\n+\t * that basename among the rename destinations. When we have\n+\t * multiple files with the same basename in either set, we do not\n+\t * know which to compare against. However, there are some\n+\t * filenames that occur in large numbers (particularly\n+\t * build-related filenames such as 'Makefile', '.gitignore', or\n+\t * 'build.gradle' that potentially exist within every single\n+\t * subdirectory), and for performance we want to be able to quickly\n+\t * find renames for these files too.\n+\t *\n+\t * The reason basename comparisons are a useful heuristic was that it\n+\t * is common for people to move files across directories while keeping\n+\t * their filename the same. If we had a way of determining or even\n+\t * making a good educated guess about which directory these non-unique\n+\t * basename files had moved the file to, we could check it.\n+\t * Luckily...\n+\t *\n+\t * When an entire directory is in fact renamed, we have two factors\n+\t * helping us out:\n+\t * (a) the original directory disappeared giving us a hint\n+\t * about when we can apply an extra heuristic.\n+\t * (a) we often have several files within that directory and\n+\t * subdirectories that are renamed without changes\n+\t * So, rules for a heuristic:\n+\t * (0) If there basename matches are non-unique (the condition under\n+\t * which this function is called) AND\n+\t * (1) the directory in which the file was found has disappeared\n+\t * (i.e. dirs_removed is non-NULL and has a relevant entry) THEN\n+\t * (2) use exact renames of files within the directory to determine\n+\t * where the directory is likely to have been renamed to. IF\n+\t * there is at least one exact rename from within that\n+\t * directory, we can proceed.\n+\t * (3) If there are multiple places the directory could have been\n+\t * renamed to based on exact renames, ignore all but one of them.\n+\t * Just use the destination with the most renames going to it.\n+\t * (4) Check if applying that directory rename to the original file\n+\t * would result in a destination filename that is in the\n+\t * potential rename set. If so, return the index of the\n+\t * destination file (the index within rename_dst).\n+\t * (5) Compare the original file and returned destination for\n+\t * similarity, and if they are sufficiently similar, record the\n+\t * rename.\n+\t *\n+\t * This function, idx_possible_rename(), is only responsible for (4).\n+\t * The conditions/steps in (1)-(3) are handled via setting up\n+\t * dir_rename_count and dir_rename_guess in\n+\t * initialize_dir_rename_info(). Steps (0) and (5) are handled by\n+\t * the caller of this function.\n+\t */\n+\tchar *old_dir, *new_dir, *new_path;\n+\tint idx;\n+\n+\tif (!info->setup)\n+\t\treturn -1;\n+\n+\told_dir = get_dirname(filename);\n+\tnew_dir = strmap_get(&info->dir_rename_guess, old_dir);\n+\tfree(old_dir);\n+\tif (!new_dir)\n+\t\treturn -1;\n+\n+\tnew_path = xstrfmt(\"%s/%s\", new_dir, get_basename(filename));\n+\n+\tidx = strintmap_get(&info->idx_map, new_path);\n+\tfree(new_path);\n+\treturn idx;\n+}\n+\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t struct dir_rename_info *info,\n-- \ngitgitgadget\n\n"},{"id":"417635","messageId":"dffecc064dd36d9ba3885d947e544a2ce010a01d.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 06/10] diffcore-rename: add a mapping of destination names to their indices","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:03Z","receivedAt":"2021-02-23T23:56:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd an idx_map member to struct dir_rename_info, which tracks a mapping\nof the full filename to the index within rename_dst where that filename\nis found. We will later use this for quickly finding an array entry in\nrename_dst given the pathname.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 22 +++++++++++++++++++++-\n 1 file changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 489e9cb0871e..db569e4a0b0a 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -368,6 +368,7 @@ static int find_exact_renames(struct diff_options *options)\n }\n \n struct dir_rename_info {\n+\tstruct strintmap idx_map;\n \tstruct strmap *dir_rename_count;\n \tunsigned setup;\n };\n@@ -509,10 +510,26 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n \t\tstrmap_init(info->dir_rename_count);\n \t}\n+\tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \n+\t/*\n+\t * Loop setting up both info->idx_map, and doing setup of\n+\t * info->dir_rename_count.\n+\t */\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n+\t\t * For non-renamed files, make idx_map contain mapping of\n+\t\t * filename -> index (index within rename_dst, that is)\n+\t\t */\n+\t\tif (!rename_dst[i].is_rename) {\n+\t\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\t\tstrintmap_set(&info->idx_map, filename, i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * For everything else (i.e. renamed files), make\n+\t\t * dir_rename_count contain a map of a map:\n \t\t * old_directory -> {new_directory -> count}\n \t\t * In other words, for every pair look at the directories for\n \t\t * the old filename and the new filename and count how many\n@@ -546,6 +563,9 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \tif (!info->setup)\n \t\treturn;\n \n+\t/* idx_map */\n+\tstrintmap_clear(&info->idx_map);\n+\n \tif (!keep_dir_rename_count) {\n \t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n \t\tstrmap_clear(info->dir_rename_count, 1);\n-- \ngitgitgadget\n\n"},{"id":"417638","messageId":"805c101cfd849db3a5defb30775c7abbfec99f68.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 10/10] diffcore-rename: use directory rename guided basename comparisons","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:07Z","receivedAt":"2021-02-24T00:00:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nHook the work from the last several patches together so that when\nbasenames in the sets of possible remaining rename sources or\ndestinations aren't unique, we can guess which directory source files\nwere renamed into. When that guess gives us a pairing of files, and\nthose files are sufficiently similar, we record the two files as a\nrename and remove them from the large matrix of comparisons for inexact\nrename detection.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n\nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 32 +++++++++++++++++++++++---------\n 2 files changed, 24 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex 80fcf9542441..8673a5c5b2f2 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -186,7 +186,7 @@ mark a file pair as a rename and stop considering other candidates for\n better matches. At most, one comparison is done per file in this\n preliminary pass; so if there are several remaining ext.txt files\n throughout the directory hierarchy after exact rename detection, this\n-preliminary step will be skipped for those files.\n+preliminary step may be skipped for those files.\n \n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 5de4497e04fa..70a484b9b63e 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -667,7 +667,6 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n-MAYBE_UNUSED\n static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n {\n \t/*\n@@ -780,8 +779,6 @@ static int find_basename_matches(struct diff_options *options,\n \tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n-\tstruct hashmap_iter iter;\n-\tstruct strmap_entry *entry;\n \n \t/*\n \t * The prefeteching stuff wants to know if it can skip prefetching\n@@ -831,17 +828,34 @@ static int find_basename_matches(struct diff_options *options,\n \t}\n \n \t/* Now look for basename matchups and do similarity estimation */\n-\tstrintmap_for_each_entry(&sources, &iter, entry) {\n-\t\tconst char *base = entry->key;\n-\t\tintptr_t src_index = (intptr_t)entry->value;\n+\tfor (i = 0; i < rename_src_nr; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base = NULL;\n+\t\tintptr_t src_index;\n \t\tintptr_t dst_index;\n-\t\tif (src_index == -1)\n-\t\t\tcontinue;\n \n-\t\tif (0 <= (dst_index = strintmap_get(&dests, base))) {\n+\t\t/* Is this basename unique among remaining sources? */\n+\t\tbase = get_basename(filename);\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tassert(src_index == -1 || src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n \t\t\tstruct diff_filespec *one, *two;\n \t\t\tint score;\n \n+\t\t\t/* Find a matching destination, if possible */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (src_index == -1 || dst_index == -1) {\n+\t\t\t\tsrc_index = i;\n+\t\t\t\tdst_index = idx_possible_rename(filename, info);\n+\t\t\t}\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n \t\t\t/* Estimate the similarity */\n \t\t\tone = rename_src[src_index].p->one;\n \t\t\ttwo = rename_dst[dst_index].p->two;\n-- \ngitgitgadget\n"},{"id":"417639","messageId":"4e095ea7c4390cb47828bbba50af876249983870.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 09/10] diffcore-rename: limit dir_rename_counts computation to relevant dirs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:06Z","receivedAt":"2021-02-24T00:00:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe are using dir_rename_counts to count the number of other directories\nthat files within a directory moved to. We only need this information\nfor directories that disappeared, though, so we can return early from\nupdate_dir_rename_counts() for other paths.\n\nWhile dirs_removed provides the relevant information for us right now,\nwe introduce a new info->relevant_source_dirs parameter because future\noptimizations will want to change how things are called somewhat.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 1e4a56adde2c..5de4497e04fa 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -371,6 +371,7 @@ struct dir_rename_info {\n \tstruct strintmap idx_map;\n \tstruct strmap dir_rename_guess;\n \tstruct strmap *dir_rename_count;\n+\tstruct strset *relevant_source_dirs;\n \tunsigned setup;\n };\n \n@@ -460,7 +461,13 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\treturn;\n \n \twhile (1) {\n+\t\t/* Get old_dir, skip if its directory isn't relevant. */\n \t\tdirname_munge(old_dir);\n+\t\tif (info->relevant_source_dirs &&\n+\t\t !strset_contains(info->relevant_source_dirs, old_dir))\n+\t\t\tbreak;\n+\n+\t\t/* Get new_dir */\n \t\tdirname_munge(new_dir);\n \n \t\t/*\n@@ -540,6 +547,9 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n \n+\t/* Setup info->relevant_source_dirs */\n+\tinfo->relevant_source_dirs = dirs_removed;\n+\n \t/*\n \t * Loop setting up both info->idx_map, and doing setup of\n \t * info->dir_rename_count.\n-- \ngitgitgadget\n\n"},{"id":"417640","messageId":"4983a1c2f908f02bc8a47f883c31652723ffde51.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 07/10] diffcore-rename: add a dir_rename_guess field to dir_rename_info","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:04Z","receivedAt":"2021-02-24T00:08:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndir_rename_counts has a mapping of a mapping, in particular, it has\n old_dir => { new_dir => count }\nWe want a simple mapping of\n old_dir => new_dir\nbased on which new_dir had the highest count for a given old_dir.\nIntroduce dir_rename_guess for this purpose.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 42 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex db569e4a0b0a..d24f104aa81c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -369,6 +369,7 @@ static int find_exact_renames(struct diff_options *options)\n \n struct dir_rename_info {\n \tstruct strintmap idx_map;\n+\tstruct strmap dir_rename_guess;\n \tstruct strmap *dir_rename_count;\n \tunsigned setup;\n };\n@@ -381,6 +382,24 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n+static const char *get_highest_rename_path(struct strintmap *counts)\n+{\n+\tint highest_count = 0;\n+\tconst char *highest_destination_dir = NULL;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrintmap_for_each_entry(counts, &iter, entry) {\n+\t\tconst char *destination_dir = entry->key;\n+\t\tintptr_t count = (intptr_t)entry->value;\n+\t\tif (count > highest_count) {\n+\t\t\thighest_count = count;\n+\t\t\thighest_destination_dir = destination_dir;\n+\t\t}\n+\t}\n+\treturn highest_destination_dir;\n+}\n+\n static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n@@ -498,6 +517,8 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t struct strmap *dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n \tint i;\n \n \tinfo->setup = 0;\n@@ -511,6 +532,7 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\tstrmap_init(info->dir_rename_count);\n \t}\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n+\tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n \n \t/*\n \t * Loop setting up both info->idx_map, and doing setup of\n@@ -539,6 +561,23 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n+\n+\t/*\n+\t * Now we collapse\n+\t * dir_rename_count: old_directory -> {new_directory -> count}\n+\t * down to\n+\t * dir_rename_guess: old_directory -> best_new_directory\n+\t * where best_new_directory is the one with the highest count.\n+\t */\n+\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\t/* entry->key is source_dir */\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tchar *best_newdir;\n+\n+\t\tbest_newdir = xstrdup(get_highest_rename_path(counts));\n+\t\tstrmap_put(&info->dir_rename_guess, entry->key,\n+\t\t\t best_newdir);\n+\t}\n }\n \n void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n@@ -566,6 +605,9 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \t/* idx_map */\n \tstrintmap_clear(&info->idx_map);\n \n+\t/* dir_rename_guess */\n+\tstrmap_clear(&info->dir_rename_guess, 1);\n+\n \tif (!keep_dir_rename_count) {\n \t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n \t\tstrmap_clear(info->dir_rename_count, 1);\n-- \ngitgitgadget\n\n"},{"id":"417641","messageId":"3a29cf9e526fba0227a7eec92c0c6bd58a7850f0.1614123848.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v2 05/10] diffcore-rename: compute dir_rename_counts in stages","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-23T23:44:02Z","receivedAt":"2021-02-24T00:09:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to first compute dir_rename_counts based just on exact renames\nto start, as that can provide us useful information in\nfind_basename_matches(). That will give us an incomplete result, which\nwe can then later augment as basename and inexact rename matches are\nfound.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 76 ++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 62 insertions(+), 14 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex aa21d4e7175c..489e9cb0871e 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -411,6 +411,28 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tchar new_dir_first_char = new_dir[0];\n \tint first_time_in_loop = 1;\n \n+\tif (!info->setup)\n+\t\t/*\n+\t\t * info->setup is 0 here in two cases: (1) all auxiliary\n+\t\t * vars (like dirs_removed) were NULL so\n+\t\t * initialize_dir_rename_info() returned early, or (2)\n+\t\t * either break detection or copy detection are active so\n+\t\t * that we never called initialize_dir_rename_info(). In\n+\t\t * the former case, we don't have enough info to know if\n+\t\t * directories were renamed (because dirs_removed lets us\n+\t\t * know about a necessary prerequisite, namely if they were\n+\t\t * removed), and in the latter, we don't care about\n+\t\t * directory renames or find_basename_matches.\n+\t\t *\n+\t\t * This matters because both basename and inexact matching\n+\t\t * will also call update_dir_rename_counts(). In either of\n+\t\t * the above two cases info->dir_rename_counts will not\n+\t\t * have been properly initialized which prevents us from\n+\t\t * updating it, but in these two cases we don't care about\n+\t\t * dir_rename_counts anyway, so we can just exit early.\n+\t\t */\n+\t\treturn;\n+\n \twhile (1) {\n \t\tdirname_munge(old_dir);\n \t\tdirname_munge(new_dir);\n@@ -471,14 +493,22 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct dir_rename_info *info,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t struct strmap *dir_rename_count)\n+static void initialize_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n+\tinfo->setup = 0;\n+\tif (!dirs_removed)\n+\t\treturn;\n \tinfo->setup = 1;\n+\n \tinfo->dir_rename_count = dir_rename_count;\n+\tif (!info->dir_rename_count) {\n+\t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n+\t\tstrmap_init(info->dir_rename_count);\n+\t}\n \n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n@@ -506,7 +536,6 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n \tstrmap_partial_clear(dir_rename_count, 1);\n }\n \n-MAYBE_UNUSED\n static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t int keep_dir_rename_count)\n@@ -561,7 +590,9 @@ static const char *get_basename(const char *filename)\n }\n \n static int find_basename_matches(struct diff_options *options,\n-\t\t\t\t int minimum_score)\n+\t\t\t\t int minimum_score,\n+\t\t\t\t struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed)\n {\n \t/*\n \t * When I checked in early 2020, over 76% of file renames in linux\n@@ -669,6 +700,8 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\tcontinue;\n \t\t\trecord_rename_pair(dst_index, src_index, score);\n \t\t\trenames++;\n+\t\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t\t one->path, two->path);\n \n \t\t\t/*\n \t\t\t * Found a rename so don't need text anymore; if we\n@@ -752,7 +785,12 @@ static int too_many_rename_candidates(int num_destinations, int num_sources,\n \treturn 1;\n }\n \n-static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, int copies)\n+static int find_renames(struct diff_score *mx,\n+\t\t\tint dst_cnt,\n+\t\t\tint minimum_score,\n+\t\t\tint copies,\n+\t\t\tstruct dir_rename_info *info,\n+\t\t\tstruct strset *dirs_removed)\n {\n \tint count = 0, i;\n \n@@ -769,6 +807,9 @@ static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, i\n \t\t\tcontinue;\n \t\trecord_rename_pair(mx[i].dst, mx[i].src, mx[i].score);\n \t\tcount++;\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t rename_src[mx[i].src].p->one->path,\n+\t\t\t\t\t rename_dst[mx[i].dst].p->two->path);\n \t}\n \treturn count;\n }\n@@ -840,6 +881,8 @@ void diffcore_rename_extended(struct diff_options *options,\n \tinfo.setup = 0;\n \tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n+\tif (dirs_removed && (break_idx || want_copies))\n+\t\tBUG(\"dirs_removed incompatible with break/copy detection\");\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n \n@@ -931,10 +974,17 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tremove_unneeded_paths_from_src(want_copies);\n \t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n \n+\t\t/* Preparation for basename-driven matching. */\n+\t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n+\t\tinitialize_dir_rename_info(&info,\n+\t\t\t\t\t dirs_removed, dir_rename_count);\n+\t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n+\n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n-\t\t\t\t\t\t min_basename_score);\n+\t\t\t\t\t\t min_basename_score,\n+\t\t\t\t\t\t &info, dirs_removed);\n \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n \n \t\t/*\n@@ -1020,18 +1070,15 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/* cost matrix sorted by most to least similar pair */\n \tSTABLE_QSORT(mx, dst_cnt * NUM_CANDIDATE_PER_DST, score_compare);\n \n-\trename_count += find_renames(mx, dst_cnt, minimum_score, 0);\n+\trename_count += find_renames(mx, dst_cnt, minimum_score, 0,\n+\t\t\t\t &info, dirs_removed);\n \tif (want_copies)\n-\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1);\n+\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1,\n+\t\t\t\t\t &info, dirs_removed);\n \tfree(mx);\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n-\t/*\n-\t * Now that renames have been computed, compute dir_rename_count */\n-\tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n-\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n@@ -1103,6 +1150,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tif (rename_dst[i].filespec_to_free)\n \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n \n+\tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n \trename_dst_nr = rename_dst_alloc = 0;\n \tFREE_AND_NULL(rename_src);\n-- \ngitgitgadget\n\n"},{"id":"417677","messageId":"bb0ef143-3050-6f3e-79ed-f06c1536f0a7@gmail.com","threadId":"55146","inReplyTo":"fec4f1d44c0694cf92c9b98f2b6519a15fb78188.1614123848.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/10] Move computation of dir_rename_count from merge-ort to diffcore-rename","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-24T15:25:34Z","receivedAt":"2021-02-24T15:55:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2021 6:43 PM, Elijah Newren via GitGitGadget wrote:\n> ... While the\n> diffstat looks large, viewing this commit with --color-moved makes it\n> clear that only about 20 lines changed. With this patch, the\n> computation of dir_rename_count is still only done after inexact rename\n> detection, but subsequent commits will add a preliminary computation of\n> dir_rename_count after exact rename detection, followed by some updates\n> after inexact rename detection.\n\nThe --color-moved recommendation is a good one. Everything seems to be\npretty standard, except this function:\n\n> +static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n> +\t\t\t\t struct strset *dirs_removed)\n> +{\n> +\tint i;\n> +\n> +\t/* Set up dir_rename_count */\n> +\tfor (i = 0; i < rename_dst_nr; ++i) {\n> +\t\t/*\n> +\t\t * Make dir_rename_count contain a map of a map:\n> +\t\t * old_directory -> {new_directory -> count}\n> +\t\t * In other words, for every pair look at the directories for\n> +\t\t * the old filename and the new filename and count how many\n> +\t\t * times that pairing occurs.\n> +\t\t */\n> +\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n> +\t\t\t\t\t rename_dst[i].p->one->path,\n> +\t\t\t\t\t rename_dst[i].p->two->path);\n> +\t}\n> +}\n\nis very similar to this:\n\n> -static void compute_rename_counts(struct diff_queue_struct *pairs,\n> -\t\t\t\t struct strmap *dir_rename_count,\n> -\t\t\t\t struct strset *dirs_removed)\n> -{\n> -\tint i;\n> -\n> -\tfor (i = 0; i < pairs->nr; ++i) {\n> -\t\tstruct diff_filepair *pair = pairs->queue[i];\n> -\n> -\t\t/* File not part of directory rename if it wasn't renamed */\n> -\t\tif (pair->status != 'R')\n> -\t\t\tcontinue;\n> -\n> -\t\t/*\n> -\t\t * Make dir_rename_count contain a map of a map:\n> -\t\t * old_directory -> {new_directory -> count}\n> -\t\t * In other words, for every pair look at the directories for\n> -\t\t * the old filename and the new filename and count how many\n> -\t\t * times that pairing occurs.\n> -\t\t */\n> -\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n> -\t\t\t\t\t pair->one->path,\n> -\t\t\t\t\t pair->two->path);\n> -\t}\n> -}\n> -\n\nbut we dropped that \"File not part of directory rename\" check.\n\nIt seems that is no longer possible to use with the new data structure,\nbut I wonder if this will cause a slowdown in the directory renames when\nmerging? Or, has the data already been filtered before calling\ncompute_dir_rename_counts()?\n\nThanks,\n-Stolee\n"},{"id":"417678","messageId":"588e7f21-9890-3aa6-4222-42946c5446fe@gmail.com","threadId":"55146","inReplyTo":"f7bdad78219de6819d0403f8957e9a0c8b4218bc.1614123848.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-24T15:37:02Z","receivedAt":"2021-02-24T15:55:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2021 6:44 PM, Elijah Newren via GitGitGadget wrote:\n> +\t\tfor (i=0; i +\tinfo->setup = 0;\n> +\tif (!dirs_removed)\n> +\t\treturn;\n> \tinfo->setup = 1;\n\nThis would probably be clearer as\n\n\tif (!dirs_removed) {\n\t\tinfo->setup = 0;\n\t\treturn;\n\t}\n\tinfo->setup = 1;\n\n> -MAYBE_UNUSED\n\nGood cleanup.\n\n> @@ -931,10 +974,17 @@ void diffcore_rename_extended(struct diff_options *options,\n> \t\tremove_unneeded_paths_from_src(want_copies);\n> \t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n> \n> +\t\t/* Preparation for basename-driven matching. */\n> +\t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n> +\t\tinitialize_dir_rename_info(&info,\n> +\t\t\t\t\t dirs_removed, dir_rename_count);\n> +\t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n> +\n\nThe parts visible in this context are pretty trivial, but this\nmethod _is_ doing a lot of work. Good to mark it with a trace\nregion so we can identify if/when it is a problem.\n\nThanks,\n-Stolee\n"},{"id":"417685","messageId":"aaf7c657-ef16-7612-166f-01560918d82d@gmail.com","threadId":"55146","inReplyTo":"cbd055ab33998962cb7712906cdad5dff3390660.1614123848.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/10] diffcore-rename: add a new idx_possible_rename function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-24T17:35:44Z","receivedAt":"2021-02-24T17:36:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2021 6:44 PM, Elijah Newren via GitGitGadget wrote:> +static char *get_dirname(const char *filename)\n> +{\n> +\tchar *slash = strrchr(filename, '/');\n> +\treturn slash ? xstrndup(filename, slash-filename) : xstrdup(\"\");\n\nMy brain interpreted \"slash-filename\" as a single token on first\nread, which confused me briefly. Inserting spaces would help\nreaders like me.\n\n> +\t * (4) Check if applying that directory rename to the original file\n> +\t * would result in a destination filename that is in the\n> +\t * potential rename set. If so, return the index of the\n> +\t * destination file (the index within rename_dst).\n\n> +\t * This function, idx_possible_rename(), is only responsible for (4).\n\nThis helps isolate the important step to care about for the implementation,\nwhile the rest of the context is important, too.\n\n> +\tchar *old_dir, *new_dir, *new_path;\n> +\tint idx;\n> +\n> +\tif (!info->setup)\n> +\t\treturn -1;\n> +\n> +\told_dir = get_dirname(filename);\n> +\tnew_dir = strmap_get(&info->dir_rename_guess, old_dir);\n> +\tfree(old_dir);\n> +\tif (!new_dir)\n> +\t\treturn -1;\n> +\n> +\tnew_path = xstrfmt(\"%s/%s\", new_dir, get_basename(filename));\n\nThis is running in a loop, so `xstrfmt()` might be overkill compared\nto something like\n\n\tstrbuf_addstr(&new_path, new_dir);\n\tstrbuf_addch(&new_path, '/');\n\tstrbuf_addstr(&new_path, get_basename(filename));\n\nbut maybe the difference is too small to notice. (notice the type\nchange to \"struct strbuf new_path = STRBUF_INIT;\")\n\n> +\n> +\tidx = strintmap_get(&info->idx_map, new_path);\n> +\tfree(new_path);\n> +\treturn idx;\n> +}\n\nDoes what it says it does.\n\nThanks,\n-Stolee\n"},{"id":"417686","messageId":"9286ffba-7927-d12f-988f-36e67fcb882d@gmail.com","threadId":"55146","inReplyTo":"805c101cfd849db3a5defb30775c7abbfec99f68.1614123848.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/10] diffcore-rename: use directory rename guided basename comparisons","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-24T17:44:54Z","receivedAt":"2021-02-24T17:45:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2021 6:44 PM, Elijah Newren via GitGitGadget wrote:\n> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> performance work; instrument with trace2_region_* calls\", 2020-10-28),\n> this change improves the performance as follows:\n> \n> Before After\n> no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n> mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n> just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n\nHooray!\n\n> +\tfor (i = 0; i < rename_src_nr; ++i) {\n> +\t\tchar *filename = rename_src[i].p->one->path;\n> +\t\tconst char *base = NULL;\n> +\t\tintptr_t src_index;\n> \t\tintptr_t dst_index;\n>\n> +\t\t/* Is this basename unique among remaining sources? */\n\nThis comment sent me down a confusing direction. Perhaps, we can\ninstead say:\n\n\t/*\n\t * If the basename is unique among remaining sources, then\n\t * src_index will equal 'i' and we can attempt to match it\n\t * to a unique basename in the destinations. Otherwise, use\n\t * directory rename heuristics, if possible.\n\t */\n\n> +\t\tbase = get_basename(filename);\n> +\t\tsrc_index = strintmap_get(&sources, base);\n> +\t\tassert(src_index == -1 || src_index == i);\n> +\n> +\t\tif (strintmap_contains(&dests, base)) {\n> \t\t\tstruct diff_filespec *one, *two;\n> \t\t\tint score;\n> \n> +\t\t\t/* Find a matching destination, if possible */\n> +\t\t\tdst_index = strintmap_get(&dests, base);\n> +\t\t\tif (src_index == -1 || dst_index == -1) {\n> +\t\t\t\tsrc_index = i;\n> +\t\t\t\tdst_index = idx_possible_rename(filename, info);\n> +\t\t\t}\n\nIt is important that 'src_index == i' from this point on, no\nmatter whether it was unique or not.\n\n> +\t\t\tif (dst_index == -1)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\t/* Ignore this dest if already used in a rename */\n> +\t\t\tif (rename_dst[dst_index].is_rename)\n> +\t\t\t\tcontinue; /* already used previously */\n> +\n\nThis seems to match all of the complicated special cases.\n\nThanks,\n-Stolee\n"},{"id":"417687","messageId":"22419bc6-d35a-7537-d388-d51c24138f5f@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/10] Optimization batch 8: use file basenames even more","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-24T17:50:15Z","receivedAt":"2021-02-24T17:51:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2021 6:43 PM, Elijah Newren via GitGitGadget wrote:\n> This series depends on en/diffcore-rename (a concatenation of what I was\n> calling ort-perf-batch-6 and ort-perf-batch-7).\n> \n> There are no changes since v1; it's just a resend a week and a half later to\n> bump it so it isn't lost.\n\nThank you for re-sending. I intended to review it before but got redirected\nand forgot to pick it up again.\n\n> === Results ===\n> \n> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> changes in just this series improves the performance as follows:\n> \n> Before Series After Series\n> no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n> mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n> just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n> \n> \n> As a reminder, before any merge-ort/diffcore-rename performance work, the\n> performance results we started with (as noted in the same commit message)\n> were:\n> \n> no-renames-am: 6.940 s ± 0.485 s\n> no-renames: 18.912 s ± 0.174 s\n> mega-renames: 5964.031 s ± 10.459 s\n> just-one-mega: 149.583 s ± 0.751 s\n\nThese are good results.\n\nI reviewed the patches and believe they do the optimizations claimed. I\nonly found some nits for comments and whitespace things.\n\nYou are very careful to create the necessary pieces and connect them\nfrom the bottom-up. However, this leads to one big \"now everything is\ndone\" commit with performance improvements. It seems that there are\nsome smaller performance improvements that could be measured if the\nlogic was instead built from the top-down with stubs for the complicated\nlogic.\n\nFor example, the final patch links the rename logic with a call to\nidx_possible_rename(). But, that could just as well always return -1\nand the implementation would be correct. Then, it would be good to see\nif the performance changes with that non-functional update. It would\nalso help me read the series in patch order and understand the context\nof the methods a bit better before seeing their implementation.\n\nThis is _not_ a recommendation that you rewrite the series. Just food\nfor thought as we continue with similar enhancements in the future.\n\nThanks,\n-Stolee\n"},{"id":"417693","messageId":"CABPp-BHJGi7m6=DgP3zDR4nLF6jaDuysgRgfDae8HY85VaLvgg@mail.gmail.com","threadId":"55146","inReplyTo":"bb0ef143-3050-6f3e-79ed-f06c1536f0a7@gmail.com","subject":"Re: [PATCH v2 01/10] Move computation of dir_rename_count from merge-ort to diffcore-rename","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-24T18:50:01Z","receivedAt":"2021-02-24T18:52:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 24, 2021 at 7:25 AM Derrick Stolee wrote:\n>\n> On 2/23/2021 6:43 PM, Elijah Newren via GitGitGadget wrote:\n> > ... While the\n> > diffstat looks large, viewing this commit with --color-moved makes it\n> > clear that only about 20 lines changed. With this patch, the\n> > computation of dir_rename_count is still only done after inexact rename\n> > detection, but subsequent commits will add a preliminary computation of\n> > dir_rename_count after exact rename detection, followed by some updates\n> > after inexact rename detection.\n>\n> The --color-moved recommendation is a good one. Everything seems to be\n> pretty standard, except this function:\n>\n> > +static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n> > + struct strset *dirs_removed)\n> > +{\n> > + int i;\n> > +\n> > + /* Set up dir_rename_count */\n> > + for (i = 0; i < rename_dst_nr; ++i) {\n> > + /*\n> > + * Make dir_rename_count contain a map of a map:\n> > + * old_directory -> {new_directory -> count}\n> > + * In other words, for every pair look at the directories for\n> > + * the old filename and the new filename and count how many\n> > + * times that pairing occurs.\n> > + */\n> > + update_dir_rename_counts(dir_rename_count, dirs_removed,\n> > + rename_dst[i].p->one->path,\n> > + rename_dst[i].p->two->path);\n> > + }\n> > +}\n>\n> is very similar to this:\n>\n> > -static void compute_rename_counts(struct diff_queue_struct *pairs,\n> > - struct strmap *dir_rename_count,\n> > - struct strset *dirs_removed)\n> > -{\n> > - int i;\n> > -\n> > - for (i = 0; i < pairs->nr; ++i) {\n> > - struct diff_filepair *pair = pairs->queue[i];\n> > -\n> > - /* File not part of directory rename if it wasn't renamed */\n> > - if (pair->status != 'R')\n> > - continue;\n> > -\n> > - /*\n> > - * Make dir_rename_count contain a map of a map:\n> > - * old_directory -> {new_directory -> count}\n> > - * In other words, for every pair look at the directories for\n> > - * the old filename and the new filename and count how many\n> > - * times that pairing occurs.\n> > - */\n> > - update_dir_rename_counts(dir_rename_count, dirs_removed,\n> > - pair->one->path,\n> > - pair->two->path);\n> > - }\n> > -}\n> > -\n>\n> but we dropped that \"File not part of directory rename\" check.\n>\n> It seems that is no longer possible to use with the new data structure,\n> but I wonder if this will cause a slowdown in the directory renames when\n> merging? Or, has the data already been filtered before calling\n> compute_dir_rename_counts()?\n\nOh, indeed, good catch. The\n if (pair->status != 'R')\n continue;\ncheck should have been translated to\n if (!rename_dst[i].is_rename)\n continue;\nSuch a check _was_ added later in the series except with more code\nthan just the \"continue\" that didn't make sense at this stage; the\nextra code made me overlook it when I was splitting. Oops. I'll add\nthis check back in; thanks for catching.\n"},{"id":"417768","messageId":"CABPp-BGL+OrWtuh4tDLA2e-XHw0=zkMCitb46jOkO_5d+vJtkw@mail.gmail.com","threadId":"55146","inReplyTo":"aaf7c657-ef16-7612-166f-01560918d82d@gmail.com","subject":"Re: [PATCH v2 08/10] diffcore-rename: add a new idx_possible_rename function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-25T01:13:57Z","receivedAt":"2021-02-25T01:15:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 24, 2021 at 9:35 AM Derrick Stolee wrote:\n>\n> On 2/23/2021 6:44 PM, Elijah Newren via GitGitGadget wrote:> +static char *get_dirname(const char *filename)\n> > +{\n> > + char *slash = strrchr(filename, '/');\n> > + return slash ? xstrndup(filename, slash-filename) : xstrdup(\"\");\n>\n> My brain interpreted \"slash-filename\" as a single token on first\n> read, which confused me briefly. Inserting spaces would help\n> readers like me.\n>\n> > + * (4) Check if applying that directory rename to the original file\n> > + * would result in a destination filename that is in the\n> > + * potential rename set. If so, return the index of the\n> > + * destination file (the index within rename_dst).\n>\n> > + * This function, idx_possible_rename(), is only responsible for (4).\n>\n> This helps isolate the important step to care about for the implementation,\n> while the rest of the context is important, too.\n>\n> > + char *old_dir, *new_dir, *new_path;\n> > + int idx;\n> > +\n> > + if (!info->setup)\n> > + return -1;\n> > +\n> > + old_dir = get_dirname(filename);\n> > + new_dir = strmap_get(&info->dir_rename_guess, old_dir);\n> > + free(old_dir);\n> > + if (!new_dir)\n> > + return -1;\n> > +\n> > + new_path = xstrfmt(\"%s/%s\", new_dir, get_basename(filename));\n>\n> This is running in a loop, so `xstrfmt()` might be overkill compared\n> to something like\n>\n> strbuf_addstr(&new_path, new_dir);\n> strbuf_addch(&new_path, '/');\n> strbuf_addstr(&new_path, get_basename(filename));\n>\n> but maybe the difference is too small to notice. (notice the type\n> change to \"struct strbuf new_path = STRBUF_INIT;\")\n\nOoh, nice find. Since this is in a loop over the renames as you point\nout, this is an O(N) improvement (with N = number of renames) rather\nthan an O(1) improvement. It does turn out to be hard to notice,\nthough. Since we still have some O(N^2) code (all the inexact rename\ndetection for which our exact- and basename-guided detection\noptimizations can't handle), with that N^2 actually being multiplied\nby the average number of lines in the given files, this improvement\ndoes seem to mostly get lost in the noise.\n\nI tried a bunch of times to measure the performance with these\nchanges. After a bunch of runs, it seems that this optimization saves\nsomewhere between 3-10ms (depending on which testcase, whether at this\npoint in the series or at the very end, etc.). It's hard to pin down,\nbecause the savings is less than the standard deviation of any given\nsets of runs. I don't think it's big enough to warrant restating the\nperformance measurements, but I'm very happy to include this\nsuggestion in my reroll.\n\n>\n> > +\n> > + idx = strintmap_get(&info->idx_map, new_path);\n> > + free(new_path);\n> > + return idx;\n> > +}\n>\n> Does what it says it does.\n>\n> Thanks,\n> -Stolee\n"},{"id":"417771","messageId":"CABPp-BH0NABbHqY1ZfcatiEryrqXLbv=qQKJKKzsGp-4=tWTEA@mail.gmail.com","threadId":"55146","inReplyTo":"22419bc6-d35a-7537-d388-d51c24138f5f@gmail.com","subject":"Re: [PATCH v2 00/10] Optimization batch 8: use file basenames even more","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-25T01:38:52Z","receivedAt":"2021-02-25T01:39:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 24, 2021 at 9:50 AM Derrick Stolee wrote:\n>\n> On 2/23/2021 6:43 PM, Elijah Newren via GitGitGadget wrote:\n> > This series depends on en/diffcore-rename (a concatenation of what I was\n> > calling ort-perf-batch-6 and ort-perf-batch-7).\n> >\n> > There are no changes since v1; it's just a resend a week and a half later to\n> > bump it so it isn't lost.\n>\n> Thank you for re-sending. I intended to review it before but got redirected\n> and forgot to pick it up again.\n>\n> > === Results ===\n> >\n> > For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> > performance work; instrument with trace2_region_* calls\", 2020-10-28), the\n> > changes in just this series improves the performance as follows:\n> >\n> > Before Series After Series\n> > no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n> > mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n> > just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n> >\n> >\n> > As a reminder, before any merge-ort/diffcore-rename performance work, the\n> > performance results we started with (as noted in the same commit message)\n> > were:\n> >\n> > no-renames-am: 6.940 s ± 0.485 s\n> > no-renames: 18.912 s ± 0.174 s\n> > mega-renames: 5964.031 s ± 10.459 s\n> > just-one-mega: 149.583 s ± 0.751 s\n>\n> These are good results.\n>\n> I reviewed the patches and believe they do the optimizations claimed. I\n> only found some nits for comments and whitespace things.\n\nThanks for taking a look; I'll get those fixed up. Also, I think your\nperformance improvement of switching from xstrfmt to a few strbuf\ncalls, even if small, counts as more than \"nits for comments and\nwhitespace things\". :-)\n\n> You are very careful to create the necessary pieces and connect them\n> from the bottom-up. However, this leads to one big \"now everything is\n> done\" commit with performance improvements. It seems that there are\n> some smaller performance improvements that could be measured if the\n> logic was instead built from the top-down with stubs for the complicated\n> logic.\n>\n> For example, the final patch links the rename logic with a call to\n> idx_possible_rename(). But, that could just as well always return -1\n> and the implementation would be correct. Then, it would be good to see\n> if the performance changes with that non-functional update. It would\n> also help me read the series in patch order and understand the context\n> of the methods a bit better before seeing their implementation.\n>\n> This is _not_ a recommendation that you rewrite the series. Just food\n> for thought as we continue with similar enhancements in the future.\n\nI can give it a shot for future relevant patch series (some of the\nseries this wouldn't be relevant for because they just include a\ncollection of patches implementing separate improvements that are just\nbatched together). A couple of the series are already structured this\nway, in fact, but the next series after this one has one patch that I\nthink I could reorder to make it more like this.\n"},{"id":"417772","messageId":"87v9ag7uyx.fsf@evledraar.gmail.com","threadId":"55146","inReplyTo":"f7bdad78219de6819d0403f8957e9a0c8b4218bc.1614123848.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-25T02:16:22Z","receivedAt":"2021-02-25T02:17:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 24 2021, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren \n> [...]\n> MAYBE_UNUSED\n> -static void cleanup_dir_rename_info(struct dir_rename_info *info)\n> +static void cleanup_dir_rename_info(struct dir_rename_info *info,\n> +\t\t\t\t struct strset *dirs_removed,\n> +\t\t\t\t int keep_dir_rename_count)\n> {\n> +\tstruct hashmap_iter iter;\n> +\tstruct strmap_entry *entry;\n> +\n> \tif (!info->setup)\n> \t\treturn;\n> \n> -\tpartial_clear_dir_rename_count(info->dir_rename_count);\n> -\tstrmap_clear(info->dir_rename_count, 1);\n> +\tif (!keep_dir_rename_count) {\n> +\t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n> +\t\tstrmap_clear(info->dir_rename_count, 1);\n> +\t\tFREE_AND_NULL(info->dir_rename_count);\n> +\t} else {\n> +\t\t/*\n> +\t\t * Although dir_rename_count was passed in\n> +\t\t * diffcore_rename_extended() and we want to keep it around and\n> +\t\t * return it to that caller, we first want to remove any data\n> +\t\t * associated with directories that weren't renamed.\n> +\t\t */\n> +\t\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n> +\t\tint i;\n> +\n> [...]\n\nI find the pattern in patch 02 and 03 and leading up to this 04/05\nconfusing to review.\n\nFirst we add a clear_dir_rename_count() in 02 but nothing uses it, then\nin 03 it's renamed to cleanup_dir_rename_info() and its code changed,\nbut still nothing uses it. Here we're changing the function nothing\nuses, and then finally in 05 we make use of it, and the MAYBE_UNUSED\nattribute is removed.\n\nI appreciate trying to split these large and complex patches into more\ndigestible pieces. I think that sometimes it's more readable to have a\npatch that adds a function and a subsequent one that uses it.\n\nBut in this case where we've gone through stages of changing code that's\nnever been used I think we're making it harder to read than not. I'd\nprefer just to see this cleanup_dir_rename_info() function pop into\nexistence in 05.\n\nJust my 0.02.\n\nStyle nit/preference: I think code like this is easier to read as:\n\n if (simple-case) {\n blah\n blah;\n return;\n }\n complex_case;\n\nThan not having the \"return\" and having most of the interesting logic in\nan indented \"else\" block. Or maybe just this on top of the whole thing\n(a -w diff, hopefully more readable, but still understandable):\n \n diff --git a/diffcore-rename.c b/diffcore-rename.c\n index 70a484b9b6..5a5c62ec79 100644\n --- a/diffcore-rename.c\n +++ b/diffcore-rename.c\n @@ -609,11 +609,12 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n }\n \n static void cleanup_dir_rename_info(struct dir_rename_info *info,\n -\t\t\t\t struct strset *dirs_removed,\n -\t\t\t\t int keep_dir_rename_count)\n +\t\t\t\t struct strset *dirs_removed)\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n +\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n +\tint i;\n \n \tif (!info->setup)\n \t\treturn;\n @@ -624,20 +625,12 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \t/* dir_rename_guess */\n \tstrmap_clear(&info->dir_rename_guess, 1);\n \n -\tif (!keep_dir_rename_count) {\n -\t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n -\t\tstrmap_clear(info->dir_rename_count, 1);\n -\t\tFREE_AND_NULL(info->dir_rename_count);\n -\t} else {\n \t/*\n \t * Although dir_rename_count was passed in\n \t * diffcore_rename_extended() and we want to keep it around and\n \t * return it to that caller, we first want to remove any data\n \t * associated with directories that weren't renamed.\n \t */\n -\t\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n -\t\tint i;\n -\n \tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n \t\tconst char *source_dir = entry->key;\n \t\tstruct strintmap *counts = entry->value;\n @@ -653,7 +646,6 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info,\n \t\t\t to_remove.items[i].string, 1);\n \tstring_list_clear(&to_remove, 0);\n }\n -}\n \n static const char *get_basename(const char *filename)\n {\n @@ -1317,7 +1309,13 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tif (rename_dst[i].filespec_to_free)\n \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n \n -\tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n +\tif (!dir_rename_count) {\n +\t\tcleanup_dir_rename_info(&info, dirs_removed);\n +\t} else {\n +\t\tpartial_clear_dir_rename_count(info.dir_rename_count);\n +\t\tstrmap_clear(info.dir_rename_count, 1);\n +\t\tFREE_AND_NULL(info.dir_rename_count);\n +\t}\n \tFREE_AND_NULL(rename_dst);\n \trename_dst_nr = rename_dst_alloc = 0;\n \tFREE_AND_NULL(rename_src);\n\nI also wonder if that strmap_clear() wouldn't be better moved into\npartial_clear_dir_rename_count():\n \n diff --git a/diffcore-rename.c b/diffcore-rename.c\n index 5a5c62ec790..5f6c5745d64 100644\n --- a/diffcore-rename.c\n +++ b/diffcore-rename.c\n @@ -596,7 +596,8 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t}\n }\n \n -void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n +void partial_clear_dir_rename_count(struct strmap *dir_rename_count,\n +\t\t\t\t int clear_strmap)\n {\n \tstruct hashmap_iter iter;\n \tstruct strmap_entry *entry;\n @@ -606,6 +607,9 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n \t\tstrintmap_clear(counts);\n \t}\n \tstrmap_partial_clear(dir_rename_count, 1);\n +\tif (clear_strmap)\n +\t\tstrmap_clear(dir_rename_count, 1);\n +\n }\n \n static void cleanup_dir_rename_info(struct dir_rename_info *info,\n @@ -1312,8 +1316,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \tif (!dir_rename_count) {\n \t\tcleanup_dir_rename_info(&info, dirs_removed);\n \t} else {\n -\t\tpartial_clear_dir_rename_count(info.dir_rename_count);\n -\t\tstrmap_clear(info.dir_rename_count, 1);\n +\t\tpartial_clear_dir_rename_count(info.dir_rename_count, 1);\n \t\tFREE_AND_NULL(info.dir_rename_count);\n \t}\n \tFREE_AND_NULL(rename_dst);\n diff --git a/diffcore.h b/diffcore.h\n index c6ba64abd19..de8667bfa04 100644\n --- a/diffcore.h\n +++ b/diffcore.h\n @@ -161,7 +161,8 @@ struct diff_filepair *diff_queue(struct diff_queue_struct *,\n \t\t\t\t struct diff_filespec *);\n void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n -void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n +void partial_clear_dir_rename_count(struct strmap *dir_rename_count,\n +\t\t\t\t int clear_strmap);\n \n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n diff --git a/merge-ort.c b/merge-ort.c\n index 467404cc0a3..0bbd49f0d78 100644\n --- a/merge-ort.c\n +++ b/merge-ort.c\n @@ -353,9 +353,9 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n \t\tstrset_func(&renames->dirs_removed[i]);\n \n -\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n -\t\tif (!reinitialize)\n -\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n +\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i],\n +\t\t\t\t\t !reinitialize);\n +\t\tfree(&renames->dir_rename_count[i]);\n \n \t\tstrmap_func(&renames->dir_renames[i], 0);\n \t}\n \n"},{"id":"417773","messageId":"87sg5k7uhf.fsf@evledraar.gmail.com","threadId":"55146","inReplyTo":"87v9ag7uyx.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 04/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-25T02:26:52Z","receivedAt":"2021-02-25T02:27:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 25 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> -\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n> -\t\tif (!reinitialize)\n> -\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n> +\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i],\n> +\t\t\t\t\t !reinitialize);\n> +\t\tfree(&renames->dir_rename_count[i]);\n> \n> \t\tstrmap_func(&renames->dir_renames[i], 0);\n> \t}\n\nThat free() wasn't supposed to be in there, I was still experimenting\nwith whether partial_clear_dir_rename_count() should also free() this\nitself.\n\nBut now that I notice I left that in there and in the meantime ran all\nthe tests, they all passed. So maybe this & the if/else in\ndiffcore_rename_extended() can lose its braces by just calling either\ncleanup_dir_rename_info() or partial_clear_dir_rename_count(). I didn't\nlook in any detail, and if the free() v.s. FREE_AND_NULL() distinction\nmattered here.\n"},{"id":"417774","messageId":"xmqqim6gyix5.fsf@gitster.g","threadId":"55146","inReplyTo":"87v9ag7uyx.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 04/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-25T02:34:30Z","receivedAt":"2021-02-25T02:35:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> But in this case where we've gone through stages of changing code that's\n> never been used I think we're making it harder to read than not. I'd\n> prefer just to see this cleanup_dir_rename_info() function pop into\n> existence in 05.\n\nI had a similar impression on parts of the earlier series where a\nnew helper without users were given as a standalone patch. I found\nit a bit disorienting.\n\n> Style nit/preference: I think code like this is easier to read as:\n>\n> if (simple-case) {\n> blah\n> blah;\n> return;\n> }\n> complex_case;\n>\n> Than not having the \"return\" and having most of the interesting logic in\n> an indented \"else\" block. Or maybe just this on top of the whole thing\n> (a -w diff, hopefully more readable, but still understandable):\n\nYes, that is also a good tip for a more readable patch, but that\napplies only for if/else at the end of the function.\n\nIn general, formulating the condition so that the smaller body comes\nfirst for \"if\" and the larger one goes to the body of \"else\" would\nmake the if/else easier to understand, as you can often hold the\ncondition and smaller body just before \"else\" in your head, and\nafter clearly understanding that part, it becomes easier to\nconcentrate on the other side, i.e. \"now we know what happens if the\ncondition is true, what about the other case? Let's read on ...\".\n\nThanks.\n"},{"id":"417855","messageId":"6afa9add40b9cc75c284c480d3048b0e32e8198a.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 01/10] diffcore-rename: use directory rename guided basename comparisons","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:10Z","receivedAt":"2021-02-26T01:59:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA previous commit noted that it is very common for people to move files\nacross directories while keeping their filename the same. The last few\ncommits took advantage of this and showed that we can accelerate rename\ndetection significantly using basenames; since files with the same\nbasename serve as likely rename candidates, we can check those first and\nremove them from the rename candidate pool if they are sufficiently\nsimilar.\n\nUnfortunately, the previous optimization was limited by the fact that\nthe remaining basenames after exact rename detection are not always\nunique. Many repositories have hundreds of build files with the same\nname (e.g. Makefile, .gitignore, build.gradle, etc.), and may even have\nhundreds of source files with the same name. (For example, the linux\nkernel has 100 setup.c, 87 irq.c, and 112 core.c files. A repository at\n$DAYJOB has a lot of ObjectFactory.java and Plugin.java files).\n\nFor these files with non-unique basenames, we are faced with the task of\nattempting to determine or guess which directory they may have been\nrelocated to. Such a task is precisely the job of directory rename\ndetection. However, there are two catches: (1) the directory rename\ndetection code has traditionally been part of the merge machinery rather\nthan diffcore-rename.c, and (2) directory rename detection currently\nruns after regular rename detection is complete. The 1st catch is just\nan implementation issue that can be overcome by some code shuffling.\nThe 2nd requires us to add a further approximation: we only have access\nto exact renames at this point, so we need to do directory rename\ndetection based on just exact renames. In some cases we won't have\nexact renames, in which case this extra optimization won't apply. We\nalso choose to not apply the optimization unless we know that the\nunderlying directory was removed, which will require extra data to be\npassed in to diffcore_rename_extended(). Also, even if we get a\nprediction about which directory a file may have relocated to, we will\nstill need to check to see if there is a file in the predicted\ndirectory, and then compare the two files to see if they meet the higher\nmin_basename_score threshold required for marking the two files as\nrenames.\n\nThis commit introduces an idx_possible_rename() function which will give\ndo this directory rename detection for us and give us the index within\nrename_dst of the resulting filename. For now, this function is\nhardcoded to return -1 (not found) and just hooks up how its results\nwould be used once we have a more complete implementation in place.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 42 ++++++++++++++++++++++++++++-------\n 2 files changed, 35 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex 80fcf9542441..8673a5c5b2f2 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -186,7 +186,7 @@ mark a file pair as a rename and stop considering other candidates for\n better matches. At most, one comparison is done per file in this\n preliminary pass; so if there are several remaining ext.txt files\n throughout the directory hierarchy after exact rename detection, this\n-preliminary step will be skipped for those files.\n+preliminary step may be skipped for those files.\n \n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 41558185ae1d..b3055683bac2 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -379,6 +379,12 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n+static int idx_possible_rename(char *filename)\n+{\n+\t/* Unconditionally return -1, \"not found\", for now */\n+\treturn -1;\n+}\n+\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score)\n {\n@@ -415,8 +421,6 @@ static int find_basename_matches(struct diff_options *options,\n \tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n-\tstruct hashmap_iter iter;\n-\tstruct strmap_entry *entry;\n \n \t/*\n \t * The prefeteching stuff wants to know if it can skip prefetching\n@@ -466,17 +470,39 @@ static int find_basename_matches(struct diff_options *options,\n \t}\n \n \t/* Now look for basename matchups and do similarity estimation */\n-\tstrintmap_for_each_entry(&sources, &iter, entry) {\n-\t\tconst char *base = entry->key;\n-\t\tintptr_t src_index = (intptr_t)entry->value;\n+\tfor (i = 0; i < rename_src_nr; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base = NULL;\n+\t\tintptr_t src_index;\n \t\tintptr_t dst_index;\n-\t\tif (src_index == -1)\n-\t\t\tcontinue;\n \n-\t\tif (0 <= (dst_index = strintmap_get(&dests, base))) {\n+\t\t/*\n+\t\t * If the basename is unique among remaining sources, then\n+\t\t * src_index will equal 'i' and we can attempt to match it\n+\t\t * to a unique basename in the destinations. Otherwise,\n+\t\t * use directory rename heuristics, if possible.\n+\t\t */\n+\t\tbase = get_basename(filename);\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tassert(src_index == -1 || src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n \t\t\tstruct diff_filespec *one, *two;\n \t\t\tint score;\n \n+\t\t\t/* Find a matching destination, if possible */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (src_index == -1 || dst_index == -1) {\n+\t\t\t\tsrc_index = i;\n+\t\t\t\tdst_index = idx_possible_rename(filename);\n+\t\t\t}\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n \t\t\t/* Estimate the similarity */\n \t\t\tone = rename_src[src_index].p->one;\n \t\t\ttwo = rename_dst[dst_index].p->two;\n-- \ngitgitgadget\n\n"},{"id":"417856","messageId":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v2.git.1614123848.gitgitgadget@gmail.com","subject":"[PATCH v3 00/10] Optimization batch 8: use file basenames even more","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:09Z","receivedAt":"2021-02-26T01:59:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on en/diffcore-rename (a concatenation of what I was\ncalling ort-perf-batch-6 and ort-perf-batch-7).\n\nChanges since v2:\n\n * Rearrange the patches in the series to have a top-down ordering rather\n than bottom-up -- as suggested by Stolee, Ævar, and Junio\n * Several comments and style improvements suggested by Stolee\n * Replace xstrfmt() with a few strbuf_add*() calls, as suggested by Stolee\n\nElijah Newren (10):\n diffcore-rename: use directory rename guided basename comparisons\n diffcore-rename: add a new idx_possible_rename function\n diffcore-rename: add a mapping of destination names to their indices\n Move computation of dir_rename_count from merge-ort to diffcore-rename\n diffcore-rename: add function for clearing dir_rename_count\n diffcore-rename: move dir_rename_counts into dir_rename_info struct\n diffcore-rename: extend cleanup_dir_rename_info()\n diffcore-rename: compute dir_rename_counts in stages\n diffcore-rename: limit dir_rename_counts computation to relevant dirs\n diffcore-rename: compute dir_rename_guess from dir_rename_counts\n\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 449 ++++++++++++++++++++++++++++++++--\n diffcore.h | 7 +\n merge-ort.c | 144 +----------\n 4 files changed, 449 insertions(+), 153 deletions(-)\n\n\nbase-commit: aeca14f748afc7fb5b65bca56ea2ebd970729814\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-844%2Fnewren%2Fort-perf-batch-8-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-844/newren/ort-perf-batch-8-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/844\n\nRange-diff vs v2:\n\n 10: 805c101cfd84 ! 1: 6afa9add40b9 diffcore-rename: use directory rename guided basename comparisons\n @@ Metadata\n ## Commit message ##\n diffcore-rename: use directory rename guided basename comparisons\n \n - Hook the work from the last several patches together so that when\n - basenames in the sets of possible remaining rename sources or\n - destinations aren't unique, we can guess which directory source files\n - were renamed into. When that guess gives us a pairing of files, and\n - those files are sufficiently similar, we record the two files as a\n - rename and remove them from the large matrix of comparisons for inexact\n - rename detection.\n + A previous commit noted that it is very common for people to move files\n + across directories while keeping their filename the same. The last few\n + commits took advantage of this and showed that we can accelerate rename\n + detection significantly using basenames; since files with the same\n + basename serve as likely rename candidates, we can check those first and\n + remove them from the rename candidate pool if they are sufficiently\n + similar.\n \n - For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n - performance work; instrument with trace2_region_* calls\", 2020-10-28),\n - this change improves the performance as follows:\n + Unfortunately, the previous optimization was limited by the fact that\n + the remaining basenames after exact rename detection are not always\n + unique. Many repositories have hundreds of build files with the same\n + name (e.g. Makefile, .gitignore, build.gradle, etc.), and may even have\n + hundreds of source files with the same name. (For example, the linux\n + kernel has 100 setup.c, 87 irq.c, and 112 core.c files. A repository at\n + $DAYJOB has a lot of ObjectFactory.java and Plugin.java files).\n \n - Before After\n - no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n - mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n - just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n + For these files with non-unique basenames, we are faced with the task of\n + attempting to determine or guess which directory they may have been\n + relocated to. Such a task is precisely the job of directory rename\n + detection. However, there are two catches: (1) the directory rename\n + detection code has traditionally been part of the merge machinery rather\n + than diffcore-rename.c, and (2) directory rename detection currently\n + runs after regular rename detection is complete. The 1st catch is just\n + an implementation issue that can be overcome by some code shuffling.\n + The 2nd requires us to add a further approximation: we only have access\n + to exact renames at this point, so we need to do directory rename\n + detection based on just exact renames. In some cases we won't have\n + exact renames, in which case this extra optimization won't apply. We\n + also choose to not apply the optimization unless we know that the\n + underlying directory was removed, which will require extra data to be\n + passed in to diffcore_rename_extended(). Also, even if we get a\n + prediction about which directory a file may have relocated to, we will\n + still need to check to see if there is a file in the predicted\n + directory, and then compare the two files to see if they meet the higher\n + min_basename_score threshold required for marking the two files as\n + renames.\n +\n + This commit introduces an idx_possible_rename() function which will give\n + do this directory rename detection for us and give us the index within\n + rename_dst of the resulting filename. For now, this function is\n + hardcoded to return -1 (not found) and just hooks up how its results\n + would be used once we have a more complete implementation in place.\n \n Signed-off-by: Elijah Newren \n \n @@ diffcore-rename.c: static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n --MAYBE_UNUSED\n - static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n ++static int idx_possible_rename(char *filename)\n ++{\n ++\t/* Unconditionally return -1, \"not found\", for now */\n ++\treturn -1;\n ++}\n ++\n + static int find_basename_matches(struct diff_options *options,\n + \t\t\t\t int minimum_score)\n {\n - \t/*\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options,\n \tint i, renames = 0;\n \tstruct strintmap sources;\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n -\t\t\tcontinue;\n \n -\t\tif (0 <= (dst_index = strintmap_get(&dests, base))) {\n -+\t\t/* Is this basename unique among remaining sources? */\n ++\t\t/*\n ++\t\t * If the basename is unique among remaining sources, then\n ++\t\t * src_index will equal 'i' and we can attempt to match it\n ++\t\t * to a unique basename in the destinations. Otherwise,\n ++\t\t * use directory rename heuristics, if possible.\n ++\t\t */\n +\t\tbase = get_basename(filename);\n +\t\tsrc_index = strintmap_get(&sources, base);\n +\t\tassert(src_index == -1 || src_index == i);\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n +\t\t\tdst_index = strintmap_get(&dests, base);\n +\t\t\tif (src_index == -1 || dst_index == -1) {\n +\t\t\t\tsrc_index = i;\n -+\t\t\t\tdst_index = idx_possible_rename(filename, info);\n ++\t\t\t\tdst_index = idx_possible_rename(filename);\n +\t\t\t}\n +\t\t\tif (dst_index == -1)\n +\t\t\t\tcontinue;\n 8: cbd055ab3399 ! 2: 40f57bcc2055 diffcore-rename: add a new idx_possible_rename function\n @@ Commit message\n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: struct dir_rename_info {\n - \tunsigned setup;\n - };\n +@@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n + \treturn renames;\n + }\n \n ++struct dir_rename_info {\n ++\tstruct strintmap idx_map;\n ++\tstruct strmap dir_rename_guess;\n ++\tstruct strmap *dir_rename_count;\n ++\tunsigned setup;\n ++};\n ++\n +static char *get_dirname(const char *filename)\n +{\n +\tchar *slash = strrchr(filename, '/');\n -+\treturn slash ? xstrndup(filename, slash-filename) : xstrdup(\"\");\n ++\treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n +}\n +\n - static void dirname_munge(char *filename)\n + static const char *get_basename(const char *filename)\n {\n - \tchar *slash = strrchr(filename, '/');\n + \t/*\n @@ diffcore-rename.c: static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n -+MAYBE_UNUSED\n +-static int idx_possible_rename(char *filename)\n +static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n -+{\n + {\n +-\t/* Unconditionally return -1, \"not found\", for now */\n +-\treturn -1;\n +\t/*\n +\t * Our comparison of files with the same basename (see\n +\t * find_basename_matches() below), is only helpful when after exact\n @@ diffcore-rename.c: static const char *get_basename(const char *filename)\n +\t * rename.\n +\t *\n +\t * This function, idx_possible_rename(), is only responsible for (4).\n -+\t * The conditions/steps in (1)-(3) are handled via setting up\n -+\t * dir_rename_count and dir_rename_guess in\n -+\t * initialize_dir_rename_info(). Steps (0) and (5) are handled by\n -+\t * the caller of this function.\n ++\t * The conditions/steps in (1)-(3) will be handled via setting up\n ++\t * dir_rename_count and dir_rename_guess in a future\n ++\t * initialize_dir_rename_info() function. Steps (0) and (5) are\n ++\t * handled by the caller of this function.\n +\t */\n -+\tchar *old_dir, *new_dir, *new_path;\n ++\tchar *old_dir, *new_dir;\n ++\tstruct strbuf new_path = STRBUF_INIT;\n +\tint idx;\n +\n +\tif (!info->setup)\n @@ diffcore-rename.c: static const char *get_basename(const char *filename)\n +\tif (!new_dir)\n +\t\treturn -1;\n +\n -+\tnew_path = xstrfmt(\"%s/%s\", new_dir, get_basename(filename));\n ++\tstrbuf_addstr(&new_path, new_dir);\n ++\tstrbuf_addch(&new_path, '/');\n ++\tstrbuf_addstr(&new_path, get_basename(filename));\n +\n -+\tidx = strintmap_get(&info->idx_map, new_path);\n -+\tfree(new_path);\n ++\tidx = strintmap_get(&info->idx_map, new_path.buf);\n ++\tstrbuf_release(&new_path);\n +\treturn idx;\n -+}\n -+\n + }\n + \n static int find_basename_matches(struct diff_options *options,\n - \t\t\t\t int minimum_score,\n - \t\t\t\t struct dir_rename_info *info,\n +-\t\t\t\t int minimum_score)\n ++\t\t\t\t int minimum_score,\n ++\t\t\t\t struct dir_rename_info *info)\n + {\n + \t/*\n + \t * When I checked in early 2020, over 76% of file renames in linux\n +@@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options,\n + \t\t\tdst_index = strintmap_get(&dests, base);\n + \t\t\tif (src_index == -1 || dst_index == -1) {\n + \t\t\t\tsrc_index = i;\n +-\t\t\t\tdst_index = idx_possible_rename(filename);\n ++\t\t\t\tdst_index = idx_possible_rename(filename, info);\n + \t\t\t}\n + \t\t\tif (dst_index == -1)\n + \t\t\t\tcontinue;\n +@@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n + \tint num_destinations, dst_cnt;\n + \tint num_sources, want_copies;\n + \tstruct progress *progress = NULL;\n ++\tstruct dir_rename_info info;\n + \n + \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n ++\tinfo.setup = 0;\n + \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n + \tif (!minimum_score)\n + \t\tminimum_score = DEFAULT_RENAME_SCORE;\n +@@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n + \t\t/* Utilize file basenames to quickly find renames. */\n + \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n + \t\trename_count += find_basename_matches(options,\n +-\t\t\t\t\t\t min_basename_score);\n ++\t\t\t\t\t\t min_basename_score,\n ++\t\t\t\t\t\t &info);\n + \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n + \n + \t\t/*\n 6: dffecc064dd3 ! 3: 0e14961574ea diffcore-rename: add a mapping of destination names to their indices\n @@ Metadata\n ## Commit message ##\n diffcore-rename: add a mapping of destination names to their indices\n \n - Add an idx_map member to struct dir_rename_info, which tracks a mapping\n - of the full filename to the index within rename_dst where that filename\n - is found. We will later use this for quickly finding an array entry in\n - rename_dst given the pathname.\n + Compute a mapping of full filename to the index within rename_dst where\n + that filename is found, and store it in idx_map. idx_possible_rename()\n + needs this to quickly finding an array entry in rename_dst given the\n + pathname.\n +\n + While at it, add placeholder initializations for dir_rename_count and\n + dir_rename_guess; these will be more fully populated in subsequent\n + commits.\n \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n +@@ diffcore-rename.c: static char *get_dirname(const char *filename)\n + \treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n }\n \n - struct dir_rename_info {\n -+\tstruct strintmap idx_map;\n - \tstruct strmap *dir_rename_count;\n - \tunsigned setup;\n - };\n -@@ diffcore-rename.c: static void initialize_dir_rename_info(struct dir_rename_info *info,\n - \t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n - \t\tstrmap_init(info->dir_rename_count);\n - \t}\n ++static void initialize_dir_rename_info(struct dir_rename_info *info)\n ++{\n ++\tint i;\n ++\n ++\tinfo->setup = 1;\n ++\n +\tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n - \n ++\tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n ++\tinfo->dir_rename_count = NULL;\n ++\n +\t/*\n -+\t * Loop setting up both info->idx_map, and doing setup of\n -+\t * info->dir_rename_count.\n ++\t * Loop setting up both info->idx_map.\n +\t */\n - \tfor (i = 0; i < rename_dst_nr; ++i) {\n - \t\t/*\n --\t\t * Make dir_rename_count contain a map of a map:\n ++\tfor (i = 0; i < rename_dst_nr; ++i) {\n ++\t\t/*\n +\t\t * For non-renamed files, make idx_map contain mapping of\n +\t\t * filename -> index (index within rename_dst, that is)\n +\t\t */\n +\t\tif (!rename_dst[i].is_rename) {\n +\t\t\tchar *filename = rename_dst[i].p->two->path;\n +\t\t\tstrintmap_set(&info->idx_map, filename, i);\n -+\t\t\tcontinue;\n +\t\t}\n ++\t}\n ++}\n ++\n ++static void cleanup_dir_rename_info(struct dir_rename_info *info)\n ++{\n ++\tif (!info->setup)\n ++\t\treturn;\n +\n -+\t\t/*\n -+\t\t * For everything else (i.e. renamed files), make\n -+\t\t * dir_rename_count contain a map of a map:\n - \t\t * old_directory -> {new_directory -> count}\n - \t\t * In other words, for every pair look at the directories for\n - \t\t * the old filename and the new filename and count how many\n -@@ diffcore-rename.c: static void cleanup_dir_rename_info(struct dir_rename_info *info,\n - \tif (!info->setup)\n - \t\treturn;\n - \n +\t/* idx_map */\n +\tstrintmap_clear(&info->idx_map);\n +\n - \tif (!keep_dir_rename_count) {\n - \t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n - \t\tstrmap_clear(info->dir_rename_count, 1);\n ++\t/* dir_rename_guess */\n ++\tstrmap_clear(&info->dir_rename_guess, 1);\n ++\n ++\t/* Nothing to do for dir_rename_count, yet */\n ++}\n ++\n + static const char *get_basename(const char *filename)\n + {\n + \t/*\n +@@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n + \t\tremove_unneeded_paths_from_src(want_copies);\n + \t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n + \n ++\t\t/* Preparation for basename-driven matching. */\n ++\t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n ++\t\tinitialize_dir_rename_info(&info);\n ++\t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n ++\n + \t\t/* Utilize file basenames to quickly find renames. */\n + \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n + \t\trename_count += find_basename_matches(options,\n +@@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n + \t\tif (rename_dst[i].filespec_to_free)\n + \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n + \n ++\tcleanup_dir_rename_info(&info);\n + \tFREE_AND_NULL(rename_dst);\n + \trename_dst_nr = rename_dst_alloc = 0;\n + \tFREE_AND_NULL(rename_src);\n 1: fec4f1d44c06 ! 4: 9b9d5b207b03 Move computation of dir_rename_count from merge-ort to diffcore-rename\n @@ Metadata\n ## Commit message ##\n Move computation of dir_rename_count from merge-ort to diffcore-rename\n \n - A previous commit noted that it is very common for people to move files\n - across directories while keeping their filename the same. The last few\n - commits took advantage of this and showed that we can accelerate rename\n - detection significantly using basenames; since files with the same\n - basename serve as likely rename candidates, we can check those first and\n - remove them from the rename candidate pool if they are sufficiently\n - similar.\n + Move the computation of dir_rename_count from merge-ort.c to\n + diffcore-rename.c, making slight adjustments to the data structures\n + based on the move. While the diffstat looks large, viewing this commit\n + with --color-moved makes it clear that only about 20 lines changed.\n \n - Unfortunately, the previous optimization was limited by the fact that\n - the remaining basenames after exact rename detection are not always\n - unique. Many repositories have hundreds of build files with the same\n - name (e.g. Makefile, .gitignore, build.gradle, etc.), and may even have\n - hundreds of source files with the same name. (For example, the linux\n - kernel has 100 setup.c, 87 irq.c, and 112 core.c files. A repository at\n - $DAYJOB has a lot of ObjectFactory.java and Plugin.java files).\n -\n - For these files with non-unique basenames, we are faced with the task of\n - attempting to determine or guess which directory they may have been\n - relocated to. Such a task is precisely the job of directory rename\n - detection. However, there are two catches: (1) the directory rename\n - detection code has traditionally been part of the merge machinery rather\n - than diffcore-rename.c, and (2) directory rename detection currently\n - runs after regular rename detection is complete. The 1st catch is just\n - an implementation issue that can be overcome by some code shuffling.\n - The 2nd requires us to add a further approximation: we only have access\n - to exact renames at this point, so we need to do directory rename\n - detection based on just exact renames. In some cases we won't have\n - exact renames, in which case this extra optimization won't apply. We\n - also choose to not apply the optimization unless we know that the\n - underlying directory was removed, which will require extra data to be\n - passed in to diffcore_rename_extended(). Also, even if we get a\n - prediction about which directory a file may have relocated to, we will\n - still need to check to see if there is a file in the predicted\n - directory, and then compare the two files to see if they meet the higher\n - min_basename_score threshold required for marking the two files as\n - renames.\n -\n - This commit and the next few will set up the necessary infrastructure to\n - do such computations. This commit merely moves the computation of\n - dir_rename_count from merge-ort.c to diffcore-rename.c, making slight\n - adjustments to the data structures based on the move. While the\n - diffstat looks large, viewing this commit with --color-moved makes it\n - clear that only about 20 lines changed. With this patch, the\n - computation of dir_rename_count is still only done after inexact rename\n - detection, but subsequent commits will add a preliminary computation of\n - dir_rename_count after exact rename detection, followed by some updates\n - after inexact rename detection.\n + With this patch, the computation of dir_rename_count is still only done\n + after inexact rename detection, but subsequent commits will add a\n + preliminary computation of dir_rename_count after exact rename\n + detection, followed by some updates after inexact rename detection.\n \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n - \treturn renames;\n +@@ diffcore-rename.c: static char *get_dirname(const char *filename)\n + \treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n }\n \n +static void dirname_munge(char *filename)\n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n +\n +\t/* Set up dir_rename_count */\n +\tfor (i = 0; i < rename_dst_nr; ++i) {\n ++\t\t/* File not part of directory rename counts if not a rename */\n ++\t\tif (!rename_dst[i].is_rename)\n ++\t\t\tcontinue;\n ++\n +\t\t/*\n +\t\t * Make dir_rename_count contain a map of a map:\n +\t\t * old_directory -> {new_directory -> count}\n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n +\t}\n +}\n +\n - static const char *get_basename(const char *filename)\n + static void initialize_dir_rename_info(struct dir_rename_info *info)\n {\n - \t/*\n + \tint i;\n @@ diffcore-rename.c: static void remove_unneeded_paths_from_src(int detecting_copies)\n \trename_src_nr = new_num_src;\n }\n @@ diffcore-rename.c: static void remove_unneeded_paths_from_src(int detecting_copi\n \tint detect_rename = options->detect_rename;\n \tint minimum_score = options->rename_score;\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n - \tstruct progress *progress = NULL;\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n + \tinfo.setup = 0;\n +\tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n 2: 612da82f049c ! 5: f286e89464ea diffcore-rename: add functions for clearing dir_rename_count\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - diffcore-rename: add functions for clearing dir_rename_count\n + diffcore-rename: add function for clearing dir_rename_count\n \n - As we adjust the usage of dir_rename_count we want to have functions for\n - clearing, or partially clearing it out. Add such functions.\n + As we adjust the usage of dir_rename_count we want to have a function\n + for clearing, or partially clearing it out. Add a\n + partial_clear_dir_rename_count() function for this purpose.\n \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n +@@ diffcore-rename.c: static void initialize_dir_rename_info(struct dir_rename_info *info)\n \t}\n }\n \n @@ diffcore-rename.c: static void compute_dir_rename_counts(struct strmap *dir_rena\n +\tstrmap_partial_clear(dir_rename_count, 1);\n +}\n +\n -+MAYBE_UNUSED\n -+static void clear_dir_rename_count(struct strmap *dir_rename_count)\n -+{\n -+\tpartial_clear_dir_rename_count(dir_rename_count);\n -+\tstrmap_clear(dir_rename_count, 1);\n -+}\n -+\n - static const char *get_basename(const char *filename)\n + static void cleanup_dir_rename_info(struct dir_rename_info *info)\n {\n - \t/*\n + \tif (!info->setup)\n \n ## diffcore.h ##\n @@ diffcore.h: struct diff_filepair *diff_queue(struct diff_queue_struct *,\n 3: 93f98fc0b264 ! 6: ab353f2e75eb diffcore-rename: move dir_rename_counts into a dir_rename_info struct\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - diffcore-rename: move dir_rename_counts into a dir_rename_info struct\n + diffcore-rename: move dir_rename_counts into dir_rename_info struct\n \n - This is a purely cosmetic change for now, but we will be adding\n - additional information to the struct and changing where and how it is\n - setup and used in subsequent patches.\n + This continues the migration of the directory rename detection code into\n + diffcore-rename, now taking the simple step of combining it with the\n + dir_rename_info struct. Future commits will then make dir_rename_counts\n + be computed in stages, and add computation of dir_rename_guess.\n \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n - \treturn renames;\n - }\n - \n -+struct dir_rename_info {\n -+\tstruct strmap *dir_rename_count;\n -+\tunsigned setup;\n -+};\n -+\n - static void dirname_munge(char *filename)\n - {\n - \tchar *slash = strrchr(filename, '/');\n @@ diffcore-rename.c: static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n @@ diffcore-rename.c: static void update_dir_rename_counts(struct strmap *dir_renam\n +\tinfo->dir_rename_count = dir_rename_count;\n +\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n - \t\t/*\n - \t\t * Make dir_rename_count contain a map of a map:\n + \t\t/* File not part of directory rename counts if not a rename */\n + \t\tif (!rename_dst[i].is_rename)\n @@ diffcore-rename.c: static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t * the old filename and the new filename and count how many\n \t\t * times that pairing occurs.\n @@ diffcore-rename.c: static void compute_dir_rename_counts(struct strmap *dir_rena\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n -@@ diffcore-rename.c: void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n - }\n +@@ diffcore-rename.c: static void cleanup_dir_rename_info(struct dir_rename_info *info)\n + \t/* dir_rename_guess */\n + \tstrmap_clear(&info->dir_rename_guess, 1);\n \n - MAYBE_UNUSED\n --static void clear_dir_rename_count(struct strmap *dir_rename_count)\n -+static void cleanup_dir_rename_info(struct dir_rename_info *info)\n - {\n --\tpartial_clear_dir_rename_count(dir_rename_count);\n --\tstrmap_clear(dir_rename_count, 1);\n -+\tif (!info->setup)\n -+\t\treturn;\n -+\n +-\t/* Nothing to do for dir_rename_count, yet */\n ++\t/* dir_rename_count */\n +\tpartial_clear_dir_rename_count(info->dir_rename_count);\n +\tstrmap_clear(info->dir_rename_count, 1);\n }\n \n static const char *get_basename(const char *filename)\n -@@ diffcore-rename.c: void diffcore_rename_extended(struct diff_options *options,\n - \tint num_destinations, dst_cnt;\n - \tint num_sources, want_copies;\n - \tstruct progress *progress = NULL;\n -+\tstruct dir_rename_info info;\n - \n - \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n -+\tinfo.setup = 0;\n - \tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n - \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n - \tif (!minimum_score)\n @@ diffcore-rename.c: void diffcore_rename_extended(struct diff_options *options,\n \t/*\n \t * Now that renames have been computed, compute dir_rename_count */\n 4: f7bdad78219d ! 7: bd50d9e53804 diffcore-rename: extend cleanup_dir_rename_info()\n @@ Commit message\n \n ## diffcore-rename.c ##\n @@ diffcore-rename.c: void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n + \tstrmap_partial_clear(dir_rename_count, 1);\n }\n \n - MAYBE_UNUSED\n -static void cleanup_dir_rename_info(struct dir_rename_info *info)\n +static void cleanup_dir_rename_info(struct dir_rename_info *info,\n +\t\t\t\t struct strset *dirs_removed,\n @@ diffcore-rename.c: void partial_clear_dir_rename_count(struct strmap *dir_rename\n {\n +\tstruct hashmap_iter iter;\n +\tstruct strmap_entry *entry;\n ++\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n ++\tint i;\n +\n \tif (!info->setup)\n \t\treturn;\n \n +@@ diffcore-rename.c: static void cleanup_dir_rename_info(struct dir_rename_info *info)\n + \tstrmap_clear(&info->dir_rename_guess, 1);\n + \n + \t/* dir_rename_count */\n -\tpartial_clear_dir_rename_count(info->dir_rename_count);\n -\tstrmap_clear(info->dir_rename_count, 1);\n +\tif (!keep_dir_rename_count) {\n +\t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n +\t\tstrmap_clear(info->dir_rename_count, 1);\n +\t\tFREE_AND_NULL(info->dir_rename_count);\n -+\t} else {\n -+\t\t/*\n -+\t\t * Although dir_rename_count was passed in\n -+\t\t * diffcore_rename_extended() and we want to keep it around and\n -+\t\t * return it to that caller, we first want to remove any data\n -+\t\t * associated with directories that weren't renamed.\n -+\t\t */\n -+\t\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n -+\t\tint i;\n ++\t\treturn;\n ++\t}\n +\n -+\t\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n -+\t\t\tconst char *source_dir = entry->key;\n -+\t\t\tstruct strintmap *counts = entry->value;\n ++\t/*\n ++\t * Although dir_rename_count was passed in\n ++\t * diffcore_rename_extended() and we want to keep it around and\n ++\t * return it to that caller, we first want to remove any data\n ++\t * associated with directories that weren't renamed.\n ++\t */\n ++\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n ++\t\tconst char *source_dir = entry->key;\n ++\t\tstruct strintmap *counts = entry->value;\n +\n -+\t\t\tif (!strset_contains(dirs_removed, source_dir)) {\n -+\t\t\t\tstring_list_append(&to_remove, source_dir);\n -+\t\t\t\tstrintmap_clear(counts);\n -+\t\t\t\tcontinue;\n -+\t\t\t}\n ++\t\tif (!strset_contains(dirs_removed, source_dir)) {\n ++\t\t\tstring_list_append(&to_remove, source_dir);\n ++\t\t\tstrintmap_clear(counts);\n ++\t\t\tcontinue;\n +\t\t}\n -+\t\tfor (i=0; idir_rename_count,\n -+\t\t\t\t to_remove.items[i].string, 1);\n -+\t\tstring_list_clear(&to_remove, 0);\n +\t}\n ++\tfor (i = 0; i < to_remove.nr; ++i)\n ++\t\tstrmap_remove(info->dir_rename_count,\n ++\t\t\t to_remove.items[i].string, 1);\n ++\tstring_list_clear(&to_remove, 0);\n }\n \n static const char *get_basename(const char *filename)\n +@@ diffcore-rename.c: void diffcore_rename_extended(struct diff_options *options,\n + \t\tif (rename_dst[i].filespec_to_free)\n + \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n + \n +-\tcleanup_dir_rename_info(&info);\n ++\tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n + \tFREE_AND_NULL(rename_dst);\n + \trename_dst_nr = rename_dst_alloc = 0;\n + \tFREE_AND_NULL(rename_src);\n 5: 3a29cf9e526f ! 8: 44cfae6505f2 diffcore-rename: compute dir_rename_counts in stages\n @@ Metadata\n ## Commit message ##\n diffcore-rename: compute dir_rename_counts in stages\n \n - We want to first compute dir_rename_counts based just on exact renames\n - to start, as that can provide us useful information in\n - find_basename_matches(). That will give us an incomplete result, which\n - we can then later augment as basename and inexact rename matches are\n - found.\n + Compute dir_rename_counts based just on exact renames to start, as that\n + can provide us useful information in find_basename_matches(). This is\n + done by moving the code from compute_dir_rename_counts() into\n + initialize_dir_rename_info(), resulting in it being computed earlier and\n + based just on exact renames. Since that's an incomplete result, we\n + augment the counts via calling update_dir_rename_counts() after each\n + basename-guide and inexact rename detection match is found.\n \n Signed-off-by: Elijah Newren \n \n @@ diffcore-rename.c: static void update_dir_rename_counts(struct dir_rename_info *\n {\n \tint i;\n \n -+\tinfo->setup = 0;\n -+\tif (!dirs_removed)\n +-\tinfo->setup = 1;\n +-\tinfo->dir_rename_count = dir_rename_count;\n +-\n +-\tfor (i = 0; i < rename_dst_nr; ++i) {\n +-\t\t/* File not part of directory rename counts if not a rename */\n +-\t\tif (!rename_dst[i].is_rename)\n +-\t\t\tcontinue;\n +-\n +-\t\t/*\n +-\t\t * Make dir_rename_count contain a map of a map:\n +-\t\t * old_directory -> {new_directory -> count}\n +-\t\t * In other words, for every pair look at the directories for\n +-\t\t * the old filename and the new filename and count how many\n +-\t\t * times that pairing occurs.\n +-\t\t */\n +-\t\tupdate_dir_rename_counts(info, dirs_removed,\n +-\t\t\t\t\t rename_dst[i].p->one->path,\n +-\t\t\t\t\t rename_dst[i].p->two->path);\n ++\tif (!dirs_removed) {\n ++\t\tinfo->setup = 0;\n +\t\treturn;\n + \t}\n +-}\n +-\n +-static void initialize_dir_rename_info(struct dir_rename_info *info)\n +-{\n +-\tint i;\n +-\n \tinfo->setup = 1;\n -+\n - \tinfo->dir_rename_count = dir_rename_count;\n + \n ++\tinfo->dir_rename_count = dir_rename_count;\n +\tif (!info->dir_rename_count) {\n +\t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n +\t\tstrmap_init(info->dir_rename_count);\n +\t}\n + \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n + \tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n +-\tinfo->dir_rename_count = NULL;\n \n + \t/*\n +-\t * Loop setting up both info->idx_map.\n ++\t * Loop setting up both info->idx_map, and doing setup of\n ++\t * info->dir_rename_count.\n + \t */\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n -@@ diffcore-rename.c: void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n - \tstrmap_partial_clear(dir_rename_count, 1);\n +@@ diffcore-rename.c: static void initialize_dir_rename_info(struct dir_rename_info *info)\n + \t\tif (!rename_dst[i].is_rename) {\n + \t\t\tchar *filename = rename_dst[i].p->two->path;\n + \t\t\tstrintmap_set(&info->idx_map, filename, i);\n ++\t\t\tcontinue;\n + \t\t}\n ++\n ++\t\t/*\n ++\t\t * For everything else (i.e. renamed files), make\n ++\t\t * dir_rename_count contain a map of a map:\n ++\t\t * old_directory -> {new_directory -> count}\n ++\t\t * In other words, for every pair look at the directories for\n ++\t\t * the old filename and the new filename and count how many\n ++\t\t * times that pairing occurs.\n ++\t\t */\n ++\t\tupdate_dir_rename_counts(info, dirs_removed,\n ++\t\t\t\t\t rename_dst[i].p->one->path,\n ++\t\t\t\t\t rename_dst[i].p->two->path);\n + \t}\n }\n \n --MAYBE_UNUSED\n - static void cleanup_dir_rename_info(struct dir_rename_info *info,\n - \t\t\t\t struct strset *dirs_removed,\n - \t\t\t\t int keep_dir_rename_count)\n -@@ diffcore-rename.c: static const char *get_basename(const char *filename)\n - }\n +@@ diffcore-rename.c: static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n \n static int find_basename_matches(struct diff_options *options,\n --\t\t\t\t int minimum_score)\n -+\t\t\t\t int minimum_score,\n + \t\t\t\t int minimum_score,\n +-\t\t\t\t struct dir_rename_info *info)\n +\t\t\t\t struct dir_rename_info *info,\n +\t\t\t\t struct strset *dirs_removed)\n {\n @@ diffcore-rename.c: void diffcore_rename_extended(struct diff_options *options,\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n \n @@ diffcore-rename.c: void diffcore_rename_extended(struct diff_options *options,\n - \t\tremove_unneeded_paths_from_src(want_copies);\n - \t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n \n -+\t\t/* Preparation for basename-driven matching. */\n -+\t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n + \t\t/* Preparation for basename-driven matching. */\n + \t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n +-\t\tinitialize_dir_rename_info(&info);\n +\t\tinitialize_dir_rename_info(&info,\n +\t\t\t\t\t dirs_removed, dir_rename_count);\n -+\t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n -+\n + \t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n + \n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n --\t\t\t\t\t\t min_basename_score);\n -+\t\t\t\t\t\t min_basename_score,\n + \t\t\t\t\t\t min_basename_score,\n +-\t\t\t\t\t\t &info);\n +\t\t\t\t\t\t &info, dirs_removed);\n \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n \n @@ diffcore-rename.c: void diffcore_rename_extended(struct diff_options *options,\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n -@@ diffcore-rename.c: void diffcore_rename_extended(struct diff_options *options,\n - \t\tif (rename_dst[i].filespec_to_free)\n - \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n - \n -+\tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n - \tFREE_AND_NULL(rename_dst);\n - \trename_dst_nr = rename_dst_alloc = 0;\n - \tFREE_AND_NULL(rename_src);\n 9: 4e095ea7c439 ! 9: 752aff3a7995 diffcore-rename: limit dir_rename_counts computation to relevant dirs\n @@ Commit message\n for directories that disappeared, though, so we can return early from\n update_dir_rename_counts() for other paths.\n \n - While dirs_removed provides the relevant information for us right now,\n - we introduce a new info->relevant_source_dirs parameter because future\n - optimizations will want to change how things are called somewhat.\n + If dirs_removed is passed to diffcore_rename_extended(), then it\n + provides the relevant bits of information for us to limit this counting\n + to relevant dirs. If dirs_removed is not passed, we would need to\n + compute some replacement in order to do this limiting. Introduce a new\n + info->relevant_source_dirs variable for this purpose, even though at\n + this stage we will only set it to dirs_removed for simplicity.\n \n Signed-off-by: Elijah Newren \n \n 7: 4983a1c2f908 ! 10: 65f7bfb735f2 diffcore-rename: add a dir_rename_guess field to dir_rename_info\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - diffcore-rename: add a dir_rename_guess field to dir_rename_info\n + diffcore-rename: compute dir_rename_guess from dir_rename_counts\n \n dir_rename_counts has a mapping of a mapping, in particular, it has\n old_dir => { new_dir => count }\n We want a simple mapping of\n old_dir => new_dir\n based on which new_dir had the highest count for a given old_dir.\n - Introduce dir_rename_guess for this purpose.\n + Compute this and store it in dir_rename_guess.\n +\n + This is the final piece of the puzzle needed to make our guesses at\n + which directory files have been moved to when basenames aren't unique.\n +\n + For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n + performance work; instrument with trace2_region_* calls\", 2020-10-28),\n + this change improves the performance as follows:\n +\n + Before After\n + no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n + mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n + just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n - \n - struct dir_rename_info {\n - \tstruct strintmap idx_map;\n -+\tstruct strmap dir_rename_guess;\n - \tstruct strmap *dir_rename_count;\n - \tunsigned setup;\n - };\n @@ diffcore-rename.c: static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n @@ diffcore-rename.c: static void initialize_dir_rename_info(struct dir_rename_info\n +\tstruct strmap_entry *entry;\n \tint i;\n \n - \tinfo->setup = 0;\n -@@ diffcore-rename.c: static void initialize_dir_rename_info(struct dir_rename_info *info,\n - \t\tstrmap_init(info->dir_rename_count);\n - \t}\n - \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n -+\tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n - \n - \t/*\n - \t * Loop setting up both info->idx_map, and doing setup of\n + \tif (!dirs_removed) {\n @@ diffcore-rename.c: static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n @@ diffcore-rename.c: static void initialize_dir_rename_info(struct dir_rename_info\n }\n \n void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n -@@ diffcore-rename.c: static void cleanup_dir_rename_info(struct dir_rename_info *info,\n - \t/* idx_map */\n - \tstrintmap_clear(&info->idx_map);\n - \n -+\t/* dir_rename_guess */\n -+\tstrmap_clear(&info->dir_rename_guess, 1);\n -+\n - \tif (!keep_dir_rename_count) {\n - \t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n - \t\tstrmap_clear(info->dir_rename_count, 1);\n +@@ diffcore-rename.c: static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n + \t * rename.\n + \t *\n + \t * This function, idx_possible_rename(), is only responsible for (4).\n +-\t * The conditions/steps in (1)-(3) will be handled via setting up\n +-\t * dir_rename_count and dir_rename_guess in a future\n +-\t * initialize_dir_rename_info() function. Steps (0) and (5) are\n +-\t * handled by the caller of this function.\n ++\t * The conditions/steps in (1)-(3) are handled via setting up\n ++\t * dir_rename_count and dir_rename_guess in\n ++\t * initialize_dir_rename_info(). Steps (0) and (5) are handled by\n ++\t * the caller of this function.\n + \t */\n + \tchar *old_dir, *new_dir;\n + \tstruct strbuf new_path = STRBUF_INIT;\n\n-- \ngitgitgadget\n"},{"id":"417857","messageId":"40f57bcc2055773a7dd35bde50c54354a643d07a.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 02/10] diffcore-rename: add a new idx_possible_rename function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:11Z","receivedAt":"2021-02-26T01:59:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nfind_basename_matches() is great when both the remaining set of possible\nrename sources and the remaining set of possible rename destinations\nhave exactly one file each with a given basename. It allows us to match\nup files that have been moved to different directories without changing\nfilenames.\n\nWhen basenames are not unique, though, we want to be able to guess which\ndirectories the source files have been moved to. Since this is the job\nof directory rename detection, we employ it. However, since it is a\ndirectory rename detection idea, we also limit it to cases where we know\nthere could have been a directory rename, i.e. where the source\ndirectory has been removed. This has to be signalled by dirs_removed\nbeing non-NULL and containing an entry for the relevant directory.\nSince merge-ort.c is the only caller that currently does so, this\noptimization is only effective for merge-ort right now. In the future,\nthis condition could be reconsidered or we could modify other callers to\npass the necessary strset.\n\nAnyway, that's a lot of background so that we can actually describe the\nnew function. Add an idx_possible_rename() function which combines the\nrecently added dir_rename_guess and idx_map fields to provide the index\nwithin rename_dst of a potential match for a given file.\n\nFuture commits will add checks after calling this function to compare\nthe resulting 'likely rename' candidates to see if the two files meet\nthe elevated min_basename_score threshold for marking them as actual\nrenames.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 100 +++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 94 insertions(+), 6 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex b3055683bac2..edb0effb6ef4 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,19 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+struct dir_rename_info {\n+\tstruct strintmap idx_map;\n+\tstruct strmap dir_rename_guess;\n+\tstruct strmap *dir_rename_count;\n+\tunsigned setup;\n+};\n+\n+static char *get_dirname(const char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\n@@ -379,14 +392,86 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n-static int idx_possible_rename(char *filename)\n+static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n {\n-\t/* Unconditionally return -1, \"not found\", for now */\n-\treturn -1;\n+\t/*\n+\t * Our comparison of files with the same basename (see\n+\t * find_basename_matches() below), is only helpful when after exact\n+\t * rename detection we have exactly one file with a given basename\n+\t * among the rename sources and also only exactly one file with\n+\t * that basename among the rename destinations. When we have\n+\t * multiple files with the same basename in either set, we do not\n+\t * know which to compare against. However, there are some\n+\t * filenames that occur in large numbers (particularly\n+\t * build-related filenames such as 'Makefile', '.gitignore', or\n+\t * 'build.gradle' that potentially exist within every single\n+\t * subdirectory), and for performance we want to be able to quickly\n+\t * find renames for these files too.\n+\t *\n+\t * The reason basename comparisons are a useful heuristic was that it\n+\t * is common for people to move files across directories while keeping\n+\t * their filename the same. If we had a way of determining or even\n+\t * making a good educated guess about which directory these non-unique\n+\t * basename files had moved the file to, we could check it.\n+\t * Luckily...\n+\t *\n+\t * When an entire directory is in fact renamed, we have two factors\n+\t * helping us out:\n+\t * (a) the original directory disappeared giving us a hint\n+\t * about when we can apply an extra heuristic.\n+\t * (a) we often have several files within that directory and\n+\t * subdirectories that are renamed without changes\n+\t * So, rules for a heuristic:\n+\t * (0) If there basename matches are non-unique (the condition under\n+\t * which this function is called) AND\n+\t * (1) the directory in which the file was found has disappeared\n+\t * (i.e. dirs_removed is non-NULL and has a relevant entry) THEN\n+\t * (2) use exact renames of files within the directory to determine\n+\t * where the directory is likely to have been renamed to. IF\n+\t * there is at least one exact rename from within that\n+\t * directory, we can proceed.\n+\t * (3) If there are multiple places the directory could have been\n+\t * renamed to based on exact renames, ignore all but one of them.\n+\t * Just use the destination with the most renames going to it.\n+\t * (4) Check if applying that directory rename to the original file\n+\t * would result in a destination filename that is in the\n+\t * potential rename set. If so, return the index of the\n+\t * destination file (the index within rename_dst).\n+\t * (5) Compare the original file and returned destination for\n+\t * similarity, and if they are sufficiently similar, record the\n+\t * rename.\n+\t *\n+\t * This function, idx_possible_rename(), is only responsible for (4).\n+\t * The conditions/steps in (1)-(3) will be handled via setting up\n+\t * dir_rename_count and dir_rename_guess in a future\n+\t * initialize_dir_rename_info() function. Steps (0) and (5) are\n+\t * handled by the caller of this function.\n+\t */\n+\tchar *old_dir, *new_dir;\n+\tstruct strbuf new_path = STRBUF_INIT;\n+\tint idx;\n+\n+\tif (!info->setup)\n+\t\treturn -1;\n+\n+\told_dir = get_dirname(filename);\n+\tnew_dir = strmap_get(&info->dir_rename_guess, old_dir);\n+\tfree(old_dir);\n+\tif (!new_dir)\n+\t\treturn -1;\n+\n+\tstrbuf_addstr(&new_path, new_dir);\n+\tstrbuf_addch(&new_path, '/');\n+\tstrbuf_addstr(&new_path, get_basename(filename));\n+\n+\tidx = strintmap_get(&info->idx_map, new_path.buf);\n+\tstrbuf_release(&new_path);\n+\treturn idx;\n }\n \n static int find_basename_matches(struct diff_options *options,\n-\t\t\t\t int minimum_score)\n+\t\t\t\t int minimum_score,\n+\t\t\t\t struct dir_rename_info *info)\n {\n \t/*\n \t * When I checked in early 2020, over 76% of file renames in linux\n@@ -494,7 +579,7 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\tdst_index = strintmap_get(&dests, base);\n \t\t\tif (src_index == -1 || dst_index == -1) {\n \t\t\t\tsrc_index = i;\n-\t\t\t\tdst_index = idx_possible_rename(filename);\n+\t\t\t\tdst_index = idx_possible_rename(filename, info);\n \t\t\t}\n \t\t\tif (dst_index == -1)\n \t\t\t\tcontinue;\n@@ -677,8 +762,10 @@ void diffcore_rename(struct diff_options *options)\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct dir_rename_info info;\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n+\tinfo.setup = 0;\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n@@ -774,7 +861,8 @@ void diffcore_rename(struct diff_options *options)\n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n-\t\t\t\t\t\t min_basename_score);\n+\t\t\t\t\t\t min_basename_score,\n+\t\t\t\t\t\t &info);\n \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n \n \t\t/*\n-- \ngitgitgadget\n\n"},{"id":"417858","messageId":"0e14961574eab3ccb3328df79aac9143af5c2278.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 03/10] diffcore-rename: add a mapping of destination names to their indices","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:12Z","receivedAt":"2021-02-26T01:59:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCompute a mapping of full filename to the index within rename_dst where\nthat filename is found, and store it in idx_map. idx_possible_rename()\nneeds this to quickly finding an array entry in rename_dst given the\npathname.\n\nWhile at it, add placeholder initializations for dir_rename_count and\ndir_rename_guess; these will be more fully populated in subsequent\ncommits.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex edb0effb6ef4..8eeb8c73664c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -380,6 +380,45 @@ static char *get_dirname(const char *filename)\n \treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n }\n \n+static void initialize_dir_rename_info(struct dir_rename_info *info)\n+{\n+\tint i;\n+\n+\tinfo->setup = 1;\n+\n+\tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n+\tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n+\tinfo->dir_rename_count = NULL;\n+\n+\t/*\n+\t * Loop setting up both info->idx_map.\n+\t */\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\t/*\n+\t\t * For non-renamed files, make idx_map contain mapping of\n+\t\t * filename -> index (index within rename_dst, that is)\n+\t\t */\n+\t\tif (!rename_dst[i].is_rename) {\n+\t\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\t\tstrintmap_set(&info->idx_map, filename, i);\n+\t\t}\n+\t}\n+}\n+\n+static void cleanup_dir_rename_info(struct dir_rename_info *info)\n+{\n+\tif (!info->setup)\n+\t\treturn;\n+\n+\t/* idx_map */\n+\tstrintmap_clear(&info->idx_map);\n+\n+\t/* dir_rename_guess */\n+\tstrmap_clear(&info->dir_rename_guess, 1);\n+\n+\t/* Nothing to do for dir_rename_count, yet */\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\n@@ -858,6 +897,11 @@ void diffcore_rename(struct diff_options *options)\n \t\tremove_unneeded_paths_from_src(want_copies);\n \t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n \n+\t\t/* Preparation for basename-driven matching. */\n+\t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n+\t\tinitialize_dir_rename_info(&info);\n+\t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n+\n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n@@ -1026,6 +1070,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tif (rename_dst[i].filespec_to_free)\n \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n \n+\tcleanup_dir_rename_info(&info);\n \tFREE_AND_NULL(rename_dst);\n \trename_dst_nr = rename_dst_alloc = 0;\n \tFREE_AND_NULL(rename_src);\n-- \ngitgitgadget\n\n"},{"id":"417859","messageId":"9b9d5b207b034ec4181020e0bf3ac46561f55b28.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 04/10] Move computation of dir_rename_count from merge-ort to diffcore-rename","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:13Z","receivedAt":"2021-02-26T01:59:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMove the computation of dir_rename_count from merge-ort.c to\ndiffcore-rename.c, making slight adjustments to the data structures\nbased on the move. While the diffstat looks large, viewing this commit\nwith --color-moved makes it clear that only about 20 lines changed.\n\nWith this patch, the computation of dir_rename_count is still only done\nafter inexact rename detection, but subsequent commits will add a\npreliminary computation of dir_rename_count after exact rename\ndetection, followed by some updates after inexact rename detection.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 138 +++++++++++++++++++++++++++++++++++++++++++++-\n diffcore.h | 5 ++\n merge-ort.c | 132 +-------------------------------------------\n 3 files changed, 145 insertions(+), 130 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 8eeb8c73664c..39e23d57e7bc 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -380,6 +380,129 @@ static char *get_dirname(const char *filename)\n \treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n }\n \n+static void dirname_munge(char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\tif (!slash)\n+\t\tslash = filename;\n+\t*slash = '\\0';\n+}\n+\n+static void increment_count(struct strmap *dir_rename_count,\n+\t\t\t char *old_dir,\n+\t\t\t char *new_dir)\n+{\n+\tstruct strintmap *counts;\n+\tstruct strmap_entry *e;\n+\n+\t/* Get the {new_dirs -> counts} mapping using old_dir */\n+\te = strmap_get_entry(dir_rename_count, old_dir);\n+\tif (e) {\n+\t\tcounts = e->value;\n+\t} else {\n+\t\tcounts = xmalloc(sizeof(*counts));\n+\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n+\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t}\n+\n+\t/* Increment the count for new_dir */\n+\tstrintmap_incr(counts, new_dir, 1);\n+}\n+\n+static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t const char *oldname,\n+\t\t\t\t const char *newname)\n+{\n+\tchar *old_dir = xstrdup(oldname);\n+\tchar *new_dir = xstrdup(newname);\n+\tchar new_dir_first_char = new_dir[0];\n+\tint first_time_in_loop = 1;\n+\n+\twhile (1) {\n+\t\tdirname_munge(old_dir);\n+\t\tdirname_munge(new_dir);\n+\n+\t\t/*\n+\t\t * When renaming\n+\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n+\t\t * then this suggests that both\n+\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n+\t\t * a/b/c/d/ => a/b/some/thing/else/\n+\t\t * so we want to increment counters for both. We do NOT,\n+\t\t * however, also want to suggest that there was the following\n+\t\t * rename:\n+\t\t * a/b/c/ => a/b/some/thing/\n+\t\t * so we need to quit at that point.\n+\t\t *\n+\t\t * Note the when first_time_in_loop, we only strip off the\n+\t\t * basename, and we don't care if that's different.\n+\t\t */\n+\t\tif (!first_time_in_loop) {\n+\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n+\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n+\t\t\tif (!*new_dir) {\n+\t\t\t\t/*\n+\t\t\t\t * Special case when renaming to root directory,\n+\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n+\t\t\t\t * something like\n+\t\t\t\t * a/b/subdir => subdir\n+\t\t\t\t * and so dirname_munge() sets things up so that\n+\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n+\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n+\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n+\t\t\t\t * in new_dir, so we have to compare differently.\n+\t\t\t\t */\n+\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n+\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t} else {\n+\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strset_contains(dirs_removed, old_dir))\n+\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\telse\n+\t\t\tbreak;\n+\n+\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n+\t\tif (!*old_dir || !*new_dir)\n+\t\t\tbreak;\n+\n+\t\tfirst_time_in_loop = 0;\n+\t}\n+\n+\t/* Free resources we don't need anymore */\n+\tfree(old_dir);\n+\tfree(new_dir);\n+}\n+\n+static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed)\n+{\n+\tint i;\n+\n+\t/* Set up dir_rename_count */\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\t/* File not part of directory rename counts if not a rename */\n+\t\tif (!rename_dst[i].is_rename)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * Make dir_rename_count contain a map of a map:\n+\t\t * old_directory -> {new_directory -> count}\n+\t\t * In other words, for every pair look at the directories for\n+\t\t * the old filename and the new filename and count how many\n+\t\t * times that pairing occurs.\n+\t\t */\n+\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\t\t\t\t rename_dst[i].p->one->path,\n+\t\t\t\t\t rename_dst[i].p->two->path);\n+\t}\n+}\n+\n static void initialize_dir_rename_info(struct dir_rename_info *info)\n {\n \tint i;\n@@ -790,7 +913,9 @@ static void remove_unneeded_paths_from_src(int detecting_copies)\n \trename_src_nr = new_num_src;\n }\n \n-void diffcore_rename(struct diff_options *options)\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count)\n {\n \tint detect_rename = options->detect_rename;\n \tint minimum_score = options->rename_score;\n@@ -805,6 +930,7 @@ void diffcore_rename(struct diff_options *options)\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n \tinfo.setup = 0;\n+\tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n@@ -999,6 +1125,11 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n+\t/*\n+\t * Now that renames have been computed, compute dir_rename_count */\n+\tif (dirs_removed && dir_rename_count)\n+\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n@@ -1082,3 +1213,8 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"write back to queue\", options->repo);\n \treturn;\n }\n+\n+void diffcore_rename(struct diff_options *options)\n+{\n+\tdiffcore_rename_extended(options, NULL, NULL);\n+}\ndiff --git a/diffcore.h b/diffcore.h\nindex d2a63c5c71f4..db55d3853071 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -8,6 +8,8 @@\n \n struct diff_options;\n struct repository;\n+struct strmap;\n+struct strset;\n struct userdiff_driver;\n \n /* This header file is internal between diff.c and its diff transformers\n@@ -161,6 +163,9 @@ void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count);\n void diffcore_merge_broken(void);\n void diffcore_pickaxe(struct diff_options *);\n void diffcore_order(const char *orderfile);\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 603d30c52170..c4467e073b45 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1302,131 +1302,6 @@ static char *handle_path_level_conflicts(struct merge_options *opt,\n \treturn new_path;\n }\n \n-static void dirname_munge(char *filename)\n-{\n-\tchar *slash = strrchr(filename, '/');\n-\tif (!slash)\n-\t\tslash = filename;\n-\t*slash = '\\0';\n-}\n-\n-static void increment_count(struct strmap *dir_rename_count,\n-\t\t\t char *old_dir,\n-\t\t\t char *new_dir)\n-{\n-\tstruct strintmap *counts;\n-\tstruct strmap_entry *e;\n-\n-\t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n-\tif (e) {\n-\t\tcounts = e->value;\n-\t} else {\n-\t\tcounts = xmalloc(sizeof(*counts));\n-\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n-\t}\n-\n-\t/* Increment the count for new_dir */\n-\tstrintmap_incr(counts, new_dir, 1);\n-}\n-\n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t const char *oldname,\n-\t\t\t\t const char *newname)\n-{\n-\tchar *old_dir = xstrdup(oldname);\n-\tchar *new_dir = xstrdup(newname);\n-\tchar new_dir_first_char = new_dir[0];\n-\tint first_time_in_loop = 1;\n-\n-\twhile (1) {\n-\t\tdirname_munge(old_dir);\n-\t\tdirname_munge(new_dir);\n-\n-\t\t/*\n-\t\t * When renaming\n-\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n-\t\t * then this suggests that both\n-\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n-\t\t * a/b/c/d/ => a/b/some/thing/else/\n-\t\t * so we want to increment counters for both. We do NOT,\n-\t\t * however, also want to suggest that there was the following\n-\t\t * rename:\n-\t\t * a/b/c/ => a/b/some/thing/\n-\t\t * so we need to quit at that point.\n-\t\t *\n-\t\t * Note the when first_time_in_loop, we only strip off the\n-\t\t * basename, and we don't care if that's different.\n-\t\t */\n-\t\tif (!first_time_in_loop) {\n-\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n-\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n-\t\t\tif (!*new_dir) {\n-\t\t\t\t/*\n-\t\t\t\t * Special case when renaming to root directory,\n-\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n-\t\t\t\t * something like\n-\t\t\t\t * a/b/subdir => subdir\n-\t\t\t\t * and so dirname_munge() sets things up so that\n-\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n-\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n-\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n-\t\t\t\t * in new_dir, so we have to compare differently.\n-\t\t\t\t */\n-\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n-\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t} else {\n-\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n-\t\telse\n-\t\t\tbreak;\n-\n-\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n-\t\tif (!*old_dir || !*new_dir)\n-\t\t\tbreak;\n-\n-\t\tfirst_time_in_loop = 0;\n-\t}\n-\n-\t/* Free resources we don't need anymore */\n-\tfree(old_dir);\n-\tfree(new_dir);\n-}\n-\n-static void compute_rename_counts(struct diff_queue_struct *pairs,\n-\t\t\t\t struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pairs->nr; ++i) {\n-\t\tstruct diff_filepair *pair = pairs->queue[i];\n-\n-\t\t/* File not part of directory rename if it wasn't renamed */\n-\t\tif (pair->status != 'R')\n-\t\t\tcontinue;\n-\n-\t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n-\t\t * old_directory -> {new_directory -> count}\n-\t\t * In other words, for every pair look at the directories for\n-\t\t * the old filename and the new filename and count how many\n-\t\t * times that pairing occurs.\n-\t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n-\t\t\t\t\t pair->one->path,\n-\t\t\t\t\t pair->two->path);\n-\t}\n-}\n-\n static void get_provisional_directory_renames(struct merge_options *opt,\n \t\t\t\t\t unsigned side,\n \t\t\t\t\t int *clean)\n@@ -1435,9 +1310,6 @@ static void get_provisional_directory_renames(struct merge_options *opt,\n \tstruct strmap_entry *entry;\n \tstruct rename_info *renames = &opt->priv->renames;\n \n-\tcompute_rename_counts(&renames->pairs[side],\n-\t\t\t &renames->dir_rename_count[side],\n-\t\t\t &renames->dirs_removed[side]);\n \t/*\n \t * Collapse\n \t * dir_rename_count: old_directory -> {new_directory -> count}\n@@ -2162,7 +2034,9 @@ static void detect_regular_renames(struct merge_options *opt,\n \n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n-\tdiffcore_rename(&diff_opts);\n+\tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t &renames->dirs_removed[side_index],\n+\t\t\t\t &renames->dir_rename_count[side_index]);\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n \tresolve_diffpair_statuses(&diff_queued_diff);\n \n-- \ngitgitgadget\n\n"},{"id":"417860","messageId":"f286e89464ea83c1972eb93ca04e54f9c0568a53.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 05/10] diffcore-rename: add function for clearing dir_rename_count","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:14Z","receivedAt":"2021-02-26T01:59:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs we adjust the usage of dir_rename_count we want to have a function\nfor clearing, or partially clearing it out. Add a\npartial_clear_dir_rename_count() function for this purpose.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 12 ++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 12 +++---------\n 3 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 39e23d57e7bc..7dd475ff9a9f 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -528,6 +528,18 @@ static void initialize_dir_rename_info(struct dir_rename_info *info)\n \t}\n }\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrmap_for_each_entry(dir_rename_count, &iter, entry) {\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tstrintmap_clear(counts);\n+\t}\n+\tstrmap_partial_clear(dir_rename_count, 1);\n+}\n+\n static void cleanup_dir_rename_info(struct dir_rename_info *info)\n {\n \tif (!info->setup)\ndiff --git a/diffcore.h b/diffcore.h\nindex db55d3853071..c6ba64abd198 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -161,6 +161,8 @@ struct diff_filepair *diff_queue(struct diff_queue_struct *,\n \t\t\t\t struct diff_filespec *);\n void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n+\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex c4467e073b45..467404cc0a35 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -351,17 +351,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \n \t/* Free memory used by various renames maps */\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstruct hashmap_iter iter;\n-\t\tstruct strmap_entry *entry;\n-\n \t\tstrset_func(&renames->dirs_removed[i]);\n \n-\t\tstrmap_for_each_entry(&renames->dir_rename_count[i],\n-\t\t\t\t &iter, entry) {\n-\t\t\tstruct strintmap *counts = entry->value;\n-\t\t\tstrintmap_clear(counts);\n-\t\t}\n-\t\tstrmap_func(&renames->dir_rename_count[i], 1);\n+\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n+\t\tif (!reinitialize)\n+\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \n \t\tstrmap_func(&renames->dir_renames[i], 0);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"417861","messageId":"ab353f2e75ebe4744cd3eb65f74f1242ba53e989.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 06/10] diffcore-rename: move dir_rename_counts into dir_rename_info struct","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:15Z","receivedAt":"2021-02-26T01:59:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis continues the migration of the directory rename detection code into\ndiffcore-rename, now taking the simple step of combining it with the\ndir_rename_info struct. Future commits will then make dir_rename_counts\nbe computed in stages, and add computation of dir_rename_guess.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 27 ++++++++++++++++-----------\n 1 file changed, 16 insertions(+), 11 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 7dd475ff9a9f..a1ccf14001f5 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -388,7 +388,7 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n-static void increment_count(struct strmap *dir_rename_count,\n+static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n {\n@@ -396,20 +396,20 @@ static void increment_count(struct strmap *dir_rename_count,\n \tstruct strmap_entry *e;\n \n \t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n+\te = strmap_get_entry(info->dir_rename_count, old_dir);\n \tif (e) {\n \t\tcounts = e->value;\n \t} else {\n \t\tcounts = xmalloc(sizeof(*counts));\n \t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t\tstrmap_put(info->dir_rename_count, old_dir, counts);\n \t}\n \n \t/* Increment the count for new_dir */\n \tstrintmap_incr(counts, new_dir, 1);\n }\n \n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t const char *oldname,\n \t\t\t\t const char *newname)\n@@ -463,7 +463,7 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t}\n \n \t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\t\tincrement_count(info, old_dir, new_dir);\n \t\telse\n \t\t\tbreak;\n \n@@ -479,12 +479,15 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n+static void compute_dir_rename_counts(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n-\t/* Set up dir_rename_count */\n+\tinfo->setup = 1;\n+\tinfo->dir_rename_count = dir_rename_count;\n+\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/* File not part of directory rename counts if not a rename */\n \t\tif (!rename_dst[i].is_rename)\n@@ -497,7 +500,7 @@ static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t * the old filename and the new filename and count how many\n \t\t * times that pairing occurs.\n \t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n@@ -551,7 +554,9 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info)\n \t/* dir_rename_guess */\n \tstrmap_clear(&info->dir_rename_guess, 1);\n \n-\t/* Nothing to do for dir_rename_count, yet */\n+\t/* dir_rename_count */\n+\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\tstrmap_clear(info->dir_rename_count, 1);\n }\n \n static const char *get_basename(const char *filename)\n@@ -1140,7 +1145,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/*\n \t * Now that renames have been computed, compute dir_rename_count */\n \tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n \n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n-- \ngitgitgadget\n\n"},{"id":"417862","messageId":"bd50d9e53804c0cf9b0a428455e9eeea100f26b2.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 07/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:16Z","receivedAt":"2021-02-26T02:00:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen diffcore_rename_extended() is passed a NULL dir_rename_count, we\nwill still want to create a temporary one for use by\nfind_basename_matches(), but have it fully deallocated before\ndiffcore_rename_extended() returns. However, when\ndiffcore_rename_extended() is passed a dir_rename_count, we want to fill\nthat strmap with appropriate values and return it. However, for our\ninterim purposes we may also add entries corresponding to directories\nthat cannot have been renamed due to still existing on both sides.\n\nExtend cleanup_dir_rename_info() to handle these two different cases,\ncleaning up the relevant bits of information for each case.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 40 ++++++++++++++++++++++++++++++++++++----\n 1 file changed, 36 insertions(+), 4 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex a1ccf14001f5..2cf9c47c6364 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -543,8 +543,15 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n \tstrmap_partial_clear(dir_rename_count, 1);\n }\n \n-static void cleanup_dir_rename_info(struct dir_rename_info *info)\n+static void cleanup_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t int keep_dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n+\tint i;\n+\n \tif (!info->setup)\n \t\treturn;\n \n@@ -555,8 +562,33 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info)\n \tstrmap_clear(&info->dir_rename_guess, 1);\n \n \t/* dir_rename_count */\n-\tpartial_clear_dir_rename_count(info->dir_rename_count);\n-\tstrmap_clear(info->dir_rename_count, 1);\n+\tif (!keep_dir_rename_count) {\n+\t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\t\tstrmap_clear(info->dir_rename_count, 1);\n+\t\tFREE_AND_NULL(info->dir_rename_count);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Although dir_rename_count was passed in\n+\t * diffcore_rename_extended() and we want to keep it around and\n+\t * return it to that caller, we first want to remove any data\n+\t * associated with directories that weren't renamed.\n+\t */\n+\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\tconst char *source_dir = entry->key;\n+\t\tstruct strintmap *counts = entry->value;\n+\n+\t\tif (!strset_contains(dirs_removed, source_dir)) {\n+\t\t\tstring_list_append(&to_remove, source_dir);\n+\t\t\tstrintmap_clear(counts);\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\tfor (i = 0; i < to_remove.nr; ++i)\n+\t\tstrmap_remove(info->dir_rename_count,\n+\t\t\t to_remove.items[i].string, 1);\n+\tstring_list_clear(&to_remove, 0);\n }\n \n static const char *get_basename(const char *filename)\n@@ -1218,7 +1250,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tif (rename_dst[i].filespec_to_free)\n \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n \n-\tcleanup_dir_rename_info(&info);\n+\tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n \trename_dst_nr = rename_dst_alloc = 0;\n \tFREE_AND_NULL(rename_src);\n-- \ngitgitgadget\n\n"},{"id":"417863","messageId":"752aff3a799544e398c4b3ff10a6b1b03dec524a.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 09/10] diffcore-rename: limit dir_rename_counts computation to relevant dirs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:18Z","receivedAt":"2021-02-26T02:00:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe are using dir_rename_counts to count the number of other directories\nthat files within a directory moved to. We only need this information\nfor directories that disappeared, though, so we can return early from\nupdate_dir_rename_counts() for other paths.\n\nIf dirs_removed is passed to diffcore_rename_extended(), then it\nprovides the relevant bits of information for us to limit this counting\nto relevant dirs. If dirs_removed is not passed, we would need to\ncompute some replacement in order to do this limiting. Introduce a new\ninfo->relevant_source_dirs variable for this purpose, even though at\nthis stage we will only set it to dirs_removed for simplicity.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 10f8f4a301e3..e5fa0cb555dd 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -371,6 +371,7 @@ struct dir_rename_info {\n \tstruct strintmap idx_map;\n \tstruct strmap dir_rename_guess;\n \tstruct strmap *dir_rename_count;\n+\tstruct strset *relevant_source_dirs;\n \tunsigned setup;\n };\n \n@@ -442,7 +443,13 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\treturn;\n \n \twhile (1) {\n+\t\t/* Get old_dir, skip if its directory isn't relevant. */\n \t\tdirname_munge(old_dir);\n+\t\tif (info->relevant_source_dirs &&\n+\t\t !strset_contains(info->relevant_source_dirs, old_dir))\n+\t\t\tbreak;\n+\n+\t\t/* Get new_dir */\n \t\tdirname_munge(new_dir);\n \n \t\t/*\n@@ -521,6 +528,9 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n \n+\t/* Setup info->relevant_source_dirs */\n+\tinfo->relevant_source_dirs = dirs_removed;\n+\n \t/*\n \t * Loop setting up both info->idx_map, and doing setup of\n \t * info->dir_rename_count.\n-- \ngitgitgadget\n\n"},{"id":"417864","messageId":"65f7bfb735f21ff774938b49ba500bf81265db94.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 10/10] diffcore-rename: compute dir_rename_guess from dir_rename_counts","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:19Z","receivedAt":"2021-02-26T02:00:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndir_rename_counts has a mapping of a mapping, in particular, it has\n old_dir => { new_dir => count }\nWe want a simple mapping of\n old_dir => new_dir\nbased on which new_dir had the highest count for a given old_dir.\nCompute this and store it in dir_rename_guess.\n\nThis is the final piece of the puzzle needed to make our guesses at\nwhich directory files have been moved to when basenames aren't unique.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 45 +++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 41 insertions(+), 4 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex e5fa0cb555dd..1fe902ed2af0 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -389,6 +389,24 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n+static const char *get_highest_rename_path(struct strintmap *counts)\n+{\n+\tint highest_count = 0;\n+\tconst char *highest_destination_dir = NULL;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrintmap_for_each_entry(counts, &iter, entry) {\n+\t\tconst char *destination_dir = entry->key;\n+\t\tintptr_t count = (intptr_t)entry->value;\n+\t\tif (count > highest_count) {\n+\t\t\thighest_count = count;\n+\t\t\thighest_destination_dir = destination_dir;\n+\t\t}\n+\t}\n+\treturn highest_destination_dir;\n+}\n+\n static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n@@ -512,6 +530,8 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t struct strmap *dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n \tint i;\n \n \tif (!dirs_removed) {\n@@ -558,6 +578,23 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n+\n+\t/*\n+\t * Now we collapse\n+\t * dir_rename_count: old_directory -> {new_directory -> count}\n+\t * down to\n+\t * dir_rename_guess: old_directory -> best_new_directory\n+\t * where best_new_directory is the one with the highest count.\n+\t */\n+\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\t/* entry->key is source_dir */\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tchar *best_newdir;\n+\n+\t\tbest_newdir = xstrdup(get_highest_rename_path(counts));\n+\t\tstrmap_put(&info->dir_rename_guess, entry->key,\n+\t\t\t best_newdir);\n+\t}\n }\n \n void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n@@ -682,10 +719,10 @@ static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n \t * rename.\n \t *\n \t * This function, idx_possible_rename(), is only responsible for (4).\n-\t * The conditions/steps in (1)-(3) will be handled via setting up\n-\t * dir_rename_count and dir_rename_guess in a future\n-\t * initialize_dir_rename_info() function. Steps (0) and (5) are\n-\t * handled by the caller of this function.\n+\t * The conditions/steps in (1)-(3) are handled via setting up\n+\t * dir_rename_count and dir_rename_guess in\n+\t * initialize_dir_rename_info(). Steps (0) and (5) are handled by\n+\t * the caller of this function.\n \t */\n \tchar *old_dir, *new_dir;\n \tstruct strbuf new_path = STRBUF_INIT;\n-- \ngitgitgadget\n"},{"id":"417865","messageId":"44cfae6505f270575185e6ddeb0696c1e6c9bc20.1614304700.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v3 08/10] diffcore-rename: compute dir_rename_counts in stages","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-26T01:58:17Z","receivedAt":"2021-02-26T02:00:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCompute dir_rename_counts based just on exact renames to start, as that\ncan provide us useful information in find_basename_matches(). This is\ndone by moving the code from compute_dir_rename_counts() into\ninitialize_dir_rename_info(), resulting in it being computed earlier and\nbased just on exact renames. Since that's an incomplete result, we\naugment the counts via calling update_dir_rename_counts() after each\nbasename-guide and inexact rename detection match is found.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 110 +++++++++++++++++++++++++++++-----------------\n 1 file changed, 70 insertions(+), 40 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 2cf9c47c6364..10f8f4a301e3 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -419,6 +419,28 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tchar new_dir_first_char = new_dir[0];\n \tint first_time_in_loop = 1;\n \n+\tif (!info->setup)\n+\t\t/*\n+\t\t * info->setup is 0 here in two cases: (1) all auxiliary\n+\t\t * vars (like dirs_removed) were NULL so\n+\t\t * initialize_dir_rename_info() returned early, or (2)\n+\t\t * either break detection or copy detection are active so\n+\t\t * that we never called initialize_dir_rename_info(). In\n+\t\t * the former case, we don't have enough info to know if\n+\t\t * directories were renamed (because dirs_removed lets us\n+\t\t * know about a necessary prerequisite, namely if they were\n+\t\t * removed), and in the latter, we don't care about\n+\t\t * directory renames or find_basename_matches.\n+\t\t *\n+\t\t * This matters because both basename and inexact matching\n+\t\t * will also call update_dir_rename_counts(). In either of\n+\t\t * the above two cases info->dir_rename_counts will not\n+\t\t * have been properly initialized which prevents us from\n+\t\t * updating it, but in these two cases we don't care about\n+\t\t * dir_rename_counts anyway, so we can just exit early.\n+\t\t */\n+\t\treturn;\n+\n \twhile (1) {\n \t\tdirname_munge(old_dir);\n \t\tdirname_munge(new_dir);\n@@ -479,45 +501,29 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct dir_rename_info *info,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t struct strmap *dir_rename_count)\n+static void initialize_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n-\tinfo->setup = 1;\n-\tinfo->dir_rename_count = dir_rename_count;\n-\n-\tfor (i = 0; i < rename_dst_nr; ++i) {\n-\t\t/* File not part of directory rename counts if not a rename */\n-\t\tif (!rename_dst[i].is_rename)\n-\t\t\tcontinue;\n-\n-\t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n-\t\t * old_directory -> {new_directory -> count}\n-\t\t * In other words, for every pair look at the directories for\n-\t\t * the old filename and the new filename and count how many\n-\t\t * times that pairing occurs.\n-\t\t */\n-\t\tupdate_dir_rename_counts(info, dirs_removed,\n-\t\t\t\t\t rename_dst[i].p->one->path,\n-\t\t\t\t\t rename_dst[i].p->two->path);\n+\tif (!dirs_removed) {\n+\t\tinfo->setup = 0;\n+\t\treturn;\n \t}\n-}\n-\n-static void initialize_dir_rename_info(struct dir_rename_info *info)\n-{\n-\tint i;\n-\n \tinfo->setup = 1;\n \n+\tinfo->dir_rename_count = dir_rename_count;\n+\tif (!info->dir_rename_count) {\n+\t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n+\t\tstrmap_init(info->dir_rename_count);\n+\t}\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n-\tinfo->dir_rename_count = NULL;\n \n \t/*\n-\t * Loop setting up both info->idx_map.\n+\t * Loop setting up both info->idx_map, and doing setup of\n+\t * info->dir_rename_count.\n \t */\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n@@ -527,7 +533,20 @@ static void initialize_dir_rename_info(struct dir_rename_info *info)\n \t\tif (!rename_dst[i].is_rename) {\n \t\t\tchar *filename = rename_dst[i].p->two->path;\n \t\t\tstrintmap_set(&info->idx_map, filename, i);\n+\t\t\tcontinue;\n \t\t}\n+\n+\t\t/*\n+\t\t * For everything else (i.e. renamed files), make\n+\t\t * dir_rename_count contain a map of a map:\n+\t\t * old_directory -> {new_directory -> count}\n+\t\t * In other words, for every pair look at the directories for\n+\t\t * the old filename and the new filename and count how many\n+\t\t * times that pairing occurs.\n+\t\t */\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t rename_dst[i].p->one->path,\n+\t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n }\n \n@@ -682,7 +701,8 @@ static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n \n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n-\t\t\t\t struct dir_rename_info *info)\n+\t\t\t\t struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed)\n {\n \t/*\n \t * When I checked in early 2020, over 76% of file renames in linux\n@@ -810,6 +830,8 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\tcontinue;\n \t\t\trecord_rename_pair(dst_index, src_index, score);\n \t\t\trenames++;\n+\t\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t\t one->path, two->path);\n \n \t\t\t/*\n \t\t\t * Found a rename so don't need text anymore; if we\n@@ -893,7 +915,12 @@ static int too_many_rename_candidates(int num_destinations, int num_sources,\n \treturn 1;\n }\n \n-static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, int copies)\n+static int find_renames(struct diff_score *mx,\n+\t\t\tint dst_cnt,\n+\t\t\tint minimum_score,\n+\t\t\tint copies,\n+\t\t\tstruct dir_rename_info *info,\n+\t\t\tstruct strset *dirs_removed)\n {\n \tint count = 0, i;\n \n@@ -910,6 +937,9 @@ static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, i\n \t\t\tcontinue;\n \t\trecord_rename_pair(mx[i].dst, mx[i].src, mx[i].score);\n \t\tcount++;\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t rename_src[mx[i].src].p->one->path,\n+\t\t\t\t\t rename_dst[mx[i].dst].p->two->path);\n \t}\n \treturn count;\n }\n@@ -981,6 +1011,8 @@ void diffcore_rename_extended(struct diff_options *options,\n \tinfo.setup = 0;\n \tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n+\tif (dirs_removed && (break_idx || want_copies))\n+\t\tBUG(\"dirs_removed incompatible with break/copy detection\");\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n \n@@ -1074,14 +1106,15 @@ void diffcore_rename_extended(struct diff_options *options,\n \n \t\t/* Preparation for basename-driven matching. */\n \t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n-\t\tinitialize_dir_rename_info(&info);\n+\t\tinitialize_dir_rename_info(&info,\n+\t\t\t\t\t dirs_removed, dir_rename_count);\n \t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n \n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n \t\t\t\t\t\t min_basename_score,\n-\t\t\t\t\t\t &info);\n+\t\t\t\t\t\t &info, dirs_removed);\n \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n \n \t\t/*\n@@ -1167,18 +1200,15 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/* cost matrix sorted by most to least similar pair */\n \tSTABLE_QSORT(mx, dst_cnt * NUM_CANDIDATE_PER_DST, score_compare);\n \n-\trename_count += find_renames(mx, dst_cnt, minimum_score, 0);\n+\trename_count += find_renames(mx, dst_cnt, minimum_score, 0,\n+\t\t\t\t &info, dirs_removed);\n \tif (want_copies)\n-\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1);\n+\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1,\n+\t\t\t\t\t &info, dirs_removed);\n \tfree(mx);\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n-\t/*\n-\t * Now that renames have been computed, compute dir_rename_count */\n-\tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n-\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n-- \ngitgitgadget\n\n"},{"id":"417896","messageId":"9edad7cc-f578-3b55-394a-08b351eaa7c5@gmail.com","threadId":"55146","inReplyTo":"40f57bcc2055773a7dd35bde50c54354a643d07a.1614304700.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/10] diffcore-rename: add a new idx_possible_rename function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-26T15:52:05Z","receivedAt":"2021-02-26T15:53:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/25/2021 8:58 PM, Elijah Newren via GitGitGadget wrote:\n> Anyway, that's a lot of background so that we can actually describe the\n> new function. Add an idx_possible_rename() function which combines the\n> recently added dir_rename_guess and idx_map fields to provide the index\n> within rename_dst of a potential match for a given file.\n\nThis paragraph could use rewording now that idx_possible_rename() is not\nnew to this commit, and dir_rename_guess() is introduced here (but not\nyet initialized). This would be a good time to introduce the\ndir_rename_info struct and how it _will_ be populated.\n\n> -static int idx_possible_rename(char *filename)\n> +static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n> {\n...\n> +\tif (!info->setup)\n> +\t\treturn -1;\n\nIt might be worth noting in the commit message that since\ninfo.setup is always zero, this implementation is not run (yet).\n\nIn general, I'm finding this a lot easier to read the code in\nthe presented order. The commit messages need to be massaged to\nmatch.\n\nThanks,\n-Stolee\n\n"},{"id":"417897","messageId":"7cdfa81a-d088-f91d-c78d-6f1d957ec8b2@gmail.com","threadId":"55146","inReplyTo":"9b9d5b207b034ec4181020e0bf3ac46561f55b28.1614304700.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 04/10] Move computation of dir_rename_count from merge-ort to diffcore-rename","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-26T15:55:58Z","receivedAt":"2021-02-26T15:56:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/25/2021 8:58 PM, Elijah Newren via GitGitGadget wrote:\n> +\tfor (i = 0; i < rename_dst_nr; ++i) {\n> +\t\t/* File not part of directory rename counts if not a rename */\n> +\t\tif (!rename_dst[i].is_rename)\n> +\t\t\tcontinue;\n> +\n...\n> -\t\t/* File not part of directory rename if it wasn't renamed */\n> -\t\tif (pair->status != 'R')\n> -\t\t\tcontinue;\n> -\nThanks for updating to include this check!\n\n-Stolee\n"},{"id":"417902","messageId":"8efa1f5e-b072-abe3-9ad0-c1535d7a8b14@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/10] Optimization batch 8: use file basenames even more","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-26T16:34:35Z","receivedAt":"2021-02-26T16:35:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/25/2021 8:58 PM, Elijah Newren via GitGitGadget wrote:\n> This series depends on en/diffcore-rename (a concatenation of what I was\n> calling ort-perf-batch-6 and ort-perf-batch-7).\n> \n> Changes since v2:\n> \n> * Rearrange the patches in the series to have a top-down ordering rather\n> than bottom-up -- as suggested by Stolee, Ævar, and Junio\n> * Several comments and style improvements suggested by Stolee\n> * Replace xstrfmt() with a few strbuf_add*() calls, as suggested by Stolee\n I like the new layout. The code looks good. The only nit I have is that\nsome of the old commit messages read awkwardly with the new ordering.\nThese are not critical, so this version gets my:\n\nReviewed-by: Derrick Stolee \n\n"},{"id":"417905","messageId":"CABPp-BHEF3YKLEDzccph_J53vjqKja_Z-0tMPif3jZFQJdU5cw@mail.gmail.com","threadId":"55146","inReplyTo":"8efa1f5e-b072-abe3-9ad0-c1535d7a8b14@gmail.com","subject":"Re: [PATCH v3 00/10] Optimization batch 8: use file basenames even more","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-26T19:28:15Z","receivedAt":"2021-02-26T19:29:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 26, 2021 at 8:34 AM Derrick Stolee wrote:\n>\n> On 2/25/2021 8:58 PM, Elijah Newren via GitGitGadget wrote:\n> > This series depends on en/diffcore-rename (a concatenation of what I was\n> > calling ort-perf-batch-6 and ort-perf-batch-7).\n> >\n> > Changes since v2:\n> >\n> > * Rearrange the patches in the series to have a top-down ordering rather\n> > than bottom-up -- as suggested by Stolee, Ævar, and Junio\n> > * Several comments and style improvements suggested by Stolee\n> > * Replace xstrfmt() with a few strbuf_add*() calls, as suggested by Stolee\n\nOh, and I also forgot to mention that I applied Ævar's suggested\ncleanup -- return-in-if-block-to-avoid-indented-else.\n\n> I like the new layout. The code looks good. The only nit I have is that\n> some of the old commit messages read awkwardly with the new ordering.\n\nDoh, I fixed some of those up, but apparently missed some. I'll clean\nthose up too.\n\n> These are not critical, so this version gets my:\n>\n> Reviewed-by: Derrick Stolee \n\nThanks, I'll include that in my reroll with the cleaned up commit messages.\n"},{"id":"417923","messageId":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v3.git.1614304699.gitgitgadget@gmail.com","subject":"[PATCH v4 00/10] Optimization batch 8: use file basenames even more","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:38Z","receivedAt":"2021-02-27T00:31:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on en/diffcore-rename (a concatenation of what I was\ncalling ort-perf-batch-6 and ort-perf-batch-7).\n\nChanges since v3:\n\n * Update the commit messages (one was out of date after the rearrangement),\n and include Stolee's Reviewed-by\n\nElijah Newren (10):\n diffcore-rename: use directory rename guided basename comparisons\n diffcore-rename: provide basic implementation of idx_possible_rename()\n diffcore-rename: add a mapping of destination names to their indices\n Move computation of dir_rename_count from merge-ort to diffcore-rename\n diffcore-rename: add function for clearing dir_rename_count\n diffcore-rename: move dir_rename_counts into dir_rename_info struct\n diffcore-rename: extend cleanup_dir_rename_info()\n diffcore-rename: compute dir_rename_counts in stages\n diffcore-rename: limit dir_rename_counts computation to relevant dirs\n diffcore-rename: compute dir_rename_guess from dir_rename_counts\n\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 449 ++++++++++++++++++++++++++++++++--\n diffcore.h | 7 +\n merge-ort.c | 144 +----------\n 4 files changed, 449 insertions(+), 153 deletions(-)\n\n\nbase-commit: aeca14f748afc7fb5b65bca56ea2ebd970729814\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-844%2Fnewren%2Fort-perf-batch-8-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-844/newren/ort-perf-batch-8-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/844\n\nRange-diff vs v3:\n\n 1: 6afa9add40b9 ! 1: 823d07532e00 diffcore-rename: use directory rename guided basename comparisons\n @@ Commit message\n min_basename_score threshold required for marking the two files as\n renames.\n \n - This commit introduces an idx_possible_rename() function which will give\n + This commit introduces an idx_possible_rename() function which will\n do this directory rename detection for us and give us the index within\n rename_dst of the resulting filename. For now, this function is\n hardcoded to return -1 (not found) and just hooks up how its results\n would be used once we have a more complete implementation in place.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## Documentation/gitdiffcore.txt ##\n 2: 40f57bcc2055 ! 2: 2dde621d7de5 diffcore-rename: add a new idx_possible_rename function\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - diffcore-rename: add a new idx_possible_rename function\n + diffcore-rename: provide basic implementation of idx_possible_rename()\n \n - find_basename_matches() is great when both the remaining set of possible\n - rename sources and the remaining set of possible rename destinations\n - have exactly one file each with a given basename. It allows us to match\n - up files that have been moved to different directories without changing\n - filenames.\n + Add a new struct dir_rename_info with various values we need inside our\n + idx_possible_rename() function introduced in the previous commit. Add a\n + basic implementation for this function showing how we plan to use the\n + variables, but which will just return early with a value of -1 (not\n + found) when those variables are not set up.\n \n - When basenames are not unique, though, we want to be able to guess which\n - directories the source files have been moved to. Since this is the job\n - of directory rename detection, we employ it. However, since it is a\n - directory rename detection idea, we also limit it to cases where we know\n - there could have been a directory rename, i.e. where the source\n - directory has been removed. This has to be signalled by dirs_removed\n - being non-NULL and containing an entry for the relevant directory.\n - Since merge-ort.c is the only caller that currently does so, this\n - optimization is only effective for merge-ort right now. In the future,\n - this condition could be reconsidered or we could modify other callers to\n - pass the necessary strset.\n -\n - Anyway, that's a lot of background so that we can actually describe the\n - new function. Add an idx_possible_rename() function which combines the\n - recently added dir_rename_guess and idx_map fields to provide the index\n - within rename_dst of a potential match for a given file.\n -\n - Future commits will add checks after calling this function to compare\n - the resulting 'likely rename' candidates to see if the two files meet\n - the elevated min_basename_score threshold for marking them as actual\n - renames.\n + Future commits will do the work necessary to set up those other\n + variables so that idx_possible_rename() does not always return -1.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 3: 0e14961574ea ! 3: 21b9cf1da30e diffcore-rename: add a mapping of destination names to their indices\n @@ Commit message\n dir_rename_guess; these will be more fully populated in subsequent\n commits.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 4: 9b9d5b207b03 ! 4: 3617b0209cc4 Move computation of dir_rename_count from merge-ort to diffcore-rename\n @@ Commit message\n preliminary computation of dir_rename_count after exact rename\n detection, followed by some updates after inexact rename detection.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 5: f286e89464ea ! 5: 2baf39d82f3e diffcore-rename: add function for clearing dir_rename_count\n @@ Commit message\n for clearing, or partially clearing it out. Add a\n partial_clear_dir_rename_count() function for this purpose.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 6: ab353f2e75eb ! 6: 02f1f7c02d32 diffcore-rename: move dir_rename_counts into dir_rename_info struct\n @@ Commit message\n dir_rename_info struct. Future commits will then make dir_rename_counts\n be computed in stages, and add computation of dir_rename_guess.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 7: bd50d9e53804 ! 7: 9c3436840534 diffcore-rename: extend cleanup_dir_rename_info()\n @@ Commit message\n Extend cleanup_dir_rename_info() to handle these two different cases,\n cleaning up the relevant bits of information for each case.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 8: 44cfae6505f2 ! 8: 6bd398d3707e diffcore-rename: compute dir_rename_counts in stages\n @@ Commit message\n augment the counts via calling update_dir_rename_counts() after each\n basename-guide and inexact rename detection match is found.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 9: 752aff3a7995 ! 9: 46304aaebf5a diffcore-rename: limit dir_rename_counts computation to relevant dirs\n @@ Commit message\n info->relevant_source_dirs variable for this purpose, even though at\n this stage we will only set it to dirs_removed for simplicity.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 10: 65f7bfb735f2 ! 10: 4be565c47208 diffcore-rename: compute dir_rename_guess from dir_rename_counts\n @@ Commit message\n mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n\n-- \ngitgitgadget\n"},{"id":"417924","messageId":"21b9cf1da30e41b4e4645a4db64f4e6d15c03518.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 03/10] diffcore-rename: add a mapping of destination names to their indices","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:41Z","receivedAt":"2021-02-27T00:32:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCompute a mapping of full filename to the index within rename_dst where\nthat filename is found, and store it in idx_map. idx_possible_rename()\nneeds this to quickly finding an array entry in rename_dst given the\npathname.\n\nWhile at it, add placeholder initializations for dir_rename_count and\ndir_rename_guess; these will be more fully populated in subsequent\ncommits.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex edb0effb6ef4..8eeb8c73664c 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -380,6 +380,45 @@ static char *get_dirname(const char *filename)\n \treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n }\n \n+static void initialize_dir_rename_info(struct dir_rename_info *info)\n+{\n+\tint i;\n+\n+\tinfo->setup = 1;\n+\n+\tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n+\tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n+\tinfo->dir_rename_count = NULL;\n+\n+\t/*\n+\t * Loop setting up both info->idx_map.\n+\t */\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\t/*\n+\t\t * For non-renamed files, make idx_map contain mapping of\n+\t\t * filename -> index (index within rename_dst, that is)\n+\t\t */\n+\t\tif (!rename_dst[i].is_rename) {\n+\t\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\t\tstrintmap_set(&info->idx_map, filename, i);\n+\t\t}\n+\t}\n+}\n+\n+static void cleanup_dir_rename_info(struct dir_rename_info *info)\n+{\n+\tif (!info->setup)\n+\t\treturn;\n+\n+\t/* idx_map */\n+\tstrintmap_clear(&info->idx_map);\n+\n+\t/* dir_rename_guess */\n+\tstrmap_clear(&info->dir_rename_guess, 1);\n+\n+\t/* Nothing to do for dir_rename_count, yet */\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\n@@ -858,6 +897,11 @@ void diffcore_rename(struct diff_options *options)\n \t\tremove_unneeded_paths_from_src(want_copies);\n \t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n \n+\t\t/* Preparation for basename-driven matching. */\n+\t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n+\t\tinitialize_dir_rename_info(&info);\n+\t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n+\n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n@@ -1026,6 +1070,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tif (rename_dst[i].filespec_to_free)\n \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n \n+\tcleanup_dir_rename_info(&info);\n \tFREE_AND_NULL(rename_dst);\n \trename_dst_nr = rename_dst_alloc = 0;\n \tFREE_AND_NULL(rename_src);\n-- \ngitgitgadget\n\n"},{"id":"417925","messageId":"2dde621d7de596d7aa0bb31245b04683de2fa3d9.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 02/10] diffcore-rename: provide basic implementation of idx_possible_rename()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:40Z","receivedAt":"2021-02-27T00:32:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd a new struct dir_rename_info with various values we need inside our\nidx_possible_rename() function introduced in the previous commit. Add a\nbasic implementation for this function showing how we plan to use the\nvariables, but which will just return early with a value of -1 (not\nfound) when those variables are not set up.\n\nFuture commits will do the work necessary to set up those other\nvariables so that idx_possible_rename() does not always return -1.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 100 +++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 94 insertions(+), 6 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex b3055683bac2..edb0effb6ef4 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,19 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+struct dir_rename_info {\n+\tstruct strintmap idx_map;\n+\tstruct strmap dir_rename_guess;\n+\tstruct strmap *dir_rename_count;\n+\tunsigned setup;\n+};\n+\n+static char *get_dirname(const char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n+}\n+\n static const char *get_basename(const char *filename)\n {\n \t/*\n@@ -379,14 +392,86 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n-static int idx_possible_rename(char *filename)\n+static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n {\n-\t/* Unconditionally return -1, \"not found\", for now */\n-\treturn -1;\n+\t/*\n+\t * Our comparison of files with the same basename (see\n+\t * find_basename_matches() below), is only helpful when after exact\n+\t * rename detection we have exactly one file with a given basename\n+\t * among the rename sources and also only exactly one file with\n+\t * that basename among the rename destinations. When we have\n+\t * multiple files with the same basename in either set, we do not\n+\t * know which to compare against. However, there are some\n+\t * filenames that occur in large numbers (particularly\n+\t * build-related filenames such as 'Makefile', '.gitignore', or\n+\t * 'build.gradle' that potentially exist within every single\n+\t * subdirectory), and for performance we want to be able to quickly\n+\t * find renames for these files too.\n+\t *\n+\t * The reason basename comparisons are a useful heuristic was that it\n+\t * is common for people to move files across directories while keeping\n+\t * their filename the same. If we had a way of determining or even\n+\t * making a good educated guess about which directory these non-unique\n+\t * basename files had moved the file to, we could check it.\n+\t * Luckily...\n+\t *\n+\t * When an entire directory is in fact renamed, we have two factors\n+\t * helping us out:\n+\t * (a) the original directory disappeared giving us a hint\n+\t * about when we can apply an extra heuristic.\n+\t * (a) we often have several files within that directory and\n+\t * subdirectories that are renamed without changes\n+\t * So, rules for a heuristic:\n+\t * (0) If there basename matches are non-unique (the condition under\n+\t * which this function is called) AND\n+\t * (1) the directory in which the file was found has disappeared\n+\t * (i.e. dirs_removed is non-NULL and has a relevant entry) THEN\n+\t * (2) use exact renames of files within the directory to determine\n+\t * where the directory is likely to have been renamed to. IF\n+\t * there is at least one exact rename from within that\n+\t * directory, we can proceed.\n+\t * (3) If there are multiple places the directory could have been\n+\t * renamed to based on exact renames, ignore all but one of them.\n+\t * Just use the destination with the most renames going to it.\n+\t * (4) Check if applying that directory rename to the original file\n+\t * would result in a destination filename that is in the\n+\t * potential rename set. If so, return the index of the\n+\t * destination file (the index within rename_dst).\n+\t * (5) Compare the original file and returned destination for\n+\t * similarity, and if they are sufficiently similar, record the\n+\t * rename.\n+\t *\n+\t * This function, idx_possible_rename(), is only responsible for (4).\n+\t * The conditions/steps in (1)-(3) will be handled via setting up\n+\t * dir_rename_count and dir_rename_guess in a future\n+\t * initialize_dir_rename_info() function. Steps (0) and (5) are\n+\t * handled by the caller of this function.\n+\t */\n+\tchar *old_dir, *new_dir;\n+\tstruct strbuf new_path = STRBUF_INIT;\n+\tint idx;\n+\n+\tif (!info->setup)\n+\t\treturn -1;\n+\n+\told_dir = get_dirname(filename);\n+\tnew_dir = strmap_get(&info->dir_rename_guess, old_dir);\n+\tfree(old_dir);\n+\tif (!new_dir)\n+\t\treturn -1;\n+\n+\tstrbuf_addstr(&new_path, new_dir);\n+\tstrbuf_addch(&new_path, '/');\n+\tstrbuf_addstr(&new_path, get_basename(filename));\n+\n+\tidx = strintmap_get(&info->idx_map, new_path.buf);\n+\tstrbuf_release(&new_path);\n+\treturn idx;\n }\n \n static int find_basename_matches(struct diff_options *options,\n-\t\t\t\t int minimum_score)\n+\t\t\t\t int minimum_score,\n+\t\t\t\t struct dir_rename_info *info)\n {\n \t/*\n \t * When I checked in early 2020, over 76% of file renames in linux\n@@ -494,7 +579,7 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\tdst_index = strintmap_get(&dests, base);\n \t\t\tif (src_index == -1 || dst_index == -1) {\n \t\t\t\tsrc_index = i;\n-\t\t\t\tdst_index = idx_possible_rename(filename);\n+\t\t\t\tdst_index = idx_possible_rename(filename, info);\n \t\t\t}\n \t\t\tif (dst_index == -1)\n \t\t\t\tcontinue;\n@@ -677,8 +762,10 @@ void diffcore_rename(struct diff_options *options)\n \tint num_destinations, dst_cnt;\n \tint num_sources, want_copies;\n \tstruct progress *progress = NULL;\n+\tstruct dir_rename_info info;\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n+\tinfo.setup = 0;\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n@@ -774,7 +861,8 @@ void diffcore_rename(struct diff_options *options)\n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n-\t\t\t\t\t\t min_basename_score);\n+\t\t\t\t\t\t min_basename_score,\n+\t\t\t\t\t\t &info);\n \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n \n \t\t/*\n-- \ngitgitgadget\n\n"},{"id":"417926","messageId":"823d07532e0077b24de0a4f2e145cc47f59d1b36.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 01/10] diffcore-rename: use directory rename guided basename comparisons","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:39Z","receivedAt":"2021-02-27T00:32:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA previous commit noted that it is very common for people to move files\nacross directories while keeping their filename the same. The last few\ncommits took advantage of this and showed that we can accelerate rename\ndetection significantly using basenames; since files with the same\nbasename serve as likely rename candidates, we can check those first and\nremove them from the rename candidate pool if they are sufficiently\nsimilar.\n\nUnfortunately, the previous optimization was limited by the fact that\nthe remaining basenames after exact rename detection are not always\nunique. Many repositories have hundreds of build files with the same\nname (e.g. Makefile, .gitignore, build.gradle, etc.), and may even have\nhundreds of source files with the same name. (For example, the linux\nkernel has 100 setup.c, 87 irq.c, and 112 core.c files. A repository at\n$DAYJOB has a lot of ObjectFactory.java and Plugin.java files).\n\nFor these files with non-unique basenames, we are faced with the task of\nattempting to determine or guess which directory they may have been\nrelocated to. Such a task is precisely the job of directory rename\ndetection. However, there are two catches: (1) the directory rename\ndetection code has traditionally been part of the merge machinery rather\nthan diffcore-rename.c, and (2) directory rename detection currently\nruns after regular rename detection is complete. The 1st catch is just\nan implementation issue that can be overcome by some code shuffling.\nThe 2nd requires us to add a further approximation: we only have access\nto exact renames at this point, so we need to do directory rename\ndetection based on just exact renames. In some cases we won't have\nexact renames, in which case this extra optimization won't apply. We\nalso choose to not apply the optimization unless we know that the\nunderlying directory was removed, which will require extra data to be\npassed in to diffcore_rename_extended(). Also, even if we get a\nprediction about which directory a file may have relocated to, we will\nstill need to check to see if there is a file in the predicted\ndirectory, and then compare the two files to see if they meet the higher\nmin_basename_score threshold required for marking the two files as\nrenames.\n\nThis commit introduces an idx_possible_rename() function which will\ndo this directory rename detection for us and give us the index within\nrename_dst of the resulting filename. For now, this function is\nhardcoded to return -1 (not found) and just hooks up how its results\nwould be used once we have a more complete implementation in place.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 2 +-\n diffcore-rename.c | 42 ++++++++++++++++++++++++++++-------\n 2 files changed, 35 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex 80fcf9542441..8673a5c5b2f2 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -186,7 +186,7 @@ mark a file pair as a rename and stop considering other candidates for\n better matches. At most, one comparison is done per file in this\n preliminary pass; so if there are several remaining ext.txt files\n throughout the directory hierarchy after exact rename detection, this\n-preliminary step will be skipped for those files.\n+preliminary step may be skipped for those files.\n \n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 41558185ae1d..b3055683bac2 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -379,6 +379,12 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n+static int idx_possible_rename(char *filename)\n+{\n+\t/* Unconditionally return -1, \"not found\", for now */\n+\treturn -1;\n+}\n+\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score)\n {\n@@ -415,8 +421,6 @@ static int find_basename_matches(struct diff_options *options,\n \tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n-\tstruct hashmap_iter iter;\n-\tstruct strmap_entry *entry;\n \n \t/*\n \t * The prefeteching stuff wants to know if it can skip prefetching\n@@ -466,17 +470,39 @@ static int find_basename_matches(struct diff_options *options,\n \t}\n \n \t/* Now look for basename matchups and do similarity estimation */\n-\tstrintmap_for_each_entry(&sources, &iter, entry) {\n-\t\tconst char *base = entry->key;\n-\t\tintptr_t src_index = (intptr_t)entry->value;\n+\tfor (i = 0; i < rename_src_nr; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base = NULL;\n+\t\tintptr_t src_index;\n \t\tintptr_t dst_index;\n-\t\tif (src_index == -1)\n-\t\t\tcontinue;\n \n-\t\tif (0 <= (dst_index = strintmap_get(&dests, base))) {\n+\t\t/*\n+\t\t * If the basename is unique among remaining sources, then\n+\t\t * src_index will equal 'i' and we can attempt to match it\n+\t\t * to a unique basename in the destinations. Otherwise,\n+\t\t * use directory rename heuristics, if possible.\n+\t\t */\n+\t\tbase = get_basename(filename);\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tassert(src_index == -1 || src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n \t\t\tstruct diff_filespec *one, *two;\n \t\t\tint score;\n \n+\t\t\t/* Find a matching destination, if possible */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (src_index == -1 || dst_index == -1) {\n+\t\t\t\tsrc_index = i;\n+\t\t\t\tdst_index = idx_possible_rename(filename);\n+\t\t\t}\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n \t\t\t/* Estimate the similarity */\n \t\t\tone = rename_src[src_index].p->one;\n \t\t\ttwo = rename_dst[dst_index].p->two;\n-- \ngitgitgadget\n\n"},{"id":"417927","messageId":"3617b0209cc44b01a75f42ca57341fedc64ca8fd.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 04/10] Move computation of dir_rename_count from merge-ort to diffcore-rename","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:42Z","receivedAt":"2021-02-27T00:32:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMove the computation of dir_rename_count from merge-ort.c to\ndiffcore-rename.c, making slight adjustments to the data structures\nbased on the move. While the diffstat looks large, viewing this commit\nwith --color-moved makes it clear that only about 20 lines changed.\n\nWith this patch, the computation of dir_rename_count is still only done\nafter inexact rename detection, but subsequent commits will add a\npreliminary computation of dir_rename_count after exact rename\ndetection, followed by some updates after inexact rename detection.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 138 +++++++++++++++++++++++++++++++++++++++++++++-\n diffcore.h | 5 ++\n merge-ort.c | 132 +-------------------------------------------\n 3 files changed, 145 insertions(+), 130 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 8eeb8c73664c..39e23d57e7bc 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -380,6 +380,129 @@ static char *get_dirname(const char *filename)\n \treturn slash ? xstrndup(filename, slash - filename) : xstrdup(\"\");\n }\n \n+static void dirname_munge(char *filename)\n+{\n+\tchar *slash = strrchr(filename, '/');\n+\tif (!slash)\n+\t\tslash = filename;\n+\t*slash = '\\0';\n+}\n+\n+static void increment_count(struct strmap *dir_rename_count,\n+\t\t\t char *old_dir,\n+\t\t\t char *new_dir)\n+{\n+\tstruct strintmap *counts;\n+\tstruct strmap_entry *e;\n+\n+\t/* Get the {new_dirs -> counts} mapping using old_dir */\n+\te = strmap_get_entry(dir_rename_count, old_dir);\n+\tif (e) {\n+\t\tcounts = e->value;\n+\t} else {\n+\t\tcounts = xmalloc(sizeof(*counts));\n+\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n+\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t}\n+\n+\t/* Increment the count for new_dir */\n+\tstrintmap_incr(counts, new_dir, 1);\n+}\n+\n+static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t const char *oldname,\n+\t\t\t\t const char *newname)\n+{\n+\tchar *old_dir = xstrdup(oldname);\n+\tchar *new_dir = xstrdup(newname);\n+\tchar new_dir_first_char = new_dir[0];\n+\tint first_time_in_loop = 1;\n+\n+\twhile (1) {\n+\t\tdirname_munge(old_dir);\n+\t\tdirname_munge(new_dir);\n+\n+\t\t/*\n+\t\t * When renaming\n+\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n+\t\t * then this suggests that both\n+\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n+\t\t * a/b/c/d/ => a/b/some/thing/else/\n+\t\t * so we want to increment counters for both. We do NOT,\n+\t\t * however, also want to suggest that there was the following\n+\t\t * rename:\n+\t\t * a/b/c/ => a/b/some/thing/\n+\t\t * so we need to quit at that point.\n+\t\t *\n+\t\t * Note the when first_time_in_loop, we only strip off the\n+\t\t * basename, and we don't care if that's different.\n+\t\t */\n+\t\tif (!first_time_in_loop) {\n+\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n+\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n+\t\t\tif (!*new_dir) {\n+\t\t\t\t/*\n+\t\t\t\t * Special case when renaming to root directory,\n+\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n+\t\t\t\t * something like\n+\t\t\t\t * a/b/subdir => subdir\n+\t\t\t\t * and so dirname_munge() sets things up so that\n+\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n+\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n+\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n+\t\t\t\t * in new_dir, so we have to compare differently.\n+\t\t\t\t */\n+\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n+\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t} else {\n+\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strset_contains(dirs_removed, old_dir))\n+\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\telse\n+\t\t\tbreak;\n+\n+\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n+\t\tif (!*old_dir || !*new_dir)\n+\t\t\tbreak;\n+\n+\t\tfirst_time_in_loop = 0;\n+\t}\n+\n+\t/* Free resources we don't need anymore */\n+\tfree(old_dir);\n+\tfree(new_dir);\n+}\n+\n+static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n+\t\t\t\t struct strset *dirs_removed)\n+{\n+\tint i;\n+\n+\t/* Set up dir_rename_count */\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\t/* File not part of directory rename counts if not a rename */\n+\t\tif (!rename_dst[i].is_rename)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * Make dir_rename_count contain a map of a map:\n+\t\t * old_directory -> {new_directory -> count}\n+\t\t * In other words, for every pair look at the directories for\n+\t\t * the old filename and the new filename and count how many\n+\t\t * times that pairing occurs.\n+\t\t */\n+\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\t\t\t\t rename_dst[i].p->one->path,\n+\t\t\t\t\t rename_dst[i].p->two->path);\n+\t}\n+}\n+\n static void initialize_dir_rename_info(struct dir_rename_info *info)\n {\n \tint i;\n@@ -790,7 +913,9 @@ static void remove_unneeded_paths_from_src(int detecting_copies)\n \trename_src_nr = new_num_src;\n }\n \n-void diffcore_rename(struct diff_options *options)\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count)\n {\n \tint detect_rename = options->detect_rename;\n \tint minimum_score = options->rename_score;\n@@ -805,6 +930,7 @@ void diffcore_rename(struct diff_options *options)\n \n \ttrace2_region_enter(\"diff\", \"setup\", options->repo);\n \tinfo.setup = 0;\n+\tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n@@ -999,6 +1125,11 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n+\t/*\n+\t * Now that renames have been computed, compute dir_rename_count */\n+\tif (dirs_removed && dir_rename_count)\n+\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n@@ -1082,3 +1213,8 @@ void diffcore_rename(struct diff_options *options)\n \ttrace2_region_leave(\"diff\", \"write back to queue\", options->repo);\n \treturn;\n }\n+\n+void diffcore_rename(struct diff_options *options)\n+{\n+\tdiffcore_rename_extended(options, NULL, NULL);\n+}\ndiff --git a/diffcore.h b/diffcore.h\nindex d2a63c5c71f4..db55d3853071 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -8,6 +8,8 @@\n \n struct diff_options;\n struct repository;\n+struct strmap;\n+struct strset;\n struct userdiff_driver;\n \n /* This header file is internal between diff.c and its diff transformers\n@@ -161,6 +163,9 @@ void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n+void diffcore_rename_extended(struct diff_options *options,\n+\t\t\t struct strset *dirs_removed,\n+\t\t\t struct strmap *dir_rename_count);\n void diffcore_merge_broken(void);\n void diffcore_pickaxe(struct diff_options *);\n void diffcore_order(const char *orderfile);\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 603d30c52170..c4467e073b45 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -1302,131 +1302,6 @@ static char *handle_path_level_conflicts(struct merge_options *opt,\n \treturn new_path;\n }\n \n-static void dirname_munge(char *filename)\n-{\n-\tchar *slash = strrchr(filename, '/');\n-\tif (!slash)\n-\t\tslash = filename;\n-\t*slash = '\\0';\n-}\n-\n-static void increment_count(struct strmap *dir_rename_count,\n-\t\t\t char *old_dir,\n-\t\t\t char *new_dir)\n-{\n-\tstruct strintmap *counts;\n-\tstruct strmap_entry *e;\n-\n-\t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n-\tif (e) {\n-\t\tcounts = e->value;\n-\t} else {\n-\t\tcounts = xmalloc(sizeof(*counts));\n-\t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n-\t}\n-\n-\t/* Increment the count for new_dir */\n-\tstrintmap_incr(counts, new_dir, 1);\n-}\n-\n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t const char *oldname,\n-\t\t\t\t const char *newname)\n-{\n-\tchar *old_dir = xstrdup(oldname);\n-\tchar *new_dir = xstrdup(newname);\n-\tchar new_dir_first_char = new_dir[0];\n-\tint first_time_in_loop = 1;\n-\n-\twhile (1) {\n-\t\tdirname_munge(old_dir);\n-\t\tdirname_munge(new_dir);\n-\n-\t\t/*\n-\t\t * When renaming\n-\t\t * \"a/b/c/d/e/foo.c\" -> \"a/b/some/thing/else/e/foo.c\"\n-\t\t * then this suggests that both\n-\t\t * a/b/c/d/e/ => a/b/some/thing/else/e/\n-\t\t * a/b/c/d/ => a/b/some/thing/else/\n-\t\t * so we want to increment counters for both. We do NOT,\n-\t\t * however, also want to suggest that there was the following\n-\t\t * rename:\n-\t\t * a/b/c/ => a/b/some/thing/\n-\t\t * so we need to quit at that point.\n-\t\t *\n-\t\t * Note the when first_time_in_loop, we only strip off the\n-\t\t * basename, and we don't care if that's different.\n-\t\t */\n-\t\tif (!first_time_in_loop) {\n-\t\t\tchar *old_sub_dir = strchr(old_dir, '\\0')+1;\n-\t\t\tchar *new_sub_dir = strchr(new_dir, '\\0')+1;\n-\t\t\tif (!*new_dir) {\n-\t\t\t\t/*\n-\t\t\t\t * Special case when renaming to root directory,\n-\t\t\t\t * i.e. when new_dir == \"\". In this case, we had\n-\t\t\t\t * something like\n-\t\t\t\t * a/b/subdir => subdir\n-\t\t\t\t * and so dirname_munge() sets things up so that\n-\t\t\t\t * old_dir = \"a/b\\0subdir\\0\"\n-\t\t\t\t * new_dir = \"\\0ubdir\\0\"\n-\t\t\t\t * We didn't have a '/' to overwrite a '\\0' onto\n-\t\t\t\t * in new_dir, so we have to compare differently.\n-\t\t\t\t */\n-\t\t\t\tif (new_dir_first_char != old_sub_dir[0] ||\n-\t\t\t\t strcmp(old_sub_dir+1, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t} else {\n-\t\t\t\tif (strcmp(old_sub_dir, new_sub_dir))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n-\t\telse\n-\t\t\tbreak;\n-\n-\t\t/* If we hit toplevel directory (\"\") for old or new dir, quit */\n-\t\tif (!*old_dir || !*new_dir)\n-\t\t\tbreak;\n-\n-\t\tfirst_time_in_loop = 0;\n-\t}\n-\n-\t/* Free resources we don't need anymore */\n-\tfree(old_dir);\n-\tfree(new_dir);\n-}\n-\n-static void compute_rename_counts(struct diff_queue_struct *pairs,\n-\t\t\t\t struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n-{\n-\tint i;\n-\n-\tfor (i = 0; i < pairs->nr; ++i) {\n-\t\tstruct diff_filepair *pair = pairs->queue[i];\n-\n-\t\t/* File not part of directory rename if it wasn't renamed */\n-\t\tif (pair->status != 'R')\n-\t\t\tcontinue;\n-\n-\t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n-\t\t * old_directory -> {new_directory -> count}\n-\t\t * In other words, for every pair look at the directories for\n-\t\t * the old filename and the new filename and count how many\n-\t\t * times that pairing occurs.\n-\t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n-\t\t\t\t\t pair->one->path,\n-\t\t\t\t\t pair->two->path);\n-\t}\n-}\n-\n static void get_provisional_directory_renames(struct merge_options *opt,\n \t\t\t\t\t unsigned side,\n \t\t\t\t\t int *clean)\n@@ -1435,9 +1310,6 @@ static void get_provisional_directory_renames(struct merge_options *opt,\n \tstruct strmap_entry *entry;\n \tstruct rename_info *renames = &opt->priv->renames;\n \n-\tcompute_rename_counts(&renames->pairs[side],\n-\t\t\t &renames->dir_rename_count[side],\n-\t\t\t &renames->dirs_removed[side]);\n \t/*\n \t * Collapse\n \t * dir_rename_count: old_directory -> {new_directory -> count}\n@@ -2162,7 +2034,9 @@ static void detect_regular_renames(struct merge_options *opt,\n \n \tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n-\tdiffcore_rename(&diff_opts);\n+\tdiffcore_rename_extended(&diff_opts,\n+\t\t\t\t &renames->dirs_removed[side_index],\n+\t\t\t\t &renames->dir_rename_count[side_index]);\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n \tresolve_diffpair_statuses(&diff_queued_diff);\n \n-- \ngitgitgadget\n\n"},{"id":"417928","messageId":"2baf39d82f3ed40925addd778356b143f516f157.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 05/10] diffcore-rename: add function for clearing dir_rename_count","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:43Z","receivedAt":"2021-02-27T00:32:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs we adjust the usage of dir_rename_count we want to have a function\nfor clearing, or partially clearing it out. Add a\npartial_clear_dir_rename_count() function for this purpose.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 12 ++++++++++++\n diffcore.h | 2 ++\n merge-ort.c | 12 +++---------\n 3 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 39e23d57e7bc..7dd475ff9a9f 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -528,6 +528,18 @@ static void initialize_dir_rename_info(struct dir_rename_info *info)\n \t}\n }\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrmap_for_each_entry(dir_rename_count, &iter, entry) {\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tstrintmap_clear(counts);\n+\t}\n+\tstrmap_partial_clear(dir_rename_count, 1);\n+}\n+\n static void cleanup_dir_rename_info(struct dir_rename_info *info)\n {\n \tif (!info->setup)\ndiff --git a/diffcore.h b/diffcore.h\nindex db55d3853071..c6ba64abd198 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -161,6 +161,8 @@ struct diff_filepair *diff_queue(struct diff_queue_struct *,\n \t\t\t\t struct diff_filespec *);\n void diff_q(struct diff_queue_struct *, struct diff_filepair *);\n \n+void partial_clear_dir_rename_count(struct strmap *dir_rename_count);\n+\n void diffcore_break(struct repository *, int);\n void diffcore_rename(struct diff_options *);\n void diffcore_rename_extended(struct diff_options *options,\ndiff --git a/merge-ort.c b/merge-ort.c\nindex c4467e073b45..467404cc0a35 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -351,17 +351,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti,\n \n \t/* Free memory used by various renames maps */\n \tfor (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) {\n-\t\tstruct hashmap_iter iter;\n-\t\tstruct strmap_entry *entry;\n-\n \t\tstrset_func(&renames->dirs_removed[i]);\n \n-\t\tstrmap_for_each_entry(&renames->dir_rename_count[i],\n-\t\t\t\t &iter, entry) {\n-\t\t\tstruct strintmap *counts = entry->value;\n-\t\t\tstrintmap_clear(counts);\n-\t\t}\n-\t\tstrmap_func(&renames->dir_rename_count[i], 1);\n+\t\tpartial_clear_dir_rename_count(&renames->dir_rename_count[i]);\n+\t\tif (!reinitialize)\n+\t\t\tstrmap_clear(&renames->dir_rename_count[i], 1);\n \n \t\tstrmap_func(&renames->dir_renames[i], 0);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"417929","messageId":"02f1f7c02d32dc37813abf5fcf767cc374b17fd3.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 06/10] diffcore-rename: move dir_rename_counts into dir_rename_info struct","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:44Z","receivedAt":"2021-02-27T00:32:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis continues the migration of the directory rename detection code into\ndiffcore-rename, now taking the simple step of combining it with the\ndir_rename_info struct. Future commits will then make dir_rename_counts\nbe computed in stages, and add computation of dir_rename_guess.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 27 ++++++++++++++++-----------\n 1 file changed, 16 insertions(+), 11 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 7dd475ff9a9f..a1ccf14001f5 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -388,7 +388,7 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n-static void increment_count(struct strmap *dir_rename_count,\n+static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n {\n@@ -396,20 +396,20 @@ static void increment_count(struct strmap *dir_rename_count,\n \tstruct strmap_entry *e;\n \n \t/* Get the {new_dirs -> counts} mapping using old_dir */\n-\te = strmap_get_entry(dir_rename_count, old_dir);\n+\te = strmap_get_entry(info->dir_rename_count, old_dir);\n \tif (e) {\n \t\tcounts = e->value;\n \t} else {\n \t\tcounts = xmalloc(sizeof(*counts));\n \t\tstrintmap_init_with_options(counts, 0, NULL, 1);\n-\t\tstrmap_put(dir_rename_count, old_dir, counts);\n+\t\tstrmap_put(info->dir_rename_count, old_dir, counts);\n \t}\n \n \t/* Increment the count for new_dir */\n \tstrintmap_incr(counts, new_dir, 1);\n }\n \n-static void update_dir_rename_counts(struct strmap *dir_rename_count,\n+static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t const char *oldname,\n \t\t\t\t const char *newname)\n@@ -463,7 +463,7 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t}\n \n \t\tif (strset_contains(dirs_removed, old_dir))\n-\t\t\tincrement_count(dir_rename_count, old_dir, new_dir);\n+\t\t\tincrement_count(info, old_dir, new_dir);\n \t\telse\n \t\t\tbreak;\n \n@@ -479,12 +479,15 @@ static void update_dir_rename_counts(struct strmap *dir_rename_count,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n-\t\t\t\t struct strset *dirs_removed)\n+static void compute_dir_rename_counts(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n-\t/* Set up dir_rename_count */\n+\tinfo->setup = 1;\n+\tinfo->dir_rename_count = dir_rename_count;\n+\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/* File not part of directory rename counts if not a rename */\n \t\tif (!rename_dst[i].is_rename)\n@@ -497,7 +500,7 @@ static void compute_dir_rename_counts(struct strmap *dir_rename_count,\n \t\t * the old filename and the new filename and count how many\n \t\t * times that pairing occurs.\n \t\t */\n-\t\tupdate_dir_rename_counts(dir_rename_count, dirs_removed,\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n@@ -551,7 +554,9 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info)\n \t/* dir_rename_guess */\n \tstrmap_clear(&info->dir_rename_guess, 1);\n \n-\t/* Nothing to do for dir_rename_count, yet */\n+\t/* dir_rename_count */\n+\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\tstrmap_clear(info->dir_rename_count, 1);\n }\n \n static const char *get_basename(const char *filename)\n@@ -1140,7 +1145,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/*\n \t * Now that renames have been computed, compute dir_rename_count */\n \tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(dir_rename_count, dirs_removed);\n+\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n \n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n-- \ngitgitgadget\n\n"},{"id":"417930","messageId":"9c3436840534266ab257ab59e04364722bf3f6e9.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 07/10] diffcore-rename: extend cleanup_dir_rename_info()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:45Z","receivedAt":"2021-02-27T00:32:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen diffcore_rename_extended() is passed a NULL dir_rename_count, we\nwill still want to create a temporary one for use by\nfind_basename_matches(), but have it fully deallocated before\ndiffcore_rename_extended() returns. However, when\ndiffcore_rename_extended() is passed a dir_rename_count, we want to fill\nthat strmap with appropriate values and return it. However, for our\ninterim purposes we may also add entries corresponding to directories\nthat cannot have been renamed due to still existing on both sides.\n\nExtend cleanup_dir_rename_info() to handle these two different cases,\ncleaning up the relevant bits of information for each case.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 40 ++++++++++++++++++++++++++++++++++++----\n 1 file changed, 36 insertions(+), 4 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex a1ccf14001f5..2cf9c47c6364 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -543,8 +543,15 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n \tstrmap_partial_clear(dir_rename_count, 1);\n }\n \n-static void cleanup_dir_rename_info(struct dir_rename_info *info)\n+static void cleanup_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t int keep_dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\tstruct string_list to_remove = STRING_LIST_INIT_NODUP;\n+\tint i;\n+\n \tif (!info->setup)\n \t\treturn;\n \n@@ -555,8 +562,33 @@ static void cleanup_dir_rename_info(struct dir_rename_info *info)\n \tstrmap_clear(&info->dir_rename_guess, 1);\n \n \t/* dir_rename_count */\n-\tpartial_clear_dir_rename_count(info->dir_rename_count);\n-\tstrmap_clear(info->dir_rename_count, 1);\n+\tif (!keep_dir_rename_count) {\n+\t\tpartial_clear_dir_rename_count(info->dir_rename_count);\n+\t\tstrmap_clear(info->dir_rename_count, 1);\n+\t\tFREE_AND_NULL(info->dir_rename_count);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Although dir_rename_count was passed in\n+\t * diffcore_rename_extended() and we want to keep it around and\n+\t * return it to that caller, we first want to remove any data\n+\t * associated with directories that weren't renamed.\n+\t */\n+\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\tconst char *source_dir = entry->key;\n+\t\tstruct strintmap *counts = entry->value;\n+\n+\t\tif (!strset_contains(dirs_removed, source_dir)) {\n+\t\t\tstring_list_append(&to_remove, source_dir);\n+\t\t\tstrintmap_clear(counts);\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\tfor (i = 0; i < to_remove.nr; ++i)\n+\t\tstrmap_remove(info->dir_rename_count,\n+\t\t\t to_remove.items[i].string, 1);\n+\tstring_list_clear(&to_remove, 0);\n }\n \n static const char *get_basename(const char *filename)\n@@ -1218,7 +1250,7 @@ void diffcore_rename_extended(struct diff_options *options,\n \t\tif (rename_dst[i].filespec_to_free)\n \t\t\tfree_filespec(rename_dst[i].filespec_to_free);\n \n-\tcleanup_dir_rename_info(&info);\n+\tcleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL);\n \tFREE_AND_NULL(rename_dst);\n \trename_dst_nr = rename_dst_alloc = 0;\n \tFREE_AND_NULL(rename_src);\n-- \ngitgitgadget\n\n"},{"id":"417931","messageId":"46304aaebf5aa4120df81284c37efcb554cface5.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 09/10] diffcore-rename: limit dir_rename_counts computation to relevant dirs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:47Z","receivedAt":"2021-02-27T00:32:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe are using dir_rename_counts to count the number of other directories\nthat files within a directory moved to. We only need this information\nfor directories that disappeared, though, so we can return early from\nupdate_dir_rename_counts() for other paths.\n\nIf dirs_removed is passed to diffcore_rename_extended(), then it\nprovides the relevant bits of information for us to limit this counting\nto relevant dirs. If dirs_removed is not passed, we would need to\ncompute some replacement in order to do this limiting. Introduce a new\ninfo->relevant_source_dirs variable for this purpose, even though at\nthis stage we will only set it to dirs_removed for simplicity.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 10f8f4a301e3..e5fa0cb555dd 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -371,6 +371,7 @@ struct dir_rename_info {\n \tstruct strintmap idx_map;\n \tstruct strmap dir_rename_guess;\n \tstruct strmap *dir_rename_count;\n+\tstruct strset *relevant_source_dirs;\n \tunsigned setup;\n };\n \n@@ -442,7 +443,13 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \t\treturn;\n \n \twhile (1) {\n+\t\t/* Get old_dir, skip if its directory isn't relevant. */\n \t\tdirname_munge(old_dir);\n+\t\tif (info->relevant_source_dirs &&\n+\t\t !strset_contains(info->relevant_source_dirs, old_dir))\n+\t\t\tbreak;\n+\n+\t\t/* Get new_dir */\n \t\tdirname_munge(new_dir);\n \n \t\t/*\n@@ -521,6 +528,9 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n \n+\t/* Setup info->relevant_source_dirs */\n+\tinfo->relevant_source_dirs = dirs_removed;\n+\n \t/*\n \t * Loop setting up both info->idx_map, and doing setup of\n \t * info->dir_rename_count.\n-- \ngitgitgadget\n\n"},{"id":"417932","messageId":"6bd398d3707eb514d8758dcff0a21cae11c897b5.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 08/10] diffcore-rename: compute dir_rename_counts in stages","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:46Z","receivedAt":"2021-02-27T00:32:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCompute dir_rename_counts based just on exact renames to start, as that\ncan provide us useful information in find_basename_matches(). This is\ndone by moving the code from compute_dir_rename_counts() into\ninitialize_dir_rename_info(), resulting in it being computed earlier and\nbased just on exact renames. Since that's an incomplete result, we\naugment the counts via calling update_dir_rename_counts() after each\nbasename-guide and inexact rename detection match is found.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 110 +++++++++++++++++++++++++++++-----------------\n 1 file changed, 70 insertions(+), 40 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 2cf9c47c6364..10f8f4a301e3 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -419,6 +419,28 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tchar new_dir_first_char = new_dir[0];\n \tint first_time_in_loop = 1;\n \n+\tif (!info->setup)\n+\t\t/*\n+\t\t * info->setup is 0 here in two cases: (1) all auxiliary\n+\t\t * vars (like dirs_removed) were NULL so\n+\t\t * initialize_dir_rename_info() returned early, or (2)\n+\t\t * either break detection or copy detection are active so\n+\t\t * that we never called initialize_dir_rename_info(). In\n+\t\t * the former case, we don't have enough info to know if\n+\t\t * directories were renamed (because dirs_removed lets us\n+\t\t * know about a necessary prerequisite, namely if they were\n+\t\t * removed), and in the latter, we don't care about\n+\t\t * directory renames or find_basename_matches.\n+\t\t *\n+\t\t * This matters because both basename and inexact matching\n+\t\t * will also call update_dir_rename_counts(). In either of\n+\t\t * the above two cases info->dir_rename_counts will not\n+\t\t * have been properly initialized which prevents us from\n+\t\t * updating it, but in these two cases we don't care about\n+\t\t * dir_rename_counts anyway, so we can just exit early.\n+\t\t */\n+\t\treturn;\n+\n \twhile (1) {\n \t\tdirname_munge(old_dir);\n \t\tdirname_munge(new_dir);\n@@ -479,45 +501,29 @@ static void update_dir_rename_counts(struct dir_rename_info *info,\n \tfree(new_dir);\n }\n \n-static void compute_dir_rename_counts(struct dir_rename_info *info,\n-\t\t\t\t struct strset *dirs_removed,\n-\t\t\t\t struct strmap *dir_rename_count)\n+static void initialize_dir_rename_info(struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed,\n+\t\t\t\t struct strmap *dir_rename_count)\n {\n \tint i;\n \n-\tinfo->setup = 1;\n-\tinfo->dir_rename_count = dir_rename_count;\n-\n-\tfor (i = 0; i < rename_dst_nr; ++i) {\n-\t\t/* File not part of directory rename counts if not a rename */\n-\t\tif (!rename_dst[i].is_rename)\n-\t\t\tcontinue;\n-\n-\t\t/*\n-\t\t * Make dir_rename_count contain a map of a map:\n-\t\t * old_directory -> {new_directory -> count}\n-\t\t * In other words, for every pair look at the directories for\n-\t\t * the old filename and the new filename and count how many\n-\t\t * times that pairing occurs.\n-\t\t */\n-\t\tupdate_dir_rename_counts(info, dirs_removed,\n-\t\t\t\t\t rename_dst[i].p->one->path,\n-\t\t\t\t\t rename_dst[i].p->two->path);\n+\tif (!dirs_removed) {\n+\t\tinfo->setup = 0;\n+\t\treturn;\n \t}\n-}\n-\n-static void initialize_dir_rename_info(struct dir_rename_info *info)\n-{\n-\tint i;\n-\n \tinfo->setup = 1;\n \n+\tinfo->dir_rename_count = dir_rename_count;\n+\tif (!info->dir_rename_count) {\n+\t\tinfo->dir_rename_count = xmalloc(sizeof(*dir_rename_count));\n+\t\tstrmap_init(info->dir_rename_count);\n+\t}\n \tstrintmap_init_with_options(&info->idx_map, -1, NULL, 0);\n \tstrmap_init_with_options(&info->dir_rename_guess, NULL, 0);\n-\tinfo->dir_rename_count = NULL;\n \n \t/*\n-\t * Loop setting up both info->idx_map.\n+\t * Loop setting up both info->idx_map, and doing setup of\n+\t * info->dir_rename_count.\n \t */\n \tfor (i = 0; i < rename_dst_nr; ++i) {\n \t\t/*\n@@ -527,7 +533,20 @@ static void initialize_dir_rename_info(struct dir_rename_info *info)\n \t\tif (!rename_dst[i].is_rename) {\n \t\t\tchar *filename = rename_dst[i].p->two->path;\n \t\t\tstrintmap_set(&info->idx_map, filename, i);\n+\t\t\tcontinue;\n \t\t}\n+\n+\t\t/*\n+\t\t * For everything else (i.e. renamed files), make\n+\t\t * dir_rename_count contain a map of a map:\n+\t\t * old_directory -> {new_directory -> count}\n+\t\t * In other words, for every pair look at the directories for\n+\t\t * the old filename and the new filename and count how many\n+\t\t * times that pairing occurs.\n+\t\t */\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t rename_dst[i].p->one->path,\n+\t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n }\n \n@@ -682,7 +701,8 @@ static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n \n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n-\t\t\t\t struct dir_rename_info *info)\n+\t\t\t\t struct dir_rename_info *info,\n+\t\t\t\t struct strset *dirs_removed)\n {\n \t/*\n \t * When I checked in early 2020, over 76% of file renames in linux\n@@ -810,6 +830,8 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\tcontinue;\n \t\t\trecord_rename_pair(dst_index, src_index, score);\n \t\t\trenames++;\n+\t\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t\t one->path, two->path);\n \n \t\t\t/*\n \t\t\t * Found a rename so don't need text anymore; if we\n@@ -893,7 +915,12 @@ static int too_many_rename_candidates(int num_destinations, int num_sources,\n \treturn 1;\n }\n \n-static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, int copies)\n+static int find_renames(struct diff_score *mx,\n+\t\t\tint dst_cnt,\n+\t\t\tint minimum_score,\n+\t\t\tint copies,\n+\t\t\tstruct dir_rename_info *info,\n+\t\t\tstruct strset *dirs_removed)\n {\n \tint count = 0, i;\n \n@@ -910,6 +937,9 @@ static int find_renames(struct diff_score *mx, int dst_cnt, int minimum_score, i\n \t\t\tcontinue;\n \t\trecord_rename_pair(mx[i].dst, mx[i].src, mx[i].score);\n \t\tcount++;\n+\t\tupdate_dir_rename_counts(info, dirs_removed,\n+\t\t\t\t\t rename_src[mx[i].src].p->one->path,\n+\t\t\t\t\t rename_dst[mx[i].dst].p->two->path);\n \t}\n \treturn count;\n }\n@@ -981,6 +1011,8 @@ void diffcore_rename_extended(struct diff_options *options,\n \tinfo.setup = 0;\n \tassert(!dir_rename_count || strmap_empty(dir_rename_count));\n \twant_copies = (detect_rename == DIFF_DETECT_COPY);\n+\tif (dirs_removed && (break_idx || want_copies))\n+\t\tBUG(\"dirs_removed incompatible with break/copy detection\");\n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n \n@@ -1074,14 +1106,15 @@ void diffcore_rename_extended(struct diff_options *options,\n \n \t\t/* Preparation for basename-driven matching. */\n \t\ttrace2_region_enter(\"diff\", \"dir rename setup\", options->repo);\n-\t\tinitialize_dir_rename_info(&info);\n+\t\tinitialize_dir_rename_info(&info,\n+\t\t\t\t\t dirs_removed, dir_rename_count);\n \t\ttrace2_region_leave(\"diff\", \"dir rename setup\", options->repo);\n \n \t\t/* Utilize file basenames to quickly find renames. */\n \t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n \t\trename_count += find_basename_matches(options,\n \t\t\t\t\t\t min_basename_score,\n-\t\t\t\t\t\t &info);\n+\t\t\t\t\t\t &info, dirs_removed);\n \t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n \n \t\t/*\n@@ -1167,18 +1200,15 @@ void diffcore_rename_extended(struct diff_options *options,\n \t/* cost matrix sorted by most to least similar pair */\n \tSTABLE_QSORT(mx, dst_cnt * NUM_CANDIDATE_PER_DST, score_compare);\n \n-\trename_count += find_renames(mx, dst_cnt, minimum_score, 0);\n+\trename_count += find_renames(mx, dst_cnt, minimum_score, 0,\n+\t\t\t\t &info, dirs_removed);\n \tif (want_copies)\n-\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1);\n+\t\trename_count += find_renames(mx, dst_cnt, minimum_score, 1,\n+\t\t\t\t\t &info, dirs_removed);\n \tfree(mx);\n \ttrace2_region_leave(\"diff\", \"inexact renames\", options->repo);\n \n cleanup:\n-\t/*\n-\t * Now that renames have been computed, compute dir_rename_count */\n-\tif (dirs_removed && dir_rename_count)\n-\t\tcompute_dir_rename_counts(&info, dirs_removed, dir_rename_count);\n-\n \t/* At this point, we have found some renames and copies and they\n \t * are recorded in rename_dst. The original list is still in *q.\n \t */\n-- \ngitgitgadget\n\n"},{"id":"417933","messageId":"4be565c472088d4144063b736308bf2a57331f45.1614385849.git.gitgitgadget@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"[PATCH v4 10/10] diffcore-rename: compute dir_rename_guess from dir_rename_counts","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-27T00:30:48Z","receivedAt":"2021-02-27T00:32:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndir_rename_counts has a mapping of a mapping, in particular, it has\n old_dir => { new_dir => count }\nWe want a simple mapping of\n old_dir => new_dir\nbased on which new_dir had the highest count for a given old_dir.\nCompute this and store it in dir_rename_guess.\n\nThis is the final piece of the puzzle needed to make our guesses at\nwhich directory files have been moved to when basenames aren't unique.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 12.775 s ± 0.062 s 12.596 s ± 0.061 s\n mega-renames: 188.754 s ± 0.284 s 130.465 s ± 0.259 s\n just-one-mega: 5.599 s ± 0.019 s 3.958 s ± 0.010 s\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 45 +++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 41 insertions(+), 4 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex e5fa0cb555dd..1fe902ed2af0 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -389,6 +389,24 @@ static void dirname_munge(char *filename)\n \t*slash = '\\0';\n }\n \n+static const char *get_highest_rename_path(struct strintmap *counts)\n+{\n+\tint highest_count = 0;\n+\tconst char *highest_destination_dir = NULL;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\tstrintmap_for_each_entry(counts, &iter, entry) {\n+\t\tconst char *destination_dir = entry->key;\n+\t\tintptr_t count = (intptr_t)entry->value;\n+\t\tif (count > highest_count) {\n+\t\t\thighest_count = count;\n+\t\t\thighest_destination_dir = destination_dir;\n+\t\t}\n+\t}\n+\treturn highest_destination_dir;\n+}\n+\n static void increment_count(struct dir_rename_info *info,\n \t\t\t char *old_dir,\n \t\t\t char *new_dir)\n@@ -512,6 +530,8 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t struct strset *dirs_removed,\n \t\t\t\t struct strmap *dir_rename_count)\n {\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n \tint i;\n \n \tif (!dirs_removed) {\n@@ -558,6 +578,23 @@ static void initialize_dir_rename_info(struct dir_rename_info *info,\n \t\t\t\t\t rename_dst[i].p->one->path,\n \t\t\t\t\t rename_dst[i].p->two->path);\n \t}\n+\n+\t/*\n+\t * Now we collapse\n+\t * dir_rename_count: old_directory -> {new_directory -> count}\n+\t * down to\n+\t * dir_rename_guess: old_directory -> best_new_directory\n+\t * where best_new_directory is the one with the highest count.\n+\t */\n+\tstrmap_for_each_entry(info->dir_rename_count, &iter, entry) {\n+\t\t/* entry->key is source_dir */\n+\t\tstruct strintmap *counts = entry->value;\n+\t\tchar *best_newdir;\n+\n+\t\tbest_newdir = xstrdup(get_highest_rename_path(counts));\n+\t\tstrmap_put(&info->dir_rename_guess, entry->key,\n+\t\t\t best_newdir);\n+\t}\n }\n \n void partial_clear_dir_rename_count(struct strmap *dir_rename_count)\n@@ -682,10 +719,10 @@ static int idx_possible_rename(char *filename, struct dir_rename_info *info)\n \t * rename.\n \t *\n \t * This function, idx_possible_rename(), is only responsible for (4).\n-\t * The conditions/steps in (1)-(3) will be handled via setting up\n-\t * dir_rename_count and dir_rename_guess in a future\n-\t * initialize_dir_rename_info() function. Steps (0) and (5) are\n-\t * handled by the caller of this function.\n+\t * The conditions/steps in (1)-(3) are handled via setting up\n+\t * dir_rename_count and dir_rename_guess in\n+\t * initialize_dir_rename_info(). Steps (0) and (5) are handled by\n+\t * the caller of this function.\n \t */\n \tchar *old_dir, *new_dir;\n \tstruct strbuf new_path = STRBUF_INIT;\n-- \ngitgitgadget\n"},{"id":"418657","messageId":"98dcac36-4c0f-e29c-2bdd-e3d89f4f9275@gmail.com","threadId":"55146","inReplyTo":"pull.844.v4.git.1614385849.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/10] Optimization batch 8: use file basenames even more","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-09T21:52:09Z","receivedAt":"2021-03-09T21:53:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/26/2021 7:30 PM, Elijah Newren via GitGitGadget wrote:\n> This series depends on en/diffcore-rename (a concatenation of what I was\n> calling ort-perf-batch-6 and ort-perf-batch-7).\n> \n> Changes since v3:\n\nI'm very late in doing this, but I reviewed the range-diff and found\nthis version to be satisfactory. Thanks!\n\n-Stolee\n"}]}