{"thread":{"id":"55112","subject":"[PATCH 1/3] diffcore-rename: compute basenames of all source and dest candidates","startedAt":"2021-02-06T22:53:10Z","lastAt":"2021-02-14T07:55:36Z","messageCount":71,"participants":["Elijah Newren via GitGitGadget","Junio C Hamano","Elijah Newren","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"416341","messageId":"377a4a39fa864a09eb1d90e1addcc5d61d6b026c.1612651937.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.git.1612651937.gitgitgadget@gmail.com","subject":"[PATCH 1/3] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-06T22:52:15Z","receivedAt":"2021-02-06T22:53:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to make use of unique basenames to help inform rename detection,\nso that more likely pairings can be checked first. Add a new function,\nnot yet used, which creates a map of the unique basenames within\nrename_src and another within rename_dst, together with the indices\nwithin rename_src/rename_dst where those basenames show up. Non-unique\nbasenames still show up in the map, but have an invalid index (-1).\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 74930716e70d..1c52077b04e5 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,59 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+MAYBE_UNUSED\n+static int find_basename_matches(struct diff_options *options,\n+\t\t\t\t int minimum_score,\n+\t\t\t\t int num_src)\n+{\n+\tint i;\n+\tstruct strintmap sources;\n+\tstruct strintmap dests;\n+\n+\t/* Create maps of basename -> fullname(s) for sources and dests */\n+\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n+\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tchar *base;\n+\n+\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n+\t\tassert(!rename_src[i].p->one->rename_used);\n+\n+\t\tbase = strrchr(filename, '/');\n+\t\tbase = (base ? base+1 : filename);\n+\n+\t\t/* Record index within rename_src (i) if basename is unique */\n+\t\tif (strintmap_contains(&sources, base))\n+\t\t\tstrintmap_set(&sources, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&sources, base, i);\n+\t}\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\tchar *base;\n+\n+\t\tif (rename_dst[i].is_rename)\n+\t\t\tcontinue; /* involved in exact match already. */\n+\n+\t\tbase = strrchr(filename, '/');\n+\t\tbase = (base ? base+1 : filename);\n+\n+\t\t/* Record index within rename_dst (i) if basename is unique */\n+\t\tif (strintmap_contains(&dests, base))\n+\t\t\tstrintmap_set(&dests, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&dests, base, i);\n+\t}\n+\n+\t/* TODO: Make use of basenames source and destination basenames */\n+\n+\tstrintmap_clear(&sources);\n+\tstrintmap_clear(&dests);\n+\n+\treturn 0;\n+}\n+\n #define NUM_CANDIDATE_PER_DST 4\n static void record_if_better(struct diff_score m[], struct diff_score *o)\n {\n-- \ngitgitgadget\n\n"},{"id":"416343","messageId":"5fb4493247ff4676abd11eb47b1db3111e043fb5.1612651937.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.git.1612651937.gitgitgadget@gmail.com","subject":"[PATCH 2/3] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-06T22:52:16Z","receivedAt":"2021-02-06T22:53:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIt is not uncommon in real world repositories for the majority of file\nrenames to not change the basename of the file; i.e. most \"renames\" are\njust a move of files into different directories. We can make use of\nthis to avoid comparing all rename source candidates with all rename\ndestination candidates, by first comparing sources to destinations with\nthe same basenames. If two files with the same basename are\nsufficiently similar, we record the rename; if not, we include those\nfiles in the more exhaustive matrix comparison.\n\nNote that this optimization might give us different results than without\nthe optimization, because it's possible that despite files with the same\nbasename being sufficiently similar to be considered a rename, there's\nan even better match between files without the same basename. I think\nthat is okay for four reasons: (1) That seems somewhat unlikely in\npractice, (2) it's easy to explain to the users what happened if it does\never occur (or even for them to intuitively figure out), and (3) as the\nnext patch will show it provides such a large performance boost that\nit's worth the tradeoff. Reason (4) takes a full paragraph to\nexplain...\n\nIf the previous three reasons aren't enough, consider what rename\ndetection already does. Break detection is not the default, meaning\nthat if files have the same _fullname_, then they are considered related\neven if they are 0% similar. In fact, in such a case, we don't even\nbother comparing the files to see if they are similar let alone\ncomparing them to all other files to see what they are most similar to.\nBasically, we override content similarity based on sufficient filename\nsimilarity. Without the filename similarity (currently implemented as\nan exact match of filename), we swing the pendulum the opposite\ndirection and say that filename similarity is irrelevant and compare a\nfull N x M matrix of sources and destinations to find out which have the\nmost similar contents. This optimization just adds another form of\nfilename similarity comparison, but augments it with a file content\nsimilarity check as well. Basically, if two files have the same\nbasename and are sufficiently similar to be considered a rename, mark\nthem as such without comparing the two to all other rename candidates.\n\nWe do not use this heuristic together with either break or copy\ndetection. The point of break detection is to say that filename\nsimilarity does not imply file content similarity, and we only want to\nknow about file content similarity. The point of copy detection is to\nuse more resources to check for additional similarities, while this is\nan optimization that uses far less resources but which might also result\nin finding slightly fewer similarities. So the idea behind this\noptimization goes against both of those features, and will be turned off\nfor both.\n\nNote that this optimization is not yet effective in any situation, as\nthe function is still unused. The next commit will hook it into the\ncode so that it is used when rename detection is wanted, but neither\ncopy nor break detection are.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 94 +++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 91 insertions(+), 3 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 1c52077b04e5..b1dda41de9b1 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -372,10 +372,48 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n {\n-\tint i;\n+\t/*\n+\t * When I checked, over 76% of file renames in linux just moved\n+\t * files to a different directory but kept the same basename. gcc\n+\t * did that with over 64% of renames, gecko did it with over 79%,\n+\t * and WebKit did it with over 89%.\n+\t *\n+\t * Therefore we can bypass the normal exhaustive NxM matrix\n+\t * comparison of similarities between all potential rename sources\n+\t * and destinations by instead using file basename as a hint, checking\n+\t * for similarity between files with the same basename, and if we\n+\t * find a pair that are sufficiently similar, record the rename\n+\t * pair and exclude those two from the NxM matrix.\n+\t *\n+\t * This *might* cause us to find a less than optimal pairing (if\n+\t * there is another file that we are even more similar to but has a\n+\t * different basename). Given the huge performance advantage\n+\t * basename matching provides, and given the frequency with which\n+\t * people use the same basename in real world projects, that's a\n+\t * trade-off we are willing to accept when doing just rename\n+\t * detection. However, if someone wants copy detection that\n+\t * implies they are willing to spend more cycles to find\n+\t * similarities between files, so it may be less likely that this\n+\t * heuristic is wanted.\n+\t */\n+\n+\tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n \n+\t/*\n+\t * The prefeteching stuff wants to know if it can skip prefetching blobs\n+\t * that are unmodified. unmodified blobs are only relevant when doing\n+\t * copy detection. find_basename_matches() is only used when detecting\n+\t * renames, not when detecting copies, so it'll only be used when a file\n+\t * only existed in the source. Since we already know that the file\n+\t * won't be unmodified, there's no point checking for it; that's just a\n+\t * waste of resources. So set skip_unmodified to 0 so that\n+\t * estimate_similarity() and prefetch() won't waste resources checking\n+\t * for something we already know is false.\n+\t */\n+\tint skip_unmodified = 0;\n+\n \t/* Create maps of basename -> fullname(s) for sources and dests */\n \tstrintmap_init_with_options(&sources, -1, NULL, 0);\n \tstrintmap_init_with_options(&dests, -1, NULL, 0);\n@@ -412,12 +450,62 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\tstrintmap_set(&dests, base, i);\n \t}\n \n-\t/* TODO: Make use of basenames source and destination basenames */\n+\t/* Now look for basename matchups and do similarity estimation */\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tchar *base = NULL;\n+\t\tintptr_t src_index;\n+\t\tintptr_t dst_index;\n+\n+\t\t/* Get the basename */\n+\t\tbase = strrchr(filename, '/');\n+\t\tbase = (base ? base+1 : filename);\n+\n+\t\t/* Find out if this basename is unique among sources */\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tif (src_index == -1)\n+\t\t\tcontinue; /* not a unique basename; skip it */\n+\t\tassert(src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n+\t\t\tstruct diff_filespec *one, *two;\n+\t\t\tint score;\n+\n+\t\t\t/* Find out if this basename is unique among dests */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue; /* not a unique basename; skip it */\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n+\t\t\t/* Estimate the similarity */\n+\t\t\tone = rename_src[src_index].p->one;\n+\t\t\ttwo = rename_dst[dst_index].p->two;\n+\t\t\tscore = estimate_similarity(options->repo, one, two,\n+\t\t\t\t\t\t minimum_score, skip_unmodified);\n+\n+\t\t\t/* If sufficiently similar, record as rename pair */\n+\t\t\tif (score < minimum_score)\n+\t\t\t\tcontinue;\n+\t\t\trecord_rename_pair(dst_index, src_index, score);\n+\t\t\trenames++;\n+\n+\t\t\t/*\n+\t\t\t * Found a rename so don't need text anymore; if we\n+\t\t\t * didn't find a rename, the filespec_blob would get\n+\t\t\t * re-used when doing the matrix of comparisons.\n+\t\t\t */\n+\t\t\tdiff_free_filespec_blob(one);\n+\t\t\tdiff_free_filespec_blob(two);\n+\t\t}\n+\t}\n \n \tstrintmap_clear(&sources);\n \tstrintmap_clear(&dests);\n \n-\treturn 0;\n+\treturn renames;\n }\n \n #define NUM_CANDIDATE_PER_DST 4\n-- \ngitgitgadget\n\n"},{"id":"416342","messageId":"pull.843.git.1612651937.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":null,"subject":"[PATCH 0/3] Optimization batch 7: use file basenames to guide rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-06T22:52:14Z","receivedAt":"2021-02-06T22:53:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on ort-perf-batch-6[1], which has not yet appeared in\nseen despite being reviewed by both Junio and Stolee.\n\nThis series uses file basenames in a basic fashion to guide rename\ndetection. It represents \"Optimization #3\" from my Git Merge 2020 talk[2],\nand is based on the fact that real world repositories tend to have a large\nmajority of the renames they have done in history be ones that do not affect\nthe basenames of the renamed files (in other words, they are simply moving\nfiles into different directories). For the testcases mentioned in commit\n557ac0350d (\"merge-ort: begin performance work; instrument with\ntrace2_region_* calls\", 2020-10-28), the changes in just this series\nimproves the performance as follows:\n\n Before Series After Series\nno-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\nmega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\njust-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n\n\nAs a reminder, before any merge-ort/diffcore-rename performance work, the\nperformance results we started with (as noted in the same commit message)\nwere:\n\nno-renames-am: 6.940 s ± 0.485 s\nno-renames: 18.912 s ± 0.174 s\nmega-renames: 5964.031 s ± 10.459 s\njust-one-mega: 149.583 s ± 0.751 s\n\n\n[1] https://lore.kernel.org/git/xmqqlfc4byt6.fsf@gitster.c.googlers.com/ [2]\nhttps://github.com/newren/presentations/blob/pdfs/merge-performance/merge-performance-slides.pdf\n\nElijah Newren (3):\n diffcore-rename: compute basenames of all source and dest candidates\n diffcore-rename: complete find_basename_matches()\n diffcore-rename: guide inexact rename detection based on basenames\n\n diffcore-rename.c | 181 +++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 177 insertions(+), 4 deletions(-)\n\n\nbase-commit: 7ae9460d3dba84122c2674b46e4339b9d42bdedd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-843%2Fnewren%2Fort-perf-batch-7-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-843/newren/ort-perf-batch-7-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/843\n-- \ngitgitgadget\n"},{"id":"416344","messageId":"1d941c35076e8d515c8ff7ef01d6b9d8c092aaa9.1612651937.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.git.1612651937.gitgitgadget@gmail.com","subject":"[PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-06T22:52:17Z","receivedAt":"2021-02-06T22:53:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMake use of the new find_basename_matches() function added in the last\ntwo patches, to find renames more rapidly in cases where we can match up\nfiles based on basenames.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\n mega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\n just-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 42 +++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 37 insertions(+), 5 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex b1dda41de9b1..206c0bbdcdfb 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,7 +367,6 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n-MAYBE_UNUSED\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n@@ -718,12 +717,45 @@ void diffcore_rename(struct diff_options *options)\n \tif (minimum_score == MAX_SCORE)\n \t\tgoto cleanup;\n \n+\tnum_sources = rename_src_nr;\n+\n+\tif (want_copies || break_idx) {\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones corresponding to exact renames\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\t} else {\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones involved in renames (found via exact match)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull exact\", options->repo);\n+\n+\t\t/* Utilize file basenames to quickly find renames. */\n+\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n+\t\trename_count += find_basename_matches(options, minimum_score,\n+\t\t\t\t\t\t rename_src_nr);\n+\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n+\n+\t\t/*\n+\t\t * Cull sources, again:\n+\t\t * - remove ones involved in renames (found via basenames)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull basename\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull basename\", options->repo);\n+\t}\n+\n \t/*\n-\t * Calculate how many renames are left\n+\t * Calculate how many rename destinations are left\n \t */\n \tnum_destinations = (rename_dst_nr - rename_count);\n-\tremove_unneeded_paths_from_src(want_copies);\n-\tnum_sources = rename_src_nr;\n+\tnum_sources = rename_src_nr; /* rename_src_nr reflects lower number */\n \n \t/* All done? */\n \tif (!num_destinations || !num_sources)\n@@ -755,7 +787,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tstruct diff_score *m;\n \n \t\tif (rename_dst[i].is_rename)\n-\t\t\tcontinue; /* dealt with exact match already. */\n+\t\t\tcontinue; /* exact or basename match already handled */\n \n \t\tm = &mx[dst_cnt * NUM_CANDIDATE_PER_DST];\n \t\tfor (j = 0; j < NUM_CANDIDATE_PER_DST; j++)\n-- \ngitgitgadget\n"},{"id":"416347","messageId":"xmqqtuqo8nfz.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"pull.843.git.1612651937.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/3] Optimization batch 7: use file basenames to guide rename detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-07T05:19:12Z","receivedAt":"2021-02-07T05:20:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> This series depends on ort-perf-batch-6[1], which has not yet appeared in\n> seen despite being reviewed by both Junio and Stolee.\n\nIt is because that one depends on something not in, but soon about\nto go in, 'master' and I didn't want to add to \"this topic depends\non top of that other topic\" mess.\n"},{"id":"416349","messageId":"CABPp-BFyWETzOM4C44HjHctSzg6Mr5BEn+OEO5RT888QX71CTA@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqtuqo8nfz.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 0/3] Optimization batch 7: use file basenames to guide rename detection","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-07T06:05:02Z","receivedAt":"2021-02-07T06:06:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Feb 6, 2021 at 9:19 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > This series depends on ort-perf-batch-6[1], which has not yet appeared in\n> > seen despite being reviewed by both Junio and Stolee.\n>\n> It is because that one depends on something not in, but soon about\n> to go in, 'master' and I didn't want to add to \"this topic depends\n> on top of that other topic\" mess.\n\nAh, makes sense.\n"},{"id":"416353","messageId":"9fbed0f9-032e-3f99-8467-f8a9cfa2d8f1@gmail.com","threadId":"55112","inReplyTo":"1d941c35076e8d515c8ff7ef01d6b9d8c092aaa9.1612651937.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-07T14:38:37Z","receivedAt":"2021-02-07T14:39:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/6/21 5:52 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Make use of the new find_basename_matches() function added in the last\n> two patches, to find renames more rapidly in cases where we can match up\n> files based on basenames.\n\nThis is a valuable heuristic.\n\n> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> performance work; instrument with trace2_region_* calls\", 2020-10-28),\n> this change improves the performance as follows:\n> \n> Before After\n> no-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\n> mega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\n> just-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n\nThese numbers are very impressive.\n\nBefore I get too deep into reviewing these patches, I do want\nto make it clear that the speed-up is coming at the cost of\na behavior change. We are restricting the \"best match\" search\nto be first among files with common base name (although maybe\nI would use 'suffix'?). If we search for a rename among all\nadditions and deletions ending the \".txt\" we might find a\nsimilarity match that is 60% and declare that a rename, even\nif there is a \".txt\" -> \".md\" pair that has a 70% match.\n\nThis could be documented in a test case, to demonstrate that\nwe are making this choice explicitly.\n\nFor example, here is a test that passes now, but would\nstart failing with your patches (if I understand them\ncorrectly):\n\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex c16486a9d41..e4c71fcf3be 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -262,4 +262,21 @@ test_expect_success 'diff-tree -l0 defaults to a big rename limit, not zero' '\n \tgrep \"myotherfile.*myfile\" actual\n '\n \n+test_expect_success 'multiple similarity choices' '\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 line10 >delete.txt &&\n+\tgit add delete.txt &&\n+\tgit commit -m \"base txt\" &&\n+\n+\trm delete.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 >add.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 >add.md &&\n+\tgit add add.txt add.md &&\n+\tgit commit -a -m \"rename\" &&\n+\tgit diff-tree -M HEAD HEAD^ >actual &&\n+\tgrep \"add.md\tdelete.txt\" actual\n+'\n+\n test_done\n\nPersonally, I'm fine with making this assumption. All of\nour renames are based on heuristics, so any opportunity\nto reduce the number of content comparisons is a win in\nmy mind. We also don't report a rename unless there _is_\nan add/delete pair that is sufficiently close in content.\n\nSo, in this way, we are changing the optimization function\nthat is used to determine the \"best\" rename available. It\nmight be good to update documentation for how we choose\nrenames:\n\n An add/delete pair is marked as a rename based on the\n following similarity function:\n\n 1. If the blob content is identical, then those files\n are marked as a rename. (Should we break ties here\n based on the basename?)\n\n 2. Among pairs whose content matches the minimum\n similarity limit, we optimize for:\n\n i. among files with the same basename (trailer\n after final '.') select pairs with highest\n similarity.\n\n ii. if no files with the same basename have the\n minimum similarity, then select pairs with\n highest similarity across all filenames.\n\nThe above was written quickly as an attempt, so it will\nrequire careful editing to actually make sense to end\nusers.\n\nThanks,\n-Stolee\n"},{"id":"416380","messageId":"xmqqsg677j2u.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"9fbed0f9-032e-3f99-8467-f8a9cfa2d8f1@gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-07T19:51:05Z","receivedAt":"2021-02-07T19:52:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Before I get too deep into reviewing these patches, I do want\n> to make it clear that the speed-up is coming at the cost of\n> a behavior change. We are restricting the \"best match\" search\n> to be first among files with common base name (although maybe\n> I would use 'suffix'?). If we search for a rename among all\n> additions and deletions ending the \".txt\" we might find a\n> similarity match that is 60% and declare that a rename, even\n> if there is a \".txt\" -> \".md\" pair that has a 70% match.\n\nYes, my initial reaction to the idea was that \"yuck, our rename\ndetection lost its purity\". diffcore-rename strived to base its\ndecision purely on content similarity, primarily because it is one\nof the oldest part of Git where the guiding principle has always\nbeen that the content is the king. I think my aversion to the \"all\nof my neighbors are relocating, so should I move to the same place\"\n(aka \"directory rename\") comes from a similar place, but in a sense\nthis was worse.\n\nAt least, until I got over the initial bump. I do not think the\nsuffix match is necessarily a bad idea, but it adds more \"magically\ndoing a wrong thing\" failure modes (e.g. the \".txt\" to \".md\" example\nwould probably have more variants that impact the real life\nprojects; \".C\" vs \".cc\" vs \".cxx\" vs \".cpp\" immediately comes to\nmind), and a tool that silently does a wrong thing because it uses\nmore magic would be a tool that is hard to explain why it did the\nwrong thing when it does.\n\n> This could be documented in a test case, to demonstrate that\n> we are making this choice explicitly.\n\nYes. I wonder if we can solve it by requiring a lot better than\nminimum match when trying the \"suffix match\" first, or something?\n\nProvided if we agree that it is a good idea to insert this between\n\"exact contents match\" and \"full matrix\", I have one question to\nElijah on what the code does.\n\nTo me, it seems that the \"full matrix\" part still uses the remaining\nsrc and dst candidates fully. But if \"A.txt\" and \"B.txt\" are still\nsurviving in the src/dst at that stage, shouldn't we be saying that\n\"no way these can be similar enough---we've checked in the middle\nstage where only the ones with the same suffix are considered and\nthis pair didn't turn into a rename\"?\n\nThanks.\n"},{"id":"416399","messageId":"CABPp-BHPgUHFFzTd7suhqj=zEXQ61vxKU6X9gZvow5a=TLg3iw@mail.gmail.com","threadId":"55112","inReplyTo":"9fbed0f9-032e-3f99-8467-f8a9cfa2d8f1@gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-08T08:27:00Z","receivedAt":"2021-02-08T08:27:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Sun, Feb 7, 2021 at 6:38 AM Derrick Stolee wrote:\n>\n> On 2/6/21 5:52 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Make use of the new find_basename_matches() function added in the last\n> > two patches, to find renames more rapidly in cases where we can match up\n> > files based on basenames.\n>\n> This is a valuable heuristic.\n>\n> > For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> > performance work; instrument with trace2_region_* calls\", 2020-10-28),\n> > this change improves the performance as follows:\n> >\n> > Before After\n> > no-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\n> > mega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\n> > just-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n>\n> These numbers are very impressive.\n>\n> Before I get too deep into reviewing these patches, I do want\n> to make it clear that the speed-up is coming at the cost of\n> a behavior change. We are restricting the \"best match\" search\n> to be first among files with common base name (although maybe\n> I would use 'suffix'?). If we search for a rename among all\n> additions and deletions ending the \".txt\" we might find a\n> similarity match that is 60% and declare that a rename, even\n> if there is a \".txt\" -> \".md\" pair that has a 70% match.\n\nI'm glad you all are open to possible behavioral changes, but I was\nproposing a much smaller behavioral change that is quite different\nthan what you have suggested here. Perhaps my wording was poor; I\napologize for forgetting that \"basename\" has different meanings in\ndifferent contexts. Let me try again; I am not treating the filename\nextension as special in any manner here; by \"basename\" I just mean the\nportion of the path ignoring any leading directories. Thus\n src/foo.txt\nmight be a good match against\n source/foo.txt\nbut this optimization as a preliminary step would not consider\nmatching src/foo.txt against any of\n source/bar.txt\n source/foo.md\nsince the basenames ('bar.txt' and 'foo.md') do not match our original\nfile's basename ('foo.txt').\n\nOf course, if this preliminary optimization step fails to find another\n\"foo.txt\" to match src/foo.txt against (or finds more than one and\nthus doesn't compare against any of them), then the fallback inexact\nrename detection matrix might match it against either of those two\nlatter paths, as it always has.\n\n> This could be documented in a test case, to demonstrate that\n> we are making this choice explicitly.\n>\n> For example, here is a test that passes now, but would\n> start failing with your patches (if I understand them\n> correctly):\n>\n> diff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\n> index c16486a9d41..e4c71fcf3be 100755\n> --- a/t/t4001-diff-rename.sh\n> +++ b/t/t4001-diff-rename.sh\n> @@ -262,4 +262,21 @@ test_expect_success 'diff-tree -l0 defaults to a big rename limit, not zero' '\n> grep \"myotherfile.*myfile\" actual\n> '\n>\n> +test_expect_success 'multiple similarity choices' '\n> + test_write_lines line1 line2 line3 line4 line5 \\\n> + line6 line7 line8 line9 line10 >delete.txt &&\n> + git add delete.txt &&\n> + git commit -m \"base txt\" &&\n> +\n> + rm delete.txt &&\n> + test_write_lines line1 line2 line3 line4 line5 \\\n> + line6 line7 line8 >add.txt &&\n> + test_write_lines line1 line2 line3 line4 line5 \\\n> + line6 line7 line8 line9 >add.md &&\n> + git add add.txt add.md &&\n> + git commit -a -m \"rename\" &&\n> + git diff-tree -M HEAD HEAD^ >actual &&\n> + grep \"add.md delete.txt\" actual\n> +'\n> +\n> test_done\n>\n> Personally, I'm fine with making this assumption. All of\n> our renames are based on heuristics, so any opportunity\n> to reduce the number of content comparisons is a win in\n> my mind. We also don't report a rename unless there _is_\n> an add/delete pair that is sufficiently close in content.\n>\n> So, in this way, we are changing the optimization function\n> that is used to determine the \"best\" rename available. It\n> might be good to update documentation for how we choose\n> renames:\n\nSeems reasonable; I'll add some commentary below on the rules...\n\n>\n> An add/delete pair is marked as a rename based on the\n> following similarity function:\n>\n\n0. Unless break detection is on, files with the same fullname are\nconsidered the same file even if their content is completely\ndifferent. (With break detection turned on, we can have e.g. both\nsrc/foo.txt -> src/bar.txt and otherdir/baz.txt -> src/foo.txt, i.e.\nsrc/foo.txt can be both a source and a destination of a rename.)\n\n[The merge machinery never turns break detection on, but\ndiffcore-rename is used by git diff, git log, etc. too, so if we're\ndocumenting the rules we should cover all the cases.]\n\n> 1. If the blob content is identical, then those files\n> are marked as a rename. (Should we break ties here\n> based on the basename?)\n\nfind_identical_files() already breaks ties based on basename_same(),\nyes. So there's another area of the code that uses basenames to guide\nrename detection already, just in a much more limited fashion.\n\n> 2. Among pairs whose content matches the minimum\n> similarity limit, we optimize for:\n>\n> i. among files with the same basename (trailer\n> after final '.') select pairs with highest\n> similarity.\n\nThis is an interesting idea, but is not what I implemented. It is\npossible that your suggestion is also a useful optimization; it'd be\nhard to know without trying. However, as noted in optimization batch\n8 that I'll be submitting later, I'm worried about having any\noptimization pre-steps doing more than O(1) comparisons per path (and\nhere you suggest comparing each .txt file with all other .txt files);\ndoing that can interact badly with optimization batch 9.\nAdditionally, unless we do something to avoid re-comparing files again\nwhen doing the later all-unmatched-files-against-each-other check,\nthen worst case behavior can approach twice as slow as the original\ncode.\n\nAnyway, the explanation I'd use for the optimization I've added in\nthis series is:\n\n i. if looking through the two sets (of add pairs, and of delete\npairs), there is exactly one file with the same basename from each\nset, and they have the minimum similarity, then mark them as a rename\n\nOptimization batch 8 will extend this particular rule.\n\nOptimization batches 9 and 10 will optimize the rename detection more,\nbut instead of using rule changes, will instead pass in a list of\n\"irrelevant\" sources that can be skipped. The trick is in determining\nsource files that are irrelevant and why. I'm not sure if we want to\nalso mention in the rules that different areas of the code (the merge\nmachinery, log --follow, etc.) can make the rename detection focus\njust on some \"relevant\" subset of files. (Which will also touch on\noptimization batch 12.)\n\n> ii. if no files with the same basename have the\n> minimum similarity, then select pairs with\n> highest similarity across all filenames.\n\nYes, this will remain as the fallback at the very end.\n\n> The above was written quickly as an attempt, so it will\n> require careful editing to actually make sense to end\n> users.\n\nYeah, and we probably also need to mention copy detection above\nsomehow too, and add more precise wording about how break detection is\ninvolved.\n"},{"id":"416403","messageId":"CABPp-BGAgi+ooq==ZY2tWif0--W4Cruz02GDvxueHe6GjQEAXQ@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqsg677j2u.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-08T08:38:00Z","receivedAt":"2021-02-08T08:42:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Feb 7, 2021 at 11:51 AM Junio C Hamano wrote:\n>\n> Derrick Stolee writes:\n>\n> > Before I get too deep into reviewing these patches, I do want\n> > to make it clear that the speed-up is coming at the cost of\n> > a behavior change. We are restricting the \"best match\" search\n> > to be first among files with common base name (although maybe\n> > I would use 'suffix'?). If we search for a rename among all\n> > additions and deletions ending the \".txt\" we might find a\n> > similarity match that is 60% and declare that a rename, even\n> > if there is a \".txt\" -> \".md\" pair that has a 70% match.\n>\n> Yes, my initial reaction to the idea was that \"yuck, our rename\n> detection lost its purity\". diffcore-rename strived to base its\n> decision purely on content similarity, primarily because it is one\n> of the oldest part of Git where the guiding principle has always\n> been that the content is the king. I think my aversion to the \"all\n> of my neighbors are relocating, so should I move to the same place\"\n> (aka \"directory rename\") comes from a similar place, but in a sense\n> this was worse.\n>\n> At least, until I got over the initial bump. I do not think the\n> suffix match is necessarily a bad idea, but it adds more \"magically\n> doing a wrong thing\" failure modes (e.g. the \".txt\" to \".md\" example\n> would probably have more variants that impact the real life\n> projects; \".C\" vs \".cc\" vs \".cxx\" vs \".cpp\" immediately comes to\n> mind), and a tool that silently does a wrong thing because it uses\n> more magic would be a tool that is hard to explain why it did the\n> wrong thing when it does.\n\nStolee explained a new algorithm different than what I have proposed,\nI think based on the apparent different meanings of \"basename\" that\nexist. I tried to clarify that in response to his email, but I wanted\nto clarify one additional thing here too:\n\ndiffcore-rename has not in the past based its decision solely on\ncontent similarity. It only does that when break detection is on.\nOtherwise, 0% content similarity is trumped by sufficient filename\nsimilarity (namely, with a filename similarity of 100%). If the\nfilename similarity wasn't sufficiently high (anything less than an\nexact match), then it completely ignored filename similarity and\nlooked only at content similarity. It thus jumped from one extreme to\nanother.\n\nMy optimization is adding an in-between state. When the basename (the\npart of the path excluding the leading directory) matches the basename\nof another file (and those basenames are unique on each side), then\ncompare content similarity and mark the files as a rename if the two\nare sufficiently similar. It is thus a position that considers both\nfilename similarity (basename match) and content similarity together.\n\n> > This could be documented in a test case, to demonstrate that\n> > we are making this choice explicitly.\n>\n> Yes. I wonder if we can solve it by requiring a lot better than\n> minimum match when trying the \"suffix match\" first, or something?\n\nThis may still be a useful idea, and was something I had considered,\nbut more in the context of more generic filename similarity\ncomparisons. We could still discuss it even when basenames match, but\nbasenames matching seems strong enough to me that I wasn't sure extra\nconfiguration knobs were warranted.\n\n> Provided if we agree that it is a good idea to insert this between\n> \"exact contents match\" and \"full matrix\", I have one question to\n> Elijah on what the code does.\n>\n> To me, it seems that the \"full matrix\" part still uses the remaining\n> src and dst candidates fully. But if \"A.txt\" and \"B.txt\" are still\n> surviving in the src/dst at that stage, shouldn't we be saying that\n> \"no way these can be similar enough---we've checked in the middle\n> stage where only the ones with the same suffix are considered and\n> this pair didn't turn into a rename\"?\n\nThis is a very good point. A.txt and B.txt will not have been\ncompared previously since their basenames do not match, but the basic\nidea is still possible. For example, A.txt could have been compared\nto source/some-module/A.txt. And I don't do anything in the final\n\"full matrix\" stage to avoid re-comparing those two files again.\nHowever, it is worth noting that A.txt will have been compared to at\nmost one other file, not N files. And thus while we are wasting some\nre-comparisons, it is at most O(N) duplicated comparisons, not O(N^2).\nI thought about that, but decided to not bother, based on the\nfollowing thinking:\n\n1) The most expensive comparison is the first one, because when we do\nthat one, we first have to populate the list of integers that lines in\nthe file hash to. Subsequent comparisons are relatively cheap since\nthis list of integers has already been computed.\n\n2) This would only save us from at most N comparisons in the N x M\nmatrix (since no file in this optimization is compared to more than\none other)\n\n3) Checking if two files have previously been compared requires more\ncode, in what is already a tight nested loop. My experience\nattempting to modify that tight loop for extra conditions (e.g. don't\ncompare files that are too large), is that it's easy to accidentally\nmake the code slower. In fact, this is in part what led to the\naddition of the remove_unneed_paths_from_src() function.\n\n4) There were plenty of other interesting ideas and maybe I was a tad lazy. :-)\n\nI think removing these already-compared cases could be done, but I\njust avoided it. If we were to do the \"attempt to match files with\nthe same extension\" optimization that Stolee outlines/invents above,\nthen we'd definitely need to consider it. Otherwise, it's just a\nminor additional optimization that someone could add to my patches.\n"},{"id":"416406","messageId":"b4eeb6a1-a801-e984-d245-f291a0cf9727@gmail.com","threadId":"55112","inReplyTo":"CABPp-BHPgUHFFzTd7suhqj=zEXQ61vxKU6X9gZvow5a=TLg3iw@mail.gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-08T11:31:13Z","receivedAt":"2021-02-08T11:35:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/8/2021 3:27 AM, Elijah Newren wrote:\n> Hi,\n> \n> On Sun, Feb 7, 2021 at 6:38 AM Derrick Stolee wrote:\n>>\n>> On 2/6/21 5:52 PM, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n>>>\n>>> Make use of the new find_basename_matches() function added in the last\n>>> two patches, to find renames more rapidly in cases where we can match up\n>>> files based on basenames.\n>>\n>> This is a valuable heuristic.\n>>\n>>> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n>>> performance work; instrument with trace2_region_* calls\", 2020-10-28),\n>>> this change improves the performance as follows:\n>>>\n>>> Before After\n>>> no-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\n>>> mega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\n>>> just-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n>>\n>> These numbers are very impressive.\n>>\n>> Before I get too deep into reviewing these patches, I do want\n>> to make it clear that the speed-up is coming at the cost of\n>> a behavior change. We are restricting the \"best match\" search\n>> to be first among files with common base name (although maybe\n>> I would use 'suffix'?). If we search for a rename among all\n>> additions and deletions ending the \".txt\" we might find a\n>> similarity match that is 60% and declare that a rename, even\n>> if there is a \".txt\" -> \".md\" pair that has a 70% match.\n> \n> I'm glad you all are open to possible behavioral changes, but I was\n> proposing a much smaller behavioral change that is quite different\n> than what you have suggested here. Perhaps my wording was poor; I\n> apologize for forgetting that \"basename\" has different meanings in\n> different contexts. Let me try again; I am not treating the filename\n> extension as special in any manner here; by \"basename\" I just mean the\n> portion of the path ignoring any leading directories. Thus\n> src/foo.txt\n> might be a good match against\n> source/foo.txt\n> but this optimization as a preliminary step would not consider\n> matching src/foo.txt against any of\n> source/bar.txt\n> source/foo.md\n> since the basenames ('bar.txt' and 'foo.md') do not match our original\n> file's basename ('foo.txt').\n> \n> Of course, if this preliminary optimization step fails to find another\n> \"foo.txt\" to match src/foo.txt against (or finds more than one and\n> thus doesn't compare against any of them), then the fallback inexact\n> rename detection matrix might match it against either of those two\n> latter paths, as it always has.\n\nThank you for making it clear that I had misunderstood what the\noptimization is actually doing. A much more narrow scope makes\nmore sense, and avoids the quadratic problem even when many files\nof the same suffix are renamed.\n\n>> This could be documented in a test case, to demonstrate that\n>> we are making this choice explicitly.\n\nMy test is thus bogus, but you could have a similar one for\nyour actual optimization.\n\n>> So, in this way, we are changing the optimization function\n>> that is used to determine the \"best\" rename available. It\n>> might be good to update documentation for how we choose\n>> renames:\n> \n> Seems reasonable; I'll add some commentary below on the rules...\n\nYour commentary is helpful. I look forward to reading your\ncarefully-written docs in the next version ;).\n\n>> i. among files with the same basename (trailer\n>> after final '.') select pairs with highest\n>> similarity.\n> \n> This is an interesting idea, but is not what I implemented.\n\nThat's what I get for reading the commit messages quickly and\ncommenting on what I _think_ is going on instead of actually\nreading the code carefully. Sorry about that.\n\n> It is\n> possible that your suggestion is also a useful optimization; it'd be\n> hard to know without trying. However, as noted in optimization batch\n> 8 that I'll be submitting later, I'm worried about having any\n> optimization pre-steps doing more than O(1) comparisons per path (and\n> here you suggest comparing each .txt file with all other .txt files);\n> doing that can interact badly with optimization batch 9.\n> Additionally, unless we do something to avoid re-comparing files again\n> when doing the later all-unmatched-files-against-each-other check,\n> then worst case behavior can approach twice as slow as the original\n> code.\n\nRight. If Git decides to reorganize all of its *.c files in one\ncommit, we would still get quadratic behavior in rename detection.\nMaybe it's not _that_ much of an improvement.\n\nThanks,\n-Stolee\n"},{"id":"416407","messageId":"57d30e7d-7727-8d98-e3ef-bcfeebf9edd3@gmail.com","threadId":"55112","inReplyTo":"CABPp-BGAgi+ooq==ZY2tWif0--W4Cruz02GDvxueHe6GjQEAXQ@mail.gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-08T11:43:59Z","receivedAt":"2021-02-08T11:47:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/8/2021 3:38 AM, Elijah Newren wrote:\n> On Sun, Feb 7, 2021 at 11:51 AM Junio C Hamano wrote:\n>>\n>> Derrick Stolee writes:\n>>\n>>> Before I get too deep into reviewing these patches, I do want\n>>> to make it clear that the speed-up is coming at the cost of\n>>> a behavior change. We are restricting the \"best match\" search\n>>> to be first among files with common base name (although maybe\n>>> I would use 'suffix'?). If we search for a rename among all\n>>> additions and deletions ending the \".txt\" we might find a\n>>> similarity match that is 60% and declare that a rename, even\n>>> if there is a \".txt\" -> \".md\" pair that has a 70% match.\n>>\n>> Yes, my initial reaction to the idea was that \"yuck, our rename\n>> detection lost its purity\". diffcore-rename strived to base its\n>> decision purely on content similarity, primarily because it is one\n>> of the oldest part of Git where the guiding principle has always\n>> been that the content is the king. I think my aversion to the \"all\n>> of my neighbors are relocating, so should I move to the same place\"\n>> (aka \"directory rename\") comes from a similar place, but in a sense\n>> this was worse.\n>>\n>> At least, until I got over the initial bump. I do not think the\n>> suffix match is necessarily a bad idea, but it adds more \"magically\n>> doing a wrong thing\" failure modes (e.g. the \".txt\" to \".md\" example\n>> would probably have more variants that impact the real life\n>> projects; \".C\" vs \".cc\" vs \".cxx\" vs \".cpp\" immediately comes to\n>> mind), and a tool that silently does a wrong thing because it uses\n>> more magic would be a tool that is hard to explain why it did the\n>> wrong thing when it does.\n> \n> Stolee explained a new algorithm different than what I have proposed,\n\nYes, sorry for adding noise. The point stands that we are changing\nthe behavior in some cases, so that must be agreed upon. What you\nare _actually_ proposing is a much smaller change than I thought,\nbut it is still worth pointing out the behavior change.\n\n> I think based on the apparent different meanings of \"basename\" that\n> exist. I tried to clarify that in response to his email, but I wanted\n> to clarify one additional thing here too:\n>> diffcore-rename has not in the past based its decision solely on\n> content similarity. It only does that when break detection is on.\n> Otherwise, 0% content similarity is trumped by sufficient filename\n> similarity (namely, with a filename similarity of 100%). If the\n> filename similarity wasn't sufficiently high (anything less than an\n> exact match), then it completely ignored filename similarity and\n> looked only at content similarity. It thus jumped from one extreme to\n> another.\n\nThis idea of optimizing first for 100% filename similarity is a\ngood perspective on Git's rename detection algorithm. The canonical\nexample of this 100% filename similarity is a rename cycle:\n\n\tA -> B\n\tB -> C\n\tC -> A\n\nEven if the OIDs are distinct and exactly match across these renames,\nwe see that there are no adds or deletes, so we do not even trigger\nrename detection and report A, B, and C as edited instead.\n\nA \"rename path\" (not cycle) such as:\n\n\tA -> B\n\tB -> C\n\ndoes trigger rename detection, but B will never be considered. Instead,\n\"A -> C\" will be checked for similarity to see if it is within the\nthreshold.\n\nOf course, I am _not_ advocating that we change this behavior. These\nsituations are incredibly rare and we should not sacrifice performance\nin the typical case to handle them.\n\n> My optimization is adding an in-between state. When the basename (the\n> part of the path excluding the leading directory) matches the basename\n> of another file (and those basenames are unique on each side), then\n> compare content similarity and mark the files as a rename if the two\n> are sufficiently similar. It is thus a position that considers both\n> filename similarity (basename match) and content similarity together.\n> \n>>> This could be documented in a test case, to demonstrate that\n>>> we are making this choice explicitly.\n>>\n>> Yes. I wonder if we can solve it by requiring a lot better than\n>> minimum match when trying the \"suffix match\" first, or something?\n> \n> This may still be a useful idea, and was something I had considered,\n> but more in the context of more generic filename similarity\n> comparisons. We could still discuss it even when basenames match, but\n> basenames matching seems strong enough to me that I wasn't sure extra\n> configuration knobs were warranted.\n\nI think this is a complication that we might not want to add to the\nheuristic, at least not at first. We might want to have a follow-up\nthat adjusts that value to be higher. A natural way would be through\na config option, so users can select something incredibly high like\n99%. Another option would be to take a minimum that is halfway between\nthe existing similarity minimum and 100%.\n\n>> Provided if we agree that it is a good idea to insert this between\n>> \"exact contents match\" and \"full matrix\", I have one question to\n>> Elijah on what the code does.\n>>\n>> To me, it seems that the \"full matrix\" part still uses the remaining\n>> src and dst candidates fully. But if \"A.txt\" and \"B.txt\" are still\n>> surviving in the src/dst at that stage, shouldn't we be saying that\n>> \"no way these can be similar enough---we've checked in the middle\n>> stage where only the ones with the same suffix are considered and\n>> this pair didn't turn into a rename\"?\n> \n> This is a very good point. A.txt and B.txt will not have been\n> compared previously since their basenames do not match, but the basic\n> idea is still possible. For example, A.txt could have been compared\n> to source/some-module/A.txt. And I don't do anything in the final\n> \"full matrix\" stage to avoid re-comparing those two files again.\n> However, it is worth noting that A.txt will have been compared to at\n> most one other file, not N files. And thus while we are wasting some\n> re-comparisons, it is at most O(N) duplicated comparisons, not O(N^2).\n> I thought about that, but decided to not bother, based on the\n> following thinking:\n> \n> 1) The most expensive comparison is the first one, because when we do\n> that one, we first have to populate the list of integers that lines in\n> the file hash to. Subsequent comparisons are relatively cheap since\n> this list of integers has already been computed.\n> \n> 2) This would only save us from at most N comparisons in the N x M\n> matrix (since no file in this optimization is compared to more than\n> one other)\n> \n> 3) Checking if two files have previously been compared requires more\n> code, in what is already a tight nested loop. My experience\n> attempting to modify that tight loop for extra conditions (e.g. don't\n> compare files that are too large), is that it's easy to accidentally\n> make the code slower. In fact, this is in part what led to the\n> addition of the remove_unneed_paths_from_src() function.\n\nEven storing a single bit to say \"these were already compared\" takes\nquadratic space. The hope is to not have quadratic behavior if it\ncan be avoided.\n\n> 4) There were plenty of other interesting ideas and maybe I was a tad lazy. :-)\n> \n> I think removing these already-compared cases could be done, but I\n> just avoided it. If we were to do the \"attempt to match files with\n> the same extension\" optimization that Stolee outlines/invents above,\n> then we'd definitely need to consider it. Otherwise, it's just a\n> minor additional optimization that someone could add to my patches.\n\nThe more I think about it, the less my idea makes sense. I'm sorry\nfor adding noise to the thread.\n\nThanks,\n-Stolee\n"},{"id":"416416","messageId":"CABPp-BH7iVoCmnDqtQeyO6kgp37MGMbHA0X-4H+YJFD6jeUarw@mail.gmail.com","threadId":"55112","inReplyTo":"b4eeb6a1-a801-e984-d245-f291a0cf9727@gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-08T16:09:59Z","receivedAt":"2021-02-08T16:11:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 8, 2021 at 3:31 AM Derrick Stolee wrote:\n>\n> On 2/8/2021 3:27 AM, Elijah Newren wrote:\n> > Hi,\n> >\n> > On Sun, Feb 7, 2021 at 6:38 AM Derrick Stolee wrote:\n> >>\n> >> On 2/6/21 5:52 PM, Elijah Newren via GitGitGadget wrote:\n> >>> From: Elijah Newren \n> >>>\n> >>> Make use of the new find_basename_matches() function added in the last\n> >>> two patches, to find renames more rapidly in cases where we can match up\n> >>> files based on basenames.\n> >>\n> >> This is a valuable heuristic.\n> >>\n> >>> For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n> >>> performance work; instrument with trace2_region_* calls\", 2020-10-28),\n> >>> this change improves the performance as follows:\n> >>>\n> >>> Before After\n> >>> no-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\n> >>> mega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\n> >>> just-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n> >>\n> >> These numbers are very impressive.\n> >>\n> >> Before I get too deep into reviewing these patches, I do want\n> >> to make it clear that the speed-up is coming at the cost of\n> >> a behavior change. We are restricting the \"best match\" search\n> >> to be first among files with common base name (although maybe\n> >> I would use 'suffix'?). If we search for a rename among all\n> >> additions and deletions ending the \".txt\" we might find a\n> >> similarity match that is 60% and declare that a rename, even\n> >> if there is a \".txt\" -> \".md\" pair that has a 70% match.\n> >\n> > I'm glad you all are open to possible behavioral changes, but I was\n> > proposing a much smaller behavioral change that is quite different\n> > than what you have suggested here. Perhaps my wording was poor; I\n> > apologize for forgetting that \"basename\" has different meanings in\n> > different contexts. Let me try again; I am not treating the filename\n> > extension as special in any manner here; by \"basename\" I just mean the\n> > portion of the path ignoring any leading directories. Thus\n> > src/foo.txt\n> > might be a good match against\n> > source/foo.txt\n> > but this optimization as a preliminary step would not consider\n> > matching src/foo.txt against any of\n> > source/bar.txt\n> > source/foo.md\n> > since the basenames ('bar.txt' and 'foo.md') do not match our original\n> > file's basename ('foo.txt').\n> >\n> > Of course, if this preliminary optimization step fails to find another\n> > \"foo.txt\" to match src/foo.txt against (or finds more than one and\n> > thus doesn't compare against any of them), then the fallback inexact\n> > rename detection matrix might match it against either of those two\n> > latter paths, as it always has.\n>\n> Thank you for making it clear that I had misunderstood what the\n> optimization is actually doing. A much more narrow scope makes\n> more sense, and avoids the quadratic problem even when many files\n> of the same suffix are renamed.\n>\n> >> This could be documented in a test case, to demonstrate that\n> >> we are making this choice explicitly.\n>\n> My test is thus bogus, but you could have a similar one for\n> your actual optimization.\n\nYes, good point.\n\n> >> So, in this way, we are changing the optimization function\n> >> that is used to determine the \"best\" rename available. It\n> >> might be good to update documentation for how we choose\n> >> renames:\n> >\n> > Seems reasonable; I'll add some commentary below on the rules...\n>\n> Your commentary is helpful. I look forward to reading your\n> carefully-written docs in the next version ;).\n\n:-)\n\n> >> i. among files with the same basename (trailer\n> >> after final '.') select pairs with highest\n> >> similarity.\n> >\n> > This is an interesting idea, but is not what I implemented.\n>\n> That's what I get for reading the commit messages quickly and\n> commenting on what I _think_ is going on instead of actually\n> reading the code carefully. Sorry about that.\n\nThere's absolutely no need to apologize. If you read all three commit\nmessages and you didn't understand the idea, then clearly there's a\nbug in my commit messages. Thanks for highlighting it; I'll figure\nout how to reword or add extra verbiage to make it clear. Something\nin these follow-up emails seemed to work, so I'll try to incorporate\nstuff from them.\n\n> > It is\n> > possible that your suggestion is also a useful optimization; it'd be\n> > hard to know without trying. However, as noted in optimization batch\n> > 8 that I'll be submitting later, I'm worried about having any\n> > optimization pre-steps doing more than O(1) comparisons per path (and\n> > here you suggest comparing each .txt file with all other .txt files);\n> > doing that can interact badly with optimization batch 9.\n> > Additionally, unless we do something to avoid re-comparing files again\n> > when doing the later all-unmatched-files-against-each-other check,\n> > then worst case behavior can approach twice as slow as the original\n> > code.\n>\n> Right. If Git decides to reorganize all of its *.c files in one\n> commit, we would still get quadratic behavior in rename detection.\n> Maybe it's not _that_ much of an improvement.\n>\n> Thanks,\n> -Stolee\n"},{"id":"416417","messageId":"CABPp-BF-YB4gjAPFWExY01ffLY1oLdUhih99AER9Sy0xaA1VfQ@mail.gmail.com","threadId":"55112","inReplyTo":"57d30e7d-7727-8d98-e3ef-bcfeebf9edd3@gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-08T16:25:00Z","receivedAt":"2021-02-08T16:26:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 8, 2021 at 3:44 AM Derrick Stolee wrote:\n>\n> On 2/8/2021 3:38 AM, Elijah Newren wrote:\n> > On Sun, Feb 7, 2021 at 11:51 AM Junio C Hamano wrote:\n> >>\n> >> Derrick Stolee writes:\n> >>\n> >>> Before I get too deep into reviewing these patches, I do want\n> >>> to make it clear that the speed-up is coming at the cost of\n> >>> a behavior change. We are restricting the \"best match\" search\n> >>> to be first among files with common base name (although maybe\n> >>> I would use 'suffix'?). If we search for a rename among all\n> >>> additions and deletions ending the \".txt\" we might find a\n> >>> similarity match that is 60% and declare that a rename, even\n> >>> if there is a \".txt\" -> \".md\" pair that has a 70% match.\n> >>\n> >> Yes, my initial reaction to the idea was that \"yuck, our rename\n> >> detection lost its purity\". diffcore-rename strived to base its\n> >> decision purely on content similarity, primarily because it is one\n> >> of the oldest part of Git where the guiding principle has always\n> >> been that the content is the king. I think my aversion to the \"all\n> >> of my neighbors are relocating, so should I move to the same place\"\n> >> (aka \"directory rename\") comes from a similar place, but in a sense\n> >> this was worse.\n> >>\n> >> At least, until I got over the initial bump. I do not think the\n> >> suffix match is necessarily a bad idea, but it adds more \"magically\n> >> doing a wrong thing\" failure modes (e.g. the \".txt\" to \".md\" example\n> >> would probably have more variants that impact the real life\n> >> projects; \".C\" vs \".cc\" vs \".cxx\" vs \".cpp\" immediately comes to\n> >> mind), and a tool that silently does a wrong thing because it uses\n> >> more magic would be a tool that is hard to explain why it did the\n> >> wrong thing when it does.\n> >\n> > Stolee explained a new algorithm different than what I have proposed,\n>\n> Yes, sorry for adding noise. The point stands that we are changing\n> the behavior in some cases, so that must be agreed upon. What you\n> are _actually_ proposing is a much smaller change than I thought,\n> but it is still worth pointing out the behavior change.\n\nAgain, no need to apologize; if my commit messages weren't clear, they\nneed to be fixed. I am much more surprised here by your repeated\npoint that it's worth pointing out the behavior change. I totally\nagree with that, and it's why I spent two paragraphs on the second\ncommit message explicitly covering this and listing four enumerated\nreasons to argue for the change. So, I thought I had done that, but\nit apparently isn't very clear...and I'm left wondering how to clarify\nit further. So, a question for you: How should I change it? Should I\njust modify the third commit message to re-highlight that it does\nchange behavior and refer to the second commit message for details?\nBecause repeating the point is the only way I can think of to make it\nclearer. Is there anything else I can or should do?\n\n> > I think based on the apparent different meanings of \"basename\" that\n> > exist. I tried to clarify that in response to his email, but I wanted\n> > to clarify one additional thing here too:\n> >> diffcore-rename has not in the past based its decision solely on\n> > content similarity. It only does that when break detection is on.\n> > Otherwise, 0% content similarity is trumped by sufficient filename\n> > similarity (namely, with a filename similarity of 100%). If the\n> > filename similarity wasn't sufficiently high (anything less than an\n> > exact match), then it completely ignored filename similarity and\n> > looked only at content similarity. It thus jumped from one extreme to\n> > another.\n>\n> This idea of optimizing first for 100% filename similarity is a\n> good perspective on Git's rename detection algorithm. The canonical\n> example of this 100% filename similarity is a rename cycle:\n>\n> A -> B\n> B -> C\n> C -> A\n>\n> Even if the OIDs are distinct and exactly match across these renames,\n> we see that there are no adds or deletes, so we do not even trigger\n> rename detection and report A, B, and C as edited instead.\n>\n> A \"rename path\" (not cycle) such as:\n>\n> A -> B\n> B -> C\n>\n> does trigger rename detection, but B will never be considered. Instead,\n> \"A -> C\" will be checked for similarity to see if it is within the\n> threshold.\n>\n> Of course, I am _not_ advocating that we change this behavior. These\n> situations are incredibly rare and we should not sacrifice performance\n> in the typical case to handle them.\n>\n> > My optimization is adding an in-between state. When the basename (the\n> > part of the path excluding the leading directory) matches the basename\n> > of another file (and those basenames are unique on each side), then\n> > compare content similarity and mark the files as a rename if the two\n> > are sufficiently similar. It is thus a position that considers both\n> > filename similarity (basename match) and content similarity together.\n> >\n> >>> This could be documented in a test case, to demonstrate that\n> >>> we are making this choice explicitly.\n> >>\n> >> Yes. I wonder if we can solve it by requiring a lot better than\n> >> minimum match when trying the \"suffix match\" first, or something?\n> >\n> > This may still be a useful idea, and was something I had considered,\n> > but more in the context of more generic filename similarity\n> > comparisons. We could still discuss it even when basenames match, but\n> > basenames matching seems strong enough to me that I wasn't sure extra\n> > configuration knobs were warranted.\n>\n> I think this is a complication that we might not want to add to the\n> heuristic, at least not at first. We might want to have a follow-up\n> that adjusts that value to be higher. A natural way would be through\n> a config option, so users can select something incredibly high like\n> 99%. Another option would be to take a minimum that is halfway between\n> the existing similarity minimum and 100%.\n>\n> >> Provided if we agree that it is a good idea to insert this between\n> >> \"exact contents match\" and \"full matrix\", I have one question to\n> >> Elijah on what the code does.\n> >>\n> >> To me, it seems that the \"full matrix\" part still uses the remaining\n> >> src and dst candidates fully. But if \"A.txt\" and \"B.txt\" are still\n> >> surviving in the src/dst at that stage, shouldn't we be saying that\n> >> \"no way these can be similar enough---we've checked in the middle\n> >> stage where only the ones with the same suffix are considered and\n> >> this pair didn't turn into a rename\"?\n> >\n> > This is a very good point. A.txt and B.txt will not have been\n> > compared previously since their basenames do not match, but the basic\n> > idea is still possible. For example, A.txt could have been compared\n> > to source/some-module/A.txt. And I don't do anything in the final\n> > \"full matrix\" stage to avoid re-comparing those two files again.\n> > However, it is worth noting that A.txt will have been compared to at\n> > most one other file, not N files. And thus while we are wasting some\n> > re-comparisons, it is at most O(N) duplicated comparisons, not O(N^2).\n> > I thought about that, but decided to not bother, based on the\n> > following thinking:\n> >\n> > 1) The most expensive comparison is the first one, because when we do\n> > that one, we first have to populate the list of integers that lines in\n> > the file hash to. Subsequent comparisons are relatively cheap since\n> > this list of integers has already been computed.\n> >\n> > 2) This would only save us from at most N comparisons in the N x M\n> > matrix (since no file in this optimization is compared to more than\n> > one other)\n> >\n> > 3) Checking if two files have previously been compared requires more\n> > code, in what is already a tight nested loop. My experience\n> > attempting to modify that tight loop for extra conditions (e.g. don't\n> > compare files that are too large), is that it's easy to accidentally\n> > make the code slower. In fact, this is in part what led to the\n> > addition of the remove_unneed_paths_from_src() function.\n>\n> Even storing a single bit to say \"these were already compared\" takes\n> quadratic space. The hope is to not have quadratic behavior if it\n> can be avoided.\n>\n> > 4) There were plenty of other interesting ideas and maybe I was a tad lazy. :-)\n> >\n> > I think removing these already-compared cases could be done, but I\n> > just avoided it. If we were to do the \"attempt to match files with\n> > the same extension\" optimization that Stolee outlines/invents above,\n> > then we'd definitely need to consider it. Otherwise, it's just a\n> > minor additional optimization that someone could add to my patches.\n>\n> The more I think about it, the less my idea makes sense. I'm sorry\n> for adding noise to the thread.\n>\n> Thanks,\n> -Stolee\n"},{"id":"416421","messageId":"xmqq4kim7964.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"CABPp-BGAgi+ooq==ZY2tWif0--W4Cruz02GDvxueHe6GjQEAXQ@mail.gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-08T17:37:23Z","receivedAt":"2021-02-08T17:38:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> idea is still possible. For example, A.txt could have been compared\n> to source/some-module/A.txt. And I don't do anything in the final\n> \"full matrix\" stage to avoid re-comparing those two files again.\n> However, it is worth noting that A.txt will have been compared to at\n> most one other file, not N files.\n\nSorry, but where does this \"at most one other file\" come from? \"It\nis rare to remove source/some-other-module/A.txt at the same time\nwhile the above is happening\"? If so, yes, that sounds like a\nsensible thing.\n\n> 1) The most expensive comparison is the first one,...\n\nYes. we keep the spanhash table across comparison.\n\n> 2) This would only save us from at most N comparisons in the N x M\n> matrix (since no file in this optimization is compared to more than\n> one other)\n\nTrue, but doesn't rename_src[] and rename_dst[] entries have the\noriginal pathname, where you can see A.txt and some-module/A.txt\nshare the same filename part cheaply? Is that more expensive than\ncomparing spanhash tables?\n\nHaving asked these, I do think it is not worth pursuing, especially\nbecause I agree with Derrick that this \"we see a new file whose name\nis the same as the one deleted from a different directory, so if\nthey are similar enough, let's declare victory and not bother\nfinding a better match\" needs to be used with higher similarity bar\nthan the normal one. If -M60 says \"only consider pairs that are\nwith at least 60% similarity index\", finding one at 60% similarity\nand stopping at it only because the pair looks to move a file from\none directory to another directory while retaining the same name,\nrejecting other paring, feels a bit too crude a heuristics. And if\nwe require higher similarity levels to short-circuit, the later full\nmatrix stage won't be helped with \"we must have already rejected\"\nlogic. A.txt and some-module/A.txt may not have been similar enough\nto short-circuit and reject others in the earlier part, but the\nfull-matrix part work at a lower bar, which may consider the pair\ngood enough to keep as match candidates.\n\nThanks.\n\n"},{"id":"416448","messageId":"CABPp-BH_DWEE-3M96e=PPNwDqeYPaax9s1kBDhS8a6GtxsW=Mg@mail.gmail.com","threadId":"55112","inReplyTo":"xmqq4kim7964.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-08T22:00:30Z","receivedAt":"2021-02-08T22:01:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Mon, Feb 8, 2021 at 9:37 AM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > idea is still possible. For example, A.txt could have been compared\n> > to source/some-module/A.txt. And I don't do anything in the final\n> > \"full matrix\" stage to avoid re-comparing those two files again.\n> > However, it is worth noting that A.txt will have been compared to at\n> > most one other file, not N files.\n>\n> Sorry, but where does this \"at most one other file\" come from? \"It\n> is rare to remove source/some-other-module/A.txt at the same time\n> while the above is happening\"? If so, yes, that sounds like a\n> sensible thing.\n\nIt comes from the current implementation. If both src/module1/A.txt\nand src/module2/A.txt were removed, then I don't have a unique 'A.txt'\nthat was deleted. In such a case, all 'A.txt' files are excluded from\nthis optimization step -- both sources and destinations. This\noptimization only kicks in for basenames where there was exactly one\nof them deleted somewhere, and exactly one of them added somewhere.\n\nOne could consider trying to compare all deleted 'A.txt' files with\nall added 'A.txt' files. I tried that, but it interacts badly with\noptimization batch 9 and tossed it aside; I will not be submitting\nsuch a method.\n\nNaturally, this \"unique basename\" limitation presents problems for\nfile basenames like .gitignore, Makefile, build.gradle, or even\nObjectFactory.java, setup.c, etc. that tend to appear in several\nlocations throughout the tree. As of this series, we have to fall\nback to the full N x M matrix comparison to detect the renames for\nsuch non-unique basenames. The next series I am planning on\nsubmitting will do something smarter for those files while still\nensuring that the preliminary step only compares any given file to at\nmost one other file.\n\n> > 1) The most expensive comparison is the first one,...\n>\n> Yes. we keep the spanhash table across comparison.\n>\n> > 2) This would only save us from at most N comparisons in the N x M\n> > matrix (since no file in this optimization is compared to more than\n> > one other)\n>\n> True, but doesn't rename_src[] and rename_dst[] entries have the\n> original pathname, where you can see A.txt and some-module/A.txt\n> share the same filename part cheaply? Is that more expensive than\n> comparing spanhash tables?\n\nFor a small enough number of renames, no, it won't be more expensive.\nBut I don't want to optimize for low numbers of renames; the code is\nfast enough for those. And with a large enough number of renames,\nyes, the basename comparisons in aggregate will be more expensive than\nthe number of spanhash array comparisons you avoid redoing. The\npreliminary step from this optimization at most only did O(N) spanhash\ncomparisons, because it would only compare any given file to at most\none other file. (Any file that didn't have a matching basename on the\nother side, or wasn't a unique basename, wouldn't have been compared\nto anything.) So, at most, we save O(N) spanhash comparisons. In\norder to avoid repeating those O(N) comparisons, you are adding O(NxM)\nbasename comparisons. Once M is large enough, the O(NxM) basename\ncomparisons you added will be more expensive than the O(N) spanhash\ncomparisons you are saving. Recall that my testcase used N and M of\napproximately 26,000. The real world repository I based it on had\nover 30K renames. And if I know of a repository with 30K renames with\nonly 50K files (at the time), I think we shouldn't be using that as an\nupper bound either.\n\n> Having asked these, I do think it is not worth pursuing, especially\n> because I agree with Derrick that this \"we see a new file whose name\n> is the same as the one deleted from a different directory, so if\n> they are similar enough, let's declare victory and not bother\n> finding a better match\" needs to be used with higher similarity bar\n> than the normal one.\n\nYou say you agree with Stolee, but that's not what I understood Stolee\nas saying at all. He said he thought it wasn't worth the complication\nof trying to use a different value for the basename minimum similarity\nthan the normal minimum similarity, at least not at first. He\nsuggested we could add that in the future at some time, and then\ntalked a bit about how to add it if we do.\n\n> If -M60 says \"only consider pairs that are\n> with at least 60% similarity index\", finding one at 60% similarity\n> and stopping at it only because the pair looks to move a file from\n> one directory to another directory while retaining the same name,\n> rejecting other paring, feels a bit too crude a heuristics. And if\n> we require higher similarity levels to short-circuit, the later full\n> matrix stage won't be helped with \"we must have already rejected\"\n> logic. A.txt and some-module/A.txt may not have been similar enough\n> to short-circuit and reject others in the earlier part, but the\n> full-matrix part work at a lower bar, which may consider the pair\n> good enough to keep as match candidates.\n\nI'm sorry, but I'm not following you. As best I can tell, you seem to\nbe suggesting that if we were to use a higher similarity bar for\nchecking same-basename files, that such a difference would end up not\naccelerating the diffcore-rename algorithm at all? Is that correct?\nIf not, I don't understand what you're saying.\n\nIf by chance my restatement is an accurate summary of your claim, then\nallow me to disabuse you of your assumptions here; you're way off. I\nwrote find_basename_matches() to take a similarity score, so that it\ncould take a different one than is used elsewhere in the algorithm. I\ndidn't think it was necessary, but it does make it easy to test your\nhypothesis. Here are some results:\n\nOriginal, not using basename-guided rename detection:\n no-renames: 13.815 s ± 0.062 s\n mega-renames: 1799.937 s ± 0.493 s\n just-one-mega: 51.289 s ± 0.019 s\n\nUsing basename_min_score = minimum_score, i.e. 50%:\n no-renames: 13.428 s ± 0.119 s\n mega-renames: 172.137 s ± 0.958 s\n just-one-mega: 5.154 s ± 0.025 s\n\nUsing basename_min_score = 0.5 * (minimum_score + MAX_SCORE), i.e. 75%:\n no-renames: 13.543 s ± 0.094 s\n mega-renames: 189.598 s ± 0.726 s\n just-one-mega: 5.647 s ± 0.016 s\n\nUsing basename_min_score = 0.1 * (minimum_score + 9*MAX_SCORE), i.e. 95%:\n no-renames: 13.733 s ± 0.086 s\n mega-renames: 353.479 s ± 2.574 s\n just-one-mega: 10.351 s ± 0.030 s\n\n\nSo, when we bump the bar for basename similarity much past your\nhypothetical 60% all the way up to 75% (i.e. just take a simple\naverage of minimum score and MAX_SCORE), we see almost identical\nspeedups (factor of 9 or so instead of 10 or so). And even when we go\nto the extreme of requiring a 95% or greater similarity in order to\npair up basenames, we still see a speed-up factor of 5-6; that's less\nthan the factor of 10 we could get by allowing basename_min_score to\nmatch minimum_score at 50%, but it's still _most_ of the speedup.\n\nGranted, this is just one testcase. It's going to vary a lot between\ntestcases and repositories and how far back or forward in history you\nare rebasing or merging, etc. The fact that this particular testcase\nwas obtained by doing a \"git mv drivers/ pilots/\" in the linux kernel\nand then finding a topic to rebase across that rename boundary makes\nit a bit special. But....even if we were only able to pair 50% of the\nfiles due to basename similarity, that would save 75% of the spanhash\ncomparisons. Even in git.git where only 16% of the renames change the\nbasename, if we could pair up 16% of the files based on basenames it'd\nsave roughly 30% of the spanhash comparisons. The numbers are\nprobably a lot better than either of those, though. Since 76% of\nrenames in the linux kernel don't change the basename, 64% of the\nrenames in gcc don't, over 79% of the renames in the gecko repository\ndon't, and over 89% of the renames in the WebKit repository don't, I\nthink this is a really valuable heuristic to use.\n"},{"id":"416462","messageId":"xmqqzh0e2kib.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"CABPp-BH_DWEE-3M96e=PPNwDqeYPaax9s1kBDhS8a6GtxsW=Mg@mail.gmail.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-08T23:43:40Z","receivedAt":"2021-02-08T23:45:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> I'm sorry, but I'm not following you. As best I can tell, you seem to\n> be suggesting that if we were to use a higher similarity bar for\n> checking same-basename files, that such a difference would end up not\n> accelerating the diffcore-rename algorithm at all?\n\nNo. If we assume we use the minimum similarity threashold in the\nnew middle step that consider only the files that were moved across\ndirectories without changing their names, and the last \"full matrix\"\nstep sees a src that did *not* pair with a dst of the same name in a\ndifferent directory surviving, we know that the pair would not be\nsimilar enough (because we are using the same \"minimum similarity\"\nin the middle step and the full matrix step) without comparing them\nagain. But if we used higher similarity in the middle step, the\nfact that such a src/dst pair surviving the middle step without\nproducing a match only means that the pair was not similar enough\nwith a raised bar used in the middle, and the full-matrix step need\nto consider the possibility that they may still be similar enough\nwhen using \"minimum similarity\" used for all the other pairs.\n\nAnd because I was assuming that requiring higher similarity in the\nmiddle step would be a prudent thing to do to avoid false matches\nthat discard better matches elsewhere, my conclusion was that it\nwould not be a useful optimization to do in the final full-matrix\nstep to see if a pair is something that was a candidate in the\nmiddle step but did not match well enough (because the fact that the\npair did not compare well enough with higher bar does not mean it\nwould not compare well to pass the lower \"minimum\" bar).\n\n\n"},{"id":"416464","messageId":"CABPp-BE9dPYgTsrAKjjmPTfy-xY56ajg-1ZYPf7X97YR0T_n3Q@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqzh0e2kib.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 3/3] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-08T23:52:45Z","receivedAt":"2021-02-08T23:53:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 8, 2021 at 3:43 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > I'm sorry, but I'm not following you. As best I can tell, you seem to\n> > be suggesting that if we were to use a higher similarity bar for\n> > checking same-basename files, that such a difference would end up not\n> > accelerating the diffcore-rename algorithm at all?\n>\n> No. If we assume we use the minimum similarity threashold in the\n> new middle step that consider only the files that were moved across\n> directories without changing their names, and the last \"full matrix\"\n> step sees a src that did *not* pair with a dst of the same name in a\n> different directory surviving, we know that the pair would not be\n> similar enough (because we are using the same \"minimum similarity\"\n> in the middle step and the full matrix step) without comparing them\n> again. But if we used higher similarity in the middle step, the\n> fact that such a src/dst pair surviving the middle step without\n> producing a match only means that the pair was not similar enough\n> with a raised bar used in the middle, and the full-matrix step need\n> to consider the possibility that they may still be similar enough\n> when using \"minimum similarity\" used for all the other pairs.\n>\n> And because I was assuming that requiring higher similarity in the\n> middle step would be a prudent thing to do to avoid false matches\n> that discard better matches elsewhere, my conclusion was that it\n> would not be a useful optimization to do in the final full-matrix\n> step to see if a pair is something that was a candidate in the\n> middle step but did not match well enough (because the fact that the\n> pair did not compare well enough with higher bar does not mean it\n> would not compare well to pass the lower \"minimum\" bar).\n\nAh, gotcha! Thanks for clarifying. Yes, yet another reason to not\neven try to avoid \"redoing\" the O(N) spanhash comparisons.\n"},{"id":"416494","messageId":"dcd0175229aa6fba576425e78875b95385acb58d.1612870326.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v2.git.1612870326.gitgitgadget@gmail.com","subject":"[PATCH v2 2/4] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-09T11:32:04Z","receivedAt":"2021-02-09T11:42:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIt is not uncommon in real world repositories for the majority of file\nrenames to not change the basename of the file; i.e. most \"renames\" are\njust a move of files into different directories. We can make use of\nthis to avoid comparing all rename source candidates with all rename\ndestination candidates, by first comparing sources to destinations with\nthe same basenames. If two files with the same basename are\nsufficiently similar, we record the rename; if not, we include those\nfiles in the more exhaustive matrix comparison.\n\nThis means we are adding a set of preliminary additional comparisons,\nbut for each file we only compare it with at most one other file. For\nexample, if there was a include/media/device.h that was deleted and a\nsrc/module/media/device.h that was added, and there were no other\ndevice.h files added or deleted between the commits being compared,\nthen these two files would be compared in the preliminary step.\n\nThis commit does not yet actually employ this new optimization, it\nmerely adds a function which can be used for this purpose. The next\ncommit will do the necessary plumbing to make use of it.\n\nNote that this optimization might give us different results than without\nthe optimization, because it's possible that despite files with the same\nbasename being sufficiently similar to be considered a rename, there's\nan even better match between files without the same basename. I think\nthat is okay for four reasons: (1) it's easy to explain to the users\nwhat happened if it does ever occur (or even for them to intuitively\nfigure out), (2) as the next patch will show it provides such a large\nperformance boost that it's worth the tradeoff, and (3) it's somewhat\nunlikely that despite having unique matching basenames that other files\nserve as better matches. Reason (4) takes a full paragraph to\nexplain...\n\nIf the previous three reasons aren't enough, consider what rename\ndetection already does. Break detection is not the default, meaning\nthat if files have the same _fullname_, then they are considered related\neven if they are 0% similar. In fact, in such a case, we don't even\nbother comparing the files to see if they are similar let alone\ncomparing them to all other files to see what they are most similar to.\nBasically, we override content similarity based on sufficient filename\nsimilarity. Without the filename similarity (currently implemented as\nan exact match of filename), we swing the pendulum the opposite\ndirection and say that filename similarity is irrelevant and compare a\nfull N x M matrix of sources and destinations to find out which have the\nmost similar contents. This optimization just adds another form of\nfilename similarity comparison, but augments it with a file content\nsimilarity check as well. Basically, if two files have the same\nbasename and are sufficiently similar to be considered a rename, mark\nthem as such without comparing the two to all other rename candidates.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 94 +++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 91 insertions(+), 3 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 1c52077b04e5..b1dda41de9b1 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -372,10 +372,48 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n {\n-\tint i;\n+\t/*\n+\t * When I checked, over 76% of file renames in linux just moved\n+\t * files to a different directory but kept the same basename. gcc\n+\t * did that with over 64% of renames, gecko did it with over 79%,\n+\t * and WebKit did it with over 89%.\n+\t *\n+\t * Therefore we can bypass the normal exhaustive NxM matrix\n+\t * comparison of similarities between all potential rename sources\n+\t * and destinations by instead using file basename as a hint, checking\n+\t * for similarity between files with the same basename, and if we\n+\t * find a pair that are sufficiently similar, record the rename\n+\t * pair and exclude those two from the NxM matrix.\n+\t *\n+\t * This *might* cause us to find a less than optimal pairing (if\n+\t * there is another file that we are even more similar to but has a\n+\t * different basename). Given the huge performance advantage\n+\t * basename matching provides, and given the frequency with which\n+\t * people use the same basename in real world projects, that's a\n+\t * trade-off we are willing to accept when doing just rename\n+\t * detection. However, if someone wants copy detection that\n+\t * implies they are willing to spend more cycles to find\n+\t * similarities between files, so it may be less likely that this\n+\t * heuristic is wanted.\n+\t */\n+\n+\tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n \n+\t/*\n+\t * The prefeteching stuff wants to know if it can skip prefetching blobs\n+\t * that are unmodified. unmodified blobs are only relevant when doing\n+\t * copy detection. find_basename_matches() is only used when detecting\n+\t * renames, not when detecting copies, so it'll only be used when a file\n+\t * only existed in the source. Since we already know that the file\n+\t * won't be unmodified, there's no point checking for it; that's just a\n+\t * waste of resources. So set skip_unmodified to 0 so that\n+\t * estimate_similarity() and prefetch() won't waste resources checking\n+\t * for something we already know is false.\n+\t */\n+\tint skip_unmodified = 0;\n+\n \t/* Create maps of basename -> fullname(s) for sources and dests */\n \tstrintmap_init_with_options(&sources, -1, NULL, 0);\n \tstrintmap_init_with_options(&dests, -1, NULL, 0);\n@@ -412,12 +450,62 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\tstrintmap_set(&dests, base, i);\n \t}\n \n-\t/* TODO: Make use of basenames source and destination basenames */\n+\t/* Now look for basename matchups and do similarity estimation */\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tchar *base = NULL;\n+\t\tintptr_t src_index;\n+\t\tintptr_t dst_index;\n+\n+\t\t/* Get the basename */\n+\t\tbase = strrchr(filename, '/');\n+\t\tbase = (base ? base+1 : filename);\n+\n+\t\t/* Find out if this basename is unique among sources */\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tif (src_index == -1)\n+\t\t\tcontinue; /* not a unique basename; skip it */\n+\t\tassert(src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n+\t\t\tstruct diff_filespec *one, *two;\n+\t\t\tint score;\n+\n+\t\t\t/* Find out if this basename is unique among dests */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue; /* not a unique basename; skip it */\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n+\t\t\t/* Estimate the similarity */\n+\t\t\tone = rename_src[src_index].p->one;\n+\t\t\ttwo = rename_dst[dst_index].p->two;\n+\t\t\tscore = estimate_similarity(options->repo, one, two,\n+\t\t\t\t\t\t minimum_score, skip_unmodified);\n+\n+\t\t\t/* If sufficiently similar, record as rename pair */\n+\t\t\tif (score < minimum_score)\n+\t\t\t\tcontinue;\n+\t\t\trecord_rename_pair(dst_index, src_index, score);\n+\t\t\trenames++;\n+\n+\t\t\t/*\n+\t\t\t * Found a rename so don't need text anymore; if we\n+\t\t\t * didn't find a rename, the filespec_blob would get\n+\t\t\t * re-used when doing the matrix of comparisons.\n+\t\t\t */\n+\t\t\tdiff_free_filespec_blob(one);\n+\t\t\tdiff_free_filespec_blob(two);\n+\t\t}\n+\t}\n \n \tstrintmap_clear(&sources);\n \tstrintmap_clear(&dests);\n \n-\treturn 0;\n+\treturn renames;\n }\n \n #define NUM_CANDIDATE_PER_DST 4\n-- \ngitgitgadget\n\n"},{"id":"416495","messageId":"ce2173aa1fb7ac3830bf8856a14558446e16a3dd.1612870326.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v2.git.1612870326.gitgitgadget@gmail.com","subject":"[PATCH v2 3/4] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-09T11:32:05Z","receivedAt":"2021-02-09T11:42:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMake use of the new find_basename_matches() function added in the last\ntwo patches, to find renames more rapidly in cases where we can match up\nfiles based on basenames. As a quick reminder (see the last two commit\nmessages for more details), this means for example that\ndocs/extensions.txt and docs/config/extensions.txt are considered likely\nrenames if there are no 'extensions.txt' files elsewhere among the added\nand deleted files, and if a similarity check confirms they are similar,\nthen they are marked as a rename without looking for a better similarity\nmatch among other files. This is a behavioral change, as covered in\nmore detail in the previous commit message.\n\nWe do not use this heuristic together with either break or copy\ndetection. The point of break detection is to say that filename\nsimilarity does not imply file content similarity, and we only want to\nknow about file content similarity. The point of copy detection is to\nuse more resources to check for additional similarities, while this is\nan optimization that uses far less resources but which might also result\nin finding slightly fewer similarities. So the idea behind this\noptimization goes against both of those features, and will be turned off\nfor both.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\n mega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\n just-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 46 +++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 41 insertions(+), 5 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex b1dda41de9b1..048a6186fd21 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,7 +367,6 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n-MAYBE_UNUSED\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n@@ -718,12 +717,49 @@ void diffcore_rename(struct diff_options *options)\n \tif (minimum_score == MAX_SCORE)\n \t\tgoto cleanup;\n \n+\tnum_sources = rename_src_nr;\n+\n+\tif (want_copies || break_idx) {\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones corresponding to exact renames\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\t} else {\n+\t\t/* Determine minimum score to match basenames */\n+\t\tint min_basename_score = (int)(5*minimum_score + 0*MAX_SCORE)/5;\n+\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones involved in renames (found via exact match)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull exact\", options->repo);\n+\n+\t\t/* Utilize file basenames to quickly find renames. */\n+\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n+\t\trename_count += find_basename_matches(options,\n+\t\t\t\t\t\t min_basename_score,\n+\t\t\t\t\t\t rename_src_nr);\n+\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n+\n+\t\t/*\n+\t\t * Cull sources, again:\n+\t\t * - remove ones involved in renames (found via basenames)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull basename\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull basename\", options->repo);\n+\t}\n+\n \t/*\n-\t * Calculate how many renames are left\n+\t * Calculate how many rename destinations are left\n \t */\n \tnum_destinations = (rename_dst_nr - rename_count);\n-\tremove_unneeded_paths_from_src(want_copies);\n-\tnum_sources = rename_src_nr;\n+\tnum_sources = rename_src_nr; /* rename_src_nr reflects lower number */\n \n \t/* All done? */\n \tif (!num_destinations || !num_sources)\n@@ -755,7 +791,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tstruct diff_score *m;\n \n \t\tif (rename_dst[i].is_rename)\n-\t\t\tcontinue; /* dealt with exact match already. */\n+\t\t\tcontinue; /* exact or basename match already handled */\n \n \t\tm = &mx[dst_cnt * NUM_CANDIDATE_PER_DST];\n \t\tfor (j = 0; j < NUM_CANDIDATE_PER_DST; j++)\n-- \ngitgitgadget\n\n"},{"id":"416496","messageId":"381a45d239bb52a70373c385d8978005c9cb4800.1612870326.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v2.git.1612870326.gitgitgadget@gmail.com","subject":"[PATCH v2 1/4] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-09T11:32:03Z","receivedAt":"2021-02-09T11:43:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to make use of unique basenames to help inform rename detection,\nso that more likely pairings can be checked first. (src/moduleA/foo.txt\nand source/module/A/foo.txt are likely related if there are no other\n'foo.txt' files among the deleted and added files.) Add a new function,\nnot yet used, which creates a map of the unique basenames within\nrename_src and another within rename_dst, together with the indices\nwithin rename_src/rename_dst where those basenames show up. Non-unique\nbasenames still show up in the map, but have an invalid index (-1).\n\nThis function was inspired by the fact that in real world repositories,\nmost renames often do not involve a basename change. Here are some\nsample repositories and the percentage of their historical renames (as of\nearly 2020) that did not involve a basename change:\n * linux: 76%\n * gcc: 64%\n * gecko: 79%\n * webkit: 89%\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 74930716e70d..1c52077b04e5 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,59 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+MAYBE_UNUSED\n+static int find_basename_matches(struct diff_options *options,\n+\t\t\t\t int minimum_score,\n+\t\t\t\t int num_src)\n+{\n+\tint i;\n+\tstruct strintmap sources;\n+\tstruct strintmap dests;\n+\n+\t/* Create maps of basename -> fullname(s) for sources and dests */\n+\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n+\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tchar *base;\n+\n+\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n+\t\tassert(!rename_src[i].p->one->rename_used);\n+\n+\t\tbase = strrchr(filename, '/');\n+\t\tbase = (base ? base+1 : filename);\n+\n+\t\t/* Record index within rename_src (i) if basename is unique */\n+\t\tif (strintmap_contains(&sources, base))\n+\t\t\tstrintmap_set(&sources, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&sources, base, i);\n+\t}\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\tchar *base;\n+\n+\t\tif (rename_dst[i].is_rename)\n+\t\t\tcontinue; /* involved in exact match already. */\n+\n+\t\tbase = strrchr(filename, '/');\n+\t\tbase = (base ? base+1 : filename);\n+\n+\t\t/* Record index within rename_dst (i) if basename is unique */\n+\t\tif (strintmap_contains(&dests, base))\n+\t\t\tstrintmap_set(&dests, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&dests, base, i);\n+\t}\n+\n+\t/* TODO: Make use of basenames source and destination basenames */\n+\n+\tstrintmap_clear(&sources);\n+\tstrintmap_clear(&dests);\n+\n+\treturn 0;\n+}\n+\n #define NUM_CANDIDATE_PER_DST 4\n static void record_if_better(struct diff_score m[], struct diff_score *o)\n {\n-- \ngitgitgadget\n\n"},{"id":"416497","messageId":"a0e75d8cd6bd32fb1ab2a209bc2079c30995b257.1612870326.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v2.git.1612870326.gitgitgadget@gmail.com","subject":"[PATCH v2 4/4] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-09T11:32:06Z","receivedAt":"2021-02-09T11:44:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe last few patches have introduced a new preliminary step when rename\ndetection is on but both break detection and copy detection are off.\nDocument this new step. While we're at it, add a testcase that checks\nthe new behavior as well.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 15 +++++++++++++++\n t/t4001-diff-rename.sh | 24 ++++++++++++++++++++++++\n 2 files changed, 39 insertions(+)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex c970d9fe438a..954ae3ef1082 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -168,6 +168,21 @@ a similarity score different from the default of 50% by giving a\n number after the \"-M\" or \"-C\" option (e.g. \"-M8\" to tell it to use\n 8/10 = 80%).\n \n+Note that when rename detection is on but both copy and break\n+detection are off, rename detection adds a preliminary step that first\n+checks files with the same basename. If files with the same basename\n+are sufficiently similar, it will mark them as renames and exclude\n+them from the later quadratic step (the one that pairwise compares all\n+unmatched files to find the \"best\" matches, determined by the highest\n+content similarity). So, for example, if docs/extensions.txt and\n+docs/config/extensions.txt have similar content, then they will be\n+marked as a rename even if it turns out that docs/extensions.txt was\n+more similar to src/extension-checks.c. At most, one comparison is\n+done per file in this preliminary pass; so if there are several\n+extensions.txt files throughout the directory hierarchy that were\n+added and deleted, this preliminary step will be skipped for those\n+files.\n+\n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\n diffcore mechanism as well as modified ones. This lets the copy\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex c16486a9d41a..bf62537c29a0 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -262,4 +262,28 @@ test_expect_success 'diff-tree -l0 defaults to a big rename limit, not zero' '\n \tgrep \"myotherfile.*myfile\" actual\n '\n \n+test_expect_success 'basename similarity vs best similarity' '\n+\tmkdir subdir &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 line10 >subdir/file.txt &&\n+\tgit add subdir/file.txt &&\n+\tgit commit -m \"base txt\" &&\n+\n+\tgit rm subdir/file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 >file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 >file.md &&\n+\tgit add file.txt file.md &&\n+\tgit commit -a -m \"rename\" &&\n+\tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n+\t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n+\t# but since same basenames are checked first...\n+\tcat >expected <<-\\EOF &&\n+\tA\tfile.md\n+\tR078\tsubdir/file.txt\tfile.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"416498","messageId":"pull.843.v2.git.1612870326.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.git.1612651937.gitgitgadget@gmail.com","subject":"[PATCH v2 0/4] Optimization batch 7: use file basenames to guide rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-09T11:32:02Z","receivedAt":"2021-02-09T11:46:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on ort-perf-batch-6[1].\n\nThis series uses file basenames in a basic fashion to guide rename\ndetection.\n\nChanges since v1:\n\n * Significant edits to the commit messages to make them explain the basic\n idea better and (hopefully) prevent misunderstandings.\n * Add a commit at the end that updates the documentation and includes a new\n testcase\n * Modify the code to make it clearer that it can already handle using a\n different score for basename comparison similarity, even if we don't use\n that ability yet (see below)\n\nChanges not yet included -- I need input about what is wanted:\n\n * Stolee suggested not creating a separate score for basename comparisons,\n at least not yet. Junio suggested it may be prudent to use a higher score\n for that than whatever -M option the user provided for normal\n comparisons...but didn't suggest whether it should be a separate\n user-specified option, or some kind of weighted average of the -M option\n and MAX_SCORE (e.g. use 60% if -M is 50%, or use 80% if -M is 75%). I\n tweaked the code to make it clearer that it already is able to handle\n such a score difference, but I'm not sure what whether we want an\n automatically computed higher value or a user-controlled possibly higher\n value.\n\n[1] https://lore.kernel.org/git/xmqqlfc4byt6.fsf@gitster.c.googlers.com/ [2]\nhttps://github.com/newren/presentations/blob/pdfs/merge-performance/merge-performance-slides.pdf\n\nElijah Newren (4):\n diffcore-rename: compute basenames of all source and dest candidates\n diffcore-rename: complete find_basename_matches()\n diffcore-rename: guide inexact rename detection based on basenames\n gitdiffcore doc: mention new preliminary step for rename detection\n\n Documentation/gitdiffcore.txt | 15 +++\n diffcore-rename.c | 185 +++++++++++++++++++++++++++++++++-\n t/t4001-diff-rename.sh | 24 +++++\n 3 files changed, 220 insertions(+), 4 deletions(-)\n\n\nbase-commit: 7ae9460d3dba84122c2674b46e4339b9d42bdedd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-843%2Fnewren%2Fort-perf-batch-7-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-843/newren/ort-perf-batch-7-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/843\n\nRange-diff vs v1:\n\n 1: 377a4a39fa86 ! 1: 381a45d239bb diffcore-rename: compute basenames of all source and dest candidates\n @@ Commit message\n diffcore-rename: compute basenames of all source and dest candidates\n \n We want to make use of unique basenames to help inform rename detection,\n - so that more likely pairings can be checked first. Add a new function,\n + so that more likely pairings can be checked first. (src/moduleA/foo.txt\n + and source/module/A/foo.txt are likely related if there are no other\n + 'foo.txt' files among the deleted and added files.) Add a new function,\n not yet used, which creates a map of the unique basenames within\n rename_src and another within rename_dst, together with the indices\n within rename_src/rename_dst where those basenames show up. Non-unique\n basenames still show up in the map, but have an invalid index (-1).\n \n + This function was inspired by the fact that in real world repositories,\n + most renames often do not involve a basename change. Here are some\n + sample repositories and the percentage of their historical renames (as of\n + early 2020) that did not involve a basename change:\n + * linux: 76%\n + * gcc: 64%\n + * gecko: 79%\n + * webkit: 89%\n +\n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 2: 5fb4493247ff ! 2: dcd0175229aa diffcore-rename: complete find_basename_matches()\n @@ Commit message\n sufficiently similar, we record the rename; if not, we include those\n files in the more exhaustive matrix comparison.\n \n + This means we are adding a set of preliminary additional comparisons,\n + but for each file we only compare it with at most one other file. For\n + example, if there was a include/media/device.h that was deleted and a\n + src/module/media/device.h that was added, and there were no other\n + device.h files added or deleted between the commits being compared,\n + then these two files would be compared in the preliminary step.\n +\n + This commit does not yet actually employ this new optimization, it\n + merely adds a function which can be used for this purpose. The next\n + commit will do the necessary plumbing to make use of it.\n +\n Note that this optimization might give us different results than without\n the optimization, because it's possible that despite files with the same\n basename being sufficiently similar to be considered a rename, there's\n an even better match between files without the same basename. I think\n - that is okay for four reasons: (1) That seems somewhat unlikely in\n - practice, (2) it's easy to explain to the users what happened if it does\n - ever occur (or even for them to intuitively figure out), and (3) as the\n - next patch will show it provides such a large performance boost that\n - it's worth the tradeoff. Reason (4) takes a full paragraph to\n + that is okay for four reasons: (1) it's easy to explain to the users\n + what happened if it does ever occur (or even for them to intuitively\n + figure out), (2) as the next patch will show it provides such a large\n + performance boost that it's worth the tradeoff, and (3) it's somewhat\n + unlikely that despite having unique matching basenames that other files\n + serve as better matches. Reason (4) takes a full paragraph to\n explain...\n \n If the previous three reasons aren't enough, consider what rename\n @@ Commit message\n basename and are sufficiently similar to be considered a rename, mark\n them as such without comparing the two to all other rename candidates.\n \n - We do not use this heuristic together with either break or copy\n - detection. The point of break detection is to say that filename\n - similarity does not imply file content similarity, and we only want to\n - know about file content similarity. The point of copy detection is to\n - use more resources to check for additional similarities, while this is\n - an optimization that uses far less resources but which might also result\n - in finding slightly fewer similarities. So the idea behind this\n - optimization goes against both of those features, and will be turned off\n - for both.\n -\n - Note that this optimization is not yet effective in any situation, as\n - the function is still unused. The next commit will hook it into the\n - code so that it is used when rename detection is wanted, but neither\n - copy nor break detection are.\n -\n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n 3: 1d941c35076e ! 3: ce2173aa1fb7 diffcore-rename: guide inexact rename detection based on basenames\n @@ Commit message\n \n Make use of the new find_basename_matches() function added in the last\n two patches, to find renames more rapidly in cases where we can match up\n - files based on basenames.\n + files based on basenames. As a quick reminder (see the last two commit\n + messages for more details), this means for example that\n + docs/extensions.txt and docs/config/extensions.txt are considered likely\n + renames if there are no 'extensions.txt' files elsewhere among the added\n + and deleted files, and if a similarity check confirms they are similar,\n + then they are marked as a rename without looking for a better similarity\n + match among other files. This is a behavioral change, as covered in\n + more detail in the previous commit message.\n +\n + We do not use this heuristic together with either break or copy\n + detection. The point of break detection is to say that filename\n + similarity does not imply file content similarity, and we only want to\n + know about file content similarity. The point of copy detection is to\n + use more resources to check for additional similarities, while this is\n + an optimization that uses far less resources but which might also result\n + in finding slightly fewer similarities. So the idea behind this\n + optimization goes against both of those features, and will be turned off\n + for both.\n \n For the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\n performance work; instrument with trace2_region_* calls\", 2020-10-28),\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n +\t\tremove_unneeded_paths_from_src(want_copies);\n +\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n +\t} else {\n ++\t\t/* Determine minimum score to match basenames */\n ++\t\tint min_basename_score = (int)(5*minimum_score + 0*MAX_SCORE)/5;\n ++\n +\t\t/*\n +\t\t * Cull sources:\n +\t\t * - remove ones involved in renames (found via exact match)\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n +\n +\t\t/* Utilize file basenames to quickly find renames. */\n +\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n -+\t\trename_count += find_basename_matches(options, minimum_score,\n ++\t\trename_count += find_basename_matches(options,\n ++\t\t\t\t\t\t min_basename_score,\n +\t\t\t\t\t\t rename_src_nr);\n +\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n +\n -: ------------ > 4: a0e75d8cd6bd gitdiffcore doc: mention new preliminary step for rename detection\n\n-- \ngitgitgadget\n"},{"id":"416502","messageId":"6aa32f94-9c8c-ad26-09ab-4c2028ef1d67@gmail.com","threadId":"55112","inReplyTo":"a0e75d8cd6bd32fb1ab2a209bc2079c30995b257.1612870326.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/4] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T12:59:09Z","receivedAt":"2021-02-09T12:59:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> The last few patches have introduced a new preliminary step when rename\n> detection is on but both break detection and copy detection are off.\n> Document this new step. While we're at it, add a testcase that checks\n> the new behavior as well.\n\nThanks for adding this documentation and test.\n\n> +Note that when rename detection is on but both copy and break\n> +detection are off, rename detection adds a preliminary step that first\n> +checks files with the same basename. If files with the same basename\n\nI find myself wanting a definition of 'basename' here, but perhaps I'm\njust being pedantic. A quick search clarifies this as a standard term [1]\nof which I was just ignorant.\n\n[1] https://man7.org/linux/man-pages/man3/basename.3.html\n\n> +are sufficiently similar, it will mark them as renames and exclude\n> +them from the later quadratic step (the one that pairwise compares all\n> +unmatched files to find the \"best\" matches, determined by the highest\n> +content similarity). So, for example, if docs/extensions.txt and\n> +docs/config/extensions.txt have similar content, then they will be\n> +marked as a rename even if it turns out that docs/extensions.txt was\n> +more similar to src/extension-checks.c. At most, one comparison is\n> +done per file in this preliminary pass; so if there are several\n> +extensions.txt files throughout the directory hierarchy that were\n> +added and deleted, this preliminary step will be skipped for those\n> +files.\n\n> +test_expect_success 'basename similarity vs best similarity' '\n> +\tmkdir subdir &&\n> +\ttest_write_lines line1 line2 line3 line4 line5 \\\n> +\t\t\t line6 line7 line8 line9 line10 >subdir/file.txt &&\n> +\tgit add subdir/file.txt &&\n> +\tgit commit -m \"base txt\" &&\n> +\n> +\tgit rm subdir/file.txt &&\n> +\ttest_write_lines line1 line2 line3 line4 line5 \\\n> +\t\t\t line6 line7 line8 >file.txt &&\n> +\ttest_write_lines line1 line2 line3 line4 line5 \\\n> +\t\t\t line6 line7 line8 line9 >file.md &&\n> +\tgit add file.txt file.md &&\n> +\tgit commit -a -m \"rename\" &&\n> +\tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n> +\t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n> +\t# but since same basenames are checked first...\n> +\tcat >expected <<-\\EOF &&\n> +\tA\tfile.md\n> +\tR078\tsubdir/file.txt\tfile.txt\n> +\tEOF\n> +\ttest_cmp expected actual\n> +'\n> +\n\nI appreciate the additional comments in this test to make it clear\nwhat you are testing. A minor nit is that the test could have been\nadded at the start of the series to document the _old_ behavior.\nThe 'expected' file would have this content:\n\n+\tcat >expected <<-\\EOF &&\n+\tA\tfile.txt\n+\tR078\tsubdir/file.txt\tfile.md\n+\tEOF\n\nThen, this test case would change the expected output in the same\npatch that introduces the behavior change.\n\nThanks,\n-Stolee\n\n"},{"id":"416503","messageId":"dfbffe97-51de-9e8b-37a4-417909358323@gmail.com","threadId":"55112","inReplyTo":"381a45d239bb52a70373c385d8978005c9cb4800.1612870326.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/4] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T13:17:53Z","receivedAt":"2021-02-09T13:18:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> We want to make use of unique basenames to help inform rename detection,\n> so that more likely pairings can be checked first. (src/moduleA/foo.txt\n> and source/module/A/foo.txt are likely related if there are no other\n> 'foo.txt' files among the deleted and added files.) Add a new function,\n> not yet used, which creates a map of the unique basenames within\n> rename_src and another within rename_dst, together with the indices\n> within rename_src/rename_dst where those basenames show up. Non-unique\n> basenames still show up in the map, but have an invalid index (-1).\n> \n> This function was inspired by the fact that in real world repositories,\n> most renames often do not involve a basename change. Here are some\n> sample repositories and the percentage of their historical renames (as of\n> early 2020) that did not involve a basename change:\n\nI found this difficult to parse. Perhaps instead\n\n \"the percentage of their renames that preserved basenames\".\n\nWe might also need something stronger, though: which percentage of renames\npreserved the basename but also had no other copy of that basename in the\nscope of all add/deletes?\n\nIs this reproducible from a shell command that could be documented here?\n\n> +MAYBE_UNUSED\n> +static int find_basename_matches(struct diff_options *options,\n> +\t\t\t\t int minimum_score,\n> +\t\t\t\t int num_src)\n> +{\n> +\tint i;\n> +\tstruct strintmap sources;\n> +\tstruct strintmap dests;\n> +\n> +\t/* Create maps of basename -> fullname(s) for sources and dests */\n> +\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n> +\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n\nInitially, I was wondering why we need the map for each side, but we will need\nto enforce uniqueness in each set, so OK.\n\n> +\tfor (i = 0; i < num_src; ++i) {\n> +\t\tchar *filename = rename_src[i].p->one->path;\n> +\t\tchar *base;\n> +\n> +\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n> +\t\tassert(!rename_src[i].p->one->rename_used);\n> +\n> +\t\tbase = strrchr(filename, '/');\n> +\t\tbase = (base ? base+1 : filename);\n\nnit: \"base + 1\"\n\nAlso, this is used here and below. Perhaps it's worth pulling out as a\nhelper? I see similar code being duplicated in these existing spots:\n\n* diff-no-index.c:append_basename()\n* help.c:append_similar_ref()\n* packfile.c:pack_basename()\n* replace-object.c:register_replace_ref()\n* setup.c:read_gitfile_gently()\n* builtin/rebase.c:cmd_rebase()\n* builtin/stash.c:do_create_stash()\n* builtin/worktree.c:add_worktree()\n* contrib/credential/gnome-keyring/git-credential-gnome-keyring.c:usage()\n* contrib/credential/libsecret/git-credential-libsecret.c:usage()\n* trace2/tr2_dst.c:tr2_dst_try_auto_path()\n\nThere are other places that use strchr(_, '/') but they seem to be related\nto peeling basenames off of paths and using the leading portion of the path.\n\n> +\t\t/* Record index within rename_src (i) if basename is unique */\n> +\t\tif (strintmap_contains(&sources, base))\n> +\t\t\tstrintmap_set(&sources, base, -1);\n> +\t\telse\n> +\t\t\tstrintmap_set(&sources, base, i);\n> +\t}\n> +\tfor (i = 0; i < rename_dst_nr; ++i) {\n> +\t\tchar *filename = rename_dst[i].p->two->path;\n> +\t\tchar *base;\n> +\n> +\t\tif (rename_dst[i].is_rename)\n> +\t\t\tcontinue; /* involved in exact match already. */\n> +\n> +\t\tbase = strrchr(filename, '/');\n> +\t\tbase = (base ? base+1 : filename);\n> +\n> +\t\t/* Record index within rename_dst (i) if basename is unique */\n> +\t\tif (strintmap_contains(&dests, base))\n> +\t\t\tstrintmap_set(&dests, base, -1);\n> +\t\telse\n> +\t\t\tstrintmap_set(&dests, base, i);\n> +\t}\n> +\n> +\t/* TODO: Make use of basenames source and destination basenames */\n> +\n> +\tstrintmap_clear(&sources);\n> +\tstrintmap_clear(&dests);\n> +\n> +\treturn 0;\n> +}\n\nThanks,\n-Stolee\n"},{"id":"416504","messageId":"b64e30ab-9cb3-ffee-ec5e-1b94529ce636@gmail.com","threadId":"55112","inReplyTo":"dcd0175229aa6fba576425e78875b95385acb58d.1612870326.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/4] diffcore-rename: complete find_basename_matches()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T13:25:48Z","receivedAt":"2021-02-09T13:26:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n> +\t/*\n> +\t * When I checked, over 76% of file renames in linux just moved\n\nPerhaps \"In late 2020,\" instead of \"When I checked\".\n\n> +\t * files to a different directory but kept the same basename. gcc\n> +\t * did that with over 64% of renames, gecko did it with over 79%,\n> +\t * and WebKit did it with over 89%.\n> +\t *\n> +\t * Therefore we can bypass the normal exhaustive NxM matrix\n> +\t * comparison of similarities between all potential rename sources\n> +\t * and destinations by instead using file basename as a hint, checking\n> +\t * for similarity between files with the same basename, and if we\n> +\t * find a pair that are sufficiently similar, record the rename\n> +\t * pair and exclude those two from the NxM matrix.\n> +\t *\n> +\t * This *might* cause us to find a less than optimal pairing (if\n> +\t * there is another file that we are even more similar to but has a\n> +\t * different basename). Given the huge performance advantage\n> +\t * basename matching provides, and given the frequency with which\n> +\t * people use the same basename in real world projects, that's a\n> +\t * trade-off we are willing to accept when doing just rename\n> +\t * detection. However, if someone wants copy detection that\n> +\t * implies they are willing to spend more cycles to find\n> +\t * similarities between files, so it may be less likely that this\n> +\t * heuristic is wanted.\n> +\t */\n> +\n> +\tint i, renames = 0;\n> \tstruct strintmap sources;\n> \tstruct strintmap dests; \n\n...\n\n> +\t * copy detection. find_basename_matches() is only used when detecting\n> +\t * renames, not when detecting copies, so it'll only be used when a file\n> +\t * only existed in the source. Since we already know that the file\n\nThere are two \"only\"s in this sentence. Just awkward, not wrong.\n\n> +\t * won't be unmodified, there's no point checking for it; that's just a\n> +\t * waste of resources. So set skip_unmodified to 0 so that\n> +\t * estimate_similarity() and prefetch() won't waste resources checking\n> +\t * for something we already know is false.\n> +\t */\n> +\tint skip_unmodified = 0;\n> +\n\n\n\n> -\t/* TODO: Make use of basenames source and destination basenames */\n> +\t/* Now look for basename matchups and do similarity estimation */\n> +\tfor (i = 0; i < num_src; ++i) {\n> +\t\tchar *filename = rename_src[i].p->one->path;\n> +\t\tchar *base = NULL;\n> +\t\tintptr_t src_index;\n> +\t\tintptr_t dst_index;\n> +\n> +\t\t/* Get the basename */\n> +\t\tbase = strrchr(filename, '/');\n> +\t\tbase = (base ? base+1 : filename);\n\nHere is the third instance of this in the same function. At minimum we should\nextract a helper for you to consume.\n\n> +\t\t/* Find out if this basename is unique among sources */\n> +\t\tsrc_index = strintmap_get(&sources, base);\n> +\t\tif (src_index == -1)\n> +\t\t\tcontinue; /* not a unique basename; skip it */\n> +\t\tassert(src_index == i);\n> +\n> +\t\tif (strintmap_contains(&dests, base)) {\n> +\t\t\tstruct diff_filespec *one, *two;\n> +\t\t\tint score;\n> +\n> +\t\t\t/* Find out if this basename is unique among dests */\n> +\t\t\tdst_index = strintmap_get(&dests, base);\n> +\t\t\tif (dst_index == -1)\n> +\t\t\t\tcontinue; /* not a unique basename; skip it */\n> +\n> +\t\t\t/* Ignore this dest if already used in a rename */\n> +\t\t\tif (rename_dst[dst_index].is_rename)\n> +\t\t\t\tcontinue; /* already used previously */\n> +\n> +\t\t\t/* Estimate the similarity */\n> +\t\t\tone = rename_src[src_index].p->one;\n> +\t\t\ttwo = rename_dst[dst_index].p->two;\n> +\t\t\tscore = estimate_similarity(options->repo, one, two,\n> +\t\t\t\t\t\t minimum_score, skip_unmodified);\n> +\n> +\t\t\t/* If sufficiently similar, record as rename pair */\n> +\t\t\tif (score < minimum_score)\n> +\t\t\t\tcontinue;\n> +\t\t\trecord_rename_pair(dst_index, src_index, score);\n> +\t\t\trenames++;\n> +\n> +\t\t\t/*\n> +\t\t\t * Found a rename so don't need text anymore; if we\n> +\t\t\t * didn't find a rename, the filespec_blob would get\n> +\t\t\t * re-used when doing the matrix of comparisons.\n> +\t\t\t */\n> +\t\t\tdiff_free_filespec_blob(one);\n> +\t\t\tdiff_free_filespec_blob(two);\n> +\t\t}\n> +\t}\n\nMakes sense to me.\n\nThanks,\n-Stolee\n"},{"id":"416505","messageId":"48a208c2-75fe-773e-aa2a-12a90dd06ddd@gmail.com","threadId":"55112","inReplyTo":"ce2173aa1fb7ac3830bf8856a14558446e16a3dd.1612870326.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/4] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T13:33:47Z","receivedAt":"2021-02-09T13:34:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n> +\tnum_sources = rename_src_nr;\n> +\n> +\tif (want_copies || break_idx) {\n> +\t\t/*\n> +\t\t * Cull sources:\n> +\t\t * - remove ones corresponding to exact renames\n> +\t\t */\n> +\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n> +\t\tremove_unneeded_paths_from_src(want_copies);\n> +\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n\nIsn't this the same as\n\n> +\t} else {\n> +\t\t/* Determine minimum score to match basenames */\n> +\t\tint min_basename_score = (int)(5*minimum_score + 0*MAX_SCORE)/5;\n> +\n> +\t\t/*\n> +\t\t * Cull sources:\n> +\t\t * - remove ones involved in renames (found via exact match)\n> +\t\t */\n> +\t\ttrace2_region_enter(\"diff\", \"cull exact\", options->repo);\n> +\t\tremove_unneeded_paths_from_src(want_copies);\n> +\t\ttrace2_region_leave(\"diff\", \"cull exact\", options->repo);\n\n...this? (except the regions are renamed)\n\nCould this be simplified as:\n\n+\tnum_sources = rename_src_nr;\n+\n+\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\tremove_unneeded_paths_from_src(want_copies);\n+\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\n+\tif (!want_copies && !break_idx) {\n+\t\t/* Determine minimum score to match basenames */\n\nI realize you probably changed the region names on purpose to distinguish\nthat there are two \"cull\" regions in the case of no copies, but I think\nthat isn't really worth different names. Better to have a consistent region\nname around the same activity in both cases.\n\n> +\t\tint min_basename_score = (int)(5*minimum_score + 0*MAX_SCORE)/5;\n\nDid you intend for this to be 5*min + 0*MAX? This seems wrong if you want\nthis value to be different from minimum_score.\n\n> +\n> +\t\t/* Utilize file basenames to quickly find renames. */\n> +\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n> +\t\trename_count += find_basename_matches(options,\n> +\t\t\t\t\t\t min_basename_score,\n> +\t\t\t\t\t\t rename_src_nr);\n> +\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n> +\n> +\t\t/*\n> +\t\t * Cull sources, again:\n> +\t\t * - remove ones involved in renames (found via basenames)\n> +\t\t */\n> +\t\ttrace2_region_enter(\"diff\", \"cull basename\", options->repo);\n> +\t\tremove_unneeded_paths_from_src(want_copies);\n> +\t\ttrace2_region_leave(\"diff\", \"cull basename\", options->repo);\n> +\t}\n> +\n\nThanks,\n-Stolee\n"},{"id":"416519","messageId":"CABPp-BEsuOiUyvbkwPC384eho8pgSWuRdcvw9t5gkXhf+_j-3A@mail.gmail.com","threadId":"55112","inReplyTo":"dfbffe97-51de-9e8b-37a4-417909358323@gmail.com","subject":"Re: [PATCH v2 1/4] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-09T16:56:42Z","receivedAt":"2021-02-09T16:57:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Tue, Feb 9, 2021 at 5:17 AM Derrick Stolee wrote:\n>\n> On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > We want to make use of unique basenames to help inform rename detection,\n> > so that more likely pairings can be checked first. (src/moduleA/foo.txt\n> > and source/module/A/foo.txt are likely related if there are no other\n> > 'foo.txt' files among the deleted and added files.) Add a new function,\n> > not yet used, which creates a map of the unique basenames within\n> > rename_src and another within rename_dst, together with the indices\n> > within rename_src/rename_dst where those basenames show up. Non-unique\n> > basenames still show up in the map, but have an invalid index (-1).\n> >\n> > This function was inspired by the fact that in real world repositories,\n> > most renames often do not involve a basename change. Here are some\n> > sample repositories and the percentage of their historical renames (as of\n> > early 2020) that did not involve a basename change:\n>\n> I found this difficult to parse. Perhaps instead\n>\n> \"the percentage of their renames that preserved basenames\".\n\nOoh, I like it; happy to make that change.\n\n> We might also need something stronger, though: which percentage of renames\n> preserved the basename but also had no other copy of that basename in the\n> scope of all add/deletes?\n\nI don't think it's useful to try to prove that this idea can save time\nor how much time we can save before we try it; I think the only\npurpose of these numbers should be to motivate the idea behind why it\nwas worth trying. If we attempt to prove how much we'll save apriori,\nthen what you have is also too weak. We would need \"percentage of\ntotal adds/deletes that are renames that preserved the basename but\nalso had no other copy of that basename in the scope of all\nadd/deletes\". But that is also wrong, actually; we need \"for any\ngiven two commits that we are likely to diff, what is the average\npercentage of total adds/deletes between them that are renames that\npreserved the basename but also had no other copy of that basename in\nthe scope of all add/deletes\". In particular, my script did not look\nat the \"any two given likely-to-be-diffed commits\" viewpoint, I simply\nadded the number of renames within individual commits that preserved\nrenames, and divided by the total number of renames in individual\ncommits. But even if we could calculate the \"any two given\nlikely-to-be-diffed commits\" viewpoint in some sane manner, it'd still\nbe misleading. The next series is going to change the \"no other copy\nof that basename in the scope of all adds/deletes\" caveat, by adding a\nway to match up _some_ of those files (when it can find a way to\ncompare any given file to exactly one of the other files with the same\nbasename). And even if you consider all the above and calculated it\nin order to try to show how much could be saved, you might need to\nstart worrying about details like the fact that the first comparison\nbetween files in diffcore-rename.c is _much_ more expensive than\nsubsequent comparisons (due to the fact that the spanhash is cached).\n\nTrying to account for all these details and describe them fully is\ncompletely beside the point, though; I didn't bother to check any of\nthis before implementing the algorithm -- I just looked up these very\nrough numbers and felt they provided sufficient motivation that there\nwas an optimization worth trying.\n\n> Is this reproducible from a shell command that could be documented here?\n\nNo, trying to parse log output with full handling of proper quoting in\nthe case of filenames with funny characters is too complex to attempt\nin shell. I was surprised by how long it turned out to be in python.\n(And I dread attempting to calculate \"something stronger\" in any\naccurate way given how involved just this rough calculation was. That\nidea seems harder to me than actually implementing this series.)\n\nIf you're curious, though, and don't care about\nquickly-hacked-together-script-not-designed-for-reuse:\nhttps://github.com/newren/git/blob/ort/rebase-testcase/count-renames.py\n\n> > +MAYBE_UNUSED\n> > +static int find_basename_matches(struct diff_options *options,\n> > + int minimum_score,\n> > + int num_src)\n> > +{\n> > + int i;\n> > + struct strintmap sources;\n> > + struct strintmap dests;\n> > +\n> > + /* Create maps of basename -> fullname(s) for sources and dests */\n> > + strintmap_init_with_options(&sources, -1, NULL, 0);\n> > + strintmap_init_with_options(&dests, -1, NULL, 0);\n>\n> Initially, I was wondering why we need the map for each side, but we will need\n> to enforce uniqueness in each set, so OK.\n>\n>> > + for (i = 0; i < num_src; ++i) {\n> > + char *filename = rename_src[i].p->one->path;\n> > + char *base;\n> > +\n> > + /* exact renames removed in remove_unneeded_paths_from_src() */\n> > + assert(!rename_src[i].p->one->rename_used);\n> > +\n> > + base = strrchr(filename, '/');\n> > + base = (base ? base+1 : filename);\n>\n> nit: \"base + 1\"\n\nWill fix.\n\n> Also, this is used here and below. Perhaps it's worth pulling out as a\n> helper? I see similar code being duplicated in these existing spots:\n>\n> * diff-no-index.c:append_basename()\n> * help.c:append_similar_ref()\n> * packfile.c:pack_basename()\n> * replace-object.c:register_replace_ref()\n> * setup.c:read_gitfile_gently()\n> * builtin/rebase.c:cmd_rebase()\n> * builtin/stash.c:do_create_stash()\n> * builtin/worktree.c:add_worktree()\n> * contrib/credential/gnome-keyring/git-credential-gnome-keyring.c:usage()\n> * contrib/credential/libsecret/git-credential-libsecret.c:usage()\n> * trace2/tr2_dst.c:tr2_dst_try_auto_path()\n\nHonestly asking: would anyone ever search for such a two-line helper\nfunction? I wouldn't have even thought to look, since it seems so\nsimple.\n\nHowever, my real concern here is that this type of change would risk\nintroducing conflicts with unrelated series. This series is the\nsecond in what will be a 9-series deep dependency chain of\noptimizations[1], and the later series are going to be longer than\nthese first two were (the latter ones are 6-11 patches each). We've\nalready discussed previously whether we possibly want to hold the\nfirst couple optimization series out of the upcoming git-2.31 release\nin order to keep the optimizations all together, but that might\nincrease the risk of conflicts with unrelated patches if we try a\nbigger tree refactor like this. (Junio never commented on that,\nthough.) It might be better to keep the series touching only\nmerge-ort.c & diffcore-rename.c, and then do cleanups like the one you\nsuggest here after the whole series.\n\nThat said, it's not a difficult initial change, so I'm mostly\nexpressing this concern out of making things harder for Junio. It'd\nbe best to get his opinion -- Junio, your thoughts?\n\n[1] https://github.com/gitgitgadget/git/pulls?q=is%3Apr+author%3Anewren+Optimization+batch\n\n> There are other places that use strchr(_, '/') but they seem to be related\n> to peeling basenames off of paths and using the leading portion of the path.\n>\n> > + /* Record index within rename_src (i) if basename is unique */\n> > + if (strintmap_contains(&sources, base))\n> > + strintmap_set(&sources, base, -1);\n> > + else\n> > + strintmap_set(&sources, base, i);\n> > + }\n> > + for (i = 0; i < rename_dst_nr; ++i) {\n> > + char *filename = rename_dst[i].p->two->path;\n> > + char *base;\n> > +\n> > + if (rename_dst[i].is_rename)\n> > + continue; /* involved in exact match already. */\n> > +\n> > + base = strrchr(filename, '/');\n> > + base = (base ? base+1 : filename);\n> > +\n> > + /* Record index within rename_dst (i) if basename is unique */\n> > + if (strintmap_contains(&dests, base))\n> > + strintmap_set(&dests, base, -1);\n> > + else\n> > + strintmap_set(&dests, base, i);\n> > + }\n> > +\n> > + /* TODO: Make use of basenames source and destination basenames */\n> > +\n> > + strintmap_clear(&sources);\n> > + strintmap_clear(&dests);\n> > +\n> > + return 0;\n> > +}\n>\n> Thanks,\n> -Stolee\n\nThanks for the review!\n"},{"id":"416522","messageId":"9d8ee4be-33d7-6c50-f14b-0b92a4ebe949@gmail.com","threadId":"55112","inReplyTo":"CABPp-BEsuOiUyvbkwPC384eho8pgSWuRdcvw9t5gkXhf+_j-3A@mail.gmail.com","subject":"Re: [PATCH v2 1/4] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T17:02:50Z","receivedAt":"2021-02-09T17:03:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2021 11:56 AM, Elijah Newren wrote:\n>> Also, this is used here and below. Perhaps it's worth pulling out as a\n>> helper? I see similar code being duplicated in these existing spots:\n>>\n>> * diff-no-index.c:append_basename()\n>> * help.c:append_similar_ref()\n>> * packfile.c:pack_basename()\n>> * replace-object.c:register_replace_ref()\n>> * setup.c:read_gitfile_gently()\n>> * builtin/rebase.c:cmd_rebase()\n>> * builtin/stash.c:do_create_stash()\n>> * builtin/worktree.c:add_worktree()\n>> * contrib/credential/gnome-keyring/git-credential-gnome-keyring.c:usage()\n>> * contrib/credential/libsecret/git-credential-libsecret.c:usage()\n>> * trace2/tr2_dst.c:tr2_dst_try_auto_path()\n> Honestly asking: would anyone ever search for such a two-line helper\n> function? I wouldn't have even thought to look, since it seems so\n> simple.\n> \n> However, my real concern here is that this type of change would risk\n> introducing conflicts with unrelated series. This series is the\n> second in what will be a 9-series deep dependency chain of\n> optimizations[1], and the later series are going to be longer than\n> these first two were (the latter ones are 6-11 patches each). We've\n> already discussed previously whether we possibly want to hold the\n> first couple optimization series out of the upcoming git-2.31 release\n> in order to keep the optimizations all together, but that might\n> increase the risk of conflicts with unrelated patches if we try a\n> bigger tree refactor like this. (Junio never commented on that,\n> though.) It might be better to keep the series touching only\n> merge-ort.c & diffcore-rename.c, and then do cleanups like the one you\n> suggest here after the whole series.\n> \n> That said, it's not a difficult initial change, so I'm mostly\n> expressing this concern out of making things harder for Junio. It'd\n> be best to get his opinion -- Junio, your thoughts?\n> \n> [1] https://github.com/gitgitgadget/git/pulls?q=is%3Apr+author%3Anewren+Optimization+batch\n \nI don't consider the step of \"go put the helper in all these other\nplaces\" necessary for the current series. However, the \"get basename\"\ncode appears a total of three times in this series, so it would be\ngood to at least extract it to a static inline method to reduce\nthe duplication isolated to this change.\n\nThanks,\n-Stolee\n\n"},{"id":"416523","messageId":"xmqqo8gt2mxh.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"6aa32f94-9c8c-ad26-09ab-4c2028ef1d67@gmail.com","subject":"Re: [PATCH v2 4/4] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-09T17:03:38Z","receivedAt":"2021-02-09T17:05:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> +Note that when rename detection is on but both copy and break\n>> +detection are off, rename detection adds a preliminary step that first\n>> +checks files with the same basename. If files with the same basename\n>\n> I find myself wanting a definition of 'basename' here, but perhaps I'm\n> just being pedantic. A quick search clarifies this as a standard term [1]\n> of which I was just ignorant.\n>\n> [1] https://man7.org/linux/man-pages/man3/basename.3.html\n>\n>> +are sufficiently similar, it will mark them as renames and exclude\n>> +them from the later quadratic step (the one that pairwise compares all\n>> +unmatched files to find the \"best\" matches, determined by the highest\n>> +content similarity).\n\nWhile I do not think `basename` is unacceptably bad, we should aim\nto do better. For \"direc/tory/hello.txt\", both \"hello.txt\" or\n\"hello\" are what would come up to people's mind with the technical\nterm \"basename\" (i.e. basename as opposed to dirname, vs basename as\nopposed to filename with .extension).\n\nAvoiding this ambiguity and using a word understandable by those not\nversed well with UNIX/POSIX lingo may be done at the same time,\nhopefully.\n\nFor example, can we frame the description around this key sentence:\n\n The heuristics is based on an observation that a file is often\n moved across directories while keeping its filename the same.\n\nThe term \"filename\" alone can be ambiguous (i.e. both \"hello.txt\"\nand \"direc/tory/hello.txt\" are valid interpretations in the earlier\nexample), but in the context of a sentence that talks about \"moved\nacross directories\", the former would become the only valid one. We\ncan even say just \"name\" and there is no ambiguity in the above \"key\nsentence\".\n\nThen keeping that in mind, we can rewrite the above you quoted like\nso without going technical and without risking ambiguity, like this:\n\n ... a preliminary step that checks if files are moved across\n directories while keeping their filenames the same. If there is\n a file added to a directory whose contents is sufficiently\n similar to a file with the same name that got deleted from a\n different directory, ...\n\n"},{"id":"416525","messageId":"CABPp-BEz7ZUUUPF6EkK0RzWq5O=kYbNFFtg=iMbk607kQ-nmbg@mail.gmail.com","threadId":"55112","inReplyTo":"b64e30ab-9cb3-ffee-ec5e-1b94529ce636@gmail.com","subject":"Re: [PATCH v2 2/4] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-09T17:17:11Z","receivedAt":"2021-02-09T17:18:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Feb 9, 2021 at 5:25 AM Derrick Stolee wrote:\n>\n> On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n> > + /*\n> > + * When I checked, over 76% of file renames in linux just moved\n>\n> Perhaps \"In late 2020,\" instead of \"When I checked\".\n\nIn early 2020 (in fact, it might have been 2019, but I have no records\nto verify the actual year), but sure I can change that.\n\n> > + * files to a different directory but kept the same basename. gcc\n> > + * did that with over 64% of renames, gecko did it with over 79%,\n> > + * and WebKit did it with over 89%.\n> > + *\n> > + * Therefore we can bypass the normal exhaustive NxM matrix\n> > + * comparison of similarities between all potential rename sources\n> > + * and destinations by instead using file basename as a hint, checking\n> > + * for similarity between files with the same basename, and if we\n> > + * find a pair that are sufficiently similar, record the rename\n> > + * pair and exclude those two from the NxM matrix.\n> > + *\n> > + * This *might* cause us to find a less than optimal pairing (if\n> > + * there is another file that we are even more similar to but has a\n> > + * different basename). Given the huge performance advantage\n> > + * basename matching provides, and given the frequency with which\n> > + * people use the same basename in real world projects, that's a\n> > + * trade-off we are willing to accept when doing just rename\n> > + * detection. However, if someone wants copy detection that\n> > + * implies they are willing to spend more cycles to find\n> > + * similarities between files, so it may be less likely that this\n> > + * heuristic is wanted.\n> > + */\n> > +\n> > + int i, renames = 0;\n> > struct strintmap sources;\n> > struct strintmap dests;\n>\n> ...\n>\n> > + * copy detection. find_basename_matches() is only used when detecting\n> > + * renames, not when detecting copies, so it'll only be used when a file\n> > + * only existed in the source. Since we already know that the file\n>\n> There are two \"only\"s in this sentence. Just awkward, not wrong.\n>\n> > + * won't be unmodified, there's no point checking for it; that's just a\n> > + * waste of resources. So set skip_unmodified to 0 so that\n> > + * estimate_similarity() and prefetch() won't waste resources checking\n> > + * for something we already know is false.\n> > + */\n> > + int skip_unmodified = 0;\n> > +\n>\n>\n>\n> > - /* TODO: Make use of basenames source and destination basenames */\n> > + /* Now look for basename matchups and do similarity estimation */\n> > + for (i = 0; i < num_src; ++i) {\n> > + char *filename = rename_src[i].p->one->path;\n> > + char *base = NULL;\n> > + intptr_t src_index;\n> > + intptr_t dst_index;\n> > +\n> > + /* Get the basename */\n> > + base = strrchr(filename, '/');\n> > + base = (base ? base+1 : filename);\n>\n> Here is the third instance of this in the same function. At minimum we should\n> extract a helper for you to consume.\n\nWhere by \"this\" you mean these last two lines, right?\n\nAnd perhaps explain why I'm not using either basename(3) or\ngitbasename() from git-compat-util.h? (The latter of which I just\nlearned about while responding to the review of this patch.)\n\nor maybe gitbasename can do the job, but the skip_dos_drive_prefix()\nand the munging of the string passed in both worry me. And the\nis_dir_sep() looks inefficient since I know I'm dealing with filenames\nas stored in git internally, and thus can only use '/' characters.\nHmm...\n\nYeah, I think I'll add my own helper in this file, since you want one,\nand just use it.\n\n> > + /* Find out if this basename is unique among sources */\n> > + src_index = strintmap_get(&sources, base);\n> > + if (src_index == -1)\n> > + continue; /* not a unique basename; skip it */\n> > + assert(src_index == i);\n> > +\n> > + if (strintmap_contains(&dests, base)) {\n> > + struct diff_filespec *one, *two;\n> > + int score;\n> > +\n> > + /* Find out if this basename is unique among dests */\n> > + dst_index = strintmap_get(&dests, base);\n> > + if (dst_index == -1)\n> > + continue; /* not a unique basename; skip it */\n> > +\n> > + /* Ignore this dest if already used in a rename */\n> > + if (rename_dst[dst_index].is_rename)\n> > + continue; /* already used previously */\n> > +\n> > + /* Estimate the similarity */\n> > + one = rename_src[src_index].p->one;\n> > + two = rename_dst[dst_index].p->two;\n> > + score = estimate_similarity(options->repo, one, two,\n> > + minimum_score, skip_unmodified);\n> > +\n> > + /* If sufficiently similar, record as rename pair */\n> > + if (score < minimum_score)\n> > + continue;\n> > + record_rename_pair(dst_index, src_index, score);\n> > + renames++;\n> > +\n> > + /*\n> > + * Found a rename so don't need text anymore; if we\n> > + * didn't find a rename, the filespec_blob would get\n> > + * re-used when doing the matrix of comparisons.\n> > + */\n> > + diff_free_filespec_blob(one);\n> > + diff_free_filespec_blob(two);\n> > + }\n> > + }\n>\n> Makes sense to me.\n>\n> Thanks,\n> -Stolee\n"},{"id":"416539","messageId":"d0154183-371f-5ada-f4dd-dab7b3267c8c@gmail.com","threadId":"55112","inReplyTo":"CABPp-BEz7ZUUUPF6EkK0RzWq5O=kYbNFFtg=iMbk607kQ-nmbg@mail.gmail.com","subject":"Re: [PATCH v2 2/4] diffcore-rename: complete find_basename_matches()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T17:34:36Z","receivedAt":"2021-02-09T17:36:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2021 12:17 PM, Elijah Newren wrote:\n> On Tue, Feb 9, 2021 at 5:25 AM Derrick Stolee wrote:\n>>\n>> On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n>>> + /* Get the basename */\n>>> + base = strrchr(filename, '/');\n>>> + base = (base ? base+1 : filename);\n>>\n>> Here is the third instance of this in the same function. At minimum we should\n>> extract a helper for you to consume.\n> \n> Where by \"this\" you mean these last two lines, right?\n\nCorrect. The reason to use a helper is to ease cognitive load when\nreading the code. These lines are identical and serve the same\npurpose. By making a \"get_basename()\" helper and using it as\n\n\tbase = get_basename(filename);\n\nmakes it easy to understand what is happening without needing\nto think carefully about it. For example, I had to remember\nthat strrchr() returns NULL when '/' is not found, not the first\ncharacter of the string.\n\n> And perhaps explain why I'm not using either basename(3) or\n> gitbasename() from git-compat-util.h? (The latter of which I just\n> learned about while responding to the review of this patch.)\n> \n> or maybe gitbasename can do the job, but the skip_dos_drive_prefix()\n> and the munging of the string passed in both worry me. And the\n> is_dir_sep() looks inefficient since I know I'm dealing with filenames\n> as stored in git internally, and thus can only use '/' characters.\n> Hmm...\n> \n> Yeah, I think I'll add my own helper in this file, since you want one,\n> and just use it.\n\nRight. I almost made a point to say \"Don't use find_last_dir_sep()\"\nbecause it uses platform-specific directory separators. Your helper\nis based on in-memory representation that always uses Unix-style paths.\n\nThanks,\n-Stolee\n\n"},{"id":"416541","messageId":"CABPp-BFfF5rxbHwP1xnr63fPmp2=NkTZ-r0=FVZ1oVhAZD_s9w@mail.gmail.com","threadId":"55112","inReplyTo":"48a208c2-75fe-773e-aa2a-12a90dd06ddd@gmail.com","subject":"Re: [PATCH v2 3/4] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-09T17:41:16Z","receivedAt":"2021-02-09T17:46:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Feb 9, 2021 at 5:33 AM Derrick Stolee wrote:\n>\n> On 2/9/2021 6:32 AM, Elijah Newren via GitGitGadget wrote:\n> > + num_sources = rename_src_nr;\n> > +\n> > + if (want_copies || break_idx) {\n> > + /*\n> > + * Cull sources:\n> > + * - remove ones corresponding to exact renames\n> > + */\n> > + trace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n> > + remove_unneeded_paths_from_src(want_copies);\n> > + trace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n>\n> Isn't this the same as\n>\n> > + } else {\n> > + /* Determine minimum score to match basenames */\n> > + int min_basename_score = (int)(5*minimum_score + 0*MAX_SCORE)/5;\n> > +\n> > + /*\n> > + * Cull sources:\n> > + * - remove ones involved in renames (found via exact match)\n> > + */\n> > + trace2_region_enter(\"diff\", \"cull exact\", options->repo);\n> > + remove_unneeded_paths_from_src(want_copies);\n> > + trace2_region_leave(\"diff\", \"cull exact\", options->repo);\n>\n> ...this? (except the regions are renamed)\n>\n> Could this be simplified as:\n>\n> + num_sources = rename_src_nr;\n> +\n> + trace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n> + remove_unneeded_paths_from_src(want_copies);\n> + trace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n> +\n> + if (!want_copies && !break_idx) {\n> + /* Determine minimum score to match basenames */\n>\n> I realize you probably changed the region names on purpose to distinguish\n> that there are two \"cull\" regions in the case of no copies, but I think\n> that isn't really worth different names. Better to have a consistent region\n> name around the same activity in both cases.\n\nActually, the reason they were split is because a later series has to\ncall remove_unneeded_paths_from_src() differently for the two\nbranches. The patch history was so dirty that the easiest way to\nclean things up was just to create completely new patches pulling off\nrelevant chunks of code and touching them up; while doing that, I\ndidn't notice that the changes I made to split out this early series\nresulted in this near-duplication.\n\nSo, I can join them...but they would just need to be split back out in\nmy \"Optimization batch 9\" series.\n\nI'm happy to fix the region name to make them the same. Is that good\nenough, or would you rather these common code regions combined for\nthis patch and then split out later?\n\n>\n> > + int min_basename_score = (int)(5*minimum_score + 0*MAX_SCORE)/5;\n>\n> Did you intend for this to be 5*min + 0*MAX? This seems wrong if you want\n> this value to be different from minimum_score.\n\nIn my cover letter I noted that I didn't know what to set this to and\nwanted input; yesterday you said it wasn't worth worrying about using\na different value, but Junio suggested we should use one (but didn't\nstate how much higher it should be or whether it should be user input\ndriven). This weird construct was here just to show that it is easy\nto feed a different score into the basename comparison than what is\nused elsewhere; I can fix it up once I get word on what Junio wants to\nsee.\n\nSince I didn't know what to use, though, and I didn't want to get a\ndifferent set of numbers for the final commit message on the speedup\nachieved if I'm just going to throw them away and recompute once I\nfind out what Junio wants here, I did intentionally set the\ncomputation to just give us minimum_score, for now.\n\n> > +\n> > + /* Utilize file basenames to quickly find renames. */\n> > + trace2_region_enter(\"diff\", \"basename matches\", options->repo);\n> > + rename_count += find_basename_matches(options,\n> > + min_basename_score,\n> > + rename_src_nr);\n> > + trace2_region_leave(\"diff\", \"basename matches\", options->repo);\n> > +\n> > + /*\n> > + * Cull sources, again:\n> > + * - remove ones involved in renames (found via basenames)\n> > + */\n> > + trace2_region_enter(\"diff\", \"cull basename\", options->repo);\n> > + remove_unneeded_paths_from_src(want_copies);\n> > + trace2_region_leave(\"diff\", \"cull basename\", options->repo);\n> > + }\n> > +\n>\n> Thanks,\n> -Stolee\n"},{"id":"416542","messageId":"CABPp-BEfCLRAgCy3x3Dm_RM=U2q9sNWX7YO4OVLrqvsvzhPxfw@mail.gmail.com","threadId":"55112","inReplyTo":"9d8ee4be-33d7-6c50-f14b-0b92a4ebe949@gmail.com","subject":"Re: [PATCH v2 1/4] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-09T17:42:18Z","receivedAt":"2021-02-09T17:47:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Feb 9, 2021 at 9:02 AM Derrick Stolee wrote:\n>\n> On 2/9/2021 11:56 AM, Elijah Newren wrote:\n> >> Also, this is used here and below. Perhaps it's worth pulling out as a\n> >> helper? I see similar code being duplicated in these existing spots:\n> >>\n> >> * diff-no-index.c:append_basename()\n> >> * help.c:append_similar_ref()\n> >> * packfile.c:pack_basename()\n> >> * replace-object.c:register_replace_ref()\n> >> * setup.c:read_gitfile_gently()\n> >> * builtin/rebase.c:cmd_rebase()\n> >> * builtin/stash.c:do_create_stash()\n> >> * builtin/worktree.c:add_worktree()\n> >> * contrib/credential/gnome-keyring/git-credential-gnome-keyring.c:usage()\n> >> * contrib/credential/libsecret/git-credential-libsecret.c:usage()\n> >> * trace2/tr2_dst.c:tr2_dst_try_auto_path()\n> > Honestly asking: would anyone ever search for such a two-line helper\n> > function? I wouldn't have even thought to look, since it seems so\n> > simple.\n> >\n> > However, my real concern here is that this type of change would risk\n> > introducing conflicts with unrelated series. This series is the\n> > second in what will be a 9-series deep dependency chain of\n> > optimizations[1], and the later series are going to be longer than\n> > these first two were (the latter ones are 6-11 patches each). We've\n> > already discussed previously whether we possibly want to hold the\n> > first couple optimization series out of the upcoming git-2.31 release\n> > in order to keep the optimizations all together, but that might\n> > increase the risk of conflicts with unrelated patches if we try a\n> > bigger tree refactor like this. (Junio never commented on that,\n> > though.) It might be better to keep the series touching only\n> > merge-ort.c & diffcore-rename.c, and then do cleanups like the one you\n> > suggest here after the whole series.\n> >\n> > That said, it's not a difficult initial change, so I'm mostly\n> > expressing this concern out of making things harder for Junio. It'd\n> > be best to get his opinion -- Junio, your thoughts?\n> >\n> > [1] https://github.com/gitgitgadget/git/pulls?q=is%3Apr+author%3Anewren+Optimization+batch\n>\n> I don't consider the step of \"go put the helper in all these other\n> places\" necessary for the current series. However, the \"get basename\"\n> code appears a total of three times in this series, so it would be\n> good to at least extract it to a static inline method to reduce\n> the duplication isolated to this change.\n\nSounds good; will do.\n"},{"id":"416543","messageId":"CABPp-BGM8N_PaS_M148DoteO3dHNq+gvmNbigHuHsgkVjocCww@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqo8gt2mxh.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 4/4] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-09T17:44:28Z","receivedAt":"2021-02-09T17:55:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Feb 9, 2021 at 9:03 AM Junio C Hamano wrote:\n>\n> Derrick Stolee writes:\n>\n> >> +Note that when rename detection is on but both copy and break\n> >> +detection are off, rename detection adds a preliminary step that first\n> >> +checks files with the same basename. If files with the same basename\n> >\n> > I find myself wanting a definition of 'basename' here, but perhaps I'm\n> > just being pedantic. A quick search clarifies this as a standard term [1]\n> > of which I was just ignorant.\n> >\n> > [1] https://man7.org/linux/man-pages/man3/basename.3.html\n> >\n> >> +are sufficiently similar, it will mark them as renames and exclude\n> >> +them from the later quadratic step (the one that pairwise compares all\n> >> +unmatched files to find the \"best\" matches, determined by the highest\n> >> +content similarity).\n>\n> While I do not think `basename` is unacceptably bad, we should aim\n> to do better. For \"direc/tory/hello.txt\", both \"hello.txt\" or\n> \"hello\" are what would come up to people's mind with the technical\n> term \"basename\" (i.e. basename as opposed to dirname, vs basename as\n> opposed to filename with .extension).\n>\n> Avoiding this ambiguity and using a word understandable by those not\n> versed well with UNIX/POSIX lingo may be done at the same time,\n> hopefully.\n>\n> For example, can we frame the description around this key sentence:\n>\n> The heuristics is based on an observation that a file is often\n> moved across directories while keeping its filename the same.\n>\n> The term \"filename\" alone can be ambiguous (i.e. both \"hello.txt\"\n> and \"direc/tory/hello.txt\" are valid interpretations in the earlier\n> example), but in the context of a sentence that talks about \"moved\n> across directories\", the former would become the only valid one. We\n> can even say just \"name\" and there is no ambiguity in the above \"key\n> sentence\".\n>\n> Then keeping that in mind, we can rewrite the above you quoted like\n> so without going technical and without risking ambiguity, like this:\n>\n> ... a preliminary step that checks if files are moved across\n> directories while keeping their filenames the same. If there is\n> a file added to a directory whose contents is sufficiently\n> similar to a file with the same name that got deleted from a\n> different directory, ...\n\nNice, I like it!\n"},{"id":"416545","messageId":"xmqqa6sd2hk3.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"CABPp-BFfF5rxbHwP1xnr63fPmp2=NkTZ-r0=FVZ1oVhAZD_s9w@mail.gmail.com","subject":"Re: [PATCH v2 3/4] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-09T18:59:40Z","receivedAt":"2021-02-09T19:15:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Since I didn't know what to use, though, and I didn't want to get a\n> different set of numbers for the final commit message on the speedup\n> achieved if I'm just going to throw them away and recompute once I\n> find out what Junio wants here, I did intentionally set the\n> computation to just give us minimum_score, for now.\n\nI thought Derrick earlier suggested \"half-way\", which I found was\nprobably a reasonable starting point. So instead of 5, divide by 8\nand multiply both by 4 or something and perhaps allow a debugging\nknob to tweak to see what works the best in the real histories\nduring the refinement phase of the feature?\n\n\n\n"},{"id":"416630","messageId":"pull.843.v3.git.1612970140.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v2.git.1612870326.gitgitgadget@gmail.com","subject":"[PATCH v3 0/5] Optimization batch 7: use file basenames to guide rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-10T15:15:35Z","receivedAt":"2021-02-10T15:16:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on ort-perf-batch-6[1].\n\nThis series uses file basenames (portion of the path after final '/',\nincluding extension) in a basic fashion to guide rename detection.\n\nChanges since v2:\n\n * insert a new patch at the beginning to test expected old behavior, then\n have later patches change the test expectation\n * tweak commit message wording to clarify that rename stats merely\n motivated the optimization\n * factor out a simple get_basename() helper; have it document why we don't\n use gitbasename()\n * use a higher required threshold to mark same-basename files as a rename,\n defaulting to average of minimum_score and MAX_SCORE (since default\n rename threshold is 50%, this implies default basename threshold is 75%)\n * provide an environment variable (undocumented) that can be used to test\n appropriate threshold for basename-sameness\n * updated the timings based on the new threshold\n * modify the documentation with Junio's suggested simpler and clearer\n wording\n * clean up the wording on a few comments\n\n[1] https://lore.kernel.org/git/xmqqlfc4byt6.fsf@gitster.c.googlers.com/ [2]\nhttps://github.com/newren/presentations/blob/pdfs/merge-performance/merge-performance-slides.pdf\n\nElijah Newren (5):\n t4001: add a test comparing basename similarity and content similarity\n diffcore-rename: compute basenames of all source and dest candidates\n diffcore-rename: complete find_basename_matches()\n diffcore-rename: guide inexact rename detection based on basenames\n gitdiffcore doc: mention new preliminary step for rename detection\n\n Documentation/gitdiffcore.txt | 17 +++\n diffcore-rename.c | 202 +++++++++++++++++++++++++++++++++-\n t/t4001-diff-rename.sh | 24 ++++\n 3 files changed, 239 insertions(+), 4 deletions(-)\n\n\nbase-commit: 7ae9460d3dba84122c2674b46e4339b9d42bdedd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-843%2Fnewren%2Fort-perf-batch-7-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-843/newren/ort-perf-batch-7-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/843\n\nRange-diff vs v2:\n\n 4: a0e75d8cd6bd ! 1: 3e6af929d135 gitdiffcore doc: mention new preliminary step for rename detection\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - gitdiffcore doc: mention new preliminary step for rename detection\n + t4001: add a test comparing basename similarity and content similarity\n \n - The last few patches have introduced a new preliminary step when rename\n - detection is on but both break detection and copy detection are off.\n - Document this new step. While we're at it, add a testcase that checks\n - the new behavior as well.\n + Add a simple test where a removed file is similar to two different added\n + files; one of them has the same basename, and the other has a slightly\n + higher content similarity. Without break detection, filename similarity\n + of 100% trumps content similarity for pairing up related files. For\n + any filename similarity less than 100%, the opposite is true -- content\n + similarity is all that matters. Add a testcase that documents this.\n \n - Signed-off-by: Elijah Newren \n + Subsequent commits will add a new rule that includes an inbetween state,\n + where a mixture of filename similarity and content similarity are\n + weighed, and which will change the outcome of this testcase.\n \n - ## Documentation/gitdiffcore.txt ##\n -@@ Documentation/gitdiffcore.txt: a similarity score different from the default of 50% by giving a\n - number after the \"-M\" or \"-C\" option (e.g. \"-M8\" to tell it to use\n - 8/10 = 80%).\n - \n -+Note that when rename detection is on but both copy and break\n -+detection are off, rename detection adds a preliminary step that first\n -+checks files with the same basename. If files with the same basename\n -+are sufficiently similar, it will mark them as renames and exclude\n -+them from the later quadratic step (the one that pairwise compares all\n -+unmatched files to find the \"best\" matches, determined by the highest\n -+content similarity). So, for example, if docs/extensions.txt and\n -+docs/config/extensions.txt have similar content, then they will be\n -+marked as a rename even if it turns out that docs/extensions.txt was\n -+more similar to src/extension-checks.c. At most, one comparison is\n -+done per file in this preliminary pass; so if there are several\n -+extensions.txt files throughout the directory hierarchy that were\n -+added and deleted, this preliminary step will be skipped for those\n -+files.\n -+\n - Note. When the \"-C\" option is used with `--find-copies-harder`\n - option, 'git diff-{asterisk}' commands feed unmodified filepairs to\n - diffcore mechanism as well as modified ones. This lets the copy\n + Signed-off-by: Elijah Newren \n \n ## t/t4001-diff-rename.sh ##\n @@ t/t4001-diff-rename.sh: test_expect_success 'diff-tree -l0 defaults to a big rename limit, not zero' '\n @@ t/t4001-diff-rename.sh: test_expect_success 'diff-tree -l0 defaults to a big ren\n +\t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n +\t# but since same basenames are checked first...\n +\tcat >expected <<-\\EOF &&\n -+\tA\tfile.md\n -+\tR078\tsubdir/file.txt\tfile.txt\n ++\tR088\tsubdir/file.txt\tfile.md\n ++\tA\tfile.txt\n +\tEOF\n +\ttest_cmp expected actual\n +'\n 1: 381a45d239bb ! 2: 4fff9b1ff57b diffcore-rename: compute basenames of all source and dest candidates\n @@ Commit message\n basenames still show up in the map, but have an invalid index (-1).\n \n This function was inspired by the fact that in real world repositories,\n - most renames often do not involve a basename change. Here are some\n - sample repositories and the percentage of their historical renames (as of\n - early 2020) that did not involve a basename change:\n + files are often moved across directories without changing names. Here\n + are some sample repositories and the percentage of their historical\n + renames (as of early 2020) that preserved basenames:\n * linux: 76%\n * gcc: 64%\n * gecko: 79%\n * webkit: 89%\n + These statistics alone don't prove that an optimization in this area\n + will help or how much it will help, since there are also unpaired adds\n + and deletes, restrictions on which basenames we consider, etc., but it\n + certainly motivated the idea to try something in this area.\n \n Signed-off-by: Elijah Newren \n \n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n ++static const char *get_basename(const char *filename)\n ++{\n ++\t/*\n ++\t * gitbasename() has to worry about special drivers, multiple\n ++\t * directory separator characters, trailing slashes, NULL or\n ++\t * empty strings, etc. We only work on filenames as stored in\n ++\t * git, and thus get to ignore all those complications.\n ++\t */\n ++\tconst char *base = strrchr(filename, '/');\n ++\treturn base ? base + 1 : filename;\n ++}\n ++\n +MAYBE_UNUSED\n +static int find_basename_matches(struct diff_options *options,\n +\t\t\t\t int minimum_score,\n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n +\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n +\tfor (i = 0; i < num_src; ++i) {\n +\t\tchar *filename = rename_src[i].p->one->path;\n -+\t\tchar *base;\n ++\t\tconst char *base;\n +\n +\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n +\t\tassert(!rename_src[i].p->one->rename_used);\n +\n -+\t\tbase = strrchr(filename, '/');\n -+\t\tbase = (base ? base+1 : filename);\n -+\n +\t\t/* Record index within rename_src (i) if basename is unique */\n ++\t\tbase = get_basename(filename);\n +\t\tif (strintmap_contains(&sources, base))\n +\t\t\tstrintmap_set(&sources, base, -1);\n +\t\telse\n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n +\t}\n +\tfor (i = 0; i < rename_dst_nr; ++i) {\n +\t\tchar *filename = rename_dst[i].p->two->path;\n -+\t\tchar *base;\n ++\t\tconst char *base;\n +\n +\t\tif (rename_dst[i].is_rename)\n +\t\t\tcontinue; /* involved in exact match already. */\n +\n -+\t\tbase = strrchr(filename, '/');\n -+\t\tbase = (base ? base+1 : filename);\n -+\n +\t\t/* Record index within rename_dst (i) if basename is unique */\n ++\t\tbase = get_basename(filename);\n +\t\tif (strintmap_contains(&dests, base))\n +\t\t\tstrintmap_set(&dests, base, -1);\n +\t\telse\n 2: dcd0175229aa ! 3: dc26881e4ed3 diffcore-rename: complete find_basename_matches()\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n {\n -\tint i;\n +\t/*\n -+\t * When I checked, over 76% of file renames in linux just moved\n -+\t * files to a different directory but kept the same basename. gcc\n -+\t * did that with over 64% of renames, gecko did it with over 79%,\n -+\t * and WebKit did it with over 89%.\n ++\t * When I checked in early 2020, over 76% of file renames in linux\n ++\t * just moved files to a different directory but kept the same\n ++\t * basename. gcc did that with over 64% of renames, gecko did it\n ++\t * with over 79%, and WebKit did it with over 89%.\n +\t *\n +\t * Therefore we can bypass the normal exhaustive NxM matrix\n +\t * comparison of similarities between all potential rename sources\n -+\t * and destinations by instead using file basename as a hint, checking\n -+\t * for similarity between files with the same basename, and if we\n -+\t * find a pair that are sufficiently similar, record the rename\n -+\t * pair and exclude those two from the NxM matrix.\n ++\t * and destinations by instead using file basename as a hint (i.e.\n ++\t * the portion of the filename after the last '/'), checking for\n ++\t * similarity between files with the same basename, and if we find\n ++\t * a pair that are sufficiently similar, record the rename pair and\n ++\t * exclude those two from the NxM matrix.\n +\t *\n +\t * This *might* cause us to find a less than optimal pairing (if\n +\t * there is another file that we are even more similar to but has a\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n +\t * basename matching provides, and given the frequency with which\n +\t * people use the same basename in real world projects, that's a\n +\t * trade-off we are willing to accept when doing just rename\n -+\t * detection. However, if someone wants copy detection that\n -+\t * implies they are willing to spend more cycles to find\n -+\t * similarities between files, so it may be less likely that this\n -+\t * heuristic is wanted.\n ++\t * detection.\n ++\t *\n ++\t * If someone wants copy detection that implies they are willing to\n ++\t * spend more cycles to find similarities between files, so it may\n ++\t * be less likely that this heuristic is wanted. If someone is\n ++\t * doing break detection, that means they do not want filename\n ++\t * similarity to imply any form of content similiarity, and thus\n ++\t * this heuristic would definitely be incompatible.\n +\t */\n +\n +\tint i, renames = 0;\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n \tstruct strintmap dests;\n \n +\t/*\n -+\t * The prefeteching stuff wants to know if it can skip prefetching blobs\n -+\t * that are unmodified. unmodified blobs are only relevant when doing\n -+\t * copy detection. find_basename_matches() is only used when detecting\n -+\t * renames, not when detecting copies, so it'll only be used when a file\n -+\t * only existed in the source. Since we already know that the file\n -+\t * won't be unmodified, there's no point checking for it; that's just a\n -+\t * waste of resources. So set skip_unmodified to 0 so that\n -+\t * estimate_similarity() and prefetch() won't waste resources checking\n -+\t * for something we already know is false.\n ++\t * The prefeteching stuff wants to know if it can skip prefetching\n ++\t * blobs that are unmodified...and will then do a little extra work\n ++\t * to verify that the oids are indeed different before prefetching.\n ++\t * Unmodified blobs are only relevant when doing copy detection;\n ++\t * when limiting to rename detection, diffcore_rename[_extended]()\n ++\t * will never be called with unmodified source paths fed to us, so\n ++\t * the extra work necessary to check if rename_src entries are\n ++\t * unmodified would be a small waste.\n +\t */\n +\tint skip_unmodified = 0;\n +\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n +\t/* Now look for basename matchups and do similarity estimation */\n +\tfor (i = 0; i < num_src; ++i) {\n +\t\tchar *filename = rename_src[i].p->one->path;\n -+\t\tchar *base = NULL;\n ++\t\tconst char *base = NULL;\n +\t\tintptr_t src_index;\n +\t\tintptr_t dst_index;\n +\n -+\t\t/* Get the basename */\n -+\t\tbase = strrchr(filename, '/');\n -+\t\tbase = (base ? base+1 : filename);\n -+\n +\t\t/* Find out if this basename is unique among sources */\n ++\t\tbase = get_basename(filename);\n +\t\tsrc_index = strintmap_get(&sources, base);\n +\t\tif (src_index == -1)\n +\t\t\tcontinue; /* not a unique basename; skip it */\n 3: ce2173aa1fb7 ! 4: 2493f4b2f55d diffcore-rename: guide inexact rename detection based on basenames\n @@ Commit message\n this change improves the performance as follows:\n \n Before After\n - no-renames: 13.815 s ± 0.062 s 13.138 s ± 0.086 s\n - mega-renames: 1799.937 s ± 0.493 s 169.488 s ± 0.494 s\n - just-one-mega: 51.289 s ± 0.019 s 5.061 s ± 0.017 s\n + no-renames: 13.815 s ± 0.062 s 13.294 s ± 0.103 s\n + mega-renames: 1799.937 s ± 0.493 s 187.248 s ± 0.882 s\n + just-one-mega: 51.289 s ± 0.019 s 5.557 s ± 0.017 s\n \n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n - \treturn renames;\n +@@ diffcore-rename.c: static const char *get_basename(const char *filename)\n + \treturn base ? base + 1 : filename;\n }\n \n -MAYBE_UNUSED\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n +\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n +\t} else {\n +\t\t/* Determine minimum score to match basenames */\n -+\t\tint min_basename_score = (int)(5*minimum_score + 0*MAX_SCORE)/5;\n ++\t\tdouble factor = 0.5;\n ++\t\tchar *basename_factor = getenv(\"GIT_BASENAME_FACTOR\");\n ++\t\tint min_basename_score;\n ++\n ++\t\tif (basename_factor)\n ++\t\t\tfactor = strtol(basename_factor, NULL, 10)/100.0;\n ++\t\tassert(factor >= 0.0 && factor <= 1.0);\n ++\t\tmin_basename_score = minimum_score +\n ++\t\t\t(int)(factor * (MAX_SCORE - minimum_score));\n +\n +\t\t/*\n +\t\t * Cull sources:\n +\t\t * - remove ones involved in renames (found via exact match)\n +\t\t */\n -+\t\ttrace2_region_enter(\"diff\", \"cull exact\", options->repo);\n ++\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n +\t\tremove_unneeded_paths_from_src(want_copies);\n -+\t\ttrace2_region_leave(\"diff\", \"cull exact\", options->repo);\n ++\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n +\n +\t\t/* Utilize file basenames to quickly find renames. */\n +\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n \n \t\tm = &mx[dst_cnt * NUM_CANDIDATE_PER_DST];\n \t\tfor (j = 0; j < NUM_CANDIDATE_PER_DST; j++)\n +\n + ## t/t4001-diff-rename.sh ##\n +@@ t/t4001-diff-rename.sh: test_expect_success 'basename similarity vs best similarity' '\n + \t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n + \t# but since same basenames are checked first...\n + \tcat >expected <<-\\EOF &&\n +-\tR088\tsubdir/file.txt\tfile.md\n +-\tA\tfile.txt\n ++\tA\tfile.md\n ++\tR078\tsubdir/file.txt\tfile.txt\n + \tEOF\n + \ttest_cmp expected actual\n + '\n -: ------------ > 5: fc72d24a3358 gitdiffcore doc: mention new preliminary step for rename detection\n\n-- \ngitgitgadget\n"},{"id":"416631","messageId":"3e6af929d135ef2dc239e2f47f92a7e2e91cbd17.1612970140.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v3.git.1612970140.gitgitgadget@gmail.com","subject":"[PATCH v3 1/5] t4001: add a test comparing basename similarity and content similarity","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-10T15:15:36Z","receivedAt":"2021-02-10T15:16:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd a simple test where a removed file is similar to two different added\nfiles; one of them has the same basename, and the other has a slightly\nhigher content similarity. Without break detection, filename similarity\nof 100% trumps content similarity for pairing up related files. For\nany filename similarity less than 100%, the opposite is true -- content\nsimilarity is all that matters. Add a testcase that documents this.\n\nSubsequent commits will add a new rule that includes an inbetween state,\nwhere a mixture of filename similarity and content similarity are\nweighed, and which will change the outcome of this testcase.\n\nSigned-off-by: Elijah Newren \n---\n t/t4001-diff-rename.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex c16486a9d41a..797343b38106 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -262,4 +262,28 @@ test_expect_success 'diff-tree -l0 defaults to a big rename limit, not zero' '\n \tgrep \"myotherfile.*myfile\" actual\n '\n \n+test_expect_success 'basename similarity vs best similarity' '\n+\tmkdir subdir &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 line10 >subdir/file.txt &&\n+\tgit add subdir/file.txt &&\n+\tgit commit -m \"base txt\" &&\n+\n+\tgit rm subdir/file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 >file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 >file.md &&\n+\tgit add file.txt file.md &&\n+\tgit commit -a -m \"rename\" &&\n+\tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n+\t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n+\t# but since same basenames are checked first...\n+\tcat >expected <<-\\EOF &&\n+\tR088\tsubdir/file.txt\tfile.md\n+\tA\tfile.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"416632","messageId":"4fff9b1ff57b62587b1cbec2031f96199a214702.1612970140.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v3.git.1612970140.gitgitgadget@gmail.com","subject":"[PATCH v3 2/5] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-10T15:15:37Z","receivedAt":"2021-02-10T15:17:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to make use of unique basenames to help inform rename detection,\nso that more likely pairings can be checked first. (src/moduleA/foo.txt\nand source/module/A/foo.txt are likely related if there are no other\n'foo.txt' files among the deleted and added files.) Add a new function,\nnot yet used, which creates a map of the unique basenames within\nrename_src and another within rename_dst, together with the indices\nwithin rename_src/rename_dst where those basenames show up. Non-unique\nbasenames still show up in the map, but have an invalid index (-1).\n\nThis function was inspired by the fact that in real world repositories,\nfiles are often moved across directories without changing names. Here\nare some sample repositories and the percentage of their historical\nrenames (as of early 2020) that preserved basenames:\n * linux: 76%\n * gcc: 64%\n * gecko: 79%\n * webkit: 89%\nThese statistics alone don't prove that an optimization in this area\nwill help or how much it will help, since there are also unpaired adds\nand deletes, restrictions on which basenames we consider, etc., but it\ncertainly motivated the idea to try something in this area.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 74930716e70d..3eb49a098adf 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,67 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+static const char *get_basename(const char *filename)\n+{\n+\t/*\n+\t * gitbasename() has to worry about special drivers, multiple\n+\t * directory separator characters, trailing slashes, NULL or\n+\t * empty strings, etc. We only work on filenames as stored in\n+\t * git, and thus get to ignore all those complications.\n+\t */\n+\tconst char *base = strrchr(filename, '/');\n+\treturn base ? base + 1 : filename;\n+}\n+\n+MAYBE_UNUSED\n+static int find_basename_matches(struct diff_options *options,\n+\t\t\t\t int minimum_score,\n+\t\t\t\t int num_src)\n+{\n+\tint i;\n+\tstruct strintmap sources;\n+\tstruct strintmap dests;\n+\n+\t/* Create maps of basename -> fullname(s) for sources and dests */\n+\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n+\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base;\n+\n+\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n+\t\tassert(!rename_src[i].p->one->rename_used);\n+\n+\t\t/* Record index within rename_src (i) if basename is unique */\n+\t\tbase = get_basename(filename);\n+\t\tif (strintmap_contains(&sources, base))\n+\t\t\tstrintmap_set(&sources, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&sources, base, i);\n+\t}\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\tconst char *base;\n+\n+\t\tif (rename_dst[i].is_rename)\n+\t\t\tcontinue; /* involved in exact match already. */\n+\n+\t\t/* Record index within rename_dst (i) if basename is unique */\n+\t\tbase = get_basename(filename);\n+\t\tif (strintmap_contains(&dests, base))\n+\t\t\tstrintmap_set(&dests, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&dests, base, i);\n+\t}\n+\n+\t/* TODO: Make use of basenames source and destination basenames */\n+\n+\tstrintmap_clear(&sources);\n+\tstrintmap_clear(&dests);\n+\n+\treturn 0;\n+}\n+\n #define NUM_CANDIDATE_PER_DST 4\n static void record_if_better(struct diff_score m[], struct diff_score *o)\n {\n-- \ngitgitgadget\n\n"},{"id":"416633","messageId":"2493f4b2f55d2b602e448db3f09da1ee3a97c81b.1612970140.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v3.git.1612970140.gitgitgadget@gmail.com","subject":"[PATCH v3 4/5] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-10T15:15:39Z","receivedAt":"2021-02-10T15:17:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMake use of the new find_basename_matches() function added in the last\ntwo patches, to find renames more rapidly in cases where we can match up\nfiles based on basenames. As a quick reminder (see the last two commit\nmessages for more details), this means for example that\ndocs/extensions.txt and docs/config/extensions.txt are considered likely\nrenames if there are no 'extensions.txt' files elsewhere among the added\nand deleted files, and if a similarity check confirms they are similar,\nthen they are marked as a rename without looking for a better similarity\nmatch among other files. This is a behavioral change, as covered in\nmore detail in the previous commit message.\n\nWe do not use this heuristic together with either break or copy\ndetection. The point of break detection is to say that filename\nsimilarity does not imply file content similarity, and we only want to\nknow about file content similarity. The point of copy detection is to\nuse more resources to check for additional similarities, while this is\nan optimization that uses far less resources but which might also result\nin finding slightly fewer similarities. So the idea behind this\noptimization goes against both of those features, and will be turned off\nfor both.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 13.815 s ± 0.062 s 13.294 s ± 0.103 s\n mega-renames: 1799.937 s ± 0.493 s 187.248 s ± 0.882 s\n just-one-mega: 51.289 s ± 0.019 s 5.557 s ± 0.017 s\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 54 ++++++++++++++++++++++++++++++++++++++----\n t/t4001-diff-rename.sh | 4 ++--\n 2 files changed, 51 insertions(+), 7 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 001645624e71..df76e475c710 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -379,7 +379,6 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n-MAYBE_UNUSED\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n@@ -727,12 +726,57 @@ void diffcore_rename(struct diff_options *options)\n \tif (minimum_score == MAX_SCORE)\n \t\tgoto cleanup;\n \n+\tnum_sources = rename_src_nr;\n+\n+\tif (want_copies || break_idx) {\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones corresponding to exact renames\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\t} else {\n+\t\t/* Determine minimum score to match basenames */\n+\t\tdouble factor = 0.5;\n+\t\tchar *basename_factor = getenv(\"GIT_BASENAME_FACTOR\");\n+\t\tint min_basename_score;\n+\n+\t\tif (basename_factor)\n+\t\t\tfactor = strtol(basename_factor, NULL, 10)/100.0;\n+\t\tassert(factor >= 0.0 && factor <= 1.0);\n+\t\tmin_basename_score = minimum_score +\n+\t\t\t(int)(factor * (MAX_SCORE - minimum_score));\n+\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones involved in renames (found via exact match)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\n+\t\t/* Utilize file basenames to quickly find renames. */\n+\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n+\t\trename_count += find_basename_matches(options,\n+\t\t\t\t\t\t min_basename_score,\n+\t\t\t\t\t\t rename_src_nr);\n+\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n+\n+\t\t/*\n+\t\t * Cull sources, again:\n+\t\t * - remove ones involved in renames (found via basenames)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull basename\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull basename\", options->repo);\n+\t}\n+\n \t/*\n-\t * Calculate how many renames are left\n+\t * Calculate how many rename destinations are left\n \t */\n \tnum_destinations = (rename_dst_nr - rename_count);\n-\tremove_unneeded_paths_from_src(want_copies);\n-\tnum_sources = rename_src_nr;\n+\tnum_sources = rename_src_nr; /* rename_src_nr reflects lower number */\n \n \t/* All done? */\n \tif (!num_destinations || !num_sources)\n@@ -764,7 +808,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tstruct diff_score *m;\n \n \t\tif (rename_dst[i].is_rename)\n-\t\t\tcontinue; /* dealt with exact match already. */\n+\t\t\tcontinue; /* exact or basename match already handled */\n \n \t\tm = &mx[dst_cnt * NUM_CANDIDATE_PER_DST];\n \t\tfor (j = 0; j < NUM_CANDIDATE_PER_DST; j++)\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex 797343b38106..bf62537c29a0 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -280,8 +280,8 @@ test_expect_success 'basename similarity vs best similarity' '\n \t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n \t# but since same basenames are checked first...\n \tcat >expected <<-\\EOF &&\n-\tR088\tsubdir/file.txt\tfile.md\n-\tA\tfile.txt\n+\tA\tfile.md\n+\tR078\tsubdir/file.txt\tfile.txt\n \tEOF\n \ttest_cmp expected actual\n '\n-- \ngitgitgadget\n\n"},{"id":"416634","messageId":"dc26881e4ed3447c6efdcd492463be294f99b8c4.1612970140.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v3.git.1612970140.gitgitgadget@gmail.com","subject":"[PATCH v3 3/5] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-10T15:15:38Z","receivedAt":"2021-02-10T15:17:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIt is not uncommon in real world repositories for the majority of file\nrenames to not change the basename of the file; i.e. most \"renames\" are\njust a move of files into different directories. We can make use of\nthis to avoid comparing all rename source candidates with all rename\ndestination candidates, by first comparing sources to destinations with\nthe same basenames. If two files with the same basename are\nsufficiently similar, we record the rename; if not, we include those\nfiles in the more exhaustive matrix comparison.\n\nThis means we are adding a set of preliminary additional comparisons,\nbut for each file we only compare it with at most one other file. For\nexample, if there was a include/media/device.h that was deleted and a\nsrc/module/media/device.h that was added, and there were no other\ndevice.h files added or deleted between the commits being compared,\nthen these two files would be compared in the preliminary step.\n\nThis commit does not yet actually employ this new optimization, it\nmerely adds a function which can be used for this purpose. The next\ncommit will do the necessary plumbing to make use of it.\n\nNote that this optimization might give us different results than without\nthe optimization, because it's possible that despite files with the same\nbasename being sufficiently similar to be considered a rename, there's\nan even better match between files without the same basename. I think\nthat is okay for four reasons: (1) it's easy to explain to the users\nwhat happened if it does ever occur (or even for them to intuitively\nfigure out), (2) as the next patch will show it provides such a large\nperformance boost that it's worth the tradeoff, and (3) it's somewhat\nunlikely that despite having unique matching basenames that other files\nserve as better matches. Reason (4) takes a full paragraph to\nexplain...\n\nIf the previous three reasons aren't enough, consider what rename\ndetection already does. Break detection is not the default, meaning\nthat if files have the same _fullname_, then they are considered related\neven if they are 0% similar. In fact, in such a case, we don't even\nbother comparing the files to see if they are similar let alone\ncomparing them to all other files to see what they are most similar to.\nBasically, we override content similarity based on sufficient filename\nsimilarity. Without the filename similarity (currently implemented as\nan exact match of filename), we swing the pendulum the opposite\ndirection and say that filename similarity is irrelevant and compare a\nfull N x M matrix of sources and destinations to find out which have the\nmost similar contents. This optimization just adds another form of\nfilename similarity comparison, but augments it with a file content\nsimilarity check as well. Basically, if two files have the same\nbasename and are sufficiently similar to be considered a rename, mark\nthem as such without comparing the two to all other rename candidates.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 95 +++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 92 insertions(+), 3 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 3eb49a098adf..001645624e71 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -384,10 +384,52 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n {\n-\tint i;\n+\t/*\n+\t * When I checked in early 2020, over 76% of file renames in linux\n+\t * just moved files to a different directory but kept the same\n+\t * basename. gcc did that with over 64% of renames, gecko did it\n+\t * with over 79%, and WebKit did it with over 89%.\n+\t *\n+\t * Therefore we can bypass the normal exhaustive NxM matrix\n+\t * comparison of similarities between all potential rename sources\n+\t * and destinations by instead using file basename as a hint (i.e.\n+\t * the portion of the filename after the last '/'), checking for\n+\t * similarity between files with the same basename, and if we find\n+\t * a pair that are sufficiently similar, record the rename pair and\n+\t * exclude those two from the NxM matrix.\n+\t *\n+\t * This *might* cause us to find a less than optimal pairing (if\n+\t * there is another file that we are even more similar to but has a\n+\t * different basename). Given the huge performance advantage\n+\t * basename matching provides, and given the frequency with which\n+\t * people use the same basename in real world projects, that's a\n+\t * trade-off we are willing to accept when doing just rename\n+\t * detection.\n+\t *\n+\t * If someone wants copy detection that implies they are willing to\n+\t * spend more cycles to find similarities between files, so it may\n+\t * be less likely that this heuristic is wanted. If someone is\n+\t * doing break detection, that means they do not want filename\n+\t * similarity to imply any form of content similiarity, and thus\n+\t * this heuristic would definitely be incompatible.\n+\t */\n+\n+\tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n \n+\t/*\n+\t * The prefeteching stuff wants to know if it can skip prefetching\n+\t * blobs that are unmodified...and will then do a little extra work\n+\t * to verify that the oids are indeed different before prefetching.\n+\t * Unmodified blobs are only relevant when doing copy detection;\n+\t * when limiting to rename detection, diffcore_rename[_extended]()\n+\t * will never be called with unmodified source paths fed to us, so\n+\t * the extra work necessary to check if rename_src entries are\n+\t * unmodified would be a small waste.\n+\t */\n+\tint skip_unmodified = 0;\n+\n \t/* Create maps of basename -> fullname(s) for sources and dests */\n \tstrintmap_init_with_options(&sources, -1, NULL, 0);\n \tstrintmap_init_with_options(&dests, -1, NULL, 0);\n@@ -420,12 +462,59 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\tstrintmap_set(&dests, base, i);\n \t}\n \n-\t/* TODO: Make use of basenames source and destination basenames */\n+\t/* Now look for basename matchups and do similarity estimation */\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base = NULL;\n+\t\tintptr_t src_index;\n+\t\tintptr_t dst_index;\n+\n+\t\t/* Find out if this basename is unique among sources */\n+\t\tbase = get_basename(filename);\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tif (src_index == -1)\n+\t\t\tcontinue; /* not a unique basename; skip it */\n+\t\tassert(src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n+\t\t\tstruct diff_filespec *one, *two;\n+\t\t\tint score;\n+\n+\t\t\t/* Find out if this basename is unique among dests */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue; /* not a unique basename; skip it */\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n+\t\t\t/* Estimate the similarity */\n+\t\t\tone = rename_src[src_index].p->one;\n+\t\t\ttwo = rename_dst[dst_index].p->two;\n+\t\t\tscore = estimate_similarity(options->repo, one, two,\n+\t\t\t\t\t\t minimum_score, skip_unmodified);\n+\n+\t\t\t/* If sufficiently similar, record as rename pair */\n+\t\t\tif (score < minimum_score)\n+\t\t\t\tcontinue;\n+\t\t\trecord_rename_pair(dst_index, src_index, score);\n+\t\t\trenames++;\n+\n+\t\t\t/*\n+\t\t\t * Found a rename so don't need text anymore; if we\n+\t\t\t * didn't find a rename, the filespec_blob would get\n+\t\t\t * re-used when doing the matrix of comparisons.\n+\t\t\t */\n+\t\t\tdiff_free_filespec_blob(one);\n+\t\t\tdiff_free_filespec_blob(two);\n+\t\t}\n+\t}\n \n \tstrintmap_clear(&sources);\n \tstrintmap_clear(&dests);\n \n-\treturn 0;\n+\treturn renames;\n }\n \n #define NUM_CANDIDATE_PER_DST 4\n-- \ngitgitgadget\n\n"},{"id":"416635","messageId":"fc72d24a3358b1c5cc2753b5f07ac60174e6452b.1612970140.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v3.git.1612970140.gitgitgadget@gmail.com","subject":"[PATCH v3 5/5] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-10T15:15:40Z","receivedAt":"2021-02-10T15:17:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe last few patches have introduced a new preliminary step when rename\ndetection is on but both break detection and copy detection are off.\nDocument this new step. While we're at it, add a testcase that checks\nthe new behavior as well.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex c970d9fe438a..36ebe364d874 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -168,6 +168,23 @@ a similarity score different from the default of 50% by giving a\n number after the \"-M\" or \"-C\" option (e.g. \"-M8\" to tell it to use\n 8/10 = 80%).\n \n+Note that when rename detection is on but both copy and break\n+detection are off, rename detection adds a preliminary step that first\n+checks if files are moved across directories while keeping their\n+filename the same. If there is a file added to a directory whose\n+contents is sufficiently similar to a file with the same name that got\n+deleted from a different directory, it will mark them as renames and\n+exclude them from the later quadratic step (the one that pairwise\n+compares all unmatched files to find the \"best\" matches, determined by\n+the highest content similarity). So, for example, if\n+docs/extensions.txt and docs/config/extensions.txt have similar\n+content, then they will be marked as a rename even if it turns out\n+that docs/extensions.txt was more similar to src/extension-checks.c.\n+At most, one comparison is done per file in this preliminary pass; so\n+if there are several extensions.txt files throughout the directory\n+hierarchy that were added and deleted, this preliminary step will be\n+skipped for those files.\n+\n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\n diffcore mechanism as well as modified ones. This lets the copy\n-- \ngitgitgadget\n"},{"id":"416643","messageId":"xmqqtuqj3mfl.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"fc72d24a3358b1c5cc2753b5f07ac60174e6452b.1612970140.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/5] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-10T16:41:18Z","receivedAt":"2021-02-10T16:43:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> From: Elijah Newren \n>\n> The last few patches have introduced a new preliminary step when rename\n> detection is on but both break detection and copy detection are off.\n> Document this new step. While we're at it, add a testcase that checks\n> the new behavior as well.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> Documentation/gitdiffcore.txt | 17 +++++++++++++++++\n> 1 file changed, 17 insertions(+)\n>\n> diff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\n> index c970d9fe438a..36ebe364d874 100644\n> --- a/Documentation/gitdiffcore.txt\n> +++ b/Documentation/gitdiffcore.txt\n> @@ -168,6 +168,23 @@ a similarity score different from the default of 50% by giving a\n> number after the \"-M\" or \"-C\" option (e.g. \"-M8\" to tell it to use\n> 8/10 = 80%).\n> \n> +Note that when rename detection is on but both copy and break\n> +detection are off, rename detection adds a preliminary step that first\n> +checks if files are moved across directories while keeping their\n> +filename the same. If there is a file added to a directory whose\n> +contents is sufficiently similar to a file with the same name that got\n> +deleted from a different directory, it will mark them as renames and\n> +exclude them from the later quadratic step (the one that pairwise\n> +compares all unmatched files to find the \"best\" matches, determined by\n> +the highest content similarity). So, for example, if\n> +docs/extensions.txt and docs/config/extensions.txt have similar\n> +content, then they will be marked as a rename even if it turns out\n> +that docs/extensions.txt was more similar to src/extension-checks.c.\n\nI'd rather use docs/extensions.md instead of src/extension-checks.c;\nit would be more realistic for .md to be similar to .txt than .c.\n\nWith a raised bar for this step, the equation changes a bit, no? \n\n So, for example, if a deleted docs/ext.txt and an added\n docs/config/ext.txt are similar enough, they will be marked as a\n rename and prevent an added docs/ext.md that may be even similar\n to the deleted docs/ext.txt from being considered as the rename\n destination in the later step. For this reason, the preliminary\n \"match same filename\" step uses a bit higher threshold to mark a\n file pair as a rename and stop considering other candidates for\n better matches.\n\nor something?\n\n> +At most, one comparison is done per file in this preliminary pass; so\n> +if there are several extensions.txt files throughout the directory\n> +hierarchy that were added and deleted, this preliminary step will be\n> +skipped for those files.\n\nOther than that, the whole series looked sensible to my cursory\nread.\n\nThanks.\n\n"},{"id":"416645","messageId":"CABPp-BHuT6nvRF0QNHzXV0qNzQWAia0H7RMATNXtDSJyWJmeRA@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqtuqj3mfl.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 5/5] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-10T17:20:50Z","receivedAt":"2021-02-10T17:22:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 10, 2021 at 8:41 AM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > From: Elijah Newren \n> >\n> > The last few patches have introduced a new preliminary step when rename\n> > detection is on but both break detection and copy detection are off.\n> > Document this new step. While we're at it, add a testcase that checks\n> > the new behavior as well.\n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > Documentation/gitdiffcore.txt | 17 +++++++++++++++++\n> > 1 file changed, 17 insertions(+)\n> >\n> > diff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\n> > index c970d9fe438a..36ebe364d874 100644\n> > --- a/Documentation/gitdiffcore.txt\n> > +++ b/Documentation/gitdiffcore.txt\n> > @@ -168,6 +168,23 @@ a similarity score different from the default of 50% by giving a\n> > number after the \"-M\" or \"-C\" option (e.g. \"-M8\" to tell it to use\n> > 8/10 = 80%).\n> >\n> > +Note that when rename detection is on but both copy and break\n> > +detection are off, rename detection adds a preliminary step that first\n> > +checks if files are moved across directories while keeping their\n> > +filename the same. If there is a file added to a directory whose\n> > +contents is sufficiently similar to a file with the same name that got\n> > +deleted from a different directory, it will mark them as renames and\n> > +exclude them from the later quadratic step (the one that pairwise\n> > +compares all unmatched files to find the \"best\" matches, determined by\n> > +the highest content similarity). So, for example, if\n> > +docs/extensions.txt and docs/config/extensions.txt have similar\n> > +content, then they will be marked as a rename even if it turns out\n> > +that docs/extensions.txt was more similar to src/extension-checks.c.\n>\n> I'd rather use docs/extensions.md instead of src/extension-checks.c;\n> it would be more realistic for .md to be similar to .txt than .c.\n>\n> With a raised bar for this step, the equation changes a bit, no?\n>\n> So, for example, if a deleted docs/ext.txt and an added\n> docs/config/ext.txt are similar enough, they will be marked as a\n> rename and prevent an added docs/ext.md that may be even similar\n> to the deleted docs/ext.txt from being considered as the rename\n> destination in the later step. For this reason, the preliminary\n> \"match same filename\" step uses a bit higher threshold to mark a\n> file pair as a rename and stop considering other candidates for\n> better matches.\n>\n> or something?\n\nGood points; I've updated the docs locally to reflect your\nsuggestions, I'll wait a bit for any other feedback and then send out\na new round with this update.\n\n> > +At most, one comparison is done per file in this preliminary pass; so\n> > +if there are several extensions.txt files throughout the directory\n> > +hierarchy that were added and deleted, this preliminary step will be\n> > +skipped for those files.\n>\n> Other than that, the whole series looked sensible to my cursory\n> read.\n\nThanks.\n"},{"id":"416728","messageId":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v3.git.1612970140.gitgitgadget@gmail.com","subject":"[PATCH v4 0/6] Optimization batch 7: use file basenames to guide rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-11T08:15:43Z","receivedAt":"2021-02-11T08:16:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on ort-perf-batch-6[1].\n\nThis series uses file basenames (portion of the path after final '/',\nincluding extension) in a basic fashion to guide rename detection.\n\nChanges since v3:\n\n * update documentation as suggested by Junio\n * NEW: add another patch at the end, to simplify patch series that will be\n submitted later (please review!)\n\n[1] https://lore.kernel.org/git/xmqqlfc4byt6.fsf@gitster.c.googlers.com/\n\nElijah Newren (6):\n t4001: add a test comparing basename similarity and content similarity\n diffcore-rename: compute basenames of all source and dest candidates\n diffcore-rename: complete find_basename_matches()\n diffcore-rename: guide inexact rename detection based on basenames\n gitdiffcore doc: mention new preliminary step for rename detection\n merge-ort: call diffcore_rename() directly\n\n Documentation/gitdiffcore.txt | 20 ++++\n diffcore-rename.c | 202 +++++++++++++++++++++++++++++++++-\n merge-ort.c | 66 +++++++++--\n t/t4001-diff-rename.sh | 24 ++++\n 4 files changed, 301 insertions(+), 11 deletions(-)\n\n\nbase-commit: 7ae9460d3dba84122c2674b46e4339b9d42bdedd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-843%2Fnewren%2Fort-perf-batch-7-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-843/newren/ort-perf-batch-7-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/843\n\nRange-diff vs v3:\n\n 1: 3e6af929d135 = 1: 3e6af929d135 t4001: add a test comparing basename similarity and content similarity\n 2: 4fff9b1ff57b = 2: 4fff9b1ff57b diffcore-rename: compute basenames of all source and dest candidates\n 3: dc26881e4ed3 = 3: dc26881e4ed3 diffcore-rename: complete find_basename_matches()\n 4: 2493f4b2f55d = 4: 2493f4b2f55d diffcore-rename: guide inexact rename detection based on basenames\n 5: fc72d24a3358 ! 5: 4e86ed3f29d4 gitdiffcore doc: mention new preliminary step for rename detection\n @@ Documentation/gitdiffcore.txt: a similarity score different from the default of\n +deleted from a different directory, it will mark them as renames and\n +exclude them from the later quadratic step (the one that pairwise\n +compares all unmatched files to find the \"best\" matches, determined by\n -+the highest content similarity). So, for example, if\n -+docs/extensions.txt and docs/config/extensions.txt have similar\n -+content, then they will be marked as a rename even if it turns out\n -+that docs/extensions.txt was more similar to src/extension-checks.c.\n -+At most, one comparison is done per file in this preliminary pass; so\n -+if there are several extensions.txt files throughout the directory\n -+hierarchy that were added and deleted, this preliminary step will be\n -+skipped for those files.\n ++the highest content similarity). So, for example, if a deleted\n ++docs/ext.txt and an added docs/config/ext.txt are similar enough, they\n ++will be marked as a rename and prevent an added docs/ext.md that may\n ++be even more similar to the deleted docs/ext.txt from being considered\n ++as the rename destination in the later step. For this reason, the\n ++preliminary \"match same filename\" step uses a bit higher threshold to\n ++mark a file pair as a rename and stop considering other candidates for\n ++better matches. At most, one comparison is done per file in this\n ++preliminary pass; so if there are several ext.txt files throughout the\n ++directory hierarchy that were added and deleted, this preliminary step\n ++will be skipped for those files.\n +\n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\n -: ------------ > 6: fedb3d323d94 merge-ort: call diffcore_rename() directly\n\n-- \ngitgitgadget\n"},{"id":"416729","messageId":"3e6af929d135ef2dc239e2f47f92a7e2e91cbd17.1613031350.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"[PATCH v4 1/6] t4001: add a test comparing basename similarity and content similarity","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-11T08:15:44Z","receivedAt":"2021-02-11T08:17:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd a simple test where a removed file is similar to two different added\nfiles; one of them has the same basename, and the other has a slightly\nhigher content similarity. Without break detection, filename similarity\nof 100% trumps content similarity for pairing up related files. For\nany filename similarity less than 100%, the opposite is true -- content\nsimilarity is all that matters. Add a testcase that documents this.\n\nSubsequent commits will add a new rule that includes an inbetween state,\nwhere a mixture of filename similarity and content similarity are\nweighed, and which will change the outcome of this testcase.\n\nSigned-off-by: Elijah Newren \n---\n t/t4001-diff-rename.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex c16486a9d41a..797343b38106 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -262,4 +262,28 @@ test_expect_success 'diff-tree -l0 defaults to a big rename limit, not zero' '\n \tgrep \"myotherfile.*myfile\" actual\n '\n \n+test_expect_success 'basename similarity vs best similarity' '\n+\tmkdir subdir &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 line10 >subdir/file.txt &&\n+\tgit add subdir/file.txt &&\n+\tgit commit -m \"base txt\" &&\n+\n+\tgit rm subdir/file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 >file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 >file.md &&\n+\tgit add file.txt file.md &&\n+\tgit commit -a -m \"rename\" &&\n+\tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n+\t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n+\t# but since same basenames are checked first...\n+\tcat >expected <<-\\EOF &&\n+\tR088\tsubdir/file.txt\tfile.md\n+\tA\tfile.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"416730","messageId":"4fff9b1ff57b62587b1cbec2031f96199a214702.1613031350.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"[PATCH v4 2/6] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-11T08:15:45Z","receivedAt":"2021-02-11T08:17:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to make use of unique basenames to help inform rename detection,\nso that more likely pairings can be checked first. (src/moduleA/foo.txt\nand source/module/A/foo.txt are likely related if there are no other\n'foo.txt' files among the deleted and added files.) Add a new function,\nnot yet used, which creates a map of the unique basenames within\nrename_src and another within rename_dst, together with the indices\nwithin rename_src/rename_dst where those basenames show up. Non-unique\nbasenames still show up in the map, but have an invalid index (-1).\n\nThis function was inspired by the fact that in real world repositories,\nfiles are often moved across directories without changing names. Here\nare some sample repositories and the percentage of their historical\nrenames (as of early 2020) that preserved basenames:\n * linux: 76%\n * gcc: 64%\n * gecko: 79%\n * webkit: 89%\nThese statistics alone don't prove that an optimization in this area\nwill help or how much it will help, since there are also unpaired adds\nand deletes, restrictions on which basenames we consider, etc., but it\ncertainly motivated the idea to try something in this area.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 61 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 74930716e70d..3eb49a098adf 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,67 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+static const char *get_basename(const char *filename)\n+{\n+\t/*\n+\t * gitbasename() has to worry about special drivers, multiple\n+\t * directory separator characters, trailing slashes, NULL or\n+\t * empty strings, etc. We only work on filenames as stored in\n+\t * git, and thus get to ignore all those complications.\n+\t */\n+\tconst char *base = strrchr(filename, '/');\n+\treturn base ? base + 1 : filename;\n+}\n+\n+MAYBE_UNUSED\n+static int find_basename_matches(struct diff_options *options,\n+\t\t\t\t int minimum_score,\n+\t\t\t\t int num_src)\n+{\n+\tint i;\n+\tstruct strintmap sources;\n+\tstruct strintmap dests;\n+\n+\t/* Create maps of basename -> fullname(s) for sources and dests */\n+\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n+\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base;\n+\n+\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n+\t\tassert(!rename_src[i].p->one->rename_used);\n+\n+\t\t/* Record index within rename_src (i) if basename is unique */\n+\t\tbase = get_basename(filename);\n+\t\tif (strintmap_contains(&sources, base))\n+\t\t\tstrintmap_set(&sources, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&sources, base, i);\n+\t}\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\tconst char *base;\n+\n+\t\tif (rename_dst[i].is_rename)\n+\t\t\tcontinue; /* involved in exact match already. */\n+\n+\t\t/* Record index within rename_dst (i) if basename is unique */\n+\t\tbase = get_basename(filename);\n+\t\tif (strintmap_contains(&dests, base))\n+\t\t\tstrintmap_set(&dests, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&dests, base, i);\n+\t}\n+\n+\t/* TODO: Make use of basenames source and destination basenames */\n+\n+\tstrintmap_clear(&sources);\n+\tstrintmap_clear(&dests);\n+\n+\treturn 0;\n+}\n+\n #define NUM_CANDIDATE_PER_DST 4\n static void record_if_better(struct diff_score m[], struct diff_score *o)\n {\n-- \ngitgitgadget\n\n"},{"id":"416731","messageId":"dc26881e4ed3447c6efdcd492463be294f99b8c4.1613031350.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"[PATCH v4 3/6] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-11T08:15:46Z","receivedAt":"2021-02-11T08:17:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIt is not uncommon in real world repositories for the majority of file\nrenames to not change the basename of the file; i.e. most \"renames\" are\njust a move of files into different directories. We can make use of\nthis to avoid comparing all rename source candidates with all rename\ndestination candidates, by first comparing sources to destinations with\nthe same basenames. If two files with the same basename are\nsufficiently similar, we record the rename; if not, we include those\nfiles in the more exhaustive matrix comparison.\n\nThis means we are adding a set of preliminary additional comparisons,\nbut for each file we only compare it with at most one other file. For\nexample, if there was a include/media/device.h that was deleted and a\nsrc/module/media/device.h that was added, and there were no other\ndevice.h files added or deleted between the commits being compared,\nthen these two files would be compared in the preliminary step.\n\nThis commit does not yet actually employ this new optimization, it\nmerely adds a function which can be used for this purpose. The next\ncommit will do the necessary plumbing to make use of it.\n\nNote that this optimization might give us different results than without\nthe optimization, because it's possible that despite files with the same\nbasename being sufficiently similar to be considered a rename, there's\nan even better match between files without the same basename. I think\nthat is okay for four reasons: (1) it's easy to explain to the users\nwhat happened if it does ever occur (or even for them to intuitively\nfigure out), (2) as the next patch will show it provides such a large\nperformance boost that it's worth the tradeoff, and (3) it's somewhat\nunlikely that despite having unique matching basenames that other files\nserve as better matches. Reason (4) takes a full paragraph to\nexplain...\n\nIf the previous three reasons aren't enough, consider what rename\ndetection already does. Break detection is not the default, meaning\nthat if files have the same _fullname_, then they are considered related\neven if they are 0% similar. In fact, in such a case, we don't even\nbother comparing the files to see if they are similar let alone\ncomparing them to all other files to see what they are most similar to.\nBasically, we override content similarity based on sufficient filename\nsimilarity. Without the filename similarity (currently implemented as\nan exact match of filename), we swing the pendulum the opposite\ndirection and say that filename similarity is irrelevant and compare a\nfull N x M matrix of sources and destinations to find out which have the\nmost similar contents. This optimization just adds another form of\nfilename similarity comparison, but augments it with a file content\nsimilarity check as well. Basically, if two files have the same\nbasename and are sufficiently similar to be considered a rename, mark\nthem as such without comparing the two to all other rename candidates.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 95 +++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 92 insertions(+), 3 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 3eb49a098adf..001645624e71 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -384,10 +384,52 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n {\n-\tint i;\n+\t/*\n+\t * When I checked in early 2020, over 76% of file renames in linux\n+\t * just moved files to a different directory but kept the same\n+\t * basename. gcc did that with over 64% of renames, gecko did it\n+\t * with over 79%, and WebKit did it with over 89%.\n+\t *\n+\t * Therefore we can bypass the normal exhaustive NxM matrix\n+\t * comparison of similarities between all potential rename sources\n+\t * and destinations by instead using file basename as a hint (i.e.\n+\t * the portion of the filename after the last '/'), checking for\n+\t * similarity between files with the same basename, and if we find\n+\t * a pair that are sufficiently similar, record the rename pair and\n+\t * exclude those two from the NxM matrix.\n+\t *\n+\t * This *might* cause us to find a less than optimal pairing (if\n+\t * there is another file that we are even more similar to but has a\n+\t * different basename). Given the huge performance advantage\n+\t * basename matching provides, and given the frequency with which\n+\t * people use the same basename in real world projects, that's a\n+\t * trade-off we are willing to accept when doing just rename\n+\t * detection.\n+\t *\n+\t * If someone wants copy detection that implies they are willing to\n+\t * spend more cycles to find similarities between files, so it may\n+\t * be less likely that this heuristic is wanted. If someone is\n+\t * doing break detection, that means they do not want filename\n+\t * similarity to imply any form of content similiarity, and thus\n+\t * this heuristic would definitely be incompatible.\n+\t */\n+\n+\tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n \n+\t/*\n+\t * The prefeteching stuff wants to know if it can skip prefetching\n+\t * blobs that are unmodified...and will then do a little extra work\n+\t * to verify that the oids are indeed different before prefetching.\n+\t * Unmodified blobs are only relevant when doing copy detection;\n+\t * when limiting to rename detection, diffcore_rename[_extended]()\n+\t * will never be called with unmodified source paths fed to us, so\n+\t * the extra work necessary to check if rename_src entries are\n+\t * unmodified would be a small waste.\n+\t */\n+\tint skip_unmodified = 0;\n+\n \t/* Create maps of basename -> fullname(s) for sources and dests */\n \tstrintmap_init_with_options(&sources, -1, NULL, 0);\n \tstrintmap_init_with_options(&dests, -1, NULL, 0);\n@@ -420,12 +462,59 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\tstrintmap_set(&dests, base, i);\n \t}\n \n-\t/* TODO: Make use of basenames source and destination basenames */\n+\t/* Now look for basename matchups and do similarity estimation */\n+\tfor (i = 0; i < num_src; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base = NULL;\n+\t\tintptr_t src_index;\n+\t\tintptr_t dst_index;\n+\n+\t\t/* Find out if this basename is unique among sources */\n+\t\tbase = get_basename(filename);\n+\t\tsrc_index = strintmap_get(&sources, base);\n+\t\tif (src_index == -1)\n+\t\t\tcontinue; /* not a unique basename; skip it */\n+\t\tassert(src_index == i);\n+\n+\t\tif (strintmap_contains(&dests, base)) {\n+\t\t\tstruct diff_filespec *one, *two;\n+\t\t\tint score;\n+\n+\t\t\t/* Find out if this basename is unique among dests */\n+\t\t\tdst_index = strintmap_get(&dests, base);\n+\t\t\tif (dst_index == -1)\n+\t\t\t\tcontinue; /* not a unique basename; skip it */\n+\n+\t\t\t/* Ignore this dest if already used in a rename */\n+\t\t\tif (rename_dst[dst_index].is_rename)\n+\t\t\t\tcontinue; /* already used previously */\n+\n+\t\t\t/* Estimate the similarity */\n+\t\t\tone = rename_src[src_index].p->one;\n+\t\t\ttwo = rename_dst[dst_index].p->two;\n+\t\t\tscore = estimate_similarity(options->repo, one, two,\n+\t\t\t\t\t\t minimum_score, skip_unmodified);\n+\n+\t\t\t/* If sufficiently similar, record as rename pair */\n+\t\t\tif (score < minimum_score)\n+\t\t\t\tcontinue;\n+\t\t\trecord_rename_pair(dst_index, src_index, score);\n+\t\t\trenames++;\n+\n+\t\t\t/*\n+\t\t\t * Found a rename so don't need text anymore; if we\n+\t\t\t * didn't find a rename, the filespec_blob would get\n+\t\t\t * re-used when doing the matrix of comparisons.\n+\t\t\t */\n+\t\t\tdiff_free_filespec_blob(one);\n+\t\t\tdiff_free_filespec_blob(two);\n+\t\t}\n+\t}\n \n \tstrintmap_clear(&sources);\n \tstrintmap_clear(&dests);\n \n-\treturn 0;\n+\treturn renames;\n }\n \n #define NUM_CANDIDATE_PER_DST 4\n-- \ngitgitgadget\n\n"},{"id":"416732","messageId":"4e86ed3f29d46add81d5a17046e2c6a574a2fff3.1613031350.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"[PATCH v4 5/6] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-11T08:15:48Z","receivedAt":"2021-02-11T08:17:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe last few patches have introduced a new preliminary step when rename\ndetection is on but both break detection and copy detection are off.\nDocument this new step. While we're at it, add a testcase that checks\nthe new behavior as well.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex c970d9fe438a..edf92d988c8f 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -168,6 +168,26 @@ a similarity score different from the default of 50% by giving a\n number after the \"-M\" or \"-C\" option (e.g. \"-M8\" to tell it to use\n 8/10 = 80%).\n \n+Note that when rename detection is on but both copy and break\n+detection are off, rename detection adds a preliminary step that first\n+checks if files are moved across directories while keeping their\n+filename the same. If there is a file added to a directory whose\n+contents is sufficiently similar to a file with the same name that got\n+deleted from a different directory, it will mark them as renames and\n+exclude them from the later quadratic step (the one that pairwise\n+compares all unmatched files to find the \"best\" matches, determined by\n+the highest content similarity). So, for example, if a deleted\n+docs/ext.txt and an added docs/config/ext.txt are similar enough, they\n+will be marked as a rename and prevent an added docs/ext.md that may\n+be even more similar to the deleted docs/ext.txt from being considered\n+as the rename destination in the later step. For this reason, the\n+preliminary \"match same filename\" step uses a bit higher threshold to\n+mark a file pair as a rename and stop considering other candidates for\n+better matches. At most, one comparison is done per file in this\n+preliminary pass; so if there are several ext.txt files throughout the\n+directory hierarchy that were added and deleted, this preliminary step\n+will be skipped for those files.\n+\n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\n diffcore mechanism as well as modified ones. This lets the copy\n-- \ngitgitgadget\n\n"},{"id":"416733","messageId":"2493f4b2f55d2b602e448db3f09da1ee3a97c81b.1613031350.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"[PATCH v4 4/6] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-11T08:15:47Z","receivedAt":"2021-02-11T08:17:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMake use of the new find_basename_matches() function added in the last\ntwo patches, to find renames more rapidly in cases where we can match up\nfiles based on basenames. As a quick reminder (see the last two commit\nmessages for more details), this means for example that\ndocs/extensions.txt and docs/config/extensions.txt are considered likely\nrenames if there are no 'extensions.txt' files elsewhere among the added\nand deleted files, and if a similarity check confirms they are similar,\nthen they are marked as a rename without looking for a better similarity\nmatch among other files. This is a behavioral change, as covered in\nmore detail in the previous commit message.\n\nWe do not use this heuristic together with either break or copy\ndetection. The point of break detection is to say that filename\nsimilarity does not imply file content similarity, and we only want to\nknow about file content similarity. The point of copy detection is to\nuse more resources to check for additional similarities, while this is\nan optimization that uses far less resources but which might also result\nin finding slightly fewer similarities. So the idea behind this\noptimization goes against both of those features, and will be turned off\nfor both.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 13.815 s ± 0.062 s 13.294 s ± 0.103 s\n mega-renames: 1799.937 s ± 0.493 s 187.248 s ± 0.882 s\n just-one-mega: 51.289 s ± 0.019 s 5.557 s ± 0.017 s\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 54 ++++++++++++++++++++++++++++++++++++++----\n t/t4001-diff-rename.sh | 4 ++--\n 2 files changed, 51 insertions(+), 7 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 001645624e71..df76e475c710 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -379,7 +379,6 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n-MAYBE_UNUSED\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score,\n \t\t\t\t int num_src)\n@@ -727,12 +726,57 @@ void diffcore_rename(struct diff_options *options)\n \tif (minimum_score == MAX_SCORE)\n \t\tgoto cleanup;\n \n+\tnum_sources = rename_src_nr;\n+\n+\tif (want_copies || break_idx) {\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones corresponding to exact renames\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\t} else {\n+\t\t/* Determine minimum score to match basenames */\n+\t\tdouble factor = 0.5;\n+\t\tchar *basename_factor = getenv(\"GIT_BASENAME_FACTOR\");\n+\t\tint min_basename_score;\n+\n+\t\tif (basename_factor)\n+\t\t\tfactor = strtol(basename_factor, NULL, 10)/100.0;\n+\t\tassert(factor >= 0.0 && factor <= 1.0);\n+\t\tmin_basename_score = minimum_score +\n+\t\t\t(int)(factor * (MAX_SCORE - minimum_score));\n+\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones involved in renames (found via exact match)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\n+\t\t/* Utilize file basenames to quickly find renames. */\n+\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n+\t\trename_count += find_basename_matches(options,\n+\t\t\t\t\t\t min_basename_score,\n+\t\t\t\t\t\t rename_src_nr);\n+\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n+\n+\t\t/*\n+\t\t * Cull sources, again:\n+\t\t * - remove ones involved in renames (found via basenames)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull basename\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull basename\", options->repo);\n+\t}\n+\n \t/*\n-\t * Calculate how many renames are left\n+\t * Calculate how many rename destinations are left\n \t */\n \tnum_destinations = (rename_dst_nr - rename_count);\n-\tremove_unneeded_paths_from_src(want_copies);\n-\tnum_sources = rename_src_nr;\n+\tnum_sources = rename_src_nr; /* rename_src_nr reflects lower number */\n \n \t/* All done? */\n \tif (!num_destinations || !num_sources)\n@@ -764,7 +808,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tstruct diff_score *m;\n \n \t\tif (rename_dst[i].is_rename)\n-\t\t\tcontinue; /* dealt with exact match already. */\n+\t\t\tcontinue; /* exact or basename match already handled */\n \n \t\tm = &mx[dst_cnt * NUM_CANDIDATE_PER_DST];\n \t\tfor (j = 0; j < NUM_CANDIDATE_PER_DST; j++)\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex 797343b38106..bf62537c29a0 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -280,8 +280,8 @@ test_expect_success 'basename similarity vs best similarity' '\n \t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n \t# but since same basenames are checked first...\n \tcat >expected <<-\\EOF &&\n-\tR088\tsubdir/file.txt\tfile.md\n-\tA\tfile.txt\n+\tA\tfile.md\n+\tR078\tsubdir/file.txt\tfile.txt\n \tEOF\n \ttest_cmp expected actual\n '\n-- \ngitgitgadget\n\n"},{"id":"416734","messageId":"fedb3d323d948dfdc6e40e036563fd56983f0ad6.1613031350.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"[PATCH v4 6/6] merge-ort: call diffcore_rename() directly","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-11T08:15:49Z","receivedAt":"2021-02-11T08:17:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to pass additional information to diffcore_rename() (or some\nvariant thereof) without plumbing that extra information through\ndiff_tree_oid() and diffcore_std(). Further, since we will need to\ngather additional special information related to diffs and are walking\nthe trees anyway in collect_merge_info(), it seems odd to have\ndiff_tree_oid()/diffcore_std() repeat those tree walks. And there may\nbe times where we can avoid traversing into a subtree in\ncollect_merge_info() (based on additional information at our disposal),\nthat the basic diff logic would be unable to take advantage of. For all\nthese reasons, just create the add and delete pairs ourself and then\ncall diffcore_rename() directly.\n\nThis change is primarily about enabling future optimizations; the\nadvantage of avoiding extra tree traversals is small compared to the\ncost of rename detection, and the advantage of avoiding the extra tree\ntraversals is somewhat offset by the extra time spent in\ncollect_merge_info() collecting the additional data anyway. However...\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 13.294 s ± 0.103 s 12.775 s ± 0.062 s\n mega-renames: 187.248 s ± 0.882 s 188.754 s ± 0.284 s\n just-one-mega: 5.557 s ± 0.017 s 5.599 s ± 0.019 s\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 59 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 931b91438cf1..603d30c52170 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -535,6 +535,23 @@ static void setup_path_info(struct merge_options *opt,\n \tresult->util = mi;\n }\n \n+static void add_pair(struct merge_options *opt,\n+\t\t struct name_entry *names,\n+\t\t const char *pathname,\n+\t\t unsigned side,\n+\t\t unsigned is_add /* if false, is_delete */)\n+{\n+\tstruct diff_filespec *one, *two;\n+\tstruct rename_info *renames = &opt->priv->renames;\n+\tint names_idx = is_add ? side : 0;\n+\n+\tone = alloc_filespec(pathname);\n+\ttwo = alloc_filespec(pathname);\n+\tfill_filespec(is_add ? two : one,\n+\t\t &names[names_idx].oid, 1, names[names_idx].mode);\n+\tdiff_queue(&renames->pairs[side], one, two);\n+}\n+\n static void collect_rename_info(struct merge_options *opt,\n \t\t\t\tstruct name_entry *names,\n \t\t\t\tconst char *dirname,\n@@ -544,6 +561,7 @@ static void collect_rename_info(struct merge_options *opt,\n \t\t\t\tunsigned match_mask)\n {\n \tstruct rename_info *renames = &opt->priv->renames;\n+\tunsigned side;\n \n \t/* Update dirs_removed, as needed */\n \tif (dirmask == 1 || dirmask == 3 || dirmask == 5) {\n@@ -554,6 +572,21 @@ static void collect_rename_info(struct merge_options *opt,\n \t\tif (sides & 2)\n \t\t\tstrset_add(&renames->dirs_removed[2], fullname);\n \t}\n+\n+\tif (filemask == 0 || filemask == 7)\n+\t\treturn;\n+\n+\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; ++side) {\n+\t\tunsigned side_mask = (1 << side);\n+\n+\t\t/* Check for deletion on side */\n+\t\tif ((filemask & 1) && !(filemask & side_mask))\n+\t\t\tadd_pair(opt, names, fullname, side, 0 /* delete */);\n+\n+\t\t/* Check for addition on side */\n+\t\tif (!(filemask & 1) && (filemask & side_mask))\n+\t\t\tadd_pair(opt, names, fullname, side, 1 /* add */);\n+\t}\n }\n \n static int collect_merge_info_callback(int n,\n@@ -2079,6 +2112,27 @@ static int process_renames(struct merge_options *opt,\n \treturn clean_merge;\n }\n \n+static void resolve_diffpair_statuses(struct diff_queue_struct *q)\n+{\n+\t/*\n+\t * A simplified version of diff_resolve_rename_copy(); would probably\n+\t * just use that function but it's static...\n+\t */\n+\tint i;\n+\tstruct diff_filepair *p;\n+\n+\tfor (i = 0; i < q->nr; ++i) {\n+\t\tp = q->queue[i];\n+\t\tp->status = 0; /* undecided */\n+\t\tif (!DIFF_FILE_VALID(p->one))\n+\t\t\tp->status = DIFF_STATUS_ADDED;\n+\t\telse if (!DIFF_FILE_VALID(p->two))\n+\t\t\tp->status = DIFF_STATUS_DELETED;\n+\t\telse if (DIFF_PAIR_RENAME(p))\n+\t\t\tp->status = DIFF_STATUS_RENAMED;\n+\t}\n+}\n+\n static int compare_pairs(const void *a_, const void *b_)\n {\n \tconst struct diff_filepair *a = *((const struct diff_filepair **)a_);\n@@ -2089,8 +2143,6 @@ static int compare_pairs(const void *a_, const void *b_)\n \n /* Call diffcore_rename() to compute which files have changed on given side */\n static void detect_regular_renames(struct merge_options *opt,\n-\t\t\t\t struct tree *merge_base,\n-\t\t\t\t struct tree *side,\n \t\t\t\t unsigned side_index)\n {\n \tstruct diff_options diff_opts;\n@@ -2108,11 +2160,11 @@ static void detect_regular_renames(struct merge_options *opt,\n \tdiff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;\n \tdiff_setup_done(&diff_opts);\n \n+\tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n-\tdiff_tree_oid(&merge_base->object.oid, &side->object.oid, \"\",\n-\t\t &diff_opts);\n-\tdiffcore_std(&diff_opts);\n+\tdiffcore_rename(&diff_opts);\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n+\tresolve_diffpair_statuses(&diff_queued_diff);\n \n \tif (diff_opts.needed_rename_limit > renames->needed_limit)\n \t\trenames->needed_limit = diff_opts.needed_rename_limit;\n@@ -2212,8 +2264,8 @@ static int detect_and_process_renames(struct merge_options *opt,\n \tmemset(&combined, 0, sizeof(combined));\n \n \ttrace2_region_enter(\"merge\", \"regular renames\", opt->repo);\n-\tdetect_regular_renames(opt, merge_base, side1, MERGE_SIDE1);\n-\tdetect_regular_renames(opt, merge_base, side2, MERGE_SIDE2);\n+\tdetect_regular_renames(opt, MERGE_SIDE1);\n+\tdetect_regular_renames(opt, MERGE_SIDE2);\n \ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n \n \ttrace2_region_enter(\"merge\", \"directory renames\", opt->repo);\n-- \ngitgitgadget\n"},{"id":"416822","messageId":"xmqq4kigvkct.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"3e6af929d135ef2dc239e2f47f92a7e2e91cbd17.1612970140.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/5] t4001: add a test comparing basename similarity and content similarity","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-13T01:15:30Z","receivedAt":"2021-02-13T01:16:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> From: Elijah Newren \n>\n> Add a simple test where a removed file is similar to two different added\n> files; one of them has the same basename, and the other has a slightly\n> higher content similarity. Without break detection, filename similarity\n> of 100% trumps content similarity for pairing up related files. For\n> any filename similarity less than 100%, the opposite is true -- content\n> similarity is all that matters. Add a testcase that documents this.\n\nI am not sure why it is the \"opposite\". When contents are similar\nto the same degree of 100%, we tiebreak with the filename. We never\nfavor a pair between the same filename over a pair between different\nfilenames with better content similarity.\n\nAnd when contents are similar to the same degree of less than 100%,\nwe do not favor a pair between the same filename over a pair between\ndifferent filenames, as long as they are similar to the same degree.\n\nSo, I do not think \"opposite\" is helping readers to understand what\nis going on.\n\n> +test_expect_success 'basename similarity vs best similarity' '\n> +\tmkdir subdir &&\n> +\ttest_write_lines line1 line2 line3 line4 line5 \\\n> +\t\t\t line6 line7 line8 line9 line10 >subdir/file.txt &&\n> +\tgit add subdir/file.txt &&\n> +\tgit commit -m \"base txt\" &&\n> +\n> +\tgit rm subdir/file.txt &&\n> +\ttest_write_lines line1 line2 line3 line4 line5 \\\n> +\t\t\t line6 line7 line8 >file.txt &&\n> +\ttest_write_lines line1 line2 line3 line4 line5 \\\n> +\t\t\t line6 line7 line8 line9 >file.md &&\n> +\tgit add file.txt file.md &&\n> +\tgit commit -a -m \"rename\" &&\n> +\tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n> +\t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n> +\t# but since same basenames are checked first...\n\nI am not sure what the second line of this comment wants to imply\nwith the ellipses here. Care to finish the sentence?\n\nOr was the second line planned to be added when we start applying\nthe \"check only the same filename first and see if we find a\nbetter-than-reasonable match\" heuristics but somehow survived\n\"rebase -i\" and ended up here?\n\n> +\tcat >expected <<-\\EOF &&\n> +\tR088\tsubdir/file.txt\tfile.md\n> +\tA\tfile.txt\n> +\tEOF\n> +\ttest_cmp expected actual\n\nThanks.\n"},{"id":"416824","messageId":"xmqqy2fsu4z8.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"4fff9b1ff57b62587b1cbec2031f96199a214702.1612970140.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/5] diffcore-rename: compute basenames of all source and dest candidates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-13T01:32:59Z","receivedAt":"2021-02-13T01:34:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> +MAYBE_UNUSED\n> +static int find_basename_matches(struct diff_options *options,\n> +\t\t\t\t int minimum_score,\n> +\t\t\t\t int num_src)\n> +{\n> +\tint i;\n> +\tstruct strintmap sources;\n> +\tstruct strintmap dests;\n> +\n> +\t/* Create maps of basename -> fullname(s) for sources and dests */\n> +\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n> +\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n> +\tfor (i = 0; i < num_src; ++i) {\n> +\t\tchar *filename = rename_src[i].p->one->path;\n> +\t\tconst char *base;\n> +\n> +\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n> +\t\tassert(!rename_src[i].p->one->rename_used);\n> +\n> +\t\t/* Record index within rename_src (i) if basename is unique */\n> +\t\tbase = get_basename(filename);\n> +\t\tif (strintmap_contains(&sources, base))\n> +\t\t\tstrintmap_set(&sources, base, -1);\n> +\t\telse\n> +\t\t\tstrintmap_set(&sources, base, i);\n> +\t}\n> +\tfor (i = 0; i < rename_dst_nr; ++i) {\n> +\t\tchar *filename = rename_dst[i].p->two->path;\n> +\t\tconst char *base;\n> +\n> +\t\tif (rename_dst[i].is_rename)\n> +\t\t\tcontinue; /* involved in exact match already. */\n> +\n> +\t\t/* Record index within rename_dst (i) if basename is unique */\n> +\t\tbase = get_basename(filename);\n> +\t\tif (strintmap_contains(&dests, base))\n> +\t\t\tstrintmap_set(&dests, base, -1);\n> +\t\telse\n> +\t\t\tstrintmap_set(&dests, base, i);\n> +\t}\n> +\n> +\t/* TODO: Make use of basenames source and destination basenames */\n\n;-)\n\nSo at this point sources and dests can be used to quickly look up,\ngiven a filename, if there is a single src among all sources, and a\nsingle dst among all dests, that have the filename.\n\nI wonder if the second loop over destinations can be \"optimized\"\nfurther by using the sources map, though. The reason you quash\nentries with -1 when you see second instance of the same name is\nbecause you intend to limit the heuristics only to a uniquely named\nfile among the removed files going to a uniquely named file among\nthe added files, right? So even if a name is unique among dests,\nif that name has duplicates on the source side, there is no point\nrecording its location. i.e.\n\n\t/* record index within dst if it is unique in both dst and src */\n\tbase = get_basename(filename);\n\tif (strintmap_contains(&sources, base) ||\n\t strintmap_contains(&dests, base))\n\t\tstrintmap_set(&dests, base, -1);\n\telse\n\t\tstrintmap_set(&dests, base, i);\n\nperhaps?\n\nI guess it depends on what actually will be written in this \"TODO\"\nspace how effective such a change would be. Presumably, you'd\niterate over &sources while skipping entries that record -1, to\nlearn (basename, i), and use the basename found there to consult\n&dests to see if it yields a non-negative integer j, to notice that\nrename_src[i] is a good candidate to match rename_dst[j]. If that\nis the case, then such a change won't help as an optimization at\nall, as we'd need to consult &dests map with the basename anyway,\nso let's scratch the above idea.\n\nIn any case, after we walk over rename_src[] and rename_dst[] once,\nthe number of entries in &sources would be smaller than rename_src[]\nso iterating over &sources, hunting for entries that record\nnon-negative index into rename_src[] would hopefully be cheaper than\nthe naive loop we've been working with. I like the idea of using the\nstrintmap for this part of the code.\n\nThanks.\n\n\n> +\tstrintmap_clear(&sources);\n> +\tstrintmap_clear(&dests);\n> +\n> +\treturn 0;\n> +}\n> +\n> #define NUM_CANDIDATE_PER_DST 4\n> static void record_if_better(struct diff_score m[], struct diff_score *o)\n> {\n"},{"id":"416825","messageId":"xmqqsg60u49i.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"dc26881e4ed3447c6efdcd492463be294f99b8c4.1612970140.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/5] diffcore-rename: complete find_basename_matches()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-13T01:48:25Z","receivedAt":"2021-02-13T01:49:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> +\t/* Now look for basename matchups and do similarity estimation */\n> +\tfor (i = 0; i < num_src; ++i) {\n> +\t\tchar *filename = rename_src[i].p->one->path;\n> +\t\tconst char *base = NULL;\n> +\t\tintptr_t src_index;\n> +\t\tintptr_t dst_index;\n> +\n> +\t\t/* Find out if this basename is unique among sources */\n> +\t\tbase = get_basename(filename);\n> +\t\tsrc_index = strintmap_get(&sources, base);\n> +\t\tif (src_index == -1)\n> +\t\t\tcontinue; /* not a unique basename; skip it */\n> +\t\tassert(src_index == i);\n> +\n> +\t\tif (strintmap_contains(&dests, base)) {\n> +\t\t\tstruct diff_filespec *one, *two;\n> +\t\t\tint score;\n> +\n> +\t\t\t/* Find out if this basename is unique among dests */\n> +\t\t\tdst_index = strintmap_get(&dests, base);\n> +\t\t\tif (dst_index == -1)\n> +\t\t\t\tcontinue; /* not a unique basename; skip it */\n\nIt would be a lot easier to read if \"we must have the same singleton\nin dests\" in a single if condition, I suspect. I.e.\n\n\t\tif (strintmap_contains(&dests, base) &&\n\t\t 0 <= (dst_index = (strintmap_get(&dests, base)))) {\n\nIt is a bit sad that we iterate over rename_src[] array, even though\nwe now have a map that presumably have fewer number of entries than\nthe original array, though.\n\n> +\t\t\t/* Ignore this dest if already used in a rename */\n> +\t\t\tif (rename_dst[dst_index].is_rename)\n> +\t\t\t\tcontinue; /* already used previously */\n\nSince we will only be matching between unique entries in src and\ndst, this \"this has been used, so we cannot use it\" will not change\nduring this loop. I wonder if the preparation done in the previous\nstep, i.e. [PATCH v3 2/5], can take advantage of this fact, i.e. a\ndst that has already been used (in the previous \"exact\" step) would\nnot even have to be in &dests map, so that the strintmap_contains()\ncheck can reject it much earlier.\n\nStepping back a bit, it appears to me that [2/5] and [3/5] considers\na source file having unique basename among the sources even if there\nare many such files with the same basename, as long as all the other\nfiles with the same basename have been matched in the previous\n\"exact\" phase. It probably does the same thing for destination\nside.\n\nIntended?\n\nIt feels incompatible with the spirit of these two steps aim for\n(i.e. only use this optimization on a pair of src/dst with UNIQUE\nbasenames). For the purpose of \"we only handle unique ones\", the\npaths that already have matched should participate in deciding if\nthe files that survived \"exact\" phase have unique basename among\nthe original inpu?\n\nThanks.\n\n"},{"id":"416826","messageId":"xmqqo8gou47m.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"2493f4b2f55d2b602e448db3f09da1ee3a97c81b.1612970140.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/5] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-13T01:49:33Z","receivedAt":"2021-02-13T01:50:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> diff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\n> index 797343b38106..bf62537c29a0 100755\n> --- a/t/t4001-diff-rename.sh\n> +++ b/t/t4001-diff-rename.sh\n> @@ -280,8 +280,8 @@ test_expect_success 'basename similarity vs best similarity' '\n> \t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n> \t# but since same basenames are checked first...\n\nHere lies the answer to my earlier question ;-)\n\n> \tcat >expected <<-\\EOF &&\n> -\tR088\tsubdir/file.txt\tfile.md\n> -\tA\tfile.txt\n> +\tA\tfile.md\n> +\tR078\tsubdir/file.txt\tfile.txt\n> \tEOF\n> \ttest_cmp expected actual\n> '\n"},{"id":"416832","messageId":"xmqqk0rcu41l.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/6] Optimization batch 7: use file basenames to guide rename detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-13T01:53:10Z","receivedAt":"2021-02-13T01:53:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> This series depends on ort-perf-batch-6[1].\n>\n> This series uses file basenames (portion of the path after final '/',\n> including extension) in a basic fashion to guide rename detection.\n>\n> Changes since v3:\n>\n> * update documentation as suggested by Junio\n> * NEW: add another patch at the end, to simplify patch series that will be\n> submitted later (please review!)\n\nSorry, by mistake I somehow read v4 and sent some comments on v3,\nbut as the above says, they are on the part that hadn't changed at\nall, and should still be relevant.\n\nThanks.\n\n"},{"id":"416835","messageId":"CABPp-BHyYOAw939vU20oA_tOSFRVm=EaLKuw+4M1oP1GDDV5cg@mail.gmail.com","threadId":"55112","inReplyTo":"xmqq4kigvkct.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 1/5] t4001: add a test comparing basename similarity and content similarity","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-13T04:50:07Z","receivedAt":"2021-02-13T04:51:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 12, 2021 at 5:15 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > From: Elijah Newren \n> >\n> > Add a simple test where a removed file is similar to two different added\n> > files; one of them has the same basename, and the other has a slightly\n> > higher content similarity. Without break detection, filename similarity\n> > of 100% trumps content similarity for pairing up related files. For\n> > any filename similarity less than 100%, the opposite is true -- content\n> > similarity is all that matters. Add a testcase that documents this.\n>\n> I am not sure why it is the \"opposite\". When contents are similar\n> to the same degree of 100%, we tiebreak with the filename. We never\n> favor a pair between the same filename over a pair between different\n> filenames with better content similarity.\n\nThis is not true. If src/main.c is 99% similar to src/foo.c, and is\n0% similar to the src/main.c in the new commit, we match the old\nsrc/main.c to the new src/main.c despite being far more similar\nsrc/foo.c. Unless break detection is turned on, we do not allow\ncontent similarity to trump (full) filename equality.\n\n> And when contents are similar to the same degree of less than 100%,\n> we do not favor a pair between the same filename over a pair between\n> different filenames, as long as they are similar to the same degree.\n\nThis is also not true; we tiebreak with filenames for inexact renames\njust like we do for exact renames (note that basename_same() is called\nboth from find_identical_files() and from the nested loop where\ninexact rename detection is done).\n\n> So, I do not think \"opposite\" is helping readers to understand what\n> is going on.\n>\n> > +test_expect_success 'basename similarity vs best similarity' '\n> > + mkdir subdir &&\n> > + test_write_lines line1 line2 line3 line4 line5 \\\n> > + line6 line7 line8 line9 line10 >subdir/file.txt &&\n> > + git add subdir/file.txt &&\n> > + git commit -m \"base txt\" &&\n> > +\n> > + git rm subdir/file.txt &&\n> > + test_write_lines line1 line2 line3 line4 line5 \\\n> > + line6 line7 line8 >file.txt &&\n> > + test_write_lines line1 line2 line3 line4 line5 \\\n> > + line6 line7 line8 line9 >file.md &&\n> > + git add file.txt file.md &&\n> > + git commit -a -m \"rename\" &&\n> > + git diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n> > + # subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n> > + # but since same basenames are checked first...\n>\n> I am not sure what the second line of this comment wants to imply\n> with the ellipses here. Care to finish the sentence?\n>\n> Or was the second line planned to be added when we start applying\n> the \"check only the same filename first and see if we find a\n> better-than-reasonable match\" heuristics but somehow survived\n> \"rebase -i\" and ended up here?\n\nOops, indeed; that is precisely what happened. Will fix.\n\n> > + cat >expected <<-\\EOF &&\n> > + R088 subdir/file.txt file.md\n> > + A file.txt\n> > + EOF\n> > + test_cmp expected actual\n>\n> Thanks.\n"},{"id":"416844","messageId":"CABPp-BF_cj1GEYT75aj9funUk1mYvtM2OvKiNb1JAZzA6E5mgQ@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqsg60u49i.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 3/5] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-13T18:34:50Z","receivedAt":"2021-02-13T18:36:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 12, 2021 at 5:48 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > + /* Now look for basename matchups and do similarity estimation */\n> > + for (i = 0; i < num_src; ++i) {\n> > + char *filename = rename_src[i].p->one->path;\n> > + const char *base = NULL;\n> > + intptr_t src_index;\n> > + intptr_t dst_index;\n> > +\n> > + /* Find out if this basename is unique among sources */\n> > + base = get_basename(filename);\n> > + src_index = strintmap_get(&sources, base);\n> > + if (src_index == -1)\n> > + continue; /* not a unique basename; skip it */\n> > + assert(src_index == i);\n> > +\n> > + if (strintmap_contains(&dests, base)) {\n> > + struct diff_filespec *one, *two;\n> > + int score;\n> > +\n> > + /* Find out if this basename is unique among dests */\n> > + dst_index = strintmap_get(&dests, base);\n> > + if (dst_index == -1)\n> > + continue; /* not a unique basename; skip it */\n>\n> It would be a lot easier to read if \"we must have the same singleton\n> in dests\" in a single if condition, I suspect. I.e.\n>\n> if (strintmap_contains(&dests, base) &&\n> 0 <= (dst_index = (strintmap_get(&dests, base)))) {\n\nI can change that. I can also simplify it further to\n\n if (0 <= (dst_index = (strintmap_get(&dests, base)))) {\n\nsince dests uses a default value of -1. That will decrease the number\nof strmap lookups here from 2 to 1.\n\n> It is a bit sad that we iterate over rename_src[] array, even though\n> we now have a map that presumably have fewer number of entries than\n> the original array, though.\n\nOh, interesting; I forgot all about that. I just looked up my\noriginal implementation from February of last year and indeed I had\ndone exactly that\n(https://github.com/newren/git/commit/43eaec6007c92b6af05e0ef0fcc047c1d1ba1de8).\nHowever, when I added a later optimization that pairs up non-unique\nbasenames, I had to switch to looping over rename_src.\n\nFor various reasons (mostly starting with the fact that I had lots of\nexperimental ideas that were tried and thrown out but with pieces kept\naround for ideas), I wasn't even close to having a clean history in my\noriginal implementation of merge-ort and the diffcore-rename\noptimizations. And it was far, far easier to achieve the goal of a\nclean history by picking out chunks of code from the end-state and\ncreating entirely new commits than attempting to use my existing\nhistory. But, of course, that method made me lose this intermediate\nstate.\n\n>\n> > + /* Ignore this dest if already used in a rename */\n> > + if (rename_dst[dst_index].is_rename)\n> > + continue; /* already used previously */\n>\n> Since we will only be matching between unique entries in src and\n> dst, this \"this has been used, so we cannot use it\" will not change\n> during this loop. I wonder if the preparation done in the previous\n> step, i.e. [PATCH v3 2/5], can take advantage of this fact, i.e. a\n> dst that has already been used (in the previous \"exact\" step) would\n> not even have to be in &dests map, so that the strintmap_contains()\n> check can reject it much earlier.\n\nGood, catch again. The previous step (v4 2/5) actually did already\ncheck this, so this if-condition will always be false at this point.\nLooking at the link above, this if-condition check wasn't there in the\noriginal, but again was added due to altered state introduced by a\nlater optimization. So, I should pull this check out of this patch\nand add it back in to the later patch.\n\n> Stepping back a bit, it appears to me that [2/5] and [3/5] considers\n> a source file having unique basename among the sources even if there\n> are many such files with the same basename, as long as all the other\n> files with the same basename have been matched in the previous\n> \"exact\" phase. It probably does the same thing for destination\n> side.\n>\n> Intended?\n>\n> It feels incompatible with the spirit of these two steps aim for\n> (i.e. only use this optimization on a pair of src/dst with UNIQUE\n> basenames). For the purpose of \"we only handle unique ones\", the\n> paths that already have matched should participate in deciding if\n> the files that survived \"exact\" phase have unique basename among\n> the original inpu?\n\nYeah, I should have been more careful with my wording. Stated a\ndifferent way, what confidence can we associate with an exact rename?\nObviously, the confidence is high as we mark them as renames. But if\nthe confidence is less than 100%, and enough less than 100% that it\ncasts a doubt on \"related\" inexact renames, then yes the basenames of\nthe exact renames should also be computed so that we can determine\nwhat basenames are truly unique. By the exact same argument, you\ncould take this a step further and say that we should calculate the\nbasenames of *all* files in the tree, not just add/delete pairs, and\nonly match up the ones via basename that are *truly* unique. After\nall, break detection exists, so perhaps we don't have full confidence\nthat files with an unchanged fullname are actually related.\n\nFrom my view, though, both are too cautious and throwing out valuable\nheuristics for common cases. Starting with break detection, it is off\nfor a reason: we think unchanged filename is a strong enough heuristic\nto just match up those files and consider the confidence of the match\nin effect 100%. Similarly, we put a lot of confidence in exact rename\ndetection. If there are multiple adds/deletes with the same basename,\nand all but one on each side are paired up by exact rename detection,\naren't the remaining two files a (very) likely rename pair? I think\nso, and believe they're worth including in the basename-based rename\ndetection step. We do require basename-based matches to meet a much\nhigher similarity scoring threshold now, which I feel already\nadequately adjusts for not doing full content similarity against all\nother files.\n\nAlso, in the next series, I find an additional way to match up files\nby basename when basenames are not unique, and which doesn't involve\npairwise comparing all the files with the same basename. I only pick\nat most one other file to compare to (and the selection is not\nrandom). So, my overall strategy for these two series is \"find which\nbasenames are likely matches\" even if I didn't word it very well.\n\nI do agree, though, that I should add some more careful wording about\nthis in the series. I'll include it in a re-roll.\n"},{"id":"416846","messageId":"xmqqft1zttev.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"CABPp-BF_cj1GEYT75aj9funUk1mYvtM2OvKiNb1JAZzA6E5mgQ@mail.gmail.com","subject":"Re: [PATCH v3 3/5] diffcore-rename: complete find_basename_matches()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-13T23:55:04Z","receivedAt":"2021-02-13T23:56:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> I can change that. I can also simplify it further to\n>\n> if (0 <= (dst_index = (strintmap_get(&dests, base)))) {\n>\n> since dests uses a default value of -1. That will decrease the number\n> of strmap lookups here from 2 to 1.\n\nWhich would be a real win, unlike what I said in the message you are\nresponding to.\n\n>> It feels incompatible with the spirit of these two steps aim for\n>> (i.e. only use this optimization on a pair of src/dst with UNIQUE\n>> basenames). For the purpose of \"we only handle unique ones\", the\n>> paths that already have matched should participate in deciding if\n>> the files that survived \"exact\" phase have unique basename among\n>> the original inpu?\n>\n> Yeah, I should have been more careful with my wording. Stated a\n> different way, what confidence can we associate with an exact rename?\n\nSuppose you start with a/Makefile, b/Makefile and c/Makefile and\nthen they all disappeared while a1/Makefile, b1/Makefile and\nc1/Makefile now are in the tree. The contents a/Makefile used to\nhave appears without any difference in a1/Makefile, the same for b\nand b1, but c/Makefile and c1/Makefile are different. The c vs c1\npair may worth investigating, so it goes through the \"same basename\"\nphase.\n\nNow, in a slightly different situation, a vs a1 are still identical,\nbut b vs b1 have only one blank line removal but without any other\nchange. It looks odd that such a change has to pessimize c vs c1\noptimization opportunity, but an interesting part of the story is\nthat we can only say \"such a change\", not \"such a miniscule change\",\nbecause we have just finished the \"exact\" phase, and we do not know\nhow big a difference b vs b1 pair actually had.\n\nThat makes me feel that this whole \"we must treat unique one that\nremains specially\" is being incoherent. If \"because we have only\nsmall number of removed and added Makefiles spread across the trees,\nfirst full-matrix matching among them without anything else with\nhigher bar may be worth an optimization\" were the optimization, then\nI would understand and support the design to omit those that have\nalready been matched in the \"exact\" phase.\n\nBut IIRC, limiting this \"same basename\" phase to unique add/del pair\nwas sold as a way to make it less likely for the heuristics to make\nmistakes, yet the definition of \"unique\", as shown above, is not all\nthat solid. That I find it rather unsatisfactory.\n\nIn other words, it is not \"what confidence do we have in exact\nphase?\" \"exact\" matching may have found perfect matching pair. But\nthe found pair should be happy just between themselves, and should\nnot have undue effect on how _other_ pairs are compared. Stopping\nthe \"exact\" pair from participating in the \"uniqueness\" definition\nis placing \"exact\" phase too much weight to affect how other filepairs\nare found.\n\n> By the exact same argument, you\n> could take this a step further and say that we should calculate the\n> basenames of *all* files in the tree, not just add/delete pairs, and\n> only match up the ones via basename that are *truly* unique. After\n> all, break detection exists, so perhaps we don't have full confidence\n> that files with an unchanged fullname are actually related.\n\nSorry, but you are not making sense. These optimizations are done\nonly when we are not using copies and breaks, no? What _other_\nchanges that kept the paths the same, or modified in place, have any\neffect on matching added and deleted pairs?\n"},{"id":"416847","messageId":"xmqqblcnttcr.fsf@gitster.c.googlers.com","threadId":"55112","inReplyTo":"CABPp-BHyYOAw939vU20oA_tOSFRVm=EaLKuw+4M1oP1GDDV5cg@mail.gmail.com","subject":"Re: [PATCH v3 1/5] t4001: add a test comparing basename similarity and content similarity","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-13T23:56:20Z","receivedAt":"2021-02-13T23:57:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> This is not true. If src/main.c is 99% similar to src/foo.c, and is\n> 0% similar to the src/main.c in the new commit, we match the old\n> src/main.c to the new src/main.c despite being far more similar\n> src/foo.c. Unless break detection is turned on, we do not allow\n> content similarity to trump (full) filename equality.\n\nAbsolutely. And we are talking about a new optimization that kicks\nin only when there is no break or no copy detection going on, no?\n"},{"id":"416848","messageId":"CABPp-BErByO=6jE+tZQHSsGJacr92AqVL7dnctE9MjQzmeQfcg@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqblcnttcr.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 1/5] t4001: add a test comparing basename similarity and content similarity","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-14T01:24:50Z","receivedAt":"2021-02-14T01:25:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Feb 13, 2021 at 3:56 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > This is not true. If src/main.c is 99% similar to src/foo.c, and is\n> > 0% similar to the src/main.c in the new commit, we match the old\n> > src/main.c to the new src/main.c despite being far more similar\n> > src/foo.c. Unless break detection is turned on, we do not allow\n> > content similarity to trump (full) filename equality.\n>\n> Absolutely. And we are talking about a new optimization that kicks\n> in only when there is no break or no copy detection going on, no?\n\nYes, precisely, we are only considering cases without break\ndetection...and thus we are considering cases where for the last 15\nyears or more, sufficiently large filename similarity (an exact\nfullname match) trumps any level of content similarity. I think it is\nuseful to note that while my optimization is adding more\nconsiderations that can overrule maximal content similarity, it is not\nthe first such code choice to do that.\n\nBut let me back up a bit...\n\nWhen I submitted the series, you and Stolee went into a long\ndiscussion about an optimization that I didn't submit, one that feels\nlooser on \"matching\" than anything I submitted, and which I think\nmight counter-intuitively reduce performance rather than aid it. (The\nperformance side only comes into view in combination with later\nseries, but it was why I harped so much since then on only comparing\nagainst at most one other file in the steps before full inexact rename\ndetection.) I was quite surprised by the diversion, but it made it\nclear to me that my descriptions and commit messages were far too\nvague and could be read to imply a completely different algorithm than\nI intended. So, I tried to be far more careful in subsequent\niterations by adding wider context and contrasts.\n\nFurther, after I wrote various things to try to clarify the\nmisunderstandings, I noticed that Stolee picked out one thing and\nstated that \"This idea of optimizing first for 100% filename\nsimilarity is a good perspective on Git's rename detection algorithm.\"\n(see https://lore.kernel.org/git/57d30e7d-7727-8d98-e3ef-bcfeebf9edd3@gmail.com/)\n So, that particular point seemed to help him understand more, and\nthus might be useful extra context for others reading along now or in\nthe future.\n\nGiven all the above, I was trying to address earlier misunderstandings\nand provide more context. Perhaps I swung the pendulum too far and\ntalked too much about other cases, or perhaps I just worded things\npoorly again. All I was attempting to do in the commit message was\npoint out the multiple basic rules with filename and content\nsimilarity, to lay the groundwork for new rules that do alternative\nweightings.\n\nAnyway, I've added a few more tweaks to try to improve the wording for\nthe next round I'll submit today. Given my track record so far, it\nwould not be surprising if it still needed more tweaks.\n"},{"id":"416849","messageId":"CAPc5daVVNaBiWt8vRv3xrXd7GvNPwRGRF=W31DUJCnZRo+9qtA@mail.gmail.com","threadId":"55112","inReplyTo":"CABPp-BErByO=6jE+tZQHSsGJacr92AqVL7dnctE9MjQzmeQfcg@mail.gmail.com","subject":"Re: [PATCH v3 1/5] t4001: add a test comparing basename similarity and content similarity","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-14T01:32:40Z","receivedAt":"2021-02-14T01:33:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I do not consider \"the same file changed in place\" the same as \"we\nseem to have lost a file in the old tree, ah, we found one that has\nthe same basename in a different directory\" at all, so your argument\nstill does not make any sense to me, sorry.\n\n2021年2月13日(土) 17:25 Elijah Newren :\n>\n> On Sat, Feb 13, 2021 at 3:56 PM Junio C Hamano wrote:\n> >\n> > Elijah Newren writes:\n> >\n> > > This is not true. If src/main.c is 99% similar to src/foo.c, and is\n> > > 0% similar to the src/main.c in the new commit, we match the old\n> > > src/main.c to the new src/main.c despite being far more similar\n> > > src/foo.c. Unless break detection is turned on, we do not allow\n> > > content similarity to trump (full) filename equality.\n> >\n> > Absolutely. And we are talking about a new optimization that kicks\n> > in only when there is no break or no copy detection going on, no?\n>\n> Yes, precisely, we are only considering cases without break\n> detection...and thus we are considering cases where for the last 15\n> years or more, sufficiently large filename similarity (an exact\n> fullname match) trumps any level of content similarity. I think it is\n> useful to note that while my optimization is adding more\n> considerations that can overrule maximal content similarity, it is not\n> the first such code choice to do that.\n>\n> But let me back up a bit...\n>\n> When I submitted the series, you and Stolee went into a long\n> discussion about an optimization that I didn't submit, one that feels\n> looser on \"matching\" than anything I submitted, and which I think\n> might counter-intuitively reduce performance rather than aid it. (The\n> performance side only comes into view in combination with later\n> series, but it was why I harped so much since then on only comparing\n> against at most one other file in the steps before full inexact rename\n> detection.) I was quite surprised by the diversion, but it made it\n> clear to me that my descriptions and commit messages were far too\n> vague and could be read to imply a completely different algorithm than\n> I intended. So, I tried to be far more careful in subsequent\n> iterations by adding wider context and contrasts.\n>\n> Further, after I wrote various things to try to clarify the\n> misunderstandings, I noticed that Stolee picked out one thing and\n> stated that \"This idea of optimizing first for 100% filename\n> similarity is a good perspective on Git's rename detection algorithm.\"\n> (see https://lore.kernel.org/git/57d30e7d-7727-8d98-e3ef-bcfeebf9edd3@gmail.com/)\n> So, that particular point seemed to help him understand more, and\n> thus might be useful extra context for others reading along now or in\n> the future.\n>\n> Given all the above, I was trying to address earlier misunderstandings\n> and provide more context. Perhaps I swung the pendulum too far and\n> talked too much about other cases, or perhaps I just worded things\n> poorly again. All I was attempting to do in the commit message was\n> point out the multiple basic rules with filename and content\n> similarity, to lay the groundwork for new rules that do alternative\n> weightings.\n>\n> Anyway, I've added a few more tweaks to try to improve the wording for\n> the next round I'll submit today. Given my track record so far, it\n> would not be surprising if it still needed more tweaks.\n"},{"id":"416853","messageId":"CABPp-BHSsiU_BN-egJBjzCV2wyE4ABXkzhfnz-QRaUx5cbTDKQ@mail.gmail.com","threadId":"55112","inReplyTo":"xmqqft1zttev.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 3/5] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-14T03:08:31Z","receivedAt":"2021-02-14T03:10:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Feb 13, 2021 at 3:55 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > I can change that. I can also simplify it further to\n> >\n> > if (0 <= (dst_index = (strintmap_get(&dests, base)))) {\n> >\n> > since dests uses a default value of -1. That will decrease the number\n> > of strmap lookups here from 2 to 1.\n>\n> Which would be a real win, unlike what I said in the message you are\n> responding to.\n\nSadly, while it's a real win, it's very temporary. The next series\nI'll submit needs to separate the two checks back out for other\nreasons.\n\n> >> It feels incompatible with the spirit of these two steps aim for\n> >> (i.e. only use this optimization on a pair of src/dst with UNIQUE\n> >> basenames). For the purpose of \"we only handle unique ones\", the\n> >> paths that already have matched should participate in deciding if\n> >> the files that survived \"exact\" phase have unique basename among\n> >> the original inpu?\n> >\n> > Yeah, I should have been more careful with my wording. Stated a\n> > different way, what confidence can we associate with an exact rename?\n>\n> Suppose you start with a/Makefile, b/Makefile and c/Makefile and\n> then they all disappeared while a1/Makefile, b1/Makefile and\n> c1/Makefile now are in the tree. The contents a/Makefile used to\n> have appears without any difference in a1/Makefile, the same for b\n> and b1, but c/Makefile and c1/Makefile are different. The c vs c1\n> pair may worth investigating, so it goes through the \"same basename\"\n> phase.\n>\n> Now, in a slightly different situation, a vs a1 are still identical,\n> but b vs b1 have only one blank line removal but without any other\n> change. It looks odd that such a change has to pessimize c vs c1\n> optimization opportunity, but an interesting part of the story is\n> that we can only say \"such a change\", not \"such a miniscule change\",\n> because we have just finished the \"exact\" phase, and we do not know\n> how big a difference b vs b1 pair actually had.\n>\n> That makes me feel that this whole \"we must treat unique one that\n> remains specially\" is being incoherent.\n\nIt's really not that special; the pessimization is not in my mind due\nto correctness reasons, but performance reasons.\n\nI need to only compare any given file to at most one other file in the\npreliminary steps. When there are multiple remaining possibilities to\ncompare, I need a method for selecting which ones to compare. I have\nsuch a method, but it's a lot more code. It was easier to submit a\nseries that was only 3 patches long and only considered the pairs that\njust happened to uniquely match up so we could talk about the general\nidea of basename matching. The next series finds ways to match up\nmore files with similar basenames.\n\n> If \"because we have only\n> small number of removed and added Makefiles spread across the trees,\n> first full-matrix matching among them without anything else with\n> higher bar may be worth an optimization\" were the optimization, then\n\nThis optimization was indeed considered...and fully implemented.\nLet's give it a name, so I can refer to it more below. How about the\n\"preliminary-matrix-of-basenames\" optimization?\n\n> I would understand and support the design to omit those that have\n> already been matched in the \"exact\" phase.\n>\n> But IIRC, limiting this \"same basename\" phase to unique add/del pair\n> was sold as a way to make it less likely for the heuristics to make\n> mistakes, yet the definition of \"unique\", as shown above, is not all\n> that solid. That I find it rather unsatisfactory.\n\nNo, I never sold it as a way to make it less likely for the heuristics\nto make mistakes. If I implied that anywhere, it was on accident.\n\nI certainly emphasized only doing one comparison per file, but not for\nthat reason. I had three reasons for mentioning\none-comparison-per-file: (1) I was trying to contrast with Stolee's\noriginal assumption about what this series was doing, to try to avoid\na repeat of the misunderstandings about the current optimization being\nsuggested. (2) The preliminary-matrix-of-basenames optimization has\nworst-case performance nearly twice as bad as without such an\noptimization. (For example, with preliminary-matrix-of-basenames, if\nnearly all unmatched files have the same basename, we end up basically\ndoing inexact rename detection on all files twice). I believe\nStolee's original assumption of what was being proposed also has such\ntwice-as-slow-as-normal worst-case performance behavior. Even though\nthe worst case performance would be fairly rare, making an algorithm\ntwice as slow by introducing an optimization felt like something I\nshould avoid. (3) Despite the theoretical problems with worst-case\nperformance, I implemented the preliminary-matrix-of-basenames\noptimization anyway. I threw the code away, because even in cases\nwith a wide variety of basenames, it slowed things down when other\noptimizations were also involved. The one clear way to work well with\nother optimizations I was working with was to only allow the\npreliminary step to compare any given file to at most one other file.\n\n> In other words, it is not \"what confidence do we have in exact\n> phase?\" \"exact\" matching may have found perfect matching pair. But\n> the found pair should be happy just between themselves, and should\n> not have undue effect on how _other_ pairs are compared. Stopping\n> the \"exact\" pair from participating in the \"uniqueness\" definition\n> is placing \"exact\" phase too much weight to affect how other filepairs\n> are found.\n\nI guess I look at this quite a bit differently. Here's my view:\n\n * If we have a reasonable and cheap way to determine that two\nparticular files are likely potential rename pairs,\n * AND checking their similarity confirms they are sufficiently\nsimilar (perhaps with a higher bar)\n * then we've found a way to avoid quadratic comparisons.\n\nWe will give up \"optimal\" matches, but as long as what we provide are\n\"reasonable\" matches I think that should suffice. I personally\nbelieve \"reasonable\" at O(N) cost trumps \"optimal\" at O(N^2).\n\nThere are several different ways to find \"likely potential rename pairs\":\n * The preliminary-matrix-of-basenames is one that I tried (but\ninteracts badly performance-wise with other optimizations).\n * https://github.com/gitgitgadget/git/issues/519 has multiple ideas.\n * Stolee's misunderstanding of my series is another\n * unique basenames among remaining pairs after exact renames is a\nreally simple one that lets me introduce \"reasonable\" matches so we\ncan discuss\n * my next series adds another\n\nThat leaves us with a big question. Are we happy with higher\nsufficient similarity bar being enough of a constraint for\n\"reasonable\" matches? If so, each of the above ideas might be able to\nhelp us. If not, we may be able to rule some of them out apriori and\navoid working on them (well, working on them any more; I've already\nimplemented three, and we have an intern who picked a project to look\nat one)\n\n> > By the exact same argument, you\n> > could take this a step further and say that we should calculate the\n> > basenames of *all* files in the tree, not just add/delete pairs, and\n> > only match up the ones via basename that are *truly* unique. After\n> > all, break detection exists, so perhaps we don't have full confidence\n> > that files with an unchanged fullname are actually related.\n>\n> Sorry, but you are not making sense. These optimizations are done\n> only when we are not using copies and breaks, no? What _other_\n> changes that kept the paths the same, or modified in place, have any\n> effect on matching added and deleted pairs?\n\nIf the optimization is presented to users as \"only compare basenames\nin a preliminary step when they are unique\", which is what I was\nunderstanding you to say, and if the user has a/Makefile and\nd/Makefile in the source tree, and a1/Makefile and d/Makefile in the\ndestination tree, then a/Makefile is not the unique \"Makefile\" in the\nsource tree.\n\nI think you're trying to make an argument about uniqueness and why it\nmatters for correctness, but I'm not following it.\n\nThe only reason uniqueness is important to me is because I was using\nit with future optimizations in mind, and knew it to be related to an\nimportant performance criteria. I tried to avoid mentioning\nuniqueness at all in the user-facing documentation, though I did try\nto explain why some files with the same basename might not be matched\nup by that step (and my next series modifies those docs a bit.)\n"},{"id":"416854","messageId":"CABPp-BGYZBKhgLPSwGc0z+WO7UGAVS6A9W+cXoDn8nZDh2ZTJw@mail.gmail.com","threadId":"55112","inReplyTo":"CAPc5daVVNaBiWt8vRv3xrXd7GvNPwRGRF=W31DUJCnZRo+9qtA@mail.gmail.com","subject":"Re: [PATCH v3 1/5] t4001: add a test comparing basename similarity and content similarity","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-02-14T03:14:09Z","receivedAt":"2021-02-14T03:15:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Feb 13, 2021 at 5:32 PM Junio C Hamano wrote:\n>\n> I do not consider \"the same file changed in place\" the same as \"we\n> seem to have lost a file in the old tree, ah, we found one that has\n> the same basename in a different directory\" at all, so your argument\n> still does not make any sense to me, sorry.\n\nI'm not set on the commit message wording, you asked why I had used\nthe terms I did and I tried to explain. I also explained how the\nwording seemed to have helped Stolee understand.\n\nIf you'd like to suggest an alternative commit message, I'm happy to take it.\n\n> 2021年2月13日(土) 17:25 Elijah Newren :\n> >\n> > On Sat, Feb 13, 2021 at 3:56 PM Junio C Hamano wrote:\n> > >\n> > > Elijah Newren writes:\n> > >\n> > > > This is not true. If src/main.c is 99% similar to src/foo.c, and is\n> > > > 0% similar to the src/main.c in the new commit, we match the old\n> > > > src/main.c to the new src/main.c despite being far more similar\n> > > > src/foo.c. Unless break detection is turned on, we do not allow\n> > > > content similarity to trump (full) filename equality.\n> > >\n> > > Absolutely. And we are talking about a new optimization that kicks\n> > > in only when there is no break or no copy detection going on, no?\n> >\n> > Yes, precisely, we are only considering cases without break\n> > detection...and thus we are considering cases where for the last 15\n> > years or more, sufficiently large filename similarity (an exact\n> > fullname match) trumps any level of content similarity. I think it is\n> > useful to note that while my optimization is adding more\n> > considerations that can overrule maximal content similarity, it is not\n> > the first such code choice to do that.\n> >\n> > But let me back up a bit...\n> >\n> > When I submitted the series, you and Stolee went into a long\n> > discussion about an optimization that I didn't submit, one that feels\n> > looser on \"matching\" than anything I submitted, and which I think\n> > might counter-intuitively reduce performance rather than aid it. (The\n> > performance side only comes into view in combination with later\n> > series, but it was why I harped so much since then on only comparing\n> > against at most one other file in the steps before full inexact rename\n> > detection.) I was quite surprised by the diversion, but it made it\n> > clear to me that my descriptions and commit messages were far too\n> > vague and could be read to imply a completely different algorithm than\n> > I intended. So, I tried to be far more careful in subsequent\n> > iterations by adding wider context and contrasts.\n> >\n> > Further, after I wrote various things to try to clarify the\n> > misunderstandings, I noticed that Stolee picked out one thing and\n> > stated that \"This idea of optimizing first for 100% filename\n> > similarity is a good perspective on Git's rename detection algorithm.\"\n> > (see https://lore.kernel.org/git/57d30e7d-7727-8d98-e3ef-bcfeebf9edd3@gmail.com/)\n> > So, that particular point seemed to help him understand more, and\n> > thus might be useful extra context for others reading along now or in\n> > the future.\n> >\n> > Given all the above, I was trying to address earlier misunderstandings\n> > and provide more context. Perhaps I swung the pendulum too far and\n> > talked too much about other cases, or perhaps I just worded things\n> > poorly again. All I was attempting to do in the commit message was\n> > point out the multiple basic rules with filename and content\n> > similarity, to lay the groundwork for new rules that do alternative\n> > weightings.\n> >\n> > Anyway, I've added a few more tweaks to try to improve the wording for\n> > the next round I'll submit today. Given my track record so far, it\n> > would not be surprising if it still needed more tweaks.\n"},{"id":"416859","messageId":"ece76429dc35ec2f7f61dd79e069cdb86cf98ee7.1613289112.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v5.git.1613289112.gitgitgadget@gmail.com","subject":"[PATCH v5 3/6] diffcore-rename: complete find_basename_matches()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:51:48Z","receivedAt":"2021-02-14T07:53:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIt is not uncommon in real world repositories for the majority of file\nrenames to not change the basename of the file; i.e. most \"renames\" are\njust a move of files into different directories. We can make use of\nthis to avoid comparing all rename source candidates with all rename\ndestination candidates, by first comparing sources to destinations with\nthe same basenames. If two files with the same basename are\nsufficiently similar, we record the rename; if not, we include those\nfiles in the more exhaustive matrix comparison.\n\nThis means we are adding a set of preliminary additional comparisons,\nbut for each file we only compare it with at most one other file. For\nexample, if there was a include/media/device.h that was deleted and a\nsrc/module/media/device.h that was added, and there are no other\ndevice.h files in the remaining sets of added and deleted files after\nexact rename detection, then these two files would be compared in the\npreliminary step.\n\nThis commit does not yet actually employ this new optimization, it\nmerely adds a function which can be used for this purpose. The next\ncommit will do the necessary plumbing to make use of it.\n\nNote that this optimization might give us different results than without\nthe optimization, because it's possible that despite files with the same\nbasename being sufficiently similar to be considered a rename, there's\nan even better match between files without the same basename. I think\nthat is okay for four reasons: (1) it's easy to explain to the users\nwhat happened if it does ever occur (or even for them to intuitively\nfigure out), (2) as the next patch will show it provides such a large\nperformance boost that it's worth the tradeoff, and (3) it's somewhat\nunlikely that despite having unique matching basenames that other files\nserve as better matches. Reason (4) takes a full paragraph to\nexplain...\n\nIf the previous three reasons aren't enough, consider what rename\ndetection already does. Break detection is not the default, meaning\nthat if files have the same _fullname_, then they are considered related\neven if they are 0% similar. In fact, in such a case, we don't even\nbother comparing the files to see if they are similar let alone\ncomparing them to all other files to see what they are most similar to.\nBasically, we override content similarity based on sufficient filename\nsimilarity. Without the filename similarity (currently implemented as\nan exact match of filename), we swing the pendulum the opposite\ndirection and say that filename similarity is irrelevant and compare a\nfull N x M matrix of sources and destinations to find out which have the\nmost similar contents. This optimization just adds another form of\nfilename similarity comparison, but augments it with a file content\nsimilarity check as well. Basically, if two files have the same\nbasename and are sufficiently similar to be considered a rename, mark\nthem as such without comparing the two to all other rename candidates.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 82 +++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 79 insertions(+), 3 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex e51f33a2184a..266d4fae48c7 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -383,9 +383,53 @@ MAYBE_UNUSED\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score)\n {\n-\tint i;\n+\t/*\n+\t * When I checked in early 2020, over 76% of file renames in linux\n+\t * just moved files to a different directory but kept the same\n+\t * basename. gcc did that with over 64% of renames, gecko did it\n+\t * with over 79%, and WebKit did it with over 89%.\n+\t *\n+\t * Therefore we can bypass the normal exhaustive NxM matrix\n+\t * comparison of similarities between all potential rename sources\n+\t * and destinations by instead using file basename as a hint (i.e.\n+\t * the portion of the filename after the last '/'), checking for\n+\t * similarity between files with the same basename, and if we find\n+\t * a pair that are sufficiently similar, record the rename pair and\n+\t * exclude those two from the NxM matrix.\n+\t *\n+\t * This *might* cause us to find a less than optimal pairing (if\n+\t * there is another file that we are even more similar to but has a\n+\t * different basename). Given the huge performance advantage\n+\t * basename matching provides, and given the frequency with which\n+\t * people use the same basename in real world projects, that's a\n+\t * trade-off we are willing to accept when doing just rename\n+\t * detection.\n+\t *\n+\t * If someone wants copy detection that implies they are willing to\n+\t * spend more cycles to find similarities between files, so it may\n+\t * be less likely that this heuristic is wanted. If someone is\n+\t * doing break detection, that means they do not want filename\n+\t * similarity to imply any form of content similiarity, and thus\n+\t * this heuristic would definitely be incompatible.\n+\t */\n+\n+\tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *entry;\n+\n+\t/*\n+\t * The prefeteching stuff wants to know if it can skip prefetching\n+\t * blobs that are unmodified...and will then do a little extra work\n+\t * to verify that the oids are indeed different before prefetching.\n+\t * Unmodified blobs are only relevant when doing copy detection;\n+\t * when limiting to rename detection, diffcore_rename[_extended]()\n+\t * will never be called with unmodified source paths fed to us, so\n+\t * the extra work necessary to check if rename_src entries are\n+\t * unmodified would be a small waste.\n+\t */\n+\tint skip_unmodified = 0;\n \n \t/*\n \t * Create maps of basename -> fullname(s) for remaining sources and\n@@ -422,12 +466,44 @@ static int find_basename_matches(struct diff_options *options,\n \t\t\tstrintmap_set(&dests, base, i);\n \t}\n \n-\t/* TODO: Make use of basenames source and destination basenames */\n+\t/* Now look for basename matchups and do similarity estimation */\n+\tstrintmap_for_each_entry(&sources, &iter, entry) {\n+\t\tconst char *base = entry->key;\n+\t\tintptr_t src_index = (intptr_t)entry->value;\n+\t\tintptr_t dst_index;\n+\t\tif (src_index == -1)\n+\t\t\tcontinue;\n+\n+\t\tif (0 <= (dst_index = strintmap_get(&dests, base))) {\n+\t\t\tstruct diff_filespec *one, *two;\n+\t\t\tint score;\n+\n+\t\t\t/* Estimate the similarity */\n+\t\t\tone = rename_src[src_index].p->one;\n+\t\t\ttwo = rename_dst[dst_index].p->two;\n+\t\t\tscore = estimate_similarity(options->repo, one, two,\n+\t\t\t\t\t\t minimum_score, skip_unmodified);\n+\n+\t\t\t/* If sufficiently similar, record as rename pair */\n+\t\t\tif (score < minimum_score)\n+\t\t\t\tcontinue;\n+\t\t\trecord_rename_pair(dst_index, src_index, score);\n+\t\t\trenames++;\n+\n+\t\t\t/*\n+\t\t\t * Found a rename so don't need text anymore; if we\n+\t\t\t * didn't find a rename, the filespec_blob would get\n+\t\t\t * re-used when doing the matrix of comparisons.\n+\t\t\t */\n+\t\t\tdiff_free_filespec_blob(one);\n+\t\t\tdiff_free_filespec_blob(two);\n+\t\t}\n+\t}\n \n \tstrintmap_clear(&sources);\n \tstrintmap_clear(&dests);\n \n-\treturn 0;\n+\treturn renames;\n }\n \n #define NUM_CANDIDATE_PER_DST 4\n-- \ngitgitgadget\n\n"},{"id":"416860","messageId":"6f5584f613500bd41fb40988dbe014939c0ea814.1613289112.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v5.git.1613289112.gitgitgadget@gmail.com","subject":"[PATCH v5 5/6] gitdiffcore doc: mention new preliminary step for rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:51:50Z","receivedAt":"2021-02-14T07:53:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe last few patches have introduced a new preliminary step when rename\ndetection is on but both break detection and copy detection are off.\nDocument this new step. While we're at it, add a testcase that checks\nthe new behavior as well.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/gitdiffcore.txt | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/Documentation/gitdiffcore.txt b/Documentation/gitdiffcore.txt\nindex c970d9fe438a..80fcf9542441 100644\n--- a/Documentation/gitdiffcore.txt\n+++ b/Documentation/gitdiffcore.txt\n@@ -168,6 +168,26 @@ a similarity score different from the default of 50% by giving a\n number after the \"-M\" or \"-C\" option (e.g. \"-M8\" to tell it to use\n 8/10 = 80%).\n \n+Note that when rename detection is on but both copy and break\n+detection are off, rename detection adds a preliminary step that first\n+checks if files are moved across directories while keeping their\n+filename the same. If there is a file added to a directory whose\n+contents is sufficiently similar to a file with the same name that got\n+deleted from a different directory, it will mark them as renames and\n+exclude them from the later quadratic step (the one that pairwise\n+compares all unmatched files to find the \"best\" matches, determined by\n+the highest content similarity). So, for example, if a deleted\n+docs/ext.txt and an added docs/config/ext.txt are similar enough, they\n+will be marked as a rename and prevent an added docs/ext.md that may\n+be even more similar to the deleted docs/ext.txt from being considered\n+as the rename destination in the later step. For this reason, the\n+preliminary \"match same filename\" step uses a bit higher threshold to\n+mark a file pair as a rename and stop considering other candidates for\n+better matches. At most, one comparison is done per file in this\n+preliminary pass; so if there are several remaining ext.txt files\n+throughout the directory hierarchy after exact rename detection, this\n+preliminary step will be skipped for those files.\n+\n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\n diffcore mechanism as well as modified ones. This lets the copy\n-- \ngitgitgadget\n\n"},{"id":"416861","messageId":"pull.843.v5.git.1613289112.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v4.git.1613031350.gitgitgadget@gmail.com","subject":"[PATCH v5 0/6] Optimization batch 7: use file basenames to guide rename detection","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:51:45Z","receivedAt":"2021-02-14T07:53:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series depends on ort-perf-batch-6. It appears Junio has appended an\nearlier round of this series on that one and called the combined series\nen/diffcore-rename. I'm still resubmitting the two separately to preserve\nthe threaded discussion in the archives and because gitgitgadget can provide\nproper range-diffs that way.\n\nThis series uses file basenames (portion of the path after final '/',\nincluding extension) in a basic fashion to guide rename detection.\n\nChanges since v4:\n\n * add wording to make it clearer that we are considering remaining\n basenames after exact rename detection\n * add three minor optimizations to patch 3. (All three will have to be\n undone by the next series, but this series is probably clearer with\n them.)\n * a typo fix or two\n * v2 of ort-perf-batch-6 added some changes around consistency of\n rename_src_nr; make similar changes in using this variable in\n find_basename_changes() for consistency\n * fix the testcase so the expected comments about the change in behavior\n only show up after we change the behavior\n * attempt a rewrite of the commit message for the new testcase, who knows\n if I'll get it right this time.\n\nElijah Newren (6):\n t4001: add a test comparing basename similarity and content similarity\n diffcore-rename: compute basenames of source and dest candidates\n diffcore-rename: complete find_basename_matches()\n diffcore-rename: guide inexact rename detection based on basenames\n gitdiffcore doc: mention new preliminary step for rename detection\n merge-ort: call diffcore_rename() directly\n\n Documentation/gitdiffcore.txt | 20 ++++\n diffcore-rename.c | 190 +++++++++++++++++++++++++++++++++-\n merge-ort.c | 66 ++++++++++--\n t/t4001-diff-rename.sh | 24 +++++\n 4 files changed, 289 insertions(+), 11 deletions(-)\n\n\nbase-commit: dd6595b45640ee9894293e8b729ef9a254564a49\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-843%2Fnewren%2Fort-perf-batch-7-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-843/newren/ort-perf-batch-7-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/843\n\nRange-diff vs v4:\n\n 1: 3e6af929d135 ! 1: 6848422e47e8 t4001: add a test comparing basename similarity and content similarity\n @@ Commit message\n \n Add a simple test where a removed file is similar to two different added\n files; one of them has the same basename, and the other has a slightly\n - higher content similarity. Without break detection, filename similarity\n - of 100% trumps content similarity for pairing up related files. For\n - any filename similarity less than 100%, the opposite is true -- content\n - similarity is all that matters. Add a testcase that documents this.\n + higher content similarity. In the current test, content similarity is\n + weighted higher than filename similarity.\n \n - Subsequent commits will add a new rule that includes an inbetween state,\n - where a mixture of filename similarity and content similarity are\n - weighed, and which will change the outcome of this testcase.\n + Subsequent commits will add a new rule that weighs a mixture of filename\n + similarity and content similarity in a manner that will change the\n + outcome of this testcase.\n \n Signed-off-by: Elijah Newren \n \n @@ t/t4001-diff-rename.sh: test_expect_success 'diff-tree -l0 defaults to a big ren\n +\tgit add file.txt file.md &&\n +\tgit commit -a -m \"rename\" &&\n +\tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n -+\t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n -+\t# but since same basenames are checked first...\n ++\t# subdir/file.txt is 88% similar to file.md and 78% similar to file.txt\n +\tcat >expected <<-\\EOF &&\n +\tR088\tsubdir/file.txt\tfile.md\n +\tA\tfile.txt\n 2: 4fff9b1ff57b ! 2: 73baae2535d0 diffcore-rename: compute basenames of all source and dest candidates\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - diffcore-rename: compute basenames of all source and dest candidates\n + diffcore-rename: compute basenames of source and dest candidates\n \n - We want to make use of unique basenames to help inform rename detection,\n - so that more likely pairings can be checked first. (src/moduleA/foo.txt\n - and source/module/A/foo.txt are likely related if there are no other\n - 'foo.txt' files among the deleted and added files.) Add a new function,\n - not yet used, which creates a map of the unique basenames within\n - rename_src and another within rename_dst, together with the indices\n - within rename_src/rename_dst where those basenames show up. Non-unique\n - basenames still show up in the map, but have an invalid index (-1).\n + We want to make use of unique basenames among remaining source and\n + destination files to help inform rename detection, so that more likely\n + pairings can be checked first. (src/moduleA/foo.txt and\n + source/module/A/foo.txt are likely related if there are no other\n + 'foo.txt' files among the remaining deleted and added files.) Add a new\n + function, not yet used, which creates a map of the unique basenames\n + within rename_src and another within rename_dst, together with the\n + indices within rename_src/rename_dst where those basenames show up.\n + Non-unique basenames still show up in the map, but have an invalid index\n + (-1).\n \n This function was inspired by the fact that in real world repositories,\n files are often moved across directories without changing names. Here\n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n +static const char *get_basename(const char *filename)\n +{\n +\t/*\n -+\t * gitbasename() has to worry about special drivers, multiple\n ++\t * gitbasename() has to worry about special drives, multiple\n +\t * directory separator characters, trailing slashes, NULL or\n +\t * empty strings, etc. We only work on filenames as stored in\n +\t * git, and thus get to ignore all those complications.\n @@ diffcore-rename.c: static int find_exact_renames(struct diff_options *options)\n +\n +MAYBE_UNUSED\n +static int find_basename_matches(struct diff_options *options,\n -+\t\t\t\t int minimum_score,\n -+\t\t\t\t int num_src)\n ++\t\t\t\t int minimum_score)\n +{\n +\tint i;\n +\tstruct strintmap sources;\n +\tstruct strintmap dests;\n +\n -+\t/* Create maps of basename -> fullname(s) for sources and dests */\n ++\t/*\n ++\t * Create maps of basename -> fullname(s) for remaining sources and\n ++\t * dests.\n ++\t */\n +\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n +\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n -+\tfor (i = 0; i < num_src; ++i) {\n ++\tfor (i = 0; i < rename_src_nr; ++i) {\n +\t\tchar *filename = rename_src[i].p->one->path;\n +\t\tconst char *base;\n +\n 3: dc26881e4ed3 ! 3: ece76429dc35 diffcore-rename: complete find_basename_matches()\n @@ Commit message\n This means we are adding a set of preliminary additional comparisons,\n but for each file we only compare it with at most one other file. For\n example, if there was a include/media/device.h that was deleted and a\n - src/module/media/device.h that was added, and there were no other\n - device.h files added or deleted between the commits being compared,\n - then these two files would be compared in the preliminary step.\n + src/module/media/device.h that was added, and there are no other\n + device.h files in the remaining sets of added and deleted files after\n + exact rename detection, then these two files would be compared in the\n + preliminary step.\n \n This commit does not yet actually employ this new optimization, it\n merely adds a function which can be used for this purpose. The next\n @@ Commit message\n Signed-off-by: Elijah Newren \n \n ## diffcore-rename.c ##\n -@@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options,\n - \t\t\t\t int minimum_score,\n - \t\t\t\t int num_src)\n +@@ diffcore-rename.c: MAYBE_UNUSED\n + static int find_basename_matches(struct diff_options *options,\n + \t\t\t\t int minimum_score)\n {\n -\tint i;\n +\t/*\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n +\tint i, renames = 0;\n \tstruct strintmap sources;\n \tstruct strintmap dests;\n - \n ++\tstruct hashmap_iter iter;\n ++\tstruct strmap_entry *entry;\n ++\n +\t/*\n +\t * The prefeteching stuff wants to know if it can skip prefetching\n +\t * blobs that are unmodified...and will then do a little extra work\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options\n +\t * unmodified would be a small waste.\n +\t */\n +\tint skip_unmodified = 0;\n -+\n - \t/* Create maps of basename -> fullname(s) for sources and dests */\n - \tstrintmap_init_with_options(&sources, -1, NULL, 0);\n - \tstrintmap_init_with_options(&dests, -1, NULL, 0);\n + \n + \t/*\n + \t * Create maps of basename -> fullname(s) for remaining sources and\n @@ diffcore-rename.c: static int find_basename_matches(struct diff_options *options,\n \t\t\tstrintmap_set(&dests, base, i);\n \t}\n \n -\t/* TODO: Make use of basenames source and destination basenames */\n +\t/* Now look for basename matchups and do similarity estimation */\n -+\tfor (i = 0; i < num_src; ++i) {\n -+\t\tchar *filename = rename_src[i].p->one->path;\n -+\t\tconst char *base = NULL;\n -+\t\tintptr_t src_index;\n ++\tstrintmap_for_each_entry(&sources, &iter, entry) {\n ++\t\tconst char *base = entry->key;\n ++\t\tintptr_t src_index = (intptr_t)entry->value;\n +\t\tintptr_t dst_index;\n -+\n -+\t\t/* Find out if this basename is unique among sources */\n -+\t\tbase = get_basename(filename);\n -+\t\tsrc_index = strintmap_get(&sources, base);\n +\t\tif (src_index == -1)\n -+\t\t\tcontinue; /* not a unique basename; skip it */\n -+\t\tassert(src_index == i);\n ++\t\t\tcontinue;\n +\n -+\t\tif (strintmap_contains(&dests, base)) {\n ++\t\tif (0 <= (dst_index = strintmap_get(&dests, base))) {\n +\t\t\tstruct diff_filespec *one, *two;\n +\t\t\tint score;\n +\n -+\t\t\t/* Find out if this basename is unique among dests */\n -+\t\t\tdst_index = strintmap_get(&dests, base);\n -+\t\t\tif (dst_index == -1)\n -+\t\t\t\tcontinue; /* not a unique basename; skip it */\n -+\n -+\t\t\t/* Ignore this dest if already used in a rename */\n -+\t\t\tif (rename_dst[dst_index].is_rename)\n -+\t\t\t\tcontinue; /* already used previously */\n -+\n +\t\t\t/* Estimate the similarity */\n +\t\t\tone = rename_src[src_index].p->one;\n +\t\t\ttwo = rename_dst[dst_index].p->two;\n 4: 2493f4b2f55d ! 4: 122902e2706f diffcore-rename: guide inexact rename detection based on basenames\n @@ Commit message\n files based on basenames. As a quick reminder (see the last two commit\n messages for more details), this means for example that\n docs/extensions.txt and docs/config/extensions.txt are considered likely\n - renames if there are no 'extensions.txt' files elsewhere among the added\n - and deleted files, and if a similarity check confirms they are similar,\n - then they are marked as a rename without looking for a better similarity\n - match among other files. This is a behavioral change, as covered in\n - more detail in the previous commit message.\n + renames if there are no remaining 'extensions.txt' files elsewhere among\n + the added and deleted files, and if a similarity check confirms they are\n + similar, then they are marked as a rename without looking for a better\n + similarity match among other files. This is a behavioral change, as\n + covered in more detail in the previous commit message.\n \n We do not use this heuristic together with either break or copy\n detection. The point of break detection is to say that filename\n @@ diffcore-rename.c: static const char *get_basename(const char *filename)\n \n -MAYBE_UNUSED\n static int find_basename_matches(struct diff_options *options,\n - \t\t\t\t int minimum_score,\n - \t\t\t\t int num_src)\n + \t\t\t\t int minimum_score)\n + {\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n \tif (minimum_score == MAX_SCORE)\n \t\tgoto cleanup;\n \n -+\tnum_sources = rename_src_nr;\n -+\n +-\t/* Calculate how many renames are left */\n +-\tnum_destinations = (rename_dst_nr - rename_count);\n +-\tremove_unneeded_paths_from_src(want_copies);\n + \tnum_sources = rename_src_nr;\n + \n +\tif (want_copies || break_idx) {\n +\t\t/*\n +\t\t * Cull sources:\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n +\t\t/* Utilize file basenames to quickly find renames. */\n +\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n +\t\trename_count += find_basename_matches(options,\n -+\t\t\t\t\t\t min_basename_score,\n -+\t\t\t\t\t\t rename_src_nr);\n ++\t\t\t\t\t\t min_basename_score);\n +\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n +\n +\t\t/*\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n +\t\ttrace2_region_leave(\"diff\", \"cull basename\", options->repo);\n +\t}\n +\n - \t/*\n --\t * Calculate how many renames are left\n -+\t * Calculate how many rename destinations are left\n - \t */\n - \tnum_destinations = (rename_dst_nr - rename_count);\n --\tremove_unneeded_paths_from_src(want_copies);\n --\tnum_sources = rename_src_nr;\n ++\t/* Calculate how many rename destinations are left */\n ++\tnum_destinations = (rename_dst_nr - rename_count);\n +\tnum_sources = rename_src_nr; /* rename_src_nr reflects lower number */\n - \n ++\n \t/* All done? */\n \tif (!num_destinations || !num_sources)\n + \t\tgoto cleanup;\n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n \t\tstruct diff_score *m;\n \n @@ diffcore-rename.c: void diffcore_rename(struct diff_options *options)\n \n ## t/t4001-diff-rename.sh ##\n @@ t/t4001-diff-rename.sh: test_expect_success 'basename similarity vs best similarity' '\n - \t# subdir/file.txt is 89% similar to file.md, 78% similar to file.txt,\n - \t# but since same basenames are checked first...\n + \tgit add file.txt file.md &&\n + \tgit commit -a -m \"rename\" &&\n + \tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n +-\t# subdir/file.txt is 88% similar to file.md and 78% similar to file.txt\n ++\t# subdir/file.txt is 88% similar to file.md, 78% similar to file.txt,\n ++\t# but since same basenames are checked first...\n \tcat >expected <<-\\EOF &&\n -\tR088\tsubdir/file.txt\tfile.md\n -\tA\tfile.txt\n 5: 4e86ed3f29d4 ! 5: 6f5584f61350 gitdiffcore doc: mention new preliminary step for rename detection\n @@ Documentation/gitdiffcore.txt: a similarity score different from the default of\n +preliminary \"match same filename\" step uses a bit higher threshold to\n +mark a file pair as a rename and stop considering other candidates for\n +better matches. At most, one comparison is done per file in this\n -+preliminary pass; so if there are several ext.txt files throughout the\n -+directory hierarchy that were added and deleted, this preliminary step\n -+will be skipped for those files.\n ++preliminary pass; so if there are several remaining ext.txt files\n ++throughout the directory hierarchy after exact rename detection, this\n ++preliminary step will be skipped for those files.\n +\n Note. When the \"-C\" option is used with `--find-copies-harder`\n option, 'git diff-{asterisk}' commands feed unmodified filepairs to\n 6: fedb3d323d94 = 6: aeca14f748af merge-ort: call diffcore_rename() directly\n\n-- \ngitgitgadget\n"},{"id":"416862","messageId":"6848422e47e8b8477e6d24b6442345aa521ccdb5.1613289112.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v5.git.1613289112.gitgitgadget@gmail.com","subject":"[PATCH v5 1/6] t4001: add a test comparing basename similarity and content similarity","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:51:46Z","receivedAt":"2021-02-14T07:53:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd a simple test where a removed file is similar to two different added\nfiles; one of them has the same basename, and the other has a slightly\nhigher content similarity. In the current test, content similarity is\nweighted higher than filename similarity.\n\nSubsequent commits will add a new rule that weighs a mixture of filename\nsimilarity and content similarity in a manner that will change the\noutcome of this testcase.\n\nSigned-off-by: Elijah Newren \n---\n t/t4001-diff-rename.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex c16486a9d41a..0f97858197e1 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -262,4 +262,27 @@ test_expect_success 'diff-tree -l0 defaults to a big rename limit, not zero' '\n \tgrep \"myotherfile.*myfile\" actual\n '\n \n+test_expect_success 'basename similarity vs best similarity' '\n+\tmkdir subdir &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 line10 >subdir/file.txt &&\n+\tgit add subdir/file.txt &&\n+\tgit commit -m \"base txt\" &&\n+\n+\tgit rm subdir/file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 >file.txt &&\n+\ttest_write_lines line1 line2 line3 line4 line5 \\\n+\t\t\t line6 line7 line8 line9 >file.md &&\n+\tgit add file.txt file.md &&\n+\tgit commit -a -m \"rename\" &&\n+\tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n+\t# subdir/file.txt is 88% similar to file.md and 78% similar to file.txt\n+\tcat >expected <<-\\EOF &&\n+\tR088\tsubdir/file.txt\tfile.md\n+\tA\tfile.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"416863","messageId":"73baae2535d0d2d50f1d05d81f575c29f9776c97.1613289112.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v5.git.1613289112.gitgitgadget@gmail.com","subject":"[PATCH v5 2/6] diffcore-rename: compute basenames of source and dest candidates","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:51:47Z","receivedAt":"2021-02-14T07:54:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to make use of unique basenames among remaining source and\ndestination files to help inform rename detection, so that more likely\npairings can be checked first. (src/moduleA/foo.txt and\nsource/module/A/foo.txt are likely related if there are no other\n'foo.txt' files among the remaining deleted and added files.) Add a new\nfunction, not yet used, which creates a map of the unique basenames\nwithin rename_src and another within rename_dst, together with the\nindices within rename_src/rename_dst where those basenames show up.\nNon-unique basenames still show up in the map, but have an invalid index\n(-1).\n\nThis function was inspired by the fact that in real world repositories,\nfiles are often moved across directories without changing names. Here\nare some sample repositories and the percentage of their historical\nrenames (as of early 2020) that preserved basenames:\n * linux: 76%\n * gcc: 64%\n * gecko: 79%\n * webkit: 89%\nThese statistics alone don't prove that an optimization in this area\nwill help or how much it will help, since there are also unpaired adds\nand deletes, restrictions on which basenames we consider, etc., but it\ncertainly motivated the idea to try something in this area.\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 63 insertions(+)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 6fd0c4a2f485..e51f33a2184a 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -367,6 +367,69 @@ static int find_exact_renames(struct diff_options *options)\n \treturn renames;\n }\n \n+static const char *get_basename(const char *filename)\n+{\n+\t/*\n+\t * gitbasename() has to worry about special drives, multiple\n+\t * directory separator characters, trailing slashes, NULL or\n+\t * empty strings, etc. We only work on filenames as stored in\n+\t * git, and thus get to ignore all those complications.\n+\t */\n+\tconst char *base = strrchr(filename, '/');\n+\treturn base ? base + 1 : filename;\n+}\n+\n+MAYBE_UNUSED\n+static int find_basename_matches(struct diff_options *options,\n+\t\t\t\t int minimum_score)\n+{\n+\tint i;\n+\tstruct strintmap sources;\n+\tstruct strintmap dests;\n+\n+\t/*\n+\t * Create maps of basename -> fullname(s) for remaining sources and\n+\t * dests.\n+\t */\n+\tstrintmap_init_with_options(&sources, -1, NULL, 0);\n+\tstrintmap_init_with_options(&dests, -1, NULL, 0);\n+\tfor (i = 0; i < rename_src_nr; ++i) {\n+\t\tchar *filename = rename_src[i].p->one->path;\n+\t\tconst char *base;\n+\n+\t\t/* exact renames removed in remove_unneeded_paths_from_src() */\n+\t\tassert(!rename_src[i].p->one->rename_used);\n+\n+\t\t/* Record index within rename_src (i) if basename is unique */\n+\t\tbase = get_basename(filename);\n+\t\tif (strintmap_contains(&sources, base))\n+\t\t\tstrintmap_set(&sources, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&sources, base, i);\n+\t}\n+\tfor (i = 0; i < rename_dst_nr; ++i) {\n+\t\tchar *filename = rename_dst[i].p->two->path;\n+\t\tconst char *base;\n+\n+\t\tif (rename_dst[i].is_rename)\n+\t\t\tcontinue; /* involved in exact match already. */\n+\n+\t\t/* Record index within rename_dst (i) if basename is unique */\n+\t\tbase = get_basename(filename);\n+\t\tif (strintmap_contains(&dests, base))\n+\t\t\tstrintmap_set(&dests, base, -1);\n+\t\telse\n+\t\t\tstrintmap_set(&dests, base, i);\n+\t}\n+\n+\t/* TODO: Make use of basenames source and destination basenames */\n+\n+\tstrintmap_clear(&sources);\n+\tstrintmap_clear(&dests);\n+\n+\treturn 0;\n+}\n+\n #define NUM_CANDIDATE_PER_DST 4\n static void record_if_better(struct diff_score m[], struct diff_score *o)\n {\n-- \ngitgitgadget\n\n"},{"id":"416864","messageId":"122902e2706f70e926e1e05952d7dee7914607f0.1613289112.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v5.git.1613289112.gitgitgadget@gmail.com","subject":"[PATCH v5 4/6] diffcore-rename: guide inexact rename detection based on basenames","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:51:49Z","receivedAt":"2021-02-14T07:55:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMake use of the new find_basename_matches() function added in the last\ntwo patches, to find renames more rapidly in cases where we can match up\nfiles based on basenames. As a quick reminder (see the last two commit\nmessages for more details), this means for example that\ndocs/extensions.txt and docs/config/extensions.txt are considered likely\nrenames if there are no remaining 'extensions.txt' files elsewhere among\nthe added and deleted files, and if a similarity check confirms they are\nsimilar, then they are marked as a rename without looking for a better\nsimilarity match among other files. This is a behavioral change, as\ncovered in more detail in the previous commit message.\n\nWe do not use this heuristic together with either break or copy\ndetection. The point of break detection is to say that filename\nsimilarity does not imply file content similarity, and we only want to\nknow about file content similarity. The point of copy detection is to\nuse more resources to check for additional similarities, while this is\nan optimization that uses far less resources but which might also result\nin finding slightly fewer similarities. So the idea behind this\noptimization goes against both of those features, and will be turned off\nfor both.\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 13.815 s ± 0.062 s 13.294 s ± 0.103 s\n mega-renames: 1799.937 s ± 0.493 s 187.248 s ± 0.882 s\n just-one-mega: 51.289 s ± 0.019 s 5.557 s ± 0.017 s\n\nSigned-off-by: Elijah Newren \n---\n diffcore-rename.c | 53 ++++++++++++++++++++++++++++++++++++++----\n t/t4001-diff-rename.sh | 7 +++---\n 2 files changed, 52 insertions(+), 8 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 266d4fae48c7..41558185ae1d 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -379,7 +379,6 @@ static const char *get_basename(const char *filename)\n \treturn base ? base + 1 : filename;\n }\n \n-MAYBE_UNUSED\n static int find_basename_matches(struct diff_options *options,\n \t\t\t\t int minimum_score)\n {\n@@ -716,11 +715,55 @@ void diffcore_rename(struct diff_options *options)\n \tif (minimum_score == MAX_SCORE)\n \t\tgoto cleanup;\n \n-\t/* Calculate how many renames are left */\n-\tnum_destinations = (rename_dst_nr - rename_count);\n-\tremove_unneeded_paths_from_src(want_copies);\n \tnum_sources = rename_src_nr;\n \n+\tif (want_copies || break_idx) {\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones corresponding to exact renames\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\t} else {\n+\t\t/* Determine minimum score to match basenames */\n+\t\tdouble factor = 0.5;\n+\t\tchar *basename_factor = getenv(\"GIT_BASENAME_FACTOR\");\n+\t\tint min_basename_score;\n+\n+\t\tif (basename_factor)\n+\t\t\tfactor = strtol(basename_factor, NULL, 10)/100.0;\n+\t\tassert(factor >= 0.0 && factor <= 1.0);\n+\t\tmin_basename_score = minimum_score +\n+\t\t\t(int)(factor * (MAX_SCORE - minimum_score));\n+\n+\t\t/*\n+\t\t * Cull sources:\n+\t\t * - remove ones involved in renames (found via exact match)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull after exact\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull after exact\", options->repo);\n+\n+\t\t/* Utilize file basenames to quickly find renames. */\n+\t\ttrace2_region_enter(\"diff\", \"basename matches\", options->repo);\n+\t\trename_count += find_basename_matches(options,\n+\t\t\t\t\t\t min_basename_score);\n+\t\ttrace2_region_leave(\"diff\", \"basename matches\", options->repo);\n+\n+\t\t/*\n+\t\t * Cull sources, again:\n+\t\t * - remove ones involved in renames (found via basenames)\n+\t\t */\n+\t\ttrace2_region_enter(\"diff\", \"cull basename\", options->repo);\n+\t\tremove_unneeded_paths_from_src(want_copies);\n+\t\ttrace2_region_leave(\"diff\", \"cull basename\", options->repo);\n+\t}\n+\n+\t/* Calculate how many rename destinations are left */\n+\tnum_destinations = (rename_dst_nr - rename_count);\n+\tnum_sources = rename_src_nr; /* rename_src_nr reflects lower number */\n+\n \t/* All done? */\n \tif (!num_destinations || !num_sources)\n \t\tgoto cleanup;\n@@ -751,7 +794,7 @@ void diffcore_rename(struct diff_options *options)\n \t\tstruct diff_score *m;\n \n \t\tif (rename_dst[i].is_rename)\n-\t\t\tcontinue; /* dealt with exact match already. */\n+\t\t\tcontinue; /* exact or basename match already handled */\n \n \t\tm = &mx[dst_cnt * NUM_CANDIDATE_PER_DST];\n \t\tfor (j = 0; j < NUM_CANDIDATE_PER_DST; j++)\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex 0f97858197e1..99a5d1bd1c3a 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -277,10 +277,11 @@ test_expect_success 'basename similarity vs best similarity' '\n \tgit add file.txt file.md &&\n \tgit commit -a -m \"rename\" &&\n \tgit diff-tree -r -M --name-status HEAD^ HEAD >actual &&\n-\t# subdir/file.txt is 88% similar to file.md and 78% similar to file.txt\n+\t# subdir/file.txt is 88% similar to file.md, 78% similar to file.txt,\n+\t# but since same basenames are checked first...\n \tcat >expected <<-\\EOF &&\n-\tR088\tsubdir/file.txt\tfile.md\n-\tA\tfile.txt\n+\tA\tfile.md\n+\tR078\tsubdir/file.txt\tfile.txt\n \tEOF\n \ttest_cmp expected actual\n '\n-- \ngitgitgadget\n\n"},{"id":"416865","messageId":"aeca14f748afc7fb5b65bca56ea2ebd970729814.1613289112.git.gitgitgadget@gmail.com","threadId":"55112","inReplyTo":"pull.843.v5.git.1613289112.gitgitgadget@gmail.com","subject":"[PATCH v5 6/6] merge-ort: call diffcore_rename() directly","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-02-14T07:51:51Z","receivedAt":"2021-02-14T07:55:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to pass additional information to diffcore_rename() (or some\nvariant thereof) without plumbing that extra information through\ndiff_tree_oid() and diffcore_std(). Further, since we will need to\ngather additional special information related to diffs and are walking\nthe trees anyway in collect_merge_info(), it seems odd to have\ndiff_tree_oid()/diffcore_std() repeat those tree walks. And there may\nbe times where we can avoid traversing into a subtree in\ncollect_merge_info() (based on additional information at our disposal),\nthat the basic diff logic would be unable to take advantage of. For all\nthese reasons, just create the add and delete pairs ourself and then\ncall diffcore_rename() directly.\n\nThis change is primarily about enabling future optimizations; the\nadvantage of avoiding extra tree traversals is small compared to the\ncost of rename detection, and the advantage of avoiding the extra tree\ntraversals is somewhat offset by the extra time spent in\ncollect_merge_info() collecting the additional data anyway. However...\n\nFor the testcases mentioned in commit 557ac0350d (\"merge-ort: begin\nperformance work; instrument with trace2_region_* calls\", 2020-10-28),\nthis change improves the performance as follows:\n\n Before After\n no-renames: 13.294 s ± 0.103 s 12.775 s ± 0.062 s\n mega-renames: 187.248 s ± 0.882 s 188.754 s ± 0.284 s\n just-one-mega: 5.557 s ± 0.017 s 5.599 s ± 0.019 s\n\nSigned-off-by: Elijah Newren \n---\n merge-ort.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 59 insertions(+), 7 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 931b91438cf1..603d30c52170 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -535,6 +535,23 @@ static void setup_path_info(struct merge_options *opt,\n \tresult->util = mi;\n }\n \n+static void add_pair(struct merge_options *opt,\n+\t\t struct name_entry *names,\n+\t\t const char *pathname,\n+\t\t unsigned side,\n+\t\t unsigned is_add /* if false, is_delete */)\n+{\n+\tstruct diff_filespec *one, *two;\n+\tstruct rename_info *renames = &opt->priv->renames;\n+\tint names_idx = is_add ? side : 0;\n+\n+\tone = alloc_filespec(pathname);\n+\ttwo = alloc_filespec(pathname);\n+\tfill_filespec(is_add ? two : one,\n+\t\t &names[names_idx].oid, 1, names[names_idx].mode);\n+\tdiff_queue(&renames->pairs[side], one, two);\n+}\n+\n static void collect_rename_info(struct merge_options *opt,\n \t\t\t\tstruct name_entry *names,\n \t\t\t\tconst char *dirname,\n@@ -544,6 +561,7 @@ static void collect_rename_info(struct merge_options *opt,\n \t\t\t\tunsigned match_mask)\n {\n \tstruct rename_info *renames = &opt->priv->renames;\n+\tunsigned side;\n \n \t/* Update dirs_removed, as needed */\n \tif (dirmask == 1 || dirmask == 3 || dirmask == 5) {\n@@ -554,6 +572,21 @@ static void collect_rename_info(struct merge_options *opt,\n \t\tif (sides & 2)\n \t\t\tstrset_add(&renames->dirs_removed[2], fullname);\n \t}\n+\n+\tif (filemask == 0 || filemask == 7)\n+\t\treturn;\n+\n+\tfor (side = MERGE_SIDE1; side <= MERGE_SIDE2; ++side) {\n+\t\tunsigned side_mask = (1 << side);\n+\n+\t\t/* Check for deletion on side */\n+\t\tif ((filemask & 1) && !(filemask & side_mask))\n+\t\t\tadd_pair(opt, names, fullname, side, 0 /* delete */);\n+\n+\t\t/* Check for addition on side */\n+\t\tif (!(filemask & 1) && (filemask & side_mask))\n+\t\t\tadd_pair(opt, names, fullname, side, 1 /* add */);\n+\t}\n }\n \n static int collect_merge_info_callback(int n,\n@@ -2079,6 +2112,27 @@ static int process_renames(struct merge_options *opt,\n \treturn clean_merge;\n }\n \n+static void resolve_diffpair_statuses(struct diff_queue_struct *q)\n+{\n+\t/*\n+\t * A simplified version of diff_resolve_rename_copy(); would probably\n+\t * just use that function but it's static...\n+\t */\n+\tint i;\n+\tstruct diff_filepair *p;\n+\n+\tfor (i = 0; i < q->nr; ++i) {\n+\t\tp = q->queue[i];\n+\t\tp->status = 0; /* undecided */\n+\t\tif (!DIFF_FILE_VALID(p->one))\n+\t\t\tp->status = DIFF_STATUS_ADDED;\n+\t\telse if (!DIFF_FILE_VALID(p->two))\n+\t\t\tp->status = DIFF_STATUS_DELETED;\n+\t\telse if (DIFF_PAIR_RENAME(p))\n+\t\t\tp->status = DIFF_STATUS_RENAMED;\n+\t}\n+}\n+\n static int compare_pairs(const void *a_, const void *b_)\n {\n \tconst struct diff_filepair *a = *((const struct diff_filepair **)a_);\n@@ -2089,8 +2143,6 @@ static int compare_pairs(const void *a_, const void *b_)\n \n /* Call diffcore_rename() to compute which files have changed on given side */\n static void detect_regular_renames(struct merge_options *opt,\n-\t\t\t\t struct tree *merge_base,\n-\t\t\t\t struct tree *side,\n \t\t\t\t unsigned side_index)\n {\n \tstruct diff_options diff_opts;\n@@ -2108,11 +2160,11 @@ static void detect_regular_renames(struct merge_options *opt,\n \tdiff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;\n \tdiff_setup_done(&diff_opts);\n \n+\tdiff_queued_diff = renames->pairs[side_index];\n \ttrace2_region_enter(\"diff\", \"diffcore_rename\", opt->repo);\n-\tdiff_tree_oid(&merge_base->object.oid, &side->object.oid, \"\",\n-\t\t &diff_opts);\n-\tdiffcore_std(&diff_opts);\n+\tdiffcore_rename(&diff_opts);\n \ttrace2_region_leave(\"diff\", \"diffcore_rename\", opt->repo);\n+\tresolve_diffpair_statuses(&diff_queued_diff);\n \n \tif (diff_opts.needed_rename_limit > renames->needed_limit)\n \t\trenames->needed_limit = diff_opts.needed_rename_limit;\n@@ -2212,8 +2264,8 @@ static int detect_and_process_renames(struct merge_options *opt,\n \tmemset(&combined, 0, sizeof(combined));\n \n \ttrace2_region_enter(\"merge\", \"regular renames\", opt->repo);\n-\tdetect_regular_renames(opt, merge_base, side1, MERGE_SIDE1);\n-\tdetect_regular_renames(opt, merge_base, side2, MERGE_SIDE2);\n+\tdetect_regular_renames(opt, MERGE_SIDE1);\n+\tdetect_regular_renames(opt, MERGE_SIDE2);\n \ttrace2_region_leave(\"merge\", \"regular renames\", opt->repo);\n \n \ttrace2_region_enter(\"merge\", \"directory renames\", opt->repo);\n-- \ngitgitgadget\n"}]}