{"thread":{"id":"10397","subject":"[PATCH] Fix diffcore-break total breakage","startedAt":"2007-10-20T19:31:31Z","lastAt":"2007-10-21T18:40:47Z","messageCount":2,"participants":["Linus Torvalds"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"56723","messageId":"alpine.LFD.0.999.0710201158580.10525@woody.linux-foundation.org","threadId":"10397","inReplyTo":null,"subject":"[PATCH] Fix diffcore-break total breakage","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-10-20T19:31:31Z","receivedAt":"2007-10-20T19:31:31Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOk, so on the kernel list, some people noticed that \"git log --follow\" \ndoesn't work too well with some files in the x86 merge, because a lot of \nfiles got renamed in very special ways.\n\nIn particular, there was a pattern of doing single commits with renames \nthat looked basically like\n\n - rename \"filename.h\" -> \"filename_64.h\"\n - create new \"filename.c\" that includes \"filename_32.h\" or \n   \"filename_64.h\" depending on whether we're 32-bit or 64-bit.\n\nwhich was preparatory for smushing the two trees together.\n\nNow, there's two issues here:\n\n - \"filename.c\" *remained*. Yes, it was a rename, but there was a new file \n   created with the old name in the same commit. This was important, \n   because we wanted each commit to compile properly, so that it was \n   bisectable, so splitting the rename into one commit and the \"create \n   helper file\" into another was *not* an option.\n\n   So we need to break associations where the contents change too much. \n   Fine. We have the -B flag for that. When we break things up, then the \n   rename detection will be able to figure out whether there are better \n   alternatives.\n\n - \"git log --follow\" didn't with with -B.\n\nNow, the second case was really simple: we use a different \"diffopt\" \nstructure for the rename detection than the basic one (which we use for \nshowing the diffs). So that second case is trivially fixed by a trivial \none-liner that just copies the break_opt values from the \"real\" diffopts \nto the one used for rename following. So now \"git log -B --follow\" works \nfine:\n\n\tdiff --git a/tree-diff.c b/tree-diff.c\n\tindex 26bdbdd..7c261fd 100644\n\t--- a/tree-diff.c\n\t+++ b/tree-diff.c\n\t@@ -319,6 +319,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co\n\t \tdiff_opts.detect_rename = DIFF_DETECT_RENAME;\n\t \tdiff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;\n\t \tdiff_opts.single_follow = opt->paths[0];\n\t+\tdiff_opts.break_opt = opt->break_opt;\n\t \tpaths[0] = NULL;\n\t \tdiff_tree_setup_paths(paths, &diff_opts);\n\t \tif (diff_setup_done(&diff_opts) < 0)\n\nhowever, the end result does *not* work. Because our diffcore-break.c \nlogic is totally bogus!\n\nIn particular:\n\n - it used to do\n\n\tif (base_size < MINIMUM_BREAK_SIZE)\n\t\treturn 0; /* we do not break too small filepair */\n\n   which basically says \"don't bother to break small files\". But that \n   \"base_size\" is the *smaller* of the two sizes, which means that if some \n   large file was rewritten into one that just includes another file, we \n   would look at the (small) result, and decide that it's smaller than the \n   break size, so it cannot be worth it to break it up! Even if the other \n   side was ten times bigger and looked *nothing* like the samell file!\n\n   That's clearly bogus. I replaced \"base_size\" with \"max_size\", so that \n   we compare the *bigger* of the filepair with the break size.\n\n - It calculated a \"merge_score\", which was the score needed to merge it \n   back together if nothing else wanted it. But even if it was *so* \n   different that we would never want to merge it back, we wouldn't \n   consider it a break! That makes no sense. So I added\n\n\tif (*merge_score_p > break_score)\n\t\treturn 1;\n\n   to make it clear that if we wouldn't want to merge it at the end, it \n   was *definitely* a break.\n\n - It compared the whole \"extent of damage\", counting all inserts and \n   deletes, but it based this score on the \"base_size\", and generated the \n   damage score with\n\n\tdelta_size = src_removed + literal_added;\n\tdamage_score = delta_size * MAX_SCORE / base_size;\n\n   but that makes no sense either, since quite often, this will result in \n   a number that is *bigger* than MAX_SCORE! Why? Because base_size is \n   (again) the smaller of the two files we compare, and when you start out \n   from a small file and add a lot (or start out from a large file and \n   remove a lot), the base_size is going to be much smaller than the \n   damage!\n\n   Again, the fix was to replace \"base_size\" with \"max_size\", at which \n   point the damage actually becomes a sane percentage of the whole.\n\nWith these changes in place, not only does \"git log -B --follow\" work for \nthe case that triggered this in the first place, ie now\n\n\tgit log -B --follow arch/x86/kernel/vmlinux_64.lds.S\n\nactually gives reasonable resulys. But I also wanted to verify it in \ngeneral, by doing a full-history\n\n\tgit log --stat -B -C\n\non my kernel tree with the old code and the new code. \n\nThere's some tweaking to be done, but generally, the new code generates \nmuch better results wrt breaking up files (and then finding better rename \ncandidates). Here's a few examples of the \"--stat\" output:\n\n - This:\n\tinclude/asm-x86/Kbuild        |    2 -\n\tinclude/asm-x86/debugreg.h    |   79 +++++++++++++++++++++++++++++++++++------\n\tinclude/asm-x86/debugreg_32.h |   64 ---------------------------------\n\tinclude/asm-x86/debugreg_64.h |   65 ---------------------------------\n\t4 files changed, 68 insertions(+), 142 deletions(-)\n\n      Becomes:\n\n\tinclude/asm-x86/Kbuild                        |    2 -\n\tinclude/asm-x86/{debugreg_64.h => debugreg.h} |    9 +++-\n\tinclude/asm-x86/debugreg_32.h                 |   64 -------------------------\n\t3 files changed, 7 insertions(+), 68 deletions(-)\n\n - This:\n\tinclude/asm-x86/bug.h    |   41 +++++++++++++++++++++++++++++++++++++++--\n\tinclude/asm-x86/bug_32.h |   37 -------------------------------------\n\tinclude/asm-x86/bug_64.h |   34 ----------------------------------\n\t3 files changed, 39 insertions(+), 73 deletions(-)\n\n      Becomes\n\n\tinclude/asm-x86/{bug_64.h => bug.h} |   20 +++++++++++++-----\n\tinclude/asm-x86/bug_32.h            |   37 -----------------------------------\n\t2 files changed, 14 insertions(+), 43 deletions(-)\n\nNow, in some other cases, it does actually turn a rename into a real \n\"delete+create\" pair, and then the diff is usually bigger, so truth in \nadvertizing: it doesn't always generate a nicer diff. But for what -B was \nmeant for, I think this is a big improvement, and I suspect those cases \nwhere it generates a bigger diff are tweakable.\n\nSo I think this diff fixes a real bug, but we might still want to tweak \nthe default values and perhaps the exact rules for when a break happens.\n\nSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>\n---\n\nHmm? At least the \"should_break()\" tests seem to make some amount of sense \nnow, I think. \n\n\t\tLinus\n\n----\n diffcore-break.c |   11 +++++++----\n tree-diff.c      |    1 +\n 2 files changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/diffcore-break.c b/diffcore-break.c\nindex ae8a7d0..c71a226 100644\n--- a/diffcore-break.c\n+++ b/diffcore-break.c\n@@ -45,8 +45,8 @@ static int should_break(struct diff_filespec *src,\n \t * The value we return is 1 if we want the pair to be broken,\n \t * or 0 if we do not.\n \t */\n-\tunsigned long delta_size, base_size, src_copied, literal_added,\n-\t\tsrc_removed;\n+\tunsigned long delta_size, base_size, max_size;\n+\tunsigned long src_copied, literal_added, src_removed;\n \n \t*merge_score_p = 0; /* assume no deletion --- \"do not break\"\n \t\t\t     * is the default.\n@@ -63,7 +63,8 @@ static int should_break(struct diff_filespec *src,\n \t\treturn 0; /* error but caught downstream */\n \n \tbase_size = ((src->size < dst->size) ? src->size : dst->size);\n-\tif (base_size < MINIMUM_BREAK_SIZE)\n+\tmax_size = ((src->size > dst->size) ? src->size : dst->size);\n+\tif (max_size < MINIMUM_BREAK_SIZE)\n \t\treturn 0; /* we do not break too small filepair */\n \n \tif (diffcore_count_changes(src, dst,\n@@ -89,12 +90,14 @@ static int should_break(struct diff_filespec *src,\n \t * less than the minimum, after rename/copy runs.\n \t */\n \t*merge_score_p = (int)(src_removed * MAX_SCORE / src->size);\n+\tif (*merge_score_p > break_score)\n+\t\treturn 1;\n \n \t/* Extent of damage, which counts both inserts and\n \t * deletes.\n \t */\n \tdelta_size = src_removed + literal_added;\n-\tif (delta_size * MAX_SCORE / base_size < break_score)\n+\tif (delta_size * MAX_SCORE / max_size < break_score)\n \t\treturn 0;\n \n \t/* If you removed a lot without adding new material, that is\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 26bdbdd..7c261fd 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -319,6 +319,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co\n \tdiff_opts.detect_rename = DIFF_DETECT_RENAME;\n \tdiff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;\n \tdiff_opts.single_follow = opt->paths[0];\n+\tdiff_opts.break_opt = opt->break_opt;\n \tpaths[0] = NULL;\n \tdiff_tree_setup_paths(paths, &diff_opts);\n \tif (diff_setup_done(&diff_opts) < 0)\n"},{"id":"56777","messageId":"alpine.LFD.0.999.0710211137480.10525@woody.linux-foundation.org","threadId":"10397","inReplyTo":"alpine.LFD.0.999.0710201158580.10525@woody.linux-foundation.org","subject":"[PATCH] Split out \"exact content match\" phase of rename detection","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-10-21T18:40:47Z","receivedAt":"2007-10-21T18:40:47Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nSplit out \"exact content match\" phase of rename detection\n\nThis makes the exact content match a separate function of its own.\nPartly to cut down a bit on the size of the diffcore_rename() function\n(which is too complex as it is), and partly because there are smarter\nways to do this than an O(m*n) loop over it all, and that function\nshould be rewritten to take that into account.\n\nWhether I'll get to the rewrite or not is not clear, but this is a\nworthy cleanup regardless.\n\nSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>\n---\n\nSo, I'm looking a bit at rename handling, since it's got the scalability \nproblems. Andy Chu of google pointed to an algorithm that doesn't use a \nm*n array, but instead a hash table to match things up, and that should be \nmuch better. No promises as to whether I'll ever actually implement it, \nbut just looking at that diffcore_rename function makes your head ache, \nand while this doesn't improve any code, at least it splits one \nindependent thing out of it..\n\n---\n diffcore-rename.c |   90 +++++++++++++++++++++++++++++++++--------------------\n 1 files changed, 56 insertions(+), 34 deletions(-)\n\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 142e537..2077a9b 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -262,6 +262,58 @@ static int compute_stays(struct diff_queue_struct *q,\n \treturn 1;\n }\n \n+/*\n+ * Find exact renames first.\n+ *\n+ * The first round matches up the up-to-date entries,\n+ * and then during the second round we try to match\n+ * cache-dirty entries as well.\n+ *\n+ * Note: the rest of the rename logic depends on this\n+ * phase also populating all the filespecs for any\n+ * entry that isn't matched up with an exact rename,\n+ * see \"is_exact_match()\".\n+ */\n+static int find_exact_renames(void)\n+{\n+\tint rename_count = 0;\n+\tint contents_too;\n+\n+\tfor (contents_too = 0; contents_too < 2; contents_too++) {\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < rename_dst_nr; i++) {\n+\t\t\tstruct diff_filespec *two = rename_dst[i].two;\n+\t\t\tint j;\n+\n+\t\t\tif (rename_dst[i].pair)\n+\t\t\t\tcontinue; /* dealt with an earlier round */\n+\t\t\tfor (j = 0; j < rename_src_nr; j++) {\n+\t\t\t\tint k;\n+\t\t\t\tstruct diff_filespec *one = rename_src[j].one;\n+\t\t\t\tif (!is_exact_match(one, two, contents_too))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\t/* see if there is a basename match, too */\n+\t\t\t\tfor (k = j; k < rename_src_nr; k++) {\n+\t\t\t\t\tone = rename_src[k].one;\n+\t\t\t\t\tif (basename_same(one, two) &&\n+\t\t\t\t\t\tis_exact_match(one, two,\n+\t\t\t\t\t\t\tcontents_too)) {\n+\t\t\t\t\t\tj = k;\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\trecord_rename_pair(i, j, (int)MAX_SCORE);\n+\t\t\t\trename_count++;\n+\t\t\t\tbreak; /* we are done with this entry */\n+\t\t\t}\n+\t\t}\n+\t}\n+\treturn rename_count;\n+}\n+\n void diffcore_rename(struct diff_options *options)\n {\n \tint detect_rename = options->detect_rename;\n@@ -270,12 +322,11 @@ void diffcore_rename(struct diff_options *options)\n \tstruct diff_queue_struct *q = &diff_queued_diff;\n \tstruct diff_queue_struct outq;\n \tstruct diff_score *mx;\n-\tint i, j, rename_count, contents_too;\n+\tint i, j, rename_count;\n \tint num_create, num_src, dst_cnt;\n \n \tif (!minimum_score)\n \t\tminimum_score = DEFAULT_RENAME_SCORE;\n-\trename_count = 0;\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n@@ -318,40 +369,11 @@ void diffcore_rename(struct diff_options *options)\n \tif (rename_dst_nr * rename_src_nr > rename_limit * rename_limit)\n \t\tgoto cleanup;\n \n-\t/* We really want to cull the candidates list early\n+\t/*\n+\t * We really want to cull the candidates list early\n \t * with cheap tests in order to avoid doing deltas.\n-\t * The first round matches up the up-to-date entries,\n-\t * and then during the second round we try to match\n-\t * cache-dirty entries as well.\n \t */\n-\tfor (contents_too = 0; contents_too < 2; contents_too++) {\n-\t\tfor (i = 0; i < rename_dst_nr; i++) {\n-\t\t\tstruct diff_filespec *two = rename_dst[i].two;\n-\t\t\tif (rename_dst[i].pair)\n-\t\t\t\tcontinue; /* dealt with an earlier round */\n-\t\t\tfor (j = 0; j < rename_src_nr; j++) {\n-\t\t\t\tint k;\n-\t\t\t\tstruct diff_filespec *one = rename_src[j].one;\n-\t\t\t\tif (!is_exact_match(one, two, contents_too))\n-\t\t\t\t\tcontinue;\n-\n-\t\t\t\t/* see if there is a basename match, too */\n-\t\t\t\tfor (k = j; k < rename_src_nr; k++) {\n-\t\t\t\t\tone = rename_src[k].one;\n-\t\t\t\t\tif (basename_same(one, two) &&\n-\t\t\t\t\t\tis_exact_match(one, two,\n-\t\t\t\t\t\t\tcontents_too)) {\n-\t\t\t\t\t\tj = k;\n-\t\t\t\t\t\tbreak;\n-\t\t\t\t\t}\n-\t\t\t\t}\n-\n-\t\t\t\trecord_rename_pair(i, j, (int)MAX_SCORE);\n-\t\t\t\trename_count++;\n-\t\t\t\tbreak; /* we are done with this entry */\n-\t\t\t}\n-\t\t}\n-\t}\n+\trename_count = find_exact_renames();\n \n \t/* Have we run out the created file pool?  If so we can avoid\n \t * doing the delta matrix altogether.\n"}]}