patchFix diffcore-break total breakage

Ok, so on the kernel list, some people noticed that "git log --follow" doesn't work too well with some files in the x86 merge, because a lot of files got renamed in very special ways.

In particular, there was a pattern of doing single commits with renames that looked basically like

 - rename "filename.h" -> "filename_64.h"
 - create new "filename.c" that includes "filename_32.h" or 
   "filename_64.h" depending on whether we're 32-bit or 64-bit.

which was preparatory for smushing the two trees together.

Now, there's two issues here:

 - "filename.c" *remained*. Yes, it was a rename, but there was a new file 
   created with the old name in the same commit. This was important, 
   because we wanted each commit to compile properly, so that it was 
   bisectable, so splitting the rename into one commit and the "create 
   helper file" into another was *not* an option.

   So we need to break associations where the contents change too much. 
   Fine. We have the -B flag for that. When we break things up, then the 
   rename detection will be able to figure out whether there are better 
   alternatives.

 - "git log --follow" didn't with with -B.

Now, the second case was really simple: we use a different "diffopt" structure for the rename detection than the basic one (which we use for showing the diffs). So that second case is trivially fixed by a trivial one-liner that just copies the break_opt values from the "real" diffopts to the one used for rename following. So now "git log -B --follow" works fine:

	diff --git a/tree-diff.c b/tree-diff.c
	index 26bdbdd..7c261fd 100644
	--- a/tree-diff.c
	+++ b/tree-diff.c
	@@ -319,6 +319,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
	 	diff_opts.detect_rename = DIFF_DETECT_RENAME;
	 	diff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;
	 	diff_opts.single_follow = opt->paths[0];
	+	diff_opts.break_opt = opt->break_opt;
	 	paths[0] = NULL;
	 	diff_tree_setup_paths(paths, &diff_opts);
	 	if (diff_setup_done(&diff_opts) < 0)

however, the end result does *not* work. Because our diffcore-break.c logic is totally bogus!

In particular:

 - it used to do

	if (base_size < MINIMUM_BREAK_SIZE)
		return 0; /* we do not break too small filepair */

   which basically says "don't bother to break small files". But that 
   "base_size" is the *smaller* of the two sizes, which means that if some 
   large file was rewritten into one that just includes another file, we 
   would look at the (small) result, and decide that it's smaller than the 
   break size, so it cannot be worth it to break it up! Even if the other 
   side was ten times bigger and looked *nothing* like the samell file!

   That's clearly bogus. I replaced "base_size" with "max_size", so that 
   we compare the *bigger* of the filepair with the break size.

 - It calculated a "merge_score", which was the score needed to merge it 
   back together if nothing else wanted it. But even if it was *so* 
   different that we would never want to merge it back, we wouldn't 
   consider it a break! That makes no sense. So I added

	if (*merge_score_p > break_score)
		return 1;

   to make it clear that if we wouldn't want to merge it at the end, it 
   was *definitely* a break.

 - It compared the whole "extent of damage", counting all inserts and 
   deletes, but it based this score on the "base_size", and generated the 
   damage score with

	delta_size = src_removed + literal_added;
	damage_score = delta_size * MAX_SCORE / base_size;

   but that makes no sense either, since quite often, this will result in 
   a number that is *bigger* than MAX_SCORE! Why? Because base_size is 
   (again) the smaller of the two files we compare, and when you start out 
   from a small file and add a lot (or start out from a large file and 
   remove a lot), the base_size is going to be much smaller than the 
   damage!

   Again, the fix was to replace "base_size" with "max_size", at which 
   point the damage actually becomes a sane percentage of the whole.

With these changes in place, not only does "git log -B --follow" work for the case that triggered this in the first place, ie now

	git log -B --follow arch/x86/kernel/vmlinux_64.lds.S

actually gives reasonable resulys. But I also wanted to verify it in general, by doing a full-history

	git log --stat -B -C

on my kernel tree with the old code and the new code. 

There's some tweaking to be done, but generally, the new code generates much better results wrt breaking up files (and then finding better rename candidates). Here's a few examples of the "--stat" output:

 - This:
	include/asm-x86/Kbuild        |    2 -
	include/asm-x86/debugreg.h    |   79 +++++++++++++++++++++++++++++++++++------
	include/asm-x86/debugreg_32.h |   64 ---------------------------------
	include/asm-x86/debugreg_64.h |   65 ---------------------------------
	4 files changed, 68 insertions(+), 142 deletions(-)

      Becomes:

	include/asm-x86/Kbuild                        |    2 -
	include/asm-x86/{debugreg_64.h => debugreg.h} |    9 +++-
	include/asm-x86/debugreg_32.h                 |   64 -------------------------
	3 files changed, 7 insertions(+), 68 deletions(-)

 - This:
	include/asm-x86/bug.h    |   41 +++++++++++++++++++++++++++++++++++++++--
	include/asm-x86/bug_32.h |   37 -------------------------------------
	include/asm-x86/bug_64.h |   34 ----------------------------------
	3 files changed, 39 insertions(+), 73 deletions(-)

      Becomes

	include/asm-x86/{bug_64.h => bug.h} |   20 +++++++++++++-----
	include/asm-x86/bug_32.h            |   37 -----------------------------------
	2 files changed, 14 insertions(+), 43 deletions(-)

Now, in some other cases, it does actually turn a rename into a real "delete+create" pair, and then the diff is usually bigger, so truth in advertizing: it doesn't always generate a nicer diff. But for what -B was meant for, I think this is a big improvement, and I suspect those cases where it generates a bigger diff are tweakable.

So I think this diff fixes a real bug, but we might still want to tweak the default values and perhaps the exact rules for when a break happens.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

Hmm? At least the "should_break()" tests seem to make some amount of sense now, I think.

		Linus

----
 diffcore-break.c |   11 +++++++----
 tree-diff.c      |    1 +
 2 files changed, 8 insertions(+), 4 deletions(-)

Split out "exact content match" phase of rename detection

This makes the exact content match a separate function of its own. Partly to cut down a bit on the size of the diffcore_rename() function (which is too complex as it is), and partly because there are smarter ways to do this than an O(m*n) loop over it all, and that function should be rewritten to take that into account.

Whether I'll get to the rewrite or not is not clear, but this is a worthy cleanup regardless.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

So, I'm looking a bit at rename handling, since it's got the scalability problems. Andy Chu of google pointed to an algorithm that doesn't use a m*n array, but instead a hash table to match things up, and that should be much better. No promises as to whether I'll ever actually implement it, but just looking at that diffcore_rename function makes your head ache, and while this doesn't improve any code, at least it splits one independent thing out of it..

---
 diffcore-rename.c |   90 +++++++++++++++++++++++++++++++++--------------------
 1 files changed, 56 insertions(+), 34 deletions(-)