Re: [PATCH/RFC 0/3] faster inexact rename handling

From: Andy C <andychup@gmail.com>
Date: Oct 31, 2007, 00:27 UTC
Message-ID: <596909b30710301727o72770d51jfa519d858203480f@mail.gmail.com>
In-Reply-To: <20071030202014.GA22733@coredump.intra.peff.net>

On 10/30/07, Jeff King <peff@peff.net> wrote:

Show 15 quoted lines

> On Tue, Oct 30, 2007 at 08:38:24AM -0700, Linus Torvalds wrote:
>
> > > with the old and new code. Pairs like Documentation/git-add-script.txt
> > > -> Documentation/git-add.txt are not found, because the file is composed
> > > almost entirely of boilerplate.
> >
> > Ok, that does imply to me that we cannot just drop boilerplate text,
> > because the fact is, lots of files contain boilerplate, but people still
> > think they are "similar".
>
> Well, the problem is that instead of just "dropping" boilerplate text,
> we fail to count it as a similarity, but it still counts towards the
> file size. It may be that just dropping it totally is the right thing
> (in which case those renames _will_ turn up, because they will be filled
> with identical non-boilerplate goodness).

Right, in the demo I make an extra pass after the inverted indexing step to prune the index -- which means eliminating the common lines *entirely* from the index (so they don't get attributed to a random file) *and* decrementing all the file sizes by 1. That way the similarity scores shouldn't get skewed.

And as you mentioned we could bump the threshold from 1 to some other small integer. Intuitively I guess you could say it is common to copy a file to 2 places or 3 places, and you don't want all the lines to get thrown out because of that. But usually you don't copy a file to 10 or 50 places.

Andy