Re: [PATCH 1/2] diffcore-rename: support rename cache

From: Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date: Nov 8, 2008, 12:00 UTC
Message-ID: <fcaeb9bf0811080400h7ea5377cvaa8d658335811c23@mail.gmail.com>
In-Reply-To: <20081108114719.GA4989@sigill.intra.peff.net>

On 11/8/08, Jeff King <peff@peff.net> wrote:

Show 17 quoted lines

> On Sat, Nov 08, 2008 at 10:24:10AM +0100, Yann Dirson wrote:
>
>  > Well, it could be that an out-of-pathspec pair would have a better
>  > score than an in-pathspec one.  Maybe cache recording should be turned
>  > off when doing pathspec limitation ?
>
>
> One thing I notice is that the cache works at the level of "here is the
>  best rename for this commit." Maybe it could go down a level and say
>  "here is the inexact rename score between these blobs". Then you would
>  still find the best score between two blobs each time, but save the
>  really computationally intensive part (which is comparing the actual
>  _content_ of the blobs).
>  That should work in the face of path limiting or any other option,
>  because it is caching something immutable: this is the similarity score
>  between two pieces of content. And then you get arbitrary tree-to-tree
>  speedups for free, since such a cache would be valid for every commit.

I did that and realized the cost was not from each diff, in --find-copies-harder case, but from the number of diffs you had to do. Even with exact matching on linux-2.6.git, it could take significant time (it was about 5 minutes in no-cache case, 1 minute without exact match cache, and less than 1 sec if everything is cached).

Show 6 quoted lines

>
>  The downsides are:
>
>   - your cache is potentially bigger, since you are caching the score of
>    every pair you look at, instead of just "good" pairs (OTOH, you are
>    not doing a per-commit cache, which helps reduce the size)

It is huge if you accidentially add --find-copies-harder to your command, considering that every new file will be compared against every files in tree (about 25k).

>   - you can still "lie" about a score to pre-seed imported SVN renames,
>    but such lying will actually apply to all commits.

-- 
Duy