Re: caching commit patch-ids for fast git-cherry

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: May 29, 2008, 22:19 UTC
Message-ID: <alpine.DEB.1.00.0805292318040.13507@racer.site.net>
In-Reply-To: <7f9d599f0805291034l6c655ccbk219dd74964c65737@mail.gmail.com>

Hi,

On Thu, 29 May 2008, Geoffrey Irving wrote:

Show 24 quoted lines

> On Thu, May 29, 2008 at 10:13 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > Hi,
> >
> > On Thu, 29 May 2008, Geoffrey Irving wrote:
> >
> >> I'm planning to use cherry picking to manage long term syncing 
> >> between cvs/perforce and git repositories.  This means I'll have 
> >> scripts running git-cherry between branches with hundreds of uncommon 
> >> commits, and I want git-cherry to be much, much, faster.
> >>
> >> It looks like I can do this by caching commit->patch-id pairs from 
> >> commit_patch_id() in patch-ids.c to a file, say 
> >> $GIT_DIR/commit-patch-id-cache.  The file would be binary and append 
> >> only, and could be blown away if .  Any suggestions / concerns before 
> >> I write this?  Is there any reusable efficient map code for storing 
> >> the commit->patch-id map, or should I just mirror the blocked storage
> >> + binary search used for struct patch_ids?
> >
> > I would store the stuff sorted, so that the lookup is fast, generation 
> > less so.
> 
> The motivation for append-only was robustness, not speed, but I don't 
> think either concern is very significant.

I think that robustness comes from "write new file and rename if all 
succeeded", not from append-only.  Think of the case where you run out of 
disk space; with append-only, it is more complicated to get back to a 
known good state.

Show 9 quoted lines

> > For inspiration, you might want to look at the "notes" branch in my 
> > personal fork:
> >
> > http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes
> 
> Cool.  I'd rather copy just that code entirely rather than use it for 
> inspiration, since it does exactly what I need.  It would be silly to 
> have two blocks of code implementing "persistent map from 20 byte hash 
> to 20 byte hash".

Hehe.  That's what I meant by "inspiration" ;-)

> I'll start by just copying the entire nodes-index implementation (with a 
> few name substitutions), and we (or I) can refactor it later if both end 
> up in the same respository.

Very nice!

Thanks, Dscho