# caching commit patch-ids for fast git-cherry 4 messages from 2008-05-29 to 2008-05-29. Participants: Geoffrey Irving, Johannes Schindelin. Thread: https://gitlist.dev/t/13721 ## Geoffrey Irving, 2008-05-29 17:01 Subject: caching commit patch-ids for fast git-cherry Message-ID: <7f9d599f0805291001mdbb4b42q6f3a1b79bc9bc4e9@mail.gmail.com> URL: https://gitlist.dev/e/7f9d599f0805291001mdbb4b42q6f3a1b79bc9bc4e9%40mail.gmail.com ``` I'm planning to use cherry picking to manage long term syncing between cvs/perforce and git repositories. This means I'll have scripts running git-cherry between branches with hundreds of uncommon commits, and I want git-cherry to be much, much, faster. It looks like I can do this by caching commit->patch-id pairs from commit_patch_id() in patch-ids.c to a file, say $GIT_DIR/commit-patch-id-cache. The file would be binary and append only, and could be blown away if . Any suggestions / concerns before I write this? Is there any reusable efficient map code for storing the commit->patch-id map, or should I just mirror the blocked storage + binary search used for struct patch_ids? Thanks, Geoffrey ``` ## Johannes Schindelin, 2008-05-29 17:13 Subject: Re: caching commit patch-ids for fast git-cherry Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0805291809340.13507%40racer.site.net In-Reply-To: <7f9d599f0805291001mdbb4b42q6f3a1b79bc9bc4e9@mail.gmail.com> ``` Hi, On Thu, 29 May 2008, Geoffrey Irving wrote: > I'm planning to use cherry picking to manage long term syncing between > cvs/perforce and git repositories. This means I'll have scripts running > git-cherry between branches with hundreds of uncommon commits, and I > want git-cherry to be much, much, faster. > > It looks like I can do this by caching commit->patch-id pairs from > commit_patch_id() in patch-ids.c to a file, say > $GIT_DIR/commit-patch-id-cache. The file would be binary and append > only, and could be blown away if . Any suggestions / concerns before > I write this? Is there any reusable efficient map code for storing > the commit->patch-id map, or should I just mirror the blocked storage > + binary search used for struct patch_ids? I would store the stuff sorted, so that the lookup is fast, generation less so. For inspiration, you might want to look at the "notes" branch in my personal fork: http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes Hth, Dscho ``` ## Geoffrey Irving, 2008-05-29 17:34 Subject: Re: caching commit patch-ids for fast git-cherry Message-ID: <7f9d599f0805291034l6c655ccbk219dd74964c65737@mail.gmail.com> URL: https://gitlist.dev/e/7f9d599f0805291034l6c655ccbk219dd74964c65737%40mail.gmail.com In-Reply-To: ``` On Thu, May 29, 2008 at 10:13 AM, Johannes Schindelin wrote: > Hi, > > On Thu, 29 May 2008, Geoffrey Irving wrote: > >> I'm planning to use cherry picking to manage long term syncing between >> cvs/perforce and git repositories. This means I'll have scripts running >> git-cherry between branches with hundreds of uncommon commits, and I >> want git-cherry to be much, much, faster. >> >> It looks like I can do this by caching commit->patch-id pairs from >> commit_patch_id() in patch-ids.c to a file, say >> $GIT_DIR/commit-patch-id-cache. The file would be binary and append >> only, and could be blown away if . Any suggestions / concerns before >> I write this? Is there any reusable efficient map code for storing >> the commit->patch-id map, or should I just mirror the blocked storage >> + binary search used for struct patch_ids? > > I would store the stuff sorted, so that the lookup is fast, generation > less so. The motivation for append-only was robustness, not speed, but I don't think either concern is very significant. > For inspiration, you might want to look at the "notes" branch in my > personal fork: > > http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes Cool. I'd rather copy just that code entirely rather than use it for inspiration, since it does exactly what I need. It would be silly to have two blocks of code implementing "persistent map from 20 byte hash to 20 byte hash". I'll start by just copying the entire nodes-index implementation (with a few name substitutions), and we (or I) can refactor it later if both end up in the same respository. Geoffrey ``` ## Johannes Schindelin, 2008-05-29 22:19 Subject: Re: caching commit patch-ids for fast git-cherry Message-ID: URL: https://gitlist.dev/e/alpine.DEB.1.00.0805292318040.13507%40racer.site.net In-Reply-To: <7f9d599f0805291034l6c655ccbk219dd74964c65737@mail.gmail.com> ``` Hi, On Thu, 29 May 2008, Geoffrey Irving wrote: > On Thu, May 29, 2008 at 10:13 AM, Johannes Schindelin > wrote: > > Hi, > > > > On Thu, 29 May 2008, Geoffrey Irving wrote: > > > >> I'm planning to use cherry picking to manage long term syncing > >> between cvs/perforce and git repositories. This means I'll have > >> scripts running git-cherry between branches with hundreds of uncommon > >> commits, and I want git-cherry to be much, much, faster. > >> > >> It looks like I can do this by caching commit->patch-id pairs from > >> commit_patch_id() in patch-ids.c to a file, say > >> $GIT_DIR/commit-patch-id-cache. The file would be binary and append > >> only, and could be blown away if . Any suggestions / concerns before > >> I write this? Is there any reusable efficient map code for storing > >> the commit->patch-id map, or should I just mirror the blocked storage > >> + binary search used for struct patch_ids? > > > > I would store the stuff sorted, so that the lookup is fast, generation > > less so. > > The motivation for append-only was robustness, not speed, but I don't > think either concern is very significant. I think that robustness comes from "write new file and rename if all succeeded", not from append-only. Think of the case where you run out of disk space; with append-only, it is more complicated to get back to a known good state. > > For inspiration, you might want to look at the "notes" branch in my > > personal fork: > > > > http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes > > Cool. I'd rather copy just that code entirely rather than use it for > inspiration, since it does exactly what I need. It would be silly to > have two blocks of code implementing "persistent map from 20 byte hash > to 20 byte hash". Hehe. That's what I meant by "inspiration" ;-) > I'll start by just copying the entire nodes-index implementation (with a > few name substitutions), and we (or I) can refactor it later if both end > up in the same respository. Very nice! Thanks, Dscho ```