{"thread":{"id":"13721","subject":"caching commit patch-ids for fast git-cherry","startedAt":"2008-05-29T17:01:58Z","lastAt":"2008-05-29T22:19:56Z","messageCount":4,"participants":["Geoffrey Irving","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"78060","messageId":"7f9d599f0805291001mdbb4b42q6f3a1b79bc9bc4e9@mail.gmail.com","threadId":"13721","inReplyTo":null,"subject":"caching commit patch-ids for fast git-cherry","fromName":"Geoffrey Irving","fromEmail":"irving@naml.us","sentAt":"2008-05-29T17:01:58Z","receivedAt":"2008-05-29T17:01:58Z","isPatch":false,"sender":{"key":"irving@naml.us","avatar":"https://gravatar.com/avatar/52d7452fcd134aac0fa12f57a3bb7ef5f3f7e73ca0ab36736d06c6a6132de718?d=mp&s=160"},"body":"I'm planning to use cherry picking to manage long term syncing between\ncvs/perforce and git repositories. This means I'll have scripts\nrunning git-cherry between branches with hundreds of uncommon commits,\nand I want git-cherry to be much, much, faster.\n\nIt looks like I can do this by caching commit->patch-id pairs from\ncommit_patch_id() in patch-ids.c to a file, say\n$GIT_DIR/commit-patch-id-cache. The file would be binary and append\nonly, and could be blown away if . Any suggestions / concerns before\nI write this? Is there any reusable efficient map code for storing\nthe commit->patch-id map, or should I just mirror the blocked storage\n+ binary search used for struct patch_ids?\n\nThanks,\nGeoffrey\n"},{"id":"78061","messageId":"alpine.DEB.1.00.0805291809340.13507@racer.site.net","threadId":"13721","inReplyTo":"7f9d599f0805291001mdbb4b42q6f3a1b79bc9bc4e9@mail.gmail.com","subject":"Re: caching commit patch-ids for fast git-cherry","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-05-29T17:13:24Z","receivedAt":"2008-05-29T17:13:24Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 29 May 2008, Geoffrey Irving wrote:\n\n> I'm planning to use cherry picking to manage long term syncing between \n> cvs/perforce and git repositories. This means I'll have scripts running \n> git-cherry between branches with hundreds of uncommon commits, and I \n> want git-cherry to be much, much, faster.\n> \n> It looks like I can do this by caching commit->patch-id pairs from\n> commit_patch_id() in patch-ids.c to a file, say\n> $GIT_DIR/commit-patch-id-cache. The file would be binary and append\n> only, and could be blown away if . Any suggestions / concerns before\n> I write this? Is there any reusable efficient map code for storing\n> the commit->patch-id map, or should I just mirror the blocked storage\n> + binary search used for struct patch_ids?\n\nI would store the stuff sorted, so that the lookup is fast, generation \nless so.\n\nFor inspiration, you might want to look at the \"notes\" branch in my \npersonal fork:\n\nhttp://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes\n\nHth,\nDscho\n"},{"id":"78066","messageId":"7f9d599f0805291034l6c655ccbk219dd74964c65737@mail.gmail.com","threadId":"13721","inReplyTo":"alpine.DEB.1.00.0805291809340.13507@racer.site.net","subject":"Re: caching commit patch-ids for fast git-cherry","fromName":"Geoffrey Irving","fromEmail":"irving@naml.us","sentAt":"2008-05-29T17:34:10Z","receivedAt":"2008-05-29T17:34:10Z","isPatch":false,"sender":{"key":"irving@naml.us","avatar":"https://gravatar.com/avatar/52d7452fcd134aac0fa12f57a3bb7ef5f3f7e73ca0ab36736d06c6a6132de718?d=mp&s=160"},"body":"On Thu, May 29, 2008 at 10:13 AM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> On Thu, 29 May 2008, Geoffrey Irving wrote:\n>\n>> I'm planning to use cherry picking to manage long term syncing between\n>> cvs/perforce and git repositories. This means I'll have scripts running\n>> git-cherry between branches with hundreds of uncommon commits, and I\n>> want git-cherry to be much, much, faster.\n>>\n>> It looks like I can do this by caching commit->patch-id pairs from\n>> commit_patch_id() in patch-ids.c to a file, say\n>> $GIT_DIR/commit-patch-id-cache. The file would be binary and append\n>> only, and could be blown away if . Any suggestions / concerns before\n>> I write this? Is there any reusable efficient map code for storing\n>> the commit->patch-id map, or should I just mirror the blocked storage\n>> + binary search used for struct patch_ids?\n>\n> I would store the stuff sorted, so that the lookup is fast, generation\n> less so.\n\nThe motivation for append-only was robustness, not speed, but I don't\nthink either concern is very significant.\n\n> For inspiration, you might want to look at the \"notes\" branch in my\n> personal fork:\n>\n> http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes\n\nCool. I'd rather copy just that code entirely rather than use it for\ninspiration, since it does exactly what I need. It would be silly to\nhave two blocks of code implementing \"persistent map from 20 byte hash\nto 20 byte hash\".\n\nI'll start by just copying the entire nodes-index implementation (with\na few name substitutions), and we (or I) can refactor it later if both\nend up in the same respository.\n\nGeoffrey\n"},{"id":"78092","messageId":"alpine.DEB.1.00.0805292318040.13507@racer.site.net","threadId":"13721","inReplyTo":"7f9d599f0805291034l6c655ccbk219dd74964c65737@mail.gmail.com","subject":"Re: caching commit patch-ids for fast git-cherry","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-05-29T22:19:56Z","receivedAt":"2008-05-29T22:19:56Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 29 May 2008, Geoffrey Irving wrote:\n\n> On Thu, May 29, 2008 at 10:13 AM, Johannes Schindelin\n> wrote:\n> > Hi,\n> >\n> > On Thu, 29 May 2008, Geoffrey Irving wrote:\n> >\n> >> I'm planning to use cherry picking to manage long term syncing \n> >> between cvs/perforce and git repositories. This means I'll have \n> >> scripts running git-cherry between branches with hundreds of uncommon \n> >> commits, and I want git-cherry to be much, much, faster.\n> >>\n> >> It looks like I can do this by caching commit->patch-id pairs from \n> >> commit_patch_id() in patch-ids.c to a file, say \n> >> $GIT_DIR/commit-patch-id-cache. The file would be binary and append \n> >> only, and could be blown away if . Any suggestions / concerns before \n> >> I write this? Is there any reusable efficient map code for storing \n> >> the commit->patch-id map, or should I just mirror the blocked storage\n> >> + binary search used for struct patch_ids?\n> >\n> > I would store the stuff sorted, so that the lookup is fast, generation \n> > less so.\n> \n> The motivation for append-only was robustness, not speed, but I don't \n> think either concern is very significant.\n\nI think that robustness comes from \"write new file and rename if all \nsucceeded\", not from append-only. Think of the case where you run out of \ndisk space; with append-only, it is more complicated to get back to a \nknown good state.\n \n> > For inspiration, you might want to look at the \"notes\" branch in my \n> > personal fork:\n> >\n> > http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/notes\n> \n> Cool. I'd rather copy just that code entirely rather than use it for \n> inspiration, since it does exactly what I need. It would be silly to \n> have two blocks of code implementing \"persistent map from 20 byte hash \n> to 20 byte hash\".\n\nHehe. That's what I meant by \"inspiration\" ;-)\n\n> I'll start by just copying the entire nodes-index implementation (with a \n> few name substitutions), and we (or I) can refactor it later if both end \n> up in the same respository.\n\nVery nice!\n\nThanks,\nDscho\n"}]}