{"thread":{"id":"19530","subject":"git-svn: importing internal externals","startedAt":"2009-05-27T16:51:29Z","lastAt":"2009-06-10T21:58:55Z","messageCount":4,"participants":["John Koleszar","Yann Dirson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"114812","messageId":"1243443089.3192.43.camel@cp-jk-linux.corp.on2.com","threadId":"19530","inReplyTo":null,"subject":"git-svn: importing internal externals","fromName":"John Koleszar","fromEmail":"john.koleszar@on2.com","sentAt":"2009-05-27T16:51:29Z","receivedAt":"2009-05-27T16:51:29Z","isPatch":false,"sender":{"key":"john.koleszar@on2.com","avatar":null},"body":"Hi,\n\nI'm working on a one-off import of an SVN repo that makes use of\n\"internal\" svn:externals; i.e. all URLs refer to different spots in the\nsame repo, potentially with peg revisions. The SVN repo holds a number\nof projects, and my plan is to import them into individual git repos,\nincorporating the history from any svn:external linked sub-projects.\n\nMy current strategy is to let git-svn fetch the project as it normally\nwould, then fix up the history on each branch by parsing the\nunhandled.log with a filter-branch script. I'm currently using an index\nfilter, which works reasonably well, but has the undesired effect of\nsquashing commits on the subproject into commits on the parent project.\nSo if the subproject is modified in rA and rC, and the project in rB and\nrD, my modified history shows only rB and rD, with rA squashed into rB\nand rC squashed into rD, when I'd really like to see all 4 commits.\n\nI have a git-svn clone of the whole repo available, and I use that to\npull out the missing external objects when recreating history. Maybe a\nbetter idea would be to resolve the externals with my index filter on\nthis whole super-repo, where the history is properly linear, then\nsomehow filter the now-populated projects from that. I had originally\nset aside this idea because I wanted git-svn to do its automagic\nbranch/tag extraction, and while I think I could get a git repo with the\nsvn layout, I'm unsure how to turn that into proper git branches and\ntags.\n\nCan anyone offer any suggestions on how to achieve this? I'm still new\nto git, so nothing is jumping out at me. I'd be happy to share my filter\nscript (about 100 lines) if someone wants to see it.\n\nThanks!\n\nJohn\n"},{"id":"114917","messageId":"20090528112542.GA24403@nan92-1-81-57-214-146.fbx.proxad.net","threadId":"19530","inReplyTo":"1243443089.3192.43.camel@cp-jk-linux.corp.on2.com","subject":"Re: git-svn: importing internal externals","fromName":"Yann Dirson","fromEmail":"ydirson@altern.org","sentAt":"2009-05-28T11:25:43Z","receivedAt":"2009-05-28T11:25:43Z","isPatch":false,"sender":{"key":"ydirson@altern.org","avatar":"https://avatars.githubusercontent.com/u/1190950?v=4"},"body":"On Wed, May 27, 2009 at 12:51:29PM -0400, John Koleszar wrote:\n> Hi,\n> \n> I'm working on a one-off import of an SVN repo that makes use of\n> \"internal\" svn:externals; i.e. all URLs refer to different spots in the\n> same repo, potentially with peg revisions. The SVN repo holds a number\n> of projects, and my plan is to import them into individual git repos,\n> incorporating the history from any svn:external linked sub-projects.\n\nI have started to work on exactly this, at fetch time instead of as a\npost-process.  I have for now only hooked parsing of the svn:externals\nproperties, and just need to find the time to resume and finish.\n\nMy plan on the user side is to provide flags to map svn urls to git urls.\n\n\n> My current strategy is to let git-svn fetch the project as it normally\n> would, then fix up the history on each branch by parsing the\n> unhandled.log with a filter-branch script. I'm currently using an index\n> filter, which works reasonably well, but has the undesired effect of\n> squashing commits on the subproject into commits on the parent project.\n> So if the subproject is modified in rA and rC, and the project in rB and\n> rD, my modified history shows only rB and rD, with rA squashed into rB\n> and rC squashed into rD, when I'd really like to see all 4 commits.\n\nYes, this is the main issue for correctness.  For this we would have to\ncheck, before processing any new commit, if any of the submodules got any\nintervening commits, and commit them first.\n\n\n> I'd be happy to share my filter\n> script (about 100 lines) if someone wants to see it.\n\nIt can be a good idea to share your script nevertheless :)\n\nBest regards,\n-- \nYann\n"},{"id":"115049","messageId":"1243631119.6276.46.camel@cp-jk-linux.corp.on2.com","threadId":"19530","inReplyTo":"20090528112542.GA24403@nan92-1-81-57-214-146.fbx.proxad.net","subject":"Re: git-svn: importing internal externals","fromName":"John Koleszar","fromEmail":"john.koleszar@on2.com","sentAt":"2009-05-29T21:05:19Z","receivedAt":"2009-05-29T21:05:19Z","isPatch":false,"sender":{"key":"john.koleszar@on2.com","avatar":null},"body":"On Thu, 2009-05-28 at 07:25 -0400, Yann Dirson wrote:\n> On Wed, May 27, 2009 at 12:51:29PM -0400, John Koleszar wrote:\n> > Hi,\n> > \n> > I'm working on a one-off import of an SVN repo that makes use of\n> > \"internal\" svn:externals; i.e. all URLs refer to different spots in the\n> > same repo, potentially with peg revisions. The SVN repo holds a number\n> > of projects, and my plan is to import them into individual git repos,\n> > incorporating the history from any svn:external linked sub-projects.\n> \n[...]\n> It can be a good idea to share your script nevertheless :)\n> \n\nI hacked on this some more and got something pretty usable (for me). It\noperates on a git-svn clone of the whole repository, propagates commits\nto different paths if referenced by an external, rearranges the tree to\nisolate each svn branch on its own head, reparents the branches to their\nproper branch points, and converts any tags branches to real git tags.\n\nDon't know what the netequitte is on this list regarding attachments for\nthis sort of thing, so I posted it here:\nhttp://github.com/jkoleszar/git-svn-internal-externals/tree/master\n\nIt's not as fast as I'd like, but it's workable, at least for small\nrepositories. Bottleneck seems to be git-update-index (100s of ms/call)\nbut I haven't looked into it too much. I'm sure I could be smarter in\nsome of my pipelines too. Some numbers (2246 revisions, ~15k files,\nCore2 6600 @ 2.4GHz, tmpfs):\n\ngit-svn fetch:\n197.03user 174.63system 22:36.59elapsed 27%CPU (0avgtext+0avgdata\n0maxresident)k 0inputs+0outputs (0major+35448577minor)pagefaults 0swaps\n\npropagating externals:\n1381.29user 744.42system 34:28.67elapsed 102%CPU (0avgtext+0avgdata\n0maxresident)k 0inputs+0outputs (2major+305234667minor)pagefaults 0swaps\n\nrearranging heads:\n46.13user 64.23system 1:52.42elapsed 98%CPU (0avgtext+0avgdata\n0maxresident)k 0inputs+0outputs (4major+28752709minor)pagefaults 0swaps\n\nreparenting branches:\n151.52user 263.50system 6:19.54elapsed 109%CPU (0avgtext+0avgdata\n0maxresident)k 0inputs+0outputs (2major+135830914minor)pagefaults 0swaps\n\nHope this is useful for someone!\n\nJohn\n"},{"id":"116042","messageId":"20090610215854.GA5741@nan92-1-81-57-214-146.fbx.proxad.net","threadId":"19530","inReplyTo":"20090528112542.GA24403@nan92-1-81-57-214-146.fbx.proxad.net","subject":"Re: git-svn: importing internal externals","fromName":"Yann Dirson","fromEmail":"ydirson@altern.org","sentAt":"2009-06-10T21:58:55Z","receivedAt":"2009-06-10T21:58:55Z","isPatch":false,"sender":{"key":"ydirson@altern.org","avatar":"https://avatars.githubusercontent.com/u/1190950?v=4"},"body":"On Thu, May 28, 2009 at 01:25:43PM +0200, Yann Dirson wrote:\n> On Wed, May 27, 2009 at 12:51:29PM -0400, John Koleszar wrote:\n> > Hi,\n> > \n> > I'm working on a one-off import of an SVN repo that makes use of\n> > \"internal\" svn:externals; i.e. all URLs refer to different spots in the\n> > same repo, potentially with peg revisions. The SVN repo holds a number\n> > of projects, and my plan is to import them into individual git repos,\n> > incorporating the history from any svn:external linked sub-projects.\n> \n> I have started to work on exactly this, at fetch time instead of as a\n> post-process.  I have for now only hooked parsing of the svn:externals\n> properties, and just need to find the time to resume and finish.\n> \n> My plan on the user side is to provide flags to map svn urls to git urls.\n\nJust a quick note to make it public that my WIP on this issue is\navailable from the t/svn-externals branch at\nhttp://repo.or.cz/w/git/ydirson.git.  Progressing slowly (as time\npermits with a 3-children family and a real-life job ;), but still\nprogressing.\n\nThis is targetted at incremental conversion - if one looks for\none-shot imports, John's script is today the way to go.\n\nBest regards,\n-- \nYann\n"}]}