{"thread":{"id":"23175","subject":"Subversion integration with git","startedAt":"2010-03-25T14:17:42Z","lastAt":"2010-03-30T14:29:56Z","messageCount":8,"participants":["David Michael Barr","Erik Faye-Lund","Ramkumar Ramachandra","Pascal Obry"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"137797","messageId":"3311ED3F-77BC-4F66-80D5-AA9E21ACF0E2@cordelta.com","threadId":"23175","inReplyTo":null,"subject":"Subversion integration with git","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-03-25T14:17:42Z","receivedAt":"2010-03-25T14:17:42Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi folks,\n\nAs my first posting to the list, I'd like to start by giving a big thank you to all the git developers and maintainers for such a great tool.\n\nUnfortunately, I still have to interact with lesser tools such as Subversion and that is what leads me to post.\n\nI'm employed on proprietary project which is supported by a large number of open source tools. The 'canonical' source repository is hosted on a Subversion server on the other side of a rather unreliable WAN link. To date I've been using a combination of git-svn, cron, and a handful of bash scripts to handle marshalling commits between our git repositories and the Subversion instance. However, whilst this solution works well for incremental commits, every time a branch is created on the remote repository it's a hassle to synchronise.\nSo I thought I'd use git-svn and standard layout - this resulted in blasting my link with so many HTTP requests that I got a stern warning from our sysadmin and I'm sure the firm on the other side of the link weren't impressed.\nAfter exploring a few solutions I used SVK to create a local mirror of the repository.\n\nWhen I pointed git-svn at the local mirror, it took 4 days, a whole lot of RAM and fell over at 90% completion with a checksum error.\n\nWhen I pointed svn-all-fast-export at the repository it had to skip three commits or would indefinitely spew garbage.\n\nWhen I pointed svn2git.py at a dump of the repository it successfully imported 50% of commits and then ran at snail's pace, ETA next century.\n\nI decided that I liked the idea of subversion dump in - git fast-import out but it had to scale well.\n\nSo I grabbed the git-fast-import documentation and the Subversion dump format documentation and tried to design a data structure that would map well between them and scale linearly with my repository.\n\nI started a new project to implement my design and am curious as to how many git users actually care about this kind of problem. While conversion is once off for most projects - there are an awful number of projects currently using Subversion. As the community and tool-chain builds around git, that will mean many desiring to make the transition. I hope to make it far less painful than it has been for me.\n\nMy project is still in the preview phase but has enough to import commit-tree structure bar symlinks and executable flags. It imports my 22000+ commit 2.8GB dump in 4 minutes. It is currently 840 non-comment lines of C. I aim to produce output that git-svn can take over from.\n\nIs it worthwhile to start a new project - or would it be better to grok the internals of existing projects and try to make them scale?\n\nBest regards,\nDavid M Barr"},{"id":"137798","messageId":"40aa078e1003250731y2c900605k9c681475621a1ff2@mail.gmail.com","threadId":"23175","inReplyTo":"3311ED3F-77BC-4F66-80D5-AA9E21ACF0E2@cordelta.com","subject":"Re: Subversion integration with git","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@googlemail.com","sentAt":"2010-03-25T14:31:57Z","receivedAt":"2010-03-25T14:31:57Z","isPatch":false,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Thu, Mar 25, 2010 at 3:17 PM, David Michael Barr\n<david.barr@cordelta.com> wrote:\n> Hi folks,\n>\n> As my first posting to the list, I'd like to start by giving a big thank you to all the git developers and maintainers for such a great tool.\n>\n> Unfortunately, I still have to interact with lesser tools such as Subversion and that is what leads me to post.\n>\n> I'm employed on proprietary project which is supported by a large number of open source tools. The 'canonical' source repository is hosted on a Subversion server on the other side of a rather unreliable WAN link. To date I've been using a combination of git-svn, cron, and a handful of bash scripts to handle marshalling commits between our git repositories and the Subversion instance. However, whilst this solution works well for incremental commits, every time a branch is created on the remote repository it's a hassle to synchronise.\n> So I thought I'd use git-svn and standard layout - this resulted in blasting my link with so many HTTP requests that I got a stern warning from our sysadmin and I'm sure the firm on the other side of the link weren't impressed.\n> After exploring a few solutions I used SVK to create a local mirror of the repository.\n>\n> When I pointed git-svn at the local mirror, it took 4 days, a whole lot of RAM and fell over at 90% completion with a checksum error.\n>\n> When I pointed svn-all-fast-export at the repository it had to skip three commits or would indefinitely spew garbage.\n>\n> When I pointed svn2git.py at a dump of the repository it successfully imported 50% of commits and then ran at snail's pace, ETA next century.\n>\n> I decided that I liked the idea of subversion dump in - git fast-import out but it had to scale well.\n>\n> So I grabbed the git-fast-import documentation and the Subversion dump format documentation and tried to design a data structure that would map well between them and scale linearly with my repository.\n>\n> I started a new project to implement my design and am curious as to how many git users actually care about this kind of problem. While conversion is once off for most projects - there are an awful number of projects currently using Subversion. As the community and tool-chain builds around git, that will mean many desiring to make the transition. I hope to make it far less painful than it has been for me.\n>\n> My project is still in the preview phase but has enough to import commit-tree structure bar symlinks and executable flags. It imports my 22000+ commit 2.8GB dump in 4 minutes. It is currently 840 non-comment lines of C. I aim to produce output that git-svn can take over from.\n>\n\nWow, your figures sounds very impressive. I'd love to have a look at\nit! I've tried to convert simiar-sized SVN repos before, but given up\ndue to the poor performance. So at work I'm currently using git-svn\nwith only parts of the history imported, and falling back to SVN when\nhaving to dig far in the history (which is not much fun).\n\n> Is it worthwhile to start a new project - or would it be better to grok the internals of existing projects and try to make them scale?\n>\n\nI think it falls very close to the native-git-svn Google SoC\nproject[1], and if you are able to share what you have I'm sure\nRamkumar (I hope you don't mind me CC'ing you, and that I spelled your\nname right) would appreciate having a look.\n\n[1]: https://git.wiki.kernel.org/index.php/SoC2010Ideas#A_remote_helper_for_svn\n\n-- \nErik \"kusma\" Faye-Lund\n"},{"id":"137804","messageId":"f3271551003251052j5863af46i3066f0f42788ba3b@mail.gmail.com","threadId":"23175","inReplyTo":"E560EF9A-AF07-4316-9047-6D1A1802F743@cordelta.com","subject":"Re: Subversion integration with git","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-03-25T17:52:17Z","receivedAt":"2010-03-25T17:52:17Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\n> I think it falls very close to the native-git-svn Google SoC\n> project[1], and if you are able to share what you have I'm sure\n> Ramkumar (I hope you don't mind me CC'ing you, and that I spelled your\n> name right) would appreciate having a look.\n\nYes. Thank you for CC'ing me, Erik.\n\n> Is it worthwhile to start a new project - or would it be better to grok the internals of existing projects and try to make them scale?\n\nHonestly, I've just started looking into this issue, so I'll wait for\nsomeone else to comment on this. As far as interest is concerned, yes-\na lot of people seem to be pretty excited about my GSoC project\nproposal [1]. My proposal has more to do with getting native support\nworking, than building a fantastic SVN importer. I certainly have\nneither the time or experience to build an SVN importer that's any\nbetter than git-svn.perl in the summer term, and I'm clear that I\ndon't intend to do this. However, if my proposal gets accepted, I\ncould work with you to get it integrated into the remote helper that\nI'll be building. Depending on the complexity of your project, this\nmight only be possible at the end of my GSoC term.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/142623\n\n-- Ram\n"},{"id":"137806","messageId":"4BABAA30.3080806@obry.net","threadId":"23175","inReplyTo":"3311ED3F-77BC-4F66-80D5-AA9E21ACF0E2@cordelta.com","subject":"Re: Subversion integration with git","fromName":"Pascal Obry","fromEmail":"pascal@obry.net","sentAt":"2010-03-25T18:23:44Z","receivedAt":"2010-03-25T18:23:44Z","isPatch":false,"sender":{"key":"pascal@obry.net","avatar":"https://avatars.githubusercontent.com/u/467069?v=4"},"body":"David,\n\n> My project is still in the preview phase but has enough to import \n> commit-tree structure bar symlinks and executable flags. It imports \n> my 22000+ commit 2.8GB dump in 4 minutes. It is currently 840 \n> non-comment lines of C. I aim to produce output that git-svn can\n> take over from.\n\nImpressive numbers! I've converted many projects using git-svn and yes\nit is slow. Just curious, does it handles branches? Can it handles not\nstandard layout (trunk/branch/tags)? When you have a git-svn compatible\noutput I would be willing to test it on a project.\n\nPascal.\n\n-- \n\n--|------------------------------------------------------\n--| Pascal Obry                           Team-Ada Member\n--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE\n--|------------------------------------------------------\n--|    http://www.obry.net  -  http://v2p.fr.eu.org\n--| \"The best way to travel is by means of imagination\"\n--|\n--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B\n"},{"id":"137818","messageId":"4170AB98-B7C4-4351-B793-351B3DE74A4F@cordelta.com","threadId":"23175","inReplyTo":"f3271551003251052j5863af46i3066f0f42788ba3b@mail.gmail.com","subject":"Re: Subversion integration with git","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-03-25T23:50:26Z","receivedAt":"2010-03-25T23:50:26Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi,\n\n>> I think it falls very close to the native-git-svn Google SoC\n>> project[1], and if you are able to share what you have I'm sure\n>> Ramkumar (I hope you don't mind me CC'ing you, and that I spelled your\n>> name right) would appreciate having a look.\n> \n> Yes. Thank you for CC'ing me, Erik.\n> \n>> Is it worthwhile to start a new project - or would it be better to grok the internals of existing projects and try to make them scale?\n> \n> However, if my proposal gets accepted, I\n> could work with you to get it integrated into the remote helper that\n> I'll be building. Depending on the complexity of your project, this\n> might only be possible at the end of my GSoC term.\n\n>From Ramkumar's proposal[1]:\n\n> The distinct components I plan to write are:\n\n> 2. An exporter for SVN repositories, which will extract all the\n> relevant revision history and metadata to import into Git.\n> 3. A remote helper for Git that takes the data from this SVN exporter,\n> and uses git-fast-import to create corresponding commits in Git.\n\nThe scope of my project roughly corresponds to these two components.\n\nWith regard to licensing issues, I opted to work only with import/export streams so that no linking is required.\n\nIn the context of GSoC, I am studying my final year of a Bachelor of Science (Computer Science x 2) at Australian National University, Canberra.\nAs for FOSS contributions, the last time I released something was when I helped port the iBurst wireless broadband driver (USB interface) [2] a few years ago; the demands of life vs. one's passion.\n"},{"id":"138003","messageId":"4F70255B-6FCC-4020-B993-A15E6DC0E774@cordelta.com","threadId":"23175","inReplyTo":"4BABAA30.3080806@obry.net","subject":"Re: Subversion integration with git","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-03-28T12:03:28Z","receivedAt":"2010-03-28T12:03:28Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Pascal,\n\n>> My project is still in the preview phase but has enough to import \n>> commit-tree structure bar symlinks and executable flags. It imports \n>> my 22000+ commit 2.8GB dump in 4 minutes. It is currently 840 \n>> non-comment lines of C. I aim to produce output that git-svn can\n>> take over from.\n> \n> Impressive numbers! I've converted many projects using git-svn and yes\n> it is slow. Just curious, does it handles branches? Can it handles not\n> standard layout (trunk/branch/tags)? When you have a git-svn compatible\n> output I would be willing to test it on a project.\n\nMy initial design target is a one-to-one translation of the subversion history to a single linear git branch. I'm working under the assumption that something like git filter-branch can be used to transform the history to a more logical representation. This should allow any subversion layout to be handled.\n\nWhen I have git-svn compatible output, I'll proudly announce the first release.\n"},{"id":"138178","messageId":"52A6E51B-058C-4B06-A271-26219B388BE7@cordelta.com","threadId":"23175","inReplyTo":"f3271551003251052j5863af46i3066f0f42788ba3b@mail.gmail.com","subject":"Re: Subversion integration with git","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-03-30T14:05:10Z","receivedAt":"2010-03-30T14:05:10Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi,\n\n>> I think it falls very close to the native-git-svn Google SoC\n>> project[1], and if you are able to share what you have I'm sure\n>> Ramkumar (I hope you don't mind me CC'ing you, and that I spelled your\n>> name right) would appreciate having a look.\n> \n> Yes. Thank you for CC'ing me, Erik.\n> \n>> Is it worthwhile to start a new project - or would it be better to grok the internals of existing projects and try to make them scale?\n> \n> ... if my proposal gets accepted, I\n> could work with you to get it integrated into the remote helper that\n> I'll be building. Depending on the complexity of your project, this\n> might only be possible at the end of my GSoC term.\n> \n> \n> \n> -- Ram\n\nI've started looking at the first piece of the pipeline, reading from a\nremote subversion URL. I stumbled upon rsvndump[2], which is\nGPLv3+ licensed and promises to produce a Subversion dump from\na remote repository. This could be piped to my utility,\nsvn-dump-fast-export[3], to produce suitable input for git fast-import.\nI believe this would address the first two components of Ram's.\nproposal and allow more focus to be given to the interesting ones.\nThat's presuming that I have a feature-complete release by the time\nthe GSoC project begins.\n\nMy project is currently under a two-clause BSD style license.\nThis is primarily because the two projects it derives from were\ndistributed under the same license, rather than any preference.\n\nAs I've included a reference to my project, I'll emphasise that it is a\nwork in progress, with a handful of known bugs.\nAt present, symlinks are damaged on update and some files\ndisappear late in the history of my test repository.\nI'm planning a rewrite of the parser once symlinks are complete.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/142623\n[2] http://rsvndump.sourceforge.net/\n[3] http://github.com/barrbrain/svn-dump-fast-export/\n"},{"id":"138182","messageId":"f3271551003300729i340ee031r3151cad845e299be@mail.gmail.com","threadId":"23175","inReplyTo":"52A6E51B-058C-4B06-A271-26219B388BE7@cordelta.com","subject":"Re: Subversion integration with git","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-03-30T14:29:56Z","receivedAt":"2010-03-30T14:29:56Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nOn Tue, Mar 30, 2010 at 7:35 PM, David Michael Barr\n<david.barr@cordelta.com> wrote:\n> I've started looking at the first piece of the pipeline, reading from a\n> remote subversion URL. I stumbled upon rsvndump[2], which is\n> GPLv3+ licensed and promises to produce a Subversion dump from\n> a remote repository. This could be piped to my utility,\n> svn-dump-fast-export[3], to produce suitable input for git fast-import.\n\nIn the latest version of my proposal, I've proposed to write a\nmirroring tool which will basically be a stripped down version of\nsvnsync [1]. For testing, I therefore recommend that you use svnsync\ninstead of rsvndump because the former is included in the official\nsource tree. If however, you do a comparison and find that rsvndump is\nactually a better alternative, let me know.\n\n> I believe this would address the first two components of Ram's.\n> proposal and allow more focus to be given to the interesting ones.\n> That's presuming that I have a feature-complete release by the time\n> the GSoC project begins.\n\nThat's fantastic! I suspect you haven't looked at the latest revision\nof my proposal. Since Gmane doesn't seem to have indexed it, I just\nsent you a mail off the list.\n\n-- Ram\n"}]}