# [GSoC update] git-remote-svn: The final one 9 messages from 2010-08-17 to 2010-08-17. Participants: Ramkumar Ramachandra, Stephen Bash, Erik Faye-Lund, Sverre Rabbelier. Thread: https://gitlist.dev/t/24769 ## Ramkumar Ramachandra, 2010-08-17 14:42 Subject: [GSoC update] git-remote-svn: The final one Message-ID: <20100817144215.GC24886@kytes> URL: https://gitlist.dev/e/20100817144215.GC24886%40kytes ``` Hi, The GSoC is over. Not that it changes anything; I'm still here. Anyway, here's a report; this isn't a movie, so don't skip the credits at the end of the email! Admittedly, I couldn't deliver everything I'd promised in the beginning, mainly because I misjudged the size/ difficulty of the project. While I thought I'd spend most of my time in Git-land converting and mapping histories, I ended up spending a huge amount of time with Subversion writing svnrdump. The upside is that we now have an excellent dumpfile importer/ exporter merged into the Subversion trunk, while the downside is that we don't have a fully functional remote helper for Subversion yet. In all, git-remote-svn involved (involves) the following things: 1. Getting the revision history out of Subversion in a sane format (dumpfile v3). This turned out to be MUCH harder than expected; fortunately, svnrdump does this better than anything else. 2. Converting the revision history from dumpfile v3 format into a git fast-import stream. svn-dump-fast-export does this better than anything else. In fact, we've even come up with a proof that nothing can even theoretically do better. Unfortunately, it only does dumpfile v2 so far, and I still haven't managed to extend it to do dumpfile v3. 3. Getting revision history into Subversion. The load functionality in svnrdump that I wrote after the mid-term does a good job of this. I don't know of any tool that even does something similar. 4. Converting the revision history from a fast-export stream to dumpfile v3. I've just started thinking about this, and haven't started work on it- it's quite a non-trivial challenge. 5. Stitching everything together with a remote helper application. This is much easier than I thought- I wrote a remote helper in one afternoon [1], but didn't get it merged because it's missing the supporting infrastructure. 6. A branch/tag mapper. I dropped this due to lack of time. It's last priority even now, and I'll work on it only when the rest of the infrastructure is in place. I had certain objectives that I'd personally set out to meet in the beginning of the GSoC. Here they are in brief, and my personal evaluation of each one. Do feel free to criticize the approach I've followed and the trade-offs that I've chosen to make. 1. Integration. I was determined to get *everything* merged. I think I've been largely successful on this front. 2. Code quality. In terms of bug-fixing and polish, I'm happy with the current state of the code, thanks to the sheer number of code reviews. 3. Community interaction. I was determined to post atleast once a week to the list, keep everyone in the loop, and remain active all the time. I think I've done justice to this. 4. Off-topic work. This may sound weird; In addition to my GSoC work, I was determined to do some other off-topic work as well to get the hang of contributing and participating in wider discussions. Although I couldn't pay full attention to this due to my heavy workload, I did manage to get some minor OT patches in. 5. Completion. I'm lagging heavily on this front. On a final note, thanks to everyone for helping! Off the top of my head, the following people have been especially awesome (in no particular order). I'm sorry if I've forgotten to include someone (!) 1. Jonathan Nieder - for his painfully detailed reviews, constant cleanups and re-rolls. 2. David Barr - for authoring svn-dump-fast-export, without which a lot of what we've achieved wouldn't have been possible. 3. Daniel Shahaf - for his endless support while I was writing svnrdump. I couldn't have finished it without him. 4. Stefan Sperling - for his detailed review of especially of the first iteration of svnrdump, which got things moving in the first place. 5. Bert Huijben - for his innumerable contributions and killer bugfixes! 6. Sverre Rabbelier - for being an awesome mentor, and being supportive throughout. 7. Will Palmer - for writing command-line parsing UI and a test script for svnrdump. 8. Sam Vilain - for his writeup on the challenges of mapping in Subversion. I realized the challenges involved, and decided not to pursue this in the SoC term. 9. Philip Martin - for authoring svnmucc, which served as the inspiration for the load functionality in svnrdump. 10. Jonas Gehrig - for authoring rsvndump, which served as the inspiration for the dump functionality in svnrdump. 11. Ævar Arnfjörð Bjarmason - for getting me access to a powerful server to run lots of validation tests; I ended up fixing many bugs that I wouldn't have otherwise been able to do. Thanks for reading. [1] http://thread.gmane.org/gmane.comp.version-control.git/147715 -- Ram ``` ## Stephen Bash, 2010-08-17 15:48 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: <15605020.115671.1282060094242.JavaMail.root@mail.hq.genarts.com> URL: https://gitlist.dev/e/15605020.115671.1282060094242.JavaMail.root%40mail.hq.genarts.com In-Reply-To: <20100817144215.GC24886@kytes> ``` Ram- First, congratulations on making it through GSoC. I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue. > 1. Getting the revision history out of Subversion in a sane format > (dumpfile v3). > ... > 2. Converting the revision history from dumpfile v3 format into a git > fast-import stream. > ... > 5. Stitching everything together with a remote helper > application. > ... > 6. A branch/tag mapper. So being selfish, what I care about is an improved method for converting SVN repos to git repos once and never going back. It sounds to me like the major hurdles remaining there are getting svn-fe to handle dumpfile v3 and handling branches/tags. Is that correct? Thanks, and keep up the good work! Stephen ``` ## Ramkumar Ramachandra, 2010-08-17 15:59 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: <20100817155933.GB6947@kytes> URL: https://gitlist.dev/e/20100817155933.GB6947%40kytes In-Reply-To: <15605020.115671.1282060094242.JavaMail.root@mail.hq.genarts.com> ``` Hi Stephen, Stephen Bash writes: > Ram- > > First, congratulations on making it through GSoC. I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue. Thank you :) > > 1. Getting the revision history out of Subversion in a sane format > > (dumpfile v3). > > ... > > 2. Converting the revision history from dumpfile v3 format into a git > > fast-import stream. > > ... > > 5. Stitching everything together with a remote helper > > application. > > ... > > 6. A branch/tag mapper. > > So being selfish, what I care about is an improved method for converting SVN repos to git repos once and never going back. It sounds to me like the major hurdles remaining there are getting svn-fe to handle dumpfile v3 and handling branches/tags. Is that correct? We're doing better than that- when the whole thing is complete, we should have perfectly seamless two-way compatibility. But yes, you're right- there are several non-trivial tasks to handle, the top three of which are: 1. Making svn-fe git-aware (and hence handle dumpfile v3) 2. Writing a fast-export to dumpfilev3 converter 3. Implementing the branch/ tag mapper in an extensible way Don't worry about it though- we're all working hard on it :) -- Ram ``` ## Erik Faye-Lund, 2010-08-17 16:10 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: URL: https://gitlist.dev/e/AANLkTikyrC07OnMJWDmC7n0ks3TdOxb%3DjPmKWbK9Nc7u%40mail.gmail.com In-Reply-To: <20100817155933.GB6947@kytes> ``` On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra wrote: > Stephen Bash writes: >> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue. +1 on the congratulation! I've also been following the progress, but from a distance. As I use git-svn to work with SVN at $dayjob, I can't wait to get something running. > We're doing better than that- when the whole thing is complete, we > should have perfectly seamless two-way compatibility. Does this include handling SVN repos where the layout have changed during it's lifetime? If so, how will this look on the git side? ``` ## Stephen Bash, 2010-08-17 16:20 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: <27168737.115723.1282062015125.JavaMail.root@mail.hq.genarts.com> URL: https://gitlist.dev/e/27168737.115723.1282062015125.JavaMail.root%40mail.hq.genarts.com In-Reply-To: <20100817155933.GB6947@kytes> ``` > > So being selfish, what I care about is an improved method for > > converting SVN repos to git repos once and never going back. It > > sounds to me like the major hurdles remaining there are getting > > svn-fe to handle dumpfile v3 and handling branches/tags. Is that > > correct? > > We're doing better than that- when the whole thing is complete, we > should have perfectly seamless two-way compatibility. That makes complete sense. It just happens I'm interested in a subset of the problem (the order has come down from on high that we will be using git for future development), so I'm curious if/how a one-time conversion simplifies things. > But yes, you're > right- there are several non-trivial tasks to handle, the top three of > which are: > 1. Making svn-fe git-aware (and hence handle dumpfile v3) > 2. Writing a fast-export to dumpfilev3 converter > 3. Implementing the branch/ tag mapper in an extensible way Thanks for clarifying. > Don't worry about it though- we're all working hard on it :) I believe that :) And you're not holding me up; I've spent my spare time this summer coming up with some Perl scripts to work with svn-fe for the one-time conversion, so I'll have my point case solved in the near future. But I'm sure the problem will come up again, and I'll be happy to use the general tool then. Thanks, Stephen ``` ## Sverre Rabbelier, 2010-08-17 16:20 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: URL: https://gitlist.dev/e/AANLkTim5RYDuius6Oa6KX0weQhwVz%2B_YP%3D1-ujb3-CuJ%40mail.gmail.com In-Reply-To: ``` Heya, On Tue, Aug 17, 2010 at 11:10, Erik Faye-Lund wrote: > On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra > wrote: >> We're doing better than that- when the whole thing is complete, we >> should have perfectly seamless two-way compatibility. > > Does this include handling SVN repos where the layout have changed > during it's lifetime? If so, how will this look on the git side? Perhaps it makes sense to start a new thread to hash that out for when the time comes to implement that? -- Cheers, Sverre Rabbelier ``` ## Ramkumar Ramachandra, 2010-08-17 16:22 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: <20100817162233.GD6947@kytes> URL: https://gitlist.dev/e/20100817162233.GD6947%40kytes In-Reply-To: ``` Hi Erik, Erik Faye-Lund writes: > On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra > wrote: > > Stephen Bash writes: > >> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue. > > +1 on the congratulation! I've also been following the progress, but > from a distance. As I use git-svn to work with SVN at $dayjob, I can't > wait to get something running. Thanks :) > > We're doing better than that- when the whole thing is complete, we > > should have perfectly seamless two-way compatibility. > > Does this include handling SVN repos where the layout have changed > during it's lifetime? If so, how will this look on the git side? Yes, but we don't know yet. There's been some speculation about re-implementing the branch/ tag mapper in git-svn on top of git-remote-svn once it's complete. Anyway, Sam's illustration has been most insightful to me on this issue [1]. Will has also started thinking about this and has some preliminary notes outlining the challenges. When the notes are more mature, I'll post them to the list so we can get a wider audience to discuss this at length :) [1] http://article.gmane.org/gmane.comp.version-control.git/150007 -- Ram ``` ## Ramkumar Ramachandra, 2010-08-17 16:41 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: <20100817164125.GE6947@kytes> URL: https://gitlist.dev/e/20100817164125.GE6947%40kytes In-Reply-To: <27168737.115723.1282062015125.JavaMail.root@mail.hq.genarts.com> ``` Hi Stephen, Stephen Bash writes: > > > So being selfish, what I care about is an improved method for > > > converting SVN repos to git repos once and never going back. It > > > sounds to me like the major hurdles remaining there are getting > > > svn-fe to handle dumpfile v3 and handling branches/tags. Is that > > > correct? > > > > We're doing better than that- when the whole thing is complete, we > > should have perfectly seamless two-way compatibility. > > That makes complete sense. It just happens I'm interested in a subset of the problem (the order has come down from on high that we will be using git for future development), so I'm curious if/how a one-time conversion simplifies things. Thanks for bringing this up- it's quite interesting :) A one-time conversion thing would have changed our priorities. Mainly, I wouldn't have been able to justify the time I invested in building svnrdump. I'd have probably settled for complete mirror + dump using a combination of existing tools. The second difference: svn-dump-fast-export is quite complex; although it was probably an unconcious decision on David's part, it makes a really good generic parser, and I suspect that it'll help us generate the dumpfile we need for loading revision history data back into Subversion. Third, it simplifies the branch/ tag mapper greatly- we'd only need to achieve a perfect one-way mapping. A symmetric mapping is harder for several reasons- consider preserving props for instance. -- Ram ``` ## Ramkumar Ramachandra, 2010-08-17 17:18 Subject: Re: [GSoC update] git-remote-svn: The final one Message-ID: <20100817171846.GG6947@kytes> URL: https://gitlist.dev/e/20100817171846.GG6947%40kytes In-Reply-To: ``` Hi Sverre, Sverre Rabbelier writes: > Heya, > > On Tue, Aug 17, 2010 at 11:10, Erik Faye-Lund wrote: > > On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra > > wrote: > >> We're doing better than that- when the whole thing is complete, we > >> should have perfectly seamless two-way compatibility. > > > > Does this include handling SVN repos where the layout have changed > > during it's lifetime? If so, how will this look on the git side? > > Perhaps it makes sense to start a new thread to hash that out for when > the time comes to implement that? Good idea. I've posted a new timeline. I'll put up some branch/ tag mapper notes next week. -- Ram ```