{"thread":{"id":"24769","subject":"[GSoC update] git-remote-svn: The final one","startedAt":"2010-08-17T14:42:18Z","lastAt":"2010-08-17T17:18:49Z","messageCount":9,"participants":["Ramkumar Ramachandra","Stephen Bash","Erik Faye-Lund","Sverre Rabbelier"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"148278","messageId":"20100817144215.GC24886@kytes","threadId":"24769","inReplyTo":null,"subject":"[GSoC update] git-remote-svn: The final one","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-08-17T14:42:18Z","receivedAt":"2010-08-17T14:42:18Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nThe GSoC is over. Not that it changes anything; I'm still\nhere. Anyway, here's a report; this isn't a movie, so don't skip the\ncredits at the end of the email!\n\nAdmittedly, I couldn't deliver everything I'd promised in the\nbeginning, mainly because I misjudged the size/ difficulty of the\nproject. While I thought I'd spend most of my time in Git-land\nconverting and mapping histories, I ended up spending a huge amount of\ntime with Subversion writing svnrdump. The upside is that we now have\nan excellent dumpfile importer/ exporter merged into the Subversion\ntrunk, while the downside is that we don't have a fully functional\nremote helper for Subversion yet. In all, git-remote-svn involved\n(involves) the following things:\n\n1. Getting the revision history out of Subversion in a sane format\n   (dumpfile v3). This turned out to be MUCH harder than expected;\n   fortunately, svnrdump does this better than anything else.\n2. Converting the revision history from dumpfile v3 format into a git\n   fast-import stream. svn-dump-fast-export does this better than\n   anything else. In fact, we've even come up with a proof that\n   nothing can even theoretically do better. Unfortunately, it only\n   does dumpfile v2 so far, and I still haven't managed to extend it\n   to do dumpfile v3.\n3. Getting revision history into Subversion. The load functionality in\n   svnrdump that I wrote after the mid-term does a good job of this. I\n   don't know of any tool that even does something similar.\n4. Converting the revision history from a fast-export stream to\n   dumpfile v3. I've just started thinking about this, and haven't\n   started work on it- it's quite a non-trivial challenge.\n5. Stitching everything together with a remote helper\n   application. This is much easier than I thought- I wrote a remote\n   helper in one afternoon [1], but didn't get it merged because it's\n   missing the supporting infrastructure.\n6. A branch/tag mapper. I dropped this due to lack of time. It's last\n   priority even now, and I'll work on it only when the rest of the\n   infrastructure is in place.\n\nI had certain objectives that I'd personally set out to meet in the\nbeginning of the GSoC. Here they are in brief, and my personal\nevaluation of each one. Do feel free to criticize the approach I've\nfollowed and the trade-offs that I've chosen to make.\n\n1. Integration. I was determined to get *everything* merged. I think\n   I've been largely successful on this front.\n2. Code quality. In terms of bug-fixing and polish, I'm happy with the\n   current state of the code, thanks to the sheer number of code\n   reviews.\n3. Community interaction. I was determined to post atleast once a week\n   to the list, keep everyone in the loop, and remain active all the\n   time. I think I've done justice to this.\n4. Off-topic work. This may sound weird; In addition to my GSoC work, I\n   was determined to do some other off-topic work as well to get the\n   hang of contributing and participating in wider\n   discussions. Although I couldn't pay full attention to this due to\n   my heavy workload, I did manage to get some minor OT patches in.\n5. Completion. I'm lagging heavily on this front.\n\nOn a final note, thanks to everyone for helping! Off the top of my\nhead, the following people have been especially awesome (in no\nparticular order). I'm sorry if I've forgotten to include someone (!)\n\n1. Jonathan Nieder - for his painfully detailed reviews, constant\n   cleanups and re-rolls.\n2. David Barr - for authoring svn-dump-fast-export, without which a\n   lot of what we've achieved wouldn't have been possible.\n3. Daniel Shahaf - for his endless support while I was writing\n   svnrdump. I couldn't have finished it without him.\n4. Stefan Sperling - for his detailed review of especially of the\n   first iteration of svnrdump, which got things moving in the first\n   place.\n5. Bert Huijben - for his innumerable contributions and killer\n   bugfixes!\n6. Sverre Rabbelier - for being an awesome mentor, and being\n   supportive throughout.\n7. Will Palmer - for writing command-line parsing UI and a test script\n   for svnrdump.\n8. Sam Vilain - for his writeup on the challenges of mapping in\n   Subversion. I realized the challenges involved, and decided not to\n   pursue this in the SoC term.\n9. Philip Martin - for authoring svnmucc, which served as the\n   inspiration for the load functionality in svnrdump.\n10. Jonas Gehrig - for authoring rsvndump, which served as the\n    inspiration for the dump functionality in svnrdump.\n11. Ævar Arnfjörð Bjarmason - for getting me access to a powerful\n    server to run lots of validation tests; I ended up fixing many\n    bugs that I wouldn't have otherwise been able to do.\n\nThanks for reading.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/147715\n\n-- Ram\n"},{"id":"148283","messageId":"15605020.115671.1282060094242.JavaMail.root@mail.hq.genarts.com","threadId":"24769","inReplyTo":"20100817144215.GC24886@kytes","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2010-08-17T15:48:14Z","receivedAt":"2010-08-17T15:48:14Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"Ram-\n\nFirst, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.\n\n> 1. Getting the revision history out of Subversion in a sane format\n> (dumpfile v3).\n>      ...\n> 2. Converting the revision history from dumpfile v3 format into a git\n> fast-import stream.\n>      ...\n> 5. Stitching everything together with a remote helper\n> application.\n>      ...\n> 6. A branch/tag mapper.\n\nSo being selfish, what I care about is an improved method for converting SVN repos to git repos once and never going back.  It sounds to me like the major hurdles remaining there are getting svn-fe to handle dumpfile v3 and handling branches/tags.  Is that correct?\n\nThanks, and keep up the good work!\n\nStephen\n"},{"id":"148284","messageId":"20100817155933.GB6947@kytes","threadId":"24769","inReplyTo":"15605020.115671.1282060094242.JavaMail.root@mail.hq.genarts.com","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-08-17T15:59:37Z","receivedAt":"2010-08-17T15:59:37Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Stephen,\n\nStephen Bash writes:\n> Ram-\n> \n> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.\n\nThank you :)\n\n> > 1. Getting the revision history out of Subversion in a sane format\n> > (dumpfile v3).\n> >      ...\n> > 2. Converting the revision history from dumpfile v3 format into a git\n> > fast-import stream.\n> >      ...\n> > 5. Stitching everything together with a remote helper\n> > application.\n> >      ...\n> > 6. A branch/tag mapper.\n> \n> So being selfish, what I care about is an improved method for converting SVN repos to git repos once and never going back.  It sounds to me like the major hurdles remaining there are getting svn-fe to handle dumpfile v3 and handling branches/tags.  Is that correct?\n\nWe're doing better than that- when the whole thing is complete, we\nshould have perfectly seamless two-way compatibility. But yes, you're\nright- there are several non-trivial tasks to handle, the top three of\nwhich are:\n1. Making svn-fe git-aware (and hence handle dumpfile v3)\n2. Writing a fast-export to dumpfilev3 converter\n3. Implementing the branch/ tag mapper in an extensible way\n\nDon't worry about it though- we're all working hard on it :)\n\n-- Ram\n"},{"id":"148285","messageId":"AANLkTikyrC07OnMJWDmC7n0ks3TdOxb=jPmKWbK9Nc7u@mail.gmail.com","threadId":"24769","inReplyTo":"20100817155933.GB6947@kytes","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2010-08-17T16:10:39Z","receivedAt":"2010-08-17T16:10:39Z","isPatch":false,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra\n<artagnon@gmail.com> wrote:\n> Stephen Bash writes:\n>> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.\n\n+1 on the congratulation! I've also been following the progress, but\nfrom a distance. As I use git-svn to work with SVN at $dayjob, I can't\nwait to get something running.\n\n> We're doing better than that- when the whole thing is complete, we\n> should have perfectly seamless two-way compatibility.\n\nDoes this include handling SVN repos where the layout have changed\nduring it's lifetime? If so, how will this look on the git side?\n"},{"id":"148286","messageId":"27168737.115723.1282062015125.JavaMail.root@mail.hq.genarts.com","threadId":"24769","inReplyTo":"20100817155933.GB6947@kytes","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2010-08-17T16:20:15Z","receivedAt":"2010-08-17T16:20:15Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"> > So being selfish, what I care about is an improved method for\n> > converting SVN repos to git repos once and never going back. It\n> > sounds to me like the major hurdles remaining there are getting\n> > svn-fe to handle dumpfile v3 and handling branches/tags. Is that\n> > correct?\n> \n> We're doing better than that- when the whole thing is complete, we\n> should have perfectly seamless two-way compatibility.\n\nThat makes complete sense.  It just happens I'm interested in a subset of the problem (the order has come down from on high that we will be using git for future development), so I'm curious if/how a one-time conversion simplifies things.\n\n> But yes, you're\n> right- there are several non-trivial tasks to handle, the top three of\n> which are:\n> 1. Making svn-fe git-aware (and hence handle dumpfile v3)\n> 2. Writing a fast-export to dumpfilev3 converter\n> 3. Implementing the branch/ tag mapper in an extensible way\n\nThanks for clarifying.\n\n> Don't worry about it though- we're all working hard on it :)\n\nI believe that :)  And you're not holding me up; I've spent my spare time this summer coming up with some Perl scripts to work with svn-fe for the one-time conversion, so I'll have my point case solved in the near future.  But I'm sure the problem will come up again, and I'll be happy to use the general tool then.\n\nThanks,\nStephen\n"},{"id":"148287","messageId":"AANLkTim5RYDuius6Oa6KX0weQhwVz+_YP=1-ujb3-CuJ@mail.gmail.com","threadId":"24769","inReplyTo":"AANLkTikyrC07OnMJWDmC7n0ks3TdOxb=jPmKWbK9Nc7u@mail.gmail.com","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-08-17T16:20:39Z","receivedAt":"2010-08-17T16:20:39Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, Aug 17, 2010 at 11:10, Erik Faye-Lund <kusmabite@gmail.com> wrote:\n> On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra\n> <artagnon@gmail.com> wrote:\n>> We're doing better than that- when the whole thing is complete, we\n>> should have perfectly seamless two-way compatibility.\n>\n> Does this include handling SVN repos where the layout have changed\n> during it's lifetime? If so, how will this look on the git side?\n\nPerhaps it makes sense to start a new thread to hash that out for when\nthe time comes to implement that?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"148288","messageId":"20100817162233.GD6947@kytes","threadId":"24769","inReplyTo":"AANLkTikyrC07OnMJWDmC7n0ks3TdOxb=jPmKWbK9Nc7u@mail.gmail.com","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-08-17T16:22:37Z","receivedAt":"2010-08-17T16:22:37Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Erik,\n\nErik Faye-Lund writes:\n> On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra\n> <artagnon@gmail.com> wrote:\n> > Stephen Bash writes:\n> >> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.\n> \n> +1 on the congratulation! I've also been following the progress, but\n> from a distance. As I use git-svn to work with SVN at $dayjob, I can't\n> wait to get something running.\n\nThanks :)\n\n> > We're doing better than that- when the whole thing is complete, we\n> > should have perfectly seamless two-way compatibility.\n> \n> Does this include handling SVN repos where the layout have changed\n> during it's lifetime? If so, how will this look on the git side?\n\nYes, but we don't know yet. There's been some speculation about\nre-implementing the branch/ tag mapper in git-svn on top of\ngit-remote-svn once it's complete. Anyway, Sam's illustration has been\nmost insightful to me on this issue [1]. Will has also started\nthinking about this and has some preliminary notes outlining the\nchallenges. When the notes are more mature, I'll post them to the list\nso we can get a wider audience to discuss this at length :)\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/150007\n\n-- Ram\n"},{"id":"148289","messageId":"20100817164125.GE6947@kytes","threadId":"24769","inReplyTo":"27168737.115723.1282062015125.JavaMail.root@mail.hq.genarts.com","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-08-17T16:41:27Z","receivedAt":"2010-08-17T16:41:27Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Stephen,\n\nStephen Bash writes:\n> > > So being selfish, what I care about is an improved method for\n> > > converting SVN repos to git repos once and never going back. It\n> > > sounds to me like the major hurdles remaining there are getting\n> > > svn-fe to handle dumpfile v3 and handling branches/tags. Is that\n> > > correct?\n> > \n> > We're doing better than that- when the whole thing is complete, we\n> > should have perfectly seamless two-way compatibility.\n> \n> That makes complete sense.  It just happens I'm interested in a subset of the problem (the order has come down from on high that we will be using git for future development), so I'm curious if/how a one-time conversion simplifies things.\n\nThanks for bringing this up- it's quite interesting :)\n\nA one-time conversion thing would have changed our priorities. Mainly,\nI wouldn't have been able to justify the time I invested in building\nsvnrdump. I'd have probably settled for complete mirror + dump using a\ncombination of existing tools. The second difference:\nsvn-dump-fast-export is quite complex; although it was probably an\nunconcious decision on David's part, it makes a really good generic\nparser, and I suspect that it'll help us generate the dumpfile we need\nfor loading revision history data back into Subversion. Third, it\nsimplifies the branch/ tag mapper greatly- we'd only need to achieve a\nperfect one-way mapping. A symmetric mapping is harder for several\nreasons- consider preserving props for instance.\n\n-- Ram\n"},{"id":"148295","messageId":"20100817171846.GG6947@kytes","threadId":"24769","inReplyTo":"AANLkTim5RYDuius6Oa6KX0weQhwVz+_YP=1-ujb3-CuJ@mail.gmail.com","subject":"Re: [GSoC update] git-remote-svn: The final one","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-08-17T17:18:49Z","receivedAt":"2010-08-17T17:18:49Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Sverre,\n\nSverre Rabbelier writes:\n> Heya,\n> \n> On Tue, Aug 17, 2010 at 11:10, Erik Faye-Lund <kusmabite@gmail.com> wrote:\n> > On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra\n> > <artagnon@gmail.com> wrote:\n> >> We're doing better than that- when the whole thing is complete, we\n> >> should have perfectly seamless two-way compatibility.\n> >\n> > Does this include handling SVN repos where the layout have changed\n> > during it's lifetime? If so, how will this look on the git side?\n> \n> Perhaps it makes sense to start a new thread to hash that out for when\n> the time comes to implement that?\n\nGood idea. I've posted a new timeline. I'll put up some branch/ tag\nmapper notes next week.\n\n-- Ram\n"}]}