{"thread":{"id":"32236","subject":"reposurgeon now writes Subversion repositories","startedAt":"2012-11-29T05:59:45Z","lastAt":"2012-11-29T13:55:57Z","messageCount":7,"participants":["Eric S. Raymond","Daniel Shahaf","Branko Čibej","Markus Schaber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"204235","messageId":"20121129055946.2D7B84065F@snark.thyrsus.com","threadId":"32236","inReplyTo":null,"subject":"reposurgeon now writes Subversion repositories","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-29T05:59:45Z","receivedAt":"2012-11-29T05:59:45Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"This is something that probably doesn't happen very often -\ncross-posting to the Subversion and git dev lists that is on-topic for\nboth :-).\n\nThe repo head version of reposurgeon can now write Subversion\nrepositories from its common git-import-stream-based representation of\nrepository histories, as well as reading them in. This joins full\nsupport for git, hg, and bzr; it means that in theory reposurgeon\ncould now be used to move revision histories from these systems to\nSubversion, as well as the other way around.\n\n(For those of you who have been living under a rock, reposurgeon is a\nmulti-VCS surgery and conversion tool. Since 2.x it does a more\nintelligent job of lifting from Subversion to anything else than any\nother tool I know of. Much more at .)\n\nPresently, writing (as opposed to reading) Subversion repos is more of\na stunt than a real production technique, and may always remain so.\nIt has serious limitations. I am posting because I think the details\nof those limitations will be of some technical interest to both\nSubversion and git developers.\n\nIndented paragraphs is the documentation from reposurgeon's manual\npage. I have added some further notes.\n\n In summary, Subversion repository histories do not round-trip through\n reposurgeon editing. File content changes are preserved but some\n metadata is unavoidably lost. Furthermore, writing out a DVCS history\n in Subversion also loses significant portions of its metadata.\n\n Writing a Subversion repository or dump stream discards author\n information, the committer's name, and the hostname part of the commit\n address; only the commit timestamp and the local part of the\n committer's email address are preserved, the latter becoming the\n Subversion author field. However, reading a Subversion repository and\n writing it out again will preserve the author fields.\n\nSubversion's metadata doesn't have separate author and committer\nproperties, and doesn't store anything but a Unix user ID as\nattribution. I don't see any way around this.\n\n Import-stream timestamps have 1-second granularity. The subsecond\n parts of Subversion commit timestamps will be lost on their way through\n reposurgeon.\n\nUnavoidable in moving from Subversion to git import streams, and one\nof two places where git's data model requires us to throw away\ninformation. \n\nHowever, I think I could preserve this information in a\nSubversion-to-Subversion editing scenario by storing the incoming\ntimestamps as floats and only truncating them on import-stream output,\nleaving the subseconds in place for Subversion output.\n\n Empty directories aren't represented in import streams. Consequently,\n reading and writing Subversion repositories preserves file content,\n but not empty directories. It is also not guaranteed that after\n editing a Subverson repository that the sequence of directory\n creations and deletions relative to other operations will be\n identical; the only guarantee is that enclosing directories will be\n created before any files in them are.\n\n When reading a Subversion repository, reposurgeon discards the special\n directory-copy nodes associated with branch creations. These can't be\n recreated if and when the repository is written back out to\n Subversion; rather, each branch copy node from the original translates\n into a branch creation plus the first set of file modifications on the\n branch.\n\nIn theory, I could relax the rules of reposurgeon's internal\nrepresentation so that empty directory-creation and deletion nodes are\nnot discarded at read time but only when outputting a git event stream.\n\nThat would bring Subversion repositories closer to round-tripping, but\nnot get all the way there. One problem is botched branch copies -\ndirectory copies with cp(1) followed by Subversion add operations.\nThis is not an uncommon malformation; reposurgeon takes it in stride,\ntreating these as though they had been real branch copies and\nsimplifying the backlinks appropriately.\n\n When reading a Subversion repository, reposurgeon also automatically\n breaks apart mixed-branch commits.\n\nIt has to. These just can't be represented in the import-stream model of\nbranching.\n\n Because of the preceding two points, it is not guaranteed that \n even revision numbers will be stable when a Subversion repository\n is read in and then written out!\n\nSo not only can Subversion repos fail to round-trip exactly, in the\npresence of lots of branch copies and mixed-branch commits the\nrelationship between the read-in and written out revision numbers\ncould get pretty unpredictable.\n\n Subversion repositories are always written with a standard\n (trunk/tags/branches) layout. Thus, a repository with a nonstandard\n shape that has been analyzed by reposurgeon won't be written out with\n the same shape.\n\nIn particular, this means linear Subversion repositories with no trunk\n(an organization some smaller projects used to use and might still)\nwill turn into branchy repos with trunk on the way out.\n\n Subversion has a concept of \"flows\"; that is, named segments of\n history corresponding to files or directories that are created when\n the path is added, cloned when the path is copied, and deleted when\n the path is deleted. This information is not preserved in import\n streams or the internal representation that reposurgeon uses. Thus,\n after editing, the flow boundaries of a Subversion history may be\n arbitrarily changed.\n\nThis is me being obsessive about documenting the details. I think it\nis doubtful that most Subversion users even know flows exist.\n\n Bugs: Presently, writing out a history to a Subversion repository does\n not create mergeinfo properties representing branch merges. It also\n loses all information about lightweight tags (though annotated tags\n are turned into Subversion-style directory copies). These bugs will\n probably be fixed in future reposurgeon releases.\n\nI'm also not sure the present code handles branchiness exactly right. \nMy next task is to write a test suite for this new feature.\n-- \n\t\tEric S. Raymond\n\nThe Constitution is not neutral. It was designed to take the\ngovernment off the backs of the people.\n\t-- Justice William O. Douglas \n"},{"id":"204240","messageId":"20121129075829.GE3424@lp-shahaf.local","threadId":"32236","inReplyTo":"20121129055946.2D7B84065F@snark.thyrsus.com","subject":"Re: reposurgeon now writes Subversion repositories","fromName":"Daniel Shahaf","fromEmail":"danielsh@elego.de","sentAt":"2012-11-29T07:58:29Z","receivedAt":"2012-11-29T07:58:29Z","isPatch":false,"sender":{"key":"danielsh@elego.de","avatar":null},"body":"Eric S. Raymond wrote on Thu, Nov 29, 2012 at 00:59:45 -0500:\n> In summary, Subversion repository histories do not round-trip through\n> reposurgeon editing. File content changes are preserved but some\n> metadata is unavoidably lost. Furthermore, writing out a DVCS history\n> in Subversion also loses significant portions of its metadata.\n> \n> Writing a Subversion repository or dump stream discards author\n> information, the committer's name, and the hostname part of the commit\n> address; only the commit timestamp and the local part of the\n> committer's email address are preserved, the latter becoming the\n> Subversion author field. However, reading a Subversion repository and\n> writing it out again will preserve the author fields.\n> \n> Subversion's metadata doesn't have separate author and committer\n> properties, and doesn't store anything but a Unix user ID as\n> attribution. I don't see any way around this.\n\nYou're not fully informed, then.\n\n1) svn:author revprops can contain any UTF-8 string. They are not\nrestricted to Unix user id's. (For example, they can contain full\nnames, if the administrator so chooses.)\n\n2) You can define custom revision properties. In your case, the easiest\nway would be to set an reposurgeon:author property, alongside the\nsvn:author property.\n\nYou might also seek community consensus to reserve an svn:foo name for\nthe \"original author\" property --- perhaps svn:original-author --- so\nthat reposurgeon and other git->svn tools can interoperate in the way\nthey transfer the \"original author\" information.\n\nI note that one can set revision properties at commit time:\n\n svn commit -m logmsg --with-revprop svn:original-author=\"Patch Submitter \"\n\n> Empty directories aren't represented in import streams. Consequently,\n> reading and writing Subversion repositories preserves file content,\n> but not empty directories. It is also not guaranteed that after\n> editing a Subverson repository that the sequence of directory\n> creations and deletions relative to other operations will be\n> identical; the only guarantee is that enclosing directories will be\n> created before any files in them are.\n\nHow does reposurgeon handle empty directories with (node) properties?\n\n% svnadmin create r\n% svnmucc -mm -U file://$PWD/r mkdir foo propset k v foo\n\n> Subversion has a concept of \"flows\"; that is, named segments of\n> history corresponding to files or directories that are created when\n> the path is added, cloned when the path is copied, and deleted when\n> the path is deleted. This information is not preserved in import\n> streams or the internal representation that reposurgeon uses. Thus,\n> after editing, the flow boundaries of a Subversion history may be\n> arbitrarily changed.\n> \n> This is me being obsessive about documenting the details. I think it\n> is doubtful that most Subversion users even know flows exist.\n> \n\nI think you're saying that adds might turn into copies, and vice-versa.\nThat is something users would notice --- it is certainly exposed in the\nUI --- even though node-id's are not exposed to clients.\n\n> \n\nCheers\n\nDaniel\n"},{"id":"204245","messageId":"50B73995.5050505@wandisco.com","threadId":"32236","inReplyTo":"20121129075829.GE3424@lp-shahaf.local","subject":"Re: reposurgeon now writes Subversion repositories","fromName":"Branko Čibej","fromEmail":"brane@wandisco.com","sentAt":"2012-11-29T10:31:49Z","receivedAt":"2012-11-29T10:31:49Z","isPatch":false,"sender":{"key":"brane@wandisco.com","avatar":null},"body":"On 29.11.2012 08:58, Daniel Shahaf wrote:\n> I think you're saying that adds might turn into copies, and\n> vice-versa. That is something users would notice --- it is certainly\n> exposed in the UI --- even though node-id's are not exposed to clients. \n\n... yet. But there are plans underway to expose them.\n\n-- \nBranko Čibej\nDirector of Subversion | WANdisco | www.wandisco.com\n"},{"id":"204248","messageId":"20121129114637.GB9264@thyrsus.com","threadId":"32236","inReplyTo":"20121129075829.GE3424@lp-shahaf.local","subject":"Re: reposurgeon now writes Subversion repositories","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-29T11:46:37Z","receivedAt":"2012-11-29T11:46:37Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Daniel Shahaf :\n> > Subversion's metadata doesn't have separate author and committer\n> > properties, and doesn't store anything but a Unix user ID as\n> > attribution. I don't see any way around this.\n> \n> You're not fully informed, then.\n> \n> 1) svn:author revprops can contain any UTF-8 string. They are not\n> restricted to Unix user id's. (For example, they can contain full\n> names, if the administrator so chooses.)\n\nRight. At one point during the development of this feature I was\naccidentally storing the full email field in this property. So I\nalready knew that this is allowed at some level. \n\nAnd, I have no trouble believing that svn log will cheerfully echo\nanything that I choose to stuff in that field. \n\nBut...\n\n(1) How much work would it be it to set up a Subversion installation \nso that when I svn commit, the tool does the right thing, e.g. puts\na DVCS-style fullname/email string in there? \n\n(2) Have the tools been tested for bugs arising from having whitespace\nin that data?\n\nReally, if it's actually easy to set up DVCS-style globally unique IDs you\nSubversion guys ought to be shouting it from the housetops. The absence\nof this capability is a serious PITA in several situations, including \nfor example migrating projects between forges.\n\nRFC: If I wrote a patch that let Subversion users set their own\ncontent string for the author field in ~/.subversion/config, would\nyou merge it? Because I'd totally write that.\n\n> 2) You can define custom revision properties. In your case, the easiest\n> way would be to set an reposurgeon:author property, alongside the\n> svn:author property.\n\nYeah, sure, I've assumed all along this wouldn't break if I tried it.\nIf I actually thought you guys were capable of designing a data model\nwith a perfectly general-looking store of key/value pairs and then\narbitrarily restricting the key set so I couldn't do that, I'd almost\nhave to find each and every one of you and kick your asses into next\nTuesday on account of blatant stupidity. I have no such plans :-).\n\nBut...what good does this capability do? OK, it would assist\nround-tripping back to gitspace, but while that's kind of cool I don't\nsee any help for a normal Subversion workflow here.\n \n> You might also seek community consensus to reserve an svn:foo name for\n> the \"original author\" property --- perhaps svn:original-author --- so\n> that reposurgeon and other git->svn tools can interoperate in the way\n> they transfer the \"original author\" information.\n\nOK. But I like the idea of letting the users set their own author\ncontent string better. Instead of another layer of kluges, why\nshouldn't Subversion join the DVCSes in the happy land of\nInternet-scoped attributions?\n\n> How does reposurgeon handle empty directories with (node) properties?\n\nCurrently by ignoring all of them except svn:ignore, which it turns \ninto .gitignore content on the gitspace side. And now vice-versa, too.\n\nNot clear what else it *could* do. I'd take suggestions.\n\n> > Subversion has a concept of \"flows\"; that is, named segments of\n> > history corresponding to files or directories that are created when\n> > the path is added, cloned when the path is copied, and deleted when\n> > the path is deleted. This information is not preserved in import\n> > streams or the internal representation that reposurgeon uses. Thus,\n> > after editing, the flow boundaries of a Subversion history may be\n> > arbitrarily changed.\n> > \n> > This is me being obsessive about documenting the details. I think it\n> > is doubtful that most Subversion users even know flows exist.\n> \n> I think you're saying that adds might turn into copies, and vice-versa.\n> That is something users would notice --- it is certainly exposed in the\n> UI --- even though node-id's are not exposed to clients.\n\nI'm saying nobody thinks of flows when they do branch copies. It's\nnot just that users don't see node IDs, it's that no part of most users'\nmental model of how Subversion works resembles them.\n-- \n\t\tEric S. Raymond\n"},{"id":"204250","messageId":"20121129134203.GJ3424@lp-shahaf.local","threadId":"32236","inReplyTo":"20121129114637.GB9264@thyrsus.com","subject":"Re: reposurgeon now writes Subversion repositories","fromName":"Daniel Shahaf","fromEmail":"danielsh@elego.de","sentAt":"2012-11-29T13:42:03Z","receivedAt":"2012-11-29T13:42:03Z","isPatch":false,"sender":{"key":"danielsh@elego.de","avatar":null},"body":"(note, other half of the thread is on dev@svn only..)\n\nEric S. Raymond wrote on Thu, Nov 29, 2012 at 06:46:37 -0500:\n> Daniel Shahaf :\n> > You might also seek community consensus to reserve an svn:foo name for\n> > the \"original author\" property --- perhaps svn:original-author --- so\n> > that reposurgeon and other git->svn tools can interoperate in the way\n> > they transfer the \"original author\" information.\n> \n> OK. But I like the idea of letting the users set their own author\n> content string better. Instead of another layer of kluges, why\n\nI don't see the kludge here --- git has a \"author\" != \"committer\"\ndistinction, svn doesn't, so if you want to grow that distinction the\nmost natural way is a new property. Storing additional information in\nsvn:author is a separate issue.\n\n> > > Subversion has a concept of \"flows\"; that is, named segments of\n> > > history corresponding to files or directories that are created when\n> > > the path is added, cloned when the path is copied, and deleted when\n> > > the path is deleted. This information is not preserved in import\n> > > streams or the internal representation that reposurgeon uses. Thus,\n> > > after editing, the flow boundaries of a Subversion history may be\n> > > arbitrarily changed.\n> > > \n> > > This is me being obsessive about documenting the details. I think it\n> > > is doubtful that most Subversion users even know flows exist.\n> > \n> > I think you're saying that adds might turn into copies, and vice-versa.\n> > That is something users would notice --- it is certainly exposed in the\n> > UI --- even though node-id's are not exposed to clients.\n> \n> I'm saying nobody thinks of flows when they do branch copies. It's\n> not just that users don't see node IDs, it's that no part of most users'\n> mental model of how Subversion works resembles them.\n\nI'm still not sure what you have in mind. I note that 'svn log' and\n'svn blame' cross both file copies and branch creation --- that's one\neffect of \"'svn cp foo bar; svn ci' causes bar to be related to foo\".\n\n> -- \n> \t\tEric S. Raymond\n"},{"id":"204251","messageId":"727D8E16AE957149B447FE368139F2B50D9951CB@SERVER10","threadId":"32236","inReplyTo":"20121129114637.GB9264@thyrsus.com","subject":"AW: reposurgeon now writes Subversion repositories","fromName":"Markus Schaber","fromEmail":"m.schaber@codesys.com","sentAt":"2012-11-29T13:43:06Z","receivedAt":"2012-11-29T13:43:06Z","isPatch":false,"sender":{"key":"m.schaber@codesys.com","avatar":null},"body":"Hi,\n\nVon: Eric S. Raymond [mailto:esr@thyrsus.com]\n> > How does reposurgeon handle empty directories with (node) properties?\n> \n> Currently by ignoring all of them except svn:ignore, which it turns\n> into .gitignore content on the gitspace side. And now vice-versa, too.\n> \n> Not clear what else it *could* do. I'd take suggestions.\n\nAFAIR, SvnBridge (which bridges SVN to Team Foundation Server for CodePlex) creates a hidden .svnproperties file where all the properties of the directory and files are stored.\n\nI'm not really sure, but maybe this could be used as some standard to bridge svn properties to non-svn VCSes.\n\nBest regards\n\nMarkus Schaber\n\nCODESYS(r) a trademark of 3S-Smart Software Solutions GmbH\n\nInspiring Automation Solutions\n\n3S-Smart Software Solutions GmbH\nDipl.-Inf. Markus Schaber | Product Development Core Technology\nMemminger Str. 151 | 87439 Kempten | Germany\nTel. +49-831-54031-979 | Fax +49-831-54031-50\n\nE-Mail: m.schaber@codesys.com | Web: http://www.codesys.com\nCODESYS internet forum: http://forum.codesys.com\n\nManaging Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915\n"},{"id":"204252","messageId":"20121129135557.GB26607@thyrsus.com","threadId":"32236","inReplyTo":"20121129134203.GJ3424@lp-shahaf.local","subject":"Re: reposurgeon now writes Subversion repositories","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-29T13:55:57Z","receivedAt":"2012-11-29T13:55:57Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Daniel Shahaf :\n> I don't see the kludge here --- git has a \"author\" != \"committer\"\n> distinction, svn doesn't, so if you want to grow that distinction the\n> most natural way is a new property. Storing additional information in\n> svn:author is a separate issue.\n\nSee my advocacy to Branko of going to Internet-scoped IDs. The kludge\nwould be maintaining the local and Internet-scoped identifications \nas different properties and having to decide which one to key on\nad-hoc. Nothing to do with the author/committer distinction. \n-- \n\t\tEric S. Raymond\n"}]}