{"thread":{"id":"12765","subject":"[SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","startedAt":"2008-03-19T04:08:25Z","lastAt":"2008-03-24T19:50:04Z","messageCount":11,"participants":["Bryan Donlan","Sam Vilain","Shawn O. Pearce","Harvey Harrison","Julian Phillips","Jakub Narebski","Johannes Schindelin","Govind Salinas"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"72432","messageId":"3e8340490803182108y40a9aec2q8e5bcb78b907bbb5@mail.gmail.com","threadId":"12765","inReplyTo":null,"subject":"[SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Bryan Donlan","fromEmail":"bdonlan@gmail.com","sentAt":"2008-03-19T04:08:25Z","receivedAt":"2008-03-19T04:08:25Z","isPatch":false,"sender":{"key":"bdonlan@gmail.com","avatar":null},"body":"Hello,\n\nI'm planning to apply for the git summer of code project. My proposal\nis based on the project idea of a subversion gateway for git,\nimplemented with a new subversion filesystem layer. A draft of my\nproposal follows; I'd appreciate any comments/questions on it before\nthe application period proper begins.\n\nThanks,\n\nBryan Donlan\n\n=== Project Goals ===\n\nI propose to implement a subversion filesystem driver (libsvn-fs-git) that\nuses a git repository as its backing store. Commits will be supported either\ndirectly in the git repository, or in the corresponding subversion repository,\nand automatically mirrored to the other side as appropriate.\n\nI intend to support the following:\n* Full or near full (possibly forbidding modification of the toplevel\ntrunk/ branches/ tags/ structure) read/write access from subversion\n* svnadmin create/dump/load to convert existing subversion repositories\n* Support for wrapping a pre-existing git repository and presenting it\nas a subversion repository\n* Support for mapping git branches and tags onto subversion branches\nand tags (and vice versa)\n* Support for syncing svn:executable with git file mode information\n* Representation of git merge data using svk:merge and/or svn:mergeinfo\n* Syncing .gitignore and svn:ignore data\n\nAs both subversion and git are written in C, this driver will also be in C.\n\nHere are some tentative milestones:\n* Read-only access from SVN to the master branch (no trunk/ etc layout)\n  = Conversion of git commit information into svn revprops\n  = git mode/.gitignore -> svn property conversion here?\n* Read-write access from SVN to the master branch\n  = Map svn usernames to git full name/email according to a configuration map\n    - how should git commits with names unknown to svn be handled? Just pass\n      them through, spaces and <@> as well?\n  = Bidirectional svn:execute and svn:ignore conversion.\n  = Copyfrom and file property information needs to be recorded\n  = Test importing a largish repository (without converting merge information)\n    to git (the svn toplevel stuff would be left as-is in the git tree)\n  = Consider developing git-svn-fs on a git-svn-fs repository itself for\n    testing purposes\n* Standard toplevel SVN layout (trunk/ tags/ branches/)\n  = SVN branch creation might come a bit later\n  = Test importing a largish repository with tags and branches carried across\n    (might not efficiently support copy-from information)\n* Merge information annotation (git->svn)\n  = Try to guess the copy source for a new tag or branch - and for merges\n* Merge information annotation (svn->git)\n* Import of a largish repository with svk or similar merge information into git,\n  and vice versa (eg, exporting git.git with merge tracking as a subversion\n  repo)\n\n=== Interfaces ===\n\nAs mentioned before, this driver will plug into the existing subversion stack\nas a filesystem driver. This immediately allows access using any of subversion's\naccess methods (direct filesystem access, mod_dav_svn, svnserve).\n\nOn the git side I intend to use libgit for all git repository access. If I find\nit lacking a necessary feature, I will attempt to add the missing interfaces\nto libgit if at all feasable.\n\nI anticipate svn-git-fs to live either in git.git's contrib or an outside\nrepository. There should be little if any changes to git itself.\n\n=== About me ===\nI am a sophomore computer science student at the University of Maine at Orono.\nI have been programming since well before I entered college, and am experienced\nin C, although I have not done much work in large (in terms of number of\ndevelopers) projects. I have experience in using Subversion, including doing\nmerges with svk, but I am somewhat less experienced with git. I hope to become\nmore familiar with git prior to, and as I progress in this project. I also have\nsome ability with Japanese... but possibly not enough yet to translate the\nstrings and documentation in this project :)\n\nThis particular project idea caught my eye partially because I have been hoping\nto convert another open-source project that I hack on [1] to git, but as one\nof the other developers is testing it primarily on windows, we've been reluctant\nto move there. A git<->svn gateway would be ideal to help ease the transition.\n(We haven't yet tried the cygwin port, so admittedly this may be moot already)\n\nI have submitted a small documentation patch to git.git recently[2], and lurked\non the mailing lists for a while during its early days, but I have not yet\nbecome actively involved in development.\n\n[1] - http://openc2e.ccdevnet.org\n[2] - commit 81fa145917c40b68a5e2cca6afc6a10cdfdbd25b\n\n=== (Tentative) design notes ===\n\nMy current plan for storing the additional information the subversion side will\nneed (fileprops, revprops, copyfrom information...) is to create an additional\nbranch on the git repository (possibly .git-svn or similar) to hold the\nnecessary metadata. Configuration, including author maps, branch/tag maps,\netc, would be on another branch (git-svn-config or similar).\n\nThe layout might look like this:\n\n/tree/{trunk/,branches/,tags/} - the tree as svn currently sees it\n/props/{trunk/,branches/,tags/} - file properties; props on directories will be\n  represented with a reserved filename (._GIT-SVN-DIRPROPS perhaps)\n  copyfrom information might be in /props, or in a seperate tree\n/revprops/NNN - revision properties for the given revision number\n/revmap/NNN - a reference to the commit hash in the .git-svn branch\n  corresponding to the given subversion revision number\n\nEach subversion commit corresponds to two .git-svn commits; one to\nupdate tree, props, and revprops, and one to update revmap. The first\ncommit will\nadditionally have the following metadata in its commit:\ngit-svn-revno NNN\ngit-svn-parent (commit hash of corresponding git-side commit)\n\nIf the commit was initiated from the svn side, the git-side commit will be\ncommitted first, and will contain a git-svn-revno field as well. The overall\ncommit will be performed while holding a git-svn-fs-specific lock (also held\nwhen replicating new git commits to the svn side).\n\nIf a commit is performed on the subversion side, the next query to the\nsubversion layer which checks the current youngest revision number will also\nscan for updated git heads, assign revision numbers, and create the necessary\nsubversion metadata in the .git-svn branch.\n"},{"id":"72478","messageId":"47E1E8BD.7000209@vilain.net","threadId":"12765","inReplyTo":"3e8340490803182108y40a9aec2q8e5bcb78b907bbb5@mail.gmail.com","subject":"Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-03-20T04:31:57Z","receivedAt":"2008-03-20T04:31:57Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Bryan Donlan wrote:\n\n> Here are some tentative milestones:\n> * Read-only access from SVN to the master branch (no trunk/ etc layout)\n>   = Conversion of git commit information into svn revprops\n>   = git mode/.gitignore -> svn property conversion here?\n\nThis seems like a large milestone.  Can you break this up any more?\n\nFor instance, your design notes on storing the necessary mapping\ninformation are good.  How about a separate milestone of having a test\nsuite for the library functions you make for accessing that information.\n\nI would be tempted to check the protocol -\nhttp://svn.collab.net/repos/svn/trunk/subversion/libsvn_ra_svn/protocol\n- and make milestones for each request type that the protocol allows\nfor.  Perhaps there is a more relevant list that you can find, such as\ngroups of tests in the back-end test suite that ships with Subversion.\nEven taking the list of svn sub-commands, and deciding which fit into\neach category would be a good enhancement.\n\n> * Read-write access from SVN to the master branch\n>   = Map svn usernames to git full name/email according to a configuration map\n>     - how should git commits with names unknown to svn be handled? Just pass\n>       them through, spaces and <@> as well?\n\nMeh.  Just ignore them, and set revprops with all of the git committer\ninformation.\n\n>   = Bidirectional svn:execute and svn:ignore conversion.\n>   = Copyfrom and file property information needs to be recorded\n>   = Test importing a largish repository (without converting merge information)\n>     to git (the svn toplevel stuff would be left as-is in the git tree)\n>   = Consider developing git-svn-fs on a git-svn-fs repository itself for\n>     testing purposes\n\nAn honourable notion, but I'd steer away from worrying about\nself-hosting, if it is irrelevant to the task at hand.  Focus more on a\nfinding a good test suite to check you supported all the operations.\nEg, can the test suite bundled with the Subversion project run against\nyour back-end?\n\n> * Standard toplevel SVN layout (trunk/ tags/ branches/)\n>   = SVN branch creation might come a bit later\n>   = Test importing a largish repository with tags and branches carried across\n>     (might not efficiently support copy-from information)\n> * Merge information annotation (git->svn)\n>   = Try to guess the copy source for a new tag or branch - and for merges\n\nI don't like this word \"guess\".  It might be dangerous to not\ndeterministically or repeatably answer a request.  If any random\ndecisions were made, or information derived based on things that might\nchange, then it should be stored in your mapping information branch.  In\nthis instance, we didn't 'guess', we decided.\n\n> * Merge information annotation (svn->git)\n> * Import of a largish repository with svk or similar merge information into git,\n>   and vice versa (eg, exporting git.git with merge tracking as a subversion\n>   repo)\n\nWhew!  That's a lot of big milestones, but it's your summer ... :)\n\nI think the merging thing is a nice-to-have, and doing it would just\nprove that you can use the metadata that you have collected well.\n\nOne thing I like about your approach is that the tracking branch itself\ncould be replicated, leaving an audit of what happened.\n\n> === Interfaces ===\n> \n> As mentioned before, this driver will plug into the existing subversion stack\n> as a filesystem driver. This immediately allows access using any of subversion's\n> access methods (direct filesystem access, mod_dav_svn, svnserve).\n> \n> On the git side I intend to use libgit for all git repository access. If I find\n> it lacking a necessary feature, I will attempt to add the missing interfaces\n> to libgit if at all feasable.\n\nAFAIK the interface for libgit is not yet finalized, so bear in mind the\napplication will possibly need porting work for each release.\n\nSam.\n"},{"id":"72483","messageId":"20080320045632.GB8410@spearce.org","threadId":"12765","inReplyTo":"3e8340490803182108y40a9aec2q8e5bcb78b907bbb5@mail.gmail.com","subject":"Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-03-20T04:56:32Z","receivedAt":"2008-03-20T04:56:32Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Bryan Donlan <bdonlan@gmail.com> wrote:\n> I'm planning to apply for the git summer of code project. My proposal\n> is based on the project idea of a subversion gateway for git,\n> implemented with a new subversion filesystem layer. A draft of my\n> proposal follows; I'd appreciate any comments/questions on it before\n> the application period proper begins.\n\nVery cool.  Have you had a chance to look at the prototype python\nimplementation of an SVN server that Julian Phillips started?\n\n  http://git.q42.co.uk/w/git_svn_server.git\n\nI'm just curious what your take is regarding this approach.  Why\nwould you choose to construct libsvn-fs-git over a standalone server?\nThere are several advantages and drawbacks to both approaches.\nI am not advocating over the other, but want to make sure you have\nthought it through for yourself.\n \n> I intend to support the following:\n> * Full or near full (possibly forbidding modification of the toplevel\n> trunk/ branches/ tags/ structure) read/write access from subversion\n\nThat's probably the only sane way to go about it; disallow read/write\non the top level, map whatever branch \"HEAD\" points to in Git to the\ntrunk/, put the other branches in branches/ and the tags under tags/.\nBlock everything else.\n\n> * Support for syncing svn:executable with git file mode information\n> * Representation of git merge data using svk:merge and/or svn:mergeinfo\n> * Syncing .gitignore and svn:ignore data\n\nThese are gravy.  Sure they are going to be difficult to make work,\nbut people can limp by without them.  Most users who want an SVN\nclient to speak to a Git repository are trying to do so from a\nplatform that does not honor executable bits (hi Windows!) and\ntelling users to edit the funny \".gitignore\" file to alter ignore\nlists is something they can work around without too much trouble\nif they are already able to modify and commit files.\n\nThough their clients won't provide the proper ignore support out\nof the box.  *sigh*\n \n> As both subversion and git are written in C, this driver will also be in C.\n\nI think you may have underestimated the challenges associated with\nlinking \"libgit.a\" (which is _not_ a library) with SVN.  Critical\nroutines within libgit that you want to be able to invoke will do\nnot so nice things like leak massive amounts of memory or cause\nyour process to terminate if the function is fed an invalid input.\n\nMost of the C code of Git is designed for single-shot execution.\nWe leak memory like mad because it is more efficient to load up what\nwe need, exit, and let the OS just return the pages to the free pool.\nLong running processes have simply not been something we do.\n \n> My current plan for storing the additional information the subversion side will\n> need (fileprops, revprops, copyfrom information...) is to create an additional\n> branch on the git repository (possibly .git-svn or similar) to hold the\n> necessary metadata. Configuration, including author maps, branch/tag maps,\n> etc, would be on another branch (git-svn-config or similar).\n> \n> The layout might look like this:\n> \n> /tree/{trunk/,branches/,tags/} - the tree as svn currently sees it\n\nI don't think you'd want to put a copy of the tree inside of a tree,\nas this can then get out of sync with changes made directly through\ngit, plus you run into issues about connecting the two histories\ntogether in a meaningful way.\n\nI would suggest having the root directory of the SVN tree be built\non the fly based upon the list of available branches and tags in\nthe Git repository (aka the output of git-show-ref).\n\n> /props/{trunk/,branches/,tags/} - file properties; props on directories will be\n>   represented with a reserved filename (._GIT-SVN-DIRPROPS perhaps)\n>   copyfrom information might be in /props, or in a seperate tree\n\nHow critical are file properties to an SVN client for proper\nfunctioning?  Given the challenges already in front of you for this\nproject I would almost encourage you to avoid dealing with file\nlevel properties.  Its hard enough to make something that speaks SVN\non the wire but reads/writes Git on disk, not to mention you have\nto somehow \"flatten\" the Git DAG down into a sequential revision\nnamespace to make the SVN clients happy.  So deferring property\nsupport until later may be wise.\n\n> /revprops/NNN - revision properties for the given revision number\n\nDitto.  Aside from the special merge properties you mentioned,\nI wonder if you can simply avoid implementing support for these\nearly on.\n\n> /revmap/NNN - a reference to the commit hash in the .git-svn branch\n>   corresponding to the given subversion revision number\n\nHow about using a simple flat file interface?  To initially prime\nthe file you can do something like:\n\n\tgit rev-list --topo-order --date-order --reverse --all >.git/svn-map\n\nand then number the revisions by the line number that they appear on.\nLocating a Git SHA-1 for a specific SVN revision would be a simple\ncase of lseek(fd, 41 * rev, SEEK_SET).  Going the other direction\nwould be more of a challenge, but is still doable.\n\nUpdating the file should just require appending new commits; if\nthe SVN client wants a new commit you append on and return the\nline number.  If Git has caused new commits not in this file you\nneed to rebuild the log.  This would have to be done incrementally,\nto prevent changing a prior SVN revision number that clients may\nalready know about.\n \n-- \nShawn.\n"},{"id":"72488","messageId":"1205993915.17607.32.camel@brick","threadId":"12765","inReplyTo":"20080320045632.GB8410@spearce.org","subject":"Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Harvey Harrison","fromEmail":"harvey.harrison@gmail.com","sentAt":"2008-03-20T06:18:34Z","receivedAt":"2008-03-20T06:18:34Z","isPatch":false,"sender":{"key":"harvey.harrison@gmail.com","avatar":null},"body":"On Thu, 2008-03-20 at 00:56 -0400, Shawn O. Pearce wrote:\n> Bryan Donlan <bdonlan@gmail.com> wrote:\n> > /revmap/NNN - a reference to the commit hash in the .git-svn branch\n> >   corresponding to the given subversion revision number\n> \n> How about using a simple flat file interface?  To initially prime\n> the file you can do something like:\n> \n> \tgit rev-list --topo-order --date-order --reverse --all >.git/svn-map\n> \n> and then number the revisions by the line number that they appear on.\n> Locating a Git SHA-1 for a specific SVN revision would be a simple\n> case of lseek(fd, 41 * rev, SEEK_SET).  Going the other direction\n> would be more of a challenge, but is still doable.\n> \n> Updating the file should just require appending new commits; if\n> the SVN client wants a new commit you append on and return the\n> line number.  If Git has caused new commits not in this file you\n> need to rebuild the log.  This would have to be done incrementally,\n> to prevent changing a prior SVN revision number that clients may\n> already know about.\n\nWhy not just copy the rev_map format git-svn already uses, it's pretty\nefficient.\n\nHarvey\n"},{"id":"72500","messageId":"Pine.LNX.4.64.0803200910410.21580@reaper.quantumfyre.co.uk","threadId":"12765","inReplyTo":"20080320045632.GB8410@spearce.org","subject":"Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2008-03-20T09:22:58Z","receivedAt":"2008-03-20T09:22:58Z","isPatch":false,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"On Thu, 20 Mar 2008, Shawn O. Pearce wrote:\n\n> Bryan Donlan <bdonlan@gmail.com> wrote:\n>> I'm planning to apply for the git summer of code project. My proposal\n>> is based on the project idea of a subversion gateway for git,\n>> implemented with a new subversion filesystem layer. A draft of my\n>> proposal follows; I'd appreciate any comments/questions on it before\n>> the application period proper begins.\n>\n> Very cool.  Have you had a chance to look at the prototype python\n> implementation of an SVN server that Julian Phillips started?\n>\n>  http://git.q42.co.uk/w/git_svn_server.git\n\n(now with partial support for 'svn log' ... ;))\n\n>> /props/{trunk/,branches/,tags/} - file properties; props on directories will be\n>>   represented with a reserved filename (._GIT-SVN-DIRPROPS perhaps)\n>>   copyfrom information might be in /props, or in a seperate tree\n>\n> How critical are file properties to an SVN client for proper\n> functioning?  Given the challenges already in front of you for this\n> project I would almost encourage you to avoid dealing with file\n> level properties.  Its hard enough to make something that speaks SVN\n> on the wire but reads/writes Git on disk, not to mention you have\n> to somehow \"flatten\" the Git DAG down into a sequential revision\n> namespace to make the SVN clients happy.  So deferring property\n> support until later may be wise.\n\nYou might need to get svn:eol-style working to prevent the svn client from \nmunging any binary files?  Can't think of any other vital properties atm.\n\n>> /revprops/NNN - revision properties for the given revision number\n>\n> Ditto.  Aside from the special merge properties you mentioned,\n> I wonder if you can simply avoid implementing support for these\n> early on.\n\nSince you have to explicitly enable revprop editing in the subversion \nrepository by enabling a hook script, I should think that this was \ndefinately something that could be left at the bottom of the TODO list ...\n\nThough you do need to be able to convert commit info into the appropriate \nrevprops (e.g. commit msg -> svn:log revprop)\n\n-- \nJulian\n\n  ---\nOften statistics are used as a drunken man uses lampposts -- for support\nrather than illumination.\n"},{"id":"72502","messageId":"frtcmc$l8m$1@ger.gmane.org","threadId":"12765","inReplyTo":"20080320045632.GB8410@spearce.org","subject":"Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-03-20T10:01:48Z","receivedAt":"2008-03-20T10:01:48Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[Cc: Shawn O. Pearce <spearce@spearce.org>, \n     Bryan Donlan <bdonlan@gmail.com>,\n     git@vger.kernel.org]\n\nShawn O. Pearce wrote:\n> Bryan Donlan <bdonlan@gmail.com> wrote:\n\n>> /revmap/NNN - a reference to the commit hash in the .git-svn branch\n>>   corresponding to the given subversion revision number\n> \n> How about using a simple flat file interface?  To initially prime\n> the file you can do something like:\n> \n>         git rev-list --topo-order --date-order --reverse --all \\\n>                >.git/svn-map \n> \n> and then number the revisions by the line number that they appear on.\n> Locating a Git SHA-1 for a specific SVN revision would be a simple\n> case of lseek(fd, 41 * rev, SEEK_SET).  Going the other direction\n> would be more of a challenge, but is still doable.\n> \n> Updating the file should just require appending new commits; if\n> the SVN client wants a new commit you append on and return the\n> line number.  If Git has caused new commits not in this file you\n> need to rebuild the log.  This would have to be done incrementally,\n> to prevent changing a prior SVN revision number that clients may\n> already know about.\n\nBy the way, have you looked into what git-svn uses? IIRC it had some\nimprovements to avoid spending more disk space on SVN revno <-> Git SHA-1\nmapping than on the repository itself...\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"72645","messageId":"3e8340490803212202r6dbaa9eel544ba2b4b8e8d0c7@mail.gmail.com","threadId":"12765","inReplyTo":"3e8340490803182108y40a9aec2q8e5bcb78b907bbb5@mail.gmail.com","subject":"Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Bryan Donlan","fromEmail":"bdonlan@gmail.com","sentAt":"2008-03-22T05:02:39Z","receivedAt":"2008-03-22T05:02:39Z","isPatch":false,"sender":{"key":"bdonlan@gmail.com","avatar":null},"body":"On Wed, Mar 19, 2008 at 12:08 AM, Bryan Donlan <bdonlan@gmail.com> wrote:\n> Hello,\n>\n>  I'm planning to apply for the git summer of code project. My proposal\n>  is based on the project idea of a subversion gateway for git,\n>  implemented with a new subversion filesystem layer. A draft of my\n>  proposal follows; I'd appreciate any comments/questions on it before\n>  the application period proper begins.\n\nHi all,\n\nThanks for all the comments. To try to avoid spamming the list, I've\nreplied in a single message, if it'd be better to reply individually\nin the future please let me know.\n\nOn Thu, Mar 20, 2008 at 12:31 AM, Sam Vilain <sam@vilain.net> wrote:\n> Bryan Donlan wrote:\n>\n>  > Here are some tentative milestones:\n>  > * Read-only access from SVN to the master branch (no trunk/ etc layout)\n>  >   = Conversion of git commit information into svn revprops\n>  >   = git mode/.gitignore -> svn property conversion here?\n>\n>  This seems like a large milestone.  Can you break this up any more?\n>\n>  For instance, your design notes on storing the necessary mapping\n>  information are good.  How about a separate milestone of having a test\n>  suite for the library functions you make for accessing that information.\n\nThat seems reasonable - eg, a milestone for cloning over a git tree\ninto the .git-svn branch.\n\n>  I would be tempted to check the protocol -\n>  http://svn.collab.net/repos/svn/trunk/subversion/libsvn_ra_svn/protocol\n>  - and make milestones for each request type that the protocol allows\n>  for.  Perhaps there is a more relevant list that you can find, such as\n>  groups of tests in the back-end test suite that ships with Subversion.\n>  Even taking the list of svn sub-commands, and deciding which fit into\n>  each category would be a good enhancement.\n\nI haven't decided I will try to take this into the subversion tree\nproper - but I could try to shoehorn on the subversion tests. That\nsaid, they tend to work by checking in test data, then verifying it,\nso they won't work until write support works.\n\nI could set milestones based on specific libsvn_fs APIs, but once I've\ngot the metadata cloned over I don't think any individual operation\nwill be particularly difficult in itself (they'd all be just giving a\nview of the /tree/ in the .git-svn branch)\n\n>  >   = Bidirectional svn:execute and svn:ignore conversion.\n>  >   = Copyfrom and file property information needs to be recorded\n>  >   = Test importing a largish repository (without converting merge information)\n>  >     to git (the svn toplevel stuff would be left as-is in the git tree)\n>  >   = Consider developing git-svn-fs on a git-svn-fs repository itself for\n>  >     testing purposes\n>\n>  An honourable notion, but I'd steer away from worrying about\n>  self-hosting, if it is irrelevant to the task at hand.  Focus more on a\n>  finding a good test suite to check you supported all the operations.\n>  Eg, can the test suite bundled with the Subversion project run against\n>  your back-end?\n\nOnce commits work, I think it should be possible to get it working\n(it's already engineered to support two backends). svnadmin create\nsupport might be needed as well though.\n\n>  > * Standard toplevel SVN layout (trunk/ tags/ branches/)\n>  >   = SVN branch creation might come a bit later\n>  >   = Test importing a largish repository with tags and branches carried across\n>  >     (might not efficiently support copy-from information)\n>  > * Merge information annotation (git->svn)\n>  >   = Try to guess the copy source for a new tag or branch - and for merges\n>\n>  I don't like this word \"guess\".  It might be dangerous to not\n>  deterministically or repeatably answer a request.  If any random\n>  decisions were made, or information derived based on things that might\n>  change, then it should be stored in your mapping information branch.  In\n>  this instance, we didn't 'guess', we decided.\n\nIndeed, it would be saved by creating subversion commits (recorded in\nthe .git-svn branch as changes to /tree etc) corresponding to any\nchanges to branches or tags. The reason I say 'guess' is because,\nsince git commits are not unique, it is difficult to attribute them to\na single branch. eg:\n\nbranch2     C\n           /\nbranch1   B-D\n         /\nmaster  A---E\n\nCommit 'B' would have to be attributed to one or the other of branch1\nor branch2, but given just the final state of (C,D,E) we can't\nuniquely determine which it should go to. Thus some kind of heuristic\nwill be needed. Merges can be worse:\n\nbranch    D-----E\n         /     / \\\nmaster  A-----B---C\n\nIf commit 'E' is from a remote branch, we won't have enough branches\nto go around, and either the merge information would be discarded\n(leaving an incomplete view of history), or an automatically-named\nbranch would need to be made with the history of the other branch.\n\n>  > * Merge information annotation (svn->git)\n>  > * Import of a largish repository with svk or similar merge information into git,\n>  >   and vice versa (eg, exporting git.git with merge tracking as a subversion\n>  >   repo)\n>\n>  Whew!  That's a lot of big milestones, but it's your summer ... :)\n>\n>  I think the merging thing is a nice-to-have, and doing it would just\n>  prove that you can use the metadata that you have collected well.\n\nOkay, I think I'll move some of the milestones into 'nice to have, but\nmight run out of time' then :)\n\n\n\nOn Thu, Mar 20, 2008 at 12:56 AM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Bryan Donlan <bdonlan@gmail.com> wrote:\n>  > I'm planning to apply for the git summer of code project. My proposal\n>  > is based on the project idea of a subversion gateway for git,\n>  > implemented with a new subversion filesystem layer. A draft of my\n>  > proposal follows; I'd appreciate any comments/questions on it before\n>  > the application period proper begins.\n>\n>  Very cool.  Have you had a chance to look at the prototype python\n>  implementation of an SVN server that Julian Phillips started?\n>\n>   http://git.q42.co.uk/w/git_svn_server.git\n>\n>  I'm just curious what your take is regarding this approach.  Why\n>  would you choose to construct libsvn-fs-git over a standalone server?\n>  There are several advantages and drawbacks to both approaches.\n>  I am not advocating over the other, but want to make sure you have\n>  thought it through for yourself.\n\nThe main reason is generality - I want to give the user the choice of\nsvn://, http://, or even (for testing, I hope) file:// access to the\nrepository. Also, it may be possible to convert a subversion\nrepository to git with just a svnadmin load, once sufficient support\nis in place.\n\nAllowing the svn server and repository access layer to go in front of\nthe git filesystem also lets me benefit from any sanity checks in\nthere, hopefully reducing the impact of any possible security bugs.\n\nFinally, it seems a bit simpler of an API than the libsvn_ra API that\nsvnserve wraps. For one, I don't need to keep track of the client's\nworking copy state, nor to I need to mess with non-blocking IO to\navoid deadlocks.\n\n>  > I intend to support the following:\n>  > * Full or near full (possibly forbidding modification of the toplevel\n>  > trunk/ branches/ tags/ structure) read/write access from subversion\n>\n>  That's probably the only sane way to go about it; disallow read/write\n>  on the top level, map whatever branch \"HEAD\" points to in Git to the\n>  trunk/, put the other branches in branches/ and the tags under tags/.\n>  Block everything else.\n\nIt'd be nice to unblock later, to allow svnadmin load to effectively\nconvert a svn repository to git.\n\n>  > * Support for syncing svn:executable with git file mode information\n>  > * Representation of git merge data using svk:merge and/or svn:mergeinfo\n>  > * Syncing .gitignore and svn:ignore data\n>\n>  These are gravy.  Sure they are going to be difficult to make work,\n>  but people can limp by without them.  Most users who want an SVN\n>  client to speak to a Git repository are trying to do so from a\n>  platform that does not honor executable bits (hi Windows!) and\n>  telling users to edit the funny \".gitignore\" file to alter ignore\n>  lists is something they can work around without too much trouble\n>  if they are already able to modify and commit files.\n>\n>  Though their clients won't provide the proper ignore support out\n>  of the box.  *sigh*\n\nMhm, perhaps I'll move this to a later (would-be-nice-if-there's-time)\nmilestone then.\n\n>  > As both subversion and git are written in C, this driver will also be in C.\n>\n>  I think you may have underestimated the challenges associated with\n>  linking \"libgit.a\" (which is _not_ a library) with SVN.  Critical\n>  routines within libgit that you want to be able to invoke will do\n>  not so nice things like leak massive amounts of memory or cause\n>  your process to terminate if the function is fed an invalid input.\n>\n>  Most of the C code of Git is designed for single-shot execution.\n>  We leak memory like mad because it is more efficient to load up what\n>  we need, exit, and let the OS just return the pages to the free pool.\n>  Long running processes have simply not been something we do.\n\nMmm, and on further inspection there's global variables everywhere, no\nlocking, and what looks like not much support for multiple git\ndirectories. I might have to skip libgit and just write my own code to\naccess the git object store - subversion requires thread safety, and\nsupport for opening multiple filesystems (or even the same filesystem\nmultiple times).\n\n>  > My current plan for storing the additional information the subversion side will\n>  > need (fileprops, revprops, copyfrom information...) is to create an additional\n>  > branch on the git repository (possibly .git-svn or similar) to hold the\n>  > necessary metadata. Configuration, including author maps, branch/tag maps,\n>  > etc, would be on another branch (git-svn-config or similar).\n>  >\n>  > The layout might look like this:\n>  >\n>  > /tree/{trunk/,branches/,tags/} - the tree as svn currently sees it\n>\n>  I don't think you'd want to put a copy of the tree inside of a tree,\n>  as this can then get out of sync with changes made directly through\n>  git, plus you run into issues about connecting the two histories\n>  together in a meaningful way.\n>\n>  I would suggest having the root directory of the SVN tree be built\n>  on the fly based upon the list of available branches and tags in\n>  the Git repository (aka the output of git-show-ref).\n\nSubversion absolutely requires that revisions be immutable - the\nclient will do things like present the server with a revision number\nand ask for all changes since then. As such, once we decide what a\nrevision looks like, we must record that and use the same tree in the\nfuture. Explicitly saving the tree seemed to me like the most\neffective way to do that - and it also means many of the filesystem\naccess APIs can simply directly inspect this git tree.\n\nAs for synchronization, it'll be necessary to explicitly convert git\ncommits to subversion revisions anyway, as actions such as making a\nnew branch, which in git doesn't need a new commit, do require a copy\noperation and commit in subversion.\n\n>  > /props/{trunk/,branches/,tags/} - file properties; props on directories will be\n>  >   represented with a reserved filename (._GIT-SVN-DIRPROPS perhaps)\n>  >   copyfrom information might be in /props, or in a seperate tree\n>\n>  How critical are file properties to an SVN client for proper\n>  functioning?  Given the challenges already in front of you for this\n>  project I would almost encourage you to avoid dealing with file\n>  level properties.  Its hard enough to make something that speaks SVN\n>  on the wire but reads/writes Git on disk, not to mention you have\n>  to somehow \"flatten\" the Git DAG down into a sequential revision\n>  namespace to make the SVN clients happy.  So deferring property\n>  support until later may be wise.\n\nOkay. Property support is probably not too difficult, but I see no\nproblem with moving it to a later milestone.\n\n>  > /revprops/NNN - revision properties for the given revision number\n>\n>  Ditto.  Aside from the special merge properties you mentioned,\n>  I wonder if you can simply avoid implementing support for these\n>  early on.\n\nMerge properties are actually file properties. Revprops are needed for\nsvn log support; they store the commit message, author, and date at\nthe least. It may be possible to get the subversion client limping\nalong without them though.\n\n>  > /revmap/NNN - a reference to the commit hash in the .git-svn branch\n>  >   corresponding to the given subversion revision number\n>\n>  How about using a simple flat file interface?  To initially prime\n>  the file you can do something like:\n>\n>         git rev-list --topo-order --date-order --reverse --all >.git/svn-map\n>\n>  and then number the revisions by the line number that they appear on.\n>  Locating a Git SHA-1 for a specific SVN revision would be a simple\n>  case of lseek(fd, 41 * rev, SEEK_SET).  Going the other direction\n>  would be more of a challenge, but is still doable.\n>\n>  Updating the file should just require appending new commits; if\n>  the SVN client wants a new commit you append on and return the\n>  line number.  If Git has caused new commits not in this file you\n>  need to rebuild the log.  This would have to be done incrementally,\n>  to prevent changing a prior SVN revision number that clients may\n>  already know about.\n\nHm, yes, that does seem a better way. Actually since the canonical\nrevision numbers are in the commits on the metadata branch anyway,\nthis can just be a cache, and not part of the git tree itself.\n\n\n\nOn Thu, Mar 20, 2008 at 2:18 AM, Harvey Harrison\n<harvey.harrison@gmail.com> wrote:\n> On Thu, 2008-03-20 at 00:56 -0400, Shawn O. Pearce wrote:\n>  > Bryan Donlan <bdonlan@gmail.com> wrote:\n>\n> > > /revmap/NNN - a reference to the commit hash in the .git-svn branch\n>  > >   corresponding to the given subversion revision number\n>  >\n>  > How about using a simple flat file interface?  To initially prime\n>  > the file you can do something like:\n>  >\n>  >       git rev-list --topo-order --date-order --reverse --all >.git/svn-map\n>  >\n>  > and then number the revisions by the line number that they appear on.\n>  > Locating a Git SHA-1 for a specific SVN revision would be a simple\n>  > case of lseek(fd, 41 * rev, SEEK_SET).  Going the other direction\n>  > would be more of a challenge, but is still doable.\n>  >\n>  > Updating the file should just require appending new commits; if\n>  > the SVN client wants a new commit you append on and return the\n>  > line number.  If Git has caused new commits not in this file you\n>  > need to rebuild the log.  This would have to be done incrementally,\n>  > to prevent changing a prior SVN revision number that clients may\n>  > already know about.\n>\n>  Why not just copy the rev_map format git-svn already uses, it's pretty\n>  efficient.\n\n2008/3/20 Jakub Narebski <jnareb@gmail.com>:\n>  By the way, have you looked into what git-svn uses? IIRC it had some\n>  improvements to avoid spending more disk space on SVN revno <-> Git SHA-1\n>  mapping than on the repository itself...\n\ngit-svn's rev_map format's designed to support gaps in revision\nnumbers. Since I need to record all revisions, the four-byte revision\nnumber field can go - but apart from that, that seems fine for the\nrevision map cache.\n\nOn Thu, Mar 20, 2008 at 5:22 AM, Julian Phillips\n<julian@quantumfyre.co.uk> wrote:\n> On Thu, 20 Mar 2008, Shawn O. Pearce wrote:\n>\n>  > Bryan Donlan <bdonlan@gmail.com> wrote:\n\n>  >> /props/{trunk/,branches/,tags/} - file properties; props on directories will be\n>  >>   represented with a reserved filename (._GIT-SVN-DIRPROPS perhaps)\n>  >>   copyfrom information might be in /props, or in a seperate tree\n>  >\n>  > How critical are file properties to an SVN client for proper\n>  > functioning?  Given the challenges already in front of you for this\n>  > project I would almost encourage you to avoid dealing with file\n>  > level properties.  Its hard enough to make something that speaks SVN\n>  > on the wire but reads/writes Git on disk, not to mention you have\n>  > to somehow \"flatten\" the Git DAG down into a sequential revision\n>  > namespace to make the SVN clients happy.  So deferring property\n>  > support until later may be wise.\n>\n>  You might need to get svn:eol-style working to prevent the svn client from\n>  munging any binary files?  Can't think of any other vital properties atm.\n\nThe subversion client won't touch your binaries /unless/ svn:eol-style\nis set. If I just return an empty set of properties it'll leave them\nalone.\n\nDealing with eol-style in particular is somewhat hard with git - it\nworks by transforming the file into a canonical end-of-line style on\nthe client before sending it to the server, and transforming back when\nit's checked out. I don't know how it'd behave if someone on the git\nside commited with a different eol style than it expects.\n\n>\n>\n>  >> /revprops/NNN - revision properties for the given revision number\n>  >\n>  > Ditto.  Aside from the special merge properties you mentioned,\n>  > I wonder if you can simply avoid implementing support for these\n>  > early on.\n>\n>  Since you have to explicitly enable revprop editing in the subversion\n>  repository by enabling a hook script, I should think that this was\n>  definately something that could be left at the bottom of the TODO list ...\n>\n>  Though you do need to be able to convert commit info into the appropriate\n>  revprops (e.g. commit msg -> svn:log revprop)\n\nRight, and at that point adding full revprop editing ought not to be too hard.\n\n\nAnyway, I'll rework my milestones a bit before I submit the proper\nproposal. Also, after looking at libgit in a bit more detail, I think\nit might be necessary to not use it after all, as subversion requires\nsupport for multiple open repositories, as well as thread safety (at\nleast when accessing different open repo from different threads).\nPerhaps a thread-safe git library would be a nice SoC project as well?\n:)\n\nThanks for the feedback,\n\nBryan Donlan\n"},{"id":"72652","messageId":"alpine.LSU.1.00.0803221229410.4124@racer.site","threadId":"12765","inReplyTo":"3e8340490803212202r6dbaa9eel544ba2b4b8e8d0c7@mail.gmail.com","subject":"thread-safe libgit.a as a GSoC project, was Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-03-22T11:35:26Z","receivedAt":"2008-03-22T11:35:26Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 22 Mar 2008, Bryan Donlan wrote:\n\n> On Wed, Mar 19, 2008 at 12:08 AM, Bryan Donlan <bdonlan@gmail.com> wrote:\n>\n> >  I'm planning to apply for the git summer of code project. My proposal \n> >  is based on the project idea of a subversion gateway for git, \n> >  implemented with a new subversion filesystem layer. A draft of my \n> >  proposal follows; I'd appreciate any comments/questions on it before \n> >  the application period proper begins.\n> \n> Thanks for all the comments. To try to avoid spamming the list, I've\n> replied in a single message, if it'd be better to reply individually\n> in the future please let me know.\n\nMy preference is to have single replies, possibly changing the subject \n(\"xyz, was Re: blabla\"), but it is maybe just me.\n\n> Also, after looking at libgit in a bit more detail, I think it might be \n> necessary to not use it after all, as subversion requires support for \n> multiple open repositories, as well as thread safety (at least when \n> accessing different open repo from different threads). Perhaps a \n> thread-safe git library would be a nice SoC project as well?\n\nAs I said on IRC yesterday, I think that such a libgit.a would be nice, \n_but_\n\n- a lot of git programs expect to be one-shot, and libgit.a shows that,\n\n- not many people will help you with your effort, but just ignore it and \n  actively introduce things that do not help libification (at least that's \n  my experience),\n\n- unless you have a proper need for such a library, I do not think there \n  is enough motivation to actually get it to completion.\n\nI once thought that libification would be nice, and important, but as I do \nnot need it myself, I reversed my opinion.\n\nCiao,\nDscho\n"},{"id":"72715","messageId":"5d46db230803221834n7c230447r2afffdaae79e4068@mail.gmail.com","threadId":"12765","inReplyTo":"alpine.LSU.1.00.0803221229410.4124@racer.site","subject":"Re: thread-safe libgit.a as a GSoC project, was Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Govind Salinas","fromEmail":"govind@sophiasuchtig.com","sentAt":"2008-03-23T01:34:38Z","receivedAt":"2008-03-23T01:34:38Z","isPatch":false,"sender":{"key":"govind@sophiasuchtig.com","avatar":null},"body":"On Sat, Mar 22, 2008 at 6:35 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n>  On Sat, 22 Mar 2008, Bryan Donlan wrote:\n>  > Also, after looking at libgit in a bit more detail, I think it might be\n>  > necessary to not use it after all, as subversion requires support for\n>  > multiple open repositories, as well as thread safety (at least when\n>  > accessing different open repo from different threads). Perhaps a\n>  > thread-safe git library would be a nice SoC project as well?\n>\n>  As I said on IRC yesterday, I think that such a libgit.a would be nice,\n>  _but_\n>\n>  - a lot of git programs expect to be one-shot, and libgit.a shows that,\n>\n>  - not many people will help you with your effort, but just ignore it and\n>   actively introduce things that do not help libification (at least that's\n>   my experience),\n>\n>  - unless you have a proper need for such a library, I do not think there\n>   is enough motivation to actually get it to completion.\n>\n\nI would use it for my pyrite work, although it will be some time before I\ncould contribute to such an effort.  I expect it would be useful for\nanyone who wants to make a language binding that uses native\ngit underneath.\n\nJust so you know *someone* will use it.\n\nThanks,\nGovind.\n"},{"id":"72718","messageId":"alpine.LSU.1.00.0803230308540.4353@racer.site","threadId":"12765","inReplyTo":"5d46db230803221834n7c230447r2afffdaae79e4068@mail.gmail.com","subject":"Re: thread-safe libgit.a as a GSoC project, was Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-03-23T02:10:12Z","receivedAt":"2008-03-23T02:10:12Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 22 Mar 2008, Govind Salinas wrote:\n\n> I would use it for my pyrite work, although it will be some time before \n> I could contribute to such an effort.  I expect it would be useful for \n> anyone who wants to make a language binding that uses native git \n> underneath.\n> \n> Just so you know *someone* will use it.\n\nI know people would use it.  My point was: those people that want to use \nit have the best starting point to make it happen, because they (should) \nactually care about libification.\n\nCiao,\nDscho\n"},{"id":"72938","messageId":"3e8340490803241250w186d6ae2l38031521c2d5c6c2@mail.gmail.com","threadId":"12765","inReplyTo":"alpine.LSU.1.00.0803221229410.4124@racer.site","subject":"Re: thread-safe libgit.a as a GSoC project, was Re: [SoC RFC] libsvn-fs-git: A git backend for the subversion filesystem","fromName":"Bryan Donlan","fromEmail":"bdonlan@gmail.com","sentAt":"2008-03-24T19:50:04Z","receivedAt":"2008-03-24T19:50:04Z","isPatch":false,"sender":{"key":"bdonlan@gmail.com","avatar":null},"body":"On Sat, Mar 22, 2008 at 7:35 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n>  On Sat, 22 Mar 2008, Bryan Donlan wrote:\n>\n>  > On Wed, Mar 19, 2008 at 12:08 AM, Bryan Donlan <bdonlan@gmail.com> wrote:\n>  >\n>  > >  I'm planning to apply for the git summer of code project. My proposal\n>  > >  is based on the project idea of a subversion gateway for git,\n>  > >  implemented with a new subversion filesystem layer. A draft of my\n>  > >  proposal follows; I'd appreciate any comments/questions on it before\n>  > >  the application period proper begins.\n>  >\n>  > Thanks for all the comments. To try to avoid spamming the list, I've\n>  > replied in a single message, if it'd be better to reply individually\n>  > in the future please let me know.\n>\n>  My preference is to have single replies, possibly changing the subject\n>  (\"xyz, was Re: blabla\"), but it is maybe just me.\n>\n>  > Also, after looking at libgit in a bit more detail, I think it might be\n>  > necessary to not use it after all, as subversion requires support for\n>  > multiple open repositories, as well as thread safety (at least when\n>  > accessing different open repo from different threads). Perhaps a\n>  > thread-safe git library would be a nice SoC project as well?\n>\n>  As I said on IRC yesterday, I think that such a libgit.a would be nice,\n>  _but_\n>\n>  - a lot of git programs expect to be one-shot, and libgit.a shows that,\n>\n>  - not many people will help you with your effort, but just ignore it and\n>   actively introduce things that do not help libification (at least that's\n>   my experience),\n>\n>  - unless you have a proper need for such a library, I do not think there\n>   is enough motivation to actually get it to completion.\n>\n>  I once thought that libification would be nice, and important, but as I do\n>  not need it myself, I reversed my opinion.\n\nAll right. If I do end up having to recreate (thread-safe,\nmultiple-git-dir-safe) logic for my project, I'll try to keep in mind\nthe possibility of spinning it off into a proper library later though\n:)\n"}]}