{"thread":{"id":"27034","subject":"Re: GSoC proposal for svn remote helper","startedAt":"2011-04-08T03:42:07Z","lastAt":"2011-04-09T23:19:31Z","messageCount":7,"participants":["Dmitry Ivankov","Jonathan Nieder","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165396","messageId":"BANLkTinHE-E5_mK8aKYv2f7yExVvfOFVRw@mail.gmail.com","threadId":"27034","inReplyTo":"BANLkTinYyxxkZpmEF2PYXMb_BjCVcbTkYw@mail.gmail.com","subject":"Re: GSoC proposal for svn remote helper","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-04-08T03:42:07Z","receivedAt":"2011-04-08T03:42:07Z","isPatch":false,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"resending in plain-text to make vger.kernel.org happy.\n\nHi.\n\nThis is the second iteration of my GSoC proposal, the first one was on\nmelange and\ngot nice responses from Jonathan Nieder, David Barr and Sverre Rabbelier.\nHowever the conclusion was that it looks too ambitious, and that I should roll\nout it to the list. So here it is in a diff-style.\n\nI would like to work on \"Remote helper for Subversion and git-svn\".\nMy major motivation is to make git-svn repository easy to clone, and to make\ngit-svn (fetch) faster on huge repositories.\n\nProject Goals:\n+ * Design and create fully functional prototype of new git-svn which is\ncloneable and quite fast. By fully functional I mean that it'll be\nable to fetch, push, etc. but probably won't have automatic tags and\nbranches discovery and like, but will allow it to be implemented on\ntop. Oh, it just hit me that given a path (read trunk) to track and a\nsvndump it looks trivial to discover all it's branches - just seek for\ncopies.\n+ * Get all the needed core git changes merged. Some of these exist already and\nonly need help with polishing, reviewing and merging.\n- * Complete git-remote-svn and get it merged.\n- * Implement new git-svn and get it merged too.\n+ * Make the prototype as close to being merged as possible.\n\nMilestones for prototype functionality:\n * Be able to track whole / as the only remote branch.\n - * Be able to dumb track some path as the only remote branch. Could be done\n either via pruning in git-remote-svn or via maintaining custom branch,\n probably with the help of git-filter-branch.\n - * Finally, be able to work with multiple svn branches. There are many ways to\n achieve this and several kinds of what features the ability to work includes,\n so there even should be a milestone for choosing the ones to implement.\n + * Be able to track several \"paths\" as branches so that they have expected\n history (whole path copying is branching), and so that these branches are\n cloneable (will be the same in different git-svn repositories tracking the\n same svn repo, under reasonable assumptions like svn:author not being\n propedited :) ).\n * Anything else that'll appear to be interesting and related.\n\n Sorry for the late submission to this list, I was really puzzled on how to\n make my proposal more realistic and still as useful for git-svn as possible.\n"},{"id":"165398","messageId":"20110408052126.GA22256@elie","threadId":"27034","inReplyTo":"BANLkTinHE-E5_mK8aKYv2f7yExVvfOFVRw@mail.gmail.com","subject":"Re: GSoC proposal for svn remote helper","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-04-08T05:21:26Z","receivedAt":"2011-04-08T05:21:26Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(+cc: Eric who brought us git-svn)\nHi Dmitry,\n\nDmitry Ivankov wrote:\n\n> This is the second iteration of my GSoC proposal\n\nGreat; let's iron this out.\n\n> I would like to work on \"Remote helper for Subversion and git-svn\".\n> My major motivation is to make git-svn repository easy to clone, and to make\n> git-svn (fetch) faster on huge repositories.\n\nSo, my new first impression is that this goal might make things hard[1].\n\nI think replacing git-svn with an imperfect emulation would not leave\npeople happy.  Existing configurations need to continue to work.\n\n> Project Goals:\n> + * Design and create fully functional prototype of new git-svn which is\n> cloneable and quite fast.\n\n*If* one does not have this goal (\"new git-svn\") then there is a\nchance to move past some of git-svn's limitations[2].\n\nAll that said, these tools could be used to speed up git-svn.  \n\n> By fully functional I mean that it'll be\n> able to fetch, push, etc. but probably won't have automatic tags and\n> branches discovery and like, but will allow it to be implemented on\n> top. Oh, it just hit me that given a path (read trunk) to track and a\n> svndump it looks trivial to discover all it's branches - just seek for\n> copies.\n\nAs mentioned before, this sounds very ambitious.  Once we have a\ntimeline showing how this breaks down into small steps it should\nhopefully be clearer way.\n\n> + * Get all the needed core git changes merged.\n\nThe following is probably controversial.  It's my opinion only.\n\nSince you can't control what other people do, I don't think it's right\nto judge your project's success or failure based on whether it gets\nmerged.  Put another way, the product of your work that can be judged\nis not whatever fraction gets accepted in git.git by the end of the\nsummer[3].\n\nSo I think the goal is whatever it is (a working and suitable \"git\nclone svn://foo\" command, say) and getting feedback by pushing changes\nupstream and responding to it is a part of how that happens.\n\nAt some point there will probably be a point of no return --- \"if the\ndesign of this patch is not right, I would have to rewrite everything\non top of a redesign of it\".  I'd encourage getting input on such\npatches _very_ early and working hard to get them merged at least to\n\"next\" (i.e., to have a rough consensus that they are suitable modulo\nsmall tweaks).  I would love it if the proposal included a timeline\npointing out some examples of this.\n\n> Some of these exist already and\n> only need help with polishing, reviewing and merging.\n\nDo you mean support for parsing \"svnadmin dump --deltas\" output?  It\nis already polished and reviewed; it's only sitting out-of-tree for\nnow because it makes the commandline usage awkward and it would be\nnice to merge some improvements to that at the same time.\n\n> + * Make the prototype as close to being merged as possible.\n\nThat's kind of vague, you know. :)\n\n> Milestones for prototype functionality:\n[list of features snipped]\n\nCould you say something about how you would go about implementing\nthese?\n\nSorry for the ramble, and thanks for working on this.\n\nCiao,\nJonathan\n\n[1] git-svn.perl is a work of art and a wonder to behold, and if your aim\nis to make a compatible replacement for it, the first step will be to\nunderstand its design deeply.  And the thing is, that much, while\nvaluable anyway, is pretty hard already.\n\nYou see, \"git svn\" has heuristics for\n\n - matching up git history to svn history by reading commit messages;\n - pushing mergy history as linear history by rebasing internally\n   (dcommit);\n - finding the branches, merges, branch renames, and so on in an\n   imperfectly structured history (find_parent etc)\n - what particular paths are relevant  (--ignore-paths)\n\nand maintains some of its own data in the repository:\n\n - a configuration scheme and wide variety of supported configurations;\n - a log for unhandled pieces of history;\n - a cache mapping svn revision numbers to git commits\n\nand people rely a lot on an odd coincidence:\n\n - using \"git svn clone\" twice with the same configuration on the same\n   repository will, at least most of the time, give the same commit\n   names.\n\n[2] Well, it mostly comes down to one limitation.  To give a quick\nsketch:\n\nIf I clone a repository with \"git svn\", then I am in a way a\nsecond-class citizen.  The history shown with \"git log\" is filled\nwith \"git-svn-id:\" lines that are not very interesting to me (the\nrevision number is still interesting, of course).  I cannot use\n\"git push\" to push my work, and in fact I cannot push my work as a\nbranch reflecting the real development history at all --- I have to\nrebase it at the same time as pushing.  Whenever I push, the commit\nnames for my work change, so other branches based on my work don't\nshow up in \"gitk\" as based on my work any more.\n\nWouldn't it be nicer to be able to do\n\n alice$ git clone svn::http://svn.apache.org/repos/asf/subversion\n alice$ cd subversion\n alice$ ... hack hack hack ...\n\n bob$ git clone 'alice:~/src/subversion'\n bob$ cd subversion\n bob$ ... hack hack hack ...;\t# make some changes on top of alice's work\n\n alice$ git fetch origin; # anything new upstream?\n alice$ git push origin; # push my changes upstream\n\n bob$ git remote add upstream svn::http://svn.apache.org/repos/asf/subversion\n bob$ git fetch upstream\n bob$ # push my changes on top of alice's (which were already pushed):\n bob$ git push upstream\n\nThat is the dream.  Because there is not a clearly appropriate\none-to-one mapping between possible svn histories and possible git\nhistories, there are going to have to be limitations[1], but that is\nan ideal to strive for.\n\nSounds hard, maybe?  Yeah, it is, but getting at least fetch support\nusing the tools David and Ram made sounds easier to me than a fully\ncompatible replacement for git-svn.\n\n[3] Meanwhile, just writing and publishing code is not enough, since\nthe code might have a fatal flaw that means no one will use it (\"ivory\ntour syndrome\").  So what do I mean by the above?\n\nAs students work, I hope they will keep the mailing list posted on\ntheir progress and find small pieces to review and merge early.  In\nresponse they might get some questions and suggestions for\nimprovement; the response to these is just as important as the code.\n\nOn one hand this feedback is an important sanity check on the broad\nfeatures of your work and a means to get the details right for\ninclusion in git (i.e., get it merged).  On the other hand, one should\nnot be tempted by interesting side tracks and avoid getting the actual\nproject done; you have to be able to say \"no, I will not be working on\nthat\".  Out of these conversations emerge better code and\ndocumentation of the design in the form of list archives.\n\nSee [4] for a better explanation of this workflow.\n\n[4] http://thread.gmane.org/gmane.comp.version-control.git/142623/focus=142877\n"},{"id":"165405","messageId":"20110408071153.GA23051@elie","threadId":"27034","inReplyTo":"20110408052126.GA22256@elie","subject":"Re: GSoC proposal for svn remote helper","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-04-08T07:11:53Z","receivedAt":"2011-04-08T07:11:53Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi again,\n\nA small clarification.\n\nJonathan Nieder wrote:\n\n> As mentioned before, this sounds very ambitious.  Once we have a\n> timeline showing how this breaks down into small steps it should\n> hopefully be clearer way.\n\nAgh, that sentence doesn't even parse.  I ought to have have said:\n\"Breaking the task into concrete steps will make it easier to see what\nis realistic.\"\n"},{"id":"165410","messageId":"BANLkTinWtqNYvZP2yamx831_U9KHZ4x3qA@mail.gmail.com","threadId":"27034","inReplyTo":"20110408052126.GA22256@elie","subject":"Re: GSoC proposal for svn remote helper","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-04-08T08:47:58Z","receivedAt":"2011-04-08T08:47:58Z","isPatch":false,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"On Fri, Apr 8, 2011 at 11:21 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> (+cc: Eric who brought us git-svn)\n> Hi Dmitry,\n>\n> Dmitry Ivankov wrote:\n>\n>> This is the second iteration of my GSoC proposal\n>\n> Great; let's iron this out.\n>\n>> I would like to work on \"Remote helper for Subversion and git-svn\".\n>> My major motivation is to make git-svn repository easy to clone, and to make\n>> git-svn (fetch) faster on huge repositories.\n>\n> So, my new first impression is that this goal might make things hard[1].\n>\n> I think replacing git-svn with an imperfect emulation would not leave\n> people happy.  Existing configurations need to continue to work.\nI should have used different names for current git-svn.perl and what\nshould be tracking svn repo in git somewhat better than git-svn.perl.\nMaybe call it git-svn-ng. It should definitely support common\nworkflows, but I think that it should not be too close in\nconfiguration and behavior details. git-svn.perl is a personal setup\nand I doubt someone shares it somehow, so transition won't be hard. My\nfocus is on git-svn-ng core operations - fast fetch, push, ability to\nclone at least upstream svn state from another git repo, and I see way\nto a complete replacement as follows:\n1) introductory step, nothing new compared to git-svn.perl, but\nexpected to be already faster on allowed operations\ngit clone svn::/svnroot (svnroot can be a path in local or remote\nrepo, this is already supported by svn layers)\n..hack..\ngit svn dcommit or like - put changes to the remote\n\n2) allow private clones, that is be able to exchange svn updates or do\ninitial clone via git\ngit clone /somewhere/git_ro_version_of_svn.git (maybe some additional\ncommand/key to get svn metadata)\ngit remote add upstream svn://svnroot (git remote update should be as\nquick as on origin)\n..hack..\ngit svn dcommit\n\n3) allow tracking of a path, something like, intermediate,\nfunctionally the same or slightly different in corner cases\ngit remote add svn svn:://svnroot/ (it's still a good idea to have\nwhole root specified)\ngit svn branch trunk svn/trunk@12 (at first it'll be ok to behave like\nas if root is svnroot/trunk, just a new syntax)\n\n4) follow tracked path, create branches\ngit remote add svn svn:://svnroot/\ngit svn branch trunk svn/trunk@12\ngit svn branch trunk svn/branches/stable@14\ngit remote update\ngit merge-base svn/trunk svn/branches/stable\n#for example svn/trunk@13\ngit remote update\n#got svn/trunk@23 -> svn/branches/fixups (maybe if option discover is\nset; here we need the root to check that the destination is ok)\ngit svn branch trunk svn/branches/old@13 (if somehow it wasn't discovered)\ngit remote update (maybe, if we store whole root history, will be fast)\nand also create svn branches with git, ugly way is to:\ngit checkout svnroot\ncp -r trunk branches/experimental\ngit add branches/experimental\ngit commit && git svn dcommit\nor maybe:\ngit svn branch trunk svn/branches/experimental\ngit checkout svn/branches/experimental\ngit commit --allow-empty -m \"create new branch\"\ngit svn dcommit\n\nThese four imho are fine as a minimal functionality - one can use it\nfor git-svn.perl replacement if perfomance and clones are more\nimportant than heuristics and easy of use.\n\n5) tweak 3-4), allow tracking branches, sharing branches, committing\nmerges, and a lot more, see below for an idea of getting rid of\nrebase.\n\nUh, that became a long story, anyway, I clearly see that there is\ngit-svn.perl which can do git<->svn interaction quite comfortable for\nusers, there is almost ready faster git<->svn transport, there are\nalready a bunch remote-helpers available on the good side, and on the\nbad it's currently hard to get even initial clone. So I consider it\nquite possible for a GSoC project to get a kind git-svn-ng that is\ncloneable, faster than git-svn.perl, and hopefully doesn't involve\ndeep understanding and patching git-svn.perl. All this with the idea\nof extending it to handle git workflows between two git-svn-ng clones\nand a svn repo, or just better git workflows inside one git-svn-ng.\n\n>\n>> Project Goals:\n>> + * Design and create fully functional prototype of new git-svn which is\n>> cloneable and quite fast.\n>\n> *If* one does not have this goal (\"new git-svn\") then there is a\n> chance to move past some of git-svn's limitations[2].\nI'll write inline in [2]\n>\n> All that said, these tools could be used to speed up git-svn.\n>\n>> By fully functional I mean that it'll be\n>> able to fetch, push, etc. but probably won't have automatic tags and\n>> branches discovery and like, but will allow it to be implemented on\n>> top. Oh, it just hit me that given a path (read trunk) to track and a\n>> svndump it looks trivial to discover all it's branches - just seek for\n>> copies.\n>\n> As mentioned before, this sounds very ambitious.  Once we have a\n> timeline showing how this breaks down into small steps it should\n> hopefully be clearer way.\nOk, I'll try to break it into some steps.\n>\n>> + * Get all the needed core git changes merged.\n>\n> The following is probably controversial.  It's my opinion only.\n>\n> Since you can't control what other people do, I don't think it's right\n> to judge your project's success or failure based on whether it gets\n> merged.  Put another way, the product of your work that can be judged\n> is not whatever fraction gets accepted in git.git by the end of the\n> summer[3].\nThat means one can't blame them if it's not merged, but also git is a\nmentor and that'd be strange to choose a goal like \"write a thing I'll\nuse myself\" :)\n\n> So I think the goal is whatever it is (a working and suitable \"git\n> clone svn://foo\" command, say) and getting feedback by pushing changes\n> upstream and responding to it is a part of how that happens.\nThat makes a great sense to keep both things in mind, yes :)\n>\n> At some point there will probably be a point of no return --- \"if the\n> design of this patch is not right, I would have to rewrite everything\n> on top of a redesign of it\".  I'd encourage getting input on such\n> patches _very_ early and working hard to get them merged at least to\n> \"next\" (i.e., to have a rough consensus that they are suitable modulo\n> small tweaks).  I would love it if the proposal included a timeline\n> pointing out some examples of this.\n>\n>> Some of these exist already and\n>> only need help with polishing, reviewing and merging.\n>\n> Do you mean support for parsing \"svnadmin dump --deltas\" output?  It\n> is already polished and reviewed; it's only sitting out-of-tree for\n> now because it makes the commandline usage awkward and it would be\n> nice to merge some improvements to that at the same time.\nYep, help in whatever way I can with this one and also I saw helpers\nbranches introducing new remote-helpers commands or extending existing\ncore functions. I hope all the needed core changes will quickly popup\non early stages.\n>\n>> + * Make the prototype as close to being merged as possible.\n>\n> That's kind of vague, you know. :)\nYep, but I don't know any good metric for this :)\n>\n>> Milestones for prototype functionality:\n> [list of features snipped]\n>\n> Could you say something about how you would go about implementing\n> these?\n>\n> Sorry for the ramble, and thanks for working on this.\nNo problem absolutely, thank you for feedback, I like the challenges.\n>\n> Ciao,\n> Jonathan\n>\n> [1] git-svn.perl is a work of art and a wonder to behold, and if your aim\n> is to make a compatible replacement for it, the first step will be to\n> understand its design deeply.  And the thing is, that much, while\n> valuable anyway, is pretty hard already.\n[skipped git-svn.perl heuristics]\n> and people rely a lot on an odd coincidence:\n>\n>  - using \"git svn clone\" twice with the same configuration on the same\n>   repository will, at least most of the time, give the same commit\n>   names.\nI want this to happen always wherever two clone & fetch sequences\nreach the same remote revision.\n\n>\n> [2] Well, it mostly comes down to one limitation.  To give a quick\n> sketch:\n>\n> If I clone a repository with \"git svn\", then I am in a way a\n> second-class citizen.  The history shown with \"git log\" is filled\n> with \"git-svn-id:\" lines that are not very interesting to me (the\n> revision number is still interesting, of course).\nAs already mentioned I didn't mean exact emulation of git-svn.\nIdeally I'd like git commit object to include only immutable svn data,\nand even more it should be the same after a round trip to the svn repo\nand back.\n\n> I cannot use\n> \"git push\" to push my work, and in fact I cannot push my work as a\n> branch reflecting the real development history at all --- I have to\n> rebase it at the same time as pushing.  Whenever I push, the commit\n> names for my work change, so other branches based on my work don't\n> show up in \"gitk\" as based on my work any more.\n>\n> Wouldn't it be nicer to be able to do\n>\n>  alice$ git clone svn::http://svn.apache.org/repos/asf/subversion\n>  alice$ cd subversion\n>  alice$ ... hack hack hack ...\n>\n>  bob$ git clone 'alice:~/src/subversion'\n>  bob$ cd subversion\n>  bob$ ... hack hack hack ...;   # make some changes on top of alice's work\n>\n>  alice$ git fetch origin; # anything new upstream?\n>  alice$ git push origin; # push my changes upstream\n>\n>  bob$ git remote add upstream svn::http://svn.apache.org/repos/asf/subversion\n>  bob$ git fetch upstream\n>  bob$ # push my changes on top of alice's (which were already pushed):\n>  bob$ git push upstream\n>\n> That is the dream.  Because there is not a clearly appropriate\n> one-to-one mapping between possible svn histories and possible git\n> histories, there are going to have to be limitations[1], but that is\n> an ideal to strive for.\nI have an idea for it, quite raw, but could work.\nWe are limited by svn:\n- commit isn't a push, it's rebase & push. So we don't control the parent ref.\n- there are no branches in svn. There are paths, but we can convert\nthem to branches of another paths.\n- there are no merges in svn. That's a trouble, but maybe we can try\nto use svn:mergeinfo to create and read multiple parent refs.\n- we definitely want to keep everything needed inside svn, or we are\nsure to diverge in different clones sometime.\n\nSo what we do:\n- get rid of svn revision: just have path@rev -> sha1 mapping\nseparately, of course history of path@rev should look like the history\nof sha1\n- learn to create and fetch back merge commits: try svn:mergeinfo\n- be sure to control the parents: don't let svn to commit on top of\nsomething different from git parent:\n-- if path wasn't changed in the repo while we were hacking, commit it\nand it'll come back as the same sha1\n-- if it was, create a svn branch of our parent, commit there, and\nthen create a merge commit of these two, commit it and get same merge\nhistory back\n-- and if we are commiting a merge, create/commit to branches as necessary\n\nNot perfect, but it hardly can be cleaner to emulate git history in\nsvn, and get it back unchanged. And it should be optional too, not all\nsvn commits need this.\n\n> Sounds hard, maybe?  Yeah, it is, but getting at least fetch support\n> using the tools David and Ram made sounds easier to me than a fully\n> compatible replacement for git-svn.\n>\n> [3] Meanwhile, just writing and publishing code is not enough, since\n> the code might have a fatal flaw that means no one will use it (\"ivory\n> tour syndrome\").  So what do I mean by the above?\n>\n> As students work, I hope they will keep the mailing list posted on\n> their progress and find small pieces to review and merge early.  In\n> response they might get some questions and suggestions for\n> improvement; the response to these is just as important as the code.\n>\n> On one hand this feedback is an important sanity check on the broad\n> features of your work and a means to get the details right for\n> inclusion in git (i.e., get it merged).  On the other hand, one should\n> not be tempted by interesting side tracks and avoid getting the actual\n> project done; you have to be able to say \"no, I will not be working on\n> that\".  Out of these conversations emerge better code and\n> documentation of the design in the form of list archives.\n>\n> See [4] for a better explanation of this workflow.\n>\n> [4] http://thread.gmane.org/gmane.comp.version-control.git/142623/focus=142877\n>\n"},{"id":"165462","messageId":"20110408223150.GA27284@elie","threadId":"27034","inReplyTo":"BANLkTinWtqNYvZP2yamx831_U9KHZ4x3qA@mail.gmail.com","subject":"Re: GSoC proposal for svn remote helper","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-04-08T22:31:50Z","receivedAt":"2011-04-08T22:31:50Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi again,\n\nDmitry Ivankov wrote:\n\n> I should have used different names for current git-svn.perl and what\n> should be tracking svn repo in git somewhat better than git-svn.perl.\n> Maybe call it git-svn-ng.\n\nAh, sorry I misunderstood.\n\n> It should definitely support common\n> workflows, but I think that it should not be too close in\n> configuration and behavior details.\n\nAt the moment I am more concerned with what its guts will look like\nthan what features it will support.  A feature list is just a way to\nadvertise how good the guts are. ;-)\n\n> I have an idea for it, quite raw, but could work.\n\nThe discussion here might also be helpful:\nhttp://thread.gmane.org/gmane.comp.version-control.git/159235/focus=159264\n\n> We are limited by svn:\n> - commit isn't a push, it's rebase & push. So we don't control the parent ref.\n\nYes, this is unfortunate.  When I send in my change A, there is no\nserver-side locking to prevent someone else from making a change B\non the same branch first, with a semantically meaningless result.\n\nsvnrdump load has to deal with the same problem.  Possible\nworkarounds:\n\n * prevent pushes to the svn repository (using some out-of-band\n   mechanism) before allowing git to push to it.  This is what svnsync\n   users typically do.\n\n * lock directory, check parent, commit if okay, unlock directory.\n   This is probably the best we can do.\n\n * extend the SVN RA protocol to provide a lock that is automatically\n   removed if the connection is interrupted, and use that.\n\nRam, does the above (#2) make sense?  (I think this would be something\nfor svn fast-import to take care of; by the time the stream gets to\n\"svnrdump load\" hopefully the relevant branches are already locked or\nprotected from contention some other way.)\n\nSo I think this is a problem but not insurmountable.  I think it's\neven possible to ignore at first (by assuming exclusive access to the\ntarget svn repo --- i.e., blame the user :)) as long as it is\ndocumented.  Thanks for a reminder.\n\nReference: http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_ra_svn/protocol\n\n> - there are no branches in svn. There are paths, but we can convert\n> them to branches of another paths.\n\nI think it's reasonable to assume a standard trunk / tags / branches\nlayout and simple \"svn cp\" records for new and renamed branches, at\nleast to start out.\n\n> - there are no merges in svn. That's a trouble, but maybe we can try\n> to use svn:mergeinfo to create and read multiple parent refs.\n\nThe hard part is distinguishing merges from cherry-picks.  Seems\noptional to start out (i.e., a prototype can assume each commit has\na single parent).\n\n> - we definitely want to keep everything needed inside svn, or we are\n> sure to diverge in different clones sometime.\n\nRight --- this means we have to make it very clear to the user when\nthey are creating history that will be unrepresentable in svn.\n\n> So what we do:\n> - get rid of svn revision: just have path@rev -> sha1 mapping\n> separately, of course history of path@rev should look like the history\n> of sha1\n\nThe current idea that's been floating in the air is to use a \"git notes\"\ntree (see git-notes(1)) to store the mapping from git commit names to svn\npath@rev specifications.  This way, (1) they can be easily shared but\n(2) they are not part of the commit name.  So it is easy to discard\nthe mapping, or to adjust it when the project moves to a different svn\nserver.\n\nMeanwhile svn-fe needs quick mapping in the other direction, revs ->\ncommits, for its work.  So one would want to cache the reverse mapping\n(with the notes as the master copy).\n\n> - learn to create and fetch back merge commits: try svn:mergeinfo\n\nSee above.\n\n> - be sure to control the parents: don't let svn to commit on top of\n> something different from git parent:\n\nThis requires more information from svn-fe (copyfrom info).  So either\nsvn-fe needs to take care of this itself, or the next program in the\npipeline would get some information from it (using an extension to the\nfast-import format, an extra output stream, or extra data in comments,\n\"progress\" commands, or the log message).\n\n> -- if path wasn't changed in the repo while we were hacking, commit it\n> and it'll come back as the same sha1\n\nHopefully. :)\n\n> -- if it was, create a svn branch of our parent, commit there, and\n> then create a merge commit of these two, commit it and get same merge\n> history back\n\nYikes.  I don't think typical projects would like the resulting\nhistory.\n\n> -- and if we are commiting a merge, create/commit to branches as necessary\n\nYes, this is an interesting question.  Given a history like this (time\nflowing left to right):\n\n         E --- F --- G\n        /             \\\n A --- B --- C -- D -- H\n\nwhere A is the latest rev of trunk/, how do we push this history to\nsvn?  Where is the name of the side branch recorded in the git\nhistory?\n\nOn answer is that it's possible to learn the historical branch name by\nparsing the commit message for \"H\".  Yuck.\n\nI'd put off pushing merges to start.\n\n> Not perfect, but it hardly can be cleaner to emulate git history in\n> svn, and get it back unchanged. And it should be optional too, not all\n> svn commits need this.\n\nI think the cleanest solution is often to reject a push if it is not\nobvious how to represent it remotely, just as though the remote server\nhad a hook that rejected it.\n\nJonathan\n"},{"id":"165480","messageId":"BANLkTimp_Bc3bSC5WxHuyFmQdNR23qPRMA@mail.gmail.com","threadId":"27034","inReplyTo":"20110408223150.GA27284@elie","subject":"Re: GSoC proposal for svn remote helper","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-04-09T08:21:42Z","receivedAt":"2011-04-09T08:21:42Z","isPatch":false,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"Hi\n\nOn Sat, Apr 9, 2011 at 4:31 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n>\n> At the moment I am more concerned with what its guts will look like\n> than what features it will support.  A feature list is just a way to\n> advertise how good the guts are. ;-)\nMy current view is following:\nUse svnrdump stream to track / in say svnroot branch. It'll be linear\n(1), commits will include actual diffs, and some revprops translated\nto their git counterparts like svn:log,svn:author,svn:date (2). And\nit'll be a bridge for svn interaction.\nIn git-notes tree store sha1 -> svn rev mapping(3), and also for each\nsvn rev store all it's revprops.\nStore and maintain /path branches - there we have some freedom of\nchoosing git parents.\nThat's all about fetch. Obviously svnrdump will be used to push\nfast-forward linear history back to svnroot, or to a /path branch\nwhich goes the same way in fast-forward case.\nAnd to be somewhat usable we want to be able to rebase-push/dcommit\n(if there is a dense stream of svn commits going, we don't want the\nuser to type git rebase, git push and fail until he gets a lucky\ntiming).\nMerges need more thinking, and may be not that necessary for a start.\nPath ignores, or even revision ignores should be possible to implement\nin the code, but just an emergency tools for a user (sometimes people\n(by mistake) commit something enormous or incompatible with\nfilesystems names or like, so that user considers it ok to trash this\nout of his history), also there could be a need for permanent path\nfilter (like track /projX, not /bin) - just the same, be ready that\nsometime it'll have to be implemented.\n\n[skipped some of svn commit \"races\" and merge tricks]\n\n>> -- if it was, create a svn branch of our parent, commit there, and\n>> then create a merge commit of these two, commit it and get same merge\n>> history back\n>\n> Yikes.  I don't think typical projects would like the resulting\n> history.\nWill make them mad, but in some cases it should be ok, if we are\npushing a lengthy topic branch they'll sometimes prefer to see it as\none commit.\n\n> Yes, this is an interesting question.  Given a history like this (time\n> flowing left to right):\n>\n>         E --- F --- G\n>        /             \\\n>  A --- B --- C -- D -- H\n>\n> where A is the latest rev of trunk/, how do we push this history to\n> svn?  Where is the name of the side branch recorded in the git\n> history?\nCould be either autogenerated with some user pattern, like\n/branches/user/tmpXX, or specified explicitly in git-notes or\nsomewhere, or maybe we have already pushed a placeholder branch to svn\nand will commit there.\n>\n> On answer is that it's possible to learn the historical branch name by\n> parsing the commit message for \"H\".  Yuck.\n>\n> I'd put off pushing merges to start.\nIt's definitely not in the minimal plan.\n\n> I think the cleanest solution is often to reject a push if it is not\n> obvious how to represent it remotely, just as though the remote server\n> had a hook that rejected it.\nMakes sense, after all plain svn users want to see svn-like history,\nbecause they still use svn.\n\n(1) In theory we could track whole svnroot merges (from svnroot2 on\nthe same repo for example, or hypothetical merge from another repo)\nbut that's hardly used by anyone.\n(2) The hardest thing is to decide which ones to store in git.\nTranslating more gives a better look and feel, translating less\nreduces the chances to get the same git objects on another clone. And\nwhat should git do if this data changes is not a trivial choice too.\n(3) And a funny thing is could happen that there are path1@rev1\npath2@rev2 that produce the same sha1. That's perfectly fine because\nthey are just refs, care should be taken when choosing a path to\ncommit to though. Also svn will distinguish them, but it's just a\ncorner case.\n"},{"id":"165518","messageId":"20110409231931.GA24345@dcvr.yhbt.net","threadId":"27034","inReplyTo":"20110408052126.GA22256@elie","subject":"Re: GSoC proposal for svn remote helper","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2011-04-09T23:19:31Z","receivedAt":"2011-04-09T23:19:31Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jonathan Nieder <jrnieder@gmail.com> wrote:\n> [1] git-svn.perl is a work of art and a wonder to behold, and if your aim\n> is to make a compatible replacement for it, the first step will be to\n> understand its design deeply.  And the thing is, that much, while\n> valuable anyway, is pretty hard already.\n\nThanks.  I credit the use of automated tests for making things work\nas well as it does, especially given how ugly the code has gotten.\n\nI think the first step to understanding much of it is to split the\nmodules into individual files and then understanding the test cases.\n\n> and people rely a lot on an odd coincidence:\n> \n>  - using \"git svn clone\" twice with the same configuration on the same\n>    repository will, at least most of the time, give the same commit\n>    names.\n\nI did design that in mind when I first started, I'm glad it still mostly\nworks after all this time :)\n\n> That is the dream.\n\n*My* dream was to replace Subversion and get people using git.  git-svn\nwas designed to be self-obsoleting from the start.  For the most part\nI consider it a success since I no longer need to use it :)\n\n-- \nEric Wong\n"}]}