{"thread":{"id":"19937","subject":"git svn's performance on cloning mono's branches/tags...","startedAt":"2009-06-26T00:39:35Z","lastAt":"2009-10-01T07:17:26Z","messageCount":10,"participants":["Hin-Tak Leung","Andreas Ericsson","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"116985","messageId":"3ace41890906251739r45b3eae9oe1b7e32886defc0f@mail.gmail.com","threadId":"19937","inReplyTo":null,"subject":"git svn's performance on cloning mono's branches/tags...","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-06-26T00:39:35Z","receivedAt":"2009-06-26T00:39:35Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"(I'm not on list so please CC) - I am trying to git svn clone mono's\nrepository, and for some strange reason it is doing a complete check\nout from revision 1 all the way to each branch/tag . Is this normal or\nam I doing anything wrong? I have git svn clone a few other things\nrecently (R, ghostscript) and I don't remember git doing this, but R\nis quite a normal svn layout and I think I only have ghostcript trunk\n(no branch/tag).\n\nWhat I did was this: I started out with just git-svn clone trunk\n(which generated the first two [now-commented out] lines), then I\nthought I also want 2.4 branch so I added the next commented-out\nsection, and ran 'git svn fetch --all'. Then I thought I like the\nwhole thing - so I commented out most of it and added the 4 lines as\nblow - which is also my current git config.\n\nThe problem is that it seems to treat every tag as a branch, and also\nseem to want to check out every revision from r1 up to r3382 for\nMONO_0_10, etc.\nto mono/tags/MONO_0_10@3382 , and r1 to r2598 for mono/tags/MONO_0_9@2598 , etc.\n\n--------------------------------------------------------------------\n[svn-remote \"svn\"]\n#\turl = svn://anonsvn.mono-project.com/source/trunk/mono\n#\tfetch = :refs/remotes/git-svn\n url = svn://anonsvn.mono-project.com/source\n tags = tags/*/mono:refs/remotes/mono/tags/*\n branches = branches/*/mono:refs/remotes/mono/branches/*\n trunk = trunk/mono:refs/remotes/git-svn\n#[svn-remote \"mono24\"]\n#\turl = svn://anonsvn.mono-project.com/source/branches/mono-2.4/mono\n#\tfetch = :refs/remotes/git-svn-mono24\n--------------------------------------------------------------------\n\nI guess my ultimate question is, how does one modify git config to\ntrack tags and branches, etc as well, *after* having already svn\nclone'd trunk?, without burning too much bandwidth?\n"},{"id":"117000","messageId":"4A445959.6090403@op5.se","threadId":"19937","inReplyTo":"3ace41890906251739r45b3eae9oe1b7e32886defc0f@mail.gmail.com","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-06-26T05:15:05Z","receivedAt":"2009-06-26T05:15:05Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Hin-Tak Leung wrote:\n> (I'm not on list so please CC) - I am trying to git svn clone mono's\n> repository, and for some strange reason it is doing a complete check\n> out from revision 1 all the way to each branch/tag . Is this normal or\n> am I doing anything wrong? I have git svn clone a few other things\n> recently (R, ghostscript) and I don't remember git doing this, but R\n> is quite a normal svn layout and I think I only have ghostcript trunk\n> (no branch/tag).\n> \n> What I did was this: I started out with just git-svn clone trunk\n> (which generated the first two [now-commented out] lines), then I\n> thought I also want 2.4 branch so I added the next commented-out\n> section, and ran 'git svn fetch --all'. Then I thought I like the\n> whole thing - so I commented out most of it and added the 4 lines as\n> blow - which is also my current git config.\n> \n> The problem is that it seems to treat every tag as a branch,\n\nThis is normal. SVN makes no real distinction between tags and branches,\nso git-svn has no way of detecting which is which. SVN also allows\ncommitting to \"tags\", so it doesn't make sense for git to store what\nSVN calls tags as git tags.\n\nAs for your other questions, I don't know.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"117008","messageId":"3ace41890906260259o3be005fq6be9d0e2c3f9af66@mail.gmail.com","threadId":"19937","inReplyTo":"4A445959.6090403@op5.se","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-06-26T09:59:49Z","receivedAt":"2009-06-26T09:59:49Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On Fri, Jun 26, 2009 at 6:15 AM, Andreas Ericsson wrote:\n> Hin-Tak Leung wrote:\n>>\n>> (I'm not on list so please CC) - I am trying to git svn clone mono's\n>> repository, and for some strange reason it is doing a complete check\n>> out from revision 1 all the way to each branch/tag . Is this normal or\n>> am I doing anything wrong? I have git svn clone a few other things\n>> recently (R, ghostscript) and I don't remember git doing this, but R\n>> is quite a normal svn layout and I think I only have ghostcript trunk\n>> (no branch/tag).\n>>\n>> What I did was this: I started out with just git-svn clone trunk\n>> (which generated the first two [now-commented out] lines), then I\n>> thought I also want 2.4 branch so I added the next commented-out\n>> section, and ran 'git svn fetch --all'. Then I thought I like the\n>> whole thing - so I commented out most of it and added the 4 lines as\n>> blow - which is also my current git config.\n>>\n>> The problem is that it seems to treat every tag as a branch,\n>\n> This is normal. SVN makes no real distinction between tags and branches,\n> so git-svn has no way of detecting which is which. SVN also allows\n> committing to \"tags\", so it doesn't make sense for git to store what\n> SVN calls tags as git tags.\n>\n> As for your other questions, I don't know.\n>\n> --\n> Andreas Ericsson                   andreas.ericsson@op5.se\n> OP5 AB                             www.op5.se\n> Tel: +46 8-230225                  Fax: +46 8-230231\n>\n> Considering the successes of the wars on alcohol, poverty, drugs and\n> terror, I think we should give some serious thought to declaring war\n> on peace.\n>\n\nI guess my question is two-fold:\n1) what is the correct/recommended way of adding tags/branches *after*\nalready cloning trunk?\n\n2) the current way of cloning svn repository is quite inefficient -\nfor every tag/branch, it does:\nr1 -> rX branch 1\nr1 -> rN tag 1\nr1 -> rY branch 2\nr1 -> rM tag 2\nr1 -> rZ branch 3\netc\n\nbut I think it is possible to track branches to the point where it\nbranches off (when 'svn copy' happens), rather than all the way to r1?\nAnd in that sense tags are just very short branches.\n\nThe current behavior is quite bandwith consuming for projects with a\nlong history and lots of tags, like mono...\n\nHin-Tak\n"},{"id":"117012","messageId":"4A44A9A9.6030008@op5.se","threadId":"19937","inReplyTo":"3ace41890906260259o3be005fq6be9d0e2c3f9af66@mail.gmail.com","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-06-26T10:57:45Z","receivedAt":"2009-06-26T10:57:45Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Hin-Tak Leung wrote:\n> On Fri, Jun 26, 2009 at 6:15 AM, Andreas Ericsson wrote:\n>> Hin-Tak Leung wrote:\n>>> (I'm not on list so please CC) - I am trying to git svn clone mono's\n>>> repository, and for some strange reason it is doing a complete check\n>>> out from revision 1 all the way to each branch/tag . Is this normal or\n>>> am I doing anything wrong? I have git svn clone a few other things\n>>> recently (R, ghostscript) and I don't remember git doing this, but R\n>>> is quite a normal svn layout and I think I only have ghostcript trunk\n>>> (no branch/tag).\n>>>\n>>> What I did was this: I started out with just git-svn clone trunk\n>>> (which generated the first two [now-commented out] lines), then I\n>>> thought I also want 2.4 branch so I added the next commented-out\n>>> section, and ran 'git svn fetch --all'. Then I thought I like the\n>>> whole thing - so I commented out most of it and added the 4 lines as\n>>> blow - which is also my current git config.\n>>>\n>>> The problem is that it seems to treat every tag as a branch,\n>> This is normal. SVN makes no real distinction between tags and branches,\n>> so git-svn has no way of detecting which is which. SVN also allows\n>> committing to \"tags\", so it doesn't make sense for git to store what\n>> SVN calls tags as git tags.\n>>\n>> As for your other questions, I don't know.\n>>\n>> --\n>> Andreas Ericsson andreas.ericsson@op5.se\n>> OP5 AB www.op5.se\n>> Tel: +46 8-230225 Fax: +46 8-230231\n>>\n>> Considering the successes of the wars on alcohol, poverty, drugs and\n>> terror, I think we should give some serious thought to declaring war\n>> on peace.\n>>\n> \n> I guess my question is two-fold:\n> 1) what is the correct/recommended way of adding tags/branches *after*\n> already cloning trunk?\n> \n\nI don't know.\n\n> 2) the current way of cloning svn repository is quite inefficient -\n> for every tag/branch, it does:\n> r1 -> rX branch 1\n> r1 -> rN tag 1\n> r1 -> rY branch 2\n> r1 -> rM tag 2\n> r1 -> rZ branch 3\n> etc\n> \n> but I think it is possible to track branches to the point where it\n> branches off (when 'svn copy' happens), rather than all the way to r1?\n\nThis isn't really a question, but a statement with a question mark.\n\n> And in that sense tags are just very short branches.\n> \n\nOften, yes, but they *can* be committed to. SVN's inability to keep\nimmutable tags is one of the reasons I never looked at it to replace\nCVS at $dayjob. But I digress..\n\n> The current behavior is quite bandwith consuming for projects with a\n> long history and lots of tags, like mono...\n> \n\nHmm. I've cloned many subversion repositories to git, and I've never\nseen the issues you're seeing. Perhaps because I always cloned the\nentire repository the first time, or because I normally do such things\nover a high-capacity connection, or because I just fire it up and\nforget about it until it's done.\n\nWhat happens if you ignore the already-cloned svn tree and just do\na new import without trying to continue the old one? Not that I'll\nactually do anything about it, as I don't have any problems with it,\nbut I'm curious even so ;-)\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"117021","messageId":"3ace41890906260644t3eddb2d2sb4ddbcb6499801@mail.gmail.com","threadId":"19937","inReplyTo":"4A44A9A9.6030008@op5.se","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-06-26T13:44:52Z","receivedAt":"2009-06-26T13:44:52Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On Fri, Jun 26, 2009 at 11:57 AM, Andreas Ericsson wrote:\n>>> Hin-Tak Leung wrote:\n>>>>\n>>>> (I'm not on list so please CC)\n\n> Hmm. I've cloned many subversion repositories to git, and I've never\n> seen the issues you're seeing. Perhaps because I always cloned the\n> entire repository the first time, or because I normally do such things\n> over a high-capacity connection, or because I just fire it up and\n> forget about it until it's done.\n\nIt is not about the capacity/throughput of the connection, but the\nlatency (round-trip time) of many small transactions - and most of the\nunnecessary...\n\n> What happens if you ignore the already-cloned svn tree and just do\n> a new import without trying to continue the old one? Not that I'll\n> actually do anything about it, as I don't have any problems with it,\n> but I'm curious even so ;-)\n\nI did wonder about that - and so I just stopped my branch-tracking\nfetch and start all over with svn init -T -t and -b then svn fetch\n--all. The first few hundred revisions are very promising: it is\npopulating the git refs for tag and branch heads\nas it goes up.\n\nThere are tutorials online about modifying git/config like I did to\nadding branches, etc... maybe the example section of git-svn can be\nupdated with a couple of sentence on 'just throw it away and start\nfrom the beginning if you change your mind from trunk-only to\ncomplex-layout' .\n\nOTOH, I think I would have liked to preserve the compressed object\nstore, and I think it is possible to graft an object store on an empty\ninit? The advantage is just that the disc space usage does not widely\nfluctuate & no gc steps in the middle.\n\nThanks for the response anyhow - I just need to remember to blow\ntrunk-only away and start over next time I change my mind:-).\n"},{"id":"117069","messageId":"3ace41890906261817y523c9321xd621fb3130941d91@mail.gmail.com","threadId":"19937","inReplyTo":"3ace41890906260644t3eddb2d2sb4ddbcb6499801@mail.gmail.com","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-06-27T01:17:25Z","receivedAt":"2009-06-27T01:17:25Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"Starting afresh with -T/-t/-b is transversing the branches and tags\nonce, rather that over and over starting from r1.\n\nHere is the starting afresh config:\n\n--------------------------\n[svn-remote \"svn\"]\n\turl = svn://anonsvn.mono-project.com/source\n\tfetch = trunk/mono:refs/remotes/trunk\n\tbranches = branches/*/mono/:refs/remotes/*\n\ttags = tags/*/mono/:refs/remotes/tags/*\n--------------------------\n\nHere is my attempt at contiuation/changing my mind after getting at trunk only:\n--------------------------\n[svn-remote \"svn\"]\n#\turl = svn://anonsvn.mono-project.com/source/trunk/mono\n#\tfetch = :refs/remotes/git-svn\n url = svn://anonsvn.mono-project.com/source\n tags = tags/*/mono:refs/remotes/mono/tags/*\n branches = branches/*/mono:refs/remotes/mono/branches/*\n trunk = trunk/mono:refs/remotes/git-svn\n-------------------------\nthe tags, branches, url entries are effectively the same, I think - so\nthe main difference is using trunk instead of fetch.\nWhy does it make any difference? The example at the bottom of\ngit-svn's man page uses the trunk notation. Maybe it should be\nchanged?\n"},{"id":"117114","messageId":"3ace41890906272008t96bfb04q7218e95055897900@mail.gmail.com","threadId":"19937","inReplyTo":"3ace41890906261817y523c9321xd621fb3130941d91@mail.gmail.com","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-06-28T03:08:01Z","receivedAt":"2009-06-28T03:08:01Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On Sat, Jun 27, 2009 at 2:17 AM, Hin-Tak Leung wrote:\n\n> the tags, branches, url entries are effectively the same, I think - so\n> the main difference is using trunk instead of fetch.\n> Why does it make any difference? The example at the bottom of\n> git-svn's man page uses the trunk notation. Maybe it should be\n> changed?\n>\n\nTo answer my question - I think the man page should be updated.\n\nAlso, while searching 'git svn speed', I found that a git clone of\nmono is available:\nhttp://www.mono-project.com/GitSVN\nA shame this page is not referenced my other devel resources on mono's\nweb site at all... also it uses a real user's account to make the\nclone (rather than annonymous svn), so to update the git clone by\ngit-svn, one needs a real dev account - that is not a problem for the\npurpose of sending in patches as the occasional dev.\n"},{"id":"117347","messageId":"20090702081615.GB11119@dcvr.yhbt.net","threadId":"19937","inReplyTo":"3ace41890906272008t96bfb04q7218e95055897900@mail.gmail.com","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-07-02T08:16:15Z","receivedAt":"2009-07-02T08:16:15Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hin-Tak Leung wrote:\n> On Sat, Jun 27, 2009 at 2:17 AM, Hin-Tak Leung wrote:\n> \n> > the tags, branches, url entries are effectively the same, I think - so\n> > the main difference is using trunk instead of fetch.\n> > Why does it make any difference? The example at the bottom of\n> > git-svn's man page uses the trunk notation. Maybe it should be\n> > changed?\n> \n> To answer my question - I think the man page should be updated.\n\nSomebody actually fixed it a while back:\n\ncommit 0e5e69a355b7bdd1af6ca33ac7ee35299bda368e\nAuthor: Wesley J. Landaker \nDate: Wed Apr 1 16:05:01 2009 -0600\n\n Documentation: git-svn: fix trunk/fetch svn-remote key typo\n\n\nAs far as performance goes, SVN's flexibility of tagging and having\nmultiple subprojects interacts quite badly with git svn's --stdlayout\nbehavior.\n\nWith --stdoulayout, git svn supports tags/branches that are directly\ndescended from the top-level of trunk:\n\n\tsvn cp $root/trunk $root/tags/0.1.0\n\nHowever, it looks like mono does things like this:\n\n\tsvn cp $root/trunk/mono $root/tags/mono-0.1.0\n\nAnd when git svn sees that $root/tags/mono-0.1.0's parent is\n$root/trunk/mono and not $root/trunk (which it's actually following),\nit will try to fetch the complete history of $root/trunk/mono\n($root/trunk may not contain all the history $root/trunk/mono contained,\neither).\n\n-- \nEric Wong\n"},{"id":"124050","messageId":"3ace41890909301504w633323b9ybec1f42c1c169225@mail.gmail.com","threadId":"19937","inReplyTo":"20090702081615.GB11119@dcvr.yhbt.net","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Hin-Tak Leung","fromEmail":"hintak.leung@gmail.com","sentAt":"2009-09-30T22:04:30Z","receivedAt":"2009-09-30T22:04:30Z","isPatch":false,"sender":{"key":"hintak.leung@gmail.com","avatar":null},"body":"On Thu, Jul 2, 2009 at 9:16 AM, Eric Wong wrote:\n> Hin-Tak Leung wrote:\n>> On Sat, Jun 27, 2009 at 2:17 AM, Hin-Tak Leung wrote:\n>>\n>> > the tags, branches, url entries are effectively the same, I think - so\n>> > the main difference is using trunk instead of fetch.\n>> > Why does it make any difference? The example at the bottom of\n>> > git-svn's man page uses the trunk notation. Maybe it should be\n>> > changed?\n>>\n>> To answer my question - I think the man page should be updated.\n>\n> Somebody actually fixed it a while back:\n>\n> commit 0e5e69a355b7bdd1af6ca33ac7ee35299bda368e\n> Author: Wesley J. Landaker \n> Date: Wed Apr 1 16:05:01 2009 -0600\n>\n> Documentation: git-svn: fix trunk/fetch svn-remote key typo\n>\n>\n> As far as performance goes, SVN's flexibility of tagging and having\n> multiple subprojects interacts quite badly with git svn's --stdlayout\n> behavior.\n>\n> With --stdoulayout, git svn supports tags/branches that are directly\n> descended from the top-level of trunk:\n>\n> svn cp $root/trunk $root/tags/0.1.0\n>\n> However, it looks like mono does things like this:\n>\n> svn cp $root/trunk/mono $root/tags/mono-0.1.0\n>\n> And when git svn sees that $root/tags/mono-0.1.0's parent is\n> $root/trunk/mono and not $root/trunk (which it's actually following),\n> it will try to fetch the complete history of $root/trunk/mono\n> ($root/trunk may not contain all the history $root/trunk/mono contained,\n> either).\n>\n> --\n> Eric Wong\n>\n\n\nHmm, I am having another problem with git-svn going back and download\neverything over and over with this:\n\ngit svn clone -s https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper\n\nI am going to do two-step init -s then fetch --all now to see if it helps.\n\n(it is probably not entirely standard layout with the extra CVSROOT?)\n"},{"id":"124066","messageId":"20091001071726.GA11370@dcvr.yhbt.net","threadId":"19937","inReplyTo":"3ace41890909301504w633323b9ybec1f42c1c169225@mail.gmail.com","subject":"Re: git svn's performance on cloning mono's branches/tags...","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-10-01T07:17:26Z","receivedAt":"2009-10-01T07:17:26Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hin-Tak Leung wrote:\n> Hmm, I am having another problem with git-svn going back and download\n> everything over and over with this:\n> \n> git svn clone -s https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper\n> \n> I am going to do two-step init -s then fetch --all now to see if it helps.\n> \n> (it is probably not entirely standard layout with the extra CVSROOT?)\n\nHi,\n\nThe ndiswrapper layout looks to be *almost* standard, the CVSROOT can\nprobably safely be ignored, and you should be able to _mostly_ track it\nwith the following config:\n\n[svn-remote \"svn\"]\n url = https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper\n fetch = trunk/ndiswrapper:refs/remotes/trunk\n branches = branches/*/ndiswrapper:refs/remotes/*\n tags = tags/*/ndiswrapper:refs/remotes/tags/*\n\nNotice how the glob can appear in the middle of the branches and tags\nconfigs on the remote side: foo/*/bar\n\nUnfortunately, some of the tags seem to be inconsistently tagged and\nsome had tags/TAGNAME/README whereas others had\ntags/TAGNAME/ndiswrapper/README (I'm using \"README\" to designate where\nthe logical top-level working directory would be).\n\n\"svn log -v https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper/tags\"\nshould help find the ones that are tagged at the wrong depth:\n\nVERSION_1_24 for example:\n\n\tfetch = tags/VERSION_1_24:refs/remotes/tags/VERSION_1_24\n\n-- \nEric Wong\n"}]}