{"thread":{"id":"21226","subject":"Efficient cloning from svn (with multiple branches/tags subdirs)","startedAt":"2009-10-13T18:13:17Z","lastAt":"2009-10-16T11:20:16Z","messageCount":9,"participants":["Bruno Harbulot","Eric Wong","Avery Pennarun","B Smith-Mannschott"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"124859","messageId":"hb2fvu$8qi$1@ger.gmane.org","threadId":"21226","inReplyTo":null,"subject":"Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Bruno Harbulot","fromEmail":"bruno.harbulot@manchester.ac.uk","sentAt":"2009-10-13T18:13:17Z","receivedAt":"2009-10-13T18:13:17Z","isPatch":false,"sender":{"key":"bruno.harbulot@manchester.ac.uk","avatar":null},"body":"Hello,\n\nI'm trying to clone an existing subversion repository (Restlet: \nhttp://restlet.tigris.org/source/browse/). I'm using Git 1.6.5. The \nlayout of the project is like this:\n trunk/\n branches/1.0\n branches/1.1\n tags/1.0/1.0b1\n tags/1.0/1.0b2\n ...\n tags/1.0/1.0.1\n ...\n tags/1.1/1.1.0\n tags/1.1/1.1.1\n ...\n\nTherefore, I've tried to use this (with and without '-T trunk', but \nthat's a separate problem):\n\n git init\n git svn init --prefix=svn/ -t tags/1.0 -t tags/1.1 -t tags/1.2 -t \ntags/2.0 -b branches/1.0 -b branches/1.1 \nhttp://restlet.tigris.org/svn/restlet\n git svn fetch\n\n\nThis takes a while (I've had to interrupt this) and this creates a \nnumber of branches such as:\n remotes/svn/tags/1.0b1\n remotes/svn/tags/1.0b2\n remotes/svn/tags/1.0b3\n remotes/svn/tags/1.0b3@1883\n remotes/svn/tags/1.0b3@323\n\n\nWhat surprises me is that it looks like it's looping over and over, \nsince sometimes it starts back from SVN revision 1 when it's trying to \nimport a new tag.\n\nTt starts like this:\n> \n> Checked through r101\n> Checked through r201\n> Checked through r301\n> A www/index.html\n> r1 = 2ec77afc2e491e2b7c825cb685101e3bcbe7a8f7 (refs/remotes/svn/tags/1.0b1@312)\n> A source/impl/License.txt\n> A source/impl/Copyright.txt\n> A source/impl/org/restlet/UniformInterface.java\n> A source/impl/org/restlet/RestletException.java\n> ...\n\nThen, when it reaches r312, it starts again at r1:\n\n> r312 = 5b40558b5bb2b4b04f9520f89b699ff6b0f50cdb (refs/remotes/svn/tags/1.0b1@312)\n> r313 = 7ebcbd9da535cfdc23aacb612271e625445a7516 (refs/remotes/svn/tags/1.0b1@1881)\n> r1882 = aed1582d4868a1be8ae8fcc0f15546822099f339 (refs/remotes/svn/tags/1.0b1)\n> Checked through r101\n> Checked through r201\n> Checked through r301\n> A www/index.html\n> r1 = 2ec77afc2e491e2b7c825cb685101e3bcbe7a8f7 (refs/remotes/svn/tags/1.0b2@321)\n> A source/impl/License.txt\n> A source/impl/Copyright.txt\n> A source/impl/org/restlet/UniformInterface.java\n> A source/impl/org/restlet/RestletException.java\n> A source/impl/org/restlet/AbstractRestlet.java\n> A source/impl/org/restlet/connector/Resolver.java\n\n(And so on for each tag).\n\nThis seems particularly inefficient and unfriendly for the resource \nprovider (I stopped as soon as I noticed). Is there a better way to do this?\n\n\nBest wishes,\n\nBruno.\n"},{"id":"124939","messageId":"20091014060307.GA17178@dcvr.yhbt.net","threadId":"21226","inReplyTo":"hb2fvu$8qi$1@ger.gmane.org","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-10-14T06:03:07Z","receivedAt":"2009-10-14T06:03:07Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Bruno Harbulot wrote:\n> Hello,\n>\n> I'm trying to clone an existing subversion repository (Restlet: \n> http://restlet.tigris.org/source/browse/). I'm using Git 1.6.5. The \n> layout of the project is like this:\n> trunk/\n> branches/1.0\n> branches/1.1\n> tags/1.0/1.0b1\n> tags/1.0/1.0b2\n> ...\n> tags/1.0/1.0.1\n> ...\n> tags/1.1/1.1.0\n> tags/1.1/1.1.1\n> ...\n\n\nHi Bruno,\n\nThat looks like there's two levels of tags. You should be able to do\nthis with your version of git in $GIT_CONFIG:\n\n\t[svn-remote \"svn\"]\n\t\turl = http://restlet.tigris.org/svn/restlet\n\t\tfetch = trunk:refs/remotes/svn/trunk\n\t\tbranches = branches/*:refs/remotes/svn/*\n\t\ttags = tags/*/*:refs/remotes/svn/tags/*/*\n\t\t; note the */* to glob at multiple levels\n\n> Therefore, I've tried to use this (with and without '-T trunk', but \n> that's a separate problem):\n>\n> git init\n> git svn init --prefix=svn/ -t tags/1.0 -t tags/1.1 -t tags/1.2 -t \n> tags/2.0 -b branches/1.0 -b branches/1.1 \n> http://restlet.tigris.org/svn/restlet\n> git svn fetch\n>\n>\n> This takes a while (I've had to interrupt this) and this creates a \n> number of branches such as:\n> remotes/svn/tags/1.0b1\n> remotes/svn/tags/1.0b2\n> remotes/svn/tags/1.0b3\n> remotes/svn/tags/1.0b3@1883\n> remotes/svn/tags/1.0b3@323\n>\n>\n> What surprises me is that it looks like it's looping over and over, \n> since sometimes it starts back from SVN revision 1 when it's trying to \n> import a new tag.\n\nYeah, that's an unfortunate thing about the flexibility of Subversion,\nbasically anything can be a \"tag\" or a directory and it's extremely\nhard for git svn to support any uncommon cases for tags/branches\nout-of-the box, so the manual config editing is needed.\n\n-- \nEric Wong\n"},{"id":"124953","messageId":"4AD594CF.5000304@manchester.ac.uk","threadId":"21226","inReplyTo":"20091014060307.GA17178@dcvr.yhbt.net","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Bruno Harbulot","fromEmail":"bruno.harbulot@manchester.ac.uk","sentAt":"2009-10-14T09:07:27Z","receivedAt":"2009-10-14T09:07:27Z","isPatch":false,"sender":{"key":"bruno.harbulot@manchester.ac.uk","avatar":null},"body":"Hi Eric,\n\nEric Wong wrote:\n> Hi Bruno,\n> \n> That looks like there's two levels of tags. You should be able to do\n> this with your version of git in $GIT_CONFIG:\n> \n> \t[svn-remote \"svn\"]\n> \t\turl = http://restlet.tigris.org/svn/restlet\n> \t\tfetch = trunk:refs/remotes/svn/trunk\n> \t\tbranches = branches/*:refs/remotes/svn/*\n> \t\ttags = tags/*/*:refs/remotes/svn/tags/*/*\n> \t\t; note the */* to glob at multiple levels\n\nThank you, here is what I had (with the multiple -t/-b):\n\n[svn-remote \"svn\"]\n url = http://restlet.tigris.org/svn/restlet\n branches = branches/1.0/*:refs/remotes/svn/*\n branches = branches/1.1/*:refs/remotes/svn/*\n tags = tags/1.0/*:refs/remotes/svn/tags/*\n tags = tags/1.1/*:refs/remotes/svn/tags/*\n tags = tags/1.2/*:refs/remotes/svn/tags/*\n tags = tags/2.0/*:refs/remotes/svn/tags/*\n\nI think the notation you suggest \"*/*\" is indeed better, since I don't \nhave to specify each tag sub-directory. However, they change so rarely \nthat it was only a minor issue.\n\n\n>> What surprises me is that it looks like it's looping over and over, \n>> since sometimes it starts back from SVN revision 1 when it's trying to \n>> import a new tag.\n> \n> Yeah, that's an unfortunate thing about the flexibility of Subversion,\n> basically anything can be a \"tag\" or a directory and it's extremely\n> hard for git svn to support any uncommon cases for tags/branches\n> out-of-the box, so the manual config editing is needed.\n\nI must admit I don't fully understand how git-svn does the import, but \neven with this manual configuration, it still tries to pull (almost) \nevery revision from revision 1 for each tag, a bit as if there was:\n for each tag:\n for revision in 1 to tag.latest revision:\n pull the revision\n\n(This isn't even for each tag, but for each modification of each tag, \nsince tags aren't really tags in SVN).\n\nWhat I'd like to be able to do (mainly for efficiency and more \nimportantly not to hammer tigris.org) is to pull each revision at most \nonce (even if it's for the directory at the top of trunk, branches and \ntags).\n\nBest wishes,\n\nBruno.\n"},{"id":"124989","messageId":"32541b130910140928jdac0187x754423e8d5c64e53@mail.gmail.com","threadId":"21226","inReplyTo":"20091014060307.GA17178@dcvr.yhbt.net","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-10-14T16:28:38Z","receivedAt":"2009-10-14T16:28:38Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Wed, Oct 14, 2009 at 2:03 AM, Eric Wong wrote:\n> Bruno Harbulot wrote:\n>> What surprises me is that it looks like it's looping over and over,\n>> since sometimes it starts back from SVN revision 1 when it's trying to\n>> import a new tag.\n>\n> Yeah, that's an unfortunate thing about the flexibility of Subversion,\n> basically anything can be a \"tag\" or a directory and it's extremely\n> hard for git svn to support any uncommon cases for tags/branches\n> out-of-the box, so the manual config editing is needed.\n\nI've been thinking about this myself for some time. One option that\nmight be \"interesting\" would be to just grab the *entire* svn tree\n(from the root), and then use git-subtree[1] to slice and dice it into\nbranches using your local copy of git (which is fast and uses no\nbandwidth) instead of during the svn fetch (which is slow and uses\nlots of bandwidth). I think it would also simplify the git-svn code\nquite a lot, at least for fetching, since there would always be a\nglobal view of the tree and SVN things like \"copy branch A to tag B\"\nwould just be exactly that.\n\nOf course I have no time to code this up myself, so I apologize for\njust dumping ideas on you without code behind them. If this inspires\nanyone, I'd be happy to help with any missing features (or\ndocumentation) this exposes in git-subtree, though.\n\nHave fun,\n\nAvery\n\n[1] http://github.com/apenwarr/git-subtree\n"},{"id":"124993","messageId":"20091014180013.GA24741@dcvr.yhbt.net","threadId":"21226","inReplyTo":"32541b130910140928jdac0187x754423e8d5c64e53@mail.gmail.com","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-10-14T18:00:17Z","receivedAt":"2009-10-14T18:00:17Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Avery Pennarun wrote:\n> On Wed, Oct 14, 2009 at 2:03 AM, Eric Wong wrote:\n> > Bruno Harbulot wrote:\n> >> What surprises me is that it looks like it's looping over and over,\n> >> since sometimes it starts back from SVN revision 1 when it's trying to\n> >> import a new tag.\n> >\n> > Yeah, that's an unfortunate thing about the flexibility of Subversion,\n> > basically anything can be a \"tag\" or a directory and it's extremely\n> > hard for git svn to support any uncommon cases for tags/branches\n> > out-of-the box, so the manual config editing is needed.\n> \n> I've been thinking about this myself for some time. One option that\n> might be \"interesting\" would be to just grab the *entire* svn tree\n> (from the root), and then use git-subtree[1] to slice and dice it into\n> branches using your local copy of git (which is fast and uses no\n> bandwidth) instead of during the svn fetch (which is slow and uses\n> lots of bandwidth). I think it would also simplify the git-svn code\n> quite a lot, at least for fetching, since there would always be a\n> global view of the tree and SVN things like \"copy branch A to tag B\"\n> would just be exactly that.\n> \n> Of course I have no time to code this up myself, so I apologize for\n> just dumping ideas on you without code behind them. If this inspires\n> anyone, I'd be happy to help with any missing features (or\n> documentation) this exposes in git-subtree, though.\n\nThis was actually the original use case of git svn back when I started.\n\n git svn clone SVNREPO_ROOT (without --stdlayout)\n\nIt's still an option if you have the disk space for the working copies,\nbut I had to create the branches/tags support since the working copies\nwould be become prohibitively large. If git-subtree could be\ntaught to work on a bare repo (git svn has a --no-checkout option)\nit might be an option, too.\n\n> Have fun,\n> \n> Avery\n> \n> [1] http://github.com/apenwarr/git-subtree\n\n-- \nEric Wong\n"},{"id":"124997","messageId":"32541b130910141126u4df7f439i3d2926c2e1db9497@mail.gmail.com","threadId":"21226","inReplyTo":"20091014180013.GA24741@dcvr.yhbt.net","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-10-14T18:26:16Z","receivedAt":"2009-10-14T18:26:16Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Wed, Oct 14, 2009 at 2:00 PM, Eric Wong wrote:\n> Avery Pennarun wrote:\n>> I've been thinking about this myself for some time.  One option that\n>> might be \"interesting\" would be to just grab the *entire* svn tree\n>> (from the root), and then use git-subtree[1] to slice and dice it into\n>> branches using your local copy of git (which is fast and uses no\n>> bandwidth) instead of during the svn fetch (which is slow and uses\n>> lots of bandwidth).  I think it would also simplify the git-svn code\n>> quite a lot, at least for fetching, since there would always be a\n>> global view of the tree and SVN things like \"copy branch A to tag B\"\n>> would just be exactly that.\n>\n> This was actually the original use case of git svn back when I started.\n>\n>  git svn clone SVNREPO_ROOT   (without --stdlayout)\n>\n> It's still an option if you have the disk space for the working copies,\n> but I had to create the branches/tags support since the working copies\n> would be become prohibitively large.  If git-subtree could be\n> taught to work on a bare repo (git svn has a --no-checkout option)\n> it might be an option, too.\n\nI've never tested git-subtree without a working tree, however, it\ndoesn't *use* the working tree for anything when splitting, so at\nworst, there might be a minor bug or two. Thus, there ought never be\na need to check out the whole huge tree (which I agree would be both\nslow and huge).\n\ndcommit might be a little weirder. Though I guess if we fixed the\ngit-svn-id tags in the split branches, you could just commit directly\ninto a branch, then fetch the new commit back from the root, then\nrebase the branch, as dcommit already does.\n\nYou know, maybe this is actually easier than I thought... I was\nthinking committing back to svn would be complicated since it requires\na working tree, but if we let you commit straight from one of the\nbranches, it shouldn't actually be too bad at all. Hmm.\n\nHave fun,\n\nAvery\n"},{"id":"125087","messageId":"4AD75A93.9050106@manchester.ac.uk","threadId":"21226","inReplyTo":"32541b130910141126u4df7f439i3d2926c2e1db9497@mail.gmail.com","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Bruno Harbulot","fromEmail":"bruno.harbulot@manchester.ac.uk","sentAt":"2009-10-15T17:23:31Z","receivedAt":"2009-10-15T17:23:31Z","isPatch":false,"sender":{"key":"bruno.harbulot@manchester.ac.uk","avatar":null},"body":"Hello,\n\nAvery Pennarun wrote:\n> On Wed, Oct 14, 2009 at 2:00 PM, Eric Wong wrote:\n>> Avery Pennarun wrote:\n>>> I've been thinking about this myself for some time. One option that\n>>> might be \"interesting\" would be to just grab the *entire* svn tree\n>>> (from the root), and then use git-subtree[1] to slice and dice it into\n>>> branches using your local copy of git (which is fast and uses no\n>>> bandwidth) instead of during the svn fetch (which is slow and uses\n>>> lots of bandwidth). I think it would also simplify the git-svn code\n>>> quite a lot, at least for fetching, since there would always be a\n>>> global view of the tree and SVN things like \"copy branch A to tag B\"\n>>> would just be exactly that.\n>> This was actually the original use case of git svn back when I started.\n>>\n>> git svn clone SVNREPO_ROOT (without --stdlayout)\n>>\n>> It's still an option if you have the disk space for the working copies,\n>> but I had to create the branches/tags support since the working copies\n>> would be become prohibitively large. If git-subtree could be\n>> taught to work on a bare repo (git svn has a --no-checkout option)\n>> it might be an option, too.\n\nThank you for your suggestions. Unfortunately, I'm not really familiar \nwith git-subtree and how it could work with git-svn, sorry.\n\nI've tried another workaround: using svnsync to pull the repository only \nonce, and only then using git-svn fetch, locally, so as to avoid too \nmuch network traffic (I don't mind too much if it loops locally). I was \nhoping to be able to change the URL of the repository to the original \none afterwards, but it doesn't seem to work so easily, because of the \ncommit IDs. I'm assuming not having the same will cause problems for \nfurther fetches (this time directly from the original SVN repository) \nand for potential dcommits.\n\nWhen I do this:\n git init\n git svn init -s --prefix=svn/ file:///path/to/local/restlet-svnroot\n git svn fetch -r 1:2\n\nI get this ID, for example:\n r2 = c69a0b98d288a6e4e8779b50962b7fc65c4622e8\n\nIf I do this using the original http://restlet.tigris.org/svn/restlet, I \nget this:\n r2 = ce3b82915e92fe1ccf6ddedacd9d74b30bd4de86\n\n\nI've even tried to install a Apache-based subversion server locally and \nmake it believe it was restlet.tigris.org (by editing /etc/hosts and \ncreating the appropriate VirtualHost), but this generates another SHA1 \nID. (That's of course not a solution that would be generalisable.)\n\nI've had a quick look at the git-svn code to see how this ID was \ngenerated, but couldn't find anything obvious.\nI realise this isn't the cleanest approach possible, but any suggestion \nwould be appreciated.\n\n\nBest wishes,\n\nBruno.\n"},{"id":"125088","messageId":"28c656e20910151029s2e053f75q56e968f313d12b21@mail.gmail.com","threadId":"21226","inReplyTo":"4AD75A93.9050106@manchester.ac.uk","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"B Smith-Mannschott","fromEmail":"bsmith.occs@gmail.com","sentAt":"2009-10-15T17:29:58Z","receivedAt":"2009-10-15T17:29:58Z","isPatch":false,"sender":{"key":"bsmith.occs@gmail.com","avatar":"https://gravatar.com/avatar/46449e07c27bb8a30e4d50c08859f7210d47f6e84526b0a3b903641231dfc1bd?d=mp&s=160"},"body":"On Thu, Oct 15, 2009 at 19:23, Bruno Harbulot\n wrote:\n> Hello,\n>\n> Avery Pennarun wrote:\n>>\n>> On Wed, Oct 14, 2009 at 2:00 PM, Eric Wong wrote:\n>>>\n>>> Avery Pennarun wrote:\n>>>>\n>>>> I've been thinking about this myself for some time.  One option that\n>>>> might be \"interesting\" would be to just grab the *entire* svn tree\n>>>> (from the root), and then use git-subtree[1] to slice and dice it into\n>>>> branches using your local copy of git (which is fast and uses no\n>>>> bandwidth) instead of during the svn fetch (which is slow and uses\n>>>> lots of bandwidth).  I think it would also simplify the git-svn code\n>>>> quite a lot, at least for fetching, since there would always be a\n>>>> global view of the tree and SVN things like \"copy branch A to tag B\"\n>>>> would just be exactly that.\n>>>\n>>> This was actually the original use case of git svn back when I started.\n>>>\n>>>  git svn clone SVNREPO_ROOT   (without --stdlayout)\n>>>\n>>> It's still an option if you have the disk space for the working copies,\n>>> but I had to create the branches/tags support since the working copies\n>>> would be become prohibitively large.  If git-subtree could be\n>>> taught to work on a bare repo (git svn has a --no-checkout option)\n>>> it might be an option, too.\n>\n> Thank you for your suggestions. Unfortunately, I'm not really familiar with\n> git-subtree and how it could work with git-svn, sorry.\n>\n> I've tried another workaround: using svnsync to pull the repository only\n> once, and only then using git-svn fetch, locally, so as to avoid too much\n> network traffic (I don't mind too much if it loops locally). I was hoping to\n> be able to change the URL of the repository to the original one afterwards,\n> but it doesn't seem to work so easily, because of the commit IDs. I'm\n> assuming not having the same will cause problems for further fetches (this\n> time directly from the original SVN repository) and for potential dcommits.\n>\n> When I do this:\n>  git init\n>  git svn init -s --prefix=svn/ file:///path/to/local/restlet-svnroot\n>  git svn fetch -r 1:2\n>\n> I get this ID, for example:\n>  r2 = c69a0b98d288a6e4e8779b50962b7fc65c4622e8\n>\n> If I do this using the original http://restlet.tigris.org/svn/restlet, I get\n> this:\n>  r2 = ce3b82915e92fe1ccf6ddedacd9d74b30bd4de86\n>\n>\n> I've even tried to install a Apache-based subversion server locally and make\n> it believe it was restlet.tigris.org (by editing /etc/hosts and creating the\n> appropriate VirtualHost), but this generates another SHA1 ID. (That's of\n> course not a solution that would be generalisable.)\n>\n> I've had a quick look at the git-svn code to see how this ID was generated,\n> but couldn't find anything obvious.\n> I realise this isn't the cleanest approach possible, but any suggestion\n> would be appreciated.\n\nWhen I 'git svn clone' from a svnsync mirror I pass\n--use-svnsync-props. Have you tried that?\n\n// Ben\n"},{"id":"125158","messageId":"4AD856F0.6030905@manchester.ac.uk","threadId":"21226","inReplyTo":"28c656e20910151029s2e053f75q56e968f313d12b21@mail.gmail.com","subject":"Re: Efficient cloning from svn (with multiple branches/tags subdirs)","fromName":"Bruno Harbulot","fromEmail":"bruno.harbulot@manchester.ac.uk","sentAt":"2009-10-16T11:20:16Z","receivedAt":"2009-10-16T11:20:16Z","isPatch":false,"sender":{"key":"bruno.harbulot@manchester.ac.uk","avatar":null},"body":"\n\nB Smith-Mannschott wrote:\n\n>> I've had a quick look at the git-svn code to see how this ID was generated,\n>> but couldn't find anything obvious.\n>> I realise this isn't the cleanest approach possible, but any suggestion\n>> would be appreciated.\n> \n> When I 'git svn clone' from a svnsync mirror I pass\n> --use-svnsync-props. Have you tried that?\n\nThank you, I hadn't noticed this option, but it was the right one indeed.\n\nBest wishes,\n\nBruno.\n"}]}