{"thread":{"id":"26636","subject":"git-svn with big subversion repository","startedAt":"2011-03-02T02:43:23Z","lastAt":"2011-03-11T00:32:55Z","messageCount":6,"participants":["John Kristian","Thomas Ferris Nicolaisen","Phil Hord","Florian Weimer","Jason Miller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"162617","messageId":"C992EE5B.CBFB%jkristian@linkedin.com","threadId":"26636","inReplyTo":null,"subject":"git-svn with big subversion repository","fromName":"John Kristian","fromEmail":"jkristian@linkedin.com","sentAt":"2011-03-02T02:43:23Z","receivedAt":"2011-03-02T02:43:23Z","isPatch":false,"sender":{"key":"jkristian@linkedin.com","avatar":null},"body":"How do you recommend using git to work with branches of a large, busy\nsubversion repository? In general, how can small teams use git for their\ntasks, and use subversion to coordinate with a larger organization?\n\ngit-svn has some trouble, I find. For example, this tries to copy the entire\nrepo starting with revision 1:\n\ngit svn clone --stdlayout svn+ssh://server/repo/project\n\nThis would take weeks, I estimate for my subversion repository.\n\nChoosing a subset of the repository enables git svn clone to cope, but then\ngit svn fetch will stall after processing a few revisions.  For example:\n\ngit svn clone --no-follow-parent --no-minimize-url \\\n --branches=branches \\\n --ignore-paths=\"^(?!branches/(TEAM_|RELEASE_))\" \\\n -r $BASE svn+ssh://server/repo/project\ngit svn fetch --no-follow-parent # stalls\n\nI don't why it stalls. I guess it's doing something that requires processing\nthe entire subversion repository.\n\nThe best I can do is clone each subversion branch into a separate svn-remote\nsection of the .git/config file, for example:\n\ngit svn clone --no-follow-parent --no-minimize-url \\\n --svn-remote=TEAM_FOO --id=TEAM_FOO \\\n -r $BASE svn+ssh://server/repo/project/branches/TEAM_FOO\ngit svn fetch --no-follow-parent\n\nThe clone runs about as long as svn checkout, and the fetch replays the\nlater revisions briskly. Sadly, the relationship between branches isn't\nfetched: git log won't tell me how a given subversion branch was copied from\nanother. I use svn for that.\n\nI'm using git version 1.7.4, git-svn version 1.7.4 (svn 1.6.5), svn version\n1.6.0 (r36650) and Mac OS X version 10.6.5. I got git from MacPorts.\n\n- John Kristian\n"},{"id":"162648","messageId":"AANLkTikWDGAYt4SL0r9JR1Wt-vabY4O6vfVyJeDeZjqU@mail.gmail.com","threadId":"26636","inReplyTo":"C992EE5B.CBFB%jkristian@linkedin.com","subject":"Re: git-svn with big subversion repository","fromName":"Thomas Ferris Nicolaisen","fromEmail":"tfnico@gmail.com","sentAt":"2011-03-02T16:09:56Z","receivedAt":"2011-03-02T16:09:56Z","isPatch":false,"sender":{"key":"tfnico@gmail.com","avatar":"https://gravatar.com/avatar/628cf28a25ca4c596c7284562100f70f0ef908bcbfadd4da1eb3d48c23658d01?d=mp&s=160"},"body":"Hi John,\n\nI've successfully run git svn clone on a repository with about 100k\nrevisions. The clone was not of the whole repository, but rather a\nsubdirectory for a project using the trunk/tags/branches structure.\nThe project is about 200k files and about 4GB.\n\nThe initial clone took hours and hours (on my macbook). I basically\nhad to leave it on over night (the svn server is here on the LAN,\nrunning over https).\n\nThe only problem I had was that the clone would occasionally exit (not\nstall, as you say). This is a know problem described here:\nhttp://bugs.debian.org/cgi-bin/bugreport.cgi?bug=526989\n\nThe solution is to just run git svn fetch, as the cloning will pick up\nwhere it stopped. To keep from having to do this yourself, loop the\nfetch in a shell script. I blogged about it here:\n<http://blog.tfnico.com/2010/07/living-with-subversion-and-git-in.html>\n\nAnd there are also some more tricks and tips for living with git+svn\nhere: <http://www.tfnico.com/presentations/git-and-subversion>\n\nYou could also investigate how the Apache folks have made their git\nmirrors here: http://git.apache.org/ - at least they have an SVN repo\nwith over a million revisions. I think they did something like\nsvn-dump + git fast-import, but I couldn't find any details on the\nfly.\n\n\nOn Wed, Mar 2, 2011 at 3:43 AM, John Kristian <jkristian@linkedin.com> wrote:\n> How do you recommend using git to work with branches of a large, busy\n> subversion repository? In general, how can small teams use git for their\n> tasks, and use subversion to coordinate with a larger organization?\n>\n> git-svn has some trouble, I find. For example, this tries to copy the entire\n> repo starting with revision 1:\n>\n> git svn clone --stdlayout svn+ssh://server/repo/project\n>\n> This would take weeks, I estimate for my subversion repository.\n>\n> Choosing a subset of the repository enables git svn clone to cope, but then\n> git svn fetch will stall after processing a few revisions.  For example:\n>\n> git svn clone --no-follow-parent --no-minimize-url \\\n>  --branches=branches \\\n>  --ignore-paths=\"^(?!branches/(TEAM_|RELEASE_))\" \\\n>  -r $BASE svn+ssh://server/repo/project\n> git svn fetch --no-follow-parent # stalls\n>\n> I don't why it stalls. I guess it's doing something that requires processing\n> the entire subversion repository.\n>\n> The best I can do is clone each subversion branch into a separate svn-remote\n> section of the .git/config file, for example:\n>\n> git svn clone --no-follow-parent --no-minimize-url \\\n>  --svn-remote=TEAM_FOO --id=TEAM_FOO \\\n>  -r $BASE svn+ssh://server/repo/project/branches/TEAM_FOO\n> git svn fetch --no-follow-parent\n>\n> The clone runs about as long as svn checkout, and the fetch replays the\n> later revisions briskly. Sadly, the relationship between branches isn't\n> fetched: git log won't tell me how a given subversion branch was copied from\n> another. I use svn for that.\n>\n> I'm using git version 1.7.4, git-svn version 1.7.4 (svn 1.6.5), svn version\n> 1.6.0 (r36650) and Mac OS X version 10.6.5. I got git from MacPorts.\n>\n> - John Kristian\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"162702","messageId":"4D6F1578.3000203@cisco.com","threadId":"26636","inReplyTo":"C992EE5B.CBFB%jkristian@linkedin.com","subject":"Re: git-svn with big subversion repository","fromName":"Phil Hord","fromEmail":"hordp@cisco.com","sentAt":"2011-03-03T04:13:44Z","receivedAt":"2011-03-03T04:13:44Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On 03/01/2011 09:43 PM, John Kristian wrote:\n> How do you recommend using git to work with branches of a large, busy\n> subversion repository? In general, how can small teams use git for their\n> tasks, and use subversion to coordinate with a larger organization?\n>\n> git-svn has some trouble, I find. For example, this tries to copy the entire\n> repo starting with revision 1:\n>\n> git svn clone --stdlayout svn+ssh://server/repo/project\n>\n> This would take weeks, I estimate for my subversion repository.\n>\n> Choosing a subset of the repository enables git svn clone to cope, but then\n> git svn fetch will stall after processing a few revisions.  For example:\n>\n> git svn clone --no-follow-parent --no-minimize-url \\\n>  --branches=branches \\\n>  --ignore-paths=\"^(?!branches/(TEAM_|RELEASE_))\" \\\n>  -r $BASE svn+ssh://server/repo/project\n> git svn fetch --no-follow-parent # stalls\n>\n> I don't why it stalls. I guess it's doing something that requires processing\n> the entire subversion repository.\n\nMy initial git-svn clone took several days and many restarts.  It was\nmuch faster on my laptop.  I found out later I had a flaky router and it\nwas dropping about 20% of my packets.  Replaced the router and the clone\ndropped to a reasonable couple-of-hours.   Is it just me?\n\nYou can optimize by cloning specific paths inside the svn repo and then\nmerging in git later.\n\nPhil\n"},{"id":"162813","messageId":"87lj0t7s9z.fsf@mid.deneb.enyo.de","threadId":"26636","inReplyTo":"C992EE5B.CBFB%jkristian@linkedin.com","subject":"Re: git-svn with big subversion repository","fromName":"Florian Weimer","fromEmail":"fw@deneb.enyo.de","sentAt":"2011-03-05T10:53:12Z","receivedAt":"2011-03-05T10:53:12Z","isPatch":false,"sender":{"key":"fw@deneb.enyo.de","avatar":null},"body":"* John Kristian:\n\n> How do you recommend using git to work with branches of a large, busy\n> subversion repository? In general, how can small teams use git for their\n> tasks, and use subversion to coordinate with a larger organization?\n\nI've used svnsync to a local repository and git-svn against that.\nThis meant that my experiments do not cause excessive load on the\nserver.\n\nYou should definitely coordinate this because this could be considered\nleaching IP if the repository is not public.\n"},{"id":"163044","messageId":"20110308215342.35b8aac1@naru.jasonmmiller.org","threadId":"26636","inReplyTo":"C992EE5B.CBFB%jkristian@linkedin.com","subject":"Re: git-svn with big subversion repository","fromName":"Jason Miller","fromEmail":"jason@milr.com","sentAt":"2011-03-09T05:53:42Z","receivedAt":"2011-03-09T05:53:42Z","isPatch":false,"sender":{"key":"jason@milr.com","avatar":null},"body":"On Wed, 2 Mar 2011 02:43:23 +0000\nJohn Kristian <jkristian@linkedin.com> wrote:\n\n> How do you recommend using git to work with branches of a large, busy\n> subversion repository? In general, how can small teams use git for\n> their tasks, and use subversion to coordinate with a larger\n> organization?\n\nI don't know if this is the same problem that you are having, but...\n\nI had to clone a repository with 200k revisions and 12000\nbranches+tags.  It was going to take weeks and weeks with a local\nsvnsync mirror on a high-end workstation with fast disks.\n\nI've never touched perl-code previous to this, but a friend pointed me\nat a good perl profiler, and I found pretty quickly the offending line\nof code in git-svn:\n\n3515         return unless ::verify_ref($self->refname.'^0');\n\nThis was basically doing a\n  system(\"git rev-parse --verify some-reference^0\")\n\nseveral times per revision fetched per branch.  When you have 12000\nbranches, that really, really adds up.  I made a change that seems to\nspeed it up by a factor of about 10-20x on my repository, but I'm still\ndigging around in git to see if I'm doing it correctly.\n\nMy basic logic is that if the above one-liner returns true then either\none of the following files will exist:\n    $ENV{GIT_DIR}/refs/remotes/$refname\n    $ENV{GIT_DIR}/refs/$refname\n    $ENV{GIT_DIR}/refs/heads/$refname\nor there will be an entry for the reference in\n    $ENV{GIT_DIR}/packed-refs\n\nFurthermore since packed-refs changes infrequently, you can cache its\ncontents.\n\nI'm still digging around in the plumbing to see if this is\nassumption is true or not.  If I find it is true, I'll likely submit a\npatch.  Now that it \"works on my machine\" I've backburnered it a bit\nsince git is more a tool I use than a project I hack on.\n\n-Jason\n"},{"id":"163191","messageId":"20110311003255.GA13814@home.jasonmmiller.org","threadId":"26636","inReplyTo":"C99D031D.D0D9%jkristian@linkedin.com","subject":"Re: git-svn with big subversion repository","fromName":"Jason Miller","fromEmail":"jason@milr.com","sentAt":"2011-03-11T00:32:55Z","receivedAt":"2011-03-11T00:32:55Z","isPatch":false,"sender":{"key":"jason@milr.com","avatar":null},"body":"On 18:14 Wed 09 Mar     , John Kristian wrote:\nMr Kristian:\n> Thanks for sharing your experience.  After patching git-svn, were you able to clone your subversion repository?\n\nIndeed I was, it took about 48 hours to to the initial import.  However,\nI forgot to mention one other important thing that was a problem.\n\nThere is a pattern in svn of doing the following:\n\n/trunk/module1\n/trunk/module2\n/trunk/module3\n\nThen some branches will be like this:\n\tsvn cp /trunk/ /branches/mybranch1\nand others might be:\n\tsvn cp /trunk/module2 /brancyes/mybranchofmodule2\n\nIf this hasn't ever been done on your repository, you can stop reading\nnow.\n\nThere is no way to represent this in Git directly, so the correct thing\nto do here is to create a git repository for each module.  Now the hard\nthing is telling git-svn how to handle this.  I ended up writing a\npython script that reads in the SVN changelog and finds all of the\nchildren of e.g. /trunk/module1.  Any that were copied from /trunk, it\nappends \"/module1\" to the path, and any that were copied from\n/trunk/module1, it leaves alone.  This then goes in the git\nconfiguration file as the list of branches to fetch.\n\n-Jason\n"}]}