{"thread":{"id":"13247","subject":"Excruciatingly slow git-svn imports","startedAt":"2008-04-24T18:54:04Z","lastAt":"2008-05-06T11:23:30Z","messageCount":9,"participants":["Geert Bosch","Steven Grimm","Eric Wong","Avery Pennarun"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"75158","messageId":"CC0F600D-71F3-493A-AA9C-AF901C1A58C4@adacore.com","threadId":"13247","inReplyTo":null,"subject":"Excruciatingly slow git-svn imports","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2008-04-24T18:54:04Z","receivedAt":"2008-04-24T18:54:04Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"I'm trying to import a 9.7G, 130K revision svn repository\nbut it seems to only import about 6K revisions per day on fast hardware\nusing a recent git (1.5.5).\n\nThis means about 20 days, or more if things slow down as the repo gets \nbigger\nAre there any tips/tricks on how to most efficiently convert large \nrepos?\nI'm using ssh+svn protocol for accessing the repository, but slowness\nseems due to local inefficiency. An strace -fcp during a minute \ngives\nthe following results:\n\n% time seconds usecs/call calls errors syscall\n------ ----------- ----------- --------- --------- ----------------\n 52.46 21.392640 17607 1215 clone\n 47.47 19.358882 3983 4860 3645 execve\n 0.05 0.019571 16 1216 wait4\n 0.01 0.003944 0 14582 1215 open\n 0.01 0.002458 0 14580 12150 access\n 0.00 0.000797 0 8500 write\n 0.00 0.000694 0 26013 read\n 0.00 0.000574 0 3693 munmap\n 0.00 0.000513 0 20659 close\n 0.00 0.000452 0 21918 mmap\n 0.00 0.000353 0 1215 stat\n 0.00 0.000234 0 12158 1215 lseek\n 0.00 0.000155 0 17013 fstat\n 0.00 0.000077 0 6075 mprotect\n 0.00 0.000076 0 8511 rt_sigaction\n 0.00 0.000074 0 6078 6078 ioctl\n 0.00 0.000049 0 2432 unlink\n 0.00 0.000033 0 2430 dup2\n 0.00 0.000033 0 7293 fcntl\n 0.00 0.000022 0 3681 brk\n 0.00 0.000022 0 1215 getppid\n 0.00 0.000019 0 1215 uname\n 0.00 0.000019 0 1215 arch_prctl\n 0.00 0.000000 0 1215 lstat\n 0.00 0.000000 0 1216 pipe\n 0.00 0.000000 0 22 mremap\n 0.00 0.000000 0 2431 dup\n 0.00 0.000000 0 1215 getcwd\n 0.00 0.000000 0 2430 getdents64\n------ ----------- ----------- --------- --------- ----------------\n100.00 40.781691 196296 24303 total\n\nSo, 99.93% of the time seems to be in clone/execve\n(including actual work done by the forked programs)\n\nIn another trace, I found the following execve calls were made:\n 22 execve(\"/homes/bosch/x86_64-linux/bin/git\",\n 2 execve(\"/homes/bosch/x86_64-linux/bin/git-commit-tree\",\n 2842 execve(\"/homes/bosch/x86_64-linux/bin/git-hash-object\",\n 22 execve(\"/opt/gnu/bin/git\",\n 2 execve(\"/opt/gnu/bin/git-commit-tree\",\n 2842 execve(\"/opt/gnu/bin/git-hash-object\",\n 22 execve(\"/opt/local/bin/git\",\n 2 execve(\"/opt/local/bin/git-commit-tree\",\n 2842 execve(\"/opt/local/bin/git-hash-object\",\n 22 execve(\"/opt/local/sbin/git\",\n 2 execve(\"/opt/local/sbin/git-commit-tree\",\n 2842 execve(\"/opt/local/sbin/git-hash-object\",\n\nI don't have git installed in either of /opt/gnu/bin, /opt/local/bin \nor /opt/local/sbin.\nThese three directories just happen to be before the one containing \ngit in my path:\n\nbosch:~/git$ echo $PATH\n/opt/gnu/bin:/opt/local/bin:/opt/local/sbin:/homes/bosch/x86_64-linux/ \nbin ...\n\nBefore trying to brush up my Perl and propose patching fixes for this\n(I doubt the extra execve's take much time at all), I was wondering why\nwe don't open a single stream to git-fast-import and have it do\nthe heavy lifting. Are there fundamental issues with this?\n\n -Geert\n"},{"id":"75160","messageId":"2C39C19D-0B81-4879-92DC-C0436DB6E84E@midwinter.com","threadId":"13247","inReplyTo":"CC0F600D-71F3-493A-AA9C-AF901C1A58C4@adacore.com","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Steven Grimm","fromEmail":"koreth@midwinter.com","sentAt":"2008-04-24T19:57:24Z","receivedAt":"2008-04-24T19:57:24Z","isPatch":false,"sender":{"key":"koreth@midwinter.com","avatar":"https://gravatar.com/avatar/71b4d2e8b62f168bdc9e9205341159e3567003b4f9e2127c617c5fa0a1f5bad2?d=mp&s=160"},"body":"On Apr 24, 2008, at 11:54 AM, Geert Bosch wrote:\n\n> I'm trying to import a 9.7G, 130K revision svn repository\n> but it seems to only import about 6K revisions per day on fast \n> hardware\n> using a recent git (1.5.5).\n\nI've found that git-svn gets slower as it runs. Try interrupting the \nclone and running \"git svn fetch\" -- it should pick up where it left \noff and will be MUCH faster if my experience is any indication. When I \nclone the big svn repository at work I usually restart it every 1000 \nrevisions or so and it finishes in a fraction of the time it takes if \nI let it do everything in a single run.\n\n-Steve\n"},{"id":"75501","messageId":"20080429070355.GF24171@muzzle","threadId":"13247","inReplyTo":"CC0F600D-71F3-493A-AA9C-AF901C1A58C4@adacore.com","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-04-29T07:03:55Z","receivedAt":"2008-04-29T07:03:55Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Geert Bosch wrote:\n\n> Before trying to brush up my Perl and propose patching fixes for this\n> (I doubt the extra execve's take much time at all), I was wondering why\n> we don't open a single stream to git-fast-import and have it do\n> the heavy lifting. Are there fundamental issues with this?\n\nLast I checked, fast-import doesn't allow rereading freshly imported\nobjects before that particular fast-import instance is finished running.\n\nSince git-svn imports deltas from SVN instead of full files, so it often\nneeds to reread objects it imported in the same run to make use of\ndeltas.\n\nHowever, Adam Roben's been working on some improvements to git-cat-file\nwhich allow it to avoid many fork+exec calls. The tests and some code\nhave some outstanding issues, but the code appears to work, and I'm sure\nAdam would love to have you test it more for him :)\n\n http://thread.gmane.org/gmane.comp.version-control.git/80240\n\n-- \nEric Wong\n"},{"id":"75503","messageId":"20080429071122.GG24171@muzzle","threadId":"13247","inReplyTo":"2C39C19D-0B81-4879-92DC-C0436DB6E84E@midwinter.com","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-04-29T07:11:22Z","receivedAt":"2008-04-29T07:11:22Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Steven Grimm wrote:\n> On Apr 24, 2008, at 11:54 AM, Geert Bosch wrote:\n> \n> >I'm trying to import a 9.7G, 130K revision svn repository\n> >but it seems to only import about 6K revisions per day on fast \n> >hardware\n> >using a recent git (1.5.5).\n> \n> I've found that git-svn gets slower as it runs. Try interrupting the \n> clone and running \"git svn fetch\" -- it should pick up where it left \n> off and will be MUCH faster if my experience is any indication. When I \n> clone the big svn repository at work I usually restart it every 1000 \n> revisions or so and it finishes in a fraction of the time it takes if \n> I let it do everything in a single run.\n\nThat's really strange to hear... The git-svn process itself does not\nstore much state other than the current revision and the log information\nfor the next 100 or so revisions it needs to import.\n\nAre you packing the repository? Which SVN protocol are you using? Does\nmemory usage of git-svn stay stable throughout the run?\n\n-- \nEric Wong\n"},{"id":"76072","messageId":"2B9E6C04-69F1-42BD-AE60-AFCE401E093E@adacore.com","threadId":"13247","inReplyTo":"20080429071122.GG24171@muzzle","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2008-05-05T04:29:35Z","receivedAt":"2008-05-05T04:29:35Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn Apr 29, 2008, at 03:11, Eric Wong wrote:\n\n>> I've found that git-svn gets slower as it runs. Try interrupting the\n>> clone and running \"git svn fetch\" -- it should pick up where it left\n>> off and will be MUCH faster if my experience is any indication. \n>> When I\n>> clone the big svn repository at work I usually restart it every 1000\n>> revisions or so and it finishes in a fraction of the time it takes if\n>> I let it do everything in a single run.\n>\n> That's really strange to hear... The git-svn process itself does not\n> store much state other than the current revision and the log \n> information\n> for the next 100 or so revisions it needs to import.\n>\n> Are you packing the repository? Which SVN protocol are you using? \n> Does\n> memory usage of git-svn stay stable throughout the run?\n\nI found the same. After about 5 days (with maybe 10 break/restarts), I \nhad\na converted repository with all 135K commits and a total size of just\nunder 1 GB. The last 100K commits took (much?) less than a day, almost \nall the\ntime was spend in the earlier ones. These commits seemed all to have \nthousands\nof files, even though most were probably the same. I'm sure this \nrepositor,\nwhich covers 15 years of development of a multi-million line project, \nhas\na lot of tags and it seemed that it just had to chew through many copies\nof the complete set of files to find out that they're all the same.\n\nIt's great git-svn can be restarted so well and doesn't get confused by\nuncleanly terminated runs. My final repository is fast and small.\nI'm still struggling with how to properly synchronize branches, but that\nprobably is mostly a matter of user education.\n\nThanks all for these great tools.\n\n -Geert\n"},{"id":"76168","messageId":"20080506032846.GA15521@untitled","threadId":"13247","inReplyTo":"2B9E6C04-69F1-42BD-AE60-AFCE401E093E@adacore.com","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-05-06T03:28:54Z","receivedAt":"2008-05-06T03:28:54Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Geert Bosch wrote:\n> On Apr 29, 2008, at 03:11, Eric Wong wrote:\n> \n> >>I've found that git-svn gets slower as it runs. Try interrupting the\n> >>clone and running \"git svn fetch\" -- it should pick up where it left\n> >>off and will be MUCH faster if my experience is any indication.\n> >>When I clone the big svn repository at work I usually restart it\n> >>every 1000 revisions or so and it finishes in a fraction of the time\n> >>it takes if I let it do everything in a single run.\n> >\n> >That's really strange to hear... The git-svn process itself does not\n> >store much state other than the current revision and the log\n> >information for the next 100 or so revisions it needs to import.\n> >\n> >Are you packing the repository? Which SVN protocol are you using?\n> >Does memory usage of git-svn stay stable throughout the run?\n> \n> I found the same. After about 5 days (with maybe 10 break/restarts), I\n> had a converted repository with all 135K commits and a total size of\n> just under 1 GB. The last 100K commits took (much?) less than a day,\n> almost all the time was spend in the earlier ones. These commits\n> seemed all to have thousands of files, even though most were probably\n> the same. I'm sure this repositor, which covers 15 years of\n> development of a multi-million line project, has a lot of tags and it\n> seemed that it just had to chew through many copies of the complete\n> set of files to find out that they're all the same.\n\nInteresting. By \"These commits seemed all to have thousands of files\",\nyou mean the first 35K that took up most of the time? If so, yes,\nthat's definitely a problem...\n\ngit-svn requests a log from SVN containing a list of all paths modified\nin each revision. By default, git-svn only requests log entries for up\nto 100 revisions at a time to reduce memory usage. However, having\nthousands of files modified for each revision would still be\nproblematic, as would having insanely long commit messages.\n\nIs this repository public by any chance? I'd like to be able to take\na look at it in case I have time and have access to decent hardware.\nAlso, what command-line arguments did you use?\n\n> It's great git-svn can be restarted so well and doesn't get confused\n> by uncleanly terminated runs. My final repository is fast and small.\n> I'm still struggling with how to properly synchronize branches, but\n> that probably is mostly a matter of user education.\n> \n> Thanks all for these great tools.\n\nYou're welcome, thanks for the feedback! Restartability in git-svn\nis one of the things I focused on from the beginning.\n\n-- \nEric Wong\n"},{"id":"76169","messageId":"32541b130805052056g450b69cfg46693bc3c0c5a1ed@mail.gmail.com","threadId":"13247","inReplyTo":"20080506032846.GA15521@untitled","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-05-06T03:56:40Z","receivedAt":"2008-05-06T03:56:40Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 5/5/08, Eric Wong wrote:\n> Interesting. By \"These commits seemed all to have thousands of files\",\n> you mean the first 35K that took up most of the time? If so, yes,\n> that's definitely a problem...\n>\n> git-svn requests a log from SVN containing a list of all paths modified\n> in each revision. By default, git-svn only requests log entries for up\n> to 100 revisions at a time to reduce memory usage. However, having\n> thousands of files modified for each revision would still be\n> problematic, as would having insanely long commit messages.\n\nOn my system, any branch that was created using \"svn cp\" of a toplevel\ndirectory seems to cause git-svn to (rather slowly) download every\nsingle file in the entire branch for the first commit on that branch,\ngiving a symptom that sounds a lot like the above \"commits with\nthousands of files\". I assumed this was just an intentional design\ndecision in git-svn, to be slow and safe instead of fast and loose.\nIs it actually supposed to do something smarter than that?\n\nThanks,\n\nAvery\n"},{"id":"76170","messageId":"20080506042508.GA23465@untitled","threadId":"13247","inReplyTo":"32541b130805052056g450b69cfg46693bc3c0c5a1ed@mail.gmail.com","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2008-05-06T04:25:08Z","receivedAt":"2008-05-06T04:25:08Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Avery Pennarun wrote:\n> On 5/5/08, Eric Wong wrote:\n> > Interesting. By \"These commits seemed all to have thousands of files\",\n> > you mean the first 35K that took up most of the time? If so, yes,\n> > that's definitely a problem...\n> >\n> > git-svn requests a log from SVN containing a list of all paths modified\n> > in each revision. By default, git-svn only requests log entries for up\n> > to 100 revisions at a time to reduce memory usage. However, having\n> > thousands of files modified for each revision would still be\n> > problematic, as would having insanely long commit messages.\n> \n> On my system, any branch that was created using \"svn cp\" of a toplevel\n> directory seems to cause git-svn to (rather slowly) download every\n> single file in the entire branch for the first commit on that branch,\n> giving a symptom that sounds a lot like the above \"commits with\n> thousands of files\". I assumed this was just an intentional design\n> decision in git-svn, to be slow and safe instead of fast and loose.\n> Is it actually supposed to do something smarter than that?\n\nWhen using \"svn cp\" on a top-level directory, it *should*\njust show up as a single file change in the log entry.\nSomething like:\n\n A /project/branch/my-new-branch (from /project/trunk:1234)\n\nThis would not take much memory at all.\nHowever, I've also occasionally seen stuff like this:\n\n A /project/branch/my-new-branch\n A /project/branch/my-new-branch/file1 (from /project/trunk/file1:1234)\n A /project/branch/my-new-branch/file2 (from /project/trunk/file2:1234)\n A /project/branch/my-new-branch/file3 (from /project/trunk/file3:1234)\n .... many more files and directories along the same lines ...\n\nThis is what I suspect Geert is seeing in his repository and causing\nproblems. Perhaps something caused by cvs2svn importing those tags into\nSVN originally?\n\n\nBut the symptom you're seeing with git-svn downloading every file seems\nto be the result of using a pre-1.4.3 version of the Perl SVN bindings\nwhich lacked a working do_switch() function. I fallback to using\ndo_update() and checking out a new tree for SVN 1.4.2 and before.\nSo yes, I'm definitely safe, slow and _lazy_ by falling back to\ndo_update() instead of doing something fancy to workaround something\nthat's already fixed in SVN :)\n\n-- \nEric Wong\n"},{"id":"76194","messageId":"4484D251-B919-40AE-849B-1162F057550C@adacore.com","threadId":"13247","inReplyTo":"20080506042508.GA23465@untitled","subject":"Re: Excruciatingly slow git-svn imports","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2008-05-06T11:23:30Z","receivedAt":"2008-05-06T11:23:30Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn May 6, 2008, at 00:25, Eric Wong wrote:\n\n> When using \"svn cp\" on a top-level directory, it *should*\n> just show up as a single file change in the log entry.\n> Something like:\n>\n> A /project/branch/my-new-branch (from /project/trunk:1234)\n>\n> This would not take much memory at all.\n> However, I've also occasionally seen stuff like this:\n>\n> A /project/branch/my-new-branch\n> A /project/branch/my-new-branch/file1 (from /project/trunk/ \n> file1:1234)\n> A /project/branch/my-new-branch/file2 (from /project/trunk/ \n> file2:1234)\n> A /project/branch/my-new-branch/file3 (from /project/trunk/ \n> file3:1234)\n> .... many more files and directories along the same lines ...\n\nThis is exactly what I'm experiencing.\n> This is what I suspect Geert is seeing in his repository and causing\n> problems. Perhaps something caused by cvs2svn importing those tags \n> into\n> SVN originally?\n\nYes, most likely. BTW, the conversion from CVS to SVN took about a day\non slightly slower hardware.\n\n -Geert\n"}]}