{"thread":{"id":"37807","subject":"Re: Anomaly with the new code - Re: git-svn performance","startedAt":"2014-10-25T05:29:45Z","lastAt":"2014-10-27T16:56:50Z","messageCount":4,"participants":["Eric Wong","Hin-Tak Leung"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"251048","messageId":"1414214985.98758.BPMail_high_carrier@web172306.mail.ir2.yahoo.com","threadId":"37807","inReplyTo":null,"subject":"Re: Anomaly with the new code - Re: git-svn performance","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2014-10-25T05:29:45Z","receivedAt":"2014-10-25T05:29:45Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"\n\n\n\n\n------------------------------\nOn Sat, Oct 25, 2014 00:34 BST Eric Wong wrote:\n\n>Hin-Tak Leung wrote:\n>> I keep tabs of a particular svn repository over many years\n>> and run \"git svn fetch --all\" every few days. So that's the old clone.\n>> Since this discussion started, I made a new one with git 2.1.0 patched\n>> with the first two patches below, a couple of weeks ago. And I ran\n>> 'git svn fetch --all' on both every few days since.\n>> \n>> I have added a few more patches, so the whole list is the 6\n>> below against 2.1.0. The latest fetch is really strange - the fetch against\n>> the new clone took almost twice as long and uses almost twice\n>> as much memory, vs against the old. 17 min, 800 MB vs 10 min 400MB.\n>> Details below. Maybe this is a performance issue about how the clones\n>> were made?\n>\n>Memory usage seems to grow with the amount of revisions fetched,\n>see below. And higher memory means slower fork() on Linux systems.\n>\n\nbut this is fetching the same number of revisions, and same revisions to keep the two clone in sync. So the issue is about how distant history is stored and used/searched, i think.\n\n>> 0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch \n>> 0002-git-svn-only-look-at-the-root-path-for-svn-mergeinfo.patch \n>> 0003-git-svn-reduce-check_cherry_pick-cache-overhead.patch \n>> 0004-git-svn-cache-only-mergeinfo-revisions.patch \n>\n>> 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch \n>\n>0006 is insufficient and incompatible with older SVN.\n>I pushed \"git-svn: reload RA every log-window-size\"\n>(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead\n>which saves much more memory:\n>\n\nit is fetching against the new clone taking twice as long and consuming twice as much memory.\n\n>http://mid.gmane.org/20141024225352.GB31716@dcvr.yhbt.net\n>\n>But there still seems to be some slow growth with many revisions\n>which is not mergeinfo-related.\n>\n>> 0007-git-svn-remove-mergeinfo-rev-caching.patch \n>\n>I think it is also safe to remove the _rev_list memoization since\n>it uses a lot of memory. The remaining caches should be tiny\n>(but useful, I think).\n"},{"id":"251039","messageId":"20141025055118.GB30872@dcvr.yhbt.net","threadId":"37807","inReplyTo":"1414214985.98758.BPMail_high_carrier@web172306.mail.ir2.yahoo.com","subject":"Re: Anomaly with the new code - Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-25T05:51:18Z","receivedAt":"2014-10-25T05:51:18Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hin-Tak Leung wrote:\n> On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote:\n> >Hin-Tak Leung wrote:\n> >> 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch \n> >\n> >0006 is insufficient and incompatible with older SVN.\n> >I pushed \"git-svn: reload RA every log-window-size\"\n> >(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead\n> >which saves much more memory:\n> >\n> it is fetching against the new clone taking twice as long and\n> consuming twice as much memory.\n\nUgh, I've only tested \"git-svn: reload RA every log-window-size\" with\nfile:// repos so far, so it looks like I'll need to setup remote repos\non my test system to test.\n"},{"id":"251091","messageId":"20141027063818.GA17332@dcvr.yhbt.net","threadId":"37807","inReplyTo":"1414214985.98758.BPMail_high_carrier@web172306.mail.ir2.yahoo.com","subject":"Re: Anomaly with the new code - Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-27T06:38:18Z","receivedAt":"2014-10-27T06:38:18Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hin-Tak Leung wrote:\n> On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote:\n> >0006 is insufficient and incompatible with older SVN.\n> >I pushed \"git-svn: reload RA every log-window-size\"\n> >(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead\n> >which saves much more memory:\n>\n> it is fetching against the new clone taking twice as long and\n> consuming twice as much memory.\n\nWhich SVN version are you using? I'm cloning (currently on r373xx)\nhttps://svn.r-project.org/R using --stdlayout and\nunable to see memory growth of the git-svn Perl process beyond 40M\n(on a 32-bit system).\n\nI also tried http:// (not https), svn+ssh:// on my local (64-bit) system\nand did not see memory growth, either:\n\n\thttp://mid.gmane.org/20141027014033.GA4189@dcvr.yhbt.net\n\nI'm using svn 1.6.17 on Debian stable in all cases.\n"},{"id":"251120","messageId":"20141027165650.GA18752@dcvr.yhbt.net","threadId":"37807","inReplyTo":"20141027063818.GA17332@dcvr.yhbt.net","subject":"Re: Anomaly with the new code - Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-27T16:56:50Z","receivedAt":"2014-10-27T16:56:50Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong wrote:\n> Which SVN version are you using? I'm cloning (currently on r373xx)\n> https://svn.r-project.org/R using --stdlayout and\n> unable to see memory growth of the git-svn Perl process beyond 40M\n> (on a 32-bit system).\n\ngit-svn hit 45M and took 11:44 to finish. My ping times to\nsvn.r-project.org is around 150ms (I'm running this from a server in\nFremont, California). I'll keep the repo around and periodically fetch\nto see how it runs.\n"}]}