# Re: Anomaly with the new code - Re: git-svn performance 4 messages from 2014-10-25 to 2014-10-27. Participants: Eric Wong, Hin-Tak Leung. Thread: https://gitlist.dev/t/37807 ## Hin-Tak Leung, 2014-10-25 05:29 Subject: Re: Anomaly with the new code - Re: git-svn performance Message-ID: <1414214985.98758.BPMail_high_carrier@web172306.mail.ir2.yahoo.com> URL: https://gitlist.dev/e/1414214985.98758.BPMail_high_carrier%40web172306.mail.ir2.yahoo.com ``` ------------------------------ On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote: >Hin-Tak Leung wrote: >> I keep tabs of a particular svn repository over many years >> and run "git svn fetch --all" every few days. So that's the old clone. >> Since this discussion started, I made a new one with git 2.1.0 patched >> with the first two patches below, a couple of weeks ago. And I ran >> 'git svn fetch --all' on both every few days since. >> >> I have added a few more patches, so the whole list is the 6 >> below against 2.1.0. The latest fetch is really strange - the fetch against >> the new clone took almost twice as long and uses almost twice >> as much memory, vs against the old. 17 min, 800 MB vs 10 min 400MB. >> Details below. Maybe this is a performance issue about how the clones >> were made? > >Memory usage seems to grow with the amount of revisions fetched, >see below. And higher memory means slower fork() on Linux systems. > but this is fetching the same number of revisions, and same revisions to keep the two clone in sync. So the issue is about how distant history is stored and used/searched, i think. >> 0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch >> 0002-git-svn-only-look-at-the-root-path-for-svn-mergeinfo.patch >> 0003-git-svn-reduce-check_cherry_pick-cache-overhead.patch >> 0004-git-svn-cache-only-mergeinfo-revisions.patch > >> 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch > >0006 is insufficient and incompatible with older SVN. >I pushed "git-svn: reload RA every log-window-size" >(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead >which saves much more memory: > it is fetching against the new clone taking twice as long and consuming twice as much memory. >http://mid.gmane.org/20141024225352.GB31716@dcvr.yhbt.net > >But there still seems to be some slow growth with many revisions >which is not mergeinfo-related. > >> 0007-git-svn-remove-mergeinfo-rev-caching.patch > >I think it is also safe to remove the _rev_list memoization since >it uses a lot of memory. The remaining caches should be tiny >(but useful, I think). ``` ## Eric Wong, 2014-10-25 05:51 Subject: Re: Anomaly with the new code - Re: git-svn performance Message-ID: <20141025055118.GB30872@dcvr.yhbt.net> URL: https://gitlist.dev/e/20141025055118.GB30872%40dcvr.yhbt.net In-Reply-To: <1414214985.98758.BPMail_high_carrier@web172306.mail.ir2.yahoo.com> ``` Hin-Tak Leung wrote: > On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote: > >Hin-Tak Leung wrote: > >> 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch > > > >0006 is insufficient and incompatible with older SVN. > >I pushed "git-svn: reload RA every log-window-size" > >(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead > >which saves much more memory: > > > it is fetching against the new clone taking twice as long and > consuming twice as much memory. Ugh, I've only tested "git-svn: reload RA every log-window-size" with file:// repos so far, so it looks like I'll need to setup remote repos on my test system to test. ``` ## Eric Wong, 2014-10-27 06:38 Subject: Re: Anomaly with the new code - Re: git-svn performance Message-ID: <20141027063818.GA17332@dcvr.yhbt.net> URL: https://gitlist.dev/e/20141027063818.GA17332%40dcvr.yhbt.net In-Reply-To: <1414214985.98758.BPMail_high_carrier@web172306.mail.ir2.yahoo.com> ``` Hin-Tak Leung wrote: > On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote: > >0006 is insufficient and incompatible with older SVN. > >I pushed "git-svn: reload RA every log-window-size" > >(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead > >which saves much more memory: > > it is fetching against the new clone taking twice as long and > consuming twice as much memory. Which SVN version are you using? I'm cloning (currently on r373xx) https://svn.r-project.org/R using --stdlayout and unable to see memory growth of the git-svn Perl process beyond 40M (on a 32-bit system). I also tried http:// (not https), svn+ssh:// on my local (64-bit) system and did not see memory growth, either: http://mid.gmane.org/20141027014033.GA4189@dcvr.yhbt.net I'm using svn 1.6.17 on Debian stable in all cases. ``` ## Eric Wong, 2014-10-27 16:56 Subject: Re: Anomaly with the new code - Re: git-svn performance Message-ID: <20141027165650.GA18752@dcvr.yhbt.net> URL: https://gitlist.dev/e/20141027165650.GA18752%40dcvr.yhbt.net In-Reply-To: <20141027063818.GA17332@dcvr.yhbt.net> ``` Eric Wong wrote: > Which SVN version are you using? I'm cloning (currently on r373xx) > https://svn.r-project.org/R using --stdlayout and > unable to see memory growth of the git-svn Perl process beyond 40M > (on a 32-bit system). git-svn hit 45M and took 11:44 to finish. My ping times to svn.r-project.org is around 150ms (I'm running this from a server in Fremont, California). I'll keep the repo around and periodically fetch to see how it runs. ```