Re: Anomaly with the new code - Re: git-svn performance

From: Eric Wong <normalperson@yhbt.net>
Date: Oct 24, 2014, 23:34 UTC
Message-ID: <20141024233411.GA18655@dcvr.yhbt.net>
In-Reply-To: <1414134386.28852.YahooMailBasic@web172306.mail.ir2.yahoo.com>

Hin-Tak Leung <htl10@users.sourceforge.net> wrote:

Show 12 quoted lines

> I keep tabs of a particular svn repository over many years
> and run "git svn fetch --all" every few days. So that's the old clone.
> Since this discussion started, I made a new one with git 2.1.0 patched
> with the first two patches below, a couple of weeks ago. And I ran
> 'git svn fetch --all' on both every few days since.
> 
> I have added a few more patches, so the whole list is the 6
> below against 2.1.0. The latest fetch is really strange - the fetch against
> the new clone took almost twice as long and uses almost twice
> as much memory, vs against the old. 17 min, 800 MB vs 10 min 400MB.
> Details below. Maybe this is a performance issue about how the clones
> were made?

Memory usage seems to grow with the amount of revisions fetched, see below. And higher memory means slower fork() on Linux systems.

> 0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch    
> 0002-git-svn-only-look-at-the-root-path-for-svn-mergeinfo.patch   
> 0003-git-svn-reduce-check_cherry_pick-cache-overhead.patch        
> 0004-git-svn-cache-only-mergeinfo-revisions.patch

> 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch

0006 is insufficient and incompatible with older SVN. I pushed "git-svn: reload RA every log-window-size" (commit dfa72fdb96befbd790f623bb2909a347176753c2) instead which saves much more memory:

http://mid.gmane.org/20141024225352.GB31716@dcvr.yhbt.net

But there still seems to be some slow growth with many revisions which is not mergeinfo-related.

> 0007-git-svn-remove-mergeinfo-rev-caching.patch

I think it is also safe to remove the _rev_list memoization since it uses a lot of memory. The remaining caches should be tiny (but useful, I think).