Re: Anomaly with the new code - Re: git-svn performance

From: Hin-Tak Leung <htl10@users.sourceforge.net>
Date: Oct 25, 2014, 05:29 UTC
Message-ID: <1414214985.98758.BPMail_high_carrier@web172306.mail.ir2.yahoo.com>

------------------------------ On Sat, Oct 25, 2014 00:34 BST Eric Wong wrote:

Show 17 quoted lines

>Hin-Tak Leung <htl10@users.sourceforge.net> wrote:
>> I keep tabs of a particular svn repository over many years
>> and run "git svn fetch --all" every few days. So that's the old clone.
>> Since this discussion started, I made a new one with git 2.1.0 patched
>> with the first two patches below, a couple of weeks ago. And I ran
>> 'git svn fetch --all' on both every few days since.
>> 
>> I have added a few more patches, so the whole list is the 6
>> below against 2.1.0. The latest fetch is really strange - the fetch against
>> the new clone took almost twice as long and uses almost twice
>> as much memory, vs against the old. 17 min, 800 MB vs 10 min 400MB.
>> Details below. Maybe this is a performance issue about how the clones
>> were made?
>
>Memory usage seems to grow with the amount of revisions fetched,
>see below.  And higher memory means slower fork() on Linux systems.
>

but this is fetching the same number of revisions, and same revisions to keep the two clone in sync. So the issue is about how distant history is stored and used/searched, i think.

Show 12 quoted lines

>> 0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch    
>> 0002-git-svn-only-look-at-the-root-path-for-svn-mergeinfo.patch   
>> 0003-git-svn-reduce-check_cherry_pick-cache-overhead.patch        
>> 0004-git-svn-cache-only-mergeinfo-revisions.patch                 
>
>> 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch   
>
>0006 is insufficient and incompatible with older SVN.
>I pushed "git-svn: reload RA every log-window-size"
>(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead
>which saves much more memory:
>

it is fetching against the new clone taking twice as long and consuming twice as much memory.

Show 10 quoted lines

>http://mid.gmane.org/20141024225352.GB31716@dcvr.yhbt.net
>
>But there still seems to be some slow growth with many revisions
>which is not mergeinfo-related.
>
>> 0007-git-svn-remove-mergeinfo-rev-caching.patch 
>
>I think it is also safe to remove the _rev_list memoization since
>it uses a lot of memory.  The remaining caches should be tiny
>(but useful, I think).