# [PATCH v3] git-svn: memoize _rev_list and rebuild 4 messages from 2014-01-23 to 2014-10-25. Participants: manjian2006@gmail.com, Eric Wong, Junio C Hamano. Thread: https://gitlist.dev/t/35716 ## manjian2006@gmail.com, 2014-01-23 02:15 Subject: [PATCH v3] git-svn: memoize _rev_list and rebuild Message-ID: <1390443319-11239-1-git-send-email-manjian2006@gmail.com> URL: https://gitlist.dev/e/1390443319-11239-1-git-send-email-manjian2006%40gmail.com ``` From: lin zuojian According to profile data, _rev_list and rebuild consume a large portion of time. Memoize the results of _rev_list and memoize rebuild internals to avoid subprocess invocation. When importing 15152 revisions on a LAN, time improved from 10 hours to 3-4 hours. Signed-off-by: lin zuojian --- perl/Git/SVN.pm | 41 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm index 5273ee8..6e804a2 100644 --- a/perl/Git/SVN.pm +++ b/perl/Git/SVN.pm @@ -1599,6 +1599,7 @@ sub tie_for_persistent_memoization { my %lookup_svn_merge_cache; my %check_cherry_pick_cache; my %has_no_changes_cache; + my %_rev_list_cache; tie_for_persistent_memoization(\%lookup_svn_merge_cache, "$cache_path/lookup_svn_merge"); @@ -1620,6 +1621,14 @@ sub tie_for_persistent_memoization { SCALAR_CACHE => ['HASH' => \%has_no_changes_cache], LIST_CACHE => 'FAULT', ; + + tie_for_persistent_memoization(\%_rev_list_cache, + "$cache_path/_rev_list"); + memoize '_rev_list', + SCALAR_CACHE => 'FAULT', + LIST_CACHE => ['HASH' => \%_rev_list_cache], + ; + } sub unmemoize_svn_mergeinfo_functions { @@ -1629,6 +1638,7 @@ sub tie_for_persistent_memoization { Memoize::unmemoize 'lookup_svn_merge'; Memoize::unmemoize 'check_cherry_pick'; Memoize::unmemoize 'has_no_changes'; + Memoize::unmemoize '_rev_list'; } sub clear_memoized_mergeinfo_caches { @@ -1959,11 +1969,25 @@ sub rebuild_from_rev_db { unlink $path or croak "unlink: $!"; } +#define a global associate map to record rebuild status +my %rebuild_status; +#define a global associate map to record rebuild verify status +my %rebuild_verify_status; + sub rebuild { my ($self) = @_; my $map_path = $self->map_path; my $partial = (-e $map_path && ! -z $map_path); - return unless ::verify_ref($self->refname.'^0'); + my $verify_key = $self->refname.'^0'; + if (!$rebuild_verify_status{$verify_key}) { + my $verify_result = ::verify_ref($verify_key); + if ($verify_result) { + $rebuild_verify_status{$verify_key} = 1; + } + } + if (!$rebuild_verify_status{$verify_key}) { + return; + } if (!$partial && ($self->use_svm_props || $self->no_metadata)) { my $rev_db = $self->rev_db_path; $self->rebuild_from_rev_db($rev_db); @@ -1977,10 +2001,21 @@ sub rebuild { print "Rebuilding $map_path ...\n" if (!$partial); my ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) : (undef, undef)); + my $key_value = ($head ? "$head.." : "") . $self->refname; + if (exists $rebuild_status{$key_value}) { + print "Done rebuilding $map_path\n" if (!$partial || !$head); + my $rev_db_path = $self->rev_db_path; + if (-f $self->rev_db_path) { + unlink $self->rev_db_path or croak "unlink: $!"; + } + $self->unlink_rev_db_symlink; + return; + } my ($log, $ctx) = - command_output_pipe(qw/rev-list --pretty=raw --reverse/, - ($head ? "$head.." : "") . $self->refname, + command_output_pipe(qw/rev-list --pretty=raw --reverse/, + $key_value, '--'); + $rebuild_status{$key_value} = 1; my $metadata_url = $self->metadata_url; remove_username($metadata_url); my $svn_uuid = $self->rewrite_uuid || $self->ra_uuid; -- 1.8.3.2 ``` ## Eric Wong, 2014-01-23 02:58 Subject: Re: [PATCH v3] git-svn: memoize _rev_list and rebuild Message-ID: <20140123025807.GA4370@dcvr.yhbt.net> URL: https://gitlist.dev/e/20140123025807.GA4370%40dcvr.yhbt.net In-Reply-To: <1390443319-11239-1-git-send-email-manjian2006@gmail.com> ``` manjian2006@gmail.com wrote: > According to profile data, _rev_list and rebuild consume a large > portion of time. Memoize the results of _rev_list and memoize > rebuild internals to avoid subprocess invocation. > > When importing 15152 revisions on a LAN, time improved from 10 > hours to 3-4 hours. > > Signed-off-by: lin zuojian Thanks! Signed-off-by: Eric Wong Pushed for Junio. The following changes since commit d9bb4be53bc5185244b4be9860562a012803bacb: Merge tag 'gitgui-0.19.0' of http://repo.or.cz/r/git-gui (2014-01-21 13:16:17 -0800) are available in the git repository at: git://git.bogomips.org/git-svn.git master for you to fetch changes up to ab0bcec9873f1fcef6c4b8825cc9e762c636ca9e: git-svn: memoize _rev_list and rebuild (2014-01-23 02:54:26 +0000) ---------------------------------------------------------------- lin zuojian (1): git-svn: memoize _rev_list and rebuild perl/Git/SVN.pm | 41 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) ``` ## Junio C Hamano, 2014-01-23 16:49 Subject: Re: [PATCH v3] git-svn: memoize _rev_list and rebuild Message-ID: URL: https://gitlist.dev/e/xmqq8uu6wsg4.fsf%40gitster.dls.corp.google.com In-Reply-To: <20140123025807.GA4370@dcvr.yhbt.net> ``` Eric Wong writes: > Pushed for Junio. Thanks. ``` ## Eric Wong, 2014-10-25 05:44 Subject: Re: [PATCH v3] git-svn: memoize _rev_list and rebuild Message-ID: <20141025054426.GA31107@dcvr.yhbt.net> URL: https://gitlist.dev/e/20141025054426.GA31107%40dcvr.yhbt.net In-Reply-To: <1390443319-11239-1-git-send-email-manjian2006@gmail.com> ``` manjian2006@gmail.com wrote: > From: lin zuojian > > According to profile data, _rev_list and rebuild consume a large > portion of time. Memoize the results of _rev_list and memoize > rebuild internals to avoid subprocess invocation. Hi, I am considering reverting _rev_list memoization because it can increases memory usage a lot, which hurting fork() performance on Linux. Can you tell me if your original results are under Linux or another kernel? I may only disable _rev_list memoization under Linux and leave other OSes unaffected, too. I've merged some patches from Jakob to improve svn:mergeinfo performance along with some followup patches to reduce memory use. However memory bloat is still a problem. Currently in my "master" branch of git://bogomips.org/git-svn Eric Wong (4): git-svn: reduce check_cherry_pick cache overhead git-svn: cache only mergeinfo revisions git-svn: clear global SVN pool between get_log invocations git-svn: remove mergeinfo rev caching Jakob Stoklund Olesen (2): git-svn: only look at the new parts of svn:mergeinfo git-svn: only look at the root path for svn:mergeinfo ```