{"thread":{"id":"35716","subject":"[PATCH v3] git-svn: memoize _rev_list and rebuild","startedAt":"2014-01-23T02:15:19Z","lastAt":"2014-10-25T05:44:26Z","messageCount":4,"participants":["manjian2006@gmail.com","Eric Wong","Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":null},"messages":[{"id":"233582","messageId":"1390443319-11239-1-git-send-email-manjian2006@gmail.com","threadId":"35716","inReplyTo":null,"subject":"[PATCH v3] git-svn: memoize _rev_list and rebuild","fromName":"","fromEmail":"manjian2006@gmail.com","sentAt":"2014-01-23T02:15:19Z","receivedAt":"2014-01-23T02:15:19Z","isPatch":true,"sender":{"key":"manjian2006@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5935381?v=4"},"body":"From: lin zuojian \n\nAccording to profile data, _rev_list and rebuild consume a large\nportion of time. Memoize the results of _rev_list and memoize\nrebuild internals to avoid subprocess invocation.\n\nWhen importing 15152 revisions on a LAN, time improved from 10\nhours to 3-4 hours.\n\nSigned-off-by: lin zuojian \n---\n perl/Git/SVN.pm | 41 ++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 38 insertions(+), 3 deletions(-)\n\ndiff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\nindex 5273ee8..6e804a2 100644\n--- a/perl/Git/SVN.pm\n+++ b/perl/Git/SVN.pm\n@@ -1599,6 +1599,7 @@ sub tie_for_persistent_memoization {\n \t\tmy %lookup_svn_merge_cache;\n \t\tmy %check_cherry_pick_cache;\n \t\tmy %has_no_changes_cache;\n+\t\tmy %_rev_list_cache;\n \n \t\ttie_for_persistent_memoization(\\%lookup_svn_merge_cache,\n \t\t \"$cache_path/lookup_svn_merge\");\n@@ -1620,6 +1621,14 @@ sub tie_for_persistent_memoization {\n \t\t\tSCALAR_CACHE => ['HASH' => \\%has_no_changes_cache],\n \t\t\tLIST_CACHE => 'FAULT',\n \t\t;\n+\n+\t\ttie_for_persistent_memoization(\\%_rev_list_cache,\n+\t\t \"$cache_path/_rev_list\");\n+\t\tmemoize '_rev_list',\n+\t\t\tSCALAR_CACHE => 'FAULT',\n+\t\t\tLIST_CACHE => ['HASH' => \\%_rev_list_cache],\n+\t\t;\n+\n \t}\n \n \tsub unmemoize_svn_mergeinfo_functions {\n@@ -1629,6 +1638,7 @@ sub tie_for_persistent_memoization {\n \t\tMemoize::unmemoize 'lookup_svn_merge';\n \t\tMemoize::unmemoize 'check_cherry_pick';\n \t\tMemoize::unmemoize 'has_no_changes';\n+\t\tMemoize::unmemoize '_rev_list';\n \t}\n \n \tsub clear_memoized_mergeinfo_caches {\n@@ -1959,11 +1969,25 @@ sub rebuild_from_rev_db {\n \tunlink $path or croak \"unlink: $!\";\n }\n \n+#define a global associate map to record rebuild status\n+my %rebuild_status;\n+#define a global associate map to record rebuild verify status\n+my %rebuild_verify_status;\n+\n sub rebuild {\n \tmy ($self) = @_;\n \tmy $map_path = $self->map_path;\n \tmy $partial = (-e $map_path && ! -z $map_path);\n-\treturn unless ::verify_ref($self->refname.'^0');\n+\tmy $verify_key = $self->refname.'^0';\n+\tif (!$rebuild_verify_status{$verify_key}) {\n+\t\tmy $verify_result = ::verify_ref($verify_key);\n+\t\tif ($verify_result) {\n+\t\t\t$rebuild_verify_status{$verify_key} = 1;\n+\t\t}\n+\t}\n+\tif (!$rebuild_verify_status{$verify_key}) {\n+\t\treturn;\n+\t}\n \tif (!$partial && ($self->use_svm_props || $self->no_metadata)) {\n \t\tmy $rev_db = $self->rev_db_path;\n \t\t$self->rebuild_from_rev_db($rev_db);\n@@ -1977,10 +2001,21 @@ sub rebuild {\n \tprint \"Rebuilding $map_path ...\\n\" if (!$partial);\n \tmy ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :\n \t\t(undef, undef));\n+\tmy $key_value = ($head ? \"$head..\" : \"\") . $self->refname;\n+\tif (exists $rebuild_status{$key_value}) {\n+\t\tprint \"Done rebuilding $map_path\\n\" if (!$partial || !$head);\n+\t\tmy $rev_db_path = $self->rev_db_path;\n+\t\tif (-f $self->rev_db_path) {\n+\t\t\tunlink $self->rev_db_path or croak \"unlink: $!\";\n+\t\t}\n+\t\t$self->unlink_rev_db_symlink;\n+\t\treturn;\n+\t}\n \tmy ($log, $ctx) =\n-\t command_output_pipe(qw/rev-list --pretty=raw --reverse/,\n-\t\t\t\t($head ? \"$head..\" : \"\") . $self->refname,\n+\t\tcommand_output_pipe(qw/rev-list --pretty=raw --reverse/,\n+\t\t\t\t$key_value,\n \t\t\t\t'--');\n+\t$rebuild_status{$key_value} = 1;\n \tmy $metadata_url = $self->metadata_url;\n \tremove_username($metadata_url);\n \tmy $svn_uuid = $self->rewrite_uuid || $self->ra_uuid;\n-- \n1.8.3.2\n"},{"id":"233585","messageId":"20140123025807.GA4370@dcvr.yhbt.net","threadId":"35716","inReplyTo":"1390443319-11239-1-git-send-email-manjian2006@gmail.com","subject":"Re: [PATCH v3] git-svn: memoize _rev_list and rebuild","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-01-23T02:58:07Z","receivedAt":"2014-01-23T02:58:07Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"manjian2006@gmail.com wrote:\n> According to profile data, _rev_list and rebuild consume a large\n> portion of time. Memoize the results of _rev_list and memoize\n> rebuild internals to avoid subprocess invocation.\n> \n> When importing 15152 revisions on a LAN, time improved from 10\n> hours to 3-4 hours.\n> \n> Signed-off-by: lin zuojian \n\nThanks!\nSigned-off-by: Eric Wong \nPushed for Junio.\n\nThe following changes since commit d9bb4be53bc5185244b4be9860562a012803bacb:\n\n Merge tag 'gitgui-0.19.0' of http://repo.or.cz/r/git-gui (2014-01-21 13:16:17 -0800)\n\nare available in the git repository at:\n\n\n git://git.bogomips.org/git-svn.git master\n\nfor you to fetch changes up to ab0bcec9873f1fcef6c4b8825cc9e762c636ca9e:\n\n git-svn: memoize _rev_list and rebuild (2014-01-23 02:54:26 +0000)\n\n----------------------------------------------------------------\nlin zuojian (1):\n git-svn: memoize _rev_list and rebuild\n\n perl/Git/SVN.pm | 41 ++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 38 insertions(+), 3 deletions(-)\n"},{"id":"233600","messageId":"xmqq8uu6wsg4.fsf@gitster.dls.corp.google.com","threadId":"35716","inReplyTo":"20140123025807.GA4370@dcvr.yhbt.net","subject":"Re: [PATCH v3] git-svn: memoize _rev_list and rebuild","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-01-23T16:49:43Z","receivedAt":"2014-01-23T16:49:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> Pushed for Junio.\n\nThanks.\n"},{"id":"251038","messageId":"20141025054426.GA31107@dcvr.yhbt.net","threadId":"35716","inReplyTo":"1390443319-11239-1-git-send-email-manjian2006@gmail.com","subject":"Re: [PATCH v3] git-svn: memoize _rev_list and rebuild","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-25T05:44:26Z","receivedAt":"2014-10-25T05:44:26Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"manjian2006@gmail.com wrote:\n> From: lin zuojian \n> \n> According to profile data, _rev_list and rebuild consume a large\n> portion of time. Memoize the results of _rev_list and memoize\n> rebuild internals to avoid subprocess invocation.\n\nHi, I am considering reverting _rev_list memoization because it\ncan increases memory usage a lot, which hurting fork() performance on\nLinux.\n\nCan you tell me if your original results are under Linux or another\nkernel? I may only disable _rev_list memoization under Linux and\nleave other OSes unaffected, too.\n\nI've merged some patches from Jakob to improve svn:mergeinfo\nperformance along with some followup patches to reduce memory\nuse. However memory bloat is still a problem.\n\nCurrently in my \"master\" branch of git://bogomips.org/git-svn\n\nEric Wong (4):\n git-svn: reduce check_cherry_pick cache overhead\n git-svn: cache only mergeinfo revisions\n git-svn: clear global SVN pool between get_log invocations\n git-svn: remove mergeinfo rev caching\n\nJakob Stoklund Olesen (2):\n git-svn: only look at the new parts of svn:mergeinfo\n git-svn: only look at the root path for svn:mergeinfo\n"}]}