{"thread":{"id":"37761","subject":"Re: git svn's performance issue and strange pauses, and other thing","startedAt":"2014-10-19T14:22:29Z","lastAt":"2014-10-19T14:22:29Z","messageCount":1,"participants":["Hin-Tak Leung"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"250829","messageId":"1413728549.57276.YahooMailBasic@web172306.mail.ir2.yahoo.com","threadId":"37761","inReplyTo":null,"subject":"Re: git svn's performance issue and strange pauses, and other thing","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2014-10-19T14:22:29Z","receivedAt":"2014-10-19T14:22:29Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"(sorry about the last blank reply - mobile phone and finger accident...)\n\n------------------------------\nOn Sun, Oct 19, 2014 05:12 BST Eric Wong wrote:\n\n>Hin-Tak Leung <htl10@users.sourceforge.net> wrote:\n> The new clone has:\n> \n> <--\n> $ ls -ltr .git/svn/.caches/\n> total 144788\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  1166138 Oct  7 13:44 lookup_svn_merge.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak 72849741 Oct  7 13:48 check_cherry_pick.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  1133855 Oct  7 13:49 has_no_changes.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak 73109005 Oct  7 13:53 _rev_list.yaml\n> -->\n> \n> The old clone has:\n>\n><snip>\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  40241189 Oct  5 16:42 lookup_svn_merge.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak 225323456 Oct  5 16:49 check_cherry_pick.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak    242547 Oct  5 16:49 has_no_changes.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  24120007 Oct  5 16:50 _rev_list.yaml\n> -->\n> \n> I had to suspend somewhat around r59000 - but it is interesting to see\n> that the max memory consumption of the later part is almost double?\n> and it also runs at 100% rather than 60% overall; I don't know what\n> to make of that - probably just smaller changes versus\n> larger ones, or different time of day and network loads (yes,\n> I guess it is just bandwidth-limited?, since the bulk of CPU time is in system\n> rather than user).\n>\n>git-svn memory usage is insane, and we need to reduce it.\n>(on Linux, fork() performance is reduced as memory size of the parent\n> grows, and I don't think we can easily call vfork() from Perl)\n>\n\nYes, I think the memory consumption is a bit crazy. I ran svn fetch on\nthe old again and it was a bit slow, so I timed the new, and here it is.\nFor just fetching 45 changes, it took 36 minutes and the memory \nconsumption shoots up to over 1GB. (there was one or two mergeinfo\nin the middle, not shown).\n\n<---\ncd ../R-2/\n[Hin-Tak@localhost R-2]$ /usr/bin/time -v git svn fetch --all\n\tM\tsrc/library/base/R/apply.R\n\tM\tsrc/library/base/man/apply.Rd\n\tM\tdoc/NEWS.Rd\nr66721 = e26e52bf4b2cdbe291d5899fd0a449f197aa2133 (refs/remotes/trunk)\n...\n\tM\tsrc/library/tools/R/utils.R\nr66765 = c64d1828ada98395892529ce59b5760de1bdc60b (refs/remotes/R-3-1-branch)\n---\n\tCommand being timed: \"git svn fetch --all\"\n\tUser time (seconds): 2042.81\n\tSystem time (seconds): 115.98\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 36:13.74\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 1019092\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 1149\n\tMinor (reclaiming a frame) page faults: 1482219\n\tVoluntary context switches: 9470\n\tInvoluntary context switches: 226683\n\tSwaps: 0\n\tFile system inputs: 358864\n\tFile system outputs: 510680\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n[Hin-Tak@localhost R-2]$ cd ../R\n--->\n\n\n> I am somwhat worry about the dramatic difference between the two .svn/.caches -\n> check_cherry_pick.yaml is 225MB in one and 73MB in the other, and also\n> _rev_list.yaml is opposite - 24MB vs 73MB. How do I reconcile that?\n>\n>Calling patterns changed, and it looks like Jakob's changes avoided some\n>calls.  The main thing to care about:\n>    Does the repository history look right?\n>\n\nI'll check soon and report. I looks superficiently okay. I suppose\nI'd need to check every branch to be sure. I know the fetch history is\ndifferent - but reflog (or the equivalent of it in svn) expires and are pruned\nafter two weeks?\n\n>The check_cherry_pick cache can be made smaller, too:\n>----------------------- 8< -----------------------------\n>From: Eric Wong <normalperson@yhbt.net>\n>Subject: [PATCH] git-svn: reduce check_cherry_pick cache overhead\n>\n>We do not need to store entire lists of commits, only the\n>number of incomplete and the first commit for reference.\n>This reduces the amount of data we need to store in memory\n>and on disk stores.\n>\n\nIs there a way of retrospectively compress/trimming the cache, or better\nstill, examine it before compressing?\n\nI intend to hold on to both the new and the old clone for a while until\nI can reconcil the differences... though I am running the same git svn code\non both now.\n\n>Signed-off-by: Eric Wong <normalperson@yhbt.net>\n>---\n> perl/Git/SVN.pm | 28 +++++++++++++++-------------\n> 1 file changed, 15 insertions(+), 13 deletions(-)\n>\n>diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\n>index 25dbcd5..b2d37cb 100644\n>--- a/perl/Git/SVN.pm\n>+++ b/perl/Git/SVN.pm\n>@@ -1537,7 +1537,7 @@ sub _rev_list {\n>     @rv;\n> }\n> \n>-sub check_cherry_pick {\n>+sub check_cherry_pick2 {\n>     my $base = shift;\n>     my $tip = shift;\n>     my $parents = shift;\n>@@ -1552,7 +1552,8 @@ sub check_cherry_pick {\n>             delete $commits{$commit};\n>         }\n>     }\n>-    return (keys %commits);\n>+    my @k = (keys %commits);\n>+    return (scalar @k, $k[0]);\n> }\n> \n> sub has_no_changes {\n>@@ -1597,7 +1598,7 @@ sub tie_for_persistent_memoization {\n>         mkpath([$cache_path]) unless -d $cache_path;\n> \n>         my %lookup_svn_merge_cache;\n>-        my %check_cherry_pick_cache;\n>+        my %check_cherry_pick2_cache;\n>         my %has_no_changes_cache;\n>         my %_rev_list_cache;\n> \n>@@ -1608,11 +1609,11 @@ sub tie_for_persistent_memoization {\n>             LIST_CACHE => ['HASH' => \\%lookup_svn_merge_cache],\n>         ;\n> \n>-        tie_for_persistent_memoization(\\%check_cherry_pick_cache,\n>-            \"$cache_path/check_cherry_pick\");\n>-        memoize 'check_cherry_pick',\n>+        tie_for_persistent_memoization(\\%check_cherry_pick2_cache,\n>+            \"$cache_path/check_cherry_pick2\");\n>+        memoize 'check_cherry_pick2',\n>             SCALAR_CACHE => 'FAULT',\n>-            LIST_CACHE => ['HASH' => \\%check_cherry_pick_cache],\n>+            LIST_CACHE => ['HASH' => \\%check_cherry_pick2_cache],\n>         ;\n> \n>         tie_for_persistent_memoization(\\%has_no_changes_cache,\n>@@ -1636,7 +1637,7 @@ sub tie_for_persistent_memoization {\n>         $memoized = 0;\n> \n>         Memoize::unmemoize 'lookup_svn_merge';\n>-        Memoize::unmemoize 'check_cherry_pick';\n>+        Memoize::unmemoize 'check_cherry_pick2';\n>         Memoize::unmemoize 'has_no_changes';\n>         Memoize::unmemoize '_rev_list';\n>     }\n>@@ -1648,7 +1649,8 @@ sub tie_for_persistent_memoization {\n>         return unless -d $cache_path;\n> \n>         for my $cache_file ((\"$cache_path/lookup_svn_merge\",\n>-                     \"$cache_path/check_cherry_pick\",\n>+                     \"$cache_path/check_cherry_pick\", # old\n>+                     \"$cache_path/check_cherry_pick2\",\n>                      \"$cache_path/has_no_changes\")) {\n>             for my $suffix (qw(yaml db)) {\n>                 my $file = \"$cache_file.$suffix\";\n>@@ -1817,15 +1819,15 @@ sub find_extra_svn_parents {\n>         }\n> \n>         # double check that there are no missing non-merge commits\n>-        my (@incomplete) = check_cherry_pick(\n>+        my ($ninc, $ifirst) = check_cherry_pick2(\n>             $merge_base, $merge_tip,\n>             $parents,\n>             @all_ranges,\n>                );\n> \n>-        if ( @incomplete ) {\n>-            warn \"W:svn cherry-pick ignored ($spec) - missing \"\n>-                .@incomplete.\" commit(s) (eg $incomplete[0])\\n\";\n>+        if ($ninc) {\n>+            warn \"W:svn cherry-pick ignored ($spec) - missing \" .\n>+                \"$ninc commit(s) (eg $ifirst)\\n\";\n>         } else {\n>             warn\n>                 \"Found merge parent ($spec): \",\n>-- \n>EW\n"}]}