{"thread":{"id":"37759","subject":"Re: git svn's performance issue and strange pauses, and other thing","startedAt":"2014-10-19T14:04:51Z","lastAt":"2014-10-19T14:04:51Z","messageCount":1,"participants":["Hin-Tak Leung"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"250827","messageId":"1413727491.1339.BPMail_high_carrier@web172302.mail.ir2.yahoo.com","threadId":"37759","inReplyTo":null,"subject":"Re: git svn's performance issue and strange pauses, and other thing","fromName":"Hin-Tak Leung","fromEmail":"hintak_leung@yahoo.co.uk","sentAt":"2014-10-19T14:04:51Z","receivedAt":"2014-10-19T14:04:51Z","isPatch":false,"sender":{"key":"hintak_leung@yahoo.co.uk","avatar":null},"body":"\n\n\n\n\n------------------------------\nOn Sun, Oct 19, 2014 05:12 BST Eric Wong wrote:\n\n>Hin-Tak Leung <htl10@users.sourceforge.net> wrote:\n> The new clone has:\n> \n> <--\n> $ ls -ltr .git/svn/.caches/\n> total 144788\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  1166138 Oct  7 13:44 lookup_svn_merge.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak 72849741 Oct  7 13:48 check_cherry_pick.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  1133855 Oct  7 13:49 has_no_changes.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak 73109005 Oct  7 13:53 _rev_list.yaml\n> -->\n> \n> The old clone has:\n>\n><snip>\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  40241189 Oct  5 16:42 lookup_svn_merge.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak 225323456 Oct  5 16:49 check_cherry_pick.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak    242547 Oct  5 16:49 has_no_changes.yaml\n> -rw-rw-r--. 1 Hin-Tak Hin-Tak  24120007 Oct  5 16:50 _rev_list.yaml\n> -->\n> \n> I had to suspend somewhat around r59000 - but it is interesting to see\n> that the max memory consumption of the later part is almost double?\n> and it also runs at 100% rather than 60% overall; I don't know what\n> to make of that - probably just smaller changes versus\n> larger ones, or different time of day and network loads (yes,\n> I guess it is just bandwidth-limited?, since the bulk of CPU time is in system\n> rather than user).\n>\n>git-svn memory usage is insane, and we need to reduce it.\n>(on Linux, fork() performance is reduced as memory size of the parent\n> grows, and I don't think we can easily call vfork() from Perl)\n>\n> I am somwhat worry about the dramatic difference between the two .svn/.caches -\n> check_cherry_pick.yaml is 225MB in one and 73MB in the other, and also\n> _rev_list.yaml is opposite - 24MB vs 73MB. How do I reconcile that?\n>\n>Calling patterns changed, and it looks like Jakob's changes avoided some\n>calls.  The main thing to care about:\n>\tDoes the repository history look right?\n>\n>The check_cherry_pick cache can be made smaller, too:\n>----------------------- 8< -----------------------------\n>From: Eric Wong <normalperson@yhbt.net>\n>Subject: [PATCH] git-svn: reduce check_cherry_pick cache overhead\n>\n>We do not need to store entire lists of commits, only the\n>number of incomplete and the first commit for reference.\n>This reduces the amount of data we need to store in memory\n>and on disk stores.\n>\n>Signed-off-by: Eric Wong <normalperson@yhbt.net>\n>---\n> perl/Git/SVN.pm | 28 +++++++++++++++-------------\n> 1 file changed, 15 insertions(+), 13 deletions(-)\n>\n>diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\n>index 25dbcd5..b2d37cb 100644\n>--- a/perl/Git/SVN.pm\n>+++ b/perl/Git/SVN.pm\n>@@ -1537,7 +1537,7 @@ sub _rev_list {\n> \t@rv;\n> }\n> \n>-sub check_cherry_pick {\n>+sub check_cherry_pick2 {\n> \tmy $base = shift;\n> \tmy $tip = shift;\n> \tmy $parents = shift;\n>@@ -1552,7 +1552,8 @@ sub check_cherry_pick {\n> \t\t\tdelete $commits{$commit};\n> \t\t}\n> \t}\n>-\treturn (keys %commits);\n>+\tmy @k = (keys %commits);\n>+\treturn (scalar @k, $k[0]);\n> }\n> \n> sub has_no_changes {\n>@@ -1597,7 +1598,7 @@ sub tie_for_persistent_memoization {\n> \t\tmkpath([$cache_path]) unless -d $cache_path;\n> \n> \t\tmy %lookup_svn_merge_cache;\n>-\t\tmy %check_cherry_pick_cache;\n>+\t\tmy %check_cherry_pick2_cache;\n> \t\tmy %has_no_changes_cache;\n> \t\tmy %_rev_list_cache;\n> \n>@@ -1608,11 +1609,11 @@ sub tie_for_persistent_memoization {\n> \t\t\tLIST_CACHE => ['HASH' => \\%lookup_svn_merge_cache],\n> \t\t;\n> \n>-\t\ttie_for_persistent_memoization(\\%check_cherry_pick_cache,\n>-\t\t    \"$cache_path/check_cherry_pick\");\n>-\t\tmemoize 'check_cherry_pick',\n>+\t\ttie_for_persistent_memoization(\\%check_cherry_pick2_cache,\n>+\t\t    \"$cache_path/check_cherry_pick2\");\n>+\t\tmemoize 'check_cherry_pick2',\n> \t\t\tSCALAR_CACHE => 'FAULT',\n>-\t\t\tLIST_CACHE => ['HASH' => \\%check_cherry_pick_cache],\n>+\t\t\tLIST_CACHE => ['HASH' => \\%check_cherry_pick2_cache],\n> \t\t;\n> \n> \t\ttie_for_persistent_memoization(\\%has_no_changes_cache,\n>@@ -1636,7 +1637,7 @@ sub tie_for_persistent_memoization {\n> \t\t$memoized = 0;\n> \n> \t\tMemoize::unmemoize 'lookup_svn_merge';\n>-\t\tMemoize::unmemoize 'check_cherry_pick';\n>+\t\tMemoize::unmemoize 'check_cherry_pick2';\n> \t\tMemoize::unmemoize 'has_no_changes';\n> \t\tMemoize::unmemoize '_rev_list';\n> \t}\n>@@ -1648,7 +1649,8 @@ sub tie_for_persistent_memoization {\n> \t\treturn unless -d $cache_path;\n> \n> \t\tfor my $cache_file ((\"$cache_path/lookup_svn_merge\",\n>-\t\t\t\t     \"$cache_path/check_cherry_pick\",\n>+\t\t\t\t     \"$cache_path/check_cherry_pick\", # old\n>+\t\t\t\t     \"$cache_path/check_cherry_pick2\",\n> \t\t\t\t     \"$cache_path/has_no_changes\")) {\n> \t\t\tfor my $suffix (qw(yaml db)) {\n> \t\t\t\tmy $file = \"$cache_file.$suffix\";\n>@@ -1817,15 +1819,15 @@ sub find_extra_svn_parents {\n> \t\t}\n> \n> \t\t# double check that there are no missing non-merge commits\n>-\t\tmy (@incomplete) = check_cherry_pick(\n>+\t\tmy ($ninc, $ifirst) = check_cherry_pick2(\n> \t\t\t$merge_base, $merge_tip,\n> \t\t\t$parents,\n> \t\t\t@all_ranges,\n> \t\t       );\n> \n>-\t\tif ( @incomplete ) {\n>-\t\t\twarn \"W:svn cherry-pick ignored ($spec) - missing \"\n>-\t\t\t\t.@incomplete.\" commit(s) (eg $incomplete[0])\\n\";\n>+\t\tif ($ninc) {\n>+\t\t\twarn \"W:svn cherry-pick ignored ($spec) - missing \" .\n>+\t\t\t\t\"$ninc commit(s) (eg $ifirst)\\n\";\n> \t\t} else {\n> \t\t\twarn\n> \t\t\t\t\"Found merge parent ($spec): \",\n>-- \n>EW\n"}]}