{"thread":{"id":"37783","subject":"Re: git-svn performance","startedAt":"2014-10-22T17:38:30Z","lastAt":"2014-10-25T00:02:22Z","messageCount":4,"participants":["Hin-Tak Leung","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"250971","messageId":"1413999510.36832.YahooMailBasic@web172305.mail.ir2.yahoo.com","threadId":"37783","inReplyTo":null,"subject":"Re: git-svn performance","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2014-10-22T17:38:30Z","receivedAt":"2014-10-22T17:38:30Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"------------------------------\nOn Tue, Oct 21, 2014 10:00 BST Eric Wong wrote:\n\n>Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:\n>> Yes, but I think you can remove cached_mergeinfo_rev too. \n>\n>Thanks, pushed the patch at the bottom, too.\n>Also started working on some memory reductions here:\n> http://mid.gmane.org/20141021033912.GA27462@dcvr.yhbt.net\n>But there seem to be more problems :<\n>\n>----------------------------8<-----------------------------\n>From: Eric Wong <normalperson@yhbt.net>\n>Date: Tue, 21 Oct 2014 06:23:22 +0000\n>Subject: [PATCH] git-svn: remove mergeinfo rev caching\n>\n>This should further reduce memory usage from the new mergeinfo\n>speedups without hurting performance too much, assuming\n>reasonable latency to the SVN server.\n>\n>Cc: Hin-Tak Leung <htl10@users.sourceforge.net>\n>Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>\n>Signed-off-by: Eric Wong <normalperson@yhbt.net>\n>---\n> perl/Git/SVN.pm | 30 +++++++++---------------------\n> 1 file changed, 9 insertions(+), 21 deletions(-)\n>\n>diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\n>index f8a75b1..4364506 100644\n>--- a/perl/Git/SVN.pm\n>+++ b/perl/Git/SVN.pm\n>@@ -1710,32 +1710,20 @@ sub mergeinfo_changes {\n>     my %minfo = map {split \":\", $_ } split \"\\n\", $mergeinfo_prop;\n>     my $old_minfo = {};\n> \n>-    # Initialize cache on the first call.\n>-    unless (defined $self->{cached_mergeinfo_rev}) {\n>-        $self->{cached_mergeinfo_rev} = {};\n>-    }\n>-\n>-    my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};\n>-    unless (defined $cached_rev && $cached_rev == $old_rev) {\n>-        my $ra = $self->ra;\n>-        # Give up if $old_path isn't in the repo.\n>-        # This is probably a merge on a subtree.\n>-        if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {\n>-            warn \"W: ignoring svn:mergeinfo on $old_path, \",\n>-                \"directory didn't exist in r$old_rev\\n\";\n>-            return {};\n>-        }\n>-    }\n>-    my (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);\n>+    my $ra = $self->ra;\n>+    # Give up if $old_path isn't in the repo.\n>+    # This is probably a merge on a subtree.\n>+    if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {\n>+        warn \"W: ignoring svn:mergeinfo on $old_path, \",\n>+            \"directory didn't exist in r$old_rev\\n\";\n>+        return {};\n>+    }\n>+    my (undef, undef, $props) = $ra->get_dir($old_path, $old_rev);\n>     if (defined $props->{\"svn:mergeinfo\"}) {\n>         my %omi = map {split \":\", $_ } split \"\\n\",\n>             $props->{\"svn:mergeinfo\"};\n>         $old_minfo = \\%omi;\n>     }\n>-    $self->{cached_mergeinfo_rev}{$old_path} = $old_rev;\n>-\n>-    # Cache the new mergeinfo.\n>-    $self->{cached_mergeinfo_rev}{$path} = $rev;\n> \n>     my %changes = ();\n>     foreach my $p (keys %minfo) {\n>-- \n>EW\n\nI'll have a look at the new changes at some point - I am still keeping the old\nclone and the new clone and just fetching from time to time to keep them\nin sync. I just tried that and fetching the same 50 commits on the old clone \ntook 1.7 GB memory vs 1.0 GB memory on the new. Details below.\nThis is just with the 2 earliest patches - I'll put the new 3 in at some point.\nSo I see some needs for retrospectively fixing old clones (maybe as part\nof garbage collection?), since most would simply use an old clone through\nthe ages... \n\nComparing trunk of old and new, I see one difference -  One short\ncommit message is missing in the *old* (the \"Add checkPoFiles etc.\" part)\nand so all the sha1 afterwards differed. Is that an old bug that's fixed\nand therefore I should throw away the old clone? \n\nDate:   Wed Apr 25 18:21:29 2012 +0000\n    Add checkPoFiles etc.\n        git-svn-id: https://svn.r-project.org/R/trunk@59188 \n\nHere is the details of fetching old and new:\n\n<---\n$ /usr/bin/time -v git svn fetch --all\n\tM\tdoc/manual/R-admin.texi\nr66784 = fc20374f26f8e03bb88c00933982e29138a6f929 (refs/remotes/trunk)\n...\n\tM\tconfigure\nr66834 = d8d1876f6aa71b3fe3773cd28a760ff945d30bdf (refs/remotes/R-3-1-branch)\n\tCommand being timed: \"git svn fetch --all\"\n\tUser time (seconds): 1520.77\n\tSystem time (seconds): 156.32\n\tPercent of CPU this job got: 98%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 28:15.82\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 1738276\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 613\n\tMinor (reclaiming a frame) page faults: 2039305\n\tVoluntary context switches: 11243\n\tInvoluntary context switches: 181507\n\tSwaps: 0\n\tFile system inputs: 658328\n\tFile system outputs: 754688\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n\n$ cd ../R-2/\n[Hin-Tak@localhost R-2]$ /usr/bin/time -v git svn fetch --all\n\tM\tdoc/manual/R-admin.texi\nr66784 = 6a08d94b456d33d85add914a1b780a972689443a (refs/remotes/trunk)\n...\n\tM\tconfigure\nr66834 = 370a6484c2a65be78dfae184b50d8f08685d389c (refs/remotes/R-3-1-branch)\n\tCommand being timed: \"git svn fetch --all\"\n\tUser time (seconds): 1507.89\n\tSystem time (seconds): 134.25\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 27:38.49\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 1026656\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 1110\n\tMinor (reclaiming a frame) page faults: 1630150\n\tVoluntary context switches: 10280\n\tInvoluntary context switches: 176444\n\tSwaps: 0\n\tFile system inputs: 361472\n\tFile system outputs: 477912\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n---->\n"},{"id":"251007","messageId":"1414134386.28852.YahooMailBasic@web172306.mail.ir2.yahoo.com","threadId":"37783","inReplyTo":"1413999510.36832.YahooMailBasic@web172305.mail.ir2.yahoo.com","subject":"Anomaly with the new code - Re: git-svn performance","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2014-10-24T07:06:26Z","receivedAt":"2014-10-24T07:06:26Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"I keep tabs of a particular svn repository over many years\nand run \"git svn fetch --all\" every few days. So that's the old clone.\nSince this discussion started, I made a new one with git 2.1.0 patched\nwith the first two patches below, a couple of weeks ago. And I ran\n'git svn fetch --all' on both every few days since.\n\nI have added a few more patches, so the whole list is the 6\nbelow against 2.1.0. The latest fetch is really strange - the fetch against\nthe new clone took almost twice as long and uses almost twice\nas much memory, vs against the old. 17 min, 800 MB vs 10 min 400MB.\nDetails below. Maybe this is a performance issue about how the clones\nwere made?\n\n0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch    \n0002-git-svn-only-look-at-the-root-path-for-svn-mergeinfo.patch   \n0003-git-svn-reduce-check_cherry_pick-cache-overhead.patch        \n0004-git-svn-cache-only-mergeinfo-revisions.patch                 \n0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch   \n0007-git-svn-remove-mergeinfo-rev-caching.patch \n\n(I dropped #5 because it doesn't seem interesting?)\n\n<---\n$ /usr/bin/time -v git svn fetch --all\n...\n\tCommand being timed: \"git svn fetch --all\"\n\tUser time (seconds): 622.20\n\tSystem time (seconds): 12.52\n\tPercent of CPU this job got: 98%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 10:42.21\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 399588\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 320\n\tMinor (reclaiming a frame) page faults: 383987\n\tVoluntary context switches: 2088\n\tInvoluntary context switches: 68304\n\tSwaps: 0\n\tFile system inputs: 168288\n\tFile system outputs: 148960\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n[Hin-Tak@localhost R]$ cd ../R-2/\n[Hin-Tak@localhost R-2]$ /usr/bin/time -v git svn fetch --all\n\tM\tsrc/library/stats/R/hclust.R\n\tM\tsrc/library/stats/R/dendrogram.R\nr66853 = 7c18b2e4084529d5912cf789c045f2eab7d4083c (refs/remotes/trunk)\n\tM\tdoc/manual/R-exts.texi\nr66854 = bc7b131e34eaf04859fede1ecedb796c0a33be02 (refs/remotes/trunk)\n\tM\tdoc/manual/R-exts.texi\nChecking svn:mergeinfo changes since r66844: 6 sources, 1 changed\nW:svn cherry-pick ignored (/trunk:66824,66854) - missing 1084 commit(s) (eg 6453a2d844e27f2963ba87142028b023c50385ef)\nr66855 = de5daf8db948732fa96c3d5b32077d8057e2a7e7 (refs/remotes/R-3-1-branch)\n\tM\tsrc/modules/internet/internet.c\nr66856 = a1e9300c6dd49ec4c3dd11f861bca0dbe3ca65b4 (refs/remotes/trunk)\n\tM\tdoc/manual/R-admin.texi\nr66857 = eb5f3175e67a806482c39def71246f5d18bf8660 (refs/remotes/trunk)\n\tM\tdoc/manual/R-admin.texi\nChecking svn:mergeinfo changes since r66855: 6 sources, 1 changed\nW:svn cherry-pick ignored (/trunk:66854,66857) - missing 1086 commit(s) (eg e8cc0c31ddeeea3f8fa1ad47105d09a2c19e1a98)\nr66858 = 10c8013f103d57c8a717b738e2a51c8d397c88f0 (refs/remotes/R-3-1-branch)\n\tM\tVERSION\nr66859 = 0f865f247da3191431bb17bcc3c307e8735dbd97 (refs/remotes/R-3-1-branch)\n\tCommand being timed: \"git svn fetch --all\"\n\tUser time (seconds): 1023.06\n\tSystem time (seconds): 15.30\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 17:27.65\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 785332\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 884\n\tMinor (reclaiming a frame) page faults: 527668\n\tVoluntary context switches: 2792\n\tInvoluntary context switches: 107718\n\tSwaps: 0\n\tFile system inputs: 194704\n\tFile system outputs: 170032\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n\n--->\n"},{"id":"251033","messageId":"20141024233411.GA18655@dcvr.yhbt.net","threadId":"37783","inReplyTo":"1414134386.28852.YahooMailBasic@web172306.mail.ir2.yahoo.com","subject":"Re: Anomaly with the new code - Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-24T23:34:11Z","receivedAt":"2014-10-24T23:34:11Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hin-Tak Leung <htl10@users.sourceforge.net> wrote:\n> I keep tabs of a particular svn repository over many years\n> and run \"git svn fetch --all\" every few days. So that's the old clone.\n> Since this discussion started, I made a new one with git 2.1.0 patched\n> with the first two patches below, a couple of weeks ago. And I ran\n> 'git svn fetch --all' on both every few days since.\n> \n> I have added a few more patches, so the whole list is the 6\n> below against 2.1.0. The latest fetch is really strange - the fetch against\n> the new clone took almost twice as long and uses almost twice\n> as much memory, vs against the old. 17 min, 800 MB vs 10 min 400MB.\n> Details below. Maybe this is a performance issue about how the clones\n> were made?\n\nMemory usage seems to grow with the amount of revisions fetched,\nsee below.  And higher memory means slower fork() on Linux systems.\n\n> 0001-git-svn-only-look-at-the-new-parts-of-svn-mergeinfo.patch    \n> 0002-git-svn-only-look-at-the-root-path-for-svn-mergeinfo.patch   \n> 0003-git-svn-reduce-check_cherry_pick-cache-overhead.patch        \n> 0004-git-svn-cache-only-mergeinfo-revisions.patch                 \n\n> 0006-git-svn-clear-global-SVN-pool-between-get_log-invoca.patch   \n\n0006 is insufficient and incompatible with older SVN.\nI pushed \"git-svn: reload RA every log-window-size\"\n(commit dfa72fdb96befbd790f623bb2909a347176753c2) instead\nwhich saves much more memory:\n\nhttp://mid.gmane.org/20141024225352.GB31716@dcvr.yhbt.net\n\nBut there still seems to be some slow growth with many revisions\nwhich is not mergeinfo-related.\n\n> 0007-git-svn-remove-mergeinfo-rev-caching.patch \n\nI think it is also safe to remove the _rev_list memoization since\nit uses a lot of memory.  The remaining caches should be tiny\n(but useful, I think).\n"},{"id":"251034","messageId":"20141025000222.GB18655@dcvr.yhbt.net","threadId":"37783","inReplyTo":"1413999510.36832.YahooMailBasic@web172305.mail.ir2.yahoo.com","subject":"Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-25T00:02:22Z","receivedAt":"2014-10-25T00:02:22Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hin-Tak Leung <htl10@users.sourceforge.net> wrote:\n> Comparing trunk of old and new, I see one difference -  One short\n> commit message is missing in the *old* (the \"Add checkPoFiles etc.\" part)\n> and so all the sha1 afterwards differed. Is that an old bug that's fixed\n> and therefore I should throw away the old clone? \n\nI don't recall a bug which would cause a revision to be skipped.\nI suppose it's alright now the new clone has that revision.\nPerhaps there was a power outage or improper shutdown?\n\nAt least we can be glad current versions see this revision...\n"}]}