Re: git-svn performance
- From
- Hin-Tak Leung <htl10@users.sourceforge.net>
- Date
- Oct 22, 2014, 17:38 UTC
- Message-ID
- <1413999510.36832.YahooMailBasic@web172305.mail.ir2.yahoo.com>
------------------------------ On Tue, Oct 21, 2014 10:00 BST Eric Wong wrote:
Show 72 quoted lines
>Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:
>> Yes, but I think you can remove cached_mergeinfo_rev too.
>
>Thanks, pushed the patch at the bottom, too.
>Also started working on some memory reductions here:
> http://mid.gmane.org/20141021033912.GA27462@dcvr.yhbt.net
>But there seem to be more problems :<
>
>----------------------------8<-----------------------------
>From: Eric Wong <normalperson@yhbt.net>
>Date: Tue, 21 Oct 2014 06:23:22 +0000
>Subject: [PATCH] git-svn: remove mergeinfo rev caching
>
>This should further reduce memory usage from the new mergeinfo
>speedups without hurting performance too much, assuming
>reasonable latency to the SVN server.
>
>Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
>Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
>Signed-off-by: Eric Wong <normalperson@yhbt.net>
>---
> perl/Git/SVN.pm | 30 +++++++++---------------------
> 1 file changed, 9 insertions(+), 21 deletions(-)
>
>diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
>index f8a75b1..4364506 100644
>--- a/perl/Git/SVN.pm
>+++ b/perl/Git/SVN.pm
>@@ -1710,32 +1710,20 @@ sub mergeinfo_changes {
> my %minfo = map {split ":", $_ } split "\n", $mergeinfo_prop;
> my $old_minfo = {};
>
>- # Initialize cache on the first call.
>- unless (defined $self->{cached_mergeinfo_rev}) {
>- $self->{cached_mergeinfo_rev} = {};
>- }
>-
>- my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};
>- unless (defined $cached_rev && $cached_rev == $old_rev) {
>- my $ra = $self->ra;
>- # Give up if $old_path isn't in the repo.
>- # This is probably a merge on a subtree.
>- if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {
>- warn "W: ignoring svn:mergeinfo on $old_path, ",
>- "directory didn't exist in r$old_rev\n";
>- return {};
>- }
>- }
>- my (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);
>+ my $ra = $self->ra;
>+ # Give up if $old_path isn't in the repo.
>+ # This is probably a merge on a subtree.
>+ if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {
>+ warn "W: ignoring svn:mergeinfo on $old_path, ",
>+ "directory didn't exist in r$old_rev\n";
>+ return {};
>+ }
>+ my (undef, undef, $props) = $ra->get_dir($old_path, $old_rev);
> if (defined $props->{"svn:mergeinfo"}) {
> my %omi = map {split ":", $_ } split "\n",
> $props->{"svn:mergeinfo"};
> $old_minfo = \%omi;
> }
>- $self->{cached_mergeinfo_rev}{$old_path} = $old_rev;
>-
>- # Cache the new mergeinfo.
>- $self->{cached_mergeinfo_rev}{$path} = $rev;
>
> my %changes = ();
> foreach my $p (keys %minfo) {
>--
>EWI'll have a look at the new changes at some point - I am still keeping the old clone and the new clone and just fetching from time to time to keep them in sync. I just tried that and fetching the same 50 commits on the old clone took 1.7 GB memory vs 1.0 GB memory on the new. Details below. This is just with the 2 earliest patches - I'll put the new 3 in at some point. So I see some needs for retrospectively fixing old clones (maybe as part of garbage collection?), since most would simply use an old clone through the ages...
Comparing trunk of old and new, I see one difference - One short commit message is missing in the *old* (the "Add checkPoFiles etc." part) and so all the sha1 afterwards differed. Is that an old bug that's fixed and therefore I should throw away the old clone?
Date: Wed Apr 25 18:21:29 2012 +0000
Add checkPoFiles etc.
git-svn-id: https://svn.r-project.org/R/trunk@59188 Here is the details of fetching old and new:
<--- $ /usr/bin/time -v git svn fetch --all M doc/manual/R-admin.texi r66784 = fc20374f26f8e03bb88c00933982e29138a6f929 (refs/remotes/trunk) ... M configure r66834 = d8d1876f6aa71b3fe3773cd28a760ff945d30bdf (refs/remotes/R-3-1-branch) Command being timed: "git svn fetch --all" User time (seconds): 1520.77 System time (seconds): 156.32 Percent of CPU this job got: 98% Elapsed (wall clock) time (h:mm:ss or m:ss): 28:15.82 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 1738276 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 613 Minor (reclaiming a frame) page faults: 2039305 Voluntary context switches: 11243 Involuntary context switches: 181507 Swaps: 0 File system inputs: 658328 File system outputs: 754688 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
$ cd ../R-2/ [Hin-Tak@localhost R-2]$ /usr/bin/time -v git svn fetch --all M doc/manual/R-admin.texi r66784 = 6a08d94b456d33d85add914a1b780a972689443a (refs/remotes/trunk) ... M configure r66834 = 370a6484c2a65be78dfae184b50d8f08685d389c (refs/remotes/R-3-1-branch) Command being timed: "git svn fetch --all" User time (seconds): 1507.89 System time (seconds): 134.25 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 27:38.49 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 1026656 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1110 Minor (reclaiming a frame) page faults: 1630150 Voluntary context switches: 10280 Involuntary context switches: 176444 Swaps: 0 File system inputs: 361472 File system outputs: 477912 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 ---->