[PATCH] git-svn: clear global SVN pool between get_log invocations

Eric Wong· Oct 21, 2014, 03:39 UTC · lore

During long fetches with many revisions, some SVN functions are not always using local pools and some gets into the global pool. Ensure we free that memory occasionally between SVN::RA::get_log calls to reduce memory growth.

This reduces memory usage over the course of fetching 10K revisions using a test repository created with the script at the end of this commit message.

As reported by time(1) on my x86-64 system:

	before: 54024k
	 after: 47344k

-----------------------------8<------------------------------
set -e
tmp=$(mktemp -d svntestrepo-XXXXXXXX)
svnadmin create "$tmp"
repo=file://"$(cd $tmp && pwd)"
svn co "$repo" "$tmp/wd"
cd "$tmp/wd"
if ! test -f a
then
	> a
	svn add a
	svn commit -m 'A'
fi

nr=10000
while test $nr -gt 0
do
	echo $nr > a
	svn commit -q -m A
	nr=$((nr - 1))
done
echo "repository created in $repo"
-----------------------------8<------------------------------

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 Pushed to master of git://bogomips.org/git-svn

 However, memory usage still seems to grow infinitely even in this simple
 fetch case, so are other memory leaks.

 perl/Git/SVN/Ra.pm | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

Show changes to perl/Git/SVN/Ra.pm +2 −4

diff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm
index a7b0119..3df4591 100644
--- a/perl/Git/SVN/Ra.pm
+++ b/perl/Git/SVN/Ra.pm
@@ -237,10 +237,6 @@ sub trees_match {
 	my $ctx = SVN::Client->new(auth => _auth_providers);
 	my $out = IO::File->new_tmpfile;
 
-	# older SVN (1.1.x) doesn't take $pool as the last parameter for
-	# $ctx->diff(), so we'll create a default one
-	my $pool = SVN::Pool->new_default_sub;
-
 	$ra_invalid = 1; # this will open a new SVN::Ra connection to $url1
 	$ctx->diff([], $url1, $rev1, $url2, $rev2, 1, 1, 0, $out, $out);
 	$out->flush;
@@ -453,6 +449,7 @@ sub gs_fetch_loop_common {
 				$_[0] = undef;
 				$self = undef;
 				$RA = undef;
+				SVN::Core->gpool->clear;
 				$self = Git::SVN::Ra->new($ra_url);
 				$ra_invalid = undef;
 			}
@@ -468,6 +465,7 @@ sub gs_fetch_loop_common {
 			my $k = "svn-remote.$g->{remote}.$g->{t}-maxRev";
 			Git::SVN::tmp_config($k, $max);
 		}
+		SVN::Core->gpool->clear;
 		last if $max >= $head;
 		$min = $max + 1;
 		$max += $inc;

-- 
EW


-- 
EW

Eric Wong· Oct 24, 2014, 22:50 UTC · re: Eric Wong · lore

Re: [PATCH] git-svn: clear global SVN pool between get_log invocations

Eric Wong <normalperson@yhbt.net> wrote:

> +				SVN::Core->gpool->clear;

Unfortunately, SVN::Core->gpool is not available in older SVN, but I'm cooking a better patch which saves even more memory.

Eric Wong· Oct 24, 2014, 22:53 UTC · re: Eric Wong · lore

[PATCH] git-svn: reload RA every log-window-size

Despite attempting to use local memory pools everywhere we can, (including our call to SVN::Ra::do_update and all subsequent reporter calls), there does not appear to be a way to force the Git::SVN::Fetcher callbacks to use a pool other than the per-SVN::Ra pool. Git::SVN::Fetcher ends up using the main RA pool which grows monotonically in size for the lifetime of the RA object.

Thus the only way to free that memory appears to be to destroy and recreate the RA connection for at every --log-window-size interval.

This reduces memory usage over the course of fetching 10K revisions using a test repository created with the script at the end of this commit message.

As reported by time(1) on my x86-64 system:

	before: 54024k
	 after: 28680k

Unfortunately, there remains some yet-to-be-tracked-down slow memory
growth which would be evident as the `nr' parameter increases in
the repository generation script:
-----------------------------8<------------------------------
set -e
tmp=$(mktemp -d svntestrepo-XXXXXXXX)
svnadmin create "$tmp"
repo=file://"$(cd $tmp && pwd)"
svn co "$repo" "$tmp/wd"
cd "$tmp/wd"
if ! test -f a
then
	> a
	svn add a
	svn commit -m 'A'
fi

nr=10000
while test $nr -gt 0
do
	echo $nr > a
	svn commit -q -m A
	nr=$((nr - 1))
done
echo "repository created in $repo"
-----------------------------8<------------------------------

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
  Eric Wong <normalperson@yhbt.net> wrote:
  > Eric Wong <normalperson@yhbt.net> wrote:
  > > +				SVN::Core->gpool->clear;
  > 
  > Unfortunately, SVN::Core->gpool is not available in older SVN,
  > but I'm cooking a better patch which saves even more memory.

 perl/Git/SVN/Ra.pm | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

Show changes to perl/Git/SVN/Ra.pm +13 −8

diff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm
index a7b0119..1828519 100644
--- a/perl/Git/SVN/Ra.pm
+++ b/perl/Git/SVN/Ra.pm
@@ -376,10 +376,19 @@ sub longest_common_path {
 sub gs_fetch_loop_common {
 	my ($self, $base, $head, $gsv, $globs) = @_;
 	return if ($base > $head);
+	my $gpool = SVN::Pool->new_default;
+	my $ra_url = $self->url;
+	my $reload_ra = sub {
+		$_[0] = undef;
+		$self = undef;
+		$RA = undef;
+		$gpool->clear;
+		$self = Git::SVN::Ra->new($ra_url);
+		$ra_invalid = undef;
+	};
 	my $inc = $_log_window_size;
 	my ($min, $max) = ($base, $head < $base + $inc ? $head : $base + $inc);
 	my $longest_path = longest_common_path($gsv, $globs);
-	my $ra_url = $self->url;
 	my $find_trailing_edge;
 	while (1) {
 		my %revs;
@@ -449,13 +458,7 @@ sub gs_fetch_loop_common {
 				        "$g->{t}-maxRev";
 				Git::SVN::tmp_config($k, $r);
 			}
-			if ($ra_invalid) {
-				$_[0] = undef;
-				$self = undef;
-				$RA = undef;
-				$self = Git::SVN::Ra->new($ra_url);
-				$ra_invalid = undef;
-			}
+			$reload_ra->() if $ra_invalid;
 		}
 		# pre-fill the .rev_db since it'll eventually get filled in
 		# with '0' x40 if something new gets committed
@@ -472,6 +475,8 @@ sub gs_fetch_loop_common {
 		$min = $max + 1;
 		$max += $inc;
 		$max = $head if ($max > $head);
+
+		$reload_ra->();
 	}
 	Git::SVN::gc();
 }

-- 
EW

patchgit-svn: clear global SVN pool between get_log invocations