{"thread":{"id":"37770","subject":"[PATCH] git-svn: clear global SVN pool between get_log invocations","startedAt":"2014-10-21T03:39:12Z","lastAt":"2014-10-24T22:53:52Z","messageCount":3,"participants":["Eric Wong"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"250884","messageId":"20141021033912.GA27462@dcvr.yhbt.net","threadId":"37770","inReplyTo":null,"subject":"[PATCH] git-svn: clear global SVN pool between get_log invocations","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-21T03:39:12Z","receivedAt":"2014-10-21T03:39:12Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"During long fetches with many revisions, some SVN functions are not\nalways using local pools and some gets into the global pool. Ensure\nwe free that memory occasionally between SVN::RA::get_log calls to\nreduce memory growth.\n\nThis reduces memory usage over the course of fetching 10K revisions\nusing a test repository created with the script at the end of this\ncommit message.\n\nAs reported by time(1) on my x86-64 system:\n\n\tbefore: 54024k\n\t after: 47344k\n\n-----------------------------8<------------------------------\nset -e\ntmp=$(mktemp -d svntestrepo-XXXXXXXX)\nsvnadmin create \"$tmp\"\nrepo=file://\"$(cd $tmp && pwd)\"\nsvn co \"$repo\" \"$tmp/wd\"\ncd \"$tmp/wd\"\nif ! test -f a\nthen\n\t> a\n\tsvn add a\n\tsvn commit -m 'A'\nfi\n\nnr=10000\nwhile test $nr -gt 0\ndo\n\techo $nr > a\n\tsvn commit -q -m A\n\tnr=$((nr - 1))\ndone\necho \"repository created in $repo\"\n-----------------------------8<------------------------------\n\nSigned-off-by: Eric Wong \n---\n Pushed to master of git://bogomips.org/git-svn\n\n However, memory usage still seems to grow infinitely even in this simple\n fetch case, so are other memory leaks.\n\n perl/Git/SVN/Ra.pm | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\nindex a7b0119..3df4591 100644\n--- a/perl/Git/SVN/Ra.pm\n+++ b/perl/Git/SVN/Ra.pm\n@@ -237,10 +237,6 @@ sub trees_match {\n \tmy $ctx = SVN::Client->new(auth => _auth_providers);\n \tmy $out = IO::File->new_tmpfile;\n \n-\t# older SVN (1.1.x) doesn't take $pool as the last parameter for\n-\t# $ctx->diff(), so we'll create a default one\n-\tmy $pool = SVN::Pool->new_default_sub;\n-\n \t$ra_invalid = 1; # this will open a new SVN::Ra connection to $url1\n \t$ctx->diff([], $url1, $rev1, $url2, $rev2, 1, 1, 0, $out, $out);\n \t$out->flush;\n@@ -453,6 +449,7 @@ sub gs_fetch_loop_common {\n \t\t\t\t$_[0] = undef;\n \t\t\t\t$self = undef;\n \t\t\t\t$RA = undef;\n+\t\t\t\tSVN::Core->gpool->clear;\n \t\t\t\t$self = Git::SVN::Ra->new($ra_url);\n \t\t\t\t$ra_invalid = undef;\n \t\t\t}\n@@ -468,6 +465,7 @@ sub gs_fetch_loop_common {\n \t\t\tmy $k = \"svn-remote.$g->{remote}.$g->{t}-maxRev\";\n \t\t\tGit::SVN::tmp_config($k, $max);\n \t\t}\n+\t\tSVN::Core->gpool->clear;\n \t\tlast if $max >= $head;\n \t\t$min = $max + 1;\n \t\t$max += $inc;\n-- \nEW\n\n\n-- \nEW\n"},{"id":"251031","messageId":"20141024225012.GA31716@dcvr.yhbt.net","threadId":"37770","inReplyTo":"20141021033912.GA27462@dcvr.yhbt.net","subject":"Re: [PATCH] git-svn: clear global SVN pool between get_log invocations","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-24T22:50:12Z","receivedAt":"2014-10-24T22:50:12Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong wrote:\n> +\t\t\t\tSVN::Core->gpool->clear;\n\nUnfortunately, SVN::Core->gpool is not available in older SVN,\nbut I'm cooking a better patch which saves even more memory.\n"},{"id":"251032","messageId":"20141024225352.GB31716@dcvr.yhbt.net","threadId":"37770","inReplyTo":"20141024225012.GA31716@dcvr.yhbt.net","subject":"[PATCH] git-svn: reload RA every log-window-size","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-24T22:53:52Z","receivedAt":"2014-10-24T22:53:52Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Despite attempting to use local memory pools everywhere we can,\n(including our call to SVN::Ra::do_update and all subsequent reporter\ncalls), there does not appear to be a way to force the Git::SVN::Fetcher\ncallbacks to use a pool other than the per-SVN::Ra pool.\nGit::SVN::Fetcher ends up using the main RA pool which grows\nmonotonically in size for the lifetime of the RA object.\n\nThus the only way to free that memory appears to be to destroy and\nrecreate the RA connection for at every --log-window-size interval.\n\nThis reduces memory usage over the course of fetching 10K revisions\nusing a test repository created with the script at the end of this\ncommit message.\n\nAs reported by time(1) on my x86-64 system:\n\n\tbefore: 54024k\n\t after: 28680k\n\nUnfortunately, there remains some yet-to-be-tracked-down slow memory\ngrowth which would be evident as the `nr' parameter increases in\nthe repository generation script:\n-----------------------------8<------------------------------\nset -e\ntmp=$(mktemp -d svntestrepo-XXXXXXXX)\nsvnadmin create \"$tmp\"\nrepo=file://\"$(cd $tmp && pwd)\"\nsvn co \"$repo\" \"$tmp/wd\"\ncd \"$tmp/wd\"\nif ! test -f a\nthen\n\t> a\n\tsvn add a\n\tsvn commit -m 'A'\nfi\n\nnr=10000\nwhile test $nr -gt 0\ndo\n\techo $nr > a\n\tsvn commit -q -m A\n\tnr=$((nr - 1))\ndone\necho \"repository created in $repo\"\n-----------------------------8<------------------------------\n\nSigned-off-by: Eric Wong \n---\n Eric Wong wrote:\n > Eric Wong wrote:\n > > +\t\t\t\tSVN::Core->gpool->clear;\n > \n > Unfortunately, SVN::Core->gpool is not available in older SVN,\n > but I'm cooking a better patch which saves even more memory.\n\n perl/Git/SVN/Ra.pm | 21 +++++++++++++--------\n 1 file changed, 13 insertions(+), 8 deletions(-)\n\ndiff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\nindex a7b0119..1828519 100644\n--- a/perl/Git/SVN/Ra.pm\n+++ b/perl/Git/SVN/Ra.pm\n@@ -376,10 +376,19 @@ sub longest_common_path {\n sub gs_fetch_loop_common {\n \tmy ($self, $base, $head, $gsv, $globs) = @_;\n \treturn if ($base > $head);\n+\tmy $gpool = SVN::Pool->new_default;\n+\tmy $ra_url = $self->url;\n+\tmy $reload_ra = sub {\n+\t\t$_[0] = undef;\n+\t\t$self = undef;\n+\t\t$RA = undef;\n+\t\t$gpool->clear;\n+\t\t$self = Git::SVN::Ra->new($ra_url);\n+\t\t$ra_invalid = undef;\n+\t};\n \tmy $inc = $_log_window_size;\n \tmy ($min, $max) = ($base, $head < $base + $inc ? $head : $base + $inc);\n \tmy $longest_path = longest_common_path($gsv, $globs);\n-\tmy $ra_url = $self->url;\n \tmy $find_trailing_edge;\n \twhile (1) {\n \t\tmy %revs;\n@@ -449,13 +458,7 @@ sub gs_fetch_loop_common {\n \t\t\t\t \"$g->{t}-maxRev\";\n \t\t\t\tGit::SVN::tmp_config($k, $r);\n \t\t\t}\n-\t\t\tif ($ra_invalid) {\n-\t\t\t\t$_[0] = undef;\n-\t\t\t\t$self = undef;\n-\t\t\t\t$RA = undef;\n-\t\t\t\t$self = Git::SVN::Ra->new($ra_url);\n-\t\t\t\t$ra_invalid = undef;\n-\t\t\t}\n+\t\t\t$reload_ra->() if $ra_invalid;\n \t\t}\n \t\t# pre-fill the .rev_db since it'll eventually get filled in\n \t\t# with '0' x40 if something new gets committed\n@@ -472,6 +475,8 @@ sub gs_fetch_loop_common {\n \t\t$min = $max + 1;\n \t\t$max += $inc;\n \t\t$max = $head if ($max > $head);\n+\n+\t\t$reload_ra->();\n \t}\n \tGit::SVN::gc();\n }\n-- \nEW\n"}]}