threads / discuss / 37752

git-svn performance

Subject: git-svn performance

## tl;dr

9 messages between Oct 17, 2014 and Oct 21, 2014.

replies: 8people: 3as markdown or json

Fabian Schmied· Oct 17, 2014, 20:47 UTC · lore
Hi,

I'm currently migrating an SVN repository to Git using git-svn (Git for Windows 1.8.3-preview20130601), and I'm experiencing severe performance problems with "git svn fetch". Commits to the SVN "trunk" are fetched very fast (a few seconds or so per SVN revision), but commits to some branches ("hotfix" branches) are currently taking about 9 minutes per revision. I fear that the time per these commits is increasing and that indeed the migration might not be finishable at all.

For the commits that take such a long time, git-svn always outputs lots of warnings about ignored SVN cherry-picks, and it tells me it can't find a revmap for the path being imported. (See [1].)

AFAICS, the offending commits take place on some branches that include a lot of manually merged ("SVN cherry-picked") revisions. Git-svn seems to be checking something (though I don't know what) that makes importing these revisions really slow. And it repeats this for every revision on these branches with increasing work to do.

Is there anything I can do to speed this up? (I already tried increasing the --log-window-size to 500, didn't have any effect.)

Thank you, best regards, Fabian

[1]
        M       foo/bar/XXX.xml
        M       foo/bar/YYY.xml
W:svn cherry-pick ignored (/branches/frob:6940-7068) - missing 12
commit(s) (eg abeaece820ceae44ebf2c06011cf43bbcbf4b1ce)
W:svn cherry-pick ignored (/branches/feature:3316-4798,4811,4827) -
missing 10 commit(s) (eg e255fff14ab1e581f21671ca8b36c0747869cf8c)
W:svn cherry-pick ignored
(/hotfixes/ZZZ.159:2131,2133,2145-2146,2148,2169) - missing 10
commit(s) (eg e04b0326c998f0611c18144b3ed8f686d3b52f4c)
W:svn cherry-pick ignored
(/hotfixes/ZZZ.333:4536,4610-4611,4625,4665,4669,4685,4713,4745,4785,4788,4908-4917,4920,4933-4944,4955,5003,5103,5174,5222,5227,
5261,5267,5306,5310,5321,5360,5416,5467,5501,5508,5599-5614,5650-5651,5757,5761-5762,5764,5778-5779,5784,5811,5814,5819,5823,5825,5836-5838,5860,5862,5873,5889,
5910,5924,5948) - missing 137 commit(s) (eg
9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)
C:\Program Files (x86)\Git\bin\perl.exe: *** WFSO timed out
W:svn cherry-pick ignored (/hotfixes/ZZZ.333.39:5696,5847) - missing
84 commit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)
W:svn cherry-pick ignored (/hotfixes/AAA:5905,6095) - missing 119
commit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)
W:svn cherry-pick ignored (/hotfixes/BBB_1.1:6971) - missing 198
commit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)
W:svn cherry-pick ignored
(/hotfixes/CCC:6134,6164,6168,6174,6206,6211,6237,6239,6244-6245,6250,6257,6269,6271,6276,6289-6292,6294,6296,6301-6302,6313,6315-6316,6329,6333,6379,6383,6394,6405,6411,6456,6478,6483,6491,6519,6537,6557)
- missing 194 commit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)
W:svn cherry-pick ignored (/hotfixes/DDD:7635) - missing 1 commit(s)
(eg 6a3ba817635eb3a9411a307924dec393311d93be)
W:svn cherry-pick ignored
(/hotfixes/EEE_1.2:7786,7794,7797,7803,7829-7830,7843,7886,7889,7933,7937,7949,7953)
- missing 80 commit(s) (eg e78b1bc68f7a9b041588a39f3fa5e1a61f98942b)
W:svn cherry-pick ignored
(/hotfixes/EEE_1.3:8159,8170,8173-8174,8177,8181-8182,8185,8187,8194-8195,8201,8203,8206,8251,8255,8257,8259-8262,8265,8280,8286,8294,8296,8304-8305,8312,8318,8323,8327,8363,8387-8388,8390,8422-8423,8432,8446,8536-8537,8548-8549,8556,8559,8566,8569,8572,8578,8597-8598,8602,8617,8619,8655,8687,8720)
- missing 104 commit(s) (eg 33febd4591f42a9d871ba330432840917b157f9e)
W:svn cherry-pick ignored
(/hotfixes/EEE_1.4:8766,8768,8770,8777-8779,8795-8796,8802-8809,8812-8814,8816-8817,8820,8823,8825,8827,8831,8836,8841,8845,8848-8852,8854-8855,8866,8868-8869,8871-8873,8875-8878,8880,8888,8892,8911-8912,8917-8918,8946,8956-8957,8964,8984,8994,9003,9008,9011,9029,9038,9040,9046-9048,9055,9086,9101,9108,9111,9113,9124,9129,9133,9138-9139,9150,9152,9154,9156,9172,9174,9188-9189,9208,9211,9217)
- missing 44 commit(s) (eg 0621fb44de682650d762c707b102bc2472c088f8)
W:svn cherry-pick ignored
(/hotfixes/EEE_1.5:9412,9421,9430,9433-9436,9439,9441,9449,9459,9468,9529,9548,9561,9568,9605-9606,9612,9614,9617,9628,9630-9631,9637,9687,9807)
- missing 41 commit(s) (eg 1bd1a9b72336bf4d3839a00348b7f2a52368c16c)
W:svn cherry-pick ignored
(/trunk:9852-9853,9857,9859,9862,9868,9872,9876,9879,9890,9895,9926-9927,9933,9953,9956,9960-9962)
- missing 60 commit(s) (eg 3322e7ffc6ab49181976d9e94c91a4556951f38a)
Couldn't find revmap for https://the-svn-server/svn/something/trunk/foo
r9963 = 597df48cb830825f9029d1cfdf45df024d7fd3dd (refs/remotes/EEE_1.6)
Eric Wong· Oct 19, 2014, 00:32 UTC · re: Fabian Schmied · lore

Re: git-svn performance

Fabian Schmied <fabian.schmied@gmail.com> wrote:
Show 23 quoted lines
> Hi,
> 
> I'm currently migrating an SVN repository to Git using git-svn (Git
> for Windows 1.8.3-preview20130601), and I'm experiencing severe
> performance problems with "git svn fetch". Commits to the SVN "trunk"
> are fetched very fast (a few seconds or so per SVN revision), but
> commits to some branches ("hotfix" branches) are currently taking
> about 9 minutes per revision. I fear that the time per these commits
> is increasing and that indeed the migration might not be finishable at
> all.
> 
> For the commits that take such a long time, git-svn always outputs
> lots of warnings about ignored SVN cherry-picks, and it tells me it
> can't find a revmap for the path being imported. (See [1].)
> 
> AFAICS, the offending commits take place on some branches that include
> a lot of manually merged ("SVN cherry-picked") revisions. Git-svn
> seems to be checking something (though I don't know what) that makes
> importing these revisions really slow. And it repeats this for every
> revision on these branches with increasing work to do.
> 
> Is there anything I can do to speed this up? (I already tried
> increasing the --log-window-size to 500, didn't have any effect.)

Can you take a look at the following two "mergeinfo-speedups" in my repo? (git://bogomips.org/git-svn)

Jakob Stoklund Olesen (2):
      git-svn: only look at the new parts of svn:mergeinfo
      git-svn: only look at the root path for svn:mergeinfo
Also downloadable here:

http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535 http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a

Hin-Tak (Cc-ed) reported good improvements with them, but also a large memory increase:

http://mid.gmane.org/1412706046.90413.YahooMailBasic@web172303.mail.ir2.yahoo.com

Jakob (or anybody else): I suppose we could tie the new cached_mergeinfo* caches to disk-backed storage to avoid the memory bloat.

Eric Wong· Oct 19, 2014, 02:29 UTC · re: Eric Wong · lore

Re: git-svn performance

Eric Wong <normalperson@yhbt.net> wrote:
> Hin-Tak (Cc-ed) reported good improvements with them, but also
> a large memory increase:
This might reduce the pathname and internal hash overheads:
------------------------8<-----------------------
From: Eric Wong <normalperson@yhbt.net>
Date: Sun, 19 Oct 2014 02:26:53 +0000
Subject: [PATCH] git-svn: simplify cached_mergeinfo layout

This reduces hash lookups for looking up cache data and will simplify tying data to disk in the next commit.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 perl/Git/SVN.pm | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
index b1a84d0..25dbcd5 100644
--- a/perl/Git/SVN.pm
+++ b/perl/Git/SVN.pm
@@ -1708,15 +1708,17 @@ sub mergeinfo_changes {
 	my %minfo = map {split ":", $_ } split "\n", $mergeinfo_prop;
 	my $old_minfo = {};
 
+	# layout: $path => [ $rev, \%mergeinfo ]
+	my $cached_mergeinfo = $self->{cached_mergeinfo};
+
 	# Initialize cache on the first call.
-	unless (defined $self->{cached_mergeinfo_rev}) {
-		$self->{cached_mergeinfo_rev} = {};
-		$self->{cached_mergeinfo} = {};
+	unless (defined $cached_mergeinfo) {
+		$cached_mergeinfo = $self->{cached_mergeinfo} = {};
 	}
 
-	my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};
-	if (defined $cached_rev && $cached_rev == $old_rev) {
-		$old_minfo = $self->{cached_mergeinfo}{$old_path};
+	my $cached = $cached_mergeinfo->{$old_path};
+	if (defined $cached && $cached->[0] == $old_rev) {
+		$old_minfo = $cached->[1];
 	} else {
 		my $ra = $self->ra;
 		# Give up if $old_path isn't in the repo.
@@ -1733,13 +1735,11 @@ sub mergeinfo_changes {
 				$props->{"svn:mergeinfo"};
 			$old_minfo = \%omi;
 		}
-		$self->{cached_mergeinfo}{$old_path} = $old_minfo;
-		$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;
+		$cached_mergeinfo->{$old_path} = [ $old_rev, $old_minfo ];
 	}
 
 	# Cache the new mergeinfo.
-	$self->{cached_mergeinfo}{$path} = \%minfo;
-	$self->{cached_mergeinfo_rev}{$path} = $rev;
+	$cached_mergeinfo->{$path} = [ $rev, \%minfo ];
 
 	my %changes = ();
 	foreach my $p (keys %minfo) {

-- 
EW
Eric Wong· Oct 19, 2014, 02:33 UTC · re: Eric Wong · lore

Re: git-svn performance

Eric Wong <normalperson@yhbt.net> wrote:
> This reduces hash lookups for looking up cache data and will
> simplify tying data to disk in the next commit.

I considered the following, but GDBM might not be readily available on non-POSIX platforms. I think the other problem is the existing caches are still in memory (whether YAML or Storable) even if disk-backed, causing a large amount of memory usage anyways.

(Both patches on top of Jakob's)
-------------------------
Subject: [RFC] git-svn: tie cached_mergeinfo to a GDBM_File store

This should reduce per-instance memory usage by allowing serialization to disk. Using the existing Memoize::Storable or YAML backends does not allow fast lookups.

GDBM_File should be available in most Perl installations
and should not pose unnecessary burden
---
 perl/Git/SVN.pm | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
index 25dbcd5..3e477c7 100644
--- a/perl/Git/SVN.pm
+++ b/perl/Git/SVN.pm
@@ -14,6 +14,7 @@ use IPC::Open3;
 use Memoize;  # core since 5.8.0, Jul 2002
 use Memoize::Storable;
 use POSIX qw(:signal_h);
+use Storable qw(freeze thaw);
 
 use Git qw(
     command
@@ -1713,10 +1714,21 @@ sub mergeinfo_changes {
 
 	# Initialize cache on the first call.
 	unless (defined $cached_mergeinfo) {
-		$cached_mergeinfo = $self->{cached_mergeinfo} = {};
+		my %hash;
+		eval '
+		require File::Temp;
+		use GDBM_File;
+		my $fh = File::Temp->new(TEMPLATE => "mergeinfo.XXXXXXXX");
+		$self->{cached_mergeinfo_fh} = $fh;
+		$fh->unlink_on_destroy(1);
+		tie %hash => "GDBM_File", $fh->filename, GDBM_WRCREAT, 0600;
+		';
+		$cached_mergeinfo = $self->{cached_mergeinfo} = \%hash;
 	}
 
 	my $cached = $cached_mergeinfo->{$old_path};
+	$cached = thaw($cached) if defined $cached;
+
 	if (defined $cached && $cached->[0] == $old_rev) {
 		$old_minfo = $cached->[1];
 	} else {
@@ -1735,11 +1747,12 @@ sub mergeinfo_changes {
 				$props->{"svn:mergeinfo"};
 			$old_minfo = \%omi;
 		}
-		$cached_mergeinfo->{$old_path} = [ $old_rev, $old_minfo ];
+		$cached_mergeinfo->{$old_path} =
+					freeze([ $old_rev, $old_minfo ]);
 	}
 
 	# Cache the new mergeinfo.
-	$cached_mergeinfo->{$path} = [ $rev, \%minfo ];
+	$cached_mergeinfo->{$path} = freeze([ $rev, \%minfo ]);
 
 	my %changes = ();
 	foreach my $p (keys %minfo) {

-- 
EW
Jakob Stoklund Olesen· Oct 19, 2014, 14:56 UTC · re: Eric Wong · lore

Re: git-svn performance

Show 10 quoted lines
> On Oct 18, 2014, at 19:33, Eric Wong <normalperson@yhbt.net> wrote:
> 
> Eric Wong <normalperson@yhbt.net> wrote:
>> This reduces hash lookups for looking up cache data and will
>> simplify tying data to disk in the next commit.
> 
> I considered the following, but GDBM might not be readily available on
> non-POSIX platforms.  I think the other problem is the existing caches
> are still in memory (whether YAML or Storable) even if disk-backed,
> causing a large amount of memory usage anyways.
If cached_mergeinfo is using too much memory, you can probably drop that cache entirely. IIRC, it didn't give that much of a speed up.
I am surprised that it is using a lot of memory, though. There is only one entry per SVN branch.
Show 66 quoted lines
> (Both patches on top of Jakob's)
> -------------------------
> Subject: [RFC] git-svn: tie cached_mergeinfo to a GDBM_File store
> 
> This should reduce per-instance memory usage by allowing
> serialization to disk.  Using the existing Memoize::Storable
> or YAML backends does not allow fast lookups.
> 
> GDBM_File should be available in most Perl installations
> and should not pose unnecessary burden
> ---
> perl/Git/SVN.pm | 19 ++++++++++++++++---
> 1 file changed, 16 insertions(+), 3 deletions(-)
> 
> diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
> index 25dbcd5..3e477c7 100644
> --- a/perl/Git/SVN.pm
> +++ b/perl/Git/SVN.pm
> @@ -14,6 +14,7 @@ use IPC::Open3;
> use Memoize;  # core since 5.8.0, Jul 2002
> use Memoize::Storable;
> use POSIX qw(:signal_h);
> +use Storable qw(freeze thaw);
> 
> use Git qw(
>     command
> @@ -1713,10 +1714,21 @@ sub mergeinfo_changes {
> 
>    # Initialize cache on the first call.
>    unless (defined $cached_mergeinfo) {
> -        $cached_mergeinfo = $self->{cached_mergeinfo} = {};
> +        my %hash;
> +        eval '
> +        require File::Temp;
> +        use GDBM_File;
> +        my $fh = File::Temp->new(TEMPLATE => "mergeinfo.XXXXXXXX");
> +        $self->{cached_mergeinfo_fh} = $fh;
> +        $fh->unlink_on_destroy(1);
> +        tie %hash => "GDBM_File", $fh->filename, GDBM_WRCREAT, 0600;
> +        ';
> +        $cached_mergeinfo = $self->{cached_mergeinfo} = \%hash;
>    }
> 
>    my $cached = $cached_mergeinfo->{$old_path};
> +    $cached = thaw($cached) if defined $cached;
> +
>    if (defined $cached && $cached->[0] == $old_rev) {
>        $old_minfo = $cached->[1];
>    } else {
> @@ -1735,11 +1747,12 @@ sub mergeinfo_changes {
>                $props->{"svn:mergeinfo"};
>            $old_minfo = \%omi;
>        }
> -        $cached_mergeinfo->{$old_path} = [ $old_rev, $old_minfo ];
> +        $cached_mergeinfo->{$old_path} =
> +                    freeze([ $old_rev, $old_minfo ]);
>    }
> 
>    # Cache the new mergeinfo.
> -    $cached_mergeinfo->{$path} = [ $rev, \%minfo ];
> +    $cached_mergeinfo->{$path} = freeze([ $rev, \%minfo ]);
> 
>    my %changes = ();
>    foreach my $p (keys %minfo) {
> -- 
> EW
Eric Wong· Oct 20, 2014, 01:16 UTC · re: Jakob Stoklund Olesen · lore

Re: git-svn performance

Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:
Show 5 quoted lines
> If cached_mergeinfo is using too much memory, you can probably drop
> that cache entirely. IIRC, it didn't give that much of a speed up.
> 
> I am surprised that it is using a lot of memory, though. There is only
> one entry per SVN branch.

Something like the below? (on top of your original two patches) Pushed to my master @ git://bogomips.org/git-svn.git

    Eric Wong (2):
          git-svn: reduce check_cherry_pick cache overhead
          git-svn: cache only mergeinfo revisions
    Jakob Stoklund Olesen (2):
          git-svn: only look at the new parts of svn:mergeinfo
          git-svn: only look at the root path for svn:mergeinfo
git-svn still seems to have some excessive memory usage problems,
even independenty of mergeinfo stuff.
--------------------------8<----------------------------
From: Eric Wong <normalperson@yhbt.net>
Date: Mon, 20 Oct 2014 01:02:53 +0000
Subject: [PATCH] git-svn: cache only mergeinfo revisions

This should reduce excessive memory usage from the new mergeinfo caches without hurting performance too much, assuming reasonable latency to the SVN server.

Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 perl/Git/SVN.pm | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
index 171af37..f8a75b1 100644
--- a/perl/Git/SVN.pm
+++ b/perl/Git/SVN.pm
@@ -1713,13 +1713,10 @@ sub mergeinfo_changes {
 	# Initialize cache on the first call.
 	unless (defined $self->{cached_mergeinfo_rev}) {
 		$self->{cached_mergeinfo_rev} = {};
-		$self->{cached_mergeinfo} = {};
 	}
 
 	my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};
-	if (defined $cached_rev && $cached_rev == $old_rev) {
-		$old_minfo = $self->{cached_mergeinfo}{$old_path};
-	} else {
+	unless (defined $cached_rev && $cached_rev == $old_rev) {
 		my $ra = $self->ra;
 		# Give up if $old_path isn't in the repo.
 		# This is probably a merge on a subtree.
@@ -1728,19 +1725,16 @@ sub mergeinfo_changes {
 				"directory didn't exist in r$old_rev\n";
 			return {};
 		}
-		my (undef, undef, $props) =
-			$self->ra->get_dir($old_path, $old_rev);
-		if (defined $props->{"svn:mergeinfo"}) {
-			my %omi = map {split ":", $_ } split "\n",
-				$props->{"svn:mergeinfo"};
-			$old_minfo = \%omi;
-		}
-		$self->{cached_mergeinfo}{$old_path} = $old_minfo;
-		$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;
 	}
+	my (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);
+	if (defined $props->{"svn:mergeinfo"}) {
+		my %omi = map {split ":", $_ } split "\n",
+			$props->{"svn:mergeinfo"};
+		$old_minfo = \%omi;
+	}
+	$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;
 
 	# Cache the new mergeinfo.
-	$self->{cached_mergeinfo}{$path} = \%minfo;
 	$self->{cached_mergeinfo_rev}{$path} = $rev;
 
 	my %changes = ();

-- 
EW
Jakob Stoklund Olesen· Oct 20, 2014, 13:46 UTC · re: Eric Wong · lore

Re: git-svn performance

Show 11 quoted lines
> On Oct 19, 2014, at 18:16, Eric Wong <normalperson@yhbt.net> wrote:
> 
> Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:
>> If cached_mergeinfo is using too much memory, you can probably drop
>> that cache entirely. IIRC, it didn't give that much of a speed up.
>> 
>> I am surprised that it is using a lot of memory, though. There is only
>> one entry per SVN branch.
> 
> Something like the below?  (on top of your original two patches)
> Pushed to my master @ git://bogomips.org/git-svn.git
Yes, but I think you can remove cached_mergeinfo_rev too.

Thanks /Jakob

Show 74 quoted lines
>    Eric Wong (2):
>          git-svn: reduce check_cherry_pick cache overhead
>          git-svn: cache only mergeinfo revisions
> 
>    Jakob Stoklund Olesen (2):
>          git-svn: only look at the new parts of svn:mergeinfo
>          git-svn: only look at the root path for svn:mergeinfo
> 
> git-svn still seems to have some excessive memory usage problems,
> even independenty of mergeinfo stuff.
> --------------------------8<----------------------------
> From: Eric Wong <normalperson@yhbt.net>
> Date: Mon, 20 Oct 2014 01:02:53 +0000
> Subject: [PATCH] git-svn: cache only mergeinfo revisions
> 
> This should reduce excessive memory usage from the new mergeinfo
> caches without hurting performance too much, assuming reasonable
> latency to the SVN server.
> 
> Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
> Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
> Signed-off-by: Eric Wong <normalperson@yhbt.net>
> ---
> perl/Git/SVN.pm | 22 ++++++++--------------
> 1 file changed, 8 insertions(+), 14 deletions(-)
> 
> diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
> index 171af37..f8a75b1 100644
> --- a/perl/Git/SVN.pm
> +++ b/perl/Git/SVN.pm
> @@ -1713,13 +1713,10 @@ sub mergeinfo_changes {
>    # Initialize cache on the first call.
>    unless (defined $self->{cached_mergeinfo_rev}) {
>        $self->{cached_mergeinfo_rev} = {};
> -        $self->{cached_mergeinfo} = {};
>    }
> 
>    my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};
> -    if (defined $cached_rev && $cached_rev == $old_rev) {
> -        $old_minfo = $self->{cached_mergeinfo}{$old_path};
> -    } else {
> +    unless (defined $cached_rev && $cached_rev == $old_rev) {
>        my $ra = $self->ra;
>        # Give up if $old_path isn't in the repo.
>        # This is probably a merge on a subtree.
> @@ -1728,19 +1725,16 @@ sub mergeinfo_changes {
>                "directory didn't exist in r$old_rev\n";
>            return {};
>        }
> -        my (undef, undef, $props) =
> -            $self->ra->get_dir($old_path, $old_rev);
> -        if (defined $props->{"svn:mergeinfo"}) {
> -            my %omi = map {split ":", $_ } split "\n",
> -                $props->{"svn:mergeinfo"};
> -            $old_minfo = \%omi;
> -        }
> -        $self->{cached_mergeinfo}{$old_path} = $old_minfo;
> -        $self->{cached_mergeinfo_rev}{$old_path} = $old_rev;
>    }
> +    my (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);
> +    if (defined $props->{"svn:mergeinfo"}) {
> +        my %omi = map {split ":", $_ } split "\n",
> +            $props->{"svn:mergeinfo"};
> +        $old_minfo = \%omi;
> +    }
> +    $self->{cached_mergeinfo_rev}{$old_path} = $old_rev;
> 
>    # Cache the new mergeinfo.
> -    $self->{cached_mergeinfo}{$path} = \%minfo;
>    $self->{cached_mergeinfo_rev}{$path} = $rev;
> 
>    my %changes = ();
> -- 
> EW
Eric Wong· Oct 21, 2014, 09:00 UTC · re: Jakob Stoklund Olesen · lore

Re: git-svn performance

Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:
> Yes, but I think you can remove cached_mergeinfo_rev too. 
Thanks, pushed the patch at the bottom, too.
Also started working on some memory reductions here:
 http://mid.gmane.org/20141021033912.GA27462@dcvr.yhbt.net
But there seem to be more problems :<
----------------------------8<-----------------------------
From: Eric Wong <normalperson@yhbt.net>
Date: Tue, 21 Oct 2014 06:23:22 +0000
Subject: [PATCH] git-svn: remove mergeinfo rev caching

This should further reduce memory usage from the new mergeinfo speedups without hurting performance too much, assuming reasonable latency to the SVN server.

Cc: Hin-Tak Leung <htl10@users.sourceforge.net>
Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 perl/Git/SVN.pm | 30 +++++++++---------------------
 1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
index f8a75b1..4364506 100644
--- a/perl/Git/SVN.pm
+++ b/perl/Git/SVN.pm
@@ -1710,32 +1710,20 @@ sub mergeinfo_changes {
 	my %minfo = map {split ":", $_ } split "\n", $mergeinfo_prop;
 	my $old_minfo = {};
 
-	# Initialize cache on the first call.
-	unless (defined $self->{cached_mergeinfo_rev}) {
-		$self->{cached_mergeinfo_rev} = {};
-	}
-
-	my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};
-	unless (defined $cached_rev && $cached_rev == $old_rev) {
-		my $ra = $self->ra;
-		# Give up if $old_path isn't in the repo.
-		# This is probably a merge on a subtree.
-		if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {
-			warn "W: ignoring svn:mergeinfo on $old_path, ",
-				"directory didn't exist in r$old_rev\n";
-			return {};
-		}
-	}
-	my (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);
+	my $ra = $self->ra;
+	# Give up if $old_path isn't in the repo.
+	# This is probably a merge on a subtree.
+	if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {
+		warn "W: ignoring svn:mergeinfo on $old_path, ",
+			"directory didn't exist in r$old_rev\n";
+		return {};
+	}
+	my (undef, undef, $props) = $ra->get_dir($old_path, $old_rev);
 	if (defined $props->{"svn:mergeinfo"}) {
 		my %omi = map {split ":", $_ } split "\n",
 			$props->{"svn:mergeinfo"};
 		$old_minfo = \%omi;
 	}
-	$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;
-
-	# Cache the new mergeinfo.
-	$self->{cached_mergeinfo_rev}{$path} = $rev;
 
 	my %changes = ();
 	foreach my $p (keys %minfo) {

-- 
EW
Fabian Schmied· Oct 19, 2014, 09:38 UTC · re: Eric Wong · lore

Re: git-svn performance

On Sun, Oct 19, 2014 at 2:32 AM, Eric Wong <normalperson@yhbt.net> wrote:
Show 11 quoted lines
> Fabian Schmied <fabian.schmied@gmail.com> wrote:
>> Hi,
>>
>> I'm currently migrating an SVN repository to Git using git-svn (Git
>> for Windows 1.8.3-preview20130601), and I'm experiencing severe
>> performance problems with "git svn fetch". Commits to the SVN "trunk"
>> are fetched very fast (a few seconds or so per SVN revision), but
>> commits to some branches ("hotfix" branches) are currently taking
>> about 9 minutes per revision. I fear that the time per these commits
>> is increasing and that indeed the migration might not be finishable at
>> all.
[...]
Show 14 quoted lines
>> Is there anything I can do to speed this up? (I already tried
>> increasing the --log-window-size to 500, didn't have any effect.)
>
> Can you take a look at the following two "mergeinfo-speedups"
> in my repo?  (git://bogomips.org/git-svn)
>
> Jakob Stoklund Olesen (2):
>       git-svn: only look at the new parts of svn:mergeinfo
>       git-svn: only look at the root path for svn:mergeinfo
>
> Also downloadable here:
>
> http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535
> http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a
[...]

Thank you _very_ much, the performance increase is tremendous: from, ATM, 15 minutes per commit (with large merge-infos) down to 15 seconds each. This means that instead of taking weeks, the migration will now complete in hours! Memory consumption might be a bit higher, but not a problem for me at all.

(I didn't apply the two additional patches you supplied, only the two ones linked above.)

Thanks again, you saved my deadline :) Fabian

← back to recent threads