{"thread":{"id":"37752","subject":"git-svn performance","startedAt":"2014-10-17T20:47:20Z","lastAt":"2014-10-21T09:00:56Z","messageCount":9,"participants":["Fabian Schmied","Eric Wong","Jakob Stoklund Olesen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"250799","messageId":"CABBCAiv0WXNzo7W9PB_o_enLjtUO_rNRb4UBEqDPeSkBj1k-Ww@mail.gmail.com","threadId":"37752","inReplyTo":null,"subject":"git-svn performance","fromName":"Fabian Schmied","fromEmail":"fabian.schmied@gmail.com","sentAt":"2014-10-17T20:47:20Z","receivedAt":"2014-10-17T20:47:20Z","isPatch":false,"sender":{"key":"fabian.schmied@gmail.com","avatar":null},"body":"Hi,\n\nI'm currently migrating an SVN repository to Git using git-svn (Git\nfor Windows 1.8.3-preview20130601), and I'm experiencing severe\nperformance problems with \"git svn fetch\". Commits to the SVN \"trunk\"\nare fetched very fast (a few seconds or so per SVN revision), but\ncommits to some branches (\"hotfix\" branches) are currently taking\nabout 9 minutes per revision. I fear that the time per these commits\nis increasing and that indeed the migration might not be finishable at\nall.\n\nFor the commits that take such a long time, git-svn always outputs\nlots of warnings about ignored SVN cherry-picks, and it tells me it\ncan't find a revmap for the path being imported. (See [1].)\n\nAFAICS, the offending commits take place on some branches that include\na lot of manually merged (\"SVN cherry-picked\") revisions. Git-svn\nseems to be checking something (though I don't know what) that makes\nimporting these revisions really slow. And it repeats this for every\nrevision on these branches with increasing work to do.\n\nIs there anything I can do to speed this up? (I already tried\nincreasing the --log-window-size to 500, didn't have any effect.)\n\nThank you, best regards,\nFabian\n\n[1]\n        M       foo/bar/XXX.xml\n        M       foo/bar/YYY.xml\nW:svn cherry-pick ignored (/branches/frob:6940-7068) - missing 12\ncommit(s) (eg abeaece820ceae44ebf2c06011cf43bbcbf4b1ce)\nW:svn cherry-pick ignored (/branches/feature:3316-4798,4811,4827) -\nmissing 10 commit(s) (eg e255fff14ab1e581f21671ca8b36c0747869cf8c)\nW:svn cherry-pick ignored\n(/hotfixes/ZZZ.159:2131,2133,2145-2146,2148,2169) - missing 10\ncommit(s) (eg e04b0326c998f0611c18144b3ed8f686d3b52f4c)\nW:svn cherry-pick ignored\n(/hotfixes/ZZZ.333:4536,4610-4611,4625,4665,4669,4685,4713,4745,4785,4788,4908-4917,4920,4933-4944,4955,5003,5103,5174,5222,5227,\n5261,5267,5306,5310,5321,5360,5416,5467,5501,5508,5599-5614,5650-5651,5757,5761-5762,5764,5778-5779,5784,5811,5814,5819,5823,5825,5836-5838,5860,5862,5873,5889,\n5910,5924,5948) - missing 137 commit(s) (eg\n9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)\nC:\\Program Files (x86)\\Git\\bin\\perl.exe: *** WFSO timed out\nW:svn cherry-pick ignored (/hotfixes/ZZZ.333.39:5696,5847) - missing\n84 commit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)\nW:svn cherry-pick ignored (/hotfixes/AAA:5905,6095) - missing 119\ncommit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)\nW:svn cherry-pick ignored (/hotfixes/BBB_1.1:6971) - missing 198\ncommit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)\nW:svn cherry-pick ignored\n(/hotfixes/CCC:6134,6164,6168,6174,6206,6211,6237,6239,6244-6245,6250,6257,6269,6271,6276,6289-6292,6294,6296,6301-6302,6313,6315-6316,6329,6333,6379,6383,6394,6405,6411,6456,6478,6483,6491,6519,6537,6557)\n- missing 194 commit(s) (eg 9daec24cbdf55200d2cdfb0cd6b3f10485e296ac)\nW:svn cherry-pick ignored (/hotfixes/DDD:7635) - missing 1 commit(s)\n(eg 6a3ba817635eb3a9411a307924dec393311d93be)\nW:svn cherry-pick ignored\n(/hotfixes/EEE_1.2:7786,7794,7797,7803,7829-7830,7843,7886,7889,7933,7937,7949,7953)\n- missing 80 commit(s) (eg e78b1bc68f7a9b041588a39f3fa5e1a61f98942b)\nW:svn cherry-pick ignored\n(/hotfixes/EEE_1.3:8159,8170,8173-8174,8177,8181-8182,8185,8187,8194-8195,8201,8203,8206,8251,8255,8257,8259-8262,8265,8280,8286,8294,8296,8304-8305,8312,8318,8323,8327,8363,8387-8388,8390,8422-8423,8432,8446,8536-8537,8548-8549,8556,8559,8566,8569,8572,8578,8597-8598,8602,8617,8619,8655,8687,8720)\n- missing 104 commit(s) (eg 33febd4591f42a9d871ba330432840917b157f9e)\nW:svn cherry-pick ignored\n(/hotfixes/EEE_1.4:8766,8768,8770,8777-8779,8795-8796,8802-8809,8812-8814,8816-8817,8820,8823,8825,8827,8831,8836,8841,8845,8848-8852,8854-8855,8866,8868-8869,8871-8873,8875-8878,8880,8888,8892,8911-8912,8917-8918,8946,8956-8957,8964,8984,8994,9003,9008,9011,9029,9038,9040,9046-9048,9055,9086,9101,9108,9111,9113,9124,9129,9133,9138-9139,9150,9152,9154,9156,9172,9174,9188-9189,9208,9211,9217)\n- missing 44 commit(s) (eg 0621fb44de682650d762c707b102bc2472c088f8)\nW:svn cherry-pick ignored\n(/hotfixes/EEE_1.5:9412,9421,9430,9433-9436,9439,9441,9449,9459,9468,9529,9548,9561,9568,9605-9606,9612,9614,9617,9628,9630-9631,9637,9687,9807)\n- missing 41 commit(s) (eg 1bd1a9b72336bf4d3839a00348b7f2a52368c16c)\nW:svn cherry-pick ignored\n(/trunk:9852-9853,9857,9859,9862,9868,9872,9876,9879,9890,9895,9926-9927,9933,9953,9956,9960-9962)\n- missing 60 commit(s) (eg 3322e7ffc6ab49181976d9e94c91a4556951f38a)\nCouldn't find revmap for https://the-svn-server/svn/something/trunk/foo\nr9963 = 597df48cb830825f9029d1cfdf45df024d7fd3dd (refs/remotes/EEE_1.6)\n"},{"id":"250813","messageId":"20141019003256.GA18532@dcvr.yhbt.net","threadId":"37752","inReplyTo":"CABBCAiv0WXNzo7W9PB_o_enLjtUO_rNRb4UBEqDPeSkBj1k-Ww@mail.gmail.com","subject":"Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-19T00:32:56Z","receivedAt":"2014-10-19T00:32:56Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Fabian Schmied <fabian.schmied@gmail.com> wrote:\n> Hi,\n> \n> I'm currently migrating an SVN repository to Git using git-svn (Git\n> for Windows 1.8.3-preview20130601), and I'm experiencing severe\n> performance problems with \"git svn fetch\". Commits to the SVN \"trunk\"\n> are fetched very fast (a few seconds or so per SVN revision), but\n> commits to some branches (\"hotfix\" branches) are currently taking\n> about 9 minutes per revision. I fear that the time per these commits\n> is increasing and that indeed the migration might not be finishable at\n> all.\n> \n> For the commits that take such a long time, git-svn always outputs\n> lots of warnings about ignored SVN cherry-picks, and it tells me it\n> can't find a revmap for the path being imported. (See [1].)\n> \n> AFAICS, the offending commits take place on some branches that include\n> a lot of manually merged (\"SVN cherry-picked\") revisions. Git-svn\n> seems to be checking something (though I don't know what) that makes\n> importing these revisions really slow. And it repeats this for every\n> revision on these branches with increasing work to do.\n> \n> Is there anything I can do to speed this up? (I already tried\n> increasing the --log-window-size to 500, didn't have any effect.)\n\nCan you take a look at the following two \"mergeinfo-speedups\"\nin my repo?  (git://bogomips.org/git-svn)\n\nJakob Stoklund Olesen (2):\n      git-svn: only look at the new parts of svn:mergeinfo\n      git-svn: only look at the root path for svn:mergeinfo\n\nAlso downloadable here:\n\nhttp://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535\nhttp://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a\n\nHin-Tak (Cc-ed) reported good improvements with them, but also\na large memory increase:\n\nhttp://mid.gmane.org/1412706046.90413.YahooMailBasic@web172303.mail.ir2.yahoo.com\n\nJakob (or anybody else): I suppose we could tie the new\ncached_mergeinfo* caches to disk-backed storage to avoid the memory\nbloat.\n"},{"id":"250817","messageId":"20141019022953.GA6537@dcvr.yhbt.net","threadId":"37752","inReplyTo":"20141019003256.GA18532@dcvr.yhbt.net","subject":"Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-19T02:29:53Z","receivedAt":"2014-10-19T02:29:53Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong <normalperson@yhbt.net> wrote:\n> Hin-Tak (Cc-ed) reported good improvements with them, but also\n> a large memory increase:\n\nThis might reduce the pathname and internal hash overheads:\n------------------------8<-----------------------\nFrom: Eric Wong <normalperson@yhbt.net>\nDate: Sun, 19 Oct 2014 02:26:53 +0000\nSubject: [PATCH] git-svn: simplify cached_mergeinfo layout\n\nThis reduces hash lookups for looking up cache data and will\nsimplify tying data to disk in the next commit.\n\nSigned-off-by: Eric Wong <normalperson@yhbt.net>\n---\n perl/Git/SVN.pm | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\nindex b1a84d0..25dbcd5 100644\n--- a/perl/Git/SVN.pm\n+++ b/perl/Git/SVN.pm\n@@ -1708,15 +1708,17 @@ sub mergeinfo_changes {\n \tmy %minfo = map {split \":\", $_ } split \"\\n\", $mergeinfo_prop;\n \tmy $old_minfo = {};\n \n+\t# layout: $path => [ $rev, \\%mergeinfo ]\n+\tmy $cached_mergeinfo = $self->{cached_mergeinfo};\n+\n \t# Initialize cache on the first call.\n-\tunless (defined $self->{cached_mergeinfo_rev}) {\n-\t\t$self->{cached_mergeinfo_rev} = {};\n-\t\t$self->{cached_mergeinfo} = {};\n+\tunless (defined $cached_mergeinfo) {\n+\t\t$cached_mergeinfo = $self->{cached_mergeinfo} = {};\n \t}\n \n-\tmy $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};\n-\tif (defined $cached_rev && $cached_rev == $old_rev) {\n-\t\t$old_minfo = $self->{cached_mergeinfo}{$old_path};\n+\tmy $cached = $cached_mergeinfo->{$old_path};\n+\tif (defined $cached && $cached->[0] == $old_rev) {\n+\t\t$old_minfo = $cached->[1];\n \t} else {\n \t\tmy $ra = $self->ra;\n \t\t# Give up if $old_path isn't in the repo.\n@@ -1733,13 +1735,11 @@ sub mergeinfo_changes {\n \t\t\t\t$props->{\"svn:mergeinfo\"};\n \t\t\t$old_minfo = \\%omi;\n \t\t}\n-\t\t$self->{cached_mergeinfo}{$old_path} = $old_minfo;\n-\t\t$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;\n+\t\t$cached_mergeinfo->{$old_path} = [ $old_rev, $old_minfo ];\n \t}\n \n \t# Cache the new mergeinfo.\n-\t$self->{cached_mergeinfo}{$path} = \\%minfo;\n-\t$self->{cached_mergeinfo_rev}{$path} = $rev;\n+\t$cached_mergeinfo->{$path} = [ $rev, \\%minfo ];\n \n \tmy %changes = ();\n \tforeach my $p (keys %minfo) {\n-- \nEW\n"},{"id":"250818","messageId":"20141019023358.GA2946@dcvr.yhbt.net","threadId":"37752","inReplyTo":"20141019022953.GA6537@dcvr.yhbt.net","subject":"Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-19T02:33:58Z","receivedAt":"2014-10-19T02:33:58Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong <normalperson@yhbt.net> wrote:\n> This reduces hash lookups for looking up cache data and will\n> simplify tying data to disk in the next commit.\n\nI considered the following, but GDBM might not be readily available on\nnon-POSIX platforms.  I think the other problem is the existing caches\nare still in memory (whether YAML or Storable) even if disk-backed,\ncausing a large amount of memory usage anyways.\n\n(Both patches on top of Jakob's)\n-------------------------\nSubject: [RFC] git-svn: tie cached_mergeinfo to a GDBM_File store\n\nThis should reduce per-instance memory usage by allowing\nserialization to disk.  Using the existing Memoize::Storable\nor YAML backends does not allow fast lookups.\n\nGDBM_File should be available in most Perl installations\nand should not pose unnecessary burden\n---\n perl/Git/SVN.pm | 19 ++++++++++++++++---\n 1 file changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\nindex 25dbcd5..3e477c7 100644\n--- a/perl/Git/SVN.pm\n+++ b/perl/Git/SVN.pm\n@@ -14,6 +14,7 @@ use IPC::Open3;\n use Memoize;  # core since 5.8.0, Jul 2002\n use Memoize::Storable;\n use POSIX qw(:signal_h);\n+use Storable qw(freeze thaw);\n \n use Git qw(\n     command\n@@ -1713,10 +1714,21 @@ sub mergeinfo_changes {\n \n \t# Initialize cache on the first call.\n \tunless (defined $cached_mergeinfo) {\n-\t\t$cached_mergeinfo = $self->{cached_mergeinfo} = {};\n+\t\tmy %hash;\n+\t\teval '\n+\t\trequire File::Temp;\n+\t\tuse GDBM_File;\n+\t\tmy $fh = File::Temp->new(TEMPLATE => \"mergeinfo.XXXXXXXX\");\n+\t\t$self->{cached_mergeinfo_fh} = $fh;\n+\t\t$fh->unlink_on_destroy(1);\n+\t\ttie %hash => \"GDBM_File\", $fh->filename, GDBM_WRCREAT, 0600;\n+\t\t';\n+\t\t$cached_mergeinfo = $self->{cached_mergeinfo} = \\%hash;\n \t}\n \n \tmy $cached = $cached_mergeinfo->{$old_path};\n+\t$cached = thaw($cached) if defined $cached;\n+\n \tif (defined $cached && $cached->[0] == $old_rev) {\n \t\t$old_minfo = $cached->[1];\n \t} else {\n@@ -1735,11 +1747,12 @@ sub mergeinfo_changes {\n \t\t\t\t$props->{\"svn:mergeinfo\"};\n \t\t\t$old_minfo = \\%omi;\n \t\t}\n-\t\t$cached_mergeinfo->{$old_path} = [ $old_rev, $old_minfo ];\n+\t\t$cached_mergeinfo->{$old_path} =\n+\t\t\t\t\tfreeze([ $old_rev, $old_minfo ]);\n \t}\n \n \t# Cache the new mergeinfo.\n-\t$cached_mergeinfo->{$path} = [ $rev, \\%minfo ];\n+\t$cached_mergeinfo->{$path} = freeze([ $rev, \\%minfo ]);\n \n \tmy %changes = ();\n \tforeach my $p (keys %minfo) {\n-- \nEW\n"},{"id":"250820","messageId":"CABBCAivMYZJ-b2CFuCxBPRBYRkhJ_sKMBRHkenfqjpJbYcx5vw@mail.gmail.com","threadId":"37752","inReplyTo":"20141019003256.GA18532@dcvr.yhbt.net","subject":"Re: git-svn performance","fromName":"Fabian Schmied","fromEmail":"fabian.schmied@gmail.com","sentAt":"2014-10-19T09:38:16Z","receivedAt":"2014-10-19T09:38:16Z","isPatch":false,"sender":{"key":"fabian.schmied@gmail.com","avatar":null},"body":"On Sun, Oct 19, 2014 at 2:32 AM, Eric Wong <normalperson@yhbt.net> wrote:\n> Fabian Schmied <fabian.schmied@gmail.com> wrote:\n>> Hi,\n>>\n>> I'm currently migrating an SVN repository to Git using git-svn (Git\n>> for Windows 1.8.3-preview20130601), and I'm experiencing severe\n>> performance problems with \"git svn fetch\". Commits to the SVN \"trunk\"\n>> are fetched very fast (a few seconds or so per SVN revision), but\n>> commits to some branches (\"hotfix\" branches) are currently taking\n>> about 9 minutes per revision. I fear that the time per these commits\n>> is increasing and that indeed the migration might not be finishable at\n>> all.\n\n[...]\n\n>> Is there anything I can do to speed this up? (I already tried\n>> increasing the --log-window-size to 500, didn't have any effect.)\n>\n> Can you take a look at the following two \"mergeinfo-speedups\"\n> in my repo?  (git://bogomips.org/git-svn)\n>\n> Jakob Stoklund Olesen (2):\n>       git-svn: only look at the new parts of svn:mergeinfo\n>       git-svn: only look at the root path for svn:mergeinfo\n>\n> Also downloadable here:\n>\n> http://bogomips.org/git-svn.git/patch?id=9b258e721b30785357535\n> http://bogomips.org/git-svn.git/patch?id=73409a2145e93b436d74a\n\n[...]\n\nThank you _very_ much, the performance increase is tremendous: from,\nATM, 15 minutes per commit (with large merge-infos) down to 15 seconds\neach. This means that instead of taking weeks, the migration will now\ncomplete in hours! Memory consumption might be a bit higher, but not a\nproblem for me at all.\n\n(I didn't apply the two additional patches you supplied, only the two\nones linked above.)\n\nThanks again, you saved my deadline :)\nFabian\n"},{"id":"250831","messageId":"B5419560-60BC-4228-B1BA-9BAD5E4D58B9@2pi.dk","threadId":"37752","inReplyTo":"20141019023358.GA2946@dcvr.yhbt.net","subject":"Re: git-svn performance","fromName":"Jakob Stoklund Olesen","fromEmail":"stoklund@2pi.dk","sentAt":"2014-10-19T14:56:11Z","receivedAt":"2014-10-19T14:56:11Z","isPatch":false,"sender":{"key":"stoklund@2pi.dk","avatar":"https://avatars.githubusercontent.com/u/12660495?v=4"},"body":"\n> On Oct 18, 2014, at 19:33, Eric Wong <normalperson@yhbt.net> wrote:\n> \n> Eric Wong <normalperson@yhbt.net> wrote:\n>> This reduces hash lookups for looking up cache data and will\n>> simplify tying data to disk in the next commit.\n> \n> I considered the following, but GDBM might not be readily available on\n> non-POSIX platforms.  I think the other problem is the existing caches\n> are still in memory (whether YAML or Storable) even if disk-backed,\n> causing a large amount of memory usage anyways.\n\nIf cached_mergeinfo is using too much memory, you can probably drop that cache entirely. IIRC, it didn't give that much of a speed up.\n\nI am surprised that it is using a lot of memory, though. There is only one entry per SVN branch.\n\n> (Both patches on top of Jakob's)\n> -------------------------\n> Subject: [RFC] git-svn: tie cached_mergeinfo to a GDBM_File store\n> \n> This should reduce per-instance memory usage by allowing\n> serialization to disk.  Using the existing Memoize::Storable\n> or YAML backends does not allow fast lookups.\n> \n> GDBM_File should be available in most Perl installations\n> and should not pose unnecessary burden\n> ---\n> perl/Git/SVN.pm | 19 ++++++++++++++++---\n> 1 file changed, 16 insertions(+), 3 deletions(-)\n> \n> diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\n> index 25dbcd5..3e477c7 100644\n> --- a/perl/Git/SVN.pm\n> +++ b/perl/Git/SVN.pm\n> @@ -14,6 +14,7 @@ use IPC::Open3;\n> use Memoize;  # core since 5.8.0, Jul 2002\n> use Memoize::Storable;\n> use POSIX qw(:signal_h);\n> +use Storable qw(freeze thaw);\n> \n> use Git qw(\n>     command\n> @@ -1713,10 +1714,21 @@ sub mergeinfo_changes {\n> \n>    # Initialize cache on the first call.\n>    unless (defined $cached_mergeinfo) {\n> -        $cached_mergeinfo = $self->{cached_mergeinfo} = {};\n> +        my %hash;\n> +        eval '\n> +        require File::Temp;\n> +        use GDBM_File;\n> +        my $fh = File::Temp->new(TEMPLATE => \"mergeinfo.XXXXXXXX\");\n> +        $self->{cached_mergeinfo_fh} = $fh;\n> +        $fh->unlink_on_destroy(1);\n> +        tie %hash => \"GDBM_File\", $fh->filename, GDBM_WRCREAT, 0600;\n> +        ';\n> +        $cached_mergeinfo = $self->{cached_mergeinfo} = \\%hash;\n>    }\n> \n>    my $cached = $cached_mergeinfo->{$old_path};\n> +    $cached = thaw($cached) if defined $cached;\n> +\n>    if (defined $cached && $cached->[0] == $old_rev) {\n>        $old_minfo = $cached->[1];\n>    } else {\n> @@ -1735,11 +1747,12 @@ sub mergeinfo_changes {\n>                $props->{\"svn:mergeinfo\"};\n>            $old_minfo = \\%omi;\n>        }\n> -        $cached_mergeinfo->{$old_path} = [ $old_rev, $old_minfo ];\n> +        $cached_mergeinfo->{$old_path} =\n> +                    freeze([ $old_rev, $old_minfo ]);\n>    }\n> \n>    # Cache the new mergeinfo.\n> -    $cached_mergeinfo->{$path} = [ $rev, \\%minfo ];\n> +    $cached_mergeinfo->{$path} = freeze([ $rev, \\%minfo ]);\n> \n>    my %changes = ();\n>    foreach my $p (keys %minfo) {\n> -- \n> EW\n"},{"id":"250842","messageId":"20141020011601.GA17345@dcvr.yhbt.net","threadId":"37752","inReplyTo":"B5419560-60BC-4228-B1BA-9BAD5E4D58B9@2pi.dk","subject":"Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-20T01:16:01Z","receivedAt":"2014-10-20T01:16:01Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:\n> If cached_mergeinfo is using too much memory, you can probably drop\n> that cache entirely. IIRC, it didn't give that much of a speed up.\n> \n> I am surprised that it is using a lot of memory, though. There is only\n> one entry per SVN branch.\n\nSomething like the below?  (on top of your original two patches)\nPushed to my master @ git://bogomips.org/git-svn.git\n\n    Eric Wong (2):\n          git-svn: reduce check_cherry_pick cache overhead\n          git-svn: cache only mergeinfo revisions\n\n    Jakob Stoklund Olesen (2):\n          git-svn: only look at the new parts of svn:mergeinfo\n          git-svn: only look at the root path for svn:mergeinfo\n\ngit-svn still seems to have some excessive memory usage problems,\neven independenty of mergeinfo stuff.\n--------------------------8<----------------------------\nFrom: Eric Wong <normalperson@yhbt.net>\nDate: Mon, 20 Oct 2014 01:02:53 +0000\nSubject: [PATCH] git-svn: cache only mergeinfo revisions\n\nThis should reduce excessive memory usage from the new mergeinfo\ncaches without hurting performance too much, assuming reasonable\nlatency to the SVN server.\n\nCc: Hin-Tak Leung <htl10@users.sourceforge.net>\nSuggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>\nSigned-off-by: Eric Wong <normalperson@yhbt.net>\n---\n perl/Git/SVN.pm | 22 ++++++++--------------\n 1 file changed, 8 insertions(+), 14 deletions(-)\n\ndiff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\nindex 171af37..f8a75b1 100644\n--- a/perl/Git/SVN.pm\n+++ b/perl/Git/SVN.pm\n@@ -1713,13 +1713,10 @@ sub mergeinfo_changes {\n \t# Initialize cache on the first call.\n \tunless (defined $self->{cached_mergeinfo_rev}) {\n \t\t$self->{cached_mergeinfo_rev} = {};\n-\t\t$self->{cached_mergeinfo} = {};\n \t}\n \n \tmy $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};\n-\tif (defined $cached_rev && $cached_rev == $old_rev) {\n-\t\t$old_minfo = $self->{cached_mergeinfo}{$old_path};\n-\t} else {\n+\tunless (defined $cached_rev && $cached_rev == $old_rev) {\n \t\tmy $ra = $self->ra;\n \t\t# Give up if $old_path isn't in the repo.\n \t\t# This is probably a merge on a subtree.\n@@ -1728,19 +1725,16 @@ sub mergeinfo_changes {\n \t\t\t\t\"directory didn't exist in r$old_rev\\n\";\n \t\t\treturn {};\n \t\t}\n-\t\tmy (undef, undef, $props) =\n-\t\t\t$self->ra->get_dir($old_path, $old_rev);\n-\t\tif (defined $props->{\"svn:mergeinfo\"}) {\n-\t\t\tmy %omi = map {split \":\", $_ } split \"\\n\",\n-\t\t\t\t$props->{\"svn:mergeinfo\"};\n-\t\t\t$old_minfo = \\%omi;\n-\t\t}\n-\t\t$self->{cached_mergeinfo}{$old_path} = $old_minfo;\n-\t\t$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;\n \t}\n+\tmy (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);\n+\tif (defined $props->{\"svn:mergeinfo\"}) {\n+\t\tmy %omi = map {split \":\", $_ } split \"\\n\",\n+\t\t\t$props->{\"svn:mergeinfo\"};\n+\t\t$old_minfo = \\%omi;\n+\t}\n+\t$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;\n \n \t# Cache the new mergeinfo.\n-\t$self->{cached_mergeinfo}{$path} = \\%minfo;\n \t$self->{cached_mergeinfo_rev}{$path} = $rev;\n \n \tmy %changes = ();\n-- \nEW\n"},{"id":"250853","messageId":"00E02C52-67BC-461B-BF46-0EEDCFFFDFB1@2pi.dk","threadId":"37752","inReplyTo":"20141020011601.GA17345@dcvr.yhbt.net","subject":"Re: git-svn performance","fromName":"Jakob Stoklund Olesen","fromEmail":"stoklund@2pi.dk","sentAt":"2014-10-20T13:46:19Z","receivedAt":"2014-10-20T13:46:19Z","isPatch":false,"sender":{"key":"stoklund@2pi.dk","avatar":"https://avatars.githubusercontent.com/u/12660495?v=4"},"body":"\n> On Oct 19, 2014, at 18:16, Eric Wong <normalperson@yhbt.net> wrote:\n> \n> Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:\n>> If cached_mergeinfo is using too much memory, you can probably drop\n>> that cache entirely. IIRC, it didn't give that much of a speed up.\n>> \n>> I am surprised that it is using a lot of memory, though. There is only\n>> one entry per SVN branch.\n> \n> Something like the below?  (on top of your original two patches)\n> Pushed to my master @ git://bogomips.org/git-svn.git\n\nYes, but I think you can remove cached_mergeinfo_rev too. \n\nThanks\n/Jakob\n\n\n>    Eric Wong (2):\n>          git-svn: reduce check_cherry_pick cache overhead\n>          git-svn: cache only mergeinfo revisions\n> \n>    Jakob Stoklund Olesen (2):\n>          git-svn: only look at the new parts of svn:mergeinfo\n>          git-svn: only look at the root path for svn:mergeinfo\n> \n> git-svn still seems to have some excessive memory usage problems,\n> even independenty of mergeinfo stuff.\n> --------------------------8<----------------------------\n> From: Eric Wong <normalperson@yhbt.net>\n> Date: Mon, 20 Oct 2014 01:02:53 +0000\n> Subject: [PATCH] git-svn: cache only mergeinfo revisions\n> \n> This should reduce excessive memory usage from the new mergeinfo\n> caches without hurting performance too much, assuming reasonable\n> latency to the SVN server.\n> \n> Cc: Hin-Tak Leung <htl10@users.sourceforge.net>\n> Suggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>\n> Signed-off-by: Eric Wong <normalperson@yhbt.net>\n> ---\n> perl/Git/SVN.pm | 22 ++++++++--------------\n> 1 file changed, 8 insertions(+), 14 deletions(-)\n> \n> diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\n> index 171af37..f8a75b1 100644\n> --- a/perl/Git/SVN.pm\n> +++ b/perl/Git/SVN.pm\n> @@ -1713,13 +1713,10 @@ sub mergeinfo_changes {\n>    # Initialize cache on the first call.\n>    unless (defined $self->{cached_mergeinfo_rev}) {\n>        $self->{cached_mergeinfo_rev} = {};\n> -        $self->{cached_mergeinfo} = {};\n>    }\n> \n>    my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};\n> -    if (defined $cached_rev && $cached_rev == $old_rev) {\n> -        $old_minfo = $self->{cached_mergeinfo}{$old_path};\n> -    } else {\n> +    unless (defined $cached_rev && $cached_rev == $old_rev) {\n>        my $ra = $self->ra;\n>        # Give up if $old_path isn't in the repo.\n>        # This is probably a merge on a subtree.\n> @@ -1728,19 +1725,16 @@ sub mergeinfo_changes {\n>                \"directory didn't exist in r$old_rev\\n\";\n>            return {};\n>        }\n> -        my (undef, undef, $props) =\n> -            $self->ra->get_dir($old_path, $old_rev);\n> -        if (defined $props->{\"svn:mergeinfo\"}) {\n> -            my %omi = map {split \":\", $_ } split \"\\n\",\n> -                $props->{\"svn:mergeinfo\"};\n> -            $old_minfo = \\%omi;\n> -        }\n> -        $self->{cached_mergeinfo}{$old_path} = $old_minfo;\n> -        $self->{cached_mergeinfo_rev}{$old_path} = $old_rev;\n>    }\n> +    my (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);\n> +    if (defined $props->{\"svn:mergeinfo\"}) {\n> +        my %omi = map {split \":\", $_ } split \"\\n\",\n> +            $props->{\"svn:mergeinfo\"};\n> +        $old_minfo = \\%omi;\n> +    }\n> +    $self->{cached_mergeinfo_rev}{$old_path} = $old_rev;\n> \n>    # Cache the new mergeinfo.\n> -    $self->{cached_mergeinfo}{$path} = \\%minfo;\n>    $self->{cached_mergeinfo_rev}{$path} = $rev;\n> \n>    my %changes = ();\n> -- \n> EW\n"},{"id":"250891","messageId":"20141021090055.GA22184@dcvr.yhbt.net","threadId":"37752","inReplyTo":"00E02C52-67BC-461B-BF46-0EEDCFFFDFB1@2pi.dk","subject":"Re: git-svn performance","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2014-10-21T09:00:56Z","receivedAt":"2014-10-21T09:00:56Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jakob Stoklund Olesen <stoklund@2pi.dk> wrote:\n> Yes, but I think you can remove cached_mergeinfo_rev too. \n\nThanks, pushed the patch at the bottom, too.\nAlso started working on some memory reductions here:\n http://mid.gmane.org/20141021033912.GA27462@dcvr.yhbt.net\nBut there seem to be more problems :<\n\n----------------------------8<-----------------------------\nFrom: Eric Wong <normalperson@yhbt.net>\nDate: Tue, 21 Oct 2014 06:23:22 +0000\nSubject: [PATCH] git-svn: remove mergeinfo rev caching\n\nThis should further reduce memory usage from the new mergeinfo\nspeedups without hurting performance too much, assuming\nreasonable latency to the SVN server.\n\nCc: Hin-Tak Leung <htl10@users.sourceforge.net>\nSuggested-by: Jakob Stoklund Olesen <stoklund@2pi.dk>\nSigned-off-by: Eric Wong <normalperson@yhbt.net>\n---\n perl/Git/SVN.pm | 30 +++++++++---------------------\n 1 file changed, 9 insertions(+), 21 deletions(-)\n\ndiff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\nindex f8a75b1..4364506 100644\n--- a/perl/Git/SVN.pm\n+++ b/perl/Git/SVN.pm\n@@ -1710,32 +1710,20 @@ sub mergeinfo_changes {\n \tmy %minfo = map {split \":\", $_ } split \"\\n\", $mergeinfo_prop;\n \tmy $old_minfo = {};\n \n-\t# Initialize cache on the first call.\n-\tunless (defined $self->{cached_mergeinfo_rev}) {\n-\t\t$self->{cached_mergeinfo_rev} = {};\n-\t}\n-\n-\tmy $cached_rev = $self->{cached_mergeinfo_rev}{$old_path};\n-\tunless (defined $cached_rev && $cached_rev == $old_rev) {\n-\t\tmy $ra = $self->ra;\n-\t\t# Give up if $old_path isn't in the repo.\n-\t\t# This is probably a merge on a subtree.\n-\t\tif ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {\n-\t\t\twarn \"W: ignoring svn:mergeinfo on $old_path, \",\n-\t\t\t\t\"directory didn't exist in r$old_rev\\n\";\n-\t\t\treturn {};\n-\t\t}\n-\t}\n-\tmy (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev);\n+\tmy $ra = $self->ra;\n+\t# Give up if $old_path isn't in the repo.\n+\t# This is probably a merge on a subtree.\n+\tif ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) {\n+\t\twarn \"W: ignoring svn:mergeinfo on $old_path, \",\n+\t\t\t\"directory didn't exist in r$old_rev\\n\";\n+\t\treturn {};\n+\t}\n+\tmy (undef, undef, $props) = $ra->get_dir($old_path, $old_rev);\n \tif (defined $props->{\"svn:mergeinfo\"}) {\n \t\tmy %omi = map {split \":\", $_ } split \"\\n\",\n \t\t\t$props->{\"svn:mergeinfo\"};\n \t\t$old_minfo = \\%omi;\n \t}\n-\t$self->{cached_mergeinfo_rev}{$old_path} = $old_rev;\n-\n-\t# Cache the new mergeinfo.\n-\t$self->{cached_mergeinfo_rev}{$path} = $rev;\n \n \tmy %changes = ();\n \tforeach my $p (keys %minfo) {\n-- \nEW\n"}]}