{"thread":{"id":"38349","subject":"git-svn metadata commands performance issue","startedAt":"2015-01-13T10:14:37Z","lastAt":"2015-01-15T11:53:38Z","messageCount":5,"participants":["Niluge kiwi","Eric Wong","David Kastrup"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"254599","messageId":"CAKdxknyCQf6HpnTZ8BMVVpEX3vcVEoTo5BmpqrkgFuxV+k7wmw@mail.gmail.com","threadId":"38349","inReplyTo":null,"subject":"git-svn metadata commands performance issue","fromName":"Niluge kiwi","fromEmail":"kiwiiii@gmail.com","sentAt":"2015-01-13T10:14:37Z","receivedAt":"2015-01-13T10:14:37Z","isPatch":false,"sender":{"key":"kiwiiii@gmail.com","avatar":null},"body":"Hi all,\n\nIn magit (http://magit.github.io/), a popular git frontend within\nemacs, there is a git-svn frontend.  With a recent refactoring, it was\ndiscovered that git-svn metadata commands (like \"git-svn info\") are\nmuch slower than git ones:\ngit svn info: 130-150ms (after warmup): get the svn revision and url.\ngit svn rebase --dry-run: 150-170ms (after warmup): get the remote\nbranch.\n\nWhereas in pure git:\ngit rev-parse --abbrev-ref HEAD@{upstream}: 2-3ms (after warmup): get\nthe remote branch\nOther git commands alike take all less than 10ms after warmup.\n\nThis is an issue for the magit developers and users: just getting a\ngit-svn status with some metadata easily take ~500ms, which is really\nslow for a UI. The equivalent UI with a pure git repository in magit\ntakes much less than 100ms to generate although more than 30 git\nprocess are forked for it.\n\nA previous version of magit-svn was much faster because it\nre-implemented the logic of git-svn from perl to elisp (the\nprogramming language in emacs), and to get the 3 previously mentioned\nvalues it took less than 10ms.\n\n\nWhat could be done about this?\nCould git-svn performance be dramatically improved?\nEven git svn --version takes ~100ms, is perl the bottleneck?\nOr should each git-svn frontend developer re-implement the git-svn\nmetadata commands themselves for better performance?\nAlso, wouldn't it be better for those frontend developers if there\nwere some git-svn porcelain commands like git has? Fast, easy to parse\nand stable input & output format?\n\nFor reference, here is the discussion about the performance issue on\nmagit-svn: https://github.com/magit/magit-svn/issues/1\n\nAnd I'm using git-svn version 2.2.1 (svn 1.6.17) on an ubuntu 12.04\n64bits Intel machine with an HDD (no SSD).\n\nThanks,\nThomas Riccardi\n"},{"id":"254687","messageId":"20150114203012.GA26023@dcvr.yhbt.net","threadId":"38349","inReplyTo":"CAKdxknyCQf6HpnTZ8BMVVpEX3vcVEoTo5BmpqrkgFuxV+k7wmw@mail.gmail.com","subject":"Re: git-svn metadata commands performance issue","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-01-14T20:30:12Z","receivedAt":"2015-01-14T20:30:12Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Niluge kiwi <kiwiiii@gmail.com> wrote:\n> Hi all,\n> \n> In magit (http://magit.github.io/), a popular git frontend within\n> emacs, there is a git-svn frontend.  With a recent refactoring, it was\n> discovered that git-svn metadata commands (like \"git-svn info\") are\n> much slower than git ones:\n> git svn info: 130-150ms (after warmup): get the svn revision and url.\n> git svn rebase --dry-run: 150-170ms (after warmup): get the remote\n> branch.\n> \n> Whereas in pure git:\n> git rev-parse --abbrev-ref HEAD@{upstream}: 2-3ms (after warmup): get\n> the remote branch\n> Other git commands alike take all less than 10ms after warmup.\n\nThanks for the bug report.  I actually see worse performance from\nmy old machines, but I'm not a very heavy git-svn user anymore.\n100ms is an eternity :<\n\n> This is an issue for the magit developers and users: just getting a\n> git-svn status with some metadata easily take ~500ms, which is really\n> slow for a UI. The equivalent UI with a pure git repository in magit\n> takes much less than 100ms to generate although more than 30 git\n> process are forked for it.\n\nHow big is the parent process which forks the git commands?  On Linux at\nleast, fork() performance is negatively impacted by parent process\nmemory size.  To avoid spawn performance problems with large parent\nprocesses, vfork() should be used, but there does not seem to be an\neasy/portable way to use vfork() from Perl.\n\n> A previous version of magit-svn was much faster because it\n> re-implemented the logic of git-svn from perl to elisp (the\n> programming language in emacs), and to get the 3 previously mentioned\n> values it took less than 10ms.\n\nI've never worked with elisp, but we can probably figure out why it's\nfaster.  Can you give us a pointer to the old elisp code?\n\n> What could be done about this?\n> Could git-svn performance be dramatically improved?\n> Even git svn --version takes ~100ms, is perl the bottleneck?\n\nThe Linux 'perf' tool reports much time is spent is from the Perl\nparser.  So we may implement lazy loading, so simple commands such as\n\"git svn info\" do not need to load and parse all the Perl code.\n\n> Or should each git-svn frontend developer re-implement the git-svn\n> metadata commands themselves for better performance?\n\nI prefer git-svn be fast enough; but you're free to reimplement\nand optimize your own code as you see fit.\n\n> Also, wouldn't it be better for those frontend developers if there\n> were some git-svn porcelain commands like git has? Fast, easy to parse\n> and stable input & output format?\n\nSure, but we don't know what you'd need beyond the current command set.\nOf course we need to be careful about adding even more code to git-svn\nas that impacts startup time, too.\n"},{"id":"254738","messageId":"20150115101434.GA15361@dcvr.yhbt.net","threadId":"38349","inReplyTo":"20150114203012.GA26023@dcvr.yhbt.net","subject":"[PATCH] git-svn: lazy load some modules","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2015-01-15T10:14:34Z","receivedAt":"2015-01-15T10:14:34Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Baby steps, pushed to my master on git://bogomips.org/git-svn\n\n--------------------------8<----------------------------\nSubject: [PATCH] git-svn: lazy load some modules\n\nWe can delay loading some modules until we need them for uncommon\ncode paths.  For example, persistent memoization is not often\nneeded, so we can avoid loading the modules for it until we\nencounter svn::mergeinfo during fetch.\n\nThis gives a tiny reduction in syscalls (from 15641 to 15305) when\nrunning \"git svn info\" and counting via \"strace -fc\".  Further,\nmore invasive work will be needed to noticeably improve performance.\n\nSigned-off-by: Eric Wong <normalperson@yhbt.net>\n---\n git-svn.perl            | 13 +++++++------\n perl/Git/SVN.pm         | 22 +++++++++++++---------\n perl/Git/SVN/Editor.pm  |  3 +--\n perl/Git/SVN/Fetcher.pm |  3 +--\n perl/Git/SVN/Ra.pm      |  5 ++++-\n 5 files changed, 26 insertions(+), 20 deletions(-)\n\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 32d109e..36f7240 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -11,14 +11,10 @@ $AUTHOR = 'Eric Wong <normalperson@yhbt.net>';\n $VERSION = '@@GIT_VERSION@@';\n \n use Carp qw/croak/;\n-use Digest::MD5;\n-use IO::File qw//;\n use File::Basename qw/dirname basename/;\n use File::Path qw/mkpath/;\n use File::Spec;\n-use File::Find;\n use Getopt::Long qw/:config gnu_getopt no_ignore_case auto_abbrev/;\n-use IPC::Open3;\n use Memoize;\n \n use Git::SVN;\n@@ -298,7 +294,6 @@ my %cmd = (\n \t\t{} ],\n );\n \n-use Term::ReadLine;\n package FakeTerm;\n sub new {\n \tmy ($class, $reason) = @_;\n@@ -313,6 +308,7 @@ package main;\n my $term;\n sub term_init {\n \t$term = eval {\n+\t\trequire Term::ReadLine;\n \t\t$ENV{\"GIT_SVN_NOTTY\"}\n \t\t\t? new Term::ReadLine 'git-svn', \\*STDIN, \\*STDOUT\n \t\t\t: new Term::ReadLine 'git-svn';\n@@ -1173,6 +1169,7 @@ sub cmd_branch {\n \t}\n \n \t::_req_svn();\n+\trequire SVN::Client;\n \n \tmy $ctx = SVN::Client->new(\n \t\tconfig => SVN::Core::config_get_config(\n@@ -1693,11 +1690,13 @@ sub cmd_reset {\n }\n \n sub cmd_gc {\n+\trequire File::Find;\n \tif (!can_compress()) {\n \t\twarn \"Compress::Zlib could not be found; unhandled.log \" .\n \t\t     \"files will not be compressed.\\n\";\n \t}\n-\tfind({ wanted => \\&gc_directory, no_chdir => 1}, \"$ENV{GIT_DIR}/svn\");\n+\tFile::Find::find({ wanted => \\&gc_directory, no_chdir => 1},\n+\t\t\t \"$ENV{GIT_DIR}/svn\");\n }\n \n ########################### utility functions #########################\n@@ -2122,6 +2121,7 @@ sub find_file_type_and_diff_status {\n sub md5sum {\n \tmy $arg = shift;\n \tmy $ref = ref $arg;\n+\trequire Digest::MD5;\n \tmy $md5 = Digest::MD5->new();\n         if ($ref eq 'GLOB' || $ref eq 'IO::File' || $ref eq 'File::Temp') {\n \t\t$md5->addfile($arg) or croak $!;\n@@ -2148,6 +2148,7 @@ sub gc_directory {\n \t\t\t$gz->gzwrite($str) or\n \t\t\t\tdie \"Unable to write: \".$gz->gzerror().\"!\\n\";\n \t\t}\n+\t\tno warnings 'once'; # $File::Find::name would warn\n \t\tunlink $_ or die \"unlink $File::Find::name: $!\\n\";\n \t} elsif (-f $_ && basename($_) eq \"index\") {\n \t\tunlink $_ or die \"unlink $_: $!\\n\";\ndiff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm\nindex 8e4af71..afa562c 100644\n--- a/perl/Git/SVN.pm\n+++ b/perl/Git/SVN.pm\n@@ -9,10 +9,8 @@ use vars qw/$_no_metadata\n \t    $_use_log_author $_add_author_from $_localtime/;\n use Carp qw/croak/;\n use File::Path qw/mkpath/;\n-use File::Copy qw/copy/;\n use IPC::Open3;\n use Memoize;  # core since 5.8.0, Jul 2002\n-use Memoize::Storable;\n use POSIX qw(:signal_h);\n \n use Git qw(\n@@ -32,11 +30,7 @@ use Git::SVN::Utils qw(\n \tadd_path_to_url\n );\n \n-my $can_use_yaml;\n-BEGIN {\n-\t$can_use_yaml = eval { require Git::SVN::Memoize::YAML; 1};\n-}\n-\n+my $memo_backend;\n our $_follow_parent  = 1;\n our $_minimize_url   = 'unset';\n our $default_repo_id = 'svn';\n@@ -1578,7 +1572,16 @@ sub tie_for_persistent_memoization {\n \tmy $hash = shift;\n \tmy $path = shift;\n \n-\tif ($can_use_yaml) {\n+\tunless ($memo_backend) {\n+\t\tif (eval { require Git::SVN::Memoize::YAML; 1}) {\n+\t\t\t$memo_backend = 1;\n+\t\t} else {\n+\t\t\trequire Memoize::Storable;\n+\t\t\t$memo_backend = -1;\n+\t\t}\n+\t}\n+\n+\tif ($memo_backend > 0) {\n \t\ttie %$hash => 'Git::SVN::Memoize::YAML', \"$path.yaml\";\n \t} else {\n \t\ttie %$hash => 'Memoize::Storable', \"$path.db\", 'nstore';\n@@ -2188,8 +2191,9 @@ sub rev_map_set {\n \t# both of these options make our .rev_db file very, very important\n \t# and we can't afford to lose it because rebuild() won't work\n \tif ($self->use_svm_props || $self->no_metadata) {\n+\t\trequire File::Copy;\n \t\t$sync = 1;\n-\t\tcopy($db, $db_lock) or die \"rev_map_set(@_): \",\n+\t\tFile::Copy::copy($db, $db_lock) or die \"rev_map_set(@_): \",\n \t\t\t\t\t   \"Failed to copy: \",\n \t\t\t\t\t   \"$db => $db_lock ($!)\\n\";\n \t} else {\ndiff --git a/perl/Git/SVN/Editor.pm b/perl/Git/SVN/Editor.pm\nindex 4088f13..c50176e 100644\n--- a/perl/Git/SVN/Editor.pm\n+++ b/perl/Git/SVN/Editor.pm\n@@ -5,7 +5,6 @@ use warnings;\n use SVN::Core;\n use SVN::Delta;\n use Carp qw/croak/;\n-use IO::File;\n use Git qw/command command_oneline command_noisy command_output_pipe\n            command_input_pipe command_close_pipe\n            command_bidi_pipe command_close_bidi_pipe/;\n@@ -586,7 +585,7 @@ The interface will change as git-svn evolves.\n =head1 DEPENDENCIES\n \n Subversion perl bindings,\n-the core L<Carp> and L<IO::File> modules,\n+the core L<Carp> module,\n and git's L<Git> helper module.\n \n C<Git::SVN::Editor> has not been tested using callers other than\ndiff --git a/perl/Git/SVN/Fetcher.pm b/perl/Git/SVN/Fetcher.pm\nindex 10edb27..6b9c6e0 100644\n--- a/perl/Git/SVN/Fetcher.pm\n+++ b/perl/Git/SVN/Fetcher.pm\n@@ -7,7 +7,6 @@ use warnings;\n use SVN::Delta;\n use Carp qw/croak/;\n use File::Basename qw/dirname/;\n-use IO::File qw//;\n use Git qw/command command_oneline command_noisy command_output_pipe\n            command_input_pipe command_close_pipe\n            command_bidi_pipe command_close_bidi_pipe/;\n@@ -600,7 +599,7 @@ developing git-svn.\n =head1 DEPENDENCIES\n \n L<SVN::Delta> from the Subversion perl bindings,\n-the core L<Carp>, L<File::Basename>, and L<IO::File> modules,\n+the core L<Carp> and L<File::Basename> modules,\n and git's L<Git> helper module.\n \n C<Git::SVN::Fetcher> has not been tested using callers other than\ndiff --git a/perl/Git/SVN/Ra.pm b/perl/Git/SVN/Ra.pm\nindex 622535e..cf36b9d 100644\n--- a/perl/Git/SVN/Ra.pm\n+++ b/perl/Git/SVN/Ra.pm\n@@ -3,7 +3,6 @@ use vars qw/@ISA $config_dir $_ignore_refs_regex $_log_window_size/;\n use strict;\n use warnings;\n use Memoize;\n-use SVN::Client;\n use Git::SVN::Utils qw(\n \tcanonicalize_url\n \tcanonicalize_path\n@@ -42,6 +41,7 @@ END {\n }\n \n sub _auth_providers () {\n+\trequire SVN::Client;\n \tmy @rv = (\n \t  SVN::Client::get_simple_provider(),\n \t  SVN::Client::get_ssl_server_trust_file_provider(),\n@@ -247,7 +247,10 @@ sub get_log {\n \t$ret;\n }\n \n+# uncommon, only for ancient SVN (<= 1.4.2)\n sub trees_match {\n+\trequire IO::File;\n+\trequire SVN::Client;\n \tmy ($self, $url1, $rev1, $url2, $rev2) = @_;\n \tmy $ctx = SVN::Client->new(auth => _auth_providers);\n \tmy $out = IO::File->new_tmpfile;\n-- \nEW\n"},{"id":"254741","messageId":"CAKdxknzztdUU+nP28R82mor=Cf7tqt98h2NwN9o0BczzT8adQw@mail.gmail.com","threadId":"38349","inReplyTo":"20150114203012.GA26023@dcvr.yhbt.net","subject":"Re: git-svn metadata commands performance issue","fromName":"Niluge kiwi","fromEmail":"kiwiiii@gmail.com","sentAt":"2015-01-15T11:26:03Z","receivedAt":"2015-01-15T11:26:03Z","isPatch":false,"sender":{"key":"kiwiiii@gmail.com","avatar":null},"body":"On Wed, Jan 14, 2015 at 9:30 PM, Eric Wong <normalperson@yhbt.net> wrote:\n> Niluge kiwi <kiwiiii@gmail.com> wrote:\n>> Hi all,\n>>\n>> In magit (http://magit.github.io/), a popular git frontend within\n>> emacs, there is a git-svn frontend.  With a recent refactoring, it was\n>> discovered that git-svn metadata commands (like \"git-svn info\") are\n>> much slower than git ones:\n>> git svn info: 130-150ms (after warmup): get the svn revision and url.\n>> git svn rebase --dry-run: 150-170ms (after warmup): get the remote\n>> branch.\n>>\n>> Whereas in pure git:\n>> git rev-parse --abbrev-ref HEAD@{upstream}: 2-3ms (after warmup): get\n>> the remote branch\n>> Other git commands alike take all less than 10ms after warmup.\n>\n> Thanks for the bug report.  I actually see worse performance from\n> my old machines, but I'm not a very heavy git-svn user anymore.\n> 100ms is an eternity :<\n>\n>> This is an issue for the magit developers and users: just getting a\n>> git-svn status with some metadata easily take ~500ms, which is really\n>> slow for a UI. The equivalent UI with a pure git repository in magit\n>> takes much less than 100ms to generate although more than 30 git\n>> process are forked for it.\n>\n> How big is the parent process which forks the git commands?  On Linux at\n> least, fork() performance is negatively impacted by parent process\n> memory size.  To avoid spawn performance problems with large parent\n> processes, vfork() should be used, but there does not seem to be an\n> easy/portable way to use vfork() from Perl.\n\nMy emacs process uses ~500MB in RAM, but all the timings from my first\nemail come from commands run from bash that uses 8Mo in RAM. I don't\nsee any significant difference between timings from emacs and bash.\n\n>\n>> A previous version of magit-svn was much faster because it\n>> re-implemented the logic of git-svn from perl to elisp (the\n>> programming language in emacs), and to get the 3 previously mentioned\n>> values it took less than 10ms.\n>\n> I've never worked with elisp, but we can probably figure out why it's\n> faster.  Can you give us a pointer to the old elisp code?\n\nThe commit that removes the old code and replaces it by simple calls\nto git-svn is here:\nhttps://github.com/magit/magit/commit/b583e43a57dee164fdd53e0772220e32c1b3fab3\n\nIt may be faster because we don't have to start perl each time...\n\n>\n>> What could be done about this?\n>> Could git-svn performance be dramatically improved?\n>> Even git svn --version takes ~100ms, is perl the bottleneck?\n>\n> The Linux 'perf' tool reports much time is spent is from the Perl\n> parser.  So we may implement lazy loading, so simple commands such as\n> \"git svn info\" do not need to load and parse all the Perl code.\n>\n>> Or should each git-svn frontend developer re-implement the git-svn\n>> metadata commands themselves for better performance?\n>\n> I prefer git-svn be fast enough; but you're free to reimplement\n> and optimize your own code as you see fit.\n\nThe magit maintainer also prefers git-svn to be fast enough, because\nit means less code to write and maintain.\n\n>\n>> Also, wouldn't it be better for those frontend developers if there\n>> were some git-svn porcelain commands like git has? Fast, easy to parse\n>> and stable input & output format?\n>\n> Sure, but we don't know what you'd need beyond the current command set.\n> Of course we need to be careful about adding even more code to git-svn\n> as that impacts startup time, too.\nI'll ask the magit maintainer to weight in here for the ideal git-svn\ncommand line features.\n\nAlso, couldn't the code be split into multiple files to have no global\nimpact on startup time when adding new features?\n"},{"id":"254743","messageId":"87sifcdzul.fsf@fencepost.gnu.org","threadId":"38349","inReplyTo":"20150114203012.GA26023@dcvr.yhbt.net","subject":"Re: git-svn metadata commands performance issue","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2015-01-15T11:53:38Z","receivedAt":"2015-01-15T11:53:38Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Eric Wong <normalperson@yhbt.net> writes:\n\n> How big is the parent process which forks the git commands?  On Linux at\n> least, fork() performance is negatively impacted by parent process\n> memory size.\n\nHuh.  I thought with the advent of demand-paging, at the very least with\ncopy-on-write, this was supposed to be sort of a non-issue.\n\nThe old original UNIX version, in contrast, consisted of swapping out\nthe current process without removing the in-memory copy.  But since the\nin-memory copy then did the exec call and since usually the exec call\nwas happy about every page of free memory (we _are_ talking about\nsomething like 64kB of total available memory here), that tended to work\nreasonably well.\n\n-- \nDavid Kastrup\n"}]}