Re: git-svn metadata commands performance issue

From: Niluge kiwi <kiwiiii@gmail.com>
Date: Jan 15, 2015, 11:26 UTC
Message-ID: <CAKdxknzztdUU+nP28R82mor=Cf7tqt98h2NwN9o0BczzT8adQw@mail.gmail.com>
In-Reply-To: <20150114203012.GA26023@dcvr.yhbt.net>

On Wed, Jan 14, 2015 at 9:30 PM, Eric Wong <normalperson@yhbt.net> wrote:

Show 31 quoted lines

> Niluge kiwi <kiwiiii@gmail.com> wrote:
>> Hi all,
>>
>> In magit (http://magit.github.io/), a popular git frontend within
>> emacs, there is a git-svn frontend.  With a recent refactoring, it was
>> discovered that git-svn metadata commands (like "git-svn info") are
>> much slower than git ones:
>> git svn info: 130-150ms (after warmup): get the svn revision and url.
>> git svn rebase --dry-run: 150-170ms (after warmup): get the remote
>> branch.
>>
>> Whereas in pure git:
>> git rev-parse --abbrev-ref HEAD@{upstream}: 2-3ms (after warmup): get
>> the remote branch
>> Other git commands alike take all less than 10ms after warmup.
>
> Thanks for the bug report.  I actually see worse performance from
> my old machines, but I'm not a very heavy git-svn user anymore.
> 100ms is an eternity :<
>
>> This is an issue for the magit developers and users: just getting a
>> git-svn status with some metadata easily take ~500ms, which is really
>> slow for a UI. The equivalent UI with a pure git repository in magit
>> takes much less than 100ms to generate although more than 30 git
>> process are forked for it.
>
> How big is the parent process which forks the git commands?  On Linux at
> least, fork() performance is negatively impacted by parent process
> memory size.  To avoid spawn performance problems with large parent
> processes, vfork() should be used, but there does not seem to be an
> easy/portable way to use vfork() from Perl.

My emacs process uses ~500MB in RAM, but all the timings from my first email come from commands run from bash that uses 8Mo in RAM. I don't see any significant difference between timings from emacs and bash.

Show 8 quoted lines

>
>> A previous version of magit-svn was much faster because it
>> re-implemented the logic of git-svn from perl to elisp (the
>> programming language in emacs), and to get the 3 previously mentioned
>> values it took less than 10ms.
>
> I've never worked with elisp, but we can probably figure out why it's
> faster.  Can you give us a pointer to the old elisp code?

The commit that removes the old code and replaces it by simple calls to git-svn is here: https://github.com/magit/magit/commit/b583e43a57dee164fdd53e0772220e32c1b3fab3

It may be faster because we don't have to start perl each time...

Show 14 quoted lines

>
>> What could be done about this?
>> Could git-svn performance be dramatically improved?
>> Even git svn --version takes ~100ms, is perl the bottleneck?
>
> The Linux 'perf' tool reports much time is spent is from the Perl
> parser.  So we may implement lazy loading, so simple commands such as
> "git svn info" do not need to load and parse all the Perl code.
>
>> Or should each git-svn frontend developer re-implement the git-svn
>> metadata commands themselves for better performance?
>
> I prefer git-svn be fast enough; but you're free to reimplement
> and optimize your own code as you see fit.

The magit maintainer also prefers git-svn to be fast enough, because it means less code to write and maintain.

Show 8 quoted lines

>
>> Also, wouldn't it be better for those frontend developers if there
>> were some git-svn porcelain commands like git has? Fast, easy to parse
>> and stable input & output format?
>
> Sure, but we don't know what you'd need beyond the current command set.
> Of course we need to be careful about adding even more code to git-svn
> as that impacts startup time, too.

I'll ask the magit maintainer to weight in here for the ideal git-svn command line features.

Also, couldn't the code be split into multiple files to have no global impact on startup time when adding new features?