Re: Slow pushes on 'pu' - even when up-to-date..

From: Stefan Beller <sbeller@google.com>
Date: Oct 3, 2016, 21:17 UTC
Message-ID: <CAGZ79kYXx+a+U3o6FPBy2mHOy4BxxuF97t6iyedPLk0Qw1Jx1A@mail.gmail.com>
In-Reply-To: <CA+55aFyos78qODyw57V=w13Ux5-8SvBqObJFAq22K+XKPWVbAA@mail.gmail.com>

On Mon, Oct 3, 2016 at 2:11 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:

Show 45 quoted lines

> This seems to be because I'm now on 'pu' as of a day or two ago in
> order to test the abbrev logic, but lookie here:
>
>     time git ls-remote ra.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux
>     .. shows all the branches and tags ..
>     real 0m0.655s
>     user 0m0.011s
>     sys 0m0.004s
>
> so the remote is fast to connect to, and with network connection
> overhead and everything, it's just over half a second. But then:
>
>     time git push ra.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux
>
> and it just sits there, and it's at 100% CPU the whole time, until it says:
>
>     Everything up-to-date
>
>     real 1m7.307s
>     user 1m2.761s
>     sys 0m0.475s
>
> Whaa? It took a *minute* of CPU time to decide that everything was up-to-date?
>
> That's just not right. The branch is entirely up-to-date:
>
>     git rev-parse HEAD
>     af79ad2b1f337a00aa150b993635b10bc68dc842
>
>     git ls-remote ra.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux master
>     af79ad2b1f337a00aa150b993635b10bc68dc842 refs/heads/master
>
> so there should be no need for any history walking. But it sure is
> doing *something*. A minute of CPU time on my machine is actually a
> pretty damn big deal.
>
> Looking at the trace, there's no IO - there's no back-and-forth about
> "I have this, do you have it?" or anything like that. The system call
> trace is just a lot of allocations, which I think means that "git
> push" is walking a lot of objects but not doing anything useful.
>
> I bisected it to commit 60cd66f "push: change submodule default to
> check", which makes little sense since I have no submodules, but there
> you go.. Apparently RECURSE_SUBMODULES_CHECK is just terminally
> broken.

Sorry for breaking you, too.

Junio complained about that when I was proposing the topic; but now the strategy seems to just wait until Heiko fixed the RECURSE_SUBMODULES_CHECK to be less broken.

* sb/push-make-submodule-check-the-default (2016-08-24) 1 commit
 - push: change submodule default to check

 Turn the default of "push.recurseSubmodules" to "check".

    Will hold to wait for hv/submodule-not-yet-pushed-fix

    This reveals that the "check" mode is too inefficient to use in
    real projects, even in ones as small as git itself.
    cf. <xmqqh9aaot49.fsf@gitster.mtv.corp.google.com>

Which itself is

* hv/submodule-not-yet-pushed-fix (2016-09-14) 2 commits
 - serialize collection of refs that contain submodule changes
 - serialize collection of changed submodules

 The code in "git push" to compute if any commit being pushed in the
 superproject binds a commit in a submodule that hasn't been pushed
 out was overly inefficient, making it unusable even for a small
 project that does not have any submodule but have a reasonable
 number of refs.

 The last two in the original series seem to break a few tests when
 queued to 'pu', and dropped for now.

 Waiting for a reroll.