Re: Drastic jump in the time required for the test suite

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Oct 20, 2016, 10:17 UTC
Message-ID: <alpine.DEB.2.20.1610201154070.3264@virtualbox>
In-Reply-To: <xmqqbmygmehv.fsf@gitster.mtv.corp.google.com>

Hi Junio,

On Wed, 19 Oct 2016, Junio C Hamano wrote:

Show 8 quoted lines

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > What I can also say for certain is that the version from yesterday (commit
> > 4ef44ce) was the first one in a week that built successfully and hence
> > reached the test phase, and it was the first version of `pu` ever to time
> > out after running for 3h.
> 
> I am sympathetic, but I'd be lying if I said I can feel your pain.

Obviously.

Show 7 quoted lines

> Admittedly I do not run _all_ the tests (I certainly know that I
> exclude the ones behind EXPENSIVE prerequisite), but after every
> rebuilding of 'jch' and 'pu', I run the testsuite (and also rebuild
> docs) before pushing them out, and "make test && make doc && make
> install install-doc" run sequentially for the four integration
> branches finishes within 15 minutes, even when I run them after
> "make clean".

You have the luxury of a system that runs shell scripts fast.

Or maybe I should put it differently: you set up the test suite in such a way that it runs fast on your system, exploiting the fact that shell scripts run fast there.

If you want to know just how harmful this reliance on shell scripting is to our goal of keeping Git portable: already moving from Linux to MacOSX costs you roughly 3x as long to run the build & test (~12mins vs ~36mins for GCC, according to https://travis-ci.org/git/git/builds/159125647).

Again, I have to repeat myself: this is not good.

> Perhaps the recent change to run the tests in parallel from slower
> to faster under prove may be helping my case.

No, you misunderstand. The tests are *already* run in parallel. And running them from slower to faster would only make a difference if the last tests were not requiring roughly the same time to run.

Also, I cannot use prove(1) because it proved to be unreliable in my setup (it does hang from time to time, for no good reason whatsoever, which would only make my situation worse, of course).

Show 7 quoted lines

> > Maybe we should start optimizing the tests...
> 
> Yup, two things that come to mind are to identify long ones and see
> if each of them can be split into two halves that can be run in
> parallel, and to go through the tests with fine toothed comb and
> remove the ones that test exactly the same thing as another test.
> The latter would be very time consuming, though.

Again, you misunderstand.

The problem is not whether or not to run the tests in parallel.

The problem is that our tests take an insanely long time to run. That is a big problem, it is no good, tests are only useful if they are cheap enough to run all the time.

So the only thing that would really count as an improvement would be to change the test suite in such a manner that it relies more on helpers in t/helper/ and less on heavy-duty shell scripting.

Of course, if you continue to resist (because the problem is obviously not affecting you personally, so why would you care), I won't even try to find the time to start on that project.

Ciao, Dscho

P.S.: After increasing the time-out to 240 minutes, the test suite still times out. I investigated a little bit and it appears that t6030-bisect-porcelain.sh now hangs in the `git bisect next` call of its "2 - bisect starts with only one bad" test case. This is specific to Windows, I cannot reproduce that problem on Linux, and I will keep you posted on my progress.