Re: Drastic jump in the time required for the test suite

From: Junio C Hamano <gitster@pobox.com>
Date: Oct 19, 2016, 17:32 UTC
Message-ID: <xmqqbmygmehv.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <alpine.DEB.2.20.1610191049040.3847@virtualbox>

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> What I can also say for certain is that the version from yesterday (commit
> 4ef44ce) was the first one in a week that built successfully and hence
> reached the test phase, and it was the first version of `pu` ever to time
> out after running for 3h.

I am sympathetic, but I'd be lying if I said I can feel your pain.

Admittedly I do not run _all_ the tests (I certainly know that I exclude the ones behind EXPENSIVE prerequisite), but after every rebuilding of 'jch' and 'pu', I run the testsuite (and also rebuild docs) before pushing them out, and "make test && make doc && make install install-doc" run sequentially for the four integration branches finishes within 15 minutes, even when I run them after "make clean".

Perhaps the recent change to run the tests in parallel from slower to faster under prove may be helping my case.

> Maybe we should start optimizing the tests...

Yup, two things that come to mind are to identify long ones and see if each of them can be split into two halves that can be run in parallel, and to go through the tests with fine toothed comb and remove the ones that test exactly the same thing as another test. The latter would be very time consuming, though.