Re: Drastic jump in the time required for the test suite

From: Jeff King <peff@peff.net>
Date: Oct 19, 2016, 20:56 UTC
Message-ID: <20161019205638.m3ytxozzmeh47ml2@sigill.intra.peff.net>
In-Reply-To: <xmqqbmygmehv.fsf@gitster.mtv.corp.google.com>

On Wed, Oct 19, 2016 at 10:32:12AM -0700, Junio C Hamano wrote:

Show 7 quoted lines

> > Maybe we should start optimizing the tests...
> 
> Yup, two things that come to mind are to identify long ones and see
> if each of them can be split into two halves that can be run in
> parallel, and to go through the tests with fine toothed comb and
> remove the ones that test exactly the same thing as another test.
> The latter would be very time consuming, though.

FWIW, I have made attempts at "split long ones into two" before, and didn't come up with much. There _are_ some tests that are much longer than others[1], but they are not longer than the whole suite takes to run. So running in slow-to-fast order means they start first, are run in parallel with the other tests, and the CPUs stay relatively full through the whole run.

Of course YMMV; the long tests on Windows may be different, or proportionally much longer (I note the worst cases almost all involve rebase, which as a shell script is presumably worse on Windows than elsewhere). And of course any reasoning about slow-to-fast order does not apply if you are not using a tool to do that for you. :)

-Peff

[1] After running "make prove" (time are in seconds):

    $ perl -MYAML -e '
        $_ = do { local $/; <> };
        # prove puts this non-YAML cruft at the end
        s/\.\.\.$//s;

        my $t = YAML::Load($_)->{tests};
        print "$_->[1] $_->[0]\n" for
          sort { $b->[1] <=> $a->[1] }
          map { [$_, $t->{$_}->{elapsed}] }
          keys(%$t);
      ' .prove | head

    43.216765165329 t3404-rebase-interactive.sh
    30.6568658351898 t3421-rebase-topology-linear.sh
    27.92564702034 t9001-send-email.sh
    15.5906939506531 t9500-gitweb-standalone-no-errors.sh
    15.4882569313049 t6030-bisect-porcelain.sh
    14.487174987793 t7610-mergetool.sh
    13.8276169300079 t3425-rebase-topology-merges.sh
    12.7450480461121 t3426-rebase-submodule.sh
    12.4915001392365 t3415-rebase-autosquash.sh
    11.8122401237488 t5572-pull-submodule.sh