Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

From: Jeff King <peff@peff.net>
Date: Jan 19, 2016, 23:27 UTC
Message-ID: <20160119232710.GA31181@sigill.intra.peff.net>
In-Reply-To: <20160119230633.GA31142@sigill.intra.peff.net>

On Tue, Jan 19, 2016 at 06:06:33PM -0500, Jeff King wrote:

Show 14 quoted lines

> > It seems that exporting something like
> > 
> >     GIT_PROVE_OPTS="--timer --state=slow,save -j8" 
> > 
> > when running "make DEFAULT_TEST_TARGET=prove test" does give me the
> > same benefit by leaving the stats from the previous run in t/.prove
> > when making the test scheduling decisions.
> 
> Yes, I've been using this on my local machine for years (which is why I
> suggested it to Lars for the Travis build). I have also noticed that my
> test runs take about as much time as the longest-running test, and do
> not fully utilize all of my processors. I suspect we could drop the
> run-time of the test suite substantially by splitting a few of the
> longer tests.

Here are the numbers for that:

  $ time make ;# configured to use prove --state=slow,save -j16
  [...]
  real    0m47.035s
  user    1m6.884s
  sys     0m19.892s

  $ grep -v '^\.\.\.' .prove |
    perl -MYAML -e '
      local $/;
      $x = YAML::Load(<>)->{tests};
      print int($x->{$_}->{elapsed}), " $_\n" for keys(%$x)
    ' |
    sort -rn |
    head
  39 t3404-rebase-interactive.sh
  29 t3421-rebase-topology-linear.sh
  27 t9001-send-email.sh
  16 t9500-gitweb-standalone-no-errors.sh
  15 t3425-rebase-topology-merges.sh
  14 t6030-bisect-porcelain.sh
  13 t7610-mergetool.sh
  13 t5572-pull-submodule.sh
  13 t3426-rebase-submodule.sh
  12 t3415-rebase-autosquash.sh

So we're running t3404 for the majority of the time. I guess that doesn't tell us how full our pipelines are for the rest of the time, though. It could be worth splitting some of those long tests and seeing if that improves run-time, though.

-Peff