Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

From: Jeff King <peff@peff.net>
Date: Jan 19, 2016, 23:06 UTC
Message-ID: <20160119230633.GA31142@sigill.intra.peff.net>
In-Reply-To: <xmqqio2p89mb.fsf@gitster.mtv.corp.google.com>

On Tue, Jan 19, 2016 at 02:53:00PM -0800, Junio C Hamano wrote:

Show 15 quoted lines

> Junio C Hamano <gitster@pobox.com> writes:
> 
> > This is cute, but isn't it useful even outside Travis's context?  I
> > am not suggesting to touch anything other than .travis.yml file in
> > this patch, but if I wanted to get the benefit from the idea in this
> > patch when I run my tests manually, I can just tell prove to use the
> > cached states, no?
> 
> It seems that exporting something like
> 
>     GIT_PROVE_OPTS="--timer --state=slow,save -j8" 
> 
> when running "make DEFAULT_TEST_TARGET=prove test" does give me the
> same benefit by leaving the stats from the previous run in t/.prove
> when making the test scheduling decisions.

Yes, I've been using this on my local machine for years (which is why I suggested it to Lars for the Travis build). I have also noticed that my test runs take about as much time as the longest-running test, and do not fully utilize all of my processors. I suspect we could drop the run-time of the test suite substantially by splitting a few of the longer tests.

You also wrote earlier:

Show 5 quoted lines

> IOW, I am confused by the beginning of the log message that says
> this is taking advantage of "the Travis-CI cache feature".  This
> improvement looks to me like using the feature of "prove" that
> allows us to run slower tests first, and does not have much to do
> with Travis.

The interesting Travis feature we are using is that we are allowed to store some data from run-to-run. So we use the Travis feature that lets us use the prove feature. :)

Show 6 quoted lines

> One thing I noticed but didn't dig further to fix was that this
> "prove --state" business did not seem to work well together with
> 
>     make T="...list of tests..." test
> 
> that limits the set of tests to perform.

Right, it does not do what you want. The "prove --state" feature is not just about ordering, but also about selecting. When run via "make", we always give prove the full list of tests. But you can also do:

  prove --state=failed

manually to just run whatever failed on the last run.

I don't know if there is a way to tell prove "use the state for ordering, but don't otherwise select from it", which would do what you want above.

You can also note that if we ever delete a test script, it will still be mentioned in prove's state file. I think prove is smart enough to realize it went away and not bother you.

-Peff