Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

From: Junio C Hamano <gitster@pobox.com>
Date: Jan 19, 2016, 22:53 UTC
Message-ID: <xmqqio2p89mb.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <xmqqmvs19w5n.fsf@gitster.mtv.corp.google.com>

Junio C Hamano <gitster@pobox.com> writes:

Show 5 quoted lines

> This is cute, but isn't it useful even outside Travis's context?  I
> am not suggesting to touch anything other than .travis.yml file in
> this patch, but if I wanted to get the benefit from the idea in this
> patch when I run my tests manually, I can just tell prove to use the
> cached states, no?

It seems that exporting something like

    GIT_PROVE_OPTS="--timer --state=slow,save -j8"

when running "make DEFAULT_TEST_TARGET=prove test" does give me the same benefit by leaving the stats from the previous run in t/.prove when making the test scheduling decisions.

One thing I noticed but didn't dig further to fix was that this "prove --state" business did not seem to work well together with

    make T="...list of tests..." test

that limits the set of tests to perform.  For example:

    $ rm -f t/.prove
    $ make -j4 GIT_PROVE_OPTS="--timer --state=slow,save -j8" \
	   T="$( cd t && echo t0???-*.sh)" \
           DEFAULT_TEST_TARGET=prove test

runs all test in 0xxx series and populates t/.prove with them. And immediately after that, with t/.prove still there:

    $ make -j4 GIT_PROVE_OPTS="--timer --state=slow,save -j8" \
         DEFAULT_TEST_TARGET=prove \
         T=t0000-basic.sh test

does not limit the test to only 0000, but ends up running all the others recorded in t/.prove file, it seems.

I would imagine that this would not affect your use case negatively, as it is unlikely that your automated tests are skipping different set of tests in each run.