Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Jan 20, 2016, 07:55 UTC
Message-ID: <alpine.DEB.2.20.1601200846010.2964@virtualbox>
In-Reply-To: <xmqqmvs19w5n.fsf@gitster.mtv.corp.google.com>

Hi Junio,

On Tue, 19 Jan 2016, Junio C Hamano wrote:

Show 25 quoted lines

> larsxschneider@gmail.com writes:
> 
> > From: Lars Schneider <larsxschneider@gmail.com>
> >
> > Use the Travis-CI cache feature to store prove test results and make
> > them available in subsequent builds. This allows to run previously
> > failed tests first and run remaining tests in slowest to fastest
> > order. As a result it is less likely that Travis-CI needs to wait for
> > a single test at the end which speeds up the test suite execution by
> > ~2 min.
> >
> > Unfortunately the cache feature is only available (for free) on the
> > Travis-CI Linux environment.
> >
> > Suggested-by: Jeff King <peff@peff.net>
> > Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> > ---
> >  .travis.yml | 8 +++++++-
> >  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> This is cute, but isn't it useful even outside Travis's context?  I am
> not suggesting to touch anything other than .travis.yml file in this
> patch, but if I wanted to get the benefit from the idea in this patch
> when I run my tests manually, I can just tell prove to use the cached
> states, no?

You are basically talking about prove's support to modify behavior based on previous runs. This patch does not introduce that support, it was introduced long ago: 5099b99 (test-lib: Adjust output to be valid TAP format, 2010-06-24). This commit also advertises that feature in t/README.

> IOW, I am confused by the beginning of the log message that says
> this is taking advantage of "the Travis-CI cache feature".

Lars' patch is really about Travis and its useful feature to retain state from previous runs. AFAICT this is the feature Lars is talking about, and it is absolutely necessary to make use of this prove feature (don't try this with BuildHive, for example, its workspaces are typically garbage-collected before the next CI run).

As such, I think it makes sense to talk about the Travis-CI feature in the commit message (and not about the prove feature because we introduced support for prove much, much earlier in Git's history, and advertised its benefits at that time, as I stated above).

Ciao, Dscho