Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

From: Lars Schneider <larsxschneider@gmail.com>
Date: Jan 20, 2016, 09:04 UTC
Message-ID: <711209F5-E034-459E-8E85-BF8BC32B2E86@gmail.com>
In-Reply-To: <xmqqmvs19w5n.fsf@gitster.mtv.corp.google.com>

On 19 Jan 2016, at 21:00, Junio C Hamano <gitster@pobox.com> wrote:

Show 38 quoted lines

> larsxschneider@gmail.com writes:
> 
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
>> Use the Travis-CI cache feature to store prove test results and make them
>> available in subsequent builds. This allows to run previously failed tests
>> first and run remaining tests in slowest to fastest order. As a result it
>> is less likely that Travis-CI needs to wait for a single test at the end
>> which speeds up the test suite execution by ~2 min.
>> 
>> Unfortunately the cache feature is only available (for free) on the
>> Travis-CI Linux environment.
>> 
>> Suggested-by: Jeff King <peff@peff.net>
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
>> ---
>> .travis.yml | 8 +++++++-
>> 1 file changed, 7 insertions(+), 1 deletion(-)
> 
> This is cute, but isn't it useful even outside Travis's context?  I
> am not suggesting to touch anything other than .travis.yml file in
> this patch, but if I wanted to get the benefit from the idea in this
> patch when I run my tests manually, I can just tell prove to use the
> cached states, no?
> 
> IOW, I am confused by the beginning of the log message that says
> this is taking advantage of "the Travis-CI cache feature".  This
> improvement looks to me like using the feature of "prove" that
> allows us to run slower tests first, and does not have much to do
> with Travis.
> 
> You are relying on the assumption that things under $HOME/ is stable
> while things under t/ (or in our source tree in general) are not,
> and I think that is a sensible thing to take advantage of, but are
> we sure that they are running in an environment where "ln -s" would
> work?  Otherwise, it may be more robust to copy $HOME/.prove to
> t/.prove before starting to test and then copy it back once the
> tests are done.

OK, looks like my wording was not ideal. One important thing to know is that $HOME is *not* stable. These TravisCI machines start *always* in a completely clean state. That's why prove cannot store and use it's cache. With the following statement I instruct Travis to cache my ".prove-cache" directory. As a consequence Travis CI will automatically restore this directory whenever it starts a new instance for the git job. It will also save the content of this directory when the job is done.

>> +cache:
>> +  directories:
>> +    - $HOME/.prove-cache

The Travis CI cache works only on a directory basis. Since I don't want to cache the entire /t directory I came up with the $HOME/.prove-cache directory. I also used a file link to leverage the automated save/restore feature for the $HOME/.prove-cache directory. If I would not use a link then I would need to copy the updated .prove file from t/ to .prove-cache after the test run.

Would the following first sentence for the commit message be less ambiguous?

"Use the Travis-CI cache feature to make the prove test results cache persistent across subsequent build jobs. This allows to run previously..."

Thanks, Lars

Show 35 quoted lines

>> 
>> diff --git a/.travis.yml b/.travis.yml
>> index c3bf9c6..f34726b 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -1,5 +1,9 @@
>> language: c
>> 
>> +cache:
>> +  directories:
>> +    - $HOME/.prove-cache
>> +
>> os:
>>   - linux
>>   - osx
>> @@ -18,7 +22,7 @@ env:
>>     - P4_VERSION="15.2"
>>     - GIT_LFS_VERSION="1.1.0"
>>     - DEFAULT_TEST_TARGET=prove
>> -    - GIT_PROVE_OPTS="--timer --jobs 3"
>> +    - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
>>     - GIT_TEST_OPTS="--verbose --tee"
>>     - CFLAGS="-g -O2 -Wall -Werror"
>>     - GIT_TEST_CLONE_2GB=YesPlease
>> @@ -67,6 +71,8 @@ before_install:
>>     p4 -V | grep Rev.;
>>     echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)";
>>     git-lfs version;
>> +    mkdir -p $HOME/.prove-cache;
>> +    ln -s $HOME/.prove-cache/.prove t/.prove;
>> 
>> before_script: make --jobs=2
>> 
>> --
>> 2.5.1