Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

From: Junio C Hamano <gitster@pobox.com>
Date: Jan 19, 2016, 20:00 UTC
Message-ID: <xmqqmvs19w5n.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <1453195469-51696-1-git-send-email-larsxschneider@gmail.com>

larsxschneider@gmail.com writes:

Show 16 quoted lines

> From: Lars Schneider <larsxschneider@gmail.com>
>
> Use the Travis-CI cache feature to store prove test results and make them
> available in subsequent builds. This allows to run previously failed tests
> first and run remaining tests in slowest to fastest order. As a result it
> is less likely that Travis-CI needs to wait for a single test at the end
> which speeds up the test suite execution by ~2 min.
>
> Unfortunately the cache feature is only available (for free) on the
> Travis-CI Linux environment.
>
> Suggested-by: Jeff King <peff@peff.net>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  .travis.yml | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)

This is cute, but isn't it useful even outside Travis's context? I am not suggesting to touch anything other than .travis.yml file in this patch, but if I wanted to get the benefit from the idea in this patch when I run my tests manually, I can just tell prove to use the cached states, no?

IOW, I am confused by the beginning of the log message that says this is taking advantage of "the Travis-CI cache feature". This improvement looks to me like using the feature of "prove" that allows us to run slower tests first, and does not have much to do with Travis.

You are relying on the assumption that things under $HOME/ is stable while things under t/ (or in our source tree in general) are not, and I think that is a sensible thing to take advantage of, but are we sure that they are running in an environment where "ln -s" would work? Otherwise, it may be more robust to copy $HOME/.prove to t/.prove before starting to test and then copy it back once the tests are done.

Thanks.

Show 35 quoted lines

>
> diff --git a/.travis.yml b/.travis.yml
> index c3bf9c6..f34726b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,5 +1,9 @@
>  language: c
>
> +cache:
> +  directories:
> +    - $HOME/.prove-cache
> +
>  os:
>    - linux
>    - osx
> @@ -18,7 +22,7 @@ env:
>      - P4_VERSION="15.2"
>      - GIT_LFS_VERSION="1.1.0"
>      - DEFAULT_TEST_TARGET=prove
> -    - GIT_PROVE_OPTS="--timer --jobs 3"
> +    - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
>      - GIT_TEST_OPTS="--verbose --tee"
>      - CFLAGS="-g -O2 -Wall -Werror"
>      - GIT_TEST_CLONE_2GB=YesPlease
> @@ -67,6 +71,8 @@ before_install:
>      p4 -V | grep Rev.;
>      echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)";
>      git-lfs version;
> +    mkdir -p $HOME/.prove-cache;
> +    ln -s $HOME/.prove-cache/.prove t/.prove;
>
>  before_script: make --jobs=2
>
> --
> 2.5.1