[PATCH] test-lib: add ability to cap the runtime of tests

Add a GIT_TEST_TIMEOUT environment variable which optimistically sets an approximate upper limit on how long any one test is allowed to run.

Once the timeout is exceeded all remaining tests are skipped, no attempt is made to stop a long running test in-progress, or deal with the edge case of the epoch changing the epoch from under us by e.g. ntpd.

On my machine median runtime for a test is around 150ms, but 8 tests take more than 10 seconds to run, with t3404-rebase-interactive.sh taking 18 seconds.

On a machine with a large number of cores these long-tail tests become the limiting factor in how long it takes to run the entire test suite, even if it's run with "prove --state=slow,save". This is because the first long-running tests started at the very beginning will still be running by the time the rest of the test suite finishes.

Speeding up the test suite by simply cataloging and skipping tests that take longer than N seconds is a hassle to maintain, and entirely skips some tests which would be nice to at least partially run, e.g. instead of entirely skipping t3404-rebase-interactive.sh we can run it for N seconds and get at least some "git rebase -i" test coverage in a fast test run.

On a 56 core Xeon E5-2690 v4 @ 2.60GHz the runtime for the test suite is cut in half with GIT_TEST_TIMEOUT=10 under prove -j56 t[0-9]*.sh. Approximate time to run all the tests with various GIT_TEST_TIMEOUT settings[1]:

    N/A     30s
    20      20s
    10      15s
    5       12s
    1       12s

Setting a timeout lower than 5-10 seconds or so starts to reach diminishing returns, e.g. t7063-status-untracked-cache.sh always takes at least 6 seconds to run since it's blocking on a single "update-index --test-untracked-cache" command. So there's room for improvement, but this simple facility gives us most of the benefits.

The number of tests on the aforementioned machine which are run with the various timeouts[2]:

    N/A     16261
    20      16037
    10      14972
    5       13952
    1       8409

While running with a timeout of 10 seconds cuts the runtime in half, over 92% of the tests are still run. The test coverage is higher than that number indicates, just taking into account the many similar tests t0027-auto-crlf.sh runs brings it up to 95%.

1. for t in '' 20 10; do printf "%s\t" $t && (time GIT_TEST_TIMEOUT=$t prove -j$(parallel --number-of-cores) --state=slow,save t[0-9]*.sh) 2>&1 | grep ^real | grep -E -o '[0-9].*'; done
2. for t in '' 20 10 5 1; do printf "%s\t" $t && (time GIT_TEST_TIMEOUT=$t prove -j$(parallel --number-of-cores) --state=slow,save -v t[0-9]*.sh) 2>&1 | grep -e ^real -e '^1\.\.' | sed 's/^1\.\.//' | awk '{s+=$1} END {print s}'; done

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/test-lib.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)