git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH] ci: avoid running the test suite _twice_

From
Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com>
Date
Nov 13, 2023, 17:00 UTC
Message-ID
<pull.1613.git.1699894837844.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>

This is a late amendment of 19ec39aab54 (ci: stop linking the `prove` cache, 2022-07-10), fixing a bug that had been hidden so far.

The bug is that the `.prove` cache stores information about previous `prove` runs (`save`) and uses them (`slow`, to run the tests in the order from longer-running to shorter ones).

This bug can cause some surprising behavior: when the Prove cache contains a reference to a test script, subsequent `prove` runs (with `--state=slow`) will run the same test script again even if said script is not specified on the `prove` command-line!

So far, this bug did not matter. Right until d8f416bbb87c (ci: run unit tests in CI, 2023-11-09) it did not matter.

But starting with that commit, we run `prove` _twice_ in CI, and with completely different sets of tests to run. Due to the bug, the second invocation re-runs all of the tests that were already run as part of the first invocation. This not only wastes build minutes, it also frequently causes the `osx-*` jobs to fail because they already take a long time and now are likely to run into a timeout.

The worst part about it is that there is actually no benefit to keep running with `--state=slow,save`, ever since we decided no longer to try to reuse the Prove cache between CI runs.

So let's just drop that Prove option and live happily ever after.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
    ci: avoid running the test suite twice
    
    This
    [https://github.com/git/git/actions/runs/6845537889/job/18614840166] is
    an example of a osx-* job that times out. Here
    [https://github.com/git/git/actions/runs/6845537889/job/18614840166#step:4:839],
    it is running t0013, and here
    [https://github.com/git/git/actions/runs/6845537889/job/18614840166#step:4:2765],
    it is run again (in the middle of the entire test suite, as part of make
    unit-tests).
    
    While this tries to fix a bug uncovered by js/doc-unit-tests, to avoid
    merge conflicts, this is based on ps/ci-gitlab.
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1613%2Fdscho%2Favoid-running-test-suite-twice-in-ci-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1613/dscho/avoid-running-test-suite-twice-in-ci-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1613
 ci/lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ci/lib.sh b/ci/lib.sh
index 6dfc90d7f53..307a8df0b5a 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -281,7 +281,7 @@ else
 fi
 
 MAKEFLAGS="$MAKEFLAGS --jobs=$JOBS"
-GIT_PROVE_OPTS="--timer --jobs $JOBS --state=failed,slow,save"
+GIT_PROVE_OPTS="--timer --jobs $JOBS"
 
 GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x"
 case "$CI_OS_NAME" in

base-commit: 0e3b67e2aa25edb7e1a5c999c87b52a7b3a7649a
-- 
gitgitgadget
Next: Jeff King
Message 1 of 12 in “ci: avoid running the test suite _twice_”
  1. ci: avoid running the test suite _twice_Johannes Schindelin via GitGitGadget, Nov 13, 2023
  2. Jeff KingNov 13, 2023
  3. Junio C HamanoNov 13, 2023
  4. Josh SteadmonNov 14, 2023
  5. Junio C HamanoNov 15, 2023
  6. Josh SteadmonNov 15, 2023
  7. Feasibility of folding `unit-tests` into `make test`, was Re: [PATCH] ci: avoid running the test suite _twice_Johannes Schindelin, Nov 16, 2023
  8. phillip.wood123@gmail.comNov 16, 2023
  9. Josh SteadmonJan 4, 2024
  10. Jeff KingNov 16, 2023
  11. ci: avoid running the test suite _twice_Johannes Schindelin via GitGitGadget, Nov 13, 2023
  12. Junio C HamanoNov 14, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.