From: Kaartic Sivaraam Date: Thu, 19 Nov 2026 07:40:31 GMT Subject: [PATCH] test-lib: run EXPENSIVE tests one at a time Message-ID: <20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com> Recently, we've acquired quite a few expensive test in our test suite. For instance, these commits add several tests that require multi-gigabytes: - 9048b48fb (t1600-index: add expensive tests for over-long tree paths, 2026-10-16) added tests to t1600 - 288266c11d (xdiff: use size_t for merge buffer size computations, 2026-10-16) added tests to t4300 - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE, 2026-03-25), added tests to t4141 We also have the memory hungry 5GB filtered hash-object test in t1007. When GIT_TEST_LONG is set, "prove --jobs" happily runs several of these at once. On the 16GB GitLab CI runners this exhausts memory: the sanitizer jobs have been hanging in t1007 until they hit the job timeout ever since the new EXPENSIVE tests in t1600 started running alongside it. Running the two under a 16GB memory cap reproduces it, with the t1007 git process getting OOM-killed. GitHub CI runners are no exception and observe the same behaviour. Take a lock shared by all test scripts around each EXPENSIVE test, so only one of them runs at a time. The lock is a directory in TEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind symlink lock. It records the holder's PID so that a lock left behind by a script that was killed with SIGKILL gets taken over by the next waiter. It is released after the test and its cleanup finish, and from the EXIT trap when a script dies or is interrupted while holding it. Cheap tests in the same scripts keep running in parallel. Only the expensive ones wait for each other. Signed-off-by: Kaartic Sivaraam --- In recent times, we've been noticing the sanitizer jobs (linux-leaks, linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in our GitLab CI runners. On some digging, it turns out that this is because of multiple memory hungry tests running in parallel. So, I thought of suggesting this patch to serialize the expensive tests at least to avoid the putting the CI runners under memory pressure. The GitHub CI runners also observed the same behaviour. Both CI runners are happy with this change. GitHub: https://github.com/sivaraam/git/actions/runs/43281285339 GitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506 --- t/README | 4 +++- t/test-lib.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/t/README b/t/README index fd7e8504b5..c156dd3af6 100644 --- a/t/README +++ b/t/README @@ -117,7 +117,9 @@ appropriately before running "make". Short options can be bundled, i.e. -l:: --long-tests:: This causes additional long-running tests to be run (where - available), for more exhaustive testing. + available), for more exhaustive testing. These tests can use a + lot of memory and disk, so only one of them runs at a time, even + when several test scripts run in parallel (e.g. "prove --jobs"). -r:: --run=:: diff --git a/t/test-lib.sh b/t/test-lib.sh index c56654b7c5..4d29172c9d 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -784,6 +784,7 @@ die () { # test script run with '--immediate' fails, or when the user hits # ctrl-C, i.e. when 'test_done' is not invoked at all. test_atexit_handler || code=$? + release_expensive_lock_ if test -n "$GIT_EXIT_OK" then exit $code @@ -1089,6 +1090,42 @@ fail_117 () { return 117 } +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of +# them running at the same time under "prove --jobs" can exhaust the +# machine. Allow only one at a time across all test scripts that share +# this TEST_OUTPUT_DIRECTORY. +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock" +have_expensive_lock_= + +acquire_expensive_lock_ () { + mkdir -p "$TEST_RESULTS_DIR" || return 1 + while ! mkdir "$expensive_lock_" 2>/dev/null + do + owner=$(cat "$expensive_lock_/pid" 2>/dev/null) + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null + then + # The holder was killed without releasing the lock. + rm -rf "$expensive_lock_" + continue + fi + if test -z "$expensive_lock_waited_" + then + say >&3 "waiting for another EXPENSIVE test to finish" + expensive_lock_waited_=t + fi + sleep 1 + done + expensive_lock_waited_= + echo $$ >"$expensive_lock_/pid" + have_expensive_lock_=t +} + +release_expensive_lock_ () { + test -n "$have_expensive_lock_" || return 0 + rm -rf "$expensive_lock_" + have_expensive_lock_= +} + test_run_ () { test_cleanup=: expecting_failure=$2 @@ -1103,6 +1140,12 @@ test_run_ () { fi fi + case ",$test_prereq," in + *,EXPENSIVE,*) + acquire_expensive_lock_ + ;; + esac + setup_malloc_check test_eval_ "$1" eval_ret=$? @@ -1115,6 +1158,7 @@ test_run_ () { test_eval_ "$test_cleanup" teardown_malloc_check fi + release_expensive_lock_ if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE" then echo "" --- base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4 change-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6