git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2] test-lib: run EXPENSIVE tests one at a time

From
Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Date
Nov 19, 2026, 17:29 UTC
Message-ID
<20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com>
In-Reply-To
<20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com>

Recently, we've acquired quite a few expensive tests in our test suite. For instance, these commits add several tests that require multi-gigabytes of memory:

  - 9048b48fb (t1600-index: add expensive tests for over-long
    tree paths, 2026-10-16) added tests to t1600
  - 288266c11d (xdiff: use size_t for merge buffer size computations,
    2026-10-16) added tests to t4300
  - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
    2026-03-25), added tests to t4141
We also have the memory hungry 5GB filtered hash-object test in t1007.

When GIT_TEST_LONG is set, "prove --jobs" happily runs several of these at once. On the 16GB GitLab CI runners this exhausts memory: the sanitizer jobs have been hanging in t1007 until they hit the job timeout ever since the new EXPENSIVE tests in t1600 started running alongside it. Running the two under a 16GB memory cap reproduces it, with the t1007 git process getting OOM-killed.

GitHub CI runners are no exception and observe the same behaviour.

Take a lock shared by all test scripts around each EXPENSIVE test, so only one of them runs at a time. The lock is a directory in TEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind symlink lock. It records the holder's PID so that a lock left behind by a script that was killed with SIGKILL gets taken over by the next waiter. It is released after the test and its cleanup finish, and from the EXIT trap when a script dies or is interrupted while holding it.

Cheap tests in the same scripts keep running in parallel. Only the expensive ones wait for each other.

Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
---
In recent times, we've been noticing the sanitizer jobs (linux-leaks,
linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
our GitLab CI runners. On some digging, it turns out that this is
because of multiple memory hungry tests running in parallel.

So, I thought of suggesting this patch to serialize the expensive tests at least to avoid the putting the CI runners under memory pressure.

The GitHub CI runners also observed the same behaviour.
Both CI runners are happy with this change.
GitHub: https://github.com/sivaraam/git/actions/runs/43281285339
GitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506
---
Changes in v2:
- Corrected commit messages as per Patrick's suggestions
- Tweaked the shell function comment to avoid mentioning 'prove --jobs'
- Made the owner and waiting variables local within acquire_expensive_lock_
Link to v1: https://patch.msgid.link/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com
---
 t/README      |  4 +++-
 t/test-lib.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 1 deletion(-)
diff --git a/t/README b/t/README
index fd7e8504b5..c156dd3af6 100644
--- a/t/README
+++ b/t/README
@@ -117,7 +117,9 @@ appropriately before running "make". Short options can be bundled, i.e.
 -l::
 --long-tests::
 	This causes additional long-running tests to be run (where
-	available), for more exhaustive testing.
+	available), for more exhaustive testing. These tests can use a
+	lot of memory and disk, so only one of them runs at a time, even
+	when several test scripts run in parallel (e.g. "prove --jobs").
 
 -r::
 --run=<test-selector>::
diff --git a/t/test-lib.sh b/t/test-lib.sh
index c56654b7c5..193bf06aea 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -784,6 +784,7 @@ die () {
 	# test script run with '--immediate' fails, or when the user hits
 	# ctrl-C, i.e. when 'test_done' is not invoked at all.
 	test_atexit_handler || code=$?
+	release_expensive_lock_
 	if test -n "$GIT_EXIT_OK"
 	then
 		exit $code
@@ -1089,6 +1090,43 @@ fail_117 () {
 	return 117
 }
 
+# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
+# them running at the same time can exhaust the machine.
+#
+# Allow only one at a time across all test scripts that share
+# this TEST_OUTPUT_DIRECTORY.
+expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
+have_expensive_lock_=
+
+acquire_expensive_lock_ () {
+	local owner waited=
+	mkdir -p "$TEST_RESULTS_DIR" || return 1
+	while ! mkdir "$expensive_lock_" 2>/dev/null
+	do
+		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
+		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
+		then
+			# The holder was killed without releasing the lock.
+			rm -rf "$expensive_lock_"
+			continue
+		fi
+		if test -z "$waited"
+		then
+			say >&3 "waiting for another EXPENSIVE test to finish"
+			waited=t
+		fi
+		sleep 1
+	done
+	echo $$ >"$expensive_lock_/pid"
+	have_expensive_lock_=t
+}
+
+release_expensive_lock_ () {
+	test -n "$have_expensive_lock_" || return 0
+	rm -rf "$expensive_lock_"
+	have_expensive_lock_=
+}
+
 test_run_ () {
 	test_cleanup=:
 	expecting_failure=$2
@@ -1103,6 +1141,12 @@ test_run_ () {
 		fi
 	fi
 
+	case ",$test_prereq," in
+	*,EXPENSIVE,*)
+		acquire_expensive_lock_
+		;;
+	esac
+
 	setup_malloc_check
 	test_eval_ "$1"
 	eval_ret=$?
@@ -1115,6 +1159,7 @@ test_run_ () {
 		test_eval_ "$test_cleanup"
 		teardown_malloc_check
 	fi
+	release_expensive_lock_
 	if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE"
 	then
 		echo ""

---
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
change-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6
Previous: Kaartic SivaraamNext: Kaartic Sivaraam
Message 4 of 5 in “test-lib: run EXPENSIVE tests one at a time”
  1. test-lib: run EXPENSIVE tests one at a timeKaartic Sivaraam, Nov 19, 2026
  2. Patrick SteinhardtNov 19, 2026
  3. Kaartic SivaraamNov 19, 2026
  4. test-lib: run EXPENSIVE tests one at a timeKaartic Sivaraam, Nov 19, 2026
  5. Kaartic SivaraamNov 19, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.