git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] test-lib: run EXPENSIVE tests one at a time

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 19, 2026, 10:07 UTC
Message-ID
<av7KRbOGWeV-Qof1@pks.im>
In-Reply-To
<20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com>
On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
> Recently, we've acquired quite a few expensive test in our test suite.
s/test/tests/
> For instance, these commits add several tests that require
> multi-gigabytes:
Multi-gigabytes what?
[snip]
> In recent times, we've been noticing the sanitizer jobs (linux-leaks,
> linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
> our GitLab CI runners. On some digging, it turns out that this is
> because of multiple memory hungry tests running in parallel.

Yeah, our pipelines are currently a mess because of this, so I'd be really happy to have this resolved soonish.

Show 9 quoted lines
> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index c56654b7c5..4d29172c9d 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -1089,6 +1090,42 @@ fail_117 () {
>  	return 117
>  }
>  
> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
s/can/may/
> +# them running at the same time under "prove --jobs" can exhaust the

I wouldn't single out `prove --jobs` here, as we know to run concurrently without prove either via our plain Makefile or via Meson.

Show 15 quoted lines
> +# machine. Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
> +	while ! mkdir "$expensive_lock_" 2>/dev/null
> +	do
> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> +		then
> +			# The holder was killed without releasing the lock.
> +			rm -rf "$expensive_lock_"
> +			continue

Okay, if the previous owner does not exist anymore we prune the lock. Two racing processes which try to acquire the lock may race in a way that the "/pid" file does not exist anymore because the other process has already deleted it. But you protect against that by checking whether we have been able to read a PID in the first place via `test -n`.

> +		fi
> +		if test -z "$expensive_lock_waited_"
> +		then
> +			say >&3 "waiting for another EXPENSIVE test to finish"

Hmm. So file descriptor 3 points to stderr or to the test output file. And if I remember correctly, we're indeed free to write to stderr whatever data we want without breaking the TAP format?

Show 5 quoted lines
> +			expensive_lock_waited_=t
> +		fi
> +		sleep 1
> +	done
> +	expensive_lock_waited_=
Should this be declared as a local variable?

Overall this is a rather pragmatic approach to solve the issue. I was thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM from EXPENSIVE_DISK, and have different locks for each of those. But I doubt adding this kind of complexity would really be worth it in the end.

So I'm fine with the approach. Onward to green pipelines again! Well, until the next thing breaks next week...

Thanks!
Patrick
Previous: Kaartic SivaraamNext: Kaartic Sivaraam
Message 2 of 5 in “test-lib: run EXPENSIVE tests one at a time”
  1. test-lib: run EXPENSIVE tests one at a timeKaartic Sivaraam, Nov 19, 2026
  2. Patrick SteinhardtNov 19, 2026
  3. Kaartic SivaraamNov 19, 2026
  4. test-lib: run EXPENSIVE tests one at a timeKaartic Sivaraam, Nov 19, 2026
  5. Kaartic SivaraamNov 19, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.