git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] test-lib: run EXPENSIVE tests one at a time

From
Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Date
Nov 19, 2026, 17:12 UTC
Message-ID
<3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com>
In-Reply-To
<av7KRbOGWeV-Qof1@pks.im>
On 11/19/26 15:37, Patrick Steinhardt wrote:
> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
>> Recently, we've acquired quite a few expensive test in our test suite.
> > s/test/tests/
Corrected it.
> >> For instance, these commits add several tests that require
>> multi-gigabytes:
> > Multi-gigabytes what?
>
Fixed it!
Show 10 quoted lines
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c56654b7c5..4d29172c9d 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1089,6 +1090,42 @@ fail_117 () {
>>   	return 117
>>   }
>>   >> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
> > s/can/may/
> Done.
>> +# them running at the same time under "prove --jobs" can exhaust the
> > I wouldn't single out `prove --jobs` here, as we know to run
> concurrently without prove either via our plain Makefile or via Meson.
>
Makes sense. Will remove the part mentioning 'prove --jobs' specifically.
Show 20 quoted lines
>> +# machine. Allow only one at a time across all test scripts that share
>> +# this TEST_OUTPUT_DIRECTORY.
>> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
>> +have_expensive_lock_=
>> +
>> +acquire_expensive_lock_ () {
>> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
>> +	while ! mkdir "$expensive_lock_" 2>/dev/null
>> +	do
>> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
>> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
>> +		then
>> +			# The holder was killed without releasing the lock.
>> +			rm -rf "$expensive_lock_"
>> +			continue
> > Okay, if the previous owner does not exist anymore we prune the lock.
> Two racing processes which try to acquire the lock may race in a way
> that the "/pid" file does not exist anymore because the other process
> has already deleted it. But you protect against that by checking whether
> we have been able to read a PID in the first place via `test -n`.
Indeed. There might still be a race in the case when two racing processes (A and B) both see the same dead PID before either acts. A would have deleted the lock, created a new one, written its PID. B which still holds the stale PID would proceed to delete the lock which is actually the live lock held by A, create a new one and write its PID. This might happen if a holder was SIGKILLed and two or more scripts were waiting at that exact moment, so it's rare. So, I wouldn't complicate the code to handle this. In the worst case, we'd have 2 expensive test cases running in place of the one we intend.
Show 8 quoted lines
> >> +		fi
>> +		if test -z "$expensive_lock_waited_"
>> +		then
>> +			say >&3 "waiting for another EXPENSIVE test to finish"
> > Hmm. So file descriptor 3 points to stderr or to the test output file.
> And if I remember correctly, we're indeed free to write to stderr
> whatever data we want without breaking the TAP format?
> Yep. With no '--verbose' or '--verbose-log' flags this message would just go to /dev/null. I suppose that will only happen when someone runs this manually. So, hopefully that's fine.
Show 7 quoted lines
>> +			expensive_lock_waited_=t
>> +		fi
>> +		sleep 1
>> +	done
>> +	expensive_lock_waited_=
> > Should this be declared as a local variable?
>
Sure. Will make 'owner' local too.
Show 6 quoted lines
> Overall this is a rather pragmatic approach to solve the issue. I was
> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
> from EXPENSIVE_DISK, and have different locks for each of those. But I
> doubt adding this kind of complexity would really be worth it in the
> end.
> Got it. Thank you for sharing your thoughts!
> So I'm fine with the approach. Onward to green pipelines again! Well,
> until the next thing breaks next week...
> Totally. It feels is like we're constantly fire-fighting in the CI these days ...
-- 
Sivaraam
Previous: Patrick SteinhardtNext: Kaartic Sivaraam
Message 3 of 5 in “test-lib: run EXPENSIVE tests one at a time”
  1. test-lib: run EXPENSIVE tests one at a timeKaartic Sivaraam, Nov 19, 2026
  2. Patrick SteinhardtNov 19, 2026
  3. Kaartic SivaraamNov 19, 2026
  4. test-lib: run EXPENSIVE tests one at a timeKaartic Sivaraam, Nov 19, 2026
  5. Kaartic SivaraamNov 19, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.