Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
On 11/19/26 15:37, Patrick Steinhardt wrote:
> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
>> Recently, we've acquired quite a few expensive test in our test suite.
> > s/test/tests/
> >> For instance, these commits add several tests that require
>> multi-gigabytes:
> > Multi-gigabytes what?
>
Show 10 quoted lines
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c56654b7c5..4d29172c9d 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1089,6 +1090,42 @@ fail_117 () {
>> return 117
>> }
>> >> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
> > s/can/may/
> Done.>> +# them running at the same time under "prove --jobs" can exhaust the
> > I wouldn't single out `prove --jobs` here, as we know to run
> concurrently without prove either via our plain Makefile or via Meson.
>
Makes sense. Will remove the part mentioning 'prove --jobs' specifically.
Show 20 quoted lines
>> +# machine. Allow only one at a time across all test scripts that share
>> +# this TEST_OUTPUT_DIRECTORY.
>> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
>> +have_expensive_lock_=
>> +
>> +acquire_expensive_lock_ () {
>> + mkdir -p "$TEST_RESULTS_DIR" || return 1
>> + while ! mkdir "$expensive_lock_" 2>/dev/null
>> + do
>> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
>> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
>> + then
>> + # The holder was killed without releasing the lock.
>> + rm -rf "$expensive_lock_"
>> + continue
> > Okay, if the previous owner does not exist anymore we prune the lock.
> Two racing processes which try to acquire the lock may race in a way
> that the "/pid" file does not exist anymore because the other process
> has already deleted it. But you protect against that by checking whether
> we have been able to read a PID in the first place via `test -n`.Indeed. There might still be a race in the case when two racing processes (A and B) both see the same dead PID before either acts. A would have deleted the lock, created a new one, written its PID. B which still holds the stale PID would proceed to delete the lock which is actually the live lock held by A, create a new one and write its PID. This might happen if a holder was SIGKILLed and two or more scripts were waiting at that exact moment, so it's rare. So, I wouldn't complicate the code to handle this. In the worst case, we'd have 2 expensive test cases running in place of the one we intend.
Show 8 quoted lines
> >> + fi
>> + if test -z "$expensive_lock_waited_"
>> + then
>> + say >&3 "waiting for another EXPENSIVE test to finish"
> > Hmm. So file descriptor 3 points to stderr or to the test output file.
> And if I remember correctly, we're indeed free to write to stderr
> whatever data we want without breaking the TAP format?
> Yep. With no '--verbose' or '--verbose-log' flags this message would just go to /dev/null. I suppose that will only happen when someone runs this manually. So, hopefully that's fine.
Show 7 quoted lines
>> + expensive_lock_waited_=t
>> + fi
>> + sleep 1
>> + done
>> + expensive_lock_waited_=
> > Should this be declared as a local variable?
>
Sure. Will make 'owner' local too.
Show 6 quoted lines
> Overall this is a rather pragmatic approach to solve the issue. I was
> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
> from EXPENSIVE_DISK, and have different locks for each of those. But I
> doubt adding this kind of complexity would really be worth it in the
> end.
> Got it. Thank you for sharing your thoughts!
> So I'm fine with the approach. Onward to green pipelines again! Well,
> until the next thing breaks next week...
> Totally. It feels is like we're constantly fire-fighting in the CI these days ...
--
Sivaraam