From: Kaartic Sivaraam Date: Thu, 19 Nov 2026 17:12:53 GMT Subject: Re: [PATCH] test-lib: run EXPENSIVE tests one at a time Message-ID: <3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com> In-Reply-To: On 11/19/26 15:37, Patrick Steinhardt wrote: > On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote: >> Recently, we've acquired quite a few expensive test in our test suite. > > s/test/tests/ Corrected it. > >> For instance, these commits add several tests that require >> multi-gigabytes: > > Multi-gigabytes what? > Fixed it! >> diff --git a/t/test-lib.sh b/t/test-lib.sh >> index c56654b7c5..4d29172c9d 100644 >> --- a/t/test-lib.sh >> +++ b/t/test-lib.sh >> @@ -1089,6 +1090,42 @@ fail_117 () { >> return 117 >> } >> >> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of > > s/can/may/ > Done. >> +# them running at the same time under "prove --jobs" can exhaust the > > I wouldn't single out `prove --jobs` here, as we know to run > concurrently without prove either via our plain Makefile or via Meson. > Makes sense. Will remove the part mentioning 'prove --jobs' specifically. >> +# machine. Allow only one at a time across all test scripts that share >> +# this TEST_OUTPUT_DIRECTORY. >> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock" >> +have_expensive_lock_= >> + >> +acquire_expensive_lock_ () { >> + mkdir -p "$TEST_RESULTS_DIR" || return 1 >> + while ! mkdir "$expensive_lock_" 2>/dev/null >> + do >> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null) >> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null >> + then >> + # The holder was killed without releasing the lock. >> + rm -rf "$expensive_lock_" >> + continue > > Okay, if the previous owner does not exist anymore we prune the lock. > Two racing processes which try to acquire the lock may race in a way > that the "/pid" file does not exist anymore because the other process > has already deleted it. But you protect against that by checking whether > we have been able to read a PID in the first place via `test -n`. Indeed. There might still be a race in the case when two racing processes (A and B) both see the same dead PID before either acts. A would have deleted the lock, created a new one, written its PID. B which still holds the stale PID would proceed to delete the lock which is actually the live lock held by A, create a new one and write its PID. This might happen if a holder was SIGKILLed and two or more scripts were waiting at that exact moment, so it's rare. So, I wouldn't complicate the code to handle this. In the worst case, we'd have 2 expensive test cases running in place of the one we intend. > >> + fi >> + if test -z "$expensive_lock_waited_" >> + then >> + say >&3 "waiting for another EXPENSIVE test to finish" > > Hmm. So file descriptor 3 points to stderr or to the test output file. > And if I remember correctly, we're indeed free to write to stderr > whatever data we want without breaking the TAP format? > Yep. With no '--verbose' or '--verbose-log' flags this message would just go to /dev/null. I suppose that will only happen when someone runs this manually. So, hopefully that's fine. >> + expensive_lock_waited_=t >> + fi >> + sleep 1 >> + done >> + expensive_lock_waited_= > > Should this be declared as a local variable? > Sure. Will make 'owner' local too. > Overall this is a rather pragmatic approach to solve the issue. I was > thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM > from EXPENSIVE_DISK, and have different locks for each of those. But I > doubt adding this kind of complexity would really be worth it in the > end. > Got it. Thank you for sharing your thoughts! > So I'm fine with the approach. Onward to green pipelines again! Well, > until the next thing breaks next week... > Totally. It feels is like we're constantly fire-fighting in the CI these days ... -- Sivaraam