Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time
- From
Junio C Hamano <gitster@pobox.com>
- Date
- Nov 20, 2026, 06:28 UTC
- Message-ID
- <xmqqpkw0rp8i.fsf@gitster.g>
- In-Reply-To
- <20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com>
Kaartic Sivaraam <kaartic.sivaraam@gmail.com> writes:
Show 6 quoted lines
> Recently, we've acquired quite a few expensive tests in our test suite. > For instance, these commits add several tests that require > multi-gigabytes of memory: > > - 9048b48fb (t1600-index: add expensive tests for over-long > tree paths, 2026-10-16) added tests to t1600
No such commit X-<. Perhaps 89048b48fb (t1600-index: add expensive tests for over-long tree paths, 2026-10-16).
Show 5 quoted lines
> - 288266c11d (xdiff: use size_t for merge buffer size computations, > 2026-10-16) added tests to t4300 > > - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE, > 2026-03-25), added tests to t4141
Inconsistent use of "," after the (close parentheses).
Show 30 quoted lines
> +# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
> +# them running at the same time can exhaust the machine.
> +#
> +# Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> + local owner waited=
> + mkdir -p "$TEST_RESULTS_DIR" || return 1
> + while ! mkdir "$expensive_lock_" 2>/dev/null
> + do
> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> + then
> + # The holder was killed without releasing the lock.
> + rm -rf "$expensive_lock_"
> + continue
> + fi
> + if test -z "$waited"
> + then
> + say >&3 "waiting for another EXPENSIVE test to finish"
> + waited=t
> + fi
> + sleep 1
> + done
> + echo $$ >"$expensive_lock_/pid"
> + have_expensive_lock_=t
> +}Hmph, is the above safe when multiple processes are working with expensive tests during normal (i.e., not involving SIGKILL) operation, or does it still have TOCTOU race on the lock file?
* T1 holds $expensive_lock_, finishes its last EXPENSIVE test, calls release_expensive_lock_ (rm -rf "$expensive_lock_"), and exits.
* Waiter T2 read owner=$(cat "$expensive_lock_/pid" 2>/dev/null) (obtaining T1's PID) just before T1 removed the directory.
* Waiter T3 runs mkdir "$expensive_lock_", succeeds, writes T3's PID to pid, and begins its EXPENSIVE test.
* T2 now checks test -n "$owner" && ! kill -0 "$owner" 2>/dev/null against T1's exited PID, finds that T1 is gone, runs rm -rf "$expensive_lock_", which deletes T3's active lock. It loops, succeeds on mkdir "$expensive_lock_", and runs its EXPENSIVE test concurrently with T3.
Also, between the time you successfully mkdir the lock directory and the time you leave this function, you can be killed by a signal with INT, TERM, or HUP, for which t/test-lib.sh installs a hander to call exit, and there is a "trap die EXIT".
You may manage to create $expensive_lock_ directory, open pid file for writing, and then get killed before you echo $$ to the pid file or assign 't' to have_expensive_lock_ variable.
When your die handler calls release_expensive_lock_, you would leave with the directory behind and exit. You may not have pid file, or the pid file may be empty. The waiting processes will not run the kill -0 "$owner" part (because they do not know who you are that held the lock and left before writing $$ in pid), the waiters will wait for the mkdir to successfully create the directory that already exists and nobody will ever remove, no?
Between the TOCTOU race allowing multiple processes entering the expensive region and deadlocking denying everybody from entering the expensive region, the latter is far worse, of course.
Show 5 quoted lines
> +release_expensive_lock_ () {
> + test -n "$have_expensive_lock_" || return 0
> + rm -rf "$expensive_lock_"
> + have_expensive_lock_=
> +}