# [PATCH] test-lib: run EXPENSIVE tests one at a time

6 messages from 2026-11-19 to 2026-11-20. Participants: Kaartic Sivaraam, Patrick Steinhardt, Junio C Hamano.
Thread: https://gitlist.dev/t/66877

## Kaartic Sivaraam, 2026-11-19 07:40

Subject: [PATCH] test-lib: run EXPENSIVE tests one at a time
Message-ID: <20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com>
URL: https://gitlist.dev/e/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0%40gmail.com

```
Recently, we've acquired quite a few expensive test in our test suite.
For instance, these commits add several tests that require
multi-gigabytes:

  - 9048b48fb (t1600-index: add expensive tests for over-long
    tree paths, 2026-10-16) added tests to t1600

  - 288266c11d (xdiff: use size_t for merge buffer size computations,
    2026-10-16) added tests to t4300

  - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
    2026-03-25), added tests to t4141

We also have the memory hungry 5GB filtered hash-object test in t1007.

When GIT_TEST_LONG is set, "prove --jobs" happily runs several of these
at once. On the 16GB GitLab CI runners this exhausts memory: the
sanitizer jobs have been hanging in t1007 until they hit the job
timeout ever since the new EXPENSIVE tests in t1600 started running
alongside it. Running the two under a 16GB memory cap reproduces it,
with the t1007 git process getting OOM-killed.

GitHub CI runners are no exception and observe the same behaviour.

Take a lock shared by all test scripts around each EXPENSIVE test, so
only one of them runs at a time. The lock is a directory in
TEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind
symlink lock. It records the holder's PID so that a lock left behind by
a script that was killed with SIGKILL gets taken over by the next
waiter. It is released after the test and its cleanup finish, and from
the EXIT trap when a script dies or is interrupted while holding it.

Cheap tests in the same scripts keep running in parallel. Only the
expensive ones wait for each other.

Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
---
In recent times, we've been noticing the sanitizer jobs (linux-leaks,
linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
our GitLab CI runners. On some digging, it turns out that this is
because of multiple memory hungry tests running in parallel.

So, I thought of suggesting this patch to serialize the expensive tests
at least to avoid the putting the CI runners under memory pressure.

The GitHub CI runners also observed the same behaviour.

Both CI runners are happy with this change.

GitHub: https://github.com/sivaraam/git/actions/runs/43281285339
GitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506
---
 t/README      |  4 +++-
 t/test-lib.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/t/README b/t/README
index fd7e8504b5..c156dd3af6 100644
--- a/t/README
+++ b/t/README
@@ -117,7 +117,9 @@ appropriately before running "make". Short options can be bundled, i.e.
 -l::
 --long-tests::
 	This causes additional long-running tests to be run (where
-	available), for more exhaustive testing.
+	available), for more exhaustive testing. These tests can use a
+	lot of memory and disk, so only one of them runs at a time, even
+	when several test scripts run in parallel (e.g. "prove --jobs").
 
 -r::
 --run=<test-selector>::
diff --git a/t/test-lib.sh b/t/test-lib.sh
index c56654b7c5..4d29172c9d 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -784,6 +784,7 @@ die () {
 	# test script run with '--immediate' fails, or when the user hits
 	# ctrl-C, i.e. when 'test_done' is not invoked at all.
 	test_atexit_handler || code=$?
+	release_expensive_lock_
 	if test -n "$GIT_EXIT_OK"
 	then
 		exit $code
@@ -1089,6 +1090,42 @@ fail_117 () {
 	return 117
 }
 
+# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
+# them running at the same time under "prove --jobs" can exhaust the
+# machine. Allow only one at a time across all test scripts that share
+# this TEST_OUTPUT_DIRECTORY.
+expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
+have_expensive_lock_=
+
+acquire_expensive_lock_ () {
+	mkdir -p "$TEST_RESULTS_DIR" || return 1
+	while ! mkdir "$expensive_lock_" 2>/dev/null
+	do
+		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
+		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
+		then
+			# The holder was killed without releasing the lock.
+			rm -rf "$expensive_lock_"
+			continue
+		fi
+		if test -z "$expensive_lock_waited_"
+		then
+			say >&3 "waiting for another EXPENSIVE test to finish"
+			expensive_lock_waited_=t
+		fi
+		sleep 1
+	done
+	expensive_lock_waited_=
+	echo $$ >"$expensive_lock_/pid"
+	have_expensive_lock_=t
+}
+
+release_expensive_lock_ () {
+	test -n "$have_expensive_lock_" || return 0
+	rm -rf "$expensive_lock_"
+	have_expensive_lock_=
+}
+
 test_run_ () {
 	test_cleanup=:
 	expecting_failure=$2
@@ -1103,6 +1140,12 @@ test_run_ () {
 		fi
 	fi
 
+	case ",$test_prereq," in
+	*,EXPENSIVE,*)
+		acquire_expensive_lock_
+		;;
+	esac
+
 	setup_malloc_check
 	test_eval_ "$1"
 	eval_ret=$?
@@ -1115,6 +1158,7 @@ test_run_ () {
 		test_eval_ "$test_cleanup"
 		teardown_malloc_check
 	fi
+	release_expensive_lock_
 	if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE"
 	then
 		echo ""

---
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
change-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6



```

## Patrick Steinhardt, 2026-11-19 10:07

Subject: Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
Message-ID: <av7KRbOGWeV-Qof1@pks.im>
URL: https://gitlist.dev/e/av7KRbOGWeV-Qof1%40pks.im
In-Reply-To: <20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com>

```
On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
> Recently, we've acquired quite a few expensive test in our test suite.

s/test/tests/

> For instance, these commits add several tests that require
> multi-gigabytes:

Multi-gigabytes what?

[snip]
> In recent times, we've been noticing the sanitizer jobs (linux-leaks,
> linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
> our GitLab CI runners. On some digging, it turns out that this is
> because of multiple memory hungry tests running in parallel.

Yeah, our pipelines are currently a mess because of this, so I'd be
really happy to have this resolved soonish.

> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index c56654b7c5..4d29172c9d 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -1089,6 +1090,42 @@ fail_117 () {
>  	return 117
>  }
>  
> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of

s/can/may/

> +# them running at the same time under "prove --jobs" can exhaust the

I wouldn't single out `prove --jobs` here, as we know to run
concurrently without prove either via our plain Makefile or via Meson.

> +# machine. Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
> +	while ! mkdir "$expensive_lock_" 2>/dev/null
> +	do
> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> +		then
> +			# The holder was killed without releasing the lock.
> +			rm -rf "$expensive_lock_"
> +			continue

Okay, if the previous owner does not exist anymore we prune the lock.
Two racing processes which try to acquire the lock may race in a way
that the "/pid" file does not exist anymore because the other process
has already deleted it. But you protect against that by checking whether
we have been able to read a PID in the first place via `test -n`.

> +		fi
> +		if test -z "$expensive_lock_waited_"
> +		then
> +			say >&3 "waiting for another EXPENSIVE test to finish"

Hmm. So file descriptor 3 points to stderr or to the test output file.
And if I remember correctly, we're indeed free to write to stderr
whatever data we want without breaking the TAP format?

> +			expensive_lock_waited_=t
> +		fi
> +		sleep 1
> +	done
> +	expensive_lock_waited_=

Should this be declared as a local variable?

Overall this is a rather pragmatic approach to solve the issue. I was
thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
from EXPENSIVE_DISK, and have different locks for each of those. But I
doubt adding this kind of complexity would really be worth it in the
end.

So I'm fine with the approach. Onward to green pipelines again! Well,
until the next thing breaks next week...

Thanks!

Patrick


```

## Kaartic Sivaraam, 2026-11-19 17:12

Subject: Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
Message-ID: <3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com>
URL: https://gitlist.dev/e/3229c599-bdae-46ef-bcad-9557965b4b43%40gmail.com
In-Reply-To: <av7KRbOGWeV-Qof1@pks.im>

```
On 11/19/26 15:37, Patrick Steinhardt wrote:
> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
>> Recently, we've acquired quite a few expensive test in our test suite.
> > s/test/tests/

Corrected it.

> >> For instance, these commits add several tests that require
>> multi-gigabytes:
> > Multi-gigabytes what?
>

Fixed it!
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c56654b7c5..4d29172c9d 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1089,6 +1090,42 @@ fail_117 () {
>>   	return 117
>>   }
>>   >> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
> > s/can/may/
> Done.

>> +# them running at the same time under "prove --jobs" can exhaust the
> > I wouldn't single out `prove --jobs` here, as we know to run
> concurrently without prove either via our plain Makefile or via Meson.
>

Makes sense. Will remove the part mentioning 'prove --jobs' specifically.

>> +# machine. Allow only one at a time across all test scripts that share
>> +# this TEST_OUTPUT_DIRECTORY.
>> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
>> +have_expensive_lock_=
>> +
>> +acquire_expensive_lock_ () {
>> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
>> +	while ! mkdir "$expensive_lock_" 2>/dev/null
>> +	do
>> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
>> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
>> +		then
>> +			# The holder was killed without releasing the lock.
>> +			rm -rf "$expensive_lock_"
>> +			continue
> > Okay, if the previous owner does not exist anymore we prune the lock.
> Two racing processes which try to acquire the lock may race in a way
> that the "/pid" file does not exist anymore because the other process
> has already deleted it. But you protect against that by checking whether
> we have been able to read a PID in the first place via `test -n`.

Indeed. There might still be a race in the case when two racing processes (A and B) both see the same dead PID before either acts. A would have deleted the lock, created a new one, written its PID. B which still holds the stale PID would proceed to delete the lock which is actually the live lock held by A, create a new one and write its PID. This might happen if a holder was SIGKILLed and two or more scripts were waiting at that exact moment, so it's rare. So, I wouldn't complicate the code to handle this. In the worst case, we'd have 2 expensive test cases running in place of the one we intend.

> >> +		fi
>> +		if test -z "$expensive_lock_waited_"
>> +		then
>> +			say >&3 "waiting for another EXPENSIVE test to finish"
> > Hmm. So file descriptor 3 points to stderr or to the test output file.
> And if I remember correctly, we're indeed free to write to stderr
> whatever data we want without breaking the TAP format?
> Yep. With no '--verbose' or '--verbose-log' flags this message would just go to /dev/null. I suppose that will only happen when someone runs this manually. So, hopefully that's fine.

>> +			expensive_lock_waited_=t
>> +		fi
>> +		sleep 1
>> +	done
>> +	expensive_lock_waited_=
> > Should this be declared as a local variable?
>

Sure. Will make 'owner' local too.

> Overall this is a rather pragmatic approach to solve the issue. I was
> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
> from EXPENSIVE_DISK, and have different locks for each of those. But I
> doubt adding this kind of complexity would really be worth it in the
> end.
> Got it. Thank you for sharing your thoughts!

> So I'm fine with the approach. Onward to green pipelines again! Well,
> until the next thing breaks next week...
> Totally. It feels is like we're constantly fire-fighting in the CI these days ...

-- 
Sivaraam



```

## Kaartic Sivaraam, 2026-11-19 17:29

Subject: [PATCH v2] test-lib: run EXPENSIVE tests one at a time
Message-ID: <20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com>
URL: https://gitlist.dev/e/20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d%40gmail.com
In-Reply-To: <20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com>

```
Recently, we've acquired quite a few expensive tests in our test suite.
For instance, these commits add several tests that require
multi-gigabytes of memory:

  - 9048b48fb (t1600-index: add expensive tests for over-long
    tree paths, 2026-10-16) added tests to t1600

  - 288266c11d (xdiff: use size_t for merge buffer size computations,
    2026-10-16) added tests to t4300

  - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
    2026-03-25), added tests to t4141

We also have the memory hungry 5GB filtered hash-object test in t1007.

When GIT_TEST_LONG is set, "prove --jobs" happily runs several of these
at once. On the 16GB GitLab CI runners this exhausts memory: the
sanitizer jobs have been hanging in t1007 until they hit the job
timeout ever since the new EXPENSIVE tests in t1600 started running
alongside it. Running the two under a 16GB memory cap reproduces it,
with the t1007 git process getting OOM-killed.

GitHub CI runners are no exception and observe the same behaviour.

Take a lock shared by all test scripts around each EXPENSIVE test, so
only one of them runs at a time. The lock is a directory in
TEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind
symlink lock. It records the holder's PID so that a lock left behind by
a script that was killed with SIGKILL gets taken over by the next
waiter. It is released after the test and its cleanup finish, and from
the EXIT trap when a script dies or is interrupted while holding it.

Cheap tests in the same scripts keep running in parallel. Only the
expensive ones wait for each other.

Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
---
In recent times, we've been noticing the sanitizer jobs (linux-leaks,
linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
our GitLab CI runners. On some digging, it turns out that this is
because of multiple memory hungry tests running in parallel.

So, I thought of suggesting this patch to serialize the expensive tests
at least to avoid the putting the CI runners under memory pressure.

The GitHub CI runners also observed the same behaviour.

Both CI runners are happy with this change.

GitHub: https://github.com/sivaraam/git/actions/runs/43281285339
GitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506
---
Changes in v2:
- Corrected commit messages as per Patrick's suggestions
- Tweaked the shell function comment to avoid mentioning 'prove --jobs'
- Made the owner and waiting variables local within acquire_expensive_lock_

Link to v1: https://patch.msgid.link/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com
---
 t/README      |  4 +++-
 t/test-lib.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 1 deletion(-)

diff --git a/t/README b/t/README
index fd7e8504b5..c156dd3af6 100644
--- a/t/README
+++ b/t/README
@@ -117,7 +117,9 @@ appropriately before running "make". Short options can be bundled, i.e.
 -l::
 --long-tests::
 	This causes additional long-running tests to be run (where
-	available), for more exhaustive testing.
+	available), for more exhaustive testing. These tests can use a
+	lot of memory and disk, so only one of them runs at a time, even
+	when several test scripts run in parallel (e.g. "prove --jobs").
 
 -r::
 --run=<test-selector>::
diff --git a/t/test-lib.sh b/t/test-lib.sh
index c56654b7c5..193bf06aea 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -784,6 +784,7 @@ die () {
 	# test script run with '--immediate' fails, or when the user hits
 	# ctrl-C, i.e. when 'test_done' is not invoked at all.
 	test_atexit_handler || code=$?
+	release_expensive_lock_
 	if test -n "$GIT_EXIT_OK"
 	then
 		exit $code
@@ -1089,6 +1090,43 @@ fail_117 () {
 	return 117
 }
 
+# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
+# them running at the same time can exhaust the machine.
+#
+# Allow only one at a time across all test scripts that share
+# this TEST_OUTPUT_DIRECTORY.
+expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
+have_expensive_lock_=
+
+acquire_expensive_lock_ () {
+	local owner waited=
+	mkdir -p "$TEST_RESULTS_DIR" || return 1
+	while ! mkdir "$expensive_lock_" 2>/dev/null
+	do
+		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
+		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
+		then
+			# The holder was killed without releasing the lock.
+			rm -rf "$expensive_lock_"
+			continue
+		fi
+		if test -z "$waited"
+		then
+			say >&3 "waiting for another EXPENSIVE test to finish"
+			waited=t
+		fi
+		sleep 1
+	done
+	echo $$ >"$expensive_lock_/pid"
+	have_expensive_lock_=t
+}
+
+release_expensive_lock_ () {
+	test -n "$have_expensive_lock_" || return 0
+	rm -rf "$expensive_lock_"
+	have_expensive_lock_=
+}
+
 test_run_ () {
 	test_cleanup=:
 	expecting_failure=$2
@@ -1103,6 +1141,12 @@ test_run_ () {
 		fi
 	fi
 
+	case ",$test_prereq," in
+	*,EXPENSIVE,*)
+		acquire_expensive_lock_
+		;;
+	esac
+
 	setup_malloc_check
 	test_eval_ "$1"
 	eval_ret=$?
@@ -1115,6 +1159,7 @@ test_run_ () {
 		test_eval_ "$test_cleanup"
 		teardown_malloc_check
 	fi
+	release_expensive_lock_
 	if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE"
 	then
 		echo ""

---
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
change-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6



```

## Kaartic Sivaraam, 2026-11-19 17:33

Subject: Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time
Message-ID: <2bcce580-f5ed-47e9-bd3e-1e238eb90d5c@gmail.com>
URL: https://gitlist.dev/e/2bcce580-f5ed-47e9-bd3e-1e238eb90d5c%40gmail.com
In-Reply-To: <20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com>

```
On 11/19/26 22:59, Kaartic Sivaraam wrote:
> Changes in v2:
> - Corrected commit messages as per Patrick's suggestions
> - Tweaked the shell function comment to avoid mentioning 'prove --jobs'
> - Made the owner and waiting variables local within acquire_expensive_lock_
> > Link to v1: https://patch.msgid.link/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com

I thought b4 would've included it by itself. Anyways, here's the range-diff:

1:  f0664ff4ee ! 1:  fbf773f39a test-lib: run EXPENSIVE tests one at a time
    @@ Metadata
      ## Commit message ##
         test-lib: run EXPENSIVE tests one at a time

    -    Recently, we've acquired quite a few expensive test in our test suite.
    +    Recently, we've acquired quite a few expensive tests in our test suite.
         For instance, these commits add several tests that require
    -    multi-gigabytes:
    +    multi-gigabytes of memory:

           - 9048b48fb (t1600-index: add expensive tests for over-long
             tree paths, 2026-10-16) added tests to t1600
    @@ t/test-lib.sh: fail_117 () {
      	return 117
      }

    -+# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
    -+# them running at the same time under "prove --jobs" can exhaust the
    -+# machine. Allow only one at a time across all test scripts that share
    ++# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
    ++# them running at the same time can exhaust the machine.
    ++#
    ++# Allow only one at a time across all test scripts that share
     +# this TEST_OUTPUT_DIRECTORY.
     +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
     +have_expensive_lock_=
     +
     +acquire_expensive_lock_ () {
    ++	local owner waited=
     +	mkdir -p "$TEST_RESULTS_DIR" || return 1
     +	while ! mkdir "$expensive_lock_" 2>/dev/null
     +	do
    @@ t/test-lib.sh: fail_117 () {
     +			rm -rf "$expensive_lock_"
     +			continue
     +		fi
    -+		if test -z "$expensive_lock_waited_"
    ++		if test -z "$waited"
     +		then
     +			say >&3 "waiting for another EXPENSIVE test to finish"
    -+			expensive_lock_waited_=t
    ++			waited=t
     +		fi
     +		sleep 1
     +	done
    -+	expensive_lock_waited_=
     +	echo $$ >"$expensive_lock_/pid"
     +	have_expensive_lock_=t
     +}

-- 
Sivaraam



```

## Junio C Hamano, 2026-11-20 06:28

Subject: Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time
Message-ID: <xmqqpkw0rp8i.fsf@gitster.g>
URL: https://gitlist.dev/e/xmqqpkw0rp8i.fsf%40gitster.g
In-Reply-To: <20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com>

```
Kaartic Sivaraam <kaartic.sivaraam@gmail.com> writes:

> Recently, we've acquired quite a few expensive tests in our test suite.
> For instance, these commits add several tests that require
> multi-gigabytes of memory:
>
>   - 9048b48fb (t1600-index: add expensive tests for over-long
>     tree paths, 2026-10-16) added tests to t1600

No such commit X-<.  Perhaps 89048b48fb (t1600-index: add expensive
tests for over-long tree paths, 2026-10-16).

>   - 288266c11d (xdiff: use size_t for merge buffer size computations,
>     2026-10-16) added tests to t4300
>
>   - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
>     2026-03-25), added tests to t4141

Inconsistent use of "," after the (close parentheses).

> +# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
> +# them running at the same time can exhaust the machine.
> +#
> +# Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> +	local owner waited=
> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
> +	while ! mkdir "$expensive_lock_" 2>/dev/null
> +	do
> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> +		then
> +			# The holder was killed without releasing the lock.
> +			rm -rf "$expensive_lock_"
> +			continue
> +		fi
> +		if test -z "$waited"
> +		then
> +			say >&3 "waiting for another EXPENSIVE test to finish"
> +			waited=t
> +		fi
> +		sleep 1
> +	done
> +	echo $$ >"$expensive_lock_/pid"
> +	have_expensive_lock_=t
> +}

Hmph, is the above safe when multiple processes are working with
expensive tests during normal (i.e., not involving SIGKILL)
operation, or does it still have TOCTOU race on the lock file?

 * T1 holds $expensive_lock_, finishes its last EXPENSIVE test,
   calls release_expensive_lock_ (rm -rf "$expensive_lock_"), and
   exits.

 * Waiter T2 read owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
   (obtaining T1's PID) just before T1 removed the directory.

 * Waiter T3 runs mkdir "$expensive_lock_", succeeds, writes T3's
   PID to pid, and begins its EXPENSIVE test.

 * T2 now checks test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
   against T1's exited PID, finds that T1 is gone, runs rm -rf
   "$expensive_lock_", which deletes T3's active lock.  It loops,
   succeeds on mkdir "$expensive_lock_", and runs its EXPENSIVE test
   concurrently with T3.

Also, between the time you successfully mkdir the lock directory and
the time you leave this function, you can be killed by a signal with
INT, TERM, or HUP, for which t/test-lib.sh installs a hander to call
exit, and there is a "trap die EXIT".

You may manage to create $expensive_lock_ directory, open pid file
for writing, and then get killed before you echo $$ to the pid file
or assign 't' to have_expensive_lock_ variable.

When your die handler calls release_expensive_lock_, you would leave
with the directory behind and exit.  You may not have pid file, or
the pid file may be empty.  The waiting processes will not run the
kill -0 "$owner" part (because they do not know who you are that held
the lock and left before writing $$ in pid), the waiters will wait
for the mkdir to successfully create the directory that already
exists and nobody will ever remove, no?

Between the TOCTOU race allowing multiple processes entering the
expensive region and deadlocking denying everybody from entering the
expensive region, the latter is far worse, of course.

> +release_expensive_lock_ () {
> +	test -n "$have_expensive_lock_" || return 0
> +	rm -rf "$expensive_lock_"
> +	have_expensive_lock_=
> +}


```
