{"thread":{"id":"66877","subject":"[PATCH] test-lib: run EXPENSIVE tests one at a time","startedAt":"2026-11-19T07:40:31Z","lastAt":"2026-11-20T06:28:13Z","messageCount":6,"participants":["Kaartic Sivaraam","Patrick Steinhardt","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"558806","messageId":"20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com","threadId":"66877","inReplyTo":null,"subject":"[PATCH] test-lib: run EXPENSIVE tests one at a time","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2026-11-19T07:40:31Z","receivedAt":"2026-11-19T07:40:31Z","isPatch":true,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"Recently, we've acquired quite a few expensive test in our test suite.\nFor instance, these commits add several tests that require\nmulti-gigabytes:\n\n  - 9048b48fb (t1600-index: add expensive tests for over-long\n    tree paths, 2026-10-16) added tests to t1600\n\n  - 288266c11d (xdiff: use size_t for merge buffer size computations,\n    2026-10-16) added tests to t4300\n\n  - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,\n    2026-03-25), added tests to t4141\n\nWe also have the memory hungry 5GB filtered hash-object test in t1007.\n\nWhen GIT_TEST_LONG is set, \"prove --jobs\" happily runs several of these\nat once. On the 16GB GitLab CI runners this exhausts memory: the\nsanitizer jobs have been hanging in t1007 until they hit the job\ntimeout ever since the new EXPENSIVE tests in t1600 started running\nalongside it. Running the two under a 16GB memory cap reproduces it,\nwith the t1007 git process getting OOM-killed.\n\nGitHub CI runners are no exception and observe the same behaviour.\n\nTake a lock shared by all test scripts around each EXPENSIVE test, so\nonly one of them runs at a time. The lock is a directory in\nTEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind\nsymlink lock. It records the holder's PID so that a lock left behind by\na script that was killed with SIGKILL gets taken over by the next\nwaiter. It is released after the test and its cleanup finish, and from\nthe EXIT trap when a script dies or is interrupted while holding it.\n\nCheap tests in the same scripts keep running in parallel. Only the\nexpensive ones wait for each other.\n\nSigned-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>\n---\nIn recent times, we've been noticing the sanitizer jobs (linux-leaks,\nlinux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in\nour GitLab CI runners. On some digging, it turns out that this is\nbecause of multiple memory hungry tests running in parallel.\n\nSo, I thought of suggesting this patch to serialize the expensive tests\nat least to avoid the putting the CI runners under memory pressure.\n\nThe GitHub CI runners also observed the same behaviour.\n\nBoth CI runners are happy with this change.\n\nGitHub: https://github.com/sivaraam/git/actions/runs/43281285339\nGitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506\n---\n t/README      |  4 +++-\n t/test-lib.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 47 insertions(+), 1 deletion(-)\n\ndiff --git a/t/README b/t/README\nindex fd7e8504b5..c156dd3af6 100644\n--- a/t/README\n+++ b/t/README\n@@ -117,7 +117,9 @@ appropriately before running \"make\". Short options can be bundled, i.e.\n -l::\n --long-tests::\n \tThis causes additional long-running tests to be run (where\n-\tavailable), for more exhaustive testing.\n+\tavailable), for more exhaustive testing. These tests can use a\n+\tlot of memory and disk, so only one of them runs at a time, even\n+\twhen several test scripts run in parallel (e.g. \"prove --jobs\").\n \n -r::\n --run=<test-selector>::\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c56654b7c5..4d29172c9d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -784,6 +784,7 @@ die () {\n \t# test script run with '--immediate' fails, or when the user hits\n \t# ctrl-C, i.e. when 'test_done' is not invoked at all.\n \ttest_atexit_handler || code=$?\n+\trelease_expensive_lock_\n \tif test -n \"$GIT_EXIT_OK\"\n \tthen\n \t\texit $code\n@@ -1089,6 +1090,42 @@ fail_117 () {\n \treturn 117\n }\n \n+# EXPENSIVE tests can need many gigabytes of memory or disk, so two of\n+# them running at the same time under \"prove --jobs\" can exhaust the\n+# machine. Allow only one at a time across all test scripts that share\n+# this TEST_OUTPUT_DIRECTORY.\n+expensive_lock_=\"$TEST_RESULTS_DIR/expensive.lock\"\n+have_expensive_lock_=\n+\n+acquire_expensive_lock_ () {\n+\tmkdir -p \"$TEST_RESULTS_DIR\" || return 1\n+\twhile ! mkdir \"$expensive_lock_\" 2>/dev/null\n+\tdo\n+\t\towner=$(cat \"$expensive_lock_/pid\" 2>/dev/null)\n+\t\tif test -n \"$owner\" && ! kill -0 \"$owner\" 2>/dev/null\n+\t\tthen\n+\t\t\t# The holder was killed without releasing the lock.\n+\t\t\trm -rf \"$expensive_lock_\"\n+\t\t\tcontinue\n+\t\tfi\n+\t\tif test -z \"$expensive_lock_waited_\"\n+\t\tthen\n+\t\t\tsay >&3 \"waiting for another EXPENSIVE test to finish\"\n+\t\t\texpensive_lock_waited_=t\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\texpensive_lock_waited_=\n+\techo $$ >\"$expensive_lock_/pid\"\n+\thave_expensive_lock_=t\n+}\n+\n+release_expensive_lock_ () {\n+\ttest -n \"$have_expensive_lock_\" || return 0\n+\trm -rf \"$expensive_lock_\"\n+\thave_expensive_lock_=\n+}\n+\n test_run_ () {\n \ttest_cleanup=:\n \texpecting_failure=$2\n@@ -1103,6 +1140,12 @@ test_run_ () {\n \t\tfi\n \tfi\n \n+\tcase \",$test_prereq,\" in\n+\t*,EXPENSIVE,*)\n+\t\tacquire_expensive_lock_\n+\t\t;;\n+\tesac\n+\n \tsetup_malloc_check\n \ttest_eval_ \"$1\"\n \teval_ret=$?\n@@ -1115,6 +1158,7 @@ test_run_ () {\n \t\ttest_eval_ \"$test_cleanup\"\n \t\tteardown_malloc_check\n \tfi\n+\trelease_expensive_lock_\n \tif test \"$verbose\" = \"t\" && test -n \"$HARNESS_ACTIVE\"\n \tthen\n \t\techo \"\"\n\n---\nbase-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4\nchange-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6\n\n\n"},{"id":"558813","messageId":"av7KRbOGWeV-Qof1@pks.im","threadId":"66877","inReplyTo":"20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com","subject":"Re: [PATCH] test-lib: run EXPENSIVE tests one at a time","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-19T10:07:01Z","receivedAt":"2026-11-19T10:07:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:\n> Recently, we've acquired quite a few expensive test in our test suite.\n\ns/test/tests/\n\n> For instance, these commits add several tests that require\n> multi-gigabytes:\n\nMulti-gigabytes what?\n\n[snip]\n> In recent times, we've been noticing the sanitizer jobs (linux-leaks,\n> linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in\n> our GitLab CI runners. On some digging, it turns out that this is\n> because of multiple memory hungry tests running in parallel.\n\nYeah, our pipelines are currently a mess because of this, so I'd be\nreally happy to have this resolved soonish.\n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index c56654b7c5..4d29172c9d 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1089,6 +1090,42 @@ fail_117 () {\n>  \treturn 117\n>  }\n>  \n> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of\n\ns/can/may/\n\n> +# them running at the same time under \"prove --jobs\" can exhaust the\n\nI wouldn't single out `prove --jobs` here, as we know to run\nconcurrently without prove either via our plain Makefile or via Meson.\n\n> +# machine. Allow only one at a time across all test scripts that share\n> +# this TEST_OUTPUT_DIRECTORY.\n> +expensive_lock_=\"$TEST_RESULTS_DIR/expensive.lock\"\n> +have_expensive_lock_=\n> +\n> +acquire_expensive_lock_ () {\n> +\tmkdir -p \"$TEST_RESULTS_DIR\" || return 1\n> +\twhile ! mkdir \"$expensive_lock_\" 2>/dev/null\n> +\tdo\n> +\t\towner=$(cat \"$expensive_lock_/pid\" 2>/dev/null)\n> +\t\tif test -n \"$owner\" && ! kill -0 \"$owner\" 2>/dev/null\n> +\t\tthen\n> +\t\t\t# The holder was killed without releasing the lock.\n> +\t\t\trm -rf \"$expensive_lock_\"\n> +\t\t\tcontinue\n\nOkay, if the previous owner does not exist anymore we prune the lock.\nTwo racing processes which try to acquire the lock may race in a way\nthat the \"/pid\" file does not exist anymore because the other process\nhas already deleted it. But you protect against that by checking whether\nwe have been able to read a PID in the first place via `test -n`.\n\n> +\t\tfi\n> +\t\tif test -z \"$expensive_lock_waited_\"\n> +\t\tthen\n> +\t\t\tsay >&3 \"waiting for another EXPENSIVE test to finish\"\n\nHmm. So file descriptor 3 points to stderr or to the test output file.\nAnd if I remember correctly, we're indeed free to write to stderr\nwhatever data we want without breaking the TAP format?\n\n> +\t\t\texpensive_lock_waited_=t\n> +\t\tfi\n> +\t\tsleep 1\n> +\tdone\n> +\texpensive_lock_waited_=\n\nShould this be declared as a local variable?\n\nOverall this is a rather pragmatic approach to solve the issue. I was\nthinking whether it would make sense to distinguish e.g. EXPENISVE_MEM\nfrom EXPENSIVE_DISK, and have different locks for each of those. But I\ndoubt adding this kind of complexity would really be worth it in the\nend.\n\nSo I'm fine with the approach. Onward to green pipelines again! Well,\nuntil the next thing breaks next week...\n\nThanks!\n\nPatrick\n\n"},{"id":"558864","messageId":"3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com","threadId":"66877","inReplyTo":"av7KRbOGWeV-Qof1@pks.im","subject":"Re: [PATCH] test-lib: run EXPENSIVE tests one at a time","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2026-11-19T17:12:53Z","receivedAt":"2026-11-19T17:12:53Z","isPatch":true,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"On 11/19/26 15:37, Patrick Steinhardt wrote:\n> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:\n>> Recently, we've acquired quite a few expensive test in our test suite.\n> > s/test/tests/\n\nCorrected it.\n\n> >> For instance, these commits add several tests that require\n>> multi-gigabytes:\n> > Multi-gigabytes what?\n>\n\nFixed it!\n>> diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> index c56654b7c5..4d29172c9d 100644\n>> --- a/t/test-lib.sh\n>> +++ b/t/test-lib.sh\n>> @@ -1089,6 +1090,42 @@ fail_117 () {\n>>   \treturn 117\n>>   }\n>>   >> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of\n> > s/can/may/\n> Done.\n\n>> +# them running at the same time under \"prove --jobs\" can exhaust the\n> > I wouldn't single out `prove --jobs` here, as we know to run\n> concurrently without prove either via our plain Makefile or via Meson.\n>\n\nMakes sense. Will remove the part mentioning 'prove --jobs' specifically.\n\n>> +# machine. Allow only one at a time across all test scripts that share\n>> +# this TEST_OUTPUT_DIRECTORY.\n>> +expensive_lock_=\"$TEST_RESULTS_DIR/expensive.lock\"\n>> +have_expensive_lock_=\n>> +\n>> +acquire_expensive_lock_ () {\n>> +\tmkdir -p \"$TEST_RESULTS_DIR\" || return 1\n>> +\twhile ! mkdir \"$expensive_lock_\" 2>/dev/null\n>> +\tdo\n>> +\t\towner=$(cat \"$expensive_lock_/pid\" 2>/dev/null)\n>> +\t\tif test -n \"$owner\" && ! kill -0 \"$owner\" 2>/dev/null\n>> +\t\tthen\n>> +\t\t\t# The holder was killed without releasing the lock.\n>> +\t\t\trm -rf \"$expensive_lock_\"\n>> +\t\t\tcontinue\n> > Okay, if the previous owner does not exist anymore we prune the lock.\n> Two racing processes which try to acquire the lock may race in a way\n> that the \"/pid\" file does not exist anymore because the other process\n> has already deleted it. But you protect against that by checking whether\n> we have been able to read a PID in the first place via `test -n`.\n\nIndeed. There might still be a race in the case when two racing processes (A and B) both see the same dead PID before either acts. A would have deleted the lock, created a new one, written its PID. B which still holds the stale PID would proceed to delete the lock which is actually the live lock held by A, create a new one and write its PID. This might happen if a holder was SIGKILLed and two or more scripts were waiting at that exact moment, so it's rare. So, I wouldn't complicate the code to handle this. In the worst case, we'd have 2 expensive test cases running in place of the one we intend.\n\n> >> +\t\tfi\n>> +\t\tif test -z \"$expensive_lock_waited_\"\n>> +\t\tthen\n>> +\t\t\tsay >&3 \"waiting for another EXPENSIVE test to finish\"\n> > Hmm. So file descriptor 3 points to stderr or to the test output file.\n> And if I remember correctly, we're indeed free to write to stderr\n> whatever data we want without breaking the TAP format?\n> Yep. With no '--verbose' or '--verbose-log' flags this message would just go to /dev/null. I suppose that will only happen when someone runs this manually. So, hopefully that's fine.\n\n>> +\t\t\texpensive_lock_waited_=t\n>> +\t\tfi\n>> +\t\tsleep 1\n>> +\tdone\n>> +\texpensive_lock_waited_=\n> > Should this be declared as a local variable?\n>\n\nSure. Will make 'owner' local too.\n\n> Overall this is a rather pragmatic approach to solve the issue. I was\n> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM\n> from EXPENSIVE_DISK, and have different locks for each of those. But I\n> doubt adding this kind of complexity would really be worth it in the\n> end.\n> Got it. Thank you for sharing your thoughts!\n\n> So I'm fine with the approach. Onward to green pipelines again! Well,\n> until the next thing breaks next week...\n> Totally. It feels is like we're constantly fire-fighting in the CI these days ...\n\n-- \nSivaraam\n\n\n"},{"id":"558867","messageId":"20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com","threadId":"66877","inReplyTo":"20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com","subject":"[PATCH v2] test-lib: run EXPENSIVE tests one at a time","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2026-11-19T17:29:15Z","receivedAt":"2026-11-19T17:29:15Z","isPatch":true,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"Recently, we've acquired quite a few expensive tests in our test suite.\nFor instance, these commits add several tests that require\nmulti-gigabytes of memory:\n\n  - 9048b48fb (t1600-index: add expensive tests for over-long\n    tree paths, 2026-10-16) added tests to t1600\n\n  - 288266c11d (xdiff: use size_t for merge buffer size computations,\n    2026-10-16) added tests to t4300\n\n  - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,\n    2026-03-25), added tests to t4141\n\nWe also have the memory hungry 5GB filtered hash-object test in t1007.\n\nWhen GIT_TEST_LONG is set, \"prove --jobs\" happily runs several of these\nat once. On the 16GB GitLab CI runners this exhausts memory: the\nsanitizer jobs have been hanging in t1007 until they hit the job\ntimeout ever since the new EXPENSIVE tests in t1600 started running\nalongside it. Running the two under a 16GB memory cap reproduces it,\nwith the t1007 git process getting OOM-killed.\n\nGitHub CI runners are no exception and observe the same behaviour.\n\nTake a lock shared by all test scripts around each EXPENSIVE test, so\nonly one of them runs at a time. The lock is a directory in\nTEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind\nsymlink lock. It records the holder's PID so that a lock left behind by\na script that was killed with SIGKILL gets taken over by the next\nwaiter. It is released after the test and its cleanup finish, and from\nthe EXIT trap when a script dies or is interrupted while holding it.\n\nCheap tests in the same scripts keep running in parallel. Only the\nexpensive ones wait for each other.\n\nSigned-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>\n---\nIn recent times, we've been noticing the sanitizer jobs (linux-leaks,\nlinux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in\nour GitLab CI runners. On some digging, it turns out that this is\nbecause of multiple memory hungry tests running in parallel.\n\nSo, I thought of suggesting this patch to serialize the expensive tests\nat least to avoid the putting the CI runners under memory pressure.\n\nThe GitHub CI runners also observed the same behaviour.\n\nBoth CI runners are happy with this change.\n\nGitHub: https://github.com/sivaraam/git/actions/runs/43281285339\nGitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506\n---\nChanges in v2:\n- Corrected commit messages as per Patrick's suggestions\n- Tweaked the shell function comment to avoid mentioning 'prove --jobs'\n- Made the owner and waiting variables local within acquire_expensive_lock_\n\nLink to v1: https://patch.msgid.link/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com\n---\n t/README      |  4 +++-\n t/test-lib.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 48 insertions(+), 1 deletion(-)\n\ndiff --git a/t/README b/t/README\nindex fd7e8504b5..c156dd3af6 100644\n--- a/t/README\n+++ b/t/README\n@@ -117,7 +117,9 @@ appropriately before running \"make\". Short options can be bundled, i.e.\n -l::\n --long-tests::\n \tThis causes additional long-running tests to be run (where\n-\tavailable), for more exhaustive testing.\n+\tavailable), for more exhaustive testing. These tests can use a\n+\tlot of memory and disk, so only one of them runs at a time, even\n+\twhen several test scripts run in parallel (e.g. \"prove --jobs\").\n \n -r::\n --run=<test-selector>::\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c56654b7c5..193bf06aea 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -784,6 +784,7 @@ die () {\n \t# test script run with '--immediate' fails, or when the user hits\n \t# ctrl-C, i.e. when 'test_done' is not invoked at all.\n \ttest_atexit_handler || code=$?\n+\trelease_expensive_lock_\n \tif test -n \"$GIT_EXIT_OK\"\n \tthen\n \t\texit $code\n@@ -1089,6 +1090,43 @@ fail_117 () {\n \treturn 117\n }\n \n+# EXPENSIVE tests may need many gigabytes of memory or disk, so two of\n+# them running at the same time can exhaust the machine.\n+#\n+# Allow only one at a time across all test scripts that share\n+# this TEST_OUTPUT_DIRECTORY.\n+expensive_lock_=\"$TEST_RESULTS_DIR/expensive.lock\"\n+have_expensive_lock_=\n+\n+acquire_expensive_lock_ () {\n+\tlocal owner waited=\n+\tmkdir -p \"$TEST_RESULTS_DIR\" || return 1\n+\twhile ! mkdir \"$expensive_lock_\" 2>/dev/null\n+\tdo\n+\t\towner=$(cat \"$expensive_lock_/pid\" 2>/dev/null)\n+\t\tif test -n \"$owner\" && ! kill -0 \"$owner\" 2>/dev/null\n+\t\tthen\n+\t\t\t# The holder was killed without releasing the lock.\n+\t\t\trm -rf \"$expensive_lock_\"\n+\t\t\tcontinue\n+\t\tfi\n+\t\tif test -z \"$waited\"\n+\t\tthen\n+\t\t\tsay >&3 \"waiting for another EXPENSIVE test to finish\"\n+\t\t\twaited=t\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\techo $$ >\"$expensive_lock_/pid\"\n+\thave_expensive_lock_=t\n+}\n+\n+release_expensive_lock_ () {\n+\ttest -n \"$have_expensive_lock_\" || return 0\n+\trm -rf \"$expensive_lock_\"\n+\thave_expensive_lock_=\n+}\n+\n test_run_ () {\n \ttest_cleanup=:\n \texpecting_failure=$2\n@@ -1103,6 +1141,12 @@ test_run_ () {\n \t\tfi\n \tfi\n \n+\tcase \",$test_prereq,\" in\n+\t*,EXPENSIVE,*)\n+\t\tacquire_expensive_lock_\n+\t\t;;\n+\tesac\n+\n \tsetup_malloc_check\n \ttest_eval_ \"$1\"\n \teval_ret=$?\n@@ -1115,6 +1159,7 @@ test_run_ () {\n \t\ttest_eval_ \"$test_cleanup\"\n \t\tteardown_malloc_check\n \tfi\n+\trelease_expensive_lock_\n \tif test \"$verbose\" = \"t\" && test -n \"$HARNESS_ACTIVE\"\n \tthen\n \t\techo \"\"\n\n---\nbase-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4\nchange-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6\n\n\n"},{"id":"558868","messageId":"2bcce580-f5ed-47e9-bd3e-1e238eb90d5c@gmail.com","threadId":"66877","inReplyTo":"20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com","subject":"Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2026-11-19T17:33:32Z","receivedAt":"2026-11-19T17:33:32Z","isPatch":true,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"On 11/19/26 22:59, Kaartic Sivaraam wrote:\n> Changes in v2:\n> - Corrected commit messages as per Patrick's suggestions\n> - Tweaked the shell function comment to avoid mentioning 'prove --jobs'\n> - Made the owner and waiting variables local within acquire_expensive_lock_\n> > Link to v1: https://patch.msgid.link/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com\n\nI thought b4 would've included it by itself. Anyways, here's the range-diff:\n\n1:  f0664ff4ee ! 1:  fbf773f39a test-lib: run EXPENSIVE tests one at a time\n    @@ Metadata\n      ## Commit message ##\n         test-lib: run EXPENSIVE tests one at a time\n\n    -    Recently, we've acquired quite a few expensive test in our test suite.\n    +    Recently, we've acquired quite a few expensive tests in our test suite.\n         For instance, these commits add several tests that require\n    -    multi-gigabytes:\n    +    multi-gigabytes of memory:\n\n           - 9048b48fb (t1600-index: add expensive tests for over-long\n             tree paths, 2026-10-16) added tests to t1600\n    @@ t/test-lib.sh: fail_117 () {\n      \treturn 117\n      }\n\n    -+# EXPENSIVE tests can need many gigabytes of memory or disk, so two of\n    -+# them running at the same time under \"prove --jobs\" can exhaust the\n    -+# machine. Allow only one at a time across all test scripts that share\n    ++# EXPENSIVE tests may need many gigabytes of memory or disk, so two of\n    ++# them running at the same time can exhaust the machine.\n    ++#\n    ++# Allow only one at a time across all test scripts that share\n     +# this TEST_OUTPUT_DIRECTORY.\n     +expensive_lock_=\"$TEST_RESULTS_DIR/expensive.lock\"\n     +have_expensive_lock_=\n     +\n     +acquire_expensive_lock_ () {\n    ++\tlocal owner waited=\n     +\tmkdir -p \"$TEST_RESULTS_DIR\" || return 1\n     +\twhile ! mkdir \"$expensive_lock_\" 2>/dev/null\n     +\tdo\n    @@ t/test-lib.sh: fail_117 () {\n     +\t\t\trm -rf \"$expensive_lock_\"\n     +\t\t\tcontinue\n     +\t\tfi\n    -+\t\tif test -z \"$expensive_lock_waited_\"\n    ++\t\tif test -z \"$waited\"\n     +\t\tthen\n     +\t\t\tsay >&3 \"waiting for another EXPENSIVE test to finish\"\n    -+\t\t\texpensive_lock_waited_=t\n    ++\t\t\twaited=t\n     +\t\tfi\n     +\t\tsleep 1\n     +\tdone\n    -+\texpensive_lock_waited_=\n     +\techo $$ >\"$expensive_lock_/pid\"\n     +\thave_expensive_lock_=t\n     +}\n\n-- \nSivaraam\n\n\n"},{"id":"558910","messageId":"xmqqpkw0rp8i.fsf@gitster.g","threadId":"66877","inReplyTo":"20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com","subject":"Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-20T06:28:13Z","receivedAt":"2026-11-20T06:28:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kaartic Sivaraam <kaartic.sivaraam@gmail.com> writes:\n\n> Recently, we've acquired quite a few expensive tests in our test suite.\n> For instance, these commits add several tests that require\n> multi-gigabytes of memory:\n>\n>   - 9048b48fb (t1600-index: add expensive tests for over-long\n>     tree paths, 2026-10-16) added tests to t1600\n\nNo such commit X-<.  Perhaps 89048b48fb (t1600-index: add expensive\ntests for over-long tree paths, 2026-10-16).\n\n>   - 288266c11d (xdiff: use size_t for merge buffer size computations,\n>     2026-10-16) added tests to t4300\n>\n>   - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,\n>     2026-03-25), added tests to t4141\n\nInconsistent use of \",\" after the (close parentheses).\n\n> +# EXPENSIVE tests may need many gigabytes of memory or disk, so two of\n> +# them running at the same time can exhaust the machine.\n> +#\n> +# Allow only one at a time across all test scripts that share\n> +# this TEST_OUTPUT_DIRECTORY.\n> +expensive_lock_=\"$TEST_RESULTS_DIR/expensive.lock\"\n> +have_expensive_lock_=\n> +\n> +acquire_expensive_lock_ () {\n> +\tlocal owner waited=\n> +\tmkdir -p \"$TEST_RESULTS_DIR\" || return 1\n> +\twhile ! mkdir \"$expensive_lock_\" 2>/dev/null\n> +\tdo\n> +\t\towner=$(cat \"$expensive_lock_/pid\" 2>/dev/null)\n> +\t\tif test -n \"$owner\" && ! kill -0 \"$owner\" 2>/dev/null\n> +\t\tthen\n> +\t\t\t# The holder was killed without releasing the lock.\n> +\t\t\trm -rf \"$expensive_lock_\"\n> +\t\t\tcontinue\n> +\t\tfi\n> +\t\tif test -z \"$waited\"\n> +\t\tthen\n> +\t\t\tsay >&3 \"waiting for another EXPENSIVE test to finish\"\n> +\t\t\twaited=t\n> +\t\tfi\n> +\t\tsleep 1\n> +\tdone\n> +\techo $$ >\"$expensive_lock_/pid\"\n> +\thave_expensive_lock_=t\n> +}\n\nHmph, is the above safe when multiple processes are working with\nexpensive tests during normal (i.e., not involving SIGKILL)\noperation, or does it still have TOCTOU race on the lock file?\n\n * T1 holds $expensive_lock_, finishes its last EXPENSIVE test,\n   calls release_expensive_lock_ (rm -rf \"$expensive_lock_\"), and\n   exits.\n\n * Waiter T2 read owner=$(cat \"$expensive_lock_/pid\" 2>/dev/null)\n   (obtaining T1's PID) just before T1 removed the directory.\n\n * Waiter T3 runs mkdir \"$expensive_lock_\", succeeds, writes T3's\n   PID to pid, and begins its EXPENSIVE test.\n\n * T2 now checks test -n \"$owner\" && ! kill -0 \"$owner\" 2>/dev/null\n   against T1's exited PID, finds that T1 is gone, runs rm -rf\n   \"$expensive_lock_\", which deletes T3's active lock.  It loops,\n   succeeds on mkdir \"$expensive_lock_\", and runs its EXPENSIVE test\n   concurrently with T3.\n\nAlso, between the time you successfully mkdir the lock directory and\nthe time you leave this function, you can be killed by a signal with\nINT, TERM, or HUP, for which t/test-lib.sh installs a hander to call\nexit, and there is a \"trap die EXIT\".\n\nYou may manage to create $expensive_lock_ directory, open pid file\nfor writing, and then get killed before you echo $$ to the pid file\nor assign 't' to have_expensive_lock_ variable.\n\nWhen your die handler calls release_expensive_lock_, you would leave\nwith the directory behind and exit.  You may not have pid file, or\nthe pid file may be empty.  The waiting processes will not run the\nkill -0 \"$owner\" part (because they do not know who you are that held\nthe lock and left before writing $$ in pid), the waiters will wait\nfor the mkdir to successfully create the directory that already\nexists and nobody will ever remove, no?\n\nBetween the TOCTOU race allowing multiple processes entering the\nexpensive region and deadlocking denying everybody from entering the\nexpensive region, the latter is far worse, of course.\n\n> +release_expensive_lock_ () {\n> +\ttest -n \"$have_expensive_lock_\" || return 0\n> +\trm -rf \"$expensive_lock_\"\n> +\thave_expensive_lock_=\n> +}\n\n"}]}