{"thread":{"id":"63587","subject":"[PATCH] t5410: avoid hangs in CI runs in the win+Meson test jobs","startedAt":"2025-06-05T10:16:49Z","lastAt":"2025-06-05T13:30:10Z","messageCount":3,"participants":["Johannes Schindelin via GitGitGadget","Patrick Steinhardt","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"519759","messageId":"pull.1932.git.1749118606047.gitgitgadget@gmail.com","threadId":"63587","inReplyTo":null,"subject":"[PATCH] t5410: avoid hangs in CI runs in the win+Meson test jobs","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-06-05T10:16:45Z","receivedAt":"2025-06-05T10:16:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nIn the GitHub workflow used in Git's CI builds, the `vs test` jobs use a\nsubset of a specific revision of Git for Windows' SDK to run Git's test\nsuite. This revision is validated by another CI workflow to ensure that\nsaid revision _can_ run Git's test suite successfully, skipping buggy\nupdates in Git for Windows' SDK.\n\nThe `win+Meson test` jobs do things differently, quite differently. They\nuse the Bash of the Git for Windows version that is installed on the\nrunners to run Git's test suite.\n\nThis difference has consequences.\n\nWhen 68cb0b5253a0 (builtin/receive-pack: add option to skip connectivity\ncheck, 2025-05-20) introduced a test case that uses `tee <file> | git\nreceive-pack` as `--receive-pack` parameter (imitating an existing\npattern in the same test script), it hit just the sweet spot to trigger\na bug in the MSYS2 runtime shipped in Git for Windows v2.49.0. This\nversion is the one currently installed on GitHub's runners.\n\nThe problem is that the `git receive-pack` process finishes while the\n`tee` process does not need to write anything anymore and therefore does\nnot receive an EOF. Instead, it should receive a SIGPIPE, but the bug in\nthe MSYS2 runtime prevents that from working as intended. As a\nconsequence, the `tee` process waits for more input from the `git.exe\nsend-pack` process but none is coming, and the test script patiently\nwaits until the 6h timeout hits.\n\nOnly every once in a while, the `git receive-pack` process manages to\nsend an EOF to the `tee` process and no hang occurs. Therefore, the\nproblem can be worked around by cancelling the clearly-hanging job after\ntwenty or so minutes and re-running it, repeating the process about half\na dozen times, until the hang was successfully avoided.\n\nThis bug in the MSYS2 runtime has been fixed in the meantime, which is\nthe reason why the same test case causes no problems in the `win test`\nand the `vs test` jobs.\n\nThis will continue to be the case until the Git for Windows version on\nthe GitHub runners is upgraded to a version that distributes a newer\nMSYS2 runtime version. However, as of time of writing, this _is_ the\nlatest Git for Windows version, and will be for another 1.5 weeks, until\nGit v2.50.0 is scheduled to appear (and shortly thereafter Git for\nWindows v2.50.0). Traditionally it takes a while before the runners pick\nup the new version.\n\nWe could just wait it out, six hours at a time.\n\nHere, I opt for an alternative: Detect the buggy MSYS2 runtime and\nsimply skip the test case. It's not like the `receive-pack` test cases\nare specific to Windows, and even then, to my chagrin the CI runs in\ngit-for-windows/git spend around ten hours of compute time each and\nevery time to run the entire test suite on all the platforms, even the\ntests that cover cross-platform code, and for Windows alone we do that\nthree times: with GCC, with MSVC, and with MSVC via Meson. Therefore, I\ndeem it more than acceptable to skip this test case in one of those\nmatrices.\n\nFor good luck, also the preceding test case is skipped in that scenario,\nas it uses the same `--receive-pack=tee <file> | git receive-pack`\npattern, even though I never observed that test case to hang in\npractice.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n    t5410: avoid hangs in CI runs in the win+Meson test jobs\n    \n    I finally had a chance to look more closely at this problem. Here is my\n    alternative to what Patrick proposed in\n    https://lore.kernel.org/git/aD7tKfXD7YxprSZh@pks.im/.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1932%2Fdscho%2Ft5410-hangs-in-win%2BMeson-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1932/dscho/t5410-hangs-in-win+Meson-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1932\n\n t/t5410-receive-pack.sh | 17 +++++++++++++++--\n 1 file changed, 15 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5410-receive-pack.sh b/t/t5410-receive-pack.sh\nindex f76a22943ef..09d6bfd2a10 100755\n--- a/t/t5410-receive-pack.sh\n+++ b/t/t5410-receive-pack.sh\n@@ -41,7 +41,19 @@ test_expect_success 'with core.alternateRefsPrefixes' '\n \ttest_cmp expect actual.haves\n '\n \n-test_expect_success 'receive-pack missing objects fails connectivity check' '\n+# The `tee.exe` shipped in Git for Windows v2.49.0 is known to hang frequently\n+# when spawned from `git.exe` and piping its output to `git.exe`. This seems\n+# related to MSYS2 runtime bug fixes regarding the signal handling; Let's just\n+# skip the tests that need to exercise this when the faulty MSYS2 runtime is\n+# detected; The test cases are exercised enough in other matrix jobs of the CI\n+# runs.\n+test_lazy_prereq TEE_DOES_NOT_HANG '\n+\ttest_have_prereq !MINGW &&\n+\tcase \"$(uname -a)\" in *3.5.7-463ebcdc.x86_64*) false;; esac\n+'\n+\n+test_expect_success TEE_DOES_NOT_HANG \\\n+\t'receive-pack missing objects fails connectivity check' '\n \ttest_when_finished rm -rf repo remote.git setup.git &&\n \n \tgit init repo &&\n@@ -62,7 +74,8 @@ test_expect_success 'receive-pack missing objects fails connectivity check' '\n \ttest_must_fail git -C remote.git cat-file -e $(git -C repo rev-parse HEAD)\n '\n \n-test_expect_success 'receive-pack missing objects bypasses connectivity check' '\n+test_expect_success TEE_DOES_NOT_HANG \\\n+\t'receive-pack missing objects bypasses connectivity check' '\n \ttest_when_finished rm -rf repo remote.git setup.git &&\n \n \tgit init repo &&\n\nbase-commit: 0d42fbd9a1f30c63cf0359a1c5aaa77020972f72\n-- \ngitgitgadget\n"},{"id":"519764","messageId":"aEF7NRgEfMhQmDw9@pks.im","threadId":"63587","inReplyTo":"pull.1932.git.1749118606047.gitgitgadget@gmail.com","subject":"Re: [PATCH] t5410: avoid hangs in CI runs in the win+Meson test jobs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-06-05T11:10:45Z","receivedAt":"2025-06-05T11:10:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jun 05, 2025 at 10:16:45AM +0000, Johannes Schindelin via GitGitGadget wrote:\n> diff --git a/t/t5410-receive-pack.sh b/t/t5410-receive-pack.sh\n> index f76a22943ef..09d6bfd2a10 100755\n> --- a/t/t5410-receive-pack.sh\n> +++ b/t/t5410-receive-pack.sh\n> @@ -41,7 +41,19 @@ test_expect_success 'with core.alternateRefsPrefixes' '\n>  \ttest_cmp expect actual.haves\n>  '\n>  \n> -test_expect_success 'receive-pack missing objects fails connectivity check' '\n> +# The `tee.exe` shipped in Git for Windows v2.49.0 is known to hang frequently\n> +# when spawned from `git.exe` and piping its output to `git.exe`. This seems\n> +# related to MSYS2 runtime bug fixes regarding the signal handling; Let's just\n> +# skip the tests that need to exercise this when the faulty MSYS2 runtime is\n> +# detected; The test cases are exercised enough in other matrix jobs of the CI\n> +# runs.\n> +test_lazy_prereq TEE_DOES_NOT_HANG '\n> +\ttest_have_prereq !MINGW &&\n> +\tcase \"$(uname -a)\" in *3.5.7-463ebcdc.x86_64*) false;; esac\n> +'\n> +\n> +test_expect_success TEE_DOES_NOT_HANG \\\n> +\t'receive-pack missing objects fails connectivity check' '\n>  \ttest_when_finished rm -rf repo remote.git setup.git &&\n>  \n>  \tgit init repo &&\n\nQuite interesting. I any case, I think this is a sensible fix for now.\nIt's a known bug, we know it's fixed, we just have to wait. And the fact\nthat this prereq will basically auto-disarm itself once we have the new\nversion is nice.\n\nI did wonder whether we can maybe rewrite the test so that we compute\nour own packfile instead of intercepting the one from git-send-pack(1).\nBut I'm not sure whether that's really worth it.\n\nThanks!\n\nPatrick\n"},{"id":"519770","messageId":"xmqqseke9uzk.fsf@gitster.g","threadId":"63587","inReplyTo":"pull.1932.git.1749118606047.gitgitgadget@gmail.com","subject":"Re: [PATCH] t5410: avoid hangs in CI runs in the win+Meson test jobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-05T13:30:07Z","receivedAt":"2025-06-05T13:30:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\nwrites:\n\n> ...\n> This bug in the MSYS2 runtime has been fixed in the meantime, which is\n> the reason why the same test case causes no problems in the `win test`\n> and the `vs test` jobs.\n>\n> This will continue to be the case until the Git for Windows version on\n> the GitHub runners is upgraded to a version that distributes a newer\n> MSYS2 runtime version. However, as of time of writing, this _is_ the\n> latest Git for Windows version, and will be for another 1.5 weeks, until\n> Git v2.50.0 is scheduled to appear (and shortly thereafter Git for\n> Windows v2.50.0). Traditionally it takes a while before the runners pick\n> up the new version.\n> ...\n>     I finally had a chance to look more closely at this problem. Here is my\n>     alternative to what Patrick proposed in\n>     https://lore.kernel.org/git/aD7tKfXD7YxprSZh@pks.im/.\n\nSuperb.  It must have taken a truly heroic effort.\n\nThanks and congratulations for finally solving the puzzle.\n\nI do agree that Patrick's \"wrap the same in a script\" smelled like\nshifting a timing issue and not truly a solution.\n\n> +# The `tee.exe` shipped in Git for Windows v2.49.0 is known to hang frequently\n> +# when spawned from `git.exe` and piping its output to `git.exe`. This seems\n> +# related to MSYS2 runtime bug fixes regarding the signal handling; Let's just\n> +# skip the tests that need to exercise this when the faulty MSYS2 runtime is\n> +# detected; The test cases are exercised enough in other matrix jobs of the CI\n> +# runs.\n> +test_lazy_prereq TEE_DOES_NOT_HANG '\n> +\ttest_have_prereq !MINGW &&\n> +\tcase \"$(uname -a)\" in *3.5.7-463ebcdc.x86_64*) false;; esac\n> +'\n\nThat's very specific ;-).\n\nAs this is not in a library-ish part, it does not have to be lazy.\nAnybody running this test script need to tell if their environment\nsatisfies the prerequisite, but lazy one does have a documentation\nvalue, I guess, and a bit of extra indirection does not hurt.\n\nWill queue.  Thanks again.\n"}]}