{"thread":{"id":"62756","subject":"[PATCH v3 02/10] t7422: fix flaky test caused by buffered stdout","startedAt":"2025-01-07T12:30:48Z","lastAt":"2025-01-09T07:33:55Z","messageCount":12,"participants":["Patrick Steinhardt","Jeff King"],"isPatch":true,"patchVersion":3,"patchTotal":10},"messages":[{"id":"510067","messageId":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250103-b4-pks-ci-fixes-v1-0-a9bb95dff833@pks.im","subject":"[PATCH v3 00/10] A couple of CI improvements","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:42Z","receivedAt":"2025-01-07T12:30:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series addresses a couple of issues I've found while\ninvestigating flaky CI jobs. Besides two more fixes for flaky jobs it\nalso removes some stale code and simplifies the setup on GitHub Actions\nto always use containerized jobs on Linux.\n\nTest runs can be found for GitLab [1] and GitHub [2].\n\nChanges in v2:\n\n  - Expand a bit on the reasoning behind the conversion to use\n    containerized jobs.\n  - Fix commit message typo.\n  - Properly fix the race in t7422 via pipe stuffing, as proposed by\n    Peff.\n  - Link to v1: https://lore.kernel.org/r/20250103-b4-pks-ci-fixes-v1-0-a9bb95dff833@pks.im\n\nChanges in v3:\n\n  - Another iteration on the SIGPIPE test, which should now finally plug\n    the race.\n  - Link to v2: https://lore.kernel.org/r/20250106-b4-pks-ci-fixes-v2-0-06ae540771b7@pks.im\n\nThanks!\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/277\n[2]: https://github.com/git/git/pull/1865\n\n---\nPatrick Steinhardt (10):\n      t0060: fix EBUSY in MinGW when setting up runtime prefix\n      t7422: fix flaky test caused by buffered stdout\n      github: adapt containerized jobs to be rootless\n      github: convert all Linux jobs to be containerized\n      github: simplify computation of the job's distro\n      gitlab-ci: remove the \"linux-old\" job\n      gitlab-ci: add linux32 job testing against i386\n      ci: stop special-casing for Ubuntu 16.04\n      ci: use latest Ubuntu release\n      ci: remove stale code for Azure Pipelines\n\n .github/workflows/main.yml  | 78 ++++++++++++++++++++++-----------------------\n .gitlab-ci.yml              | 19 ++++++-----\n ci/install-dependencies.sh  |  6 ++--\n ci/lib.sh                   | 34 +++-----------------\n ci/print-test-failures.sh   |  5 ---\n t/t0060-path-utils.sh       | 10 +++---\n t/t7422-submodule-output.sh | 43 ++++++++++++++++++++++---\n 7 files changed, 100 insertions(+), 95 deletions(-)\n\nRange-diff versus v2:\n\n 1:  924cc137a7 =  1:  2d20c22e1c t0060: fix EBUSY in MinGW when setting up runtime prefix\n 2:  85f732d57f <  -:  ---------- t7422: fix flaky test caused by buffered stdout\n -:  ---------- >  2:  97e94a22d0 t7422: fix flaky test caused by buffered stdout\n 3:  96ffed8ad9 =  3:  5f105f2d04 github: adapt containerized jobs to be rootless\n 4:  a276e87563 =  4:  ffcb18fe34 github: convert all Linux jobs to be containerized\n 5:  736d660c31 =  5:  e7a9dc276c github: simplify computation of the job's distro\n 6:  0c2e227cb8 =  6:  7e1f6b651a gitlab-ci: remove the \"linux-old\" job\n 7:  60c3cb0d76 =  7:  03b4a82fc0 gitlab-ci: add linux32 job testing against i386\n 8:  216c043aac =  8:  df57d16eb9 ci: stop special-casing for Ubuntu 16.04\n 9:  234b741805 =  9:  7c63294ace ci: use latest Ubuntu release\n10:  421852878a = 10:  d47387d596 ci: remove stale code for Azure Pipelines\n\n---\nbase-commit: 1b4e9a5f8b5f048972c21fe8acafe0404096f694\nchange-id: 20250103-b4-pks-ci-fixes-2d0a23fb5c78\n\n"},{"id":"510065","messageId":"20250107-b4-pks-ci-fixes-v3-2-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 02/10] t7422: fix flaky test caused by buffered stdout","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:44Z","receivedAt":"2025-01-07T12:30:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One test in t7422 asserts that `git submodule status --recursive`\nproperly handles SIGPIPE. This test is flaky though and may sometimes\nnot see a SIGPIPE at all:\n\n    expecting success of 7422.18 'git submodule status --recursive propagates SIGPIPE':\n            { git submodule status --recursive 2>err; echo $?>status; } |\n                    grep -q X/S &&\n            test_must_be_empty err &&\n            test_match_signal 13 \"$(cat status)\"\n    ++ git submodule status --recursive\n    ++ grep -q X/S\n    ++ echo 0\n    ++ test_must_be_empty err\n    ++ test 1 -ne 1\n    ++ test_path_is_file err\n    ++ test 1 -ne 1\n    ++ test -f err\n    ++ test -s err\n    +++ cat status\n    ++ test_match_signal 13 0\n    ++ test 0 = 141\n    ++ test 0 = 269\n    ++ return 1\n    error: last command exited with $?=1\n    not ok 18 - git submodule status --recursive propagates SIGPIPE\n\nThe issue is caused by us using grep(1) to terminate the pipe on the\nfirst matching line in the recursing git-submodule(1) process. Standard\nstreams are typically buffered though, so this condition is racy and may\ncause us to terminate the pipe after git-submodule(1) has already\nexited, and in that case we wouldn't see the expected signal.\n\nFix the issue by generating a couple thousand nested submodules and\nmatching on the first nested submodule. This ensures that the recursive\ngit-submodule(1) process completely fills its stdout buffer, which makes\nsubsequent writes block until the downstream consumer of the pipe either\nfully drains it or closes it.\n\nTo verify that this works as expected one can apply the following patch\nto the preimage of this commit, which used to reliably trigger the race:\n\n    diff --git a/t/t7422-submodule-output.sh b/t/t7422-submodule-output.sh\n    index 3c5177cc30..df6001f8a0 100755\n    --- a/t/t7422-submodule-output.sh\n    +++ b/t/t7422-submodule-output.sh\n    @@ -202,7 +202,7 @@ test_expect_success !MINGW 'git submodule status --recursive propagates SIGPIPE'\n     \t\tcd repo &&\n     \t\tGIT_ALLOW_PROTOCOL=file git submodule add \"$(pwd)\"/../submodule &&\n     \t\t{ git submodule status --recursive 2>err; echo $?>status; } |\n    -\t\t\tgrep -q recursive-submodule-path-1 &&\n    +\t\t\t{ sleep 1 && grep -q recursive-submodule-path-1 && sleep 1; } &&\n     \t\ttest_must_be_empty err &&\n     \t\ttest_match_signal 13 \"$(cat status)\"\n     \t)\n\nWith the pipe-stuffing workaround the test runs successfully.\n\nHelped-by: Jeff King <peff@peff.net>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n t/t7422-submodule-output.sh | 43 +++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 39 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t7422-submodule-output.sh b/t/t7422-submodule-output.sh\nindex f21e9203678b94701281d5339ae8bfe53d5de0ed..023a5cbdc44bac2389fca45cf7017750627c4ce9 100755\n--- a/t/t7422-submodule-output.sh\n+++ b/t/t7422-submodule-output.sh\n@@ -167,10 +167,45 @@ do\n done\n \n test_expect_success !MINGW 'git submodule status --recursive propagates SIGPIPE' '\n-\t{ git submodule status --recursive 2>err; echo $?>status; } |\n-\t\tgrep -q X/S &&\n-\ttest_must_be_empty err &&\n-\ttest_match_signal 13 \"$(cat status)\"\n+\t# The test setup is somewhat involved because triggering a SIGPIPE is\n+\t# racy with buffered pipes. To avoid the raciness we thus need to make\n+\t# sure that the subprocess in question fills the buffers completely,\n+\t# which requires a couple thousand submodules in total.\n+\ttest_when_finished \"rm -rf submodule repo\" &&\n+\tgit init submodule &&\n+\t(\n+\t\tcd submodule &&\n+\t\ttest_commit initial &&\n+\n+\t\tCOMMIT=$(git rev-parse HEAD) &&\n+\t\tfor i in $(test_seq 2000)\n+\t\tdo\n+\t\t\tprintf \"[submodule \\\"sm-$i\\\"]\\npath = recursive-submodule-path-$i\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >gitmodules &&\n+\t\tBLOB=$(git hash-object -w --stdin <gitmodules) &&\n+\n+\t\tprintf \"100644 blob $BLOB\\t.gitmodules\\n\" >tree &&\n+\t\tfor i in $(test_seq 2000)\n+\t\tdo\n+\t\t\tprintf \"160000 commit $COMMIT\\trecursive-submodule-path-%d\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >>tree &&\n+\t\tTREE=$(git mktree <tree) &&\n+\n+\t\tCOMMIT=$(git commit-tree \"$TREE\") &&\n+\t\tgit reset --hard \"$COMMIT\"\n+\t) &&\n+\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tGIT_ALLOW_PROTOCOL=file git submodule add \"$(pwd)\"/../submodule &&\n+\t\t{ git submodule status --recursive 2>err; echo $?>status; } |\n+\t\t\tgrep -q recursive-submodule-path-1 &&\n+\t\ttest_must_be_empty err &&\n+\t\ttest_match_signal 13 \"$(cat status)\"\n+\t)\n '\n \n test_done\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510066","messageId":"20250107-b4-pks-ci-fixes-v3-1-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 01/10] t0060: fix EBUSY in MinGW when setting up runtime prefix","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:43Z","receivedAt":"2025-01-07T12:30:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Two of our tests in t0060 verify that the runtime prefix functionality\nworks as expected by creating a separate directory hierarchy, copying\nthe Git executable in there and then creating scripts relative to that\nexecutable.\n\nThese tests fail quite regularly in GitLab CI with the following error:\n\n    expecting success of 0060.218 '%(prefix)/ works':\n            mkdir -p pretend/bin &&\n            cp \"$GIT_EXEC_PATH\"/git$X pretend/bin/ &&\n            git config yes.path \"%(prefix)/yes\" &&\n            GIT_EXEC_PATH= ./pretend/bin/git config --path yes.path >actual &&\n            echo \"$(pwd)/pretend/yes\" >expect &&\n            test_cmp expect actual\n    ++ mkdir -p pretend/bin\n    ++ cp /c/GitLab-Runner/builds/gitlab-org/git/git.exe pretend/bin/\n    cp: cannot create regular file 'pretend/bin/git.exe': Device or resource busy\n    error: last command exited with $?=1\n    not ok 218 - %(prefix)/ works\n\nSeemingly, the \"git.exe\" binary we are trying to overwrite is still\nbeing held open. It is somewhat puzzling why exactly that is: while the\npreceding test _does_ write to and execute the same path, it should have\nexited and shouldn't keep any backgrounded processes around. So it must\nbe held open by something else, either in MinGW or in Windows itself.\n\nWhile the root cause is puzzling, the workaround is trivial enough:\ninstead of writing the file twice we simply pull the common setup into a\nseparate test case so that we won't observe EBUSY in the first place.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n t/t0060-path-utils.sh | 10 ++++++----\n 1 file changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t0060-path-utils.sh b/t/t0060-path-utils.sh\nindex dbb2e73bcd912ae6a804603ff54e4c609966fa5d..8545cdfab559b4e247cb2699965e637529fd930a 100755\n--- a/t/t0060-path-utils.sh\n+++ b/t/t0060-path-utils.sh\n@@ -592,17 +592,19 @@ test_lazy_prereq CAN_EXEC_IN_PWD '\n \t./git rev-parse\n '\n \n+test_expect_success !VALGRIND,RUNTIME_PREFIX,CAN_EXEC_IN_PWD 'setup runtime prefix' '\n+\tmkdir -p pretend/bin &&\n+\tcp \"$GIT_EXEC_PATH\"/git$X pretend/bin/\n+'\n+\n test_expect_success !VALGRIND,RUNTIME_PREFIX,CAN_EXEC_IN_PWD 'RUNTIME_PREFIX works' '\n-\tmkdir -p pretend/bin pretend/libexec/git-core &&\n+\tmkdir -p pretend/libexec/git-core &&\n \techo \"echo HERE\" | write_script pretend/libexec/git-core/git-here &&\n-\tcp \"$GIT_EXEC_PATH\"/git$X pretend/bin/ &&\n \tGIT_EXEC_PATH= ./pretend/bin/git here >actual &&\n \techo HERE >expect &&\n \ttest_cmp expect actual'\n \n test_expect_success !VALGRIND,RUNTIME_PREFIX,CAN_EXEC_IN_PWD '%(prefix)/ works' '\n-\tmkdir -p pretend/bin &&\n-\tcp \"$GIT_EXEC_PATH\"/git$X pretend/bin/ &&\n \tgit config yes.path \"%(prefix)/yes\" &&\n \tGIT_EXEC_PATH= ./pretend/bin/git config --path yes.path >actual &&\n \techo \"$(pwd)/pretend/yes\" >expect &&\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510068","messageId":"20250107-b4-pks-ci-fixes-v3-3-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 03/10] github: adapt containerized jobs to be rootless","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:45Z","receivedAt":"2025-01-07T12:30:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The containerized jobs in GitHub Actions run as root, giving them\nspecial permissions to for example delete files even when the user\nshouldn't be able to due to file permissions. This limitation keeps us\nfrom using containerized jobs for most of our Ubuntu-based jobs as it\ncauses a number of tests to fail.\n\nAdapt the jobs to create a separate user that executes the test suite.\nThis follows similar infrastructure that we already have in GitLab CI.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n .github/workflows/main.yml | 6 ++++--\n ci/install-dependencies.sh | 2 +-\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 900be9957a23fcaa64e1aefd0c8638c5f84b7997..b02f5873a540b458d38e7951b4ee3d5ca598ae23 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -371,10 +371,12 @@ jobs:\n       run: apt -q update && apt -q -y install libc6-amd64 lib64stdc++6\n     - uses: actions/checkout@v4\n     - run: ci/install-dependencies.sh\n-    - run: ci/run-build-and-tests.sh\n+    - run: useradd builder --create-home\n+    - run: chown -R builder .\n+    - run: sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh\n     - name: print test failures\n       if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n-      run: ci/print-test-failures.sh\n+      run: sudo --preserve-env --set-home --user=builder ci/print-test-failures.sh\n     - name: Upload failed tests' directories\n       if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n       uses: actions/upload-artifact@v4\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex d1cb9fa8785388b3674fcea4dd682abc0725c968..ecb5b9d36c20d3e7e96148ac628a96c62642c308 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -31,7 +31,7 @@ alpine-*)\n \t;;\n fedora-*|almalinux-*)\n \tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n ubuntu-*|ubuntu32-*|debian-*)\n \t# Required so that apt doesn't wait for user input on certain packages.\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510070","messageId":"20250107-b4-pks-ci-fixes-v3-4-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 04/10] github: convert all Linux jobs to be containerized","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:46Z","receivedAt":"2025-01-07T12:30:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have split the CI jobs in GitHub Workflows into two categories:\n\n  - Those running on a machine pool directly.\n\n  - Those running in a container on the machine pool.\n\nThe latter is more flexible because it allows us to freely pick whatever\ncontainer image we want to use for a specific job, while the former only\nallows us to pick from a handful of different distros. The containerized\njobs do not have any significant downsides to the best of my knowledge:\n\n  - They aren't significantly slower to start up. A quick comparison by\n    Peff shows that the difference is mostly lost in the noise:\n\n            job             |  old | new\n        --------------------|------|------\n        linux-TEST-vars      11m30s 10m54s\n        linux-asan-ubsan     30m26s 31m14s\n        linux-gcc             9m47s 10m6s\n        linux-gcc-default     9m47s  9m41s\n        linux-leaks          25m50s 25m21s\n        linux-meson          10m36s 10m41s\n        linux-reftable       10m25s 10m23s\n        linux-reftable-leaks 27m18s 27m28s\n        linux-sha256          9m54s 10m31s\n\n    Some jobs are a bit faster, some are a bit slower, but there does\n    not seem to be any significant change.\n\n  - Containerized jobs run as root, which keeps a couple of tests from\n    running. This has been addressed in the preceding commit though,\n    where we now use setpriv(1) to run tests as a separate user.\n\n  - GitHub injects a Node binary into containerized jobs, which is\n    dynamically linked. This has led to some issues in the past [1], but\n    only for our 32 bit jobs. The issues have since been resolved.\n\nOverall there seem to be no downsides, but the upside is that we have\nmore control over the exact image that these jobs use. Convert the Linux\njobs accordingly.\n\n[1]: https://lore.kernel.org/git/20240912094841.GD589828@coredump.intra.peff.net/\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n .github/workflows/main.yml | 68 ++++++++++++++++++++++++++--------------------\n 1 file changed, 39 insertions(+), 29 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex b02f5873a540b458d38e7951b4ee3d5ca598ae23..8e5847da4fab009ad699c18e1a5a336a8b45c3ed 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -259,20 +259,6 @@ jobs:\n       fail-fast: false\n       matrix:\n         vector:\n-          - jobname: linux-sha256\n-            cc: clang\n-            pool: ubuntu-latest\n-          - jobname: linux-reftable\n-            cc: clang\n-            pool: ubuntu-latest\n-          - jobname: linux-gcc\n-            cc: gcc\n-            cc_package: gcc-8\n-            pool: ubuntu-20.04\n-          - jobname: linux-TEST-vars\n-            cc: gcc\n-            cc_package: gcc-8\n-            pool: ubuntu-20.04\n           - jobname: osx-clang\n             cc: clang\n             pool: macos-13\n@@ -285,21 +271,6 @@ jobs:\n           - jobname: osx-meson\n             cc: clang\n             pool: macos-13\n-          - jobname: linux-gcc-default\n-            cc: gcc\n-            pool: ubuntu-latest\n-          - jobname: linux-leaks\n-            cc: gcc\n-            pool: ubuntu-latest\n-          - jobname: linux-reftable-leaks\n-            cc: gcc\n-            pool: ubuntu-latest\n-          - jobname: linux-asan-ubsan\n-            cc: clang\n-            pool: ubuntu-latest\n-          - jobname: linux-meson\n-            cc: gcc\n-            pool: ubuntu-latest\n     env:\n       CC: ${{matrix.vector.cc}}\n       CC_PACKAGE: ${{matrix.vector.cc_package}}\n@@ -342,6 +313,44 @@ jobs:\n       fail-fast: false\n       matrix:\n         vector:\n+        - jobname: linux-sha256\n+          image: ubuntu:latest\n+          cc: clang\n+          distro: ubuntu-latest\n+        - jobname: linux-reftable\n+          image: ubuntu:latest\n+          cc: clang\n+          distro: ubuntu-latest\n+        - jobname: linux-gcc\n+          image: ubuntu:20.04\n+          cc: gcc\n+          cc_package: gcc-8\n+          distro: ubuntu-20.04\n+        - jobname: linux-TEST-vars\n+          image: ubuntu:20.04\n+          cc: gcc\n+          cc_package: gcc-8\n+          distro: ubuntu-20.04\n+        - jobname: linux-gcc-default\n+          image: ubuntu:latest\n+          cc: gcc\n+          distro: ubuntu-latest\n+        - jobname: linux-leaks\n+          image: ubuntu:latest\n+          cc: gcc\n+          distro: ubuntu-latest\n+        - jobname: linux-reftable-leaks\n+          image: ubuntu:latest\n+          cc: gcc\n+          distro: ubuntu-latest\n+        - jobname: linux-asan-ubsan\n+          image: ubuntu:latest\n+          cc: clang\n+          distro: ubuntu-latest\n+        - jobname: linux-meson\n+          image: ubuntu:latest\n+          cc: gcc\n+          distro: ubuntu-latest\n         - jobname: linux-musl\n           image: alpine\n           distro: alpine-latest\n@@ -363,6 +372,7 @@ jobs:\n     env:\n       jobname: ${{matrix.vector.jobname}}\n       distro: ${{matrix.vector.distro}}\n+      CC: ${{matrix.vector.cc}}\n     runs-on: ubuntu-latest\n     container: ${{matrix.vector.image}}\n     steps:\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510069","messageId":"20250107-b4-pks-ci-fixes-v3-6-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 06/10] gitlab-ci: remove the \"linux-old\" job","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:48Z","receivedAt":"2025-01-07T12:30:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"linux-old\" job was historically testing against the oldest\nsupported LTS release of Ubuntu. But with c85bcb5de1 (gitlab-ci: switch\nfrom Ubuntu 16.04 to 20.04, 2024-10-31) it has been converted to test\nagainst Ubuntu 20.04, which already gets exercised in a couple of other\nCI jobs. It's thus not adding any significant test coverage.\n\nDrop the job.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n .gitlab-ci.yml | 3 ---\n 1 file changed, 3 deletions(-)\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 9254e01583306e67dc12b6b9e0015183e1108655..00bc727865031620752771af4a9030c7de1b73df 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -36,9 +36,6 @@ test:linux:\n       fi\n   parallel:\n     matrix:\n-      - jobname: linux-old\n-        image: ubuntu:20.04\n-        CC: gcc\n       - jobname: linux-sha256\n         image: ubuntu:latest\n         CC: clang\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510072","messageId":"20250107-b4-pks-ci-fixes-v3-5-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 05/10] github: simplify computation of the job's distro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:47Z","receivedAt":"2025-01-07T12:30:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We explicitly list the distro of Linux-based jobs, but it is equivalent\nto the name of the image in almost all cases, except that colons are\nreplaced with dashes. Drop the redundant information and massage it in\nour CI scripts, which is equivalent to how we do it in GitLab CI.\n\nThere are a couple of exceptions:\n\n  - The \"linux32\" job, whose distro name is different than the image\n    name. This is handled by adapting all sites to use the new name.\n\n  - The \"alpine\" and \"fedora\" jobs, neither of which specify a tag for\n    their image. This is handled by adding the \"latest\" tag.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n .github/workflows/main.yml | 22 ++++------------------\n ci/install-dependencies.sh |  4 ++--\n ci/lib.sh                  |  2 ++\n 3 files changed, 8 insertions(+), 20 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 8e5847da4fab009ad699c18e1a5a336a8b45c3ed..b54da639a650682495994e3c7b137eab4e6cb3bf 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -275,7 +275,7 @@ jobs:\n       CC: ${{matrix.vector.cc}}\n       CC_PACKAGE: ${{matrix.vector.cc_package}}\n       jobname: ${{matrix.vector.jobname}}\n-      distro: ${{matrix.vector.pool}}\n+      CI_JOB_IMAGE: ${{matrix.vector.pool}}\n       TEST_OUTPUT_DIRECTORY: ${{github.workspace}}/t\n     runs-on: ${{matrix.vector.pool}}\n     steps:\n@@ -316,63 +316,49 @@ jobs:\n         - jobname: linux-sha256\n           image: ubuntu:latest\n           cc: clang\n-          distro: ubuntu-latest\n         - jobname: linux-reftable\n           image: ubuntu:latest\n           cc: clang\n-          distro: ubuntu-latest\n         - jobname: linux-gcc\n           image: ubuntu:20.04\n           cc: gcc\n           cc_package: gcc-8\n-          distro: ubuntu-20.04\n         - jobname: linux-TEST-vars\n           image: ubuntu:20.04\n           cc: gcc\n           cc_package: gcc-8\n-          distro: ubuntu-20.04\n         - jobname: linux-gcc-default\n           image: ubuntu:latest\n           cc: gcc\n-          distro: ubuntu-latest\n         - jobname: linux-leaks\n           image: ubuntu:latest\n           cc: gcc\n-          distro: ubuntu-latest\n         - jobname: linux-reftable-leaks\n           image: ubuntu:latest\n           cc: gcc\n-          distro: ubuntu-latest\n         - jobname: linux-asan-ubsan\n           image: ubuntu:latest\n           cc: clang\n-          distro: ubuntu-latest\n         - jobname: linux-meson\n           image: ubuntu:latest\n           cc: gcc\n-          distro: ubuntu-latest\n         - jobname: linux-musl\n-          image: alpine\n-          distro: alpine-latest\n+          image: alpine:latest\n         # Supported until 2025-04-02.\n         - jobname: linux32\n           image: i386/ubuntu:focal\n-          distro: ubuntu32-20.04\n         - jobname: pedantic\n-          image: fedora\n-          distro: fedora-latest\n+          image: fedora:latest\n         # A RHEL 8 compatible distro.  Supported until 2029-05-31.\n         - jobname: almalinux-8\n           image: almalinux:8\n-          distro: almalinux-8\n         # Supported until 2026-08-31.\n         - jobname: debian-11\n           image: debian:11\n-          distro: debian-11\n     env:\n       jobname: ${{matrix.vector.jobname}}\n-      distro: ${{matrix.vector.distro}}\n       CC: ${{matrix.vector.cc}}\n+      CI_JOB_IMAGE: ${{matrix.vector.image}}\n     runs-on: ubuntu-latest\n     container: ${{matrix.vector.image}}\n     steps:\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex ecb5b9d36c20d3e7e96148ac628a96c62642c308..d5a959e25ff3236656ff3416b81732ec5c2107c1 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -33,7 +33,7 @@ fedora-*|almalinux-*)\n \tdnf -yq update >/dev/null &&\n \tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n-ubuntu-*|ubuntu32-*|debian-*)\n+ubuntu-*|i386/ubuntu-*|debian-*)\n \t# Required so that apt doesn't wait for user input on certain packages.\n \texport DEBIAN_FRONTEND=noninteractive\n \n@@ -42,7 +42,7 @@ ubuntu-*|ubuntu32-*|debian-*)\n \t\tSVN='libsvn-perl subversion'\n \t\tLANGUAGES='language-pack-is'\n \t\t;;\n-\tubuntu32-*)\n+\ti386/ubuntu-*)\n \t\tSVN=\n \t\tLANGUAGES='language-pack-is'\n \t\t;;\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 8885ee3c3f86c62e8783d27756b8779bd491e7e6..f8b68ab8a6546802756fd516ca15a2c97223da5f 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -246,6 +246,8 @@ then\n \n \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n \tJOBS=10\n+\n+\tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n elif test true = \"$GITLAB_CI\"\n then\n \tCI_TYPE=gitlab-ci\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510071","messageId":"20250107-b4-pks-ci-fixes-v3-7-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 07/10] gitlab-ci: add linux32 job testing against i386","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:49Z","receivedAt":"2025-01-07T12:30:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add another job to GitLab CI that tests against the i386 architecture.\nThis job is equivalent to the same job in GitHub Workflows.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n .gitlab-ci.yml | 2 ++\n ci/lib.sh      | 2 +-\n 2 files changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 00bc727865031620752771af4a9030c7de1b73df..29e9056dd5010f8843e42aeae8410973c825de54 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -66,6 +66,8 @@ test:linux:\n         image: fedora:latest\n       - jobname: linux-musl\n         image: alpine:latest\n+      - jobname: linux32\n+        image: i386/ubuntu:20.04\n       - jobname: linux-meson\n         image: ubuntu:latest\n         CC: gcc\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex f8b68ab8a6546802756fd516ca15a2c97223da5f..2293849ada3b45873f80e4392ab93c65657d0f13 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -269,7 +269,7 @@ then\n \t\tCI_OS_NAME=osx\n \t\tJOBS=$(nproc)\n \t\t;;\n-\t*,alpine:*|*,fedora:*|*,ubuntu:*)\n+\t*,alpine:*|*,fedora:*|*,ubuntu:*|*,i386/ubuntu:*)\n \t\tCI_OS_NAME=linux\n \t\tJOBS=$(nproc)\n \t\t;;\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510074","messageId":"20250107-b4-pks-ci-fixes-v3-8-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 08/10] ci: stop special-casing for Ubuntu 16.04","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:50Z","receivedAt":"2025-01-07T12:30:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"With c85bcb5de1 (gitlab-ci: switch from Ubuntu 16.04 to 20.04,\n2024-10-31) we have adapted the last CI job to stop using Ubuntu 16.04\nin favor of Ubuntu 20.04. Remove the special-casing we still have in our\nCI scripts.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n ci/lib.sh | 9 +--------\n 1 file changed, 1 insertion(+), 8 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 2293849ada3b45873f80e4392ab93c65657d0f13..77a4aabdb8fb416c1733f02d02145b6bc0849998 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -347,14 +347,7 @@ ubuntu-*)\n \tfi\n \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE\"\n \n-\tcase \"$distro\" in\n-\tubuntu-16.04)\n-\t\t# Apache is too old for HTTP/2.\n-\t\t;;\n-\t*)\n-\t\texport GIT_TEST_HTTPD=true\n-\t\t;;\n-\tesac\n+\texport GIT_TEST_HTTPD=true\n \n \t# The Linux build installs the defined dependency versions below.\n \t# The OS X build installs much more recent versions, whichever\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510073","messageId":"20250107-b4-pks-ci-fixes-v3-9-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 09/10] ci: use latest Ubuntu release","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:51Z","receivedAt":"2025-01-07T12:30:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Both GitHub Actions and GitLab CI use the \"ubuntu:latest\" tag as the\ndefault image for most jobs. This tag is somewhat misleading though, as\nit does not refer to the latest release of Ubuntu, but to the latest LTS\nrelease thereof. But as we already have a couple of jobs exercising the\noldest LTS release of Ubuntu that Git still supports, it would make more\nsense to test the oldest and youngest versions of Ubuntu.\n\nAdapt these jobs to instead use the \"ubuntu:rolling\" tag, which refers\nto the actual latest release, which currently is Ubuntu 24.10.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n .github/workflows/main.yml | 14 +++++++-------\n .gitlab-ci.yml             | 14 +++++++-------\n 2 files changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex b54da639a650682495994e3c7b137eab4e6cb3bf..b90381ae015edf9db5aa4b8c0ace9bb5c549c37b 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -314,10 +314,10 @@ jobs:\n       matrix:\n         vector:\n         - jobname: linux-sha256\n-          image: ubuntu:latest\n+          image: ubuntu:rolling\n           cc: clang\n         - jobname: linux-reftable\n-          image: ubuntu:latest\n+          image: ubuntu:rolling\n           cc: clang\n         - jobname: linux-gcc\n           image: ubuntu:20.04\n@@ -328,19 +328,19 @@ jobs:\n           cc: gcc\n           cc_package: gcc-8\n         - jobname: linux-gcc-default\n-          image: ubuntu:latest\n+          image: ubuntu:rolling\n           cc: gcc\n         - jobname: linux-leaks\n-          image: ubuntu:latest\n+          image: ubuntu:rolling\n           cc: gcc\n         - jobname: linux-reftable-leaks\n-          image: ubuntu:latest\n+          image: ubuntu:rolling\n           cc: gcc\n         - jobname: linux-asan-ubsan\n-          image: ubuntu:latest\n+          image: ubuntu:rolling\n           cc: clang\n         - jobname: linux-meson\n-          image: ubuntu:latest\n+          image: ubuntu:rolling\n           cc: gcc\n         - jobname: linux-musl\n           image: alpine:latest\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 29e9056dd5010f8843e42aeae8410973c825de54..8ed3ff5f0373d70b6f609dc5292dda2dd7fd8f88 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -37,10 +37,10 @@ test:linux:\n   parallel:\n     matrix:\n       - jobname: linux-sha256\n-        image: ubuntu:latest\n+        image: ubuntu:rolling\n         CC: clang\n       - jobname: linux-reftable\n-        image: ubuntu:latest\n+        image: ubuntu:rolling\n         CC: clang\n       - jobname: linux-gcc\n         image: ubuntu:20.04\n@@ -51,16 +51,16 @@ test:linux:\n         CC: gcc\n         CC_PACKAGE: gcc-8\n       - jobname: linux-gcc-default\n-        image: ubuntu:latest\n+        image: ubuntu:rolling\n         CC: gcc\n       - jobname: linux-leaks\n-        image: ubuntu:latest\n+        image: ubuntu:rolling\n         CC: gcc\n       - jobname: linux-reftable-leaks\n-        image: ubuntu:latest\n+        image: ubuntu:rolling\n         CC: gcc\n       - jobname: linux-asan-ubsan\n-        image: ubuntu:latest\n+        image: ubuntu:rolling\n         CC: clang\n       - jobname: pedantic\n         image: fedora:latest\n@@ -69,7 +69,7 @@ test:linux:\n       - jobname: linux32\n         image: i386/ubuntu:20.04\n       - jobname: linux-meson\n-        image: ubuntu:latest\n+        image: ubuntu:rolling\n         CC: gcc\n   artifacts:\n     paths:\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510075","messageId":"20250107-b4-pks-ci-fixes-v3-10-546a0ebc8481@pks.im","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-0-546a0ebc8481@pks.im","subject":"[PATCH v3 10/10] ci: remove stale code for Azure Pipelines","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-07T12:30:52Z","receivedAt":"2025-01-07T12:30:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Support for Azure Pipelines has been retired in 6081d3898f (ci: retire\nthe Azure Pipelines definition, 2020-04-11) in favor of GitHub Actions.\nOur CI library still has some infrastructure left for Azure though that\nis now unused. Remove it.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n ci/lib.sh                 | 21 +--------------------\n ci/print-test-failures.sh |  5 -----\n 2 files changed, 1 insertion(+), 25 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 77a4aabdb8fb416c1733f02d02145b6bc0849998..4003354f16c048b969c0bb4340d2ee2777767300 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -206,26 +206,7 @@ export TERM=${TERM:-dumb}\n # Clear MAKEFLAGS that may come from the outside world.\n export MAKEFLAGS=\n \n-if test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n-then\n-\tCI_TYPE=azure-pipelines\n-\t# We are running in Azure Pipelines\n-\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n-\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n-\tCI_JOB_ID=\"$BUILD_BUILDID\"\n-\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n-\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n-\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n-\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n-\tCC=\"${CC:-gcc}\"\n-\n-\t# use a subdirectory of the cache dir (because the file share is shared\n-\t# among *all* phases)\n-\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n-\n-\tGIT_TEST_OPTS=\"--write-junit-xml\"\n-\tJOBS=10\n-elif test true = \"$GITHUB_ACTIONS\"\n+if test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n \tCI_BRANCH=\"$GITHUB_REF\"\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 655687dd827e5b3e4d4879803b0d4499e7751380..dc910e51609cd7344b1ad03fdb4e820e47ad3a88 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -39,11 +39,6 @@ do\n \t\ttest_name=\"${test_name##*/}\"\n \t\ttrash_dir=\"trash directory.$test_name\"\n \t\tcase \"$CI_TYPE\" in\n-\t\tazure-pipelines)\n-\t\t\tmkdir -p failed-test-artifacts\n-\t\t\tmv \"$trash_dir\" failed-test-artifacts\n-\t\t\tcontinue\n-\t\t\t;;\n \t\tgithub-actions)\n \t\t\tmkdir -p failed-test-artifacts\n \t\t\techo \"FAILED_TEST_ARTIFACTS=${TEST_OUTPUT_DIRECTORY:t}/failed-test-artifacts\" >>$GITHUB_ENV\n\n-- \n2.48.0.rc1.245.gb3e6e7acbc.dirty\n\n"},{"id":"510219","messageId":"20250109073354.GB2735258@coredump.intra.peff.net","threadId":"62756","inReplyTo":"20250107-b4-pks-ci-fixes-v3-2-546a0ebc8481@pks.im","subject":"Re: [PATCH v3 02/10] t7422: fix flaky test caused by buffered stdout","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-09T07:33:54Z","receivedAt":"2025-01-09T07:33:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 07, 2025 at 01:30:44PM +0100, Patrick Steinhardt wrote:\n\n> The issue is caused by us using grep(1) to terminate the pipe on the\n> first matching line in the recursing git-submodule(1) process. Standard\n> streams are typically buffered though, so this condition is racy and may\n> cause us to terminate the pipe after git-submodule(1) has already\n> exited, and in that case we wouldn't see the expected signal.\n\nThis patch looks good to me overall, but I think there are a few small\ninaccuracies in the commit message.\n\nI don't think buffering is relevant here. Especially since in the\noriginal test there isn't any buffering (the output comes from separate\nrecursive status processes, so each line gets its own write() call).\n\nThe race you're seeing is:\n\n  1. git-submodule (or its child process) writes the first X/S line\n     we're trying to match\n\n  2. grep matches the line\n\n  3a. grep exits, closing the pipe\n\n  3b. git-submodule (or its children) writes the rest of its lines.\n\nSteps 3a and 3b happen at the same time without any guarantees. If 3a\nhappens first, we get SIGPIPE. Otherwise, we don't (and the test fails).\n\nAnd when git-submodule exits is not important. The critical timing is\nwhen it does its final write(). If the pipe closes after that, even if\nit is still running, it will not notice. (I admit that one is a\nmicro-nit, though).\n\n> Fix the issue by generating a couple thousand nested submodules and\n> matching on the first nested submodule. This ensures that the recursive\n> git-submodule(1) process completely fills its stdout buffer, which makes\n> subsequent writes block until the downstream consumer of the pipe either\n> fully drains it or closes it.\n\nOne more micro-nit: \"fully drains\" is not accurate. If grep reads\nanother 4096 bytes, then that opens up 4096 more bytes in the pipe\nbuffer. And git-submodule can then write to it. Not important for our\ncase, since we \"know\" grep will not read more after matching, but will\njust close. So it is really more like \"block until the downstream\nconsumer either reads more or closes it\".\n\nThat \"know\" is a little bit of an assumption. In particular, there is no\nreason grep could not read into a 1MB buffer, consuming the whole input,\nmatch within it, and only then exit. And then we'd be racy again. In\npractice I'm not too worried about this. I'd be surprised by a buffer\nbigger than 8k (which is what I saw when I strace'd it on my Linux\nsystem), and your generated input is something like 160k. That should\nfill even a generous pipe buffer plus grep input buffer.\n\n> +\tgit init submodule &&\n> +\t(\n> +\t\tcd submodule &&\n> +\t\ttest_commit initial &&\n> +\n> +\t\tCOMMIT=$(git rev-parse HEAD) &&\n> +\t\tfor i in $(test_seq 2000)\n> +\t\tdo\n> +\t\t\tprintf \"[submodule \\\"sm-$i\\\"]\\npath = recursive-submodule-path-$i\\n\" \"$i\" ||\n> +\t\t\treturn 1\n> +\t\tdone >gitmodules &&\n> +\t\tBLOB=$(git hash-object -w --stdin <gitmodules) &&\n> +\n> +\t\tprintf \"100644 blob $BLOB\\t.gitmodules\\n\" >tree &&\n> +\t\tfor i in $(test_seq 2000)\n> +\t\tdo\n> +\t\t\tprintf \"160000 commit $COMMIT\\trecursive-submodule-path-%d\\n\" \"$i\" ||\n> +\t\t\treturn 1\n> +\t\tdone >>tree &&\n> +\t\tTREE=$(git mktree <tree) &&\n> +\n> +\t\tCOMMIT=$(git commit-tree \"$TREE\") &&\n> +\t\tgit reset --hard \"$COMMIT\"\n> +\t) &&\n\nOK, so the submodule has a huge number of missing children. But that's\nenough for us, because the \"-\" lines generated by \"submodule status\" are\nfine. So you're able to generate the whole thing with only printf\nprocesses running in the loops. Very clever.\n\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tGIT_ALLOW_PROTOCOL=file git submodule add \"$(pwd)\"/../submodule &&\n> +\t\t{ git submodule status --recursive 2>err; echo $?>status; } |\n> +\t\t\tgrep -q recursive-submodule-path-1 &&\n> +\t\ttest_must_be_empty err &&\n> +\t\ttest_match_signal 13 \"$(cat status)\"\n> +\t)\n\nAnd then adding another repo wrapping it is what makes it recursive.\nNice.\n\n-Peff\n"}]}