{"thread":{"id":"65901","subject":"[PATCH] ci(dockerized): reduce the PID limit for private repositories","startedAt":"2026-07-01T07:04:48Z","lastAt":"2026-07-05T05:28:04Z","messageCount":5,"participants":["Johannes Schindelin via GitGitGadget","Junio C Hamano","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"546840","messageId":"pull.2164.git.1782889484346.gitgitgadget@gmail.com","threadId":"65901","inReplyTo":null,"subject":"[PATCH] ci(dockerized): reduce the PID limit for private repositories","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-07-01T07:04:43Z","receivedAt":"2026-07-01T07:04:48Z","isPatch":true,"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nEvery once in a while I need to verify that Microsoft Git's test suite\npasses for changes that are not yet meant for public consumption, and\nsince it was (made) too difficult to keep up a working Azure Pipeline\ndefinition, I have to use GitHub Actions in a private GitHub repository\nfor that purpose.\n\nIn these tests, basically all Dockerized CI jobs fail consistently. The\nsymptom is something like:\n\n  error: cannot create async thread: Resource temporarily unavailable\n\nin the middle of a test, typically in the t5xxx-t6xxx range. The first\nsuch error is immediately followed by plenty more of these errors, and\nnot a single test succeeds afterwards.\n\nAt first, I thought that maybe the massive parallelism I enjoy there is\nthe problem, and I thought that the cgroups limits might be shared\nbetween the many containers that run on essentially the same physical\nmachine. But even reducing the matrix to just a single of those\nDockerized jobs runs into the very same problems.\n\nThe underlying reason seems to be a substantial difference in the hosted\nrunners that execute these Dockerized jobs: forcing the PID limit of the\ncontainer to a high number lets the jobs pass, even when running the\ncomplete matrix of all 13 Dockerized jobs concurrently. But that's not\nthe only difference: The jobs seem to take a lot longer in these\ncontainers than, say, in the containers made available to\nhttps://github.com/git/git.\n\nWhen forcing a PID limit of 64k in that private repository, the jobs\ncompleted successfully, but they also took a lot longer, between 2x to\n2.5x longer, i.e. painfully much longer. Reducing the PID limit to 16k,\nthe CI jobs still passed, but took an equally long amount of time.\nReducing the PID limit to 8k caused the errors to reappear.\n\nHere are the numbers from three example runs, the first one forcing the\nPID and nproc limit to 65536, the second one to 16384, the third run is\nfrom the public git/git repository:\n\nJob                           | 64k     | 16k     | reference\n------------------------------|---------|---------|---------\nalmalinux-8                   | 19m 3s  | 16m 0s  | 9m 36s\ndebian-11                     | 20m 31s | 20m 3s  | 8m 5s\nfedora-breaking-changes-meson | 16m 29s | 19m 19s | 9m 40s\nlinux-asan-ubsan              | 1h 10m  | 1h 11m  | 34m 36s\nlinux-breaking-changes        | 25m 39s | 25m 58s | 13m 15s\nlinux-leaks                   | 1h 9m   | 1h 10m  | 33m 30s\nlinux-meson                   | 28m 9s  | 27m 4s  | 13m 45s\nlinux-musl-meson              | 16m 32s | 13m 39s | 8m 6s\nlinux-reftable-leaks          | 1h 13m  | 1h 13m  | 34m 34s\nlinux-reftable                | 26m 2s  | 25m 48s | 13m 31s\nlinux-sha256                  | 26m 12s | 26m 3s  | 12m 36s\nlinux-TEST-vars               | 26m 5s  | 25m 21s | 13m 25s\nlinux32                       | 21m 16s | 19m 57s | 10m 44s\n\nIt does not look as if the PID limit is the reason for the longer\nruntime, seeing as the 64k vs 16k timings deviate no more than as is\nusual with GitHub workflows. So let's go for 16k.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n    ci(dockerized): reduce the PID limit for private repositories\n    \n    I needed to craft this patch while developing fixes for vulnerabilities\n    which eventually were published as Git for Windows v2.53.0(3).\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-2164%2Fdscho%2Fraise-pid-limit-in-private-repositories-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2164/dscho/raise-pid-limit-in-private-repositories-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/2164\n\n .github/workflows/main.yml | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex cf341d74db..85cfedf5b0 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -420,7 +420,9 @@ jobs:\n       CI_JOB_IMAGE: ${{matrix.vector.image}}\n       CUSTOM_PATH: /custom\n     runs-on: ubuntu-latest\n-    container: ${{matrix.vector.image}}\n+    container:\n+      image: ${{ matrix.vector.image }}\n+      options: ${{ github.repository_visibility == 'private' && '--pids-limit 16384 --ulimit nproc=16384:16384 --ulimit nofile=32768:32768' || '' }}\n     steps:\n     - name: prepare libc6 for actions\n       if: matrix.vector.jobname == 'linux32'\n\nbase-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc\n-- \ngitgitgadget\n"},{"id":"546924","messageId":"xmqq5x2yps4c.fsf@gitster.g","threadId":"65901","inReplyTo":"pull.2164.git.1782889484346.gitgitgadget@gmail.com","subject":"Re: [PATCH] ci(dockerized): reduce the PID limit for private repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-07-01T16:54:59Z","receivedAt":"2026-07-01T16:55:01Z","isPatch":true,"body":"\"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\nwrites:\n\n> The underlying reason seems to be a substantial difference in the hosted\n> runners that execute these Dockerized jobs: forcing the PID limit of the\n> container to a high number lets the jobs pass, even when running the\n> complete matrix of all 13 Dockerized jobs concurrently.\n\nIs the \"reduce\" in the title accurate?  The above description tells\nme that what you did was to \"raise\" the PID limit (i.e., forcing the\nPID limit to a high number), presumably because the default PID\nlimit is way too low for the tests to pass?\n\nAnd that fix turns constant failures into success (albeit the tests\nrun very slowly, which is a separate topic that you discuss next).\n\n> But that's not\n> the only difference: The jobs seem to take a lot longer in these\n> containers than, say, in the containers made available to\n> https://github.com/git/git.\n>\n> When forcing a PID limit of 64k in that private repository, the jobs\n> completed successfully, but they also took a lot longer, between 2x to\n> 2.5x longer, i.e. painfully much longer. Reducing the PID limit to 16k,\n> the CI jobs still passed, but took an equally long amount of time.\n> Reducing the PID limit to 8k caused the errors to reappear.\n> ...\n> It does not look as if the PID limit is the reason for the longer\n> runtime, seeing as the 64k vs 16k timings deviate no more than as is\n> usual with GitHub workflows. So let's go for 16k.\n\nSo 8k is too low to make them pass, just like the default setting\n(whatever it is), but 16k is sufficient, so this patch settles at\nthat number, which makes sense.\n\n> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n> index cf341d74db..85cfedf5b0 100644\n> --- a/.github/workflows/main.yml\n> +++ b/.github/workflows/main.yml\n> @@ -420,7 +420,9 @@ jobs:\n>        CI_JOB_IMAGE: ${{matrix.vector.image}}\n>        CUSTOM_PATH: /custom\n>      runs-on: ubuntu-latest\n> -    container: ${{matrix.vector.image}}\n> +    container:\n> +      image: ${{ matrix.vector.image }}\n> +      options: ${{ github.repository_visibility == 'private' && '--pids-limit 16384 --ulimit nproc=16384:16384 --ulimit nofile=32768:32768' || '' }}\n>      steps:\n>      - name: prepare libc6 for actions\n>        if: matrix.vector.jobname == 'linux32'\n>\n> base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc\n"},{"id":"547126","messageId":"c2986e77-1622-a148-98a9-2f7ce6717827@gmx.de","threadId":"65901","inReplyTo":"xmqq5x2yps4c.fsf@gitster.g","subject":"Re: [PATCH] ci(dockerized): reduce the PID limit for private repositories","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2026-07-04T08:46:56Z","receivedAt":"2026-07-04T08:47:01Z","isPatch":true,"body":"Hi Junio,\n\nOn Wed, 1 Jul 2026, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\n> writes:\n> \n> > The underlying reason seems to be a substantial difference in the hosted\n> > runners that execute these Dockerized jobs: forcing the PID limit of the\n> > container to a high number lets the jobs pass, even when running the\n> > complete matrix of all 13 Dockerized jobs concurrently.\n> \n> Is the \"reduce\" in the title accurate?  The above description tells\n> me that what you did was to \"raise\" the PID limit (i.e., forcing the\n> PID limit to a high number), presumably because the default PID\n> limit is way too low for the tests to pass?\n\nYou are right; The subject is wrong. The patch raises the limit from\nthe hosted-runner default to 16384. The subject is a left-over from a\nhypothesis that my experiments refuted.\n\nv2 will retitle to \"ci(dockerized): raise the PID limit for private\nrepositories\". No code change: 16k remains the sweet spot between the\nfailures we saw at 8k and the slow runs at 64k.\n\nCiao,\nJohannes\n"},{"id":"547127","messageId":"pull.2164.v2.git.1783155124926.gitgitgadget@gmail.com","threadId":"65901","inReplyTo":"pull.2164.git.1782889484346.gitgitgadget@gmail.com","subject":"[PATCH v2] ci(dockerized): raise the PID limit for private repositories","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-07-04T08:52:04Z","receivedAt":"2026-07-04T08:52:08Z","isPatch":true,"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nEvery once in a while I need to verify that Microsoft Git's test suite\npasses for changes that are not yet meant for public consumption, and\nsince it was (made) too difficult to keep up a working Azure Pipeline\ndefinition, I have to use GitHub Actions in a private GitHub repository\nfor that purpose.\n\nIn these tests, basically all Dockerized CI jobs fail consistently. The\nsymptom is something like:\n\n  error: cannot create async thread: Resource temporarily unavailable\n\nin the middle of a test, typically in the t5xxx-t6xxx range. The first\nsuch error is immediately followed by plenty more of these errors, and\nnot a single test succeeds afterwards.\n\nAt first, I thought that maybe the massive parallelism I enjoy there is\nthe problem, and I thought that the cgroups limits might be shared\nbetween the many containers that run on essentially the same physical\nmachine. But even reducing the matrix to just a single of those\nDockerized jobs runs into the very same problems.\n\nThe underlying reason seems to be a substantial difference in the hosted\nrunners that execute these Dockerized jobs: forcing the PID limit of the\ncontainer to a high number lets the jobs pass, even when running the\ncomplete matrix of all 13 Dockerized jobs concurrently. But that's not\nthe only difference: The jobs seem to take a lot longer in these\ncontainers than, say, in the containers made available to\nhttps://github.com/git/git.\n\nWhen forcing a PID limit of 64k in that private repository, the jobs\ncompleted successfully, but they also took a lot longer, between 2x to\n2.5x longer, i.e. painfully much longer. Reducing the PID limit to 16k,\nthe CI jobs still passed, but took an equally long amount of time.\nReducing the PID limit to 8k caused the errors to reappear.\n\nHere are the numbers from three example runs, the first one forcing the\nPID and nproc limit to 65536, the second one to 16384, the third run is\nfrom the public git/git repository:\n\nJob                           | 64k     | 16k     | reference\n------------------------------|---------|---------|---------\nalmalinux-8                   | 19m 3s  | 16m 0s  | 9m 36s\ndebian-11                     | 20m 31s | 20m 3s  | 8m 5s\nfedora-breaking-changes-meson | 16m 29s | 19m 19s | 9m 40s\nlinux-asan-ubsan              | 1h 10m  | 1h 11m  | 34m 36s\nlinux-breaking-changes        | 25m 39s | 25m 58s | 13m 15s\nlinux-leaks                   | 1h 9m   | 1h 10m  | 33m 30s\nlinux-meson                   | 28m 9s  | 27m 4s  | 13m 45s\nlinux-musl-meson              | 16m 32s | 13m 39s | 8m 6s\nlinux-reftable-leaks          | 1h 13m  | 1h 13m  | 34m 34s\nlinux-reftable                | 26m 2s  | 25m 48s | 13m 31s\nlinux-sha256                  | 26m 12s | 26m 3s  | 12m 36s\nlinux-TEST-vars               | 26m 5s  | 25m 21s | 13m 25s\nlinux32                       | 21m 16s | 19m 57s | 10m 44s\n\nIt does not look as if the PID limit is the reason for the longer\nruntime, seeing as the 64k vs 16k timings deviate no more than as is\nusual with GitHub workflows. So let's go for 16k.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n    ci(dockerized): reduce the PID limit for private repositories\n    \n    I needed to craft this patch while developing fixes for vulnerabilities\n    which eventually were published as Git for Windows v2.53.0(3).\n    \n    Changes since v1:\n    \n     * Reworded the commit message's title to reflect the actual intent.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-2164%2Fdscho%2Fraise-pid-limit-in-private-repositories-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2164/dscho/raise-pid-limit-in-private-repositories-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/2164\n\nRange-diff vs v1:\n\n 1:  77aa18442c ! 1:  671d03ad33 ci(dockerized): reduce the PID limit for private repositories\n     @@ Metadata\n      Author: Johannes Schindelin <johannes.schindelin@gmx.de>\n      \n       ## Commit message ##\n     -    ci(dockerized): reduce the PID limit for private repositories\n     +    ci(dockerized): raise the PID limit for private repositories\n      \n          Every once in a while I need to verify that Microsoft Git's test suite\n          passes for changes that are not yet meant for public consumption, and\n\n\n .github/workflows/main.yml | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex cf341d74db..85cfedf5b0 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -420,7 +420,9 @@ jobs:\n       CI_JOB_IMAGE: ${{matrix.vector.image}}\n       CUSTOM_PATH: /custom\n     runs-on: ubuntu-latest\n-    container: ${{matrix.vector.image}}\n+    container:\n+      image: ${{ matrix.vector.image }}\n+      options: ${{ github.repository_visibility == 'private' && '--pids-limit 16384 --ulimit nproc=16384:16384 --ulimit nofile=32768:32768' || '' }}\n     steps:\n     - name: prepare libc6 for actions\n       if: matrix.vector.jobname == 'linux32'\n\nbase-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc\n-- \ngitgitgadget\n"},{"id":"547148","messageId":"xmqqh5medmzh.fsf@gitster.g","threadId":"65901","inReplyTo":"pull.2164.v2.git.1783155124926.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] ci(dockerized): raise the PID limit for private repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-07-05T05:28:02Z","receivedAt":"2026-07-05T05:28:04Z","isPatch":true,"body":"\"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\nwrites:\n\n> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n>\n> Every once in a while I need to verify that Microsoft Git's test suite\n> passes for changes that are not yet meant for public consumption, and\n> since it was (made) too difficult to keep up a working Azure Pipeline\n> definition, I have to use GitHub Actions in a private GitHub repository\n> for that purpose.\n\nGreat.\n\nThe updated subject no longer confuses me ;-).  Shall we\nmark the topic for 'next'?\n\nThanks.\n"}]}