{"thread":{"id":"65114","subject":"[PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits","startedAt":"2026-03-02T11:55:08Z","lastAt":"2026-03-03T06:32:59Z","messageCount":4,"participants":["Patrick Steinhardt","Justin Tobler"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"537504","messageId":"20260302-pks-msvc-meson-xargs-v1-1-8e42abd879ce@pks.im","threadId":"65114","inReplyTo":null,"subject":"[PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-02T11:55:02Z","receivedAt":"2026-03-02T11:55:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have started to see the following assert happen in our GitLab CI\npipelines for jobs that use Windows with Meson:\n\n  assertion \"bc_ctl.arg_max >= LINE_MAX\" failed: file \"xargs.c\", line 512, function: main\n\nThe assert in question verifies that we have enough room available to\npass at least `LINE_MAX` many bytes via the command line. The xargs(1)\nbinary in those jobs comes from Git for Windows, which in turn sources\nthe binaries from MSYS2, and has the following limits in place:\n\n  $ & \"C:/Program Files/Git/usr/bin/bash.exe\" -l -c 'xargs --show-limits </dev/null'\n  Your environment variables take up 17373 bytes\n  POSIX upper limit on argument length (this system): 12579\n  POSIX smallest allowable upper limit on argument length (all systems): 4096\n  Maximum length of command we could actually use: 18446744073709546822\n  Size of command buffer we are actually using: 12579\n  Maximum parallelism (--max-procs must be no greater): 2147483647\n\nWhat's interesting to see is the limit of 16 exabits for the maximum\ncommand line length. This value might seem a bit high, and it is indeed\nthe result of an underflow: our environment is larger than the POSIX\nupper limit on argument length, and the value is computed by subtracting\nthe former from the latter. So what we get is the result of `2^64 -\n(17373 - 12579)`.\n\nThis makes it clear that the problem here is the size of our environment\nvariables. A listing sorted by length yields the following result:\n\n  $ Get-ChildItem \"Env:\" |\n      Sort-Object { $_.Value.Length } -Descending |\n      Select-Object Name, @{Name=\"Length\"; Expression={$_.Value.Length}}\n  Name                                          Length\n  ----                                          ------\n  GITLAB_FEATURES                                 6386\n  Path                                             706\n  PSModulePath                                     229\n\nThe GITLAB_FEATURES environment variable makes up for roughly a third of\nthe complete environment. This variable is a comma-separated list of\nfeatures available for the GitLab instance, and seemingly it has been\ngrowing over time as GitLab added more and more features.\n\nFix the issue by unsetting the environment variable in \"ci/lib.sh\". This\nensures that the environment variables are now smaller than the upper\nlimit on argument length again, and that in turn fixes the assert in\nxargs(1).\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\nHi,\n\nthis patch series fixes another issue that we saw creeping into GitLab\nCI jobs for MSVC+Windows. The root cause is that our environment\nvariables have grown too large, and thus xargs(1) was hitting an assert.\n\nA test run of this can be found at [1].\n\nThanks!\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/514\n---\n ci/lib.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 3ecbf147db..42a2b6a318 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -231,6 +231,10 @@ then\n \tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n elif test true = \"$GITLAB_CI\"\n then\n+\t# This environment is multiple kB in size and may cause us to exceed\n+\t# xargs(1) limits on Windows.\n+\tunset GITLAB_FEATURES\n+\n \tCI_TYPE=gitlab-ci\n \tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n \tCI_COMMIT=\"$CI_COMMIT_SHA\"\n\n---\nbase-commit: 2cc71917514657b93014134350864f4849edfc83\nchange-id: 20260302-pks-msvc-meson-xargs-c4bc35496078\n\n"},{"id":"537557","messageId":"aaXArnhYbtX9gsUU@denethor","threadId":"65114","inReplyTo":"20260302-pks-msvc-meson-xargs-v1-1-8e42abd879ce@pks.im","subject":"Re: [PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-02T17:11:52Z","receivedAt":"2026-03-02T17:11:56Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/03/02 12:55PM, Patrick Steinhardt wrote:\n> We have started to see the following assert happen in our GitLab CI\n> pipelines for jobs that use Windows with Meson:\n> \n>   assertion \"bc_ctl.arg_max >= LINE_MAX\" failed: file \"xargs.c\", line 512, function: main\n> \n> The assert in question verifies that we have enough room available to\n> pass at least `LINE_MAX` many bytes via the command line. The xargs(1)\n> binary in those jobs comes from Git for Windows, which in turn sources\n> the binaries from MSYS2, and has the following limits in place:\n> \n>   $ & \"C:/Program Files/Git/usr/bin/bash.exe\" -l -c 'xargs --show-limits </dev/null'\n>   Your environment variables take up 17373 bytes\n>   POSIX upper limit on argument length (this system): 12579\n>   POSIX smallest allowable upper limit on argument length (all systems): 4096\n>   Maximum length of command we could actually use: 18446744073709546822\n>   Size of command buffer we are actually using: 12579\n>   Maximum parallelism (--max-procs must be no greater): 2147483647\n> \n> What's interesting to see is the limit of 16 exabits for the maximum\n> command line length. This value might seem a bit high, and it is indeed\n> the result of an underflow: our environment is larger than the POSIX\n> upper limit on argument length, and the value is computed by subtracting\n> the former from the latter. So what we get is the result of `2^64 -\n> (17373 - 12579)`.\n\nOk so the problem here is that the environment variables are taking up\n17373 bytes which exceeds the upper limit here of 12579.\n\n> This makes it clear that the problem here is the size of our environment\n> variables. A listing sorted by length yields the following result:\n> \n>   $ Get-ChildItem \"Env:\" |\n>       Sort-Object { $_.Value.Length } -Descending |\n>       Select-Object Name, @{Name=\"Length\"; Expression={$_.Value.Length}}\n>   Name                                          Length\n>   ----                                          ------\n>   GITLAB_FEATURES                                 6386\n>   Path                                             706\n>   PSModulePath                                     229\n> \n> The GITLAB_FEATURES environment variable makes up for roughly a third of\n> the complete environment. This variable is a comma-separated list of\n> features available for the GitLab instance, and seemingly it has been\n> growing over time as GitLab added more and more features.\n> \n> Fix the issue by unsetting the environment variable in \"ci/lib.sh\". This\n> ensures that the environment variables are now smaller than the upper\n> limit on argument length again, and that in turn fixes the assert in\n> xargs(1).\n\nSo if we unset GITLAB_FEATURES, that puts us at 10987 bytes (17373 -\n6386) which would be under the upper limit. Unsetting this environment\nvariable seems like a reasonable means to mitigate this problem. Naive\nquestion: is the upper limit something we could increase for the\nenvironment? \n\n[snip]\n>  ci/lib.sh | 4 ++++\n>  1 file changed, 4 insertions(+)\n> \n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 3ecbf147db..42a2b6a318 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -231,6 +231,10 @@ then\n>  \tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n>  elif test true = \"$GITLAB_CI\"\n>  then\n> +\t# This environment is multiple kB in size and may cause us to exceed\n> +\t# xargs(1) limits on Windows.\n> +\tunset GITLAB_FEATURES\n\nNow when running in GitLab CI we unset the large envvar. Looks good.\n\n-Justin\n"},{"id":"537646","messageId":"aaZ8nyJFjFqct2Ri@pks.im","threadId":"65114","inReplyTo":"aaXArnhYbtX9gsUU@denethor","subject":"Re: [PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-03-03T06:15:59Z","receivedAt":"2026-03-03T06:16:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Mar 02, 2026 at 11:11:52AM -0600, Justin Tobler wrote:\n> On 26/03/02 12:55PM, Patrick Steinhardt wrote:\n[snip]\n> > The GITLAB_FEATURES environment variable makes up for roughly a third of\n> > the complete environment. This variable is a comma-separated list of\n> > features available for the GitLab instance, and seemingly it has been\n> > growing over time as GitLab added more and more features.\n> > \n> > Fix the issue by unsetting the environment variable in \"ci/lib.sh\". This\n> > ensures that the environment variables are now smaller than the upper\n> > limit on argument length again, and that in turn fixes the assert in\n> > xargs(1).\n> \n> So if we unset GITLAB_FEATURES, that puts us at 10987 bytes (17373 -\n> 6386) which would be under the upper limit. Unsetting this environment\n> variable seems like a reasonable means to mitigate this problem. Naive\n> question: is the upper limit something we could increase for the\n> environment?\n\nUnfortunately not. Under normal Linux systems you'd be able to do that,\nbut in MSYS2 the limits are hardcoded as far as I could see.\n\nThanks!\n\nPatrick\n"},{"id":"537648","messageId":"aaZ_yngsx-CXAx0p@denethor","threadId":"65114","inReplyTo":"aaZ8nyJFjFqct2Ri@pks.im","subject":"Re: [PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-03-03T06:32:56Z","receivedAt":"2026-03-03T06:32:59Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 26/03/03 07:15AM, Patrick Steinhardt wrote:\n> On Mon, Mar 02, 2026 at 11:11:52AM -0600, Justin Tobler wrote:\n> > On 26/03/02 12:55PM, Patrick Steinhardt wrote:\n> [snip]\n> > > The GITLAB_FEATURES environment variable makes up for roughly a third of\n> > > the complete environment. This variable is a comma-separated list of\n> > > features available for the GitLab instance, and seemingly it has been\n> > > growing over time as GitLab added more and more features.\n> > > \n> > > Fix the issue by unsetting the environment variable in \"ci/lib.sh\". This\n> > > ensures that the environment variables are now smaller than the upper\n> > > limit on argument length again, and that in turn fixes the assert in\n> > > xargs(1).\n> > \n> > So if we unset GITLAB_FEATURES, that puts us at 10987 bytes (17373 -\n> > 6386) which would be under the upper limit. Unsetting this environment\n> > variable seems like a reasonable means to mitigate this problem. Naive\n> > question: is the upper limit something we could increase for the\n> > environment?\n> \n> Unfortunately not. Under normal Linux systems you'd be able to do that,\n> but in MSYS2 the limits are hardcoded as far as I could see.\n\nAh ok, good to know. I was wondering if there would be value in trying\nto increase the headroom we have in case the overall size of the\nenvironment variables increases over time again, but it sounds like this\nwouldn't be possible. Hopefully we should be good though. :)\n\nThanks,\n-Justin\n"}]}