{"thread":{"id":"63014","subject":"[PATCH] gitlab-ci: fix \"msvc-meson\" test job succeeding despite test failures","startedAt":"2025-02-27T13:39:33Z","lastAt":"2025-02-27T13:39:33Z","messageCount":1,"participants":["Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"513146","messageId":"20250227-b4-pks-gitlab-ci-fix-msvc-meson-tests-v1-1-0420abde3807@pks.im","threadId":"63014","inReplyTo":null,"subject":"[PATCH] gitlab-ci: fix \"msvc-meson\" test job succeeding despite test failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-27T13:39:22Z","receivedAt":"2025-02-27T13:39:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have recently noticed that the \"msvc-meson\" test job in GitLab CI\nsucceeds even if there are failures. This is somewhat puzzling because\nwe use exactly the same command as we do on GitHub Actions, and there\nthe jobs fail as exected.\n\nAs it turns out, this is another weirdness of the GitLab CI hosted\nrunner for Windows [1]: by default, even successful commands will not\nmake the job fail. Interestingly though, this depends on what exactly\nthe command is that you're running -- the MinGW-based job for example\nworks alright and does fail as expected.\n\nThe root cause here seems to be specific behaviour of PowerShell. The\ninvocation of `ForEach-Object` does not bubble up any errors in case the\ninvocation of `meson test` fails, and thus we don't notice the error.\nThis is specific to executing the command in a loop: other build steps\nwhere we execute commands directly fail as expected.\n\nThis is because the specific version of PowerShell that we use in the\nrunner does not know about `PSNativeCommandUseErrorActionPreference`\nyet, which controls whether native commands like \"meson.exe\" honor the\n`ErrorActionPreference` variable. The preference has been introduced\nwith PowerShell 7.3 and is default-enabled since PowerShell 7.4, but\nGitLab's hosted runners still seem to use PowerShell 5.1. Consequently,\nwhen tests fail, we won't bubble up the error at all from the loop and\nthus the job doesn't fail. This isn't an issue in other cases though\nwhere we execute native commands directly, as the GitLab runner knows to\ncheck the last error code after every command.\n\nThe same thing doesn't seem to be an issue on GitHub Actions, most\nlikely because it uses PowerShell 7.4. Curiously, the preference for\n`PSNativeCommandUseErrorActionPreference` is disabled there, but the\njobs fail as expected regardless of that. It's puzzling, but I do not\nhave enough PowerShell expertise to give a definitive answer as to why\nit works there.\n\nIn any case, Meson 1.8 will likely get support for slicing tests [1], so\nwe can eventually get rid of the whole PowerShell script. For now, work\naround the issue by explicitly exiting out of the loop with a non-zero\nerror code if we see that Meson has failed.\n\n[1]: https://github.com/mesonbuild/meson/pull/14092\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\nHi,\n\nthis fix addresses the issue found and discussed in the thread at [1]. A\nfailing test run can be seen at [2].\n\nThanks!\n\nPatrick\n\n[1]: <xmqqh64gg0pu.fsf@gitster.g>\n[2]: https://gitlab.com/gitlab-org/git/-/jobs/9262132042\n---\n .gitlab-ci.yml | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 3f29181708f..a10bb8372d7 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -179,7 +179,7 @@ test:msvc-meson:\n     - job: \"build:msvc-meson\"\n       artifacts: true\n   script:\n-    - meson test -C build --list | Select-Object -Skip 1 | Select-String .* | Group-Object -Property { $_.LineNumber % $Env:CI_NODE_TOTAL + 1 } | Where-Object Name -EQ $Env:CI_NODE_INDEX | ForEach-Object { meson test -C build --no-rebuild --print-errorlogs $_.Group }\n+    - meson test -C build --list | Select-Object -Skip 1 | Select-String .* | Group-Object -Property { $_.LineNumber % $Env:CI_NODE_TOTAL + 1 } | Where-Object Name -EQ $Env:CI_NODE_INDEX | ForEach-Object { meson test -C build --no-rebuild --print-errorlogs $_.Group; if (!$?) { exit $LASTEXITCODE } }\n   parallel: 10\n \n test:fuzz-smoke-tests:\n\n---\nbase-commit: 5a526e5e18ddb9a7dfc5a2967d21d6154df64a4f\nchange-id: 20250226-b4-pks-gitlab-ci-fix-msvc-meson-tests-77a314732729\n\n"}]}