threads / patch / 64239

patch, 5 partsgitlab-ci: some fixes for failures on Windows

Subject: [PATCH 0/5] gitlab-ci: some fixes for failures on Windows

## tl;dr

10 messages between Oct 2, 2025 and Oct 6, 2025. Diffs are folded; open one to read it.

replies: 9people: 3as markdown or json

Patrick Steinhardt· Oct 2, 2025, 11:04 UTC · lore
Hi,

this small patch series fixes some intermittent failures we have observed with our Windows CI runners. Furthermore, it fixes one persistent test failure in one of the Windows runners that was introduced recently via tests for git-last-modified(1).

The pipeline at [1] shows that tests succeed now.
Thanks!
Patrick
[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/426
---
Patrick Steinhardt (5):
      gitlab-ci: dedup instructions to disable realtime monitoring
      gitlab-ci: ignore failures to disable realtime monitoring
      gitlab-ci: drop workaround for Python certificate store on Windows
      gitlab-ci: upload Meson test logs as JUnit reports
      t8020: fix test failure due to indeterministic tag sorting
 .gitlab-ci.yml           | 29 +++++++++++++++++------------
 t/t8020-last-modified.sh | 34 +++++++++++++++++++---------------
 2 files changed, 36 insertions(+), 27 deletions(-)

--- base-commit: 821f583da6d30a84249f75f33501504d597bc16b change-id: 20251001-pks-gitlab-ci-windows-improvements-16dd3f911688

Patrick Steinhardt· Oct 2, 2025, 11:04 UTC · re: Patrick Steinhardt · lore

[PATCH 1/5] gitlab-ci: dedup instructions to disable realtime monitoring

The instruction to disable realtime monitoring are shared across all of our Windows-based jobs. Deduplicate it so that we can more readily iterate on it.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
Show changes to .gitlab-ci.yml +6 −3
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index cf122e706f..552c033fb0 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -112,6 +112,9 @@ test:osx:
       - t/failed-test-artifacts
     when: on_failure
 
+.windows_before_script: &windows_before_script
+  - Set-MpPreference -DisableRealtimeMonitoring $true
+
 build:mingw64:
   stage: build
   tags:
@@ -119,7 +122,7 @@ build:mingw64:
   variables:
     NO_PERL: 1
   before_script:
-    - Set-MpPreference -DisableRealtimeMonitoring $true
+    - *windows_before_script
     - ./ci/install-sdk.ps1 -directory "git-sdk"
   script:
     - git-sdk/usr/bin/bash.exe -l -c 'ci/make-test-artifacts.sh artifacts'
@@ -136,7 +139,7 @@ test:mingw64:
     - job: "build:mingw64"
       artifacts: true
   before_script:
-    - Set-MpPreference -DisableRealtimeMonitoring $true
+    - *windows_before_script
     - git-sdk/usr/bin/bash.exe -l -c 'tar xf artifacts/artifacts.tar.gz'
     - New-Item -Path .git/info -ItemType Directory
     - New-Item .git/info/exclude -ItemType File -Value "/git-sdk"
@@ -150,7 +153,7 @@ test:mingw64:
   tags:
     - saas-windows-medium-amd64
   before_script:
-    - Set-MpPreference -DisableRealtimeMonitoring $true
+    - *windows_before_script
     - choco install -y git meson ninja openssl
     - Import-Module $env:ChocolateyInstall\helpers\chocolateyProfile.psm1
     - refreshenv
-- 
2.51.0.700.g236ee7b076.dirty
Patrick Steinhardt· Oct 2, 2025, 11:04 UTC · re: Patrick Steinhardt · lore

[PATCH 2/5] gitlab-ci: ignore failures to disable realtime monitoring

We have recently introduced a change to disable realtime monitoring for Windows job in GitLab CI. This change led (and still leads) to a quite significant speedup.

But there's a catch: seemingly, some of the runners we use already have realtime monitoring disabled. On such a machine, trying to disable the feature again leads to an error that causes the whole job to fail.

Safeguard against such failures by explicitly ignoring them.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Show changes to .gitlab-ci.yml +4 −1
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 552c033fb0..ed4dc9db94 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -113,7 +113,10 @@ test:osx:
     when: on_failure
 
 .windows_before_script: &windows_before_script
-  - Set-MpPreference -DisableRealtimeMonitoring $true
+  # Disabling realtime monitoring fails on some of the runners, but it
+  # significantly speeds up test execution in the case where it works. We thus
+  # try our luck, but ignore any failures.
+  - Set-MpPreference -DisableRealtimeMonitoring $true; $true
 
 build:mingw64:
   stage: build
-- 
2.51.0.700.g236ee7b076.dirty
Patrick Steinhardt· Oct 2, 2025, 11:04 UTC · re: Patrick Steinhardt · lore

[PATCH 3/5] gitlab-ci: drop workaround for Python certificate store on Windows

On Windows, we have been running into some issues in the past where the certificate store for Python is broken on the GitLab CI runners using Windows. The consequence was that we weren't able to establish any SSL connections via Python, but we need that feature so that we can download the Meson wraps. The workaround we employed was to import certificates from the cURL project into the certificate store via OpenSSL.

This is obviously an ugly workaround. But even more importantly, this workaround fails every time Chocolatey updates its OpenSSL packages. The problem here is that the old OpenSSL package installer will be removed immediately once the newer version was published, But the Chocolatey community repository may not yet have propagated the new version of this package to all of its caches. The result is that for a couple hours (or sometimes even one or two days) we always fail to install OpenSSL until the new version was propagated.

Luckily though, it turns out that the workaround doesn't seem to be required anymore. Drop it to work around the intermittent failures and to clean up some now-unneeded legacy cruft.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)
Show changes to .gitlab-ci.yml +1 −9
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ed4dc9db94..b388154078 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -157,17 +157,9 @@ test:mingw64:
     - saas-windows-medium-amd64
   before_script:
     - *windows_before_script
-    - choco install -y git meson ninja openssl
+    - choco install -y git meson ninja
     - Import-Module $env:ChocolateyInstall\helpers\chocolateyProfile.psm1
     - refreshenv
-    # The certificate store for Python on Windows is broken and fails to fetch
-    # certificates, see https://bugs.python.org/issue36011. This seems to
-    # mostly be an issue with how the GitLab image is set up as it is a
-    # non-issue on GitHub Actions. Work around the issue by importing
-    # cetrificates manually.
-    - Invoke-WebRequest https://curl.haxx.se/ca/cacert.pem -OutFile cacert.pem
-    - openssl pkcs12 -export -nokeys -in cacert.pem -out certs.pfx -passout "pass:"
-    - Import-PfxCertificate -CertStoreLocation Cert:\LocalMachine\Root -FilePath certs.pfx
 
 build:msvc-meson:
   extends: .msvc-meson
-- 
2.51.0.700.g236ee7b076.dirty
Toon Claes· Oct 6, 2025, 10:19 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 3/5] gitlab-ci: drop workaround for Python certificate store on Windows

Patrick Steinhardt <ps@pks.im> writes:
Show 19 quoted lines
> On Windows, we have been running into some issues in the past where the
> certificate store for Python is broken on the GitLab CI runners using
> Windows. The consequence was that we weren't able to establish any SSL
> connections via Python, but we need that feature so that we can download
> the Meson wraps. The workaround we employed was to import certificates
> from the cURL project into the certificate store via OpenSSL.
>
> This is obviously an ugly workaround. But even more importantly, this
> workaround fails every time Chocolatey updates its OpenSSL packages. The
> problem here is that the old OpenSSL package installer will be removed
> immediately once the newer version was published, But the Chocolatey
> community repository may not yet have propagated the new version of this
> package to all of its caches. The result is that for a couple hours (or
> sometimes even one or two days) we always fail to install OpenSSL until
> the new version was propagated.
>
> Luckily though, it turns out that the workaround doesn't seem to be
> required anymore. Drop it to work around the intermittent failures and
> to clean up some now-unneeded legacy cruft.

I assume this is a result of this change: https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/-/merge_requests/72

Makes sense to clean this up.
-- 
Cheers,
Toon
Patrick Steinhardt· Oct 2, 2025, 11:04 UTC · re: Patrick Steinhardt · lore

[PATCH 4/5] gitlab-ci: upload Meson test logs as JUnit reports

When running tests, Meson knows to output both a test log as well as a JUnit test report that collates results. We don't currently upload these results in our GitLab CI at all, which makes it hard to see which tests ran, but also which of our tests may have failed.

Upload these JUnit reports as artifacts to make this information more accessible. Note that we also do this for some jobs that don't use Meson and thus don't generate these reports in the first place. GitLab CI handles missing reports gracefully though, so there is no reason to special-case those jobs that don't use Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 7 +++++++
 1 file changed, 7 insertions(+)
Show changes to .gitlab-ci.yml +7 −0
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b388154078..85401b34a5 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -70,6 +70,8 @@ test:linux:
   artifacts:
     paths:
       - t/failed-test-artifacts
+    reports:
+      junit: build/meson-logs/testlog.junit.xml
     when: on_failure
 
 test:osx:
@@ -110,6 +112,8 @@ test:osx:
   artifacts:
     paths:
       - t/failed-test-artifacts
+    reports:
+      junit: build/meson-logs/testlog.junit.xml
     when: on_failure
 
 .windows_before_script: &windows_before_script
@@ -181,6 +185,9 @@ test:msvc-meson:
   script:
     - meson test -C build --no-rebuild --print-errorlogs --slice $Env:CI_NODE_INDEX/$Env:CI_NODE_TOTAL
   parallel: 10
+  artifacts:
+    reports:
+      junit: build/meson-logs/testlog.junit.xml
 
 test:fuzz-smoke-tests:
   image: ubuntu:latest
-- 
2.51.0.700.g236ee7b076.dirty
Patrick Steinhardt· Oct 2, 2025, 11:04 UTC · re: Patrick Steinhardt · lore

[PATCH 5/5] t8020: fix test failure due to indeterministic tag sorting

In e6c06e87a2 (last-modified: fix bug when some paths remain unhandled, 2025-09-18), we have fixed a bug where under certain circumstances, git-last-modified(1) would BUG because there's still some unhandled paths. The fix claims that the root cause here is criss-cross merges, and it adds a test case that seemingly exercises this.

Curiously, this test case fails on some systems because the actual output does not match our expectations:

    diff --git a/expect b/actual
    index 5271607..bdc620e 100644
    --- a/expect
    --- b/actual
    @@ -1,3 +1,3 @@
     km3 a
    -k2 k
    +km2 k
     1 file
    error: last command exited with $?=1
    not ok 15 - last-modified with subdir and criss-cross merge

The output we see is git-name-rev(1) with `--annotate-stdin`. What it does is to take the output of git-last-modified(1), which contains object IDs of the blamed commits, and convert those object IDs into the names of the corresponding tags. Interestingly, we indeed have both "k2" and "km2" as tags, and even more interestingly both of these tags point to the same commit. So the output we get isn't _wrong_, as the tags are ambiguous.

But why do both of these tags point to the same commit? "km2" really is supposed to be a merge, but due to the way the test is constructed the merge turns into a fast-forward merge. Which means that the resulting does not even contain a criss-cross merge in the first place! A quick test though shows that the test indeed triggers the bug, so the initial analysis that the behaviour is triggered by such merges must be wrong.

And it is: seemingly, the issue isn't with criss-cross merges, but rather with a graph where different files in the same directory were modified on both sides of a merge.

Refactor the test so that we explicitly test for this specific situation instead of mentioning the "criss-cross merge" red herring. As the test is very specific to the actual layout of the repository we also adapt it to use its own standalone repository.

Note that this requires us to drop the `test_when_finished` call in `check_last_modified` because it's not supported to execute that function in a subshell.

This refactoring also fixes the original tag ambiguity that caused us to fail on some platforms.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t8020-last-modified.sh | 34 +++++++++++++++++++---------------
 1 file changed, 19 insertions(+), 15 deletions(-)
Show changes to t/t8020-last-modified.sh +19 −15
diff --git a/t/t8020-last-modified.sh b/t/t8020-last-modified.sh
index e13aad1439..61f00bc15c 100755
--- a/t/t8020-last-modified.sh
+++ b/t/t8020-last-modified.sh
@@ -33,7 +33,6 @@ check_last_modified() {
 	done &&
 
 	cat >expect &&
-	test_when_finished "rm -f tmp.*" &&
 	git ${indir:+-C "$indir"} last-modified "$@" >tmp.1 &&
 	git name-rev --annotate-stdin --name-only --tags \
 		<tmp.1 >tmp.2 &&
@@ -128,20 +127,25 @@ test_expect_success 'only last-modified files in the current tree' '
 	EOF
 '
 
-test_expect_success 'last-modified with subdir and criss-cross merge' '
-	git checkout -b branch-k1 1 &&
-	mkdir -p a k &&
-	test_commit k1 a/file2 &&
-	git checkout -b branch-k2 &&
-	test_commit k2 k/file2 &&
-	git checkout branch-k1 &&
-	test_merge km2 branch-k2 &&
-	test_merge km3 3 &&
-	check_last_modified <<-\EOF
-	km3 a
-	k2 k
-	1 file
-	EOF
+test_expect_success 'subdirectory modified via merge' '
+	test_when_finished rm -rf repo &&
+	git init repo &&
+	(
+		cd repo &&
+		test_commit base &&
+		git switch --create left &&
+		mkdir subdir &&
+		test_commit left subdir/left &&
+		git switch --create right base &&
+		mkdir subdir &&
+		test_commit right subdir/right &&
+		git switch - &&
+		test_merge merge right &&
+		check_last_modified <<-\EOF
+		merge subdir
+		base base.t
+		EOF
+	)
 '
 
 test_expect_success 'cross merge boundaries in blaming' '
-- 
2.51.0.700.g236ee7b076.dirty
Kristoffer Haugsbakk· Oct 2, 2025, 11:10 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 5/5] t8020: fix test failure due to indeterministic tag sorting

On Thu, Oct 2, 2025, at 13:04, Patrick Steinhardt wrote:
Show 6 quoted lines
> [snip]
>
> But why do both of these tags point to the same commit? "km2" really is
> supposed to be a merge, but due to the way the test is constructed the
> merge turns into a fast-forward merge. Which means that the resulting
> does not even contain a criss-cross merge in the first place! A quick
“the resulting <noun>”

By the way these intro (paragraph) sentences like “But why do” and “And it is” really help connect the whole narrative in the message. I’ve noticed the same thing on some other recent commits/patches.

> test though shows that the test indeed triggers the bug, so the initial
> analysis that the behaviour is triggered by such merges must be wrong.
>
>[snip]
Patrick Steinhardt· Oct 2, 2025, 11:24 UTC · re: Kristoffer Haugsbakk · lore

Re: [PATCH 5/5] t8020: fix test failure due to indeterministic tag sorting

On Thu, Oct 02, 2025 at 01:10:53PM +0200, Kristoffer Haugsbakk wrote:
Show 9 quoted lines
> On Thu, Oct 2, 2025, at 13:04, Patrick Steinhardt wrote:
> > [snip]
> >
> > But why do both of these tags point to the same commit? "km2" really is
> > supposed to be a merge, but due to the way the test is constructed the
> > merge turns into a fast-forward merge. Which means that the resulting
> > does not even contain a criss-cross merge in the first place! A quick
> 
> “the resulting <noun>”

Ah, right. What's missing here is "commit graph". Fixed locally, will send out with the next iteration.

Thanks!
Patrick
Toon Claes· Oct 6, 2025, 10:20 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 5/5] t8020: fix test failure due to indeterministic tag sorting

Patrick Steinhardt <ps@pks.im> writes:
Show 28 quoted lines
> In e6c06e87a2 (last-modified: fix bug when some paths remain unhandled,
> 2025-09-18), we have fixed a bug where under certain circumstances,
> git-last-modified(1) would BUG because there's still some unhandled
> paths. The fix claims that the root cause here is criss-cross merges,
> and it adds a test case that seemingly exercises this.
>
> Curiously, this test case fails on some systems because the actual
> output does not match our expectations:
>
>     diff --git a/expect b/actual
>     index 5271607..bdc620e 100644
>     --- a/expect
>     --- b/actual
>     @@ -1,3 +1,3 @@
>      km3 a
>     -k2 k
>     +km2 k
>      1 file
>     error: last command exited with $?=1
>     not ok 15 - last-modified with subdir and criss-cross merge
>
> The output we see is git-name-rev(1) with `--annotate-stdin`. What it
> does is to take the output of git-last-modified(1), which contains
> object IDs of the blamed commits, and convert those object IDs into the
> names of the corresponding tags. Interestingly, we indeed have both "k2"
> and "km2" as tags, and even more interestingly both of these tags point
> to the same commit. So the output we get isn't _wrong_, as the tags are
> ambiguous.
Ahha, I see...
> But why do both of these tags point to the same commit? "km2" really is
> supposed to be a merge, but due to the way the test is constructed the
> merge turns into a fast-forward merge.
Ohw :o
> Which means that the resulting
> does not even contain a criss-cross merge in the first place! A quick
> test though shows that the test indeed triggers the bug, so the initial
> analysis that the behaviour is triggered by such merges must be wrong.

I've tried various things, this felt like the simplest case to reproduce. Apparantly not realizing it was wrong.

> And it is: seemingly, the issue isn't with criss-cross merges, but
> rather with a graph where different files in the same directory were
> modified on both sides of a merge.

Well yes, it felt like it was something like this. But like said, it _seemed_ I needed a criss-cross merge.

> Refactor the test so that we explicitly test for this specific situation
> instead of mentioning the "criss-cross merge" red herring.
Well, thanks for this cleanup!
> As the test
> is very specific to the actual layout of the repository we also adapt it
> to use its own standalone repository.

I should have done that in the first place. Trying to shoe horn this test in the existing repo might have guided me to take incorrect conclusions.

> Note that this requires us to drop the `test_when_finished` call in
> `check_last_modified` because it's not supported to execute that
> function in a subshell.

This is not an issue because these files are always overwritten before comparing, so this the `rm` wasn't strictly needed.

> This refactoring also fixes the original tag ambiguity that caused us to
> fail on some platforms.
Awesome! 
Show 55 quoted lines
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/t8020-last-modified.sh | 34 +++++++++++++++++++---------------
>  1 file changed, 19 insertions(+), 15 deletions(-)
>
> diff --git a/t/t8020-last-modified.sh b/t/t8020-last-modified.sh
> index e13aad1439..61f00bc15c 100755
> --- a/t/t8020-last-modified.sh
> +++ b/t/t8020-last-modified.sh
> @@ -33,7 +33,6 @@ check_last_modified() {
>  	done &&
>  
>  	cat >expect &&
> -	test_when_finished "rm -f tmp.*" &&
>  	git ${indir:+-C "$indir"} last-modified "$@" >tmp.1 &&
>  	git name-rev --annotate-stdin --name-only --tags \
>  		<tmp.1 >tmp.2 &&
> @@ -128,20 +127,25 @@ test_expect_success 'only last-modified files in the current tree' '
>  	EOF
>  '
>  
> -test_expect_success 'last-modified with subdir and criss-cross merge' '
> -	git checkout -b branch-k1 1 &&
> -	mkdir -p a k &&
> -	test_commit k1 a/file2 &&
> -	git checkout -b branch-k2 &&
> -	test_commit k2 k/file2 &&
> -	git checkout branch-k1 &&
> -	test_merge km2 branch-k2 &&
> -	test_merge km3 3 &&
> -	check_last_modified <<-\EOF
> -	km3 a
> -	k2 k
> -	1 file
> -	EOF
> +test_expect_success 'subdirectory modified via merge' '
> +	test_when_finished rm -rf repo &&
> +	git init repo &&
> +	(
> +		cd repo &&
> +		test_commit base &&
> +		git switch --create left &&
> +		mkdir subdir &&
> +		test_commit left subdir/left &&
> +		git switch --create right base &&
> +		mkdir subdir &&
> +		test_commit right subdir/right &&
> +		git switch - &&
> +		test_merge merge right &&
> +		check_last_modified <<-\EOF
> +		merge subdir
> +		base base.t
> +		EOF
> +	)
>  '
Looks good to me.
-- 
Cheers,
Toon

← back to recent threads