threads / discuss / 64480

CI Failures

Subject: CI Failures

## tl;dr

8 messages between Nov 14, 2025 and Nov 17, 2025.

replies: 7people: 3as markdown or json

Phillip Wood· Nov 14, 2025, 20:00 UTC · lore

I'm seeing all the Ubuntu based jobs fail. Judging by the error messages at [1] it appears that their version of sudo no longer supports "--preserve-env" which breaks all of our scripts. Does anyone know how to make sure we install the C version of sudo rather than the rust version? Failing that I guess we could switch to using "runuser"

Thanks
Phillip

[1] https://github.com/phillipwood/git/actions/runs/19371231059/job/55427466699

Phillip Wood· Nov 14, 2025, 20:11 UTC · re: Phillip Wood · lore

Re: CI Failures

On 14/11/2025 20:00, Phillip Wood wrote:
Show 5 quoted lines
> I'm seeing all the Ubuntu based jobs fail. Judging by the error messages 
> at [1] it appears that their version of sudo no longer supports "-- 
> preserve-env" which breaks all of our scripts. Does anyone know how to 
> make sure we install the C version of sudo rather than the rust version? 
> Failing that I guess we could switch to using "runuser"

Sorry for the noise, I see this was fixed last month, I had accidentally rebased onto a stale branch.

Thanks
Phillip
Show 6 quoted lines
> Thanks
> 
> Phillip
> 
> [1] https://github.com/phillipwood/git/actions/runs/19371231059/ 
> job/55427466699
Junio C Hamano· Nov 14, 2025, 20:22 UTC · re: Phillip Wood · lore

Re: CI Failures

Phillip Wood <phillip.wood123@gmail.com> writes:
Show 9 quoted lines
> On 14/11/2025 20:00, Phillip Wood wrote:
>> I'm seeing all the Ubuntu based jobs fail. Judging by the error messages 
>> at [1] it appears that their version of sudo no longer supports "-- 
>> preserve-env" which breaks all of our scripts. Does anyone know how to 
>> make sure we install the C version of sudo rather than the rust version? 
>> Failing that I guess we could switch to using "runuser"
>
> Sorry for the noise, I see this was fixed last month, I had accidentally 
> rebased onto a stale branch.

I think we've merged the fix down to the maint-2.51 track but not to maint-2.50 and below. fddb4842 (ci: fix broken jobs on Ubuntu 25.10 caused by switch to sudo-rs(1), 2025-10-10) cannot be merged below there, but you should be able to cherry-pick if needed.

Junio C Hamano· Nov 14, 2025, 20:34 UTC · re: Junio C Hamano · lore

CI Failures (osx with p4)

Speaking of CI failures, I think we are seeing unusually high rate of failures from random osx jobs failing to successfully run t98xx series. Is p4 on macOSX still a thing? As macOSX tests tend to take very long to complete compared to other jobs, I am tempted to suggest dropping p4 tests from there.

Opinions?
----- >8 -----
Subject: [PATCH] CI: drop Perforce tests from macOSX jobs

It seems that we seem to be getting unusually higher rate of failures from random macOSX jobs failing to run t98xx series, even though we haven't changed Perforce interaction part of our codebase for quite some time, and the failures seem to be limited to macOSX jobs and no other platforms.

Most importantly, nobody from the macOSX community seems to have raised an issue here, reporting any failures.

Perhaps we should stop testing the combination of P4 and macOSX to save cycles, electricity, and my sanity ;-)

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 ci/install-dependencies.sh | 4 ----
 1 file changed, 4 deletions(-)
diff --git c/ci/install-dependencies.sh w/ci/install-dependencies.sh
index 50628ee2dd..3f718dad41 100755
--- c/ci/install-dependencies.sh
+++ w/ci/install-dependencies.sh
@@ -109,10 +109,6 @@ macos-*)
 	brew link --force gettext
 
 	mkdir -p "$CUSTOM_PATH"
-	wget -q "$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz" &&
-	tar -xf helix-core-server.tgz -C "$CUSTOM_PATH" p4 p4d &&
-	sudo xattr -d com.apple.quarantine "$CUSTOM_PATH/p4" "$CUSTOM_PATH/p4d" 2>/dev/null || true
-	rm helix-core-server.tgz
 
 	case "$jobname" in
 	osx-meson)
Koji Nakamaru· Nov 16, 2025, 15:23 UTC · re: Junio C Hamano · lore

Re: CI Failures (osx with p4)

On Sat, Nov 15, 2025 at 5:34 AM Junio C Hamano <gitster@pobox.com> wrote:
Show 45 quoted lines
>
> Speaking of CI failures, I think we are seeing unusually high rate
> of failures from random osx jobs failing to successfully run t98xx
> series.  Is p4 on macOSX still a thing?  As macOSX tests tend to
> take very long to complete compared to other jobs, I am tempted to
> suggest dropping p4 tests from there.
>
> Opinions?
>
> ----- >8 -----
> Subject: [PATCH] CI: drop Perforce tests from macOSX jobs
>
> It seems that we seem to be getting unusually higher rate of
> failures from random macOSX jobs failing to run t98xx series, even
> though we haven't changed Perforce interaction part of our codebase
> for quite some time, and the failures seem to be limited to macOSX
> jobs and no other platforms.
>
> Most importantly, nobody from the macOSX community seems to have
> raised an issue here, reporting any failures.
>
> Perhaps we should stop testing the combination of P4 and macOSX to
> save cycles, electricity, and my sanity ;-)
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>  ci/install-dependencies.sh | 4 ----
>  1 file changed, 4 deletions(-)
>
> diff --git c/ci/install-dependencies.sh w/ci/install-dependencies.sh
> index 50628ee2dd..3f718dad41 100755
> --- c/ci/install-dependencies.sh
> +++ w/ci/install-dependencies.sh
> @@ -109,10 +109,6 @@ macos-*)
>         brew link --force gettext
>
>         mkdir -p "$CUSTOM_PATH"
> -       wget -q "$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz" &&
> -       tar -xf helix-core-server.tgz -C "$CUSTOM_PATH" p4 p4d &&
> -       sudo xattr -d com.apple.quarantine "$CUSTOM_PATH/p4" "$CUSTOM_PATH/p4d" 2>/dev/null || true
> -       rm helix-core-server.tgz
>
>         case "$jobname" in
>         osx-meson)
>

The cause might be that the macos-14 and later images are for arm64 [1], but install-dependencies.sh still downloads the x86_64 Perforce commands. This leads to Rosetta translation overhead. The arm64 Perforce commands can be downloaded with the following modification. I confirmed that this modified version runs without any error [2].

[1]: https://github.com/actions/runner-images?tab=readme-ov-file#available-images [2]: https://github.com/KojiNakamaru/git/actions/runs/19407172619

--
diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
index 50628ee2dd..16ee593d30 100755
--- a/ci/install-dependencies.sh
+++ b/ci/install-dependencies.sh
@@ -7,7 +7,7 @@

 begin_group "Install dependencies"

-P4WHENCE=https://cdist2.perforce.com/perforce/r23.2
+P4WHENCE=https://cdist2.perforce.com/perforce/r24.2
 LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION
 JGITWHENCE=https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh
 CARGO_MSRV_VERSION=0.18.4
@@ -109,7 +109,7 @@ macos-*)
  brew link --force gettext

  mkdir -p "$CUSTOM_PATH"
- wget -q "$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz" &&
+ wget -q "$P4WHENCE/bin.macosx12arm64/helix-core-server.tgz" &&
  tar -xf helix-core-server.tgz -C "$CUSTOM_PATH" p4 p4d &&
  sudo xattr -d com.apple.quarantine "$CUSTOM_PATH/p4"
"$CUSTOM_PATH/p4d" 2>/dev/null || true
  rm helix-core-server.tgz
Junio C Hamano· Nov 16, 2025, 17:18 UTC · re: Koji Nakamaru · lore

Re: CI Failures (osx with p4)

Koji Nakamaru <koji.nakamaru@gree.net> writes:
Show 8 quoted lines
> The cause might be that the macos-14 and later images are for arm64 [1],
> but install-dependencies.sh still downloads the x86_64 Perforce
> commands. This leads to Rosetta translation overhead. The arm64 Perforce
> commands can be downloaded with the following modification. I confirmed
> that this modified version runs without any error [2].
>
> [1]: https://github.com/actions/runner-images?tab=readme-ov-file#available-images
> [2]: https://github.com/KojiNakamaru/git/actions/runs/19407172619

Ah, sounds great. Or switch to macos-14-large, which seems to be still x86-64? I have no strong preference, but given that newer units are all running on Apple silicon, perhaps testing with them, instead of what was discontinued a few years ago, may probably be still a better choice.

Show 11 quoted lines
> --
> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
> index 50628ee2dd..16ee593d30 100755
> --- a/ci/install-dependencies.sh
> +++ b/ci/install-dependencies.sh
> @@ -7,7 +7,7 @@
>
>  begin_group "Install dependencies"
>
> -P4WHENCE=https://cdist2.perforce.com/perforce/r23.2
> +P4WHENCE=https://cdist2.perforce.com/perforce/r24.2

Just being curious, other than "because it is newer than r23.2", was there a particular reason why r24.2 was picked? It is not the latest that has arm64 macos (r25.1 also has bin.macosx12arm64 subdirectory at the above distribution site), and r23.2 also has bin.macosx12arm64 subdirectory, too, which makes the above update curious if it comes without explanation, given that this would also affect Ubuntu that slurps bin.linux26x86_64 from there.

Show 9 quoted lines
>  LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION
>  JGITWHENCE=https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh
>  CARGO_MSRV_VERSION=0.18.4
> @@ -109,7 +109,7 @@ macos-*)
>   brew link --force gettext
>
>   mkdir -p "$CUSTOM_PATH"
> - wget -q "$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz" &&
> + wget -q "$P4WHENCE/bin.macosx12arm64/helix-core-server.tgz" &&

You explained this change very well. I didn't consider that the rosetta-2 may get in the way.

>   tar -xf helix-core-server.tgz -C "$CUSTOM_PATH" p4 p4d &&
>   sudo xattr -d com.apple.quarantine "$CUSTOM_PATH/p4"
> "$CUSTOM_PATH/p4d" 2>/dev/null || true
>   rm helix-core-server.tgz
Thanks.
Koji Nakamaru· Nov 16, 2025, 23:25 UTC · re: Junio C Hamano · lore

Re: CI Failures (osx with p4)

On Mon, Nov 17, 2025 at 2:18 AM Junio C Hamano <gitster@pobox.com> wrote:
Show 19 quoted lines
> > --
> > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
> > index 50628ee2dd..16ee593d30 100755
> > --- a/ci/install-dependencies.sh
> > +++ b/ci/install-dependencies.sh
> > @@ -7,7 +7,7 @@
> >
> >  begin_group "Install dependencies"
> >
> > -P4WHENCE=https://cdist2.perforce.com/perforce/r23.2
> > +P4WHENCE=https://cdist2.perforce.com/perforce/r24.2
>
> Just being curious, other than "because it is newer than r23.2", was
> there a particular reason why r24.2 was picked?  It is not the
> latest that has arm64 macos (r25.1 also has bin.macosx12arm64
> subdirectory at the above distribution site), and r23.2 also has
> bin.macosx12arm64 subdirectory, too, which makes the above update
> curious if it comes without explanation, given that this would also
> affect Ubuntu that slurps bin.linux26x86_64 from there.

When I first looked in r23.2 in the browser, somehow I could not find bin.macosx12arm64, so I looked for another version that included it. It now exists in r23.2 as you mentioned, and I totally agree that it is better to keep using r23.2. I confirmed r23.2 runs without any error [1].

[1]: https://github.com/KojiNakamaru/git/actions/runs/19413263298
Junio C Hamano· Nov 17, 2025, 00:25 UTC · re: Koji Nakamaru · lore

Re: CI Failures (osx with p4)

Koji Nakamaru <koji.nakamaru@gree.net> writes:
Show 28 quoted lines
> On Mon, Nov 17, 2025 at 2:18 AM Junio C Hamano <gitster@pobox.com> wrote:
>> > --
>> > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
>> > index 50628ee2dd..16ee593d30 100755
>> > --- a/ci/install-dependencies.sh
>> > +++ b/ci/install-dependencies.sh
>> > @@ -7,7 +7,7 @@
>> >
>> >  begin_group "Install dependencies"
>> >
>> > -P4WHENCE=https://cdist2.perforce.com/perforce/r23.2
>> > +P4WHENCE=https://cdist2.perforce.com/perforce/r24.2
>>
>> Just being curious, other than "because it is newer than r23.2", was
>> there a particular reason why r24.2 was picked?  It is not the
>> latest that has arm64 macos (r25.1 also has bin.macosx12arm64
>> subdirectory at the above distribution site), and r23.2 also has
>> bin.macosx12arm64 subdirectory, too, which makes the above update
>> curious if it comes without explanation, given that this would also
>> affect Ubuntu that slurps bin.linux26x86_64 from there.
>
> When I first looked in r23.2 in the browser, somehow I could not find
> bin.macosx12arm64, so I looked for another version that included
> it. It now exists in r23.2 as you mentioned, and I totally agree that
> it is better to keep using r23.2. I confirmed r23.2 runs without any
> error [1].
>
> [1]: https://github.com/KojiNakamaru/git/actions/runs/19413263298
Thanks!

← back to recent threads