Re: [RFH] Why do osx CI jobs so unreliable?
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Jun 22, 2026, 09:47 UTC
- Message-ID
- <ajkEzhdqzmAePk_P@pks.im>
- In-Reply-To
- <aji9MOE-NTHKXYqn@pks.im>
On Mon, Jun 22, 2026 at 06:42:24AM +0200, Patrick Steinhardt wrote:
> On Sun, Jun 21, 2026 at 05:34:07PM -0400, Jeff King wrote: > > On Sat, Jun 20, 2026 at 08:33:13AM -0700, Michael Montalbo wrote:
[snip]
Show 34 quoted lines
> > > When it is wedged the whole chain sits at 0% CPU. upload-pack is > > > blocked in write() on the ls-refs advertisement, curl blocked in > > > select(). So it looks like an HTTP/2 flow-control stall on the > > > response side. The same stall resets itself after ~60-85s on my Linux > > > box and on a bare-metal Mac, but not on the GitHub runner; I haven't > > > pinned down why yet. > > > > We had some HTTP/2 stalls/deadlocks in the past, and they were dependent > > on libcurl and apache (actually h2_mod) versions. IIRC some of the > > non-TLS code paths for HTTP/2 were not well tested, which led to > > 8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course > > after that commit those cleartext code paths should not be a problem, so > > that is probably not exactly the issue now. > > > > But it might be worth checking the versions you're running locally > > versus what's in the GitHub runner. > > I didn't observe any similar hangs in GitLab's CI systems, so I wonder > whether this is because of different versions of curl. And indeed we use > different versions: > > - On GitHub we use 8.6.0. > > - On GitLab we use 8.7.1. > > Now this of course doesn't mean that updating the curl version is the > fix to this whole issue, as there's a ton of other factors that could > play a role in whether or not the test hangs. So while we could just > upgrade parts of the stack and cross our fingers, but that feels rather > unsatisfactory. Still, one place to start could be to update our build > images to macOS 15. > > But the big question to me is whether the hang is because of a bug in > Git with how we drive curl, a bug in curl itself, or a bug in Apache.
I noticed that a osx-clang job failed today in t5551 [1]. This time it didn't hang, but produced an actual error:
2026-06-22T09:25:45.1984230Z ++ git -C too-many-refs fetch -q --tags
2026-06-22T09:25:45.1984420Z error: RPC failed; curl 18 transfer closed with outstanding read data remaining
2026-06-22T09:25:45.1984520Z fatal: expected flush after ref listing
2026-06-22T09:25:45.1984610Z error: last command exited with $?=128
2026-06-22T09:25:45.1984660Z ++ rm -f tags
2026-06-22T09:25:45.1984710Z ++ :
2026-06-22T09:25:45.1984830Z not ok 35 - http can handle enormous ref negotiationThere was a second test failing similarly.
Patrick
[1]: https://github.com/git/git/actions/runs/27940620478/job/82672854726