Re: [RFH] Why do osx CI jobs so unreliable?
- From
Jeff King <peff@peff.net>
- Date
- Jun 21, 2026, 21:34 UTC
- Message-ID
- <20260621213407.GC2297179@coredump.intra.peff.net>
- In-Reply-To
- <CAC2Qwm+9sh=ks1fuux415JGdDJ38Jq6eZrSH7-qzQxYCoy+Aug@mail.gmail.com>
On Sat, Jun 20, 2026 at 08:33:13AM -0700, Michael Montalbo wrote:
Show 19 quoted lines
> Patrick Steinhardt <ps@pks.im> writes: > > So I strongly suspect that it most be one of the t555* tests. > > [...] > > Maybe this is something that's specific to GitHub's environment... > > I think you're right it's t5551/t5559. The runs Junio linked: > > osx-clang cancelled 360min > osx-gcc cancelled 360min > osx-reftable success 35min > osx-meson success 61min > > All four run the same t5551/t5559 under EXPENSIVE. The two that > finished differ in just two ways, which look like the levers: > osx-reftable generates the 100k-ref advertisement in ~24ms vs ~1.2s > for loose refs on macOS (so much less time mid-response), and > osx-meson runs tests at nproc while the prove jobs hardcode --jobs=10 > on a 3-core runner (over recent master/next the prove jobs hang ~40%, > meson ~10%).
If the problem is a racy deadlock, there is a reasonable chance that some jobs may simply be lucky. Even if things like packing refs help, I suspect the problem may still be lurking. Maybe I'm just a pessimist, though. ;)
Show 6 quoted lines
> When it is wedged the whole chain sits at 0% CPU. upload-pack is > blocked in write() on the ls-refs advertisement, curl blocked in > select(). So it looks like an HTTP/2 flow-control stall on the > response side. The same stall resets itself after ~60-85s on my Linux > box and on a bare-metal Mac, but not on the GitHub runner; I haven't > pinned down why yet.
We had some HTTP/2 stalls/deadlocks in the past, and they were dependent on libcurl and apache (actually h2_mod) versions. IIRC some of the non-TLS code paths for HTTP/2 were not well tested, which led to 8f2146dbf1 (t5559: make SSL/TLS the default, 2023-02-23). Of course after that commit those cleartext code paths should not be a problem, so that is probably not exactly the issue now.
But it might be worth checking the versions you're running locally versus what's in the GitHub runner.
-Peff