Re: [RFH] Why do osx CI jobs so unreliable?
- From
Michael Montalbo <mmontalbo@gmail.com>
- Date
- Jun 20, 2026, 15:33 UTC
- Message-ID
- <CAC2Qwm+9sh=ks1fuux415JGdDJ38Jq6eZrSH7-qzQxYCoy+Aug@mail.gmail.com>
Patrick Steinhardt <ps@pks.im> writes:
> So I strongly suspect that it most be one of the t555* tests. > [...] > Maybe this is something that's specific to GitHub's environment...
I think you're right it's t5551/t5559. The runs Junio linked:
osx-clang cancelled 360min osx-gcc cancelled 360min osx-reftable success 35min osx-meson success 61min
All four run the same t5551/t5559 under EXPENSIVE. The two that finished differ in just two ways, which look like the levers: osx-reftable generates the 100k-ref advertisement in ~24ms vs ~1.2s for loose refs on macOS (so much less time mid-response), and osx-meson runs tests at nproc while the prove jobs hardcode --jobs=10 on a 3-core runner (over recent master/next the prove jobs hang ~40%, meson ~10%).
When it is wedged the whole chain sits at 0% CPU. upload-pack is blocked in write() on the ls-refs advertisement, curl blocked in select(). So it looks like an HTTP/2 flow-control stall on the response side. The same stall resets itself after ~60-85s on my Linux box and on a bare-metal Mac, but not on the GitHub runner; I haven't pinned down why yet.
On the chance those two levers are the fix, a branch off master:
https://github.com/mmontalbo/git/tree/mm/macos-ci-hang-fix
- pack the refs in t5551's enormous-ref-negotiation test (doesn't
change what it checks on the wire, just avoids re-reading 100k loose
files to advertise them, like reftable already does)
- use the core count for $JOBS on the GitHub macOS path, matching the
GitLab branch in the same ci/lib.sh and what meson doesI ran the two macOS jobs under EXPENSIVE about eight times with these and they all finished in ~30-44min instead of hanging. Happy to send out a patch if it's helpful.