git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFH] Why do osx CI jobs so unreliable?

From
Michael Montalbo <mmontalbo@gmail.com>
Date
Jun 26, 2026, 03:27 UTC
Message-ID
<CAC2QwmJA2TH6BmO0O61qRYvV2pqURUk0dTXpkJtb9e-TZNZDZQ@mail.gmail.com>
In-Reply-To
<ajkOoRhqaAcy6gBg@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 5 quoted lines
> I think the issue is rather simple: we're hitting timeouts in Apache.
> [...]
> This is because our keepalive mechanisms aren't helping [...]
> Whether that's the same issue like we see in macOS sometimes is a
> different question.

I think that is the trigger for issues we've been seeing. I spent some time investigating the Apache side over the last week and maybe found a mod_http2 bug, which I filed upstream with a potential fix:

  bug:  https://bz.apache.org/bugzilla/show_bug.cgi?id=70131
  fix:  https://github.com/mmontalbo/httpd/pull/2

To Patrick's earlier question of whether this is a Git, curl, or Apache bug: as best I can tell it's Apache. I could reproduce it with no Git involved at all (just Apache and a small CGI that goes quiet past the Timeout), and across several curl versions (8.6.0, which is what the GitHub runners use, up to 8.20.0), so I don't think bumping curl would help. It also seems to wear two faces from the same trigger: over HTTP/1.1 Apache closes the connection and curl bails with the "transfer closed" error (which looks like what you hit with Timeout=1, and the recent failures on both macOS and Linux), and over HTTP/2 it does not reliably reset the stream, so the client just waits, which is the six-hour macOS hang. I share the pessimism from earlier in the thread, though: I think the real fix is upstream in Apache, and anything we do on our side mostly just bounds the symptom in the meantime.

Given there could be a potential reliability issue with an upstream dependency like Apache, I was considering what mitigation strategies might help:

  - Enforce some kind of lower bound speed limit and a client-side
    timeout so runs that wedge fail fast (and loudly) instead of
    hanging.
  - Potentially provide some affordance for retrying flaky tests
    that might fail due to upstream dependencies. Git already has
    some HTTP retry support (http.maxRetries and friends, added
    recently), but as far as I can tell it only triggers on HTTP 429
    rate limiting, so it would not catch a stall like this on its
    own. A test-level retry is not something I like that much, since
    it might encourage papering over flakiness that should be
    resolved, but it was a consideration vs requiring a fresh CI run
    to resolve the flake.
  - Make slow tests faster by optimizing the test itself and/or
    the test runner configuration (e.g., job number matching
    cores) so wedges become less likely.

For the first one, I think Git already provides some affordances. There is a stall-based timeout that just ships disabled: as I understand it http.lowSpeedLimit sets a bytes/sec floor and http.lowSpeedTime how long a transfer can sit below it before curl gives up, so it would catch a wedged connection without punishing one that is just slow. Enabling it for the http tests might look something like:

    diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
    @@ GIT_TRACE=$GIT_TRACE; export GIT_TRACE
    +# Abort a transfer that makes essentially no progress for a while,
    +# so a wedged connection fails in seconds instead of hanging to the
    +# job cap. Tiny limit, generous window, so it only trips on a true
    +# stall; override either var, or set the limit to 0, to disable.
    +GIT_HTTP_LOW_SPEED_LIMIT=${GIT_HTTP_LOW_SPEED_LIMIT-1}
    +GIT_HTTP_LOW_SPEED_TIME=${GIT_HTTP_LOW_SPEED_TIME-60}
    +export GIT_HTTP_LOW_SPEED_LIMIT GIT_HTTP_LOW_SPEED_TIME

I went conservative on the values on purpose: a floor of 1 byte/sec should only really fire on a true zero-progress stall, not on something that is just crawling on a slow runner, and the 60s window is generous for the same reason. When I tried it locally against a stall-proxy it did turn an otherwise indefinite hang into a bounded abort (a tighter limit/window brings that down to single-digit seconds). It probably does not need to be suite-wide either; it could be scoped per-command with git -c, which the http tests already lean on for this kind of thing (t5551 passes http.postbuffer and http.extraheader that way), if a narrower blast radius feels safer.

I only dug into the first option in any depth, since I wanted to sanity-check the direction before writing patches. Does turning on a stall timeout for the http tests seem reasonable? Are there other strategies that we should implement?

Previous: Patrick SteinhardtNext: Jeff King
Message 7 of 42 in “Re: [RFH] Why do osx CI jobs so unreliable?”
  1. Michael MontalboJun 20, 2026
  2. Jeff KingJun 21, 2026
  3. Patrick SteinhardtJun 22, 2026
  4. Patrick SteinhardtJun 22, 2026
  5. Patrick SteinhardtJun 22, 2026
  6. Patrick SteinhardtJun 22, 2026
  7. Michael MontalboJun 26, 2026
  8. Jeff KingJun 26, 2026
  9. Patrick SteinhardtJun 26, 2026
  10. Junio C HamanoJun 26, 2026
  11. Michael MontalboJun 26, 2026
  12. 0/3 fixing expensive http test timeoutsJeff King, Jun 28, 2026
  13. 1/3 t/lib-httpd: bump apache timeoutJeff King, Jun 28, 2026
  14. Michael MontalboJul 2, 2026
  15. 2/3 t5551: put many-tags case into its own repoJeff King, Jun 28, 2026
  16. Junio C HamanoJun 28, 2026
  17. Jeff KingJun 29, 2026
  18. Junio C HamanoJun 29, 2026
  19. Jeff KingJun 29, 2026
  20. 3/3 t5551: pack refs after creating many tagsJeff King, Jun 28, 2026
  21. Junio C HamanoJun 28, 2026
  22. Patrick SteinhardtJun 29, 2026
  23. Jeff KingJun 29, 2026
  24. Patrick SteinhardtJun 30, 2026
  25. Jeff KingJun 30, 2026
  26. weird quadratic reftable behavior, was: Re: [PATCH 3/3] t5551: pack refs after creating many tagsJeff King, Jun 30, 2026
  27. Patrick SteinhardtJul 1, 2026
  28. Jeff KingJul 1, 2026
  29. Kristofer KarlssonJul 1, 2026
  30. Patrick SteinhardtJul 1, 2026
  31. Kristofer KarlssonJul 3, 2026
  32. Patrick SteinhardtJul 6, 2026
  33. Kristofer KarlssonJul 6, 2026
  34. Patrick SteinhardtJun 29, 2026
  35. Junio C HamanoJun 29, 2026
  36. Patrick SteinhardtJun 29, 2026
  37. Junio C HamanoJun 29, 2026
  38. Patrick SteinhardtJun 30, 2026
  39. Junio C HamanoJun 30, 2026
  40. Patrick SteinhardtJul 1, 2026
  41. Jeff KingJun 26, 2026
  42. Junio C HamanoJun 22, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.