git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFH] Why do osx CI jobs so unreliable?

From
Jeff King <peff@peff.net>
Date
Jun 26, 2026, 23:43 UTC
Message-ID
<20260626234312.GA3156205@coredump.intra.peff.net>
In-Reply-To
<aj5ZaZK7xylfs4Xw@pks.im>
On Fri, Jun 26, 2026 at 12:50:17PM +0200, Patrick Steinhardt wrote:
Show 6 quoted lines
> > Thanks both of you for digging into this. I'm not familiar enough with
> > Apache's code to pass confident judgement, but your findings certainly
> > convinced me that this is just an apache bug.
> 
> The bug manifests both with HTTP/1.1 and HTTP/2 though, so this wouldn't
> fully fix the flakes we see, right?
If I understand the situation correctly, there are really two problems.

First, the EXPENSIVE tests in t5551 sometimes trigger a timeout with Apache's stock settings. This presumably became a problem recently due to 7a094d68a2 (ci: run expensive tests on push builds to integration branches, 2026-05-08). The same problem exists in t5559, which just wraps t5551 but tells us to use http2.

This timeout will cause test failures in t5551, because we aren't able to complete a request we expected to. Obviously bad and annoying.

The second problem is that when Apache hits the timeout in HTTP/2 mode, it hangs forever. And then the CI job hangs for 6 hours until it's killed, which is an even more annoying failure.

So the root cause is the same (a timeout), but the effect depends on HTTP/1.1 vs HTTP/2. I was able to reproduce both cases on my local Debian unstable system by dropping the timeout as you suggested. Running t5551 with GIT_TEST_LONG yields a failure, whereas running t5559 yields a hang.

We can mitigate both cases by bumping the timeout value, since it's addressing the root cause.

There's an open question of whether this is just papering over a problem that real users might experience, and whether Git should be doing more to keep the connection alive. I think it's probably OK to ignore this in practice. This is an intentionally large request being served by a very underpowered platform. The default apache timeout is 60s. If a real-world server is seeing ls-refs requests take that long then they probably need to reconsider some other decisions, from ref packing to better hardware to dropping some users. ;) I don't think trying to insert keepalives at the Git layer here is worth the trouble.

To give a sense of the time options, here are a few timings from my local machine, timing "git upload-pack . </dev/null >/dev/null" in t5551's big repo.git (that's a v0 advertisement, but it should be roughly the same work as the v2 ls-refs).

  cold-cache, refs not packed:
  real	0m9.973s
  user	0m0.354s
  sys	0m1.364s
  warm cache, refs not packed:
  real	0m0.410s
  user	0m0.153s
  sys	0m0.257s
  cold-cache, refs packed:
  real	0m0.149s
  user	0m0.086s
  sys	0m0.035s
  warm cache, refs packed:
  real	0m0.069s
  user	0m0.054s
  sys	0m0.016s

So 10s is pretty abysmal (and on an SSD, no less). I would expect the cache to be warm (we just wrote these refs!) but I could also believe that CI systems are under heavy I/O and memory pressure, so we sometimes end up crossing the 60s mark.

So bumping Apache's timeout to 600s or something would probably be a fine mitigation. That's still not _solving_ the problem, but presumably an order of magnitude is enough for it to never come up in practice.

Michael suggested packing the refs as a mitigation. I was lukewarm on that in my previous email, because it wasn't clear to me how close we were on the timeout budget, and if it would just make the race less frequent (rather than never happen). But seeing those cold-cache numbers makes me think it might be worth doing just on principle to make the tests more efficient, and any timeout mitigation is a bonus.

Of course the pack-refs process (and the initial ref writes) will still have to touch all of those loose files, so those will still be slow. But they're not on a timeout, and I suspect we read the result many more times than we write/pack (the test failures we are seeing are not in the expensive tests, but just "normal" tests that are stuck with the gigantic ref state).

-Peff
Previous: Patrick SteinhardtNext: Junio C Hamano
Message 41 of 42 in “Re: [RFH] Why do osx CI jobs so unreliable?”
  1. Michael MontalboJun 20, 2026
  2. Jeff KingJun 21, 2026
  3. Patrick SteinhardtJun 22, 2026
  4. Patrick SteinhardtJun 22, 2026
  5. Patrick SteinhardtJun 22, 2026
  6. Patrick SteinhardtJun 22, 2026
  7. Michael MontalboJun 26, 2026
  8. Jeff KingJun 26, 2026
  9. Patrick SteinhardtJun 26, 2026
  10. Junio C HamanoJun 26, 2026
  11. Michael MontalboJun 26, 2026
  12. 0/3 fixing expensive http test timeoutsJeff King, Jun 28, 2026
  13. 1/3 t/lib-httpd: bump apache timeoutJeff King, Jun 28, 2026
  14. Michael MontalboJul 2, 2026
  15. 2/3 t5551: put many-tags case into its own repoJeff King, Jun 28, 2026
  16. Junio C HamanoJun 28, 2026
  17. Jeff KingJun 29, 2026
  18. Junio C HamanoJun 29, 2026
  19. Jeff KingJun 29, 2026
  20. 3/3 t5551: pack refs after creating many tagsJeff King, Jun 28, 2026
  21. Junio C HamanoJun 28, 2026
  22. Patrick SteinhardtJun 29, 2026
  23. Jeff KingJun 29, 2026
  24. Patrick SteinhardtJun 30, 2026
  25. Jeff KingJun 30, 2026
  26. weird quadratic reftable behavior, was: Re: [PATCH 3/3] t5551: pack refs after creating many tagsJeff King, Jun 30, 2026
  27. Patrick SteinhardtJul 1, 2026
  28. Jeff KingJul 1, 2026
  29. Kristofer KarlssonJul 1, 2026
  30. Patrick SteinhardtJul 1, 2026
  31. Kristofer KarlssonJul 3, 2026
  32. Patrick SteinhardtJul 6, 2026
  33. Kristofer KarlssonJul 6, 2026
  34. Patrick SteinhardtJun 29, 2026
  35. Junio C HamanoJun 29, 2026
  36. Patrick SteinhardtJun 29, 2026
  37. Junio C HamanoJun 29, 2026
  38. Patrick SteinhardtJun 30, 2026
  39. Junio C HamanoJun 30, 2026
  40. Patrick SteinhardtJul 1, 2026
  41. Jeff KingJun 26, 2026
  42. Junio C HamanoJun 22, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.