git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 1/3] t/lib-httpd: bump apache timeout

From
Jeff King <peff@peff.net>
Date
Jun 28, 2026, 08:00 UTC
Message-ID
<20260628080009.GA107826@coredump.intra.peff.net>
In-Reply-To
<20260628075716.GA3525066@coredump.intra.peff.net>

Since enabling more tests with 7a094d68a2 (ci: run expensive tests on push builds to integration branches, 2026-05-08), we sometimes see test failures or timeouts in GitHub CI. The culprit seems to be the "enormous ref negotiation" test in t5551, which creates ~100k tag refs in our http server-side repo.

Iterating through the loose refs of this repo to generate a ref advertisement can take a long time, especially on a platform with slow I/O. On my otherwise unloaded local machine, a cold cache ref advertisement takes ~10s. On a busy CI machine running tests in parallel, it can presumably top 60s, which runs afoul of Apache's default CGI timeout.

The result in t5551 is a test failure, where Apache simply hangs up the connection and the client reports an error. But worse, t5559 runs the same test with HTTP/2, and a bug in Apache causes the connection to hang indefinitely! We eventually see this as a CI timeout after 6 hours.

Let's bump Apache's timeout to something much larger: 600 seconds. This doesn't eliminate the possibility of a timeout, but it makes it much less likely. It should eliminate both the test failures and the CI timeouts in practice, and it protects us from running into similar problems with other tests in the future.

There are two counter-arguments to consider.

One, could/should we just make the test faster? Probably yes. The biggest mistake here is having such an absurd number of unpacked refs on a system which is bottle-necked on I/O. But I think it's worth bumping the timeout so that we can fix this (and possibly other) correctness issues, and then consider performance separately (which we'll do in subsequent patches).

And two, is this just papering over a problem that users might see in the real world? We could teach Git to handle this case more gracefully with optimizations or keep-alives. But I think it's really an artificial situation. You need a combination of this silly number of loose refs, plus a very heavily loaded system. If you were trying to run a real server and it took more than 60s to generate the ref advertisement, I don't think the timeout is your biggest problem. Your crappy service is, and you should adjust your resources to match your load. I.e., it is probably reasonable for Git to assume that advertisements happen fast-ish and don't need protocol-level keepalives.

Though the patch here is small, tons of work went into analyzing the problem. Many thanks to the contributors credited below.

Helped-by: Michael Montalbo <mmontalbo@gmail.com>
Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Jeff King <peff@peff.net>
---
I didn't reference Michael's bugzilla report directly, because you can't
read it without a login. :(
Maybe it's worth doing anyway?
 t/lib-httpd/apache.conf | 1 +
 1 file changed, 1 insertion(+)
diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index 40a690b0bb..4149fc1078 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf
@@ -4,6 +4,7 @@ DocumentRoot www
 LogFormat "%h %l %u %t \"%r\" %>s %b" common
 CustomLog access.log common
 ErrorLog error.log
+Timeout 600
 <IfModule !mod_log_config.c>
 	LoadModule log_config_module modules/mod_log_config.so
 </IfModule>
-- 
2.55.0.rc2.353.gf769b6597e
Previous: Jeff KingNext: Michael Montalbo
Message 13 of 42 in “Re: [RFH] Why do osx CI jobs so unreliable?”
  1. Michael MontalboJun 20, 2026
  2. Jeff KingJun 21, 2026
  3. Patrick SteinhardtJun 22, 2026
  4. Patrick SteinhardtJun 22, 2026
  5. Patrick SteinhardtJun 22, 2026
  6. Patrick SteinhardtJun 22, 2026
  7. Michael MontalboJun 26, 2026
  8. Jeff KingJun 26, 2026
  9. Patrick SteinhardtJun 26, 2026
  10. Junio C HamanoJun 26, 2026
  11. Michael MontalboJun 26, 2026
  12. 0/3 fixing expensive http test timeoutsJeff King, Jun 28, 2026
  13. 1/3 t/lib-httpd: bump apache timeoutJeff King, Jun 28, 2026
  14. Michael MontalboJul 2, 2026
  15. 2/3 t5551: put many-tags case into its own repoJeff King, Jun 28, 2026
  16. Junio C HamanoJun 28, 2026
  17. Jeff KingJun 29, 2026
  18. Junio C HamanoJun 29, 2026
  19. Jeff KingJun 29, 2026
  20. 3/3 t5551: pack refs after creating many tagsJeff King, Jun 28, 2026
  21. Junio C HamanoJun 28, 2026
  22. Patrick SteinhardtJun 29, 2026
  23. Jeff KingJun 29, 2026
  24. Patrick SteinhardtJun 30, 2026
  25. Jeff KingJun 30, 2026
  26. weird quadratic reftable behavior, was: Re: [PATCH 3/3] t5551: pack refs after creating many tagsJeff King, Jun 30, 2026
  27. Patrick SteinhardtJul 1, 2026
  28. Jeff KingJul 1, 2026
  29. Kristofer KarlssonJul 1, 2026
  30. Patrick SteinhardtJul 1, 2026
  31. Kristofer KarlssonJul 3, 2026
  32. Patrick SteinhardtJul 6, 2026
  33. Kristofer KarlssonJul 6, 2026
  34. Patrick SteinhardtJun 29, 2026
  35. Junio C HamanoJun 29, 2026
  36. Patrick SteinhardtJun 29, 2026
  37. Junio C HamanoJun 29, 2026
  38. Patrick SteinhardtJun 30, 2026
  39. Junio C HamanoJun 30, 2026
  40. Patrick SteinhardtJul 1, 2026
  41. Jeff KingJun 26, 2026
  42. Junio C HamanoJun 22, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.