git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 02/10] upload-pack: adapt keepalives based on buffering

From
Patrick Steinhardt <ps@pks.im>
Date
Mar 10, 2026, 12:08 UTC
Message-ID
<abAJ2iMVa9rwGgr5@pks.im>
In-Reply-To
<20260305005611.GB4943@coredump.intra.peff.net>
On Wed, Mar 04, 2026 at 07:56:11PM -0500, Jeff King wrote:
Show 17 quoted lines
> On Tue, Mar 03, 2026 at 04:00:17PM +0100, Patrick Steinhardt wrote:
> 
> > The most important edge case here happens in `relay_pack_data()`. When
> > we haven't seen the initial "PACK" signature from git-pack-objects(1)
> > yet we buffer incoming data. So in the worst case, if each of the bytes
> > of that signature arrive shortly before the configured keepalive
> > timeout, then we may not send out any data for a time period that is
> > (almost) four times as long as the configured timeout.
> 
> Thanks for laying out this case. I think this is all-but-impossible in
> practice, as anybody writing "PACK" is going to do so all at once. Even
> 4 separate write() calls would be fine, as long as it does not pause in
> between!
> 
> I think there's another one, too. If we are getting packfile_uris, and
> pack-objects writes half a line, we will pause waiting for the complete
> line to show up. This also seems quite unlikely in practice.

It could happen in case the data was written by the pack-objects hook. But I fully agree that this is a rather unlikely scenario.

Show 38 quoted lines
> > This edge case is rather unlikely to matter in practice. But in a
> > subsequent commit we're going to adapt our buffering mechanism to become
> > more aggressive, which makes it more likely that we don't send any data
> > for an extended amount of time.
> > 
> > Adapt the logic so that instead of using a fixed timeout on every call
> > to poll(3p), we instead figure out how much time has passed since the
> > last-sent data.
> 
> OK. That should not be too bad to do, though...
> 
> > @@ -365,10 +373,14 @@ static void create_pack_file(struct upload_pack_data *pack_data,
> >  	 */
> >  
> >  	while (1) {
> > +		uint64_t now_ms = getnanotime() / 1000000;
> 
> ...now we are talking about wall-clock time since the epoch. What
> happens if time goes backwards due to a clock reset?
> 
> Then now_ms may be less than last_sent_ms, and then here:
> 
> > +		} else {
> > +			/*
> > +			 * The polling timeout needs to be adjusted based on
> > +			 * the time we have sent our last package. The longer
> > +			 * it's been in the past, the shorter the timeout
> > +			 * becomes until we eventually don't block at all.
> > +			 */
> > +			polltimeout_ms = 1000 * pack_data->keepalive - (now_ms - last_sent_ms);
> > +			if (polltimeout_ms < 0)
> > +				polltimeout_ms = 0;
> > +		}
> 
> ...we end up with a value higher than the keepalive, and we wait too
> long. That's probably an OK outcome for such an exceptional condition.
> The worst case if you set your clock back is that we fail to send
> keepalives until the next actual data chunk arrives.

Right. It's rather unlikely to happen, and in addition to that we use a monotonic clock in `getnanotime()` on systems with `CLOCK_MONOTONIC` and on Windows. Which probably covers almost all platforms out there.

Patrick
Previous: Jeff KingNext: Patrick Steinhardt
Message 17 of 53 in “upload-pack: reduce lock contention when writing packfile data”
  1. 0/2 upload-pack: reduce lock contention when writing packfile dataPatrick Steinhardt, Feb 27, 2026
  2. 1/2 upload-pack: fix debug statement when flushing packfile dataPatrick Steinhardt, Feb 27, 2026
  3. 2/2 upload-pack: reduce lock contention when writing packfile dataPatrick Steinhardt, Feb 27, 2026
  4. brian m. carlsonFeb 27, 2026
  5. Patrick SteinhardtFeb 27, 2026
  6. Junio C HamanoFeb 27, 2026
  7. Jeff KingFeb 27, 2026
  8. Patrick SteinhardtMar 2, 2026
  9. Jeff KingMar 2, 2026
  10. Patrick SteinhardtMar 3, 2026
  11. Jeff KingMar 3, 2026
  12. Patrick SteinhardtMar 3, 2026
  13. 00/10 upload-pack: reduce lock contention when writing packfile dataPatrick Steinhardt, Mar 3, 2026
  14. 01/10 upload-pack: fix debug statement when flushing packfile dataPatrick Steinhardt, Mar 3, 2026
  15. 02/10 upload-pack: adapt keepalives based on bufferingPatrick Steinhardt, Mar 3, 2026
  16. Jeff KingMar 5, 2026
  17. Patrick SteinhardtMar 10, 2026
  18. 03/10 upload-pack: reduce lock contention when writing packfile dataPatrick Steinhardt, Mar 3, 2026
  19. Jeff KingMar 5, 2026
  20. Patrick SteinhardtMar 10, 2026
  21. 04/10 git-compat-util: introduce `cast_size_t_to_ssize_t()`Patrick Steinhardt, Mar 3, 2026
  22. 05/10 compat/posix: introduce writev(3p) wrapperPatrick Steinhardt, Mar 3, 2026
  23. Junio C HamanoMar 4, 2026
  24. Jeff KingMar 5, 2026
  25. brian m. carlsonMar 5, 2026
  26. Johannes SixtMar 5, 2026
  27. brian m. carlsonMar 5, 2026
  28. Patrick SteinhardtMar 10, 2026
  29. 06/10 wrapper: introduce writev(3p) wrappersPatrick Steinhardt, Mar 3, 2026
  30. 07/10 sideband: use writev(3p) to send pktlinesPatrick Steinhardt, Mar 3, 2026
  31. Junio C HamanoMar 4, 2026
  32. 08/10 csum-file: introduce `hashfd_ext()`Patrick Steinhardt, Mar 3, 2026
  33. Junio C HamanoMar 4, 2026
  34. Patrick SteinhardtMar 10, 2026
  35. 09/10 csum-file: drop `hashfd_throughput()`Patrick Steinhardt, Mar 3, 2026
  36. 10/10 builtin/pack-objects: reduce lock contention when writing packfile dataPatrick Steinhardt, Mar 3, 2026
  37. 00/10 upload-pack: reduce lock contention when writing packfile dataPatrick Steinhardt, Mar 10, 2026
  38. 01/10 upload-pack: fix debug statement when flushing packfile dataPatrick Steinhardt, Mar 10, 2026
  39. 02/10 upload-pack: adapt keepalives based on bufferingPatrick Steinhardt, Mar 10, 2026
  40. 03/10 upload-pack: prefer flushing data over sending keepalivePatrick Steinhardt, Mar 10, 2026
  41. Junio C HamanoMar 10, 2026
  42. Patrick SteinhardtMar 10, 2026
  43. 04/10 upload-pack: reduce lock contention when writing packfile dataPatrick Steinhardt, Mar 10, 2026
  44. 05/10 compat/posix: introduce writev(3p) wrapperPatrick Steinhardt, Mar 10, 2026
  45. Junio C HamanoMar 10, 2026
  46. 06/10 wrapper: introduce writev(3p) wrappersPatrick Steinhardt, Mar 10, 2026
  47. 07/10 sideband: use writev(3p) to send pktlinesPatrick Steinhardt, Mar 10, 2026
  48. 08/10 csum-file: introduce `hashfd_ext()`Patrick Steinhardt, Mar 10, 2026
  49. 09/10 csum-file: drop `hashfd_throughput()`Patrick Steinhardt, Mar 10, 2026
  50. 10/10 builtin/pack-objects: reduce lock contention when writing packfile dataPatrick Steinhardt, Mar 10, 2026
  51. Junio C HamanoMar 10, 2026
  52. Johannes SixtMar 10, 2026
  53. Patrick SteinhardtMar 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.