git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 0/12] push progress reporting and keepalives

From
Jeff King <peff@peff.net>
Date
Jul 15, 2016, 10:25 UTC
Message-ID
<20160715102506.GA23164@sigill.intra.peff.net>

If you push a large number of objects, the server side may have to chew CPU for a long time processing the input (delta resolution, connectivity check, and whatever any hooks choose to do). During this time, you get no feedback that anything is happening, unless the hooks feel like writing something to stderr. For a repository the size of linux.git, a full push from scratch can take several minutes.

This is annoying and confusing to the user, who wonders if the connection has hung. But it can also cause problems on systems that have other timeouts (e.g., firewalls dropping TCP sessions, or web proxies dropping requests that produce no response within a certain time).

This patch series adds two new features:
 1. Progress reporting for the CPU-intensive parts of receive-pack.
 2. A keepalive mechanism similar to what we use in upload-pack
    (basically sending zero-length packets on sideband 1 while the client
    is waiting for us to speak).

Both are enabled for any client which speaks the sideband protocol. Existing versions of git handle the new behavior just fine (the progress reporting is easy, because they were expecting stderr messages anyway; the keepalive works because the demuxer just relays zero bytes back to send-pack).

I also tested with both JGit and libgit2 clients, and both seem to handle the zero-length packets just fine.

There are unfortunately no automated tests, as it's hard to simulate the effect. My manual testing involved inserting "sleep" statements into index-pack (and hooks with manual sleeps), and then using "strace" to confirm that the keepalives were sent.

  [01/12]: check_everything_connected: always pass --quiet to rev-list
  [02/12]: rev-list: add optional progress reporting
  [03/12]: check_everything_connected: convert to argv_array
  [04/12]: check_everything_connected: use a struct with named options
  [05/12]: check_connected: relay errors to alternate descriptor
  [06/12]: check_connected: add progress flag
  [07/12]: clone: use a real progress meter for connectivity check
  [08/12]: index-pack: add flag for showing delta-resolution progress
  [09/12]: receive-pack: turn on index-pack resolving progress
  [10/12]: receive-pack: relay connectivity errors to sideband
  [11/12]: receive-pack: turn on connectivity progress
  [12/12]: receive-pack: send keepalives during quiet periods
-Peff
Next: Jeff King
Message 1 of 26 in “push progress reporting and keepalives”
  1. 0/12 push progress reporting and keepalivesJeff King, Jul 15, 2016
  2. 01/12 check_everything_connected: always pass --quiet to rev-listJeff King, Jul 15, 2016
  3. 02/12 rev-list: add optional progress reportingJeff King, Jul 15, 2016
  4. Junio C HamanoJul 15, 2016
  5. Jeff KingJul 16, 2016
  6. 03/12 check_everything_connected: convert to argv_arrayJeff King, Jul 15, 2016
  7. 04/12 check_everything_connected: use a struct with named optionsJeff King, Jul 15, 2016
  8. Junio C HamanoJul 15, 2016
  9. Jeff KingJul 16, 2016
  10. 05/12 check_connected: relay errors to alternate descriptorJeff King, Jul 15, 2016
  11. Junio C HamanoJul 15, 2016
  12. Jeff KingJul 16, 2016
  13. 06/12 check_connected: add progress flagJeff King, Jul 15, 2016
  14. 07/12 clone: use a real progress meter for connectivity checkJeff King, Jul 15, 2016
  15. 08/12 index-pack: add flag for showing delta-resolution progressJeff King, Jul 15, 2016
  16. 09/12 receive-pack: turn on index-pack resolving progressJeff King, Jul 15, 2016
  17. 10/12 receive-pack: relay connectivity errors to sidebandJeff King, Jul 15, 2016
  18. 11/12 receive-pack: turn on connectivity progressJeff King, Jul 15, 2016
  19. 12/12 receive-pack: send keepalives during quiet periodsJeff King, Jul 15, 2016
  20. Stefan BellerJul 15, 2016
  21. Jeff KingJul 16, 2016
  22. Stefan BellerJul 19, 2016
  23. Jeff KingJul 19, 2016
  24. Stefan BellerJul 19, 2016
  25. Jeff KingJul 20, 2016
  26. Junio C HamanoJul 15, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.