git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCH 0/1] maintenance: separate parallelism safe and unsafe tasks

From
Calvin Wan <calvinwan@google.com>
Date
Nov 8, 2024, 17:31 UTC
Message-ID
<20241108173112.1240584-1-calvinwan@google.com>

Unless a user changes the config options `gc.autoDetach` or `maintenance.autoDetach`, Git by default runs maintenance and gc tasks in the background. This is because maintenance and gc tasks, especially in large repositories, can take a long time to run. Therefore, they _should_ be as nonintrusive as possible to other Git commands, especially porcelain ones.

However, this is not the case as discovered earlier[1] -- certain maintenance and gc tasks are not safe to run in parallel with other commands. The consequences of such are that scripts with commands that trigger maintenance/gc can race and crash. Users can also run into unexpected errors from porcelain commands that touch common files such as HEAD.lock, unaware that a background maintenance/gc task is the one holding the lock.

As Patrick points out[2], the two unsafe commands are `git reflog expire --all`, invoked by gc, and `git pack-refs --all --prune`, invoked by maintenance. We can create two buckets for subtasks -- one for async safe tasks and one for async unsafe tasks. When `[maintenance, gc].autoDetach` is not set or set to true, maintenance will run the unsafe tasks first before detaching to run the safe tasks.

This series is in RFC to see if the general direction of the patch is going in the right direction. I left a couple of WIPs in the first patch documenting what still needs to be done if the direction is palatable.

[1] https://lore.kernel.org/git/CAFySSZBCKUiY5DO3fz340a0dTb0zUDNKxaTYU0LAqsBD2RMwSg@mail.gmail.com/ [2] https://lore.kernel.org/git/ZxeilMDwq0Z3krhz@pks.im

Calvin Wan (1):
  maintenance: separate parallelism safe and unsafe tasks
 builtin/gc.c           | 173 ++++++++++++++++++++++++++++++++++++-----
 t/t7900-maintenance.sh |  24 +++---
 2 files changed, 168 insertions(+), 29 deletions(-)
-- 
2.47.0.277.g8800431eea-goog
Next: Calvin Wan
Message 1 of 13 in “maintenance: separate parallelism safe and unsafe tasks”
  1. 0/1 maintenance: separate parallelism safe and unsafe tasksCalvin Wan, Nov 8, 2024
  2. 1/1 maintenance: separate parallelism safe and unsafe tasksCalvin Wan, Nov 8, 2024
  3. Patrick SteinhardtNov 11, 2024
  4. Calvin WanNov 11, 2024
  5. Patrick SteinhardtNov 12, 2024
  6. Calvin WanNov 15, 2024
  7. Junio C HamanoNov 18, 2024
  8. Patrick SteinhardtNov 18, 2024
  9. Junio C HamanoNov 11, 2024
  10. Junio C HamanoNov 11, 2024
  11. Calvin WanNov 11, 2024
  12. Junio C HamanoNov 11, 2024
  13. Calvin WanNov 11, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.