git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Race condition between repack and loose-objects maintenance task

From
Gregory Szorc <gregory.szorc@gmail.com>
Date
Jun 29, 2022, 16:55 UTC
Message-ID
<CAKQoGamCrRMqtzziuzi8mL6E7uA3SC1WXiMGT_4rpbk1jcu_OQ@mail.gmail.com>

`git repack -A` (such as invoked by `git gc`) loosens unreachable objects from packfiles.

`git maintenance`'s `loose-objects` task consolidates loose objects into a packfile.

As neither process takes a lock or is otherwise aware of the existence of the other, there is a race condition:

1. `git repack -A` creates loose objects
2. `git maintenance`'s `loose-objects` task deletes those loose objects
3. `git repack -A` fails to find the loose objects it just created and
aborts with `fatal: unable to add recent objects`

I see this failure with regularity in a custom environment where high mutation volume repositories invoke `git maintenance run` with regularity (effectively in reaction to repo mutations). We have a Git housekeeping strategy that frequently invokes `git maintenance` with all but the `gc` task. On a ~daily frequency, we invoke `git gc` to purge accumulated garbage. This `git gc` frequently fails due to the aforementioned race condition since the long wall time of `git gc` practically guarantees a `git maintenance` would have been triggered since the repo is mutated with such high frequency.

In my scenario, I believe we have a workaround by omitting `--task=loose-objects` from the frequent `git maintenance` invocation if there exists a `gc.pid` file. However, this is only a partial solution, as `git repack` and the `loose-objects` task aren't aware of each other. (I suppose running the `gc` maintenance task and relying on the maintenance lock file could also work. However, we're invoking `git gc` explicitly since `git maintenance` doesn't allow passing custom arguments through to `git gc`.)

Gregory
Next: Taylor Blau
Message 1 of 16 in “Race condition between repack and loose-objects maintenance task”
  1. Gregory SzorcJun 29, 2022
  2. Taylor BlauJun 29, 2022
  3. Taylor BlauJun 29, 2022
  4. Gregory SzorcJun 29, 2022
  5. Taylor BlauJun 29, 2022
  6. Gregory SzorcJun 30, 2022
  7. Taylor BlauJun 30, 2022
  8. Taylor BlauJun 30, 2022
  9. Junio C HamanoJun 30, 2022
  10. Gregory SzorcJul 5, 2022
  11. Ævar Arnfjörð BjarmasonJul 6, 2022
  12. Gregory SzorcJul 20, 2022
  13. Ævar Arnfjörð BjarmasonJul 20, 2022
  14. Gregory SzorcJul 26, 2022
  15. Ævar Arnfjörð BjarmasonJul 26, 2022
  16. Gregory SzorcJul 20, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.