git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Race condition between repack and loose-objects maintenance task

From
Gregory Szorc <gregory.szorc@gmail.com>
Date
Jun 29, 2022, 17:16 UTC
Message-ID
<CAKQoGakSFaNm10ZeTKc8XtTcD0JW19CZP1OwA4j7W__iBQaJfg@mail.gmail.com>
In-Reply-To
<YryF+vkosJOXf+mQ@nand.local>
On Wed, Jun 29, 2022 at 10:03 AM Taylor Blau <me@ttaylorr.com> wrote:
Show 25 quoted lines
>
> Hi Greg,
>
> On Wed, Jun 29, 2022 at 09:55:54AM -0700, Gregory Szorc wrote:
> > 1. `git repack -A` creates loose objects
> > 2. `git maintenance`'s `loose-objects` task deletes those loose objects
> > 3. `git repack -A` fails to find the loose objects it just created and
> > aborts with `fatal: unable to add recent objects`
>
> This is a somewhat well-known race that occurs when one Git process
> decides unreachable objects are safe to be deleted, but an incoming push
> or reference update makes those to-be-deleted objects reachable before
> they are actually removed, leaving the repository in a corrupt state.
>
> I'm surprised that the loose-objects maintenance task deletes those
> objects, though, since it just runs `prune-packed` or (the equivalent
> of) `repack -d`, neither of which will actually delete objects from the
> repository.
>
> I see that Stolee is already on the CC list, so perhaps he could chime
> in on the above.
>
> In either case, my recommendation would be to keep those unreachable
> objects which haven't yet aged out of the repository around for longer,
> which will decrease the likelihood of seeing the race.

We had to lower gc.pruneExpire from its default of 1 hour because otherwise this would result in the creation of thousands of loose objects. This is normally acceptable. However, on NFS, the churn from lots of file creation and deletion resulted in acceptable performance degradation. We had to lower gc.pruneExpire to minimize the damage.

Show 9 quoted lines
> If your
> repository has a lot of unreachable objects (such that storing each
> one of them individually as loose is impractical), I would recommend
> using cruft packs (e.g., by running either `git repack --cruft -d` or
> `git gc --cruft`) to collect those unreachable objects together into a
> single pack.
>
> See Documentation/technical/cruft-packs.txt for more information about
> cruft packs.

Yes, I saw this new feature in the Git release this week and am hoping to roll it out to better mitigate this general problem! With cruft packs, I anticipate being able to restore a longer gc.pruneExpire value as they should mitigate the NFS performance badness we're working around. Thank you for confirming it should help with the problem.

Previous: Taylor BlauNext: Taylor Blau
Message 4 of 16 in “Race condition between repack and loose-objects maintenance task”
  1. Gregory SzorcJun 29, 2022
  2. Taylor BlauJun 29, 2022
  3. Taylor BlauJun 29, 2022
  4. Gregory SzorcJun 29, 2022
  5. Taylor BlauJun 29, 2022
  6. Gregory SzorcJun 30, 2022
  7. Taylor BlauJun 30, 2022
  8. Taylor BlauJun 30, 2022
  9. Junio C HamanoJun 30, 2022
  10. Gregory SzorcJul 5, 2022
  11. Ævar Arnfjörð BjarmasonJul 6, 2022
  12. Gregory SzorcJul 20, 2022
  13. Ævar Arnfjörð BjarmasonJul 20, 2022
  14. Gregory SzorcJul 26, 2022
  15. Ævar Arnfjörð BjarmasonJul 26, 2022
  16. Gregory SzorcJul 20, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.