git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 03/10] builtin/gc.c: ignore cruft packs with `--keep-largest-pack`

From
Jeff King <peff@peff.net>
Date
Apr 18, 2023, 10:39 UTC
Message-ID
<20230418103909.GD508219@coredump.intra.peff.net>
In-Reply-To
<ZD3QLMs8/+DLKZM6@nand.local>
On Mon, Apr 17, 2023 at 07:03:08PM -0400, Taylor Blau wrote:
Show 33 quoted lines
> On Mon, Apr 17, 2023 at 03:54:35PM -0700, Junio C Hamano wrote:
> > Taylor Blau <me@ttaylorr.com> writes:
> >
> > >   - The same is true for `gc.bigPackThreshold`, if the size of the cruft
> > >     pack exceeds the limit set by the caller.
> >
> > This is not as cut-and-dried clear as the previous one.  "This pack
> > is so large that it is not worth rewriting it only to expunge a
> > handful of objects that are no longer reachable from it" is the main
> > motivation to use this configuration, but doesn't some part of the
> > same reasoning apply equally to a large cruft pack?  But let's
> > assume that the configuration is totally irrelevant to cruft packs
> > and read on.
> 
> This is an inherent design trade-off. I imagine that callers who want to
> avoid rewriting their (large) cruft packs would prefer to generate a new
> cruft pack on top with just the recently accumulated unreachable
> objects.
> 
> That kind of works, except if you need to prune objects that are packed
> in an earlier cruft pack. If you have `gc.bigPackThreshold`, there is no
> way to do this if you need to expire objects that are in cruft packs
> above that threshold.
> 
> A user may find themselves frustrated when trying to `git gc --prune`
> some sensitive object(s) from their repository doesn't appear to work,
> only to discover that `gc.bigPackThreshold` is set somewhere in their
> configuration.
> 
> Writing (largely) the same cruft pack to expunge a few objects isn't
> ideal, but it is better than the status quo. And if you have so many
> unreachable objects that this is a concern, it is probably time to prune
> anyway.

Yeah, what your patch does makes sense to me as a default behavior. In a pre-cruft-pack world, those objects would all be left alone by gc.bigPackThreshol (because they're loose), and the essence of cruft-packs is creating a parallel universe where those ejected-to-loose objects just happen to be stored in a more efficient format.

Show 7 quoted lines
> It is possible that in the future we could support writing multiple
> cruft packs (we already handle the presence of multiple cruft packs
> fine, just don't expose an easy way for the user to write >1 of them).
> And at that point we would be able to relax this patch a bit and allow
> `gc.bigPackThreshold` to cover cruft packs, too. But in the meantime,
> the benefit of avoiding loose object explosions outweighs the possible
> drawbacks here, IMHO.

I wondered if that interface might be an option to say "hey, I have a gigantic cruft file I want to carry forward, please leave it alone".

But if you have a giant cruft pack that is making your "git gc" too slow, it will eventually age out on its own. And if you're impatient, then "git gc --prune=now" is probably the right tool.

And If you really did want to keep rolling it forward for some reason, then I'd think marking it with ".keep" would be the best thing (and maybe even dropping the mtimes file? I'm not sure a how a kept-cruft pack does or should behave).

-Peff
Previous: Taylor BlauNext: Derrick Stolee
Message 9 of 48 in “gc: enable cruft packs by default”
  1. 00/10 gc: enable cruft packs by defaultTaylor Blau, Apr 17, 2023
  2. 01/10 pack-write.c: plug a leak in stage_tmp_packfiles()Taylor Blau, Apr 17, 2023
  3. Jeff KingApr 18, 2023
  4. Taylor BlauApr 18, 2023
  5. 02/10 builtin/repack.c: fix incorrect reference to '-C'Taylor Blau, Apr 17, 2023
  6. 03/10 builtin/gc.c: ignore cruft packs with `--keep-largest-pack`Taylor Blau, Apr 17, 2023
  7. Junio C HamanoApr 17, 2023
  8. Taylor BlauApr 17, 2023
  9. Jeff KingApr 18, 2023
  10. Derrick StoleeApr 18, 2023
  11. 04/10 t/t5304-prune.sh: prepare for `gc --cruft` by defaultTaylor Blau, Apr 17, 2023
  12. 05/10 t/t9300-fast-import.sh: prepare for `gc --cruft` by defaultTaylor Blau, Apr 17, 2023
  13. Jeff KingApr 18, 2023
  14. Taylor BlauApr 18, 2023
  15. 06/10 t/t6500-gc.sh: refactor cruft pack testsTaylor Blau, Apr 17, 2023
  16. 07/10 t/t6500-gc.sh: add additional test casesTaylor Blau, Apr 17, 2023
  17. Jeff KingApr 18, 2023
  18. Taylor BlauApr 18, 2023
  19. 08/10 t/t6501-freshen-objects.sh: prepare for `gc --cruft` by defaultTaylor Blau, Apr 17, 2023
  20. Jeff KingApr 18, 2023
  21. Taylor BlauApr 18, 2023
  22. Jeff KingApr 22, 2023
  23. 09/10 builtin/gc.c: make `gc.cruftPacks` enabled by defaultTaylor Blau, Apr 17, 2023
  24. Jeff KingApr 18, 2023
  25. Taylor BlauApr 18, 2023
  26. 10/10 repository.h: drop unused `gc_cruft_packs`Taylor Blau, Apr 17, 2023
  27. Jeff KingApr 18, 2023
  28. Jeff KingApr 18, 2023
  29. Taylor BlauApr 18, 2023
  30. 00/10 gc: enable cruft packs by defaultTaylor Blau, Apr 18, 2023
  31. 01/10 pack-write.c: plug a leak in stage_tmp_packfiles()Taylor Blau, Apr 18, 2023
  32. Junio C HamanoApr 19, 2023
  33. Taylor BlauApr 20, 2023
  34. Junio C HamanoApr 20, 2023
  35. 02/10 builtin/repack.c: fix incorrect reference to '-C'Taylor Blau, Apr 18, 2023
  36. 03/10 builtin/gc.c: ignore cruft packs with `--keep-largest-pack`Taylor Blau, Apr 18, 2023
  37. 04/10 t/t5304-prune.sh: prepare for `gc --cruft` by defaultTaylor Blau, Apr 18, 2023
  38. 05/10 t/t6501-freshen-objects.sh: prepare for `gc --cruft` by defaultTaylor Blau, Apr 18, 2023
  39. 06/10 t/t6500-gc.sh: refactor cruft pack testsTaylor Blau, Apr 18, 2023
  40. 07/10 t/t6500-gc.sh: add additional test casesTaylor Blau, Apr 18, 2023
  41. 08/10 t/t9300-fast-import.sh: prepare for `gc --cruft` by defaultTaylor Blau, Apr 18, 2023
  42. 09/10 builtin/gc.c: make `gc.cruftPacks` enabled by defaultTaylor Blau, Apr 18, 2023
  43. Junio C HamanoApr 19, 2023
  44. Taylor BlauApr 20, 2023
  45. Junio C HamanoApr 20, 2023
  46. Taylor BlauApr 20, 2023
  47. 10/10 repository.h: drop unused `gc_cruft_packs`Taylor Blau, Apr 18, 2023
  48. Junio C HamanoApr 19, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.