git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 2/2] builtin/pack-objects.c: introduce `pack.recentObjectsHook`

From
Taylor Blau <me@ttaylorr.com>
Date
May 15, 2023, 20:49 UTC
Message-ID
<ZGKavGdYrFvjVex7@nand.local>
In-Reply-To
<20230512214542.GB2495860@coredump.intra.peff.net>
On Fri, May 12, 2023 at 05:45:42PM -0400, Jeff King wrote:
Show 36 quoted lines
> On Fri, May 12, 2023 at 05:24:56PM -0400, Jeff King wrote:
>
> > > This patch introduces a new configuration, `pack.recentObjectsHook`
> > > which allows the caller to specify a program (or set of programs) whose
> > > output is treated as a set of objects to treat as recent, regardless of
> > > their true age.
> >
> > I was going to complain about putting this in the "pack" section,
> > because I thought by touching reachable.c, we'd also affect git-prune.
> > But I don't think we do, because it does its own direct mtime check on
> > the loose objects.
> >
> > But I'm not sure that's the right behavior.
> >
> > It feels like even before your patch, this is a huge gap in our
> > object-retention strategy.  During repacking, we try to avoid dropping
> > objects which are reachable from recent-but-unreachable things we're
> > keeping (since otherwise it effectively corrupts those recent objects,
> > making them less valuable to keep). But git-prune will happily drop them
> > anyway!
> >
> > And I think the same thing would apply to your hook. If the hook says
> > "object XYZ is precious even if unreachable, keep it", then git-prune
> > ignoring that seems like it would be a source of errors.
> >
> > I suspect both could be fixed by having git-prune trigger the same
> > add_unseen_recent_objects_to_traversal() call either as part of
> > the perform_reachability_traversal() walk, or maybe in its own walk (I
> > think maybe it has to be its own because the second walk should avoid
> > complaining about missing objects).
>
> <phew> I am happy to say that I was wrong here, and git-prune behaves as
> it should, courtesy of d3038d22f9 (prune: keep objects reachable from
> recent objects, 2014-10-15). The magic happens in mark_reachable_objects(),
> which handles walking the recent objects by calling...you guessed it,
> add_unseen_recent_objects_to_traversal().

Phew. Thanks for digging into it before I was able to respond. I'm glad that this works (though I agree that we should add a test).

Show 8 quoted lines
> So it does the right thing now, and your patch should kick in
> automatically for git-prune, too. But I think we'd want two things:
>
>   1. Should the config variable name be made more generic to match?
>      Maybe "core" is too broad (though certainly I'd expect it to apply
>      anywhere in Git where we check recent-ness of objects), but perhaps
>      "gc" would make sense (even though it is not strictly part of the
>      gc command, it is within that realm of concepts).

"core" does feel pretty broad. There's some precedence for adding hook-like configuration there, at least with `core.alternateRefsCommand`. But I think that was an appropriate choice given the scope of that feature.

I think that calling it "gc.recentObjectsHook" makes the most sense.
>   2. We probably want a test covering git-prune in this situation.
Yup.

Thanks, Taylor

Previous: Jeff KingNext: Taylor Blau
Message 14 of 28 in “pack-objects: introduce `pack.extraRecentObjectsHook`”
  1. 0/2 pack-objects: introduce `pack.extraRecentObjectsHook`Taylor Blau, May 11, 2023
  2. 1/2 reachable.c: extract `obj_is_recent()`Taylor Blau, May 11, 2023
  3. 2/2 builtin/pack-objects.c: introduce `pack.recentObjectsHook`Taylor Blau, May 11, 2023
  4. Jeff KingMay 12, 2023
  5. Taylor BlauMay 15, 2023
  6. Jeff KingMay 12, 2023
  7. Taylor BlauMay 12, 2023
  8. Jeff KingMay 12, 2023
  9. Jeff KingMay 12, 2023
  10. Jeff KingMay 12, 2023
  11. Junio C HamanoMay 12, 2023
  12. Jeff KingMay 13, 2023
  13. Jeff KingMay 13, 2023
  14. Taylor BlauMay 15, 2023
  15. Taylor BlauMay 15, 2023
  16. Taylor BlauMay 11, 2023
  17. Junio C HamanoMay 11, 2023
  18. Taylor BlauMay 11, 2023
  19. 0/2 gc: introduce `gc.recentObjectsHook`Taylor Blau, May 16, 2023
  20. 1/2 reachable.c: extract `obj_is_recent()`Taylor Blau, May 16, 2023
  21. 2/2 gc: introduce `gc.recentObjectsHook`Taylor Blau, May 16, 2023
  22. Glen ChooMay 24, 2023
  23. Taylor BlauJun 7, 2023
  24. 0/2 gc: introduce `gc.recentObjectsHook`Taylor Blau, Jun 7, 2023
  25. 1/2 reachable.c: extract `obj_is_recent()`Taylor Blau, Jun 7, 2023
  26. 2/2 gc: introduce `gc.recentObjectsHook`Taylor Blau, Jun 7, 2023
  27. Glen ChooJun 9, 2023
  28. Junio C HamanoJun 12, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.