git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 12/14] builtin/pack-objects: use `packfile_store_for_each_object()`

From
Patrick Steinhardt <ps@pks.im>
Date
Jan 30, 2026, 12:57 UTC
Message-ID
<aXyqPhWDNeKJv2re@pks.im>
In-Reply-To
<20260129110839.GA1285720@coredump.intra.peff.net>
On Thu, Jan 29, 2026 at 06:08:39AM -0500, Jeff King wrote:
Show 70 quoted lines
> On Mon, Jan 26, 2026 at 09:53:18AM +0100, Patrick Steinhardt wrote:
> 
> > > Yes, the end result is the same, both your patch and what I wrote here
> > > implement the same GC-specific definition of an object's "mtime". I am
> > > not following the argument about pluggability, though. The concern I
> > > have above is that we are pushing domain-specific logic into the object
> > > storage backend, not the other way around.
> > 
> > To expand on the pluggability bit: every time you add a new backend
> > you'll have to extend the above logic to understand how it represents
> > the mtime. That by itself might be doable, but let's for example
> > consider a backend that is a black box to us (like a shared library that
> > may plug in arbitrary storage logic). In that case you would not even be
> > able to derive the information unless you have a generic layer that lets
> > you convey it to the caller.
> > 
> > So overall I agree with you that there are nuances here, and that the
> > mtimep pointer _can_ be used incorrectly. But I still think that the
> > concept is generic enough across backends, and the refactored logic
> > still works as extended. I'll try to expand the docs and commit message
> > a bit to cover this discussion.
> 
> There's a related concept that I saw while reading some of the earlier
> patches. When you converted fsck, I wondered how you would handle the
> call to read_loose_object(), which takes an actual path. And it needs to
> do so, because we want to make sure we are opening and reading that
> particular copy of the object, and not one from elsewhere.
> 
> The answer is that you punted on it for this series, and we still get
> the path via for_each_loose_file_in_source(). ;) That is OK, but I think
> it will eventually run into the same issue: we will need some kind of
> cursor or context for the iterator to be able to get extended
> information about a particular copy of an object.
> 
> I think there are probably two approaches here:
> 
>   1. The abstract odb API tries to share as little as possible. It gives
>      the caller back an opaque context struct, and that struct can be
>      handed back to the odb to get object contents or other information
>      (perhaps even an mtime!). Under the hood for the current odb
>      implementation this is probably just a pointer to a string with the
>      filesystem path for loose objects, and the usual packed_git/offset
>      pair for packed objects.
> 
>   2. The odb API provides a set of information that a particular backend
>      _might_ implement, and callers can poke at that information and
>      decide how to handle it when it's not available. And so that might
>      include a filesystem path for loose objects, which some backends
>      may choose to leave NULL.
> 
> Option (1) presents a cleaner API for the odb, but it's also more
> restrictive. Anything that a caller _might_ want to do has to be pushed
> down into the API, and it has to start learning about things like
> mtimes. And how to decide what "mtime" means for non-filesystem
> backends.
> 
> Option (2) pushes more work onto the callers. They need to not only look
> up the mtimes themselves (like they do now), but they have to decide how
> to handle the case when no path is available. Which in the worst case
> means a special case for each type of backend, though I think in
> practice they'd probably fall into rough groups.
> 
> I think one thing that appeals to me about option 2, though, is that it
> keeps a lot of the specialized "business logic" together in those
> callers. Most code doesn't are about concepts like mtime or specific
> copies of objects. But when it does, like in repack or fsck, there are
> often subtle assumptions and interpretations. I'd rather see all of that
> lumped together in the fsck code than have it split half-and-half
> between them and the odb code (which is really going to be some backends
> idea of how its concepts can be shoe-horned into the abstract API).

Yup. One thing that I'm planning to do in one of the subsequent patch series is to expand `struct object_info` to handle this.

Right now, the sturcture contains a `whence` pointer that tells us which backend the information is stored in. But that concept can be extended to surface more info: instead of only telling the caller the type, we can instead return the actual source that the object has been looked up in.

Furthermore, the `struct object_info::u` union already contains enough information for us to uniquely identify a specific option. So what we would do then is to call `odb_source_read_object_info()` on the specific source and pass it the union.

The loose source wouldn't have to do anything in that case, as the location of the object is deterministic and there can only be one copy. But the packed source would inspect `u.packed.pack` and thus know which specific object we refer to.

This still hinges on a couple intermediate steps, but I think with this plan we should be able to handle this issue in a way where the caller doesn't need to know _anything_ about how exactly the ODB source itself works.

Thanks!
Patrick
Previous: Jeff KingNext: Patrick Steinhardt
Message 100 of 120 in “odb: introduce `odb_for_each_object()`”
  1. 00/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 15, 2026
  2. 01/14 odb: rename `FOR_EACH_OBJECT_*` flagsPatrick Steinhardt, Jan 15, 2026
  3. Justin ToblerJan 15, 2026
  4. 02/14 odb: fix flags parameter to be unsignedPatrick Steinhardt, Jan 15, 2026
  5. 03/14 object-file: extract function to read object info from pathPatrick Steinhardt, Jan 15, 2026
  6. Justin ToblerJan 15, 2026
  7. Patrick SteinhardtJan 16, 2026
  8. Karthik NayakJan 20, 2026
  9. 04/14 object-file: introduce function to iterate through objectsPatrick Steinhardt, Jan 15, 2026
  10. Justin ToblerJan 15, 2026
  11. Patrick SteinhardtJan 16, 2026
  12. Karthik NayakJan 20, 2026
  13. 05/14 packfile: extract function to iterate through objects of a storePatrick Steinhardt, Jan 15, 2026
  14. 06/14 packfile: introduce function to iterate through objectsPatrick Steinhardt, Jan 15, 2026
  15. 07/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 15, 2026
  16. Justin ToblerJan 15, 2026
  17. Patrick SteinhardtJan 16, 2026
  18. Justin ToblerJan 16, 2026
  19. Patrick SteinhardtJan 19, 2026
  20. Karthik NayakJan 20, 2026
  21. Patrick SteinhardtJan 21, 2026
  22. 08/14 builtin/fsck: refactor to use `odb_for_each_object()`Patrick Steinhardt, Jan 15, 2026
  23. Justin ToblerJan 15, 2026
  24. 09/14 treewide: enumerate promisor objects via `odb_for_each_object()`Patrick Steinhardt, Jan 15, 2026
  25. 10/14 treewide: drop uses of `for_each_{loose,packed}_object()`Patrick Steinhardt, Jan 15, 2026
  26. Justin ToblerJan 15, 2026
  27. Patrick SteinhardtJan 16, 2026
  28. Justin ToblerJan 16, 2026
  29. Patrick SteinhardtJan 19, 2026
  30. 11/14 odb: introduce mtime fields for object info requestsPatrick Steinhardt, Jan 15, 2026
  31. 12/14 builtin/pack-objects: use `packfile_store_for_each_object()`Patrick Steinhardt, Jan 15, 2026
  32. 13/14 reachable: convert to use `odb_for_each_object()`Patrick Steinhardt, Jan 15, 2026
  33. 14/14 odb: drop unused `for_each_{loose,packed}_object()` functionsPatrick Steinhardt, Jan 15, 2026
  34. Junio C HamanoJan 15, 2026
  35. Patrick SteinhardtJan 16, 2026
  36. Junio C HamanoJan 16, 2026
  37. 00/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 20, 2026
  38. 01/14 odb: rename `FOR_EACH_OBJECT_*` flagsPatrick Steinhardt, Jan 20, 2026
  39. 02/14 odb: fix flags parameter to be unsignedPatrick Steinhardt, Jan 20, 2026
  40. 03/14 object-file: extract function to read object info from pathPatrick Steinhardt, Jan 20, 2026
  41. 04/14 object-file: introduce function to iterate through objectsPatrick Steinhardt, Jan 20, 2026
  42. 05/14 packfile: extract function to iterate through objects of a storePatrick Steinhardt, Jan 20, 2026
  43. 06/14 packfile: introduce function to iterate through objectsPatrick Steinhardt, Jan 20, 2026
  44. 07/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 20, 2026
  45. 08/14 builtin/fsck: refactor to use `odb_for_each_object()`Patrick Steinhardt, Jan 20, 2026
  46. 09/14 treewide: enumerate promisor objects via `odb_for_each_object()`Patrick Steinhardt, Jan 20, 2026
  47. 10/14 treewide: drop uses of `for_each_{loose,packed}_object()`Patrick Steinhardt, Jan 20, 2026
  48. 11/14 odb: introduce mtime fields for object info requestsPatrick Steinhardt, Jan 20, 2026
  49. 12/14 builtin/pack-objects: use `packfile_store_for_each_object()`Patrick Steinhardt, Jan 20, 2026
  50. 13/14 reachable: convert to use `odb_for_each_object()`Patrick Steinhardt, Jan 20, 2026
  51. 14/14 odb: drop unused `for_each_{loose,packed}_object()` functionsPatrick Steinhardt, Jan 20, 2026
  52. 00/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 21, 2026
  53. 01/14 odb: rename `FOR_EACH_OBJECT_*` flagsPatrick Steinhardt, Jan 21, 2026
  54. 02/14 odb: fix flags parameter to be unsignedPatrick Steinhardt, Jan 21, 2026
  55. Jeff KingJan 21, 2026
  56. Taylor BlauJan 22, 2026
  57. Junio C HamanoJan 22, 2026
  58. Jeff KingJan 22, 2026
  59. Patrick SteinhardtJan 23, 2026
  60. Junio C HamanoJan 26, 2026
  61. Patrick SteinhardtJan 22, 2026
  62. Taylor BlauJan 22, 2026
  63. 03/14 object-file: extract function to read object info from pathPatrick Steinhardt, Jan 21, 2026
  64. Taylor BlauJan 22, 2026
  65. Patrick SteinhardtJan 22, 2026
  66. Taylor BlauJan 22, 2026
  67. 04/14 object-file: introduce function to iterate through objectsPatrick Steinhardt, Jan 21, 2026
  68. Taylor BlauJan 22, 2026
  69. Patrick SteinhardtJan 22, 2026
  70. Taylor BlauJan 23, 2026
  71. 05/14 packfile: extract function to iterate through objects of a storePatrick Steinhardt, Jan 21, 2026
  72. Taylor BlauJan 22, 2026
  73. 06/14 packfile: introduce function to iterate through objectsPatrick Steinhardt, Jan 21, 2026
  74. Taylor BlauJan 23, 2026
  75. Patrick SteinhardtJan 23, 2026
  76. Chris TorekJan 23, 2026
  77. Junio C HamanoJan 23, 2026
  78. Taylor BlauJan 23, 2026
  79. 07/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 21, 2026
  80. Taylor BlauJan 23, 2026
  81. 08/14 builtin/fsck: refactor to use `odb_for_each_object()`Patrick Steinhardt, Jan 21, 2026
  82. Taylor BlauJan 23, 2026
  83. Patrick SteinhardtJan 23, 2026
  84. 09/14 treewide: enumerate promisor objects via `odb_for_each_object()`Patrick Steinhardt, Jan 21, 2026
  85. Taylor BlauJan 23, 2026
  86. 10/14 treewide: drop uses of `for_each_{loose,packed}_object()`Patrick Steinhardt, Jan 21, 2026
  87. Taylor BlauJan 23, 2026
  88. Patrick SteinhardtJan 23, 2026
  89. 11/14 odb: introduce mtime fields for object info requestsPatrick Steinhardt, Jan 21, 2026
  90. Taylor BlauJan 23, 2026
  91. Patrick SteinhardtJan 23, 2026
  92. Taylor BlauJan 23, 2026
  93. Patrick SteinhardtJan 26, 2026
  94. 12/14 builtin/pack-objects: use `packfile_store_for_each_object()`Patrick Steinhardt, Jan 21, 2026
  95. Taylor BlauJan 23, 2026
  96. Patrick SteinhardtJan 23, 2026
  97. Taylor BlauJan 23, 2026
  98. Patrick SteinhardtJan 26, 2026
  99. Jeff KingJan 29, 2026
  100. Patrick SteinhardtJan 30, 2026
  101. 13/14 reachable: convert to use `odb_for_each_object()`Patrick Steinhardt, Jan 21, 2026
  102. 14/14 odb: drop unused `for_each_{loose,packed}_object()` functionsPatrick Steinhardt, Jan 21, 2026
  103. Taylor BlauJan 22, 2026
  104. Junio C HamanoJan 22, 2026
  105. 00/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 26, 2026
  106. 01/14 odb: rename `FOR_EACH_OBJECT_*` flagsPatrick Steinhardt, Jan 26, 2026
  107. 02/14 odb: fix flags parameter to be unsignedPatrick Steinhardt, Jan 26, 2026
  108. 03/14 object-file: extract function to read object info from pathPatrick Steinhardt, Jan 26, 2026
  109. 04/14 object-file: introduce function to iterate through objectsPatrick Steinhardt, Jan 26, 2026
  110. 05/14 packfile: extract function to iterate through objects of a storePatrick Steinhardt, Jan 26, 2026
  111. 06/14 packfile: introduce function to iterate through objectsPatrick Steinhardt, Jan 26, 2026
  112. 07/14 odb: introduce `odb_for_each_object()`Patrick Steinhardt, Jan 26, 2026
  113. 08/14 builtin/fsck: refactor to use `odb_for_each_object()`Patrick Steinhardt, Jan 26, 2026
  114. 09/14 treewide: enumerate promisor objects via `odb_for_each_object()`Patrick Steinhardt, Jan 26, 2026
  115. 10/14 treewide: drop uses of `for_each_{loose,packed}_object()`Patrick Steinhardt, Jan 26, 2026
  116. 11/14 odb: introduce mtime fields for object info requestsPatrick Steinhardt, Jan 26, 2026
  117. 12/14 builtin/pack-objects: use `packfile_store_for_each_object()`Patrick Steinhardt, Jan 26, 2026
  118. 13/14 reachable: convert to use `odb_for_each_object()`Patrick Steinhardt, Jan 26, 2026
  119. 14/14 odb: drop unused `for_each_{loose,packed}_object()` functionsPatrick Steinhardt, Jan 26, 2026
  120. Junio C HamanoFeb 20, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.