git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 11/16] packfile: always add packfiles to MRU when adding a pack

From
Patrick Steinhardt <ps@pks.im>
Date
Sep 2, 2025, 08:50 UTC
Message-ID
<aLav6D4PhIHsUd36@pks.im>
In-Reply-To
<aK5ZmNxE4zhNt7Zg@nand.local>
On Tue, Aug 26, 2025 at 09:04:24PM -0400, Taylor Blau wrote:
Show 40 quoted lines
> On Thu, Aug 21, 2025 at 09:39:09AM +0200, Patrick Steinhardt wrote:
> > When adding a packfile to it store we add it both to the list and map of
> > packfiles, but we don't append it to the most-recently-used list of
> > packs. We do know to add the packfile to the MRU list as soon as we
> > access any of its objects, but in between we're being inconistent. It
> > doesn't help that there are some subsystems that _do_ add the packfile
> > to the MRU after having added it, which only adds to the confusion.
> >
> > Refactor the code so that we unconditionally add packfiles to the MRU
> > when adding them to a packfile store.
> 
> I am a little confused why prepare_midx_pack() wants to add packs to the
> MRU cache so eagerly, and the commit which introduced that behavior
> (commit af96fe3392 (midx: add packs to packed_git linked list,
> 2019-04-29)) doesn't focus on that area in detail.
> 
> (Note that commit af96fe3392 *does* discuss a separate cache's behavior
> regarding the open file descriptor limit, but that LRU cache is a
> different one than the MRU cache we're discussing here.)
> 
> What I do wonder about is why af96fe3392 adds packs to the MRU cache in
> the first place. As far as I can tell, we never move MIDX'd packs to
> the front of the MRU cache at all. There are two spots that call
> list_move() on the MRU cache, which are:
> 
>  - packfile.c::find_pack_entry(), which enumerates MIDX'd
>    packs in a separate loop earlier on in the function, and ignores
>    packs in the MRU cache whose p->multi_pack_index bit is set.
> 
>  - builtin/pack-objects.c::want_object_in_pack_mtime(), which also
>    enumerates MIDX'd packs in a separate loop, though it does not
>    explicitly ignore packs in the MRU cache with the multi_pack_index
>    bit set.
> 
> In practice, though, I think these two are equivalent, since
> want_object_in_pack_mtime() will return before it gets to the MRU cache
> if it found the object in a MIDX'd pack.
> 
> So I don't think we need to be adding MIDX'd packs to the MRU cache in
> the first place.

I think the status quo is quite confusing. There are callers which directly iterate through the list of packfiles in MRU order, and that list is not guaranteed right now to even contain all packfiles that are tracked in the packfile store. The list is complete when we only load packfiles from disk, but if we ever manually add a packfile to the store in-memory the list is not up-to-date anymore. I also couldn't find a reason for that distinction.

Despite being confusing, there's another motivation here as discussed with Peff in [1]: we can drop the distinction between the MRU list and the "normal" list altogether. Ensuring that all packfiles are always stored in the MRU is a prerequisite for that subsequent change. I already got a patch series pending that does this refactoring.

That being said, I wouldn't mind moving this change into that subsequent patch series, either. It doesn't really have a strong reason to exist yet, but once we remove the distinction between the two packfile lists we have a much stronger argument.

[1]: <20250820192008.GA1662788@coredump.intra.peff.net>
Patrick
Previous: Taylor BlauNext: Patrick Steinhardt
Message 76 of 181 in “packfile: carve out a new packfile store”
  1. 00/16 packfile: carve out a new packfile storePatrick Steinhardt, Aug 19, 2025
  2. 01/16 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  3. Karthik NayakAug 19, 2025
  4. Patrick SteinhardtAug 20, 2025
  5. Junio C HamanoAug 19, 2025
  6. Patrick SteinhardtAug 20, 2025
  7. 02/16 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  8. 03/16 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  9. Karthik NayakAug 19, 2025
  10. Junio C HamanoAug 19, 2025
  11. Karthik NayakAug 20, 2025
  12. Junio C HamanoAug 22, 2025
  13. Documentation: note styling for bit fieldsKarthik Nayak, Aug 26, 2025
  14. Patrick SteinhardtAug 20, 2025
  15. Junio C HamanoAug 20, 2025
  16. 04/16 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  17. 05/16 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  18. Karthik NayakAug 20, 2025
  19. Jeff KingAug 20, 2025
  20. Patrick SteinhardtAug 21, 2025
  21. 06/16 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  22. Junio C HamanoAug 19, 2025
  23. Patrick SteinhardtAug 20, 2025
  24. 07/16 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Aug 19, 2025
  25. Junio C HamanoAug 19, 2025
  26. 08/16 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  27. 09/16 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Aug 19, 2025
  28. Karthik NayakAug 20, 2025
  29. 10/16 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  30. 11/16 packfile: always add packfiles to MRU when adding a packPatrick Steinhardt, Aug 19, 2025
  31. Karthik NayakAug 20, 2025
  32. 12/16 packfile: introduce function to load and add packfilesPatrick Steinhardt, Aug 19, 2025
  33. Karthik NayakAug 20, 2025
  34. Patrick SteinhardtAug 21, 2025
  35. 13/16 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Aug 19, 2025
  36. 14/16 packfile: remove `get_packed_git()`Patrick Steinhardt, Aug 19, 2025
  37. Karthik NayakAug 20, 2025
  38. Patrick SteinhardtAug 21, 2025
  39. Karthik NayakAug 20, 2025
  40. 15/16 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  41. Karthik NayakAug 20, 2025
  42. Patrick SteinhardtAug 21, 2025
  43. 16/16 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  44. Junio C HamanoAug 19, 2025
  45. Karthik NayakAug 20, 2025
  46. 00/16 packfile: carve out a new packfile storePatrick Steinhardt, Aug 21, 2025
  47. 01/16 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  48. 02/16 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  49. Taylor BlauAug 25, 2025
  50. Patrick SteinhardtSep 2, 2025
  51. Taylor BlauSep 2, 2025
  52. Junio C HamanoSep 2, 2025
  53. Patrick SteinhardtSep 3, 2025
  54. Taylor BlauSep 11, 2025
  55. Patrick SteinhardtSep 15, 2025
  56. 03/16 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  57. Taylor BlauAug 26, 2025
  58. 04/16 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  59. Taylor BlauAug 26, 2025
  60. 05/16 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  61. 06/16 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  62. Taylor BlauAug 26, 2025
  63. Patrick SteinhardtSep 2, 2025
  64. 07/16 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Aug 21, 2025
  65. Taylor BlauAug 26, 2025
  66. 08/16 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  67. Taylor BlauAug 26, 2025
  68. 09/16 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Aug 21, 2025
  69. Taylor BlauAug 26, 2025
  70. Patrick SteinhardtSep 2, 2025
  71. 10/16 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  72. Taylor BlauAug 26, 2025
  73. Patrick SteinhardtSep 2, 2025
  74. 11/16 packfile: always add packfiles to MRU when adding a packPatrick Steinhardt, Aug 21, 2025
  75. Taylor BlauAug 27, 2025
  76. Patrick SteinhardtSep 2, 2025
  77. 12/16 packfile: introduce function to load and add packfilesPatrick Steinhardt, Aug 21, 2025
  78. Taylor BlauAug 27, 2025
  79. 13/16 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Aug 21, 2025
  80. Taylor BlauAug 27, 2025
  81. 14/16 packfile: remove `get_packed_git()`Patrick Steinhardt, Aug 21, 2025
  82. Taylor BlauAug 27, 2025
  83. Patrick SteinhardtSep 2, 2025
  84. Taylor BlauSep 11, 2025
  85. Patrick SteinhardtSep 15, 2025
  86. 15/16 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  87. Taylor BlauAug 27, 2025
  88. Patrick SteinhardtSep 2, 2025
  89. Taylor BlauSep 11, 2025
  90. Patrick SteinhardtSep 15, 2025
  91. 16/16 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  92. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 2, 2025
  93. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  94. Karthik NayakSep 9, 2025
  95. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  96. Karthik NayakSep 9, 2025
  97. Patrick SteinhardtSep 9, 2025
  98. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  99. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  100. Karthik NayakSep 9, 2025
  101. Patrick SteinhardtSep 9, 2025
  102. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  103. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  104. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 2, 2025
  105. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  106. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 2, 2025
  107. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  108. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 2, 2025
  109. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 2, 2025
  110. 13/15 packfile: remove `get_packed_git()`Patrick Steinhardt, Sep 2, 2025
  111. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  112. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  113. Junio C HamanoSep 2, 2025
  114. Taylor BlauSep 11, 2025
  115. Karthik NayakSep 9, 2025
  116. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 9, 2025
  117. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  118. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  119. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  120. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  121. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  122. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  123. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 9, 2025
  124. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  125. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 9, 2025
  126. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  127. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 9, 2025
  128. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 9, 2025
  129. 13/15 packfile: remove `get_packed_git()`Patrick Steinhardt, Sep 9, 2025
  130. Taylor BlauSep 11, 2025
  131. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  132. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  133. Karthik NayakSep 10, 2025
  134. Taylor BlauSep 11, 2025
  135. Taylor BlauSep 11, 2025
  136. Patrick SteinhardtSep 15, 2025
  137. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 15, 2025
  138. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  139. Justin ToblerSep 17, 2025
  140. Patrick SteinhardtSep 23, 2025
  141. Justin ToblerSep 24, 2025
  142. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  143. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  144. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  145. Justin ToblerSep 17, 2025
  146. Patrick SteinhardtSep 23, 2025
  147. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  148. Justin ToblerSep 17, 2025
  149. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  150. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 15, 2025
  151. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  152. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 15, 2025
  153. Justin ToblerSep 17, 2025
  154. Patrick SteinhardtSep 23, 2025
  155. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  156. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 15, 2025
  157. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 15, 2025
  158. 13/15 packfile: refactor `get_packed_git()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  159. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  160. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  161. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 23, 2025
  162. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  163. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  164. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  165. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  166. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  167. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  168. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 23, 2025
  169. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  170. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 23, 2025
  171. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  172. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 23, 2025
  173. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 23, 2025
  174. 13/15 packfile: refactor `get_packed_git()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  175. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  176. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  177. Justin ToblerSep 24, 2025
  178. Junio C HamanoSep 25, 2025
  179. Patrick SteinhardtSep 26, 2025
  180. Taylor BlauSep 28, 2025
  181. Patrick SteinhardtSep 29, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.