git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/7] odb: track multi-pack-indices via their object sources

From
Justin Tobler <jltobler@gmail.com>
Date
Jul 15, 2025, 21:59 UTC
Message-ID
<dhfzgevg4bpv42tmtjkhjdvfiy35qh5wcue4x3b7dwjjwju4q6@2kmwfyr5g3yj>
In-Reply-To
<20250715-b4-pks-midx-via-odb-alternate-v2-0-b0ca0b4b516e@pks.im>
On 25/07/15 01:29PM, Patrick Steinhardt wrote:
Show 43 quoted lines
> Hi,
> 
> multi-pack-indices are tracked via `struct multi_pack_index`. This data
> structure is stored inside `struct object_database`, which is the global
> database that spans across all of the object sources
> 
> This layout causes two problems:
> 
>   - Multi-pack indices aren't global to an object database, but instead
>     there can be one multi-pack index per object source. This creates a
>     mismatch between the on-disk layout and how things are organized in
>     the object database subsystems and makes some parts, like figuring
>     out whether an object source has an MIDX, quite awkward.
> 
>   - Multi-pack indices are an implementation detail of how efficient
>     access for packfiles work. As such, they are neither relevant in the
>     context of loose objects, nor in a potential future where we have
>     pluggable backends.
> 
> This patch series thus refactors the codebase to stop tracking MIDX's
> globally. Instead, they are being pushed down one level so that every
> `struct odb_source` has an optional MIDX itself. This simplifies some of
> our code and will make it easier in a future iteration to move the data
> into a packfile-specific object source backend.
> 
> Changes in v2:
>   - Changed the base of this series. It is now built on top of
>     a30f80fde92 (The eighth batch, 2025-07-08) with "ps/object-store" at
>     841a03b4046 (odb: rename `read_object_with_reference()`, 2025-07-01)
>     and "tb/midx-avoid-cruft-packs" at 5ee86c273bf (repack: exclude
>     cruft pack(s) from the MIDX where possible, 2025-06-23) merged into
>     it.
>   - Re-explain the split between object databases and object sources
>     to help readers out a bit, given that this is a rather recent
>     change.
>   - Rename `struct odb_source::multi_pack_index` to `struct
>     odb_source::midx`.
>   - Fix some overly long lines when looping through the individual
>     sources.
>   - Drop the patch that guards re-loading MIDXs, as we already have the
>     guard via `packed_git_initialized`.
>   - Remove some while-at-it changes to make the diffs easier to read.
>   - Link to v1: https://lore.kernel.org/r/20250709-b4-pks-midx-via-odb-alternate-v1-0-f31150d21331@pks.im
Thanks Patrick! From the range-diff, this version looks good to me :)
-Justin
Previous: Patrick SteinhardtNext: Junio C Hamano
Message 40 of 43 in “odb: track multi-pack-indices via their object sources”
  1. 0/8 odb: track multi-pack-indices via their object sourcesPatrick Steinhardt, Jul 9, 2025
  2. 1/8 midx: start tracking per object database sourcePatrick Steinhardt, Jul 9, 2025
  3. Justin ToblerJul 10, 2025
  4. Taylor BlauJul 10, 2025
  5. Taylor BlauJul 10, 2025
  6. Patrick SteinhardtJul 15, 2025
  7. Patrick SteinhardtJul 15, 2025
  8. 2/8 packfile: refactor `prepare_packed_git_one()` to work on sourcesPatrick Steinhardt, Jul 9, 2025
  9. Justin ToblerJul 10, 2025
  10. Taylor BlauJul 10, 2025
  11. Patrick SteinhardtJul 15, 2025
  12. 3/8 midx: stop using linked list when closing MIDXPatrick Steinhardt, Jul 9, 2025
  13. Justin ToblerJul 10, 2025
  14. Patrick SteinhardtJul 15, 2025
  15. Taylor BlauJul 10, 2025
  16. Patrick SteinhardtJul 15, 2025
  17. 4/8 midx: track whether we have loaded the MIDXPatrick Steinhardt, Jul 9, 2025
  18. Justin ToblerJul 10, 2025
  19. Taylor BlauJul 10, 2025
  20. Patrick SteinhardtJul 15, 2025
  21. 5/8 packfile: refactor `get_multi_pack_index()` to work on sourcesPatrick Steinhardt, Jul 9, 2025
  22. Justin ToblerJul 10, 2025
  23. Taylor BlauJul 10, 2025
  24. Patrick SteinhardtJul 15, 2025
  25. 6/8 packfile: stop using linked MIDX list in `find_pack_entry()`Patrick Steinhardt, Jul 9, 2025
  26. 7/8 packfile: stop using linked MIDX list in `get_all_packs()`Patrick Steinhardt, Jul 9, 2025
  27. 8/8 midx: remove now-unused linked list of multi-pack indicesPatrick Steinhardt, Jul 9, 2025
  28. Justin ToblerJul 10, 2025
  29. Junio C HamanoJul 9, 2025
  30. Taylor BlauJul 10, 2025
  31. Patrick SteinhardtJul 15, 2025
  32. 0/7 odb: track multi-pack-indices via their object sourcesPatrick Steinhardt, Jul 15, 2025
  33. 1/7 midx: start tracking per object database sourcePatrick Steinhardt, Jul 15, 2025
  34. 2/7 packfile: refactor `prepare_packed_git_one()` to work on sourcesPatrick Steinhardt, Jul 15, 2025
  35. 3/7 midx: stop using linked list when closing MIDXPatrick Steinhardt, Jul 15, 2025
  36. 4/7 packfile: refactor `get_multi_pack_index()` to work on sourcesPatrick Steinhardt, Jul 15, 2025
  37. 5/7 packfile: stop using linked MIDX list in `find_pack_entry()`Patrick Steinhardt, Jul 15, 2025
  38. 6/7 packfile: stop using linked MIDX list in `get_all_packs()`Patrick Steinhardt, Jul 15, 2025
  39. 7/7 midx: remove now-unused linked list of multi-pack indicesPatrick Steinhardt, Jul 15, 2025
  40. Justin ToblerJul 15, 2025
  41. Junio C HamanoJul 23, 2025
  42. Patrick SteinhardtJul 24, 2025
  43. Taylor BlauAug 12, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.