git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 2/4] midx-write.c: guard against incremental MIDXs in want_included_pack()

From
Taylor Blau <me@ttaylorr.com>
Date
May 28, 2025, 22:59 UTC
Message-ID
<cf3ab81a0808ace5a7d81aab6227730fa899350f.1748473122.git.me@ttaylorr.com>
In-Reply-To
<cover.1748473122.git.me@ttaylorr.com>

The function want_included_pack() is used to determine whether or not a the packfile corresponding to some given pack_int_id should be included in a 'git multi-pack-index repack' operation.

This spot looks like it would be broken, particularly in:
    struct packed_git *p;
    if (prepare_midx_pack(r, m, pack_int_id))
        return 0;
    p = m->packs[pack_int_id];

, when pack_int_id is greater than m->num_pack_in_base (supposing that m->num_packs_in_base is non-zero, or equivalently that m->base_midx is non-NULL).

Suppose we have two MIDXs in an incremental MIDX chain, each having two packs:

  - M0 = {packs: [P0, P1], base_midx: NULL}
  - M1 = {packs: [P2, P3], base_midx: M0}

noting that each pack is identified by its global pack_int_id within the chain.

So if you do something like:
    want_included_pack(the_repository, M1, pack_kept_objects, 2);

that function will call prepare_midx_pack(), which is smart enough to realize that the pack of interest is in the current layer (M1), and knows how to adjust its global pack_int_id into an index into the current layer's 'packs' array.

But the following line:
    p = m->packs[pack_int_id]; /* m is M1, pack_int_id is 2 */

looks broken, since each layer of the MIDX only maintains an array of the packs stored within that layer, and 'm' wasn't adjusted back to point at M1->base_midx (M0).

The right thing to do would be:
    struct packed_git *p;
    if (prepare_midx_pack(r, m, pack_int_id))
        return 0;
    /* open-code midx.c::midx_for_pack(), which is private */
    while (m && pack_int_id < m->num_packs_in_base)
        m = m->base_midx;
    if (!m)
        BUG("broken midx?");
    if (pack_int_id >= m->num_packs + m->num_packs_in_base)
        BUG("broken pack_int_id?");
    p = m->packs[pack_int_id - m->num_packs_in_base];

But that would be overkill, since this function never deals with incremental MIDXs having more than one layer! To see why, observe that want_included_pack() has two callers:

  - midx-write.c::fill_included_packs_all()
  - midx-write.c::fill_included_packs_batch()

and those functions' sole caller is in midx-write.c::midx_repack(), which dispatches a call to one or the other depending on whether or not the batch_size is non-zero.

And at the very top of midx_repack(), we have a guard against non-trivial incremental MIDX chains:

    if (m->base_midx)
        die(_("cannot repack an incremental multi-pack-index"));

So want_included_pack() is OK because it will never encounter a situation where it has to chase backwards through the '->base_midx' pointer. But that is not immediately clear from reading the code, and is too fragile for my comfort. Make this more clear by adding an ASSERT() to the above effect.

Apply the same treatment to each of the fill_included_packs-related functions as well, since those are deceptively OK by the same reasoning.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 midx-write.c | 7 +++++++
 1 file changed, 7 insertions(+)
diff --git a/midx-write.c b/midx-write.c
index dd3b3070e5..e4a3830d45 100644
--- a/midx-write.c
+++ b/midx-write.c
@@ -1636,6 +1636,9 @@ static int want_included_pack(struct repository *r,
 			      uint32_t pack_int_id)
 {
 	struct packed_git *p;
+
+	ASSERT(m && !m->base_midx);
+
 	if (prepare_midx_pack(r, m, pack_int_id))
 		return 0;
 	p = m->packs[pack_int_id];
@@ -1655,6 +1658,8 @@ static void fill_included_packs_all(struct repository *r,
 	uint32_t i;
 	int pack_kept_objects = 0;
 
+	ASSERT(m && !m->base_midx);
+
 	repo_config_get_bool(r, "repack.packkeptobjects", &pack_kept_objects);
 
 	for (i = 0; i < m->num_packs; i++) {
@@ -1675,6 +1680,8 @@ static void fill_included_packs_batch(struct repository *r,
 	struct repack_info *pack_info;
 	int pack_kept_objects = 0;
 
+	ASSERT(m && !m->base_midx);
+
 	CALLOC_ARRAY(pack_info, m->num_packs);
 
 	repo_config_get_bool(r, "repack.packkeptobjects", &pack_kept_objects);
-- 
2.49.0.640.ga4de40e6a8
Previous: Taylor BlauNext: Taylor Blau
Message 41 of 53 in “packfile: avoid access(3p) calls for missing packs”
  1. packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 16, 2025
  2. Junio C HamanoMay 16, 2025
  3. Jeff KingMay 19, 2025
  4. Junio C HamanoMay 19, 2025
  5. Patrick SteinhardtMay 20, 2025
  6. Jeff KingMay 22, 2025
  7. Taylor BlauMay 23, 2025
  8. Jeff KingMay 23, 2025
  9. 0/2 packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 20, 2025
  10. 1/2 packfile: explain ordering of how we look up auxiliary pack filesPatrick Steinhardt, May 20, 2025
  11. Taylor BlauMay 23, 2025
  12. 2/2 midx: stop repeatedly looking up nonexistent packfilesPatrick Steinhardt, May 20, 2025
  13. Jeff KingMay 22, 2025
  14. Junio C HamanoMay 22, 2025
  15. Jeff KingMay 22, 2025
  16. Junio C HamanoMay 22, 2025
  17. Taylor BlauMay 23, 2025
  18. Jeff KingMay 23, 2025
  19. Taylor BlauMay 23, 2025
  20. 0/5 midx: improve prepare_midx_pack() ergonomicsTaylor Blau, May 25, 2025
  21. 1/5 pack-bitmap.c: fix broken warning() when missing MIDX'd packTaylor Blau, May 25, 2025
  22. Patrick SteinhardtMay 26, 2025
  23. Taylor BlauMay 28, 2025
  24. 2/5 midx-write.c: guard against incremental MIDXs in want_included_pack()Taylor Blau, May 25, 2025
  25. Patrick SteinhardtMay 26, 2025
  26. Taylor BlauMay 28, 2025
  27. 3/5 midx-write.c: simplify fill_packs_from_midx()Taylor Blau, May 25, 2025
  28. Patrick SteinhardtMay 26, 2025
  29. Taylor BlauMay 28, 2025
  30. 4/5 midx-write.c: extract inner loop from fill_packs_from_midx()Taylor Blau, May 25, 2025
  31. 5/5 midx: return a `packed_git` pointer from `prepare_midx_pack()`Taylor Blau, May 25, 2025
  32. Patrick SteinhardtMay 26, 2025
  33. Taylor BlauMay 28, 2025
  34. Patrick SteinhardtMay 28, 2025
  35. 0/4 midx: improve prepare_midx_pack() ergonomicsTaylor Blau, May 28, 2025
  36. 1/4 midx: access pack names through `nth_midxed_pack_name()`Taylor Blau, May 28, 2025
  37. Junio C HamanoMay 29, 2025
  38. Taylor BlauJun 3, 2025
  39. Junio C HamanoMay 29, 2025
  40. Taylor BlauJun 3, 2025
  41. 2/4 midx-write.c: guard against incremental MIDXs in want_included_pack()Taylor Blau, May 28, 2025
  42. 3/4 midx-write.c: extract inner loop from fill_packs_from_midx()Taylor Blau, May 28, 2025
  43. 4/4 midx: return a `packed_git` pointer from `prepare_midx_pack()`Taylor Blau, May 28, 2025
  44. Jeff KingMay 30, 2025
  45. Taylor BlauJun 3, 2025
  46. Junio C HamanoAug 28, 2025
  47. Taylor BlauMay 23, 2025
  48. Jeff KingMay 23, 2025
  49. Junio C HamanoMay 21, 2025
  50. 0/2 packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 28, 2025
  51. 2/2 midx: stop repeatedly looking up nonexistent packfilesPatrick Steinhardt, May 28, 2025
  52. 1/2 packfile: explain ordering of how we look up auxiliary pack filesPatrick Steinhardt, May 28, 2025
  53. Jeff KingMay 30, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.