Re: [PATCH v2 2/2] midx: stop repeatedly looking up nonexistent packfiles

From: Jeff King <peff@peff.net>
Date: May 22, 2025, 16:59 UTC
Message-ID: <20250522165924.GA1613@coredump.intra.peff.net>
In-Reply-To: <xmqqbjrk7ikx.fsf@gitster.g>

On Thu, May 22, 2025 at 08:47:58AM -0700, Junio C Hamano wrote:

Show 6 quoted lines

> So the idea is to leave m->packs[] for unused pack NULL instead of
> magic value, so that users of that array do not need to check?
> 
> I think that is a lot safer than the magic "we know this fails"
> value that existing callers that have long trusted that a non-NULL
> .packs[] element is a valid pack.

Right. It would protect any further users of that assumption that we missed.

Show 12 quoted lines

> By the way, I suspect I am not reading the code correctly, but I am
> not sure what fill_midx_entry() does with a failed case.
> 
> 	midx_for_object(&m, pos);
> 	pack_int_id = nth_midxed_pack_int_id(m, pos);
> 
> 	if (prepare_midx_pack(r, m, pack_int_id))
> 		return 0;
> 
> With or without cached failure, this should return 0 when and only
> when m->packs[pack_int_id] is a usable pack.  But what about the
> access on the next line?

I think there's a subtlety here with incremental midx's, in that a pack id can be "global" within the whole midx chain, or a local index into a specific chain element's list of packs.

In fill_midx_entry(), I think we get such a global id back from nth_midxed_pack_int_id(). And then when we hand that to prepare_midx_pack(), it is converted into a local midx/id pair with:

  pack_int_id = midx_for_pack(&m, pack_int_id);

where the end of that function is the same:

  return pack_int_id - m->num_packs_in_base;

you saw elsewhere. So at that point we have a local midx and index, which is what prepare_midx_pack() fills via the m->packs[pack_int_id] field.

So when you say "only when m->packs[pack_int_id] is a usable pack", you are talking about the local m/pack_int_id within that function.

But back in the caller...

Show 6 quoted lines

> 	p = m->packs[pack_int_id - m->num_packs_in_base];
> 
> Do we have any guarantee that we called prepare_midx_pack() for 
> the pack at (pack_int_id - m->num_packs_in_base)th slot?  Can p
> be NULL here?  And with the magic "we know this fails" value, can p
> be that magic value?

Our pack_int_id is the global one, so it needs to be adjusted. But this pack pointer we access is the same one that was filled (or not) by prepare_midx_pack(). So it cannot be NULL or the magic "fails" value, because prepare_midx_pack() returned 0.

So I think this code is fine.

One thing that did puzzle me: in prepare_midx_pack() we not only adjust the pack_int_id, but we may walk back through the midx chain to find the correct multi_pack_index struct. Wouldn't the caller need to do the same?

The answer is that it does. The midx_for_object() call in fill_midx_entry() does that same walk, storing the result in its local "m" variable. So the walk backwards in prepare_midx_pack() is superfluous for this particular caller, who we know is already handing us the desired multi_pack_index struct, and it could just do:

  pack_int_id -= m->num_packs_in_base;

rather than calling midx_for_pack(). But the same is not necessarily true for other callers, so we should continue calling that function.

I suspect this would all be a bit more obvious if prepare_midx_pack() simply returned the pack pointer, avoiding the need for callers to look at m->packs at all (and making it a true cache, internally only to prepare_midx_pack()).

Looking at other callers of prepare_midx_pack():

  - in fill_packs_from_midx(), we do not adjust our "m" to match the
    index. But that is OK, because we adjust our local index (which we
    get by iterating from 0 to m->num_packs) to a global index when
    calling the function:

      if (prepare_midx_pack(ctx->repo, m, m->num_packs_in_base + i))
	...
      open_pack_index(m->packs[i]);

    which is fine.

  - I'm less sure of the call in expire_midx_packs(). It iterates over
    num_packs in the same way, but does:

      if (prepare_midx_pack(r, m, i))
              continue;

    and then looks at m->packs[i]. That would be wrong if "m" is not the
    first item in the chain. Ah, I see. Earlier we do:

      if (m->base_midx)
              die(_("cannot expire packs from an incremental multi-pack-index"));

    so we know that the global and local ids are equivalent in this
    instance (since the "base" midx . Still seems a bit fragile.

  - There's a similar case in midx-write.c:want_included_pack(). That
    one seems to have the same local/global confusion, but I do not
    obviously see anything preventing it from being fed a non-base midx.
    So it might possibly be buggy?

    Likewise fill_included_packs_batch() in the same file.

    In both cases I think if prepare_midx_pack() returned a pointer, we
    could just use it directly.

  - In nth_bitmapped_pack(), we call midx_for_pack() ourselves to get
    the right midx. That's good, but getting a pack pointer from
    prepare_midx_pack() wouldn't help us, because we still look at the
    midx struct for other reasons.

So I dunno. It might be a useful refactor, but it doesn't make the problem go away entirely. I'm suspicious of those calls in midx-write.c, but Taylor can probably say more about whether they're wrong or if there's some less-obvious reason we'd only see a base midx there.

-Peff