git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 4/4] midx: return a `packed_git` pointer from `prepare_midx_pack()`

From
Taylor Blau <me@ttaylorr.com>
Date
Jun 3, 2025, 22:27 UTC
Message-ID
<aD92wu6Mk/S9Qm4t@nand.local>
In-Reply-To
<20250530065034.GC1321283@coredump.intra.peff.net>
On Fri, May 30, 2025 at 02:50:34AM -0400, Jeff King wrote:
Show 23 quoted lines
> On Wed, May 28, 2025 at 06:59:09PM -0400, Taylor Blau wrote:
>
> > Let's instead have prepare_midx_pack() return a pointer to the
> > packed_git structure itself, hiding the above as an implementation
> > detail of prepare_midx_pack(). This patch turns the above snippet into:
> >
> >     struct packed_git *p = prepare_midx_pack(the_repository, some_midx,
> >                                              some_pack_int_id);
> >     if (!p)
> >         die("could not load pack xyz");
> >
> > making it far easier and less error-prone to access packs by their
> > pack_int_id in a MIDX chain.
>
> So obviously I like this direction, but a few small comments:
>
> > (In the future, we may want to consider similar treatment for, e.g., the
> > pack_names array. Likewise, it might make sense to rename the "packs"
> > member of the MIDX structure to suggest that it shouldn't be accessed
> > directly outside of midx.c.)
>
> Is this note still valid for v2? It looks like patch 1 adds
> nth_midxed_pack_name() and tries to use it everywhere.

Yeah, we should get rid of this. I had written it before I wrote what is now the first patch in this series, and neglected to remove it before sending out the latest round.

Show 27 quoted lines
> > @@ -1649,9 +1646,9 @@ static int want_included_pack(struct repository *r,
> >
> >  	ASSERT(m && !m->base_midx);
> >
> > -	if (prepare_midx_pack(r, m, pack_int_id))
> > +	p = prepare_midx_pack(r, m, pack_int_id);
> > +	if (!p)
> >  		return 0;
> > -	p = m->packs[pack_int_id];
> >  	if (!pack_kept_objects && p->pack_keep)
> >  		return 0;
> >  	if (p->is_cruft)
>
> The ASSERT() in the context is from earlier in the series. But do we
> need it once we have this patch? We no longer look at pack_int_id except
> to pass it to prepare_midx_pack(), which handles non-base midx slices
> just fine.
>
> So we could loosen the assertion now. Or we could wait for later when
> somebody wants/needs to do so, but I'm not sure how easy they would find
> it to dig in the history. They would find the commit that added the
> ASSERT(), but may not realize that this later commit made it OK to
> loosen.
>
> I didn't check the other ASSERT() spots from that earlier patch (IIRC,
> some of them may actually look use the pack_int_id for other things, and
> wouldn't be ready for non-base slices).

We could loosen the assertion here, but part of me likes keeping it as a self-documenting note that this function is only intended to be used for non-incremental MIDXs.

Show 25 quoted lines
> > -int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,
> > -		      uint32_t pack_int_id)
> > +struct packed_git *prepare_midx_pack(struct repository *r,
> > +				     struct multi_pack_index *m,
> > +				     uint32_t pack_int_id)
>
> We used to return "1" for failure and "0" for success. Now we're
> reversed: we return NULL for failure and non-zero for success.
>
> So code like:
>
>   if (prepare_midx_pack(...))
> 	return error("yikes");
>
> needs to be updated, but the compiler won't help us because it is happy
> to convert both an int and a pointer into a boolean check.
>
> Should we rename the function to make sure we catch any callers for
> topics in flight?
>
> I'd have thought we could call it nth_midxed_pack(), but that seems to
> exist already, with the caveat that it never prepares the pack, but only
> serves what's in the cache. I wonder if we could simply replace that
> with what prepare_midx_pack() does, but it may be more conservative to
> leave the two separate. So I guess nth_midxed_pack_load() or something.

In general there aren't a ton of in-flight changes in the MIDX code at any given time, so I think we could get away without renaming it. But I don't mind erring on the side of caution here, either.

Thanks, Taylor

Previous: Jeff KingNext: Junio C Hamano
Message 45 of 53 in “packfile: avoid access(3p) calls for missing packs”
  1. packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 16, 2025
  2. Junio C HamanoMay 16, 2025
  3. Jeff KingMay 19, 2025
  4. Junio C HamanoMay 19, 2025
  5. Patrick SteinhardtMay 20, 2025
  6. Jeff KingMay 22, 2025
  7. Taylor BlauMay 23, 2025
  8. Jeff KingMay 23, 2025
  9. 0/2 packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 20, 2025
  10. 1/2 packfile: explain ordering of how we look up auxiliary pack filesPatrick Steinhardt, May 20, 2025
  11. Taylor BlauMay 23, 2025
  12. 2/2 midx: stop repeatedly looking up nonexistent packfilesPatrick Steinhardt, May 20, 2025
  13. Jeff KingMay 22, 2025
  14. Junio C HamanoMay 22, 2025
  15. Jeff KingMay 22, 2025
  16. Junio C HamanoMay 22, 2025
  17. Taylor BlauMay 23, 2025
  18. Jeff KingMay 23, 2025
  19. Taylor BlauMay 23, 2025
  20. 0/5 midx: improve prepare_midx_pack() ergonomicsTaylor Blau, May 25, 2025
  21. 1/5 pack-bitmap.c: fix broken warning() when missing MIDX'd packTaylor Blau, May 25, 2025
  22. Patrick SteinhardtMay 26, 2025
  23. Taylor BlauMay 28, 2025
  24. 2/5 midx-write.c: guard against incremental MIDXs in want_included_pack()Taylor Blau, May 25, 2025
  25. Patrick SteinhardtMay 26, 2025
  26. Taylor BlauMay 28, 2025
  27. 3/5 midx-write.c: simplify fill_packs_from_midx()Taylor Blau, May 25, 2025
  28. Patrick SteinhardtMay 26, 2025
  29. Taylor BlauMay 28, 2025
  30. 4/5 midx-write.c: extract inner loop from fill_packs_from_midx()Taylor Blau, May 25, 2025
  31. 5/5 midx: return a `packed_git` pointer from `prepare_midx_pack()`Taylor Blau, May 25, 2025
  32. Patrick SteinhardtMay 26, 2025
  33. Taylor BlauMay 28, 2025
  34. Patrick SteinhardtMay 28, 2025
  35. 0/4 midx: improve prepare_midx_pack() ergonomicsTaylor Blau, May 28, 2025
  36. 1/4 midx: access pack names through `nth_midxed_pack_name()`Taylor Blau, May 28, 2025
  37. Junio C HamanoMay 29, 2025
  38. Taylor BlauJun 3, 2025
  39. Junio C HamanoMay 29, 2025
  40. Taylor BlauJun 3, 2025
  41. 2/4 midx-write.c: guard against incremental MIDXs in want_included_pack()Taylor Blau, May 28, 2025
  42. 3/4 midx-write.c: extract inner loop from fill_packs_from_midx()Taylor Blau, May 28, 2025
  43. 4/4 midx: return a `packed_git` pointer from `prepare_midx_pack()`Taylor Blau, May 28, 2025
  44. Jeff KingMay 30, 2025
  45. Taylor BlauJun 3, 2025
  46. Junio C HamanoAug 28, 2025
  47. Taylor BlauMay 23, 2025
  48. Jeff KingMay 23, 2025
  49. Junio C HamanoMay 21, 2025
  50. 0/2 packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 28, 2025
  51. 2/2 midx: stop repeatedly looking up nonexistent packfilesPatrick Steinhardt, May 28, 2025
  52. 1/2 packfile: explain ordering of how we look up auxiliary pack filesPatrick Steinhardt, May 28, 2025
  53. Jeff KingMay 30, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.