git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] builtin/repack: don't regenerate MIDX unless needed

From
Patrick Steinhardt <ps@pks.im>
Date
Dec 10, 2025, 09:40 UTC
Message-ID
<aTlABTCWRPNvUEGc@pks.im>
In-Reply-To
<aTjfj45uFl/f3b4K@nand.local>
On Tue, Dec 09, 2025 at 09:48:47PM -0500, Taylor Blau wrote:
Show 27 quoted lines
> On Mon, Dec 08, 2025 at 07:27:15PM +0100, Patrick Steinhardt wrote:
> > Address this issue by introducing a new function that determines whether
> > a rewrite of the MIDX would cause any user-visible changes. This covers
> > the following cases:
> >
> >   - No multi-pack index exists at all.
> >
> >   - The user asked us to write a bitmap, and we don't have any.
> >
> >   - The request preferred pack is different than the one that we have.
> >
> >   - The packfiles covered by the MIDX are changing.
> 
> I can't think of any cases beyond the ones you listed here that would
> require us to regenerate the MIDX. One kind-of-exception here would be:
> 
>     $ git repack [...] --write-midx --write-bitmap-index
>     $ git repack [...] --write-midx
> 
> where the second repack would generate an identical MIDX, but does not
> want to retain a bitmap. That case is already handled in the MIDX
> writing code if you search for "want_bitmap".
> 
> That makes me wonder whether the repack layer is the most appropriate
> one to handle this logic. It seems like write_midx_internal() would
> reasonably be able to detect whether or not the MIDX we have already is
> up-to-date with respect to the given input.

One upside of having it in git-repack(1) is that we need to care about less situations in general as we are operating on a higher level. And because of that we can make more assumptions.

That being said, putting it into `write_midx_internal()` has the benefit that we're of course covering more potential cases where we can avoid a needless rewrite of the MIDX, and that we have better information to decide whether it would be needed or not.

Show 27 quoted lines
> I think that makes some things about your patch easier and other things
> a little harder ;-).
> 
>  - On the "easier" front: while both the MIDX code and the portion of
>    the repack code that drives it receive the same set of packs to
>    include, the MIDX code already has the packs it would compare
>    in a standard format. That would avoid you having to handle ends_with
>    ends_with(include_name, ".idx") and ends_with(existing_name, ".pack")
>    as special cases, which would be nice.
> 
>  - On the "harder" front: when driving MIDX generation with the
>    '--stdin-packs' option, we *don't* load an existing MIDX ever since
>    0c5a62f14bc (midx-write.c: do not read existing MIDX with
>    `packs_to_include`, 2024-06-11).
> 
> I don't think that "harder" one is a show-stopper, though. Commit
> t0c5a62f14bc has enough gory details around how we generate pack IDs and
> various subtle assumptions about how and when we load packs that I am
> very hesitant to recommend changing it given its fragility (though we
> should examine and harden any fragilities within midx-write.c, maybe
> just separately ;-)).
> 
> So I don't think that we should make that change ahead of this patch.
> While you can't rely on being able to read 'ctx.m', I think you could
> load the MIDX belonging to "source" ad-hoc after we have computed the
> packs to fill from the MIDX's perspective, which is right around where
> that want_bitmap code lives.

Yeah, we're already loading the MIDX on-demand because in git-repack(1) as we have closed the object database at the point in time where we're about to write the MIDX. So overall the change is rather easy to make.

Also, now that I see that we already have some short-circuiting conditions in git-multi-pack-index(1) I guess it makes sense to extend those checks. They explicitly don't cover `--stdin-packs` rewrites right now, so adding that check is an obvious improvement.

One thing I'm a bit torn on is whether or not to handle preferred packs in the adjusted logic. We don't do so right now either, so I think I'll drop this for now. I can see an argument that us not handling a changed preferred pack is a bug though, in which case I'm happy to iterate.

Thanks for your feedback!
Patrick
Previous: Taylor BlauNext: Patrick Steinhardt
Message 8 of 16 in “builtin/repack: avoid rewriting up-to-date MIDX”
  1. 0/2 builtin/repack: avoid rewriting up-to-date MIDXPatrick Steinhardt, Dec 8, 2025
  2. 1/2 midx: fix `BUG()` when getting preferred pack without a reverse indexPatrick Steinhardt, Dec 8, 2025
  3. Taylor BlauDec 10, 2025
  4. Patrick SteinhardtDec 10, 2025
  5. Taylor BlauDec 18, 2025
  6. 2/2 builtin/repack: don't regenerate MIDX unless neededPatrick Steinhardt, Dec 8, 2025
  7. Taylor BlauDec 10, 2025
  8. Patrick SteinhardtDec 10, 2025
  9. 0/3 builtin/repack: avoid rewriting up-to-date MIDXPatrick Steinhardt, Dec 10, 2025
  10. 1/3 midx: fix `BUG()` when getting preferred pack without a reverse indexPatrick Steinhardt, Dec 10, 2025
  11. 2/3 midx-write: extract function to test whether MIDX needs updatingPatrick Steinhardt, Dec 10, 2025
  12. 3/3 midx-write: skip rewriting MIDX with `--stdin-packs` unless neededPatrick Steinhardt, Dec 10, 2025
  13. Junio C HamanoDec 11, 2025
  14. Patrick SteinhardtDec 12, 2025
  15. Taylor BlauDec 18, 2025
  16. Patrick SteinhardtDec 19, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.