git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 6/8] midx-write.c: support reading an existing MIDX with `packs_to_include`

From
Taylor Blau <me@ttaylorr.com>
Date
May 29, 2024, 22:46 UTC
Message-ID
<ZlewQ0NXmJTw4hWt@nand.local>
In-Reply-To
<ZlbdxJmIBvgV_VIF@tanuki>
On Wed, May 29, 2024 at 09:48:20AM +0200, Patrick Steinhardt wrote:
Show 15 quoted lines
> On Thu, May 23, 2024 at 12:38:19PM -0400, Taylor Blau wrote:
> > Avoid unconditionally copying all packs from an existing MIDX into a new
> > MIDX by checking that packs added via `fill_packs_from_midx()` don't
> > appear in the `to_include` set, if one was provided.
> >
> > Do so by calling `should_include_pack()` from both `add_pack_to_midx()`
> > and `fill_packs_from_midx()`.
>
> This is missing an explanation why exactly we want that. Is the current
> behaviour a bug? Is it a preparation for a future change? Is this change
> expected to modify any existing behaviour?
>
> Reading through the patch we now unconditionally load the existing MIDX
> when writing a new one, but I'm not exactly sure what the effect of that
> is going to be.

Very fair. The short answer is that this is a prerequisite for the incremental MIDX series that I'm working on. The longer answer is that an incremental MIDX-aware writer needs to be able to consult with the existing MIDX (if one exists) and exclude any objects which already appear in an earlier layer of the MIDX. This is done because we cannot have the same object appear in multiple layers of the MIDX, for reasons that are probably not interesting to this series.

I put a more concise version of the explanation above into this patch which I'll send another round of in v2 of this series shortly.

Show 40 quoted lines
> [snip]
> > diff --git a/midx-write.c b/midx-write.c
> > index 9712ac044f..36ac4ab65b 100644
> > --- a/midx-write.c
> > +++ b/midx-write.c
> > @@ -101,27 +101,13 @@ struct write_midx_context {
> >  };
> >
> >  static int should_include_pack(const struct write_midx_context *ctx,
> > -			       const char *file_name)
> > +			       const char *file_name,
> > +			       int exclude_from_midx)
> >  {
> > -	/*
> > -	 * Note that at most one of ctx->m and ctx->to_include are set,
> > -	 * so we are testing midx_contains_pack() and
> > -	 * string_list_has_string() independently (guarded by the
> > -	 * appropriate NULL checks).
> > -	 *
> > -	 * We could support passing to_include while reusing an existing
> > -	 * MIDX, but don't currently since the reuse process drags
> > -	 * forward all packs from an existing MIDX (without checking
> > -	 * whether or not they appear in the to_include list).
> > -	 *
> > -	 * If we added support for that, these next two conditional
> > -	 * should be performed independently (likely checking
> > -	 * to_include before the existing MIDX).
> > -	 */
> > -	if (ctx->m && midx_contains_pack(ctx->m, file_name))
> > +	if (exclude_from_midx && ctx->m && midx_contains_pack(ctx->m, file_name))
> >  		return 0;
> > -	else if (ctx->to_include &&
> > -		 !string_list_has_string(ctx->to_include, file_name))
> > +	if (ctx->to_include && !string_list_has_string(ctx->to_include,
> > +						       file_name))
>
> The second branch is a no-op change, right? The only change was that you
> converted from `else if` to `if`. I'd propose to either keep this as-is,
> or to do this change in the preceding patch already that introduces this
> function.

It is a no-op, but I would rather keep these separate to keep the previous step a pure code movement rather than introducing any textual changes.

Thanks, Taylor

Previous: Patrick SteinhardtNext: Taylor Blau
Message 17 of 34 in “midx-write: miscellaneous clean-ups for incremental MIDXs”
  1. 0/8 midx-write: miscellaneous clean-ups for incremental MIDXsTaylor Blau, May 23, 2024
  2. 1/8 midx-write.c: tolerate `--preferred-pack` without bitmapsTaylor Blau, May 23, 2024
  3. Patrick SteinhardtMay 29, 2024
  4. Taylor BlauMay 29, 2024
  5. 2/8 midx-write.c: reduce argument count for `get_sorted_entries()`Taylor Blau, May 23, 2024
  6. Patrick SteinhardtMay 29, 2024
  7. Taylor BlauMay 29, 2024
  8. 3/8 midx-write.c: pass `start_pack` to `get_sorted_entries()`Taylor Blau, May 23, 2024
  9. Patrick SteinhardtMay 29, 2024
  10. Taylor BlauMay 29, 2024
  11. 4/8 midx-write.c: extract `should_include_pack()`Taylor Blau, May 23, 2024
  12. Patrick SteinhardtMay 29, 2024
  13. Taylor BlauMay 29, 2024
  14. 5/8 midx-write.c: extract `fill_packs_from_midx()`Taylor Blau, May 23, 2024
  15. 6/8 midx-write.c: support reading an existing MIDX with `packs_to_include`Taylor Blau, May 23, 2024
  16. Patrick SteinhardtMay 29, 2024
  17. Taylor BlauMay 29, 2024
  18. 7/8 midx: replace `get_midx_rev_filename()` with a generic helperTaylor Blau, May 23, 2024
  19. 8/8 pack-bitmap.c: reimplement `midx_bitmap_filename()` with helperTaylor Blau, May 23, 2024
  20. 0/8 midx-write: miscellaneous clean-ups for incremental MIDXsTaylor Blau, May 29, 2024
  21. 1/8 midx-write.c: tolerate `--preferred-pack` without bitmapsTaylor Blau, May 29, 2024
  22. 2/8 midx-write.c: reduce argument count for `get_sorted_entries()`Taylor Blau, May 29, 2024
  23. 3/8 midx-write.c: pass `start_pack` to `compute_sorted_entries()`Taylor Blau, May 29, 2024
  24. Jeff KingMay 30, 2024
  25. 4/8 midx-write.c: extract `should_include_pack()`Taylor Blau, May 29, 2024
  26. 5/8 midx-write.c: extract `fill_packs_from_midx()`Taylor Blau, May 29, 2024
  27. 6/8 midx-write.c: support reading an existing MIDX with `packs_to_include`Taylor Blau, May 29, 2024
  28. Jeff KingMay 30, 2024
  29. 7/8 midx: replace `get_midx_rev_filename()` with a generic helperTaylor Blau, May 29, 2024
  30. Jeff KingMay 30, 2024
  31. 8/8 pack-bitmap.c: reimplement `midx_bitmap_filename()` with helperTaylor Blau, May 29, 2024
  32. Jeff KingMay 30, 2024
  33. Taylor BlauMay 30, 2024
  34. Patrick SteinhardtMay 31, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.