git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/3] midx-write: include packs above custom incremental base

From
Taylor Blau <ttaylorr@openai.com>
Date
Aug 13, 2026, 20:50 UTC
Message-ID
<an4uIQA09rDCwwBp@com-79390>
In-Reply-To
<an2FAWvyfX2LuGsG@pks.im>
On Thu, Aug 13, 2026 at 10:49:05AM +0200, Patrick Steinhardt wrote:
Show 23 quoted lines
> On Fri, Jun 12, 2026 at 04:07:14PM -0400, Taylor Blau wrote:
> > diff --git a/midx-write.c b/midx-write.c
> > index aa438775ebd..c50fdb5c6d1 100644
> > --- a/midx-write.c
> > +++ b/midx-write.c
> > @@ -133,8 +133,17 @@ static uint32_t midx_pack_perm(struct write_midx_context *ctx,
> >  static int should_include_pack(const struct write_midx_context *ctx,
> >  			       const char *file_name)
> >  {
> > +	struct multi_pack_index *m = ctx->m;
> >  	/*
> > -	 * Note that at most one of ctx->m and ctx->to_include are set,
> > +	 * When writing incrementally, ctx->m may contain layers above
> > +	 * the selected base MIDX, which must be included in the new
> > +	 * layer.
> > +	 */
> > +	if (ctx->incremental)
> > +		m = ctx->base_midx;
> > +	/*
> > +	 * Note that at most one of m and ctx->to_include are set,
>
> Is that true? With "--stdin-packs --incremental --base=<foo>" I'd expect
> that we have both set now.

That invariant holds for `ctx->m`j, but not for the local m after the assignment above. With '--stdin-packs', `write_midx_internal()` leaves `ctx->m` unset, but can still set `ctx->base_midx` for an incremental write. Once we assign `m = ctx->base_midx`, both `m` and `ctx->to_include` can indeed be non-NULL.

The filtering still does the right thing: packs covered by the selected base are excluded, and the remaining packs are checked against the stdin list. But the comment is wrong, so I'll fix it.

Show 5 quoted lines
> Okay, previously we were always checking against `ctx->m`, so we
> would exclude packs that are contained in the current MIDX. And that
> includes the case where parts of the current MIDX are supposed to be
> thrown away because we want to write a new layer that excludes all
> layers starting at the base.

On the non- '--stdin-packs' path, yes. With '--stdin-packs', `ctx->m` is `NULL` and the old code already checks `ctx->base_midx`. The problem appears when the previous patch starts honoring '--base' on the ordinary write path. Since `ctx->m` still refers to the entire existing chain, it excludes packs from layers above the selected base.

Show 7 quoted lines
> This is fixed by instead always comparing against the base MIDX in case
> "--incremental" was passed. When the user passes "--base=none" we don't
> have any base, and consequently we'd include all packs. Otherwise, we'll
> exclude all packs that are already covered by our base, but include all
> the other ones.
>
> That feels sensible to me.
Exactly.

Thanks, Taylor

Previous: Patrick SteinhardtNext: Junio C Hamano
Message 11 of 14 in “midx: honor custom bases for incremental writes”
  1. 0/3 midx: honor custom bases for incremental writesTaylor Blau, Jun 12, 2026
  2. 1/3 t5334: expose shared `nth_line()` helperTaylor Blau, Jun 12, 2026
  3. Patrick SteinhardtAug 13, 2026
  4. Taylor BlauAug 13, 2026
  5. 2/3 midx: pass custom '--base' through incremental writesTaylor Blau, Jun 12, 2026
  6. Patrick SteinhardtAug 13, 2026
  7. Taylor BlauAug 13, 2026
  8. Patrick SteinhardtAug 14, 2026
  9. 3/3 midx-write: include packs above custom incremental baseTaylor Blau, Jun 12, 2026
  10. Patrick SteinhardtAug 13, 2026
  11. Taylor BlauAug 13, 2026
  12. Junio C HamanoAug 26, 2026
  13. Taylor BlauAug 26, 2026
  14. Patrick SteinhardtAug 31, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.