Volume XXII, number 279Tuesday, October 6, 2026Latest message 44 minutes ago

The Git List

News and archive of git@vger.kernel.org, since April 2005

patch, 3 partsmidx: honor custom bases for incremental writes

14 messages between Jun 12, 2026 and Aug 31, 2026, from Taylor Blau, Patrick Steinhardt, Junio C Hamano.

Plain Markdown or JSON for tools and agents. Diffs are folded; open one to read it.

Taylor BlauJun 12, 2026, 20:07 UTC on lore

SZEDER noticed[1] that t5334 was trying to call `nth_line()`, despite that helper living only in t5335.

Fixing that should have made the test exercise `git multi-pack-index write --incremental --base=...`. Instead, it uncovered another wrinkle, which is that the normal MIDX write path parsed "--bases" without actually passing it down to the MIDX writer.

This short series fixes both issues. It is structured as follows:
 * The first patch moves `nth_line()` to lib-midx.sh so that t5334 and
   t5335 use the same helper.
 * The second patch threads the parsed `--base` value through
   `write_midx_file()`, and consequently marks two t5334 cases as known
   breakages.
 * The final patch fixes the pack inclusion check and marks the tests
   successful again.

The result is that `--base=none` and `--base=<hash>` now correctly produce detached incremental layers that include any packs above the selected base, preserving reachability closure for bitmaps.

Thanks in advance for your review!
[1]: https://lore.kernel.org/git/aiuaf3fKJ6kIITrf@szeder.dev/
Taylor Blau (3):
  t5334: expose shared `nth_line()` helper
  midx: pass custom '--base' through incremental writes
  midx-write: include packs above custom incremental base
 builtin/multi-pack-index.c              |  3 ++-
 builtin/repack.c                        |  2 +-
 midx-write.c                            | 18 +++++++++++++-----
 midx.h                                  |  2 +-
 t/lib-midx.sh                           |  6 ++++++
 t/t5334-incremental-multi-pack-index.sh | 20 +++++++++++++++++---
 t/t5335-compact-multi-pack-index.sh     |  7 +------
 7 files changed, 41 insertions(+), 17 deletions(-)
base-commit: 3e65291872de10c3f0bf05ea8c24187e7a71ebf0
-- 
2.55.0.rc0.3.g7bf7c87b605
Taylor BlauJun 12, 2026, 20:07 UTC in reply to Taylor Blau on lore

[PATCH 1/3] t5334: expose shared `nth_line()` helper

Since commit 0cd2255e64b (midx: support custom `--base` for incremental MIDX writes, 2026-05-19), t5334 has referred to a non-existent helper function 'nth_line', which is defined in t5335, but not here.

Move the helper to lib-midx.sh so that both tests can use the same implementation. Ensure likewise that `nth_line()` remains visible from within t5335 by sourcing lib-midx.sh there appropriately.

Curiously, t5334 passes both before and after this change. Before this change, the failed command substitution leaves '--base' with an empty value, and after this change, the custom base value is still ignored by the normal incremental write path. The following commits will explain and address that behavior.

Noticed-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 t/lib-midx.sh                       | 6 ++++++
 t/t5335-compact-multi-pack-index.sh | 7 +------
 2 files changed, 7 insertions(+), 6 deletions(-)
Show changes to 2 files +7 −6

t/lib-midx.sh, t/t5335-compact-multi-pack-index.sh

diff --git a/t/lib-midx.sh b/t/lib-midx.sh
index e38c609604c..b522dbdb0f4 100644
--- a/t/lib-midx.sh
+++ b/t/lib-midx.sh
@@ -34,3 +34,9 @@ compare_results_with_midx () {
 		midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted
 	'
 }
+
+nth_line() {
+	local n="$1"
+	shift
+	awk "NR==$n" "$@"
+}
diff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh
index ec1dafe89fc..6a4b799b9c9 100755
--- a/t/t5335-compact-multi-pack-index.sh
+++ b/t/t5335-compact-multi-pack-index.sh
@@ -3,6 +3,7 @@
 test_description='multi-pack-index compaction'
 
 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-midx.sh
 
 GIT_TEST_MULTI_PACK_INDEX=0
 GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
@@ -13,12 +14,6 @@ packdir=$objdir/pack
 midxdir=$packdir/multi-pack-index.d
 midx_chain=$midxdir/multi-pack-index-chain
 
-nth_line() {
-	local n="$1"
-	shift
-	awk "NR==$n" "$@"
-}
-
 write_packs () {
 	for c in "$@"
 	do
-- 
2.55.0.rc0.3.g7bf7c87b605
Taylor BlauJun 12, 2026, 20:07 UTC in reply to Taylor Blau on lore

[PATCH 2/3] midx: pass custom '--base' through incremental writes

The 'multi-pack-index' builtin parses '--base' for incremental writes, but the normal write path does not pass that value through to `write_midx_file()`.

As a result, something like:
    $ git multi-pack-index write --incremental --base=<base>

behaves as if no custom base had been given (unless the caller used the '--stdin-packs' path).

Thread the parsed base through `write_midx_file()`, and update the repack caller to pass NULL for the new argument where no custom base selection is needed.

This exposes a pre-existing problem in incremental writes with custom bases: the writer skips packs from the full existing MIDX chain, even when the caller selected an older base or no base at all.

The affected t5334 cases fail while trying to write MIDX bitmaps. The detached layer omits packs above the selected base, and thus the resulting MIDX does not have a reachability closure, making it impossible to generate reachability bitmaps.

Mark those tests as expected failures accordingly. The following commit will fix the broken behavior and restore these tests.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 builtin/multi-pack-index.c              |  3 ++-
 builtin/repack.c                        |  2 +-
 midx-write.c                            |  2 ++
 midx.h                                  |  2 +-
 t/t5334-incremental-multi-pack-index.sh | 24 +++++++++++++++++++-----
 5 files changed, 25 insertions(+), 8 deletions(-)
Show changes to 5 files +25 −8

builtin/multi-pack-index.c, builtin/repack.c, midx-write.c, midx.h, t/t5334-incremental-multi-pack-index.sh

diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
index 00ffb36394d..949bfa796b2 100644
--- a/builtin/multi-pack-index.c
+++ b/builtin/multi-pack-index.c
@@ -224,7 +224,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,
 	}
 
 	ret = write_midx_file(source, opts.preferred_pack,
-			      opts.refs_snapshot, opts.flags);
+			      opts.refs_snapshot, opts.incremental_base,
+			      opts.flags);
 
 	free(opts.refs_snapshot);
 	return ret;
diff --git a/builtin/repack.c b/builtin/repack.c
index 1524a9c13ad..0092a72a996 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -629,7 +629,7 @@ int cmd_repack(int argc,
 		unsigned flags = 0;
 		if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))
 			flags |= MIDX_WRITE_INCREMENTAL;
-		write_midx_file(existing.source, NULL, NULL, flags);
+		write_midx_file(existing.source, NULL, NULL, NULL, flags);
 	}
 
 cleanup:
diff --git a/midx-write.c b/midx-write.c
index 561e9eedc0e..aa438775ebd 100644
--- a/midx-write.c
+++ b/midx-write.c
@@ -1850,12 +1850,14 @@ static int write_midx_internal(struct write_midx_opts *opts)
 int write_midx_file(struct odb_source *source,
 		    const char *preferred_pack_name,
 		    const char *refs_snapshot,
+		    const char *incremental_base,
 		    unsigned flags)
 {
 	struct write_midx_opts opts = {
 		.source = source,
 		.preferred_pack_name = preferred_pack_name,
 		.refs_snapshot = refs_snapshot,
+		.incremental_base = incremental_base,
 		.flags = flags,
 	};
 
diff --git a/midx.h b/midx.h
index 63853a03a47..92ed29d913d 100644
--- a/midx.h
+++ b/midx.h
@@ -131,7 +131,7 @@ int prepare_multi_pack_index_one(struct odb_source *source);
  */
 int write_midx_file(struct odb_source *source,
 		    const char *preferred_pack_name, const char *refs_snapshot,
-		    unsigned flags);
+		    const char *incremental_base, unsigned flags);
 int write_midx_file_only(struct odb_source *source,
 			 struct string_list *packs_to_include,
 			 const char *preferred_pack_name,
diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh
index 68a103d13d2..69e96bf8d93 100755
--- a/t/t5334-incremental-multi-pack-index.sh
+++ b/t/t5334-incremental-multi-pack-index.sh
@@ -119,7 +119,7 @@ test_expect_success 'write MIDX layer with --base without --no-write-chain-file'
 	test_grep "cannot use --base without --no-write-chain-file" err
 '
 
-test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '
+test_expect_failure 'write MIDX layer with --base=none and --no-write-chain-file' '
 	test_commit base-none &&
 	git repack -d &&
 
@@ -128,19 +128,33 @@ test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file
 		--no-write-chain-file --base=none)" &&
 
 	test_cmp "$midx_chain.bak" "$midx_chain" &&
-	test_path_is_file "$midxdir/multi-pack-index-$layer.midx"
+	test_path_is_file "$midxdir/multi-pack-index-$layer.midx" &&
+
+	echo "$layer" >"$midx_chain" &&
+	test-tool read-midx --show-objects "$objdir" "$layer" >midx.objects &&
+	test_grep "^$(git rev-parse 2.2) " midx.objects &&
+	cp "$midx_chain.bak" "$midx_chain"
 '
 
-test_expect_success 'write MIDX layer with --base=<hash> and --no-write-chain-file' '
+test_expect_failure 'write MIDX layer with --base=<hash> and --no-write-chain-file' '
 	test_commit base-hash &&
 	git repack -d &&
 
 	cp "$midx_chain" "$midx_chain.bak" &&
+	base="$(nth_line 1 "$midx_chain")" &&
 	layer="$(git multi-pack-index write --bitmap --incremental \
-		--no-write-chain-file --base="$(nth_line 1 "$midx_chain")")" &&
+		--no-write-chain-file --base="$base")" &&
 
 	test_cmp "$midx_chain.bak" "$midx_chain" &&
-	test_path_is_file "$midxdir/multi-pack-index-$layer.midx"
+	test_path_is_file "$midxdir/multi-pack-index-$layer.midx" &&
+
+	{
+		echo "$base" &&
+		echo "$layer"
+	} >"$midx_chain" &&
+	test-tool read-midx --show-objects "$objdir" "$layer" >midx.objects &&
+	test_grep "^$(git rev-parse 2.2) " midx.objects &&
+	cp "$midx_chain.bak" "$midx_chain"
 '
 
 for reuse in false single multi
-- 
2.55.0.rc0.3.g7bf7c87b605
Taylor BlauJun 12, 2026, 20:07 UTC in reply to Taylor Blau on lore

[PATCH 3/3] midx-write: include packs above custom incremental base

The previous commit made '--base' take effect on the normal incremental write path, which exposed an existing assumption in our helper function `should_include_pack()`, which is that any pack already present in `ctx->m` was skipped.

That is only correct for non-incremental writes. For incremental writes, `ctx->base_midx` is the boundary that should be excluded from the new layer. If the caller selects an older base, or no base at all, then packs from layers above that base have to be included in the detached layer so that its bitmap has reachability closure.

Teach `should_include_pack()` to choose the MIDX used for pack exclusion based on whether or not we are performing an incremental write. When doing so, use `ctx->base_midx`, and use `ctx->m` otherwise.

The t5334 cases from the previous commit can now be marked as successful.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 midx-write.c                            | 16 +++++++++++-----
 t/t5334-incremental-multi-pack-index.sh |  4 ++--
 2 files changed, 13 insertions(+), 7 deletions(-)
Show changes to 2 files +13 −7

midx-write.c, t/t5334-incremental-multi-pack-index.sh

diff --git a/midx-write.c b/midx-write.c
index aa438775ebd..c50fdb5c6d1 100644
--- a/midx-write.c
+++ b/midx-write.c
@@ -133,8 +133,17 @@ static uint32_t midx_pack_perm(struct write_midx_context *ctx,
 static int should_include_pack(const struct write_midx_context *ctx,
 			       const char *file_name)
 {
+	struct multi_pack_index *m = ctx->m;
 	/*
-	 * Note that at most one of ctx->m and ctx->to_include are set,
+	 * When writing incrementally, ctx->m may contain layers above
+	 * the selected base MIDX, which must be included in the new
+	 * layer.
+	 */
+	if (ctx->incremental)
+		m = ctx->base_midx;
+
+	/*
+	 * Note that at most one of m and ctx->to_include are set,
 	 * so we are testing midx_contains_pack() and
 	 * string_list_has_string() independently (guarded by the
 	 * appropriate NULL checks).
@@ -148,10 +157,7 @@ static int should_include_pack(const struct write_midx_context *ctx,
 	 * should be performed independently (likely checking
 	 * to_include before the existing MIDX).
 	 */
-	if (ctx->m && midx_contains_pack(ctx->m, file_name))
-		return 0;
-	else if (ctx->base_midx && midx_contains_pack(ctx->base_midx,
-						      file_name))
+	if (m && midx_contains_pack(m, file_name))
 		return 0;
 	else if (ctx->to_include &&
 		 !string_list_has_string(ctx->to_include, file_name))
diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh
index 69e96bf8d93..84ff6120978 100755
--- a/t/t5334-incremental-multi-pack-index.sh
+++ b/t/t5334-incremental-multi-pack-index.sh
@@ -119,7 +119,7 @@ test_expect_success 'write MIDX layer with --base without --no-write-chain-file'
 	test_grep "cannot use --base without --no-write-chain-file" err
 '
 
-test_expect_failure 'write MIDX layer with --base=none and --no-write-chain-file' '
+test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '
 	test_commit base-none &&
 	git repack -d &&
 
@@ -136,7 +136,7 @@ test_expect_failure 'write MIDX layer with --base=none and --no-write-chain-file
 	cp "$midx_chain.bak" "$midx_chain"
 '
 
-test_expect_failure 'write MIDX layer with --base=<hash> and --no-write-chain-file' '
+test_expect_success 'write MIDX layer with --base=<hash> and --no-write-chain-file' '
 	test_commit base-hash &&
 	git repack -d &&
 
-- 
2.55.0.rc0.3.g7bf7c87b605
Patrick SteinhardtAug 13, 2026, 08:48 UTC in reply to Taylor Blau on lore

Re: [PATCH 1/3] t5334: expose shared `nth_line()` helper

On Fri, Jun 12, 2026 at 04:07:08PM -0400, Taylor Blau wrote:
Show 14 quoted lines
> diff --git a/t/lib-midx.sh b/t/lib-midx.sh
> index e38c609604c..b522dbdb0f4 100644
> --- a/t/lib-midx.sh
> +++ b/t/lib-midx.sh
> @@ -34,3 +34,9 @@ compare_results_with_midx () {
>  		midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted
>  	'
>  }
> +
> +nth_line() {
> +	local n="$1"
> +	shift
> +	awk "NR==$n" "$@"
> +}

It feels a bit weird to have such a general function in "lib-midx.sh", but so be it.

Patrick
Patrick SteinhardtAug 13, 2026, 08:49 UTC in reply to Taylor Blau on lore

Re: [PATCH 2/3] midx: pass custom '--base' through incremental writes

On Fri, Jun 12, 2026 at 04:07:11PM -0400, Taylor Blau wrote:
Show 10 quoted lines
> The 'multi-pack-index' builtin parses '--base' for incremental writes,
> but the normal write path does not pass that value through to
> `write_midx_file()`.
> 
> As a result, something like:
> 
>     $ git multi-pack-index write --incremental --base=<base>
> 
> behaves as if no custom base had been given (unless the caller used the
> '--stdin-packs' path).

I'm a bit confused. Is the "normal" write path the one that generates a completely new, full MIDX? I assume not, and that you use "normal" to discern between whether or not we pass "--stdin-packs"? I think that could be made a bit more explicit.

*goes looking into the code* Yeah, seems like the distinction indeed is whether "--stdin-packs" was passed in the first place.

Show 7 quoted lines
> Thread the parsed base through `write_midx_file()`, and update the
> repack caller to pass NULL for the new argument where no custom base
> selection is needed.
> 
> This exposes a pre-existing problem in incremental writes with custom
> bases: the writer skips packs from the full existing MIDX chain, even
> when the caller selected an older base or no base at all.

So as the "normal" write path didn't honor this option at all, I assume this bug here then refers to "--stdin-packs" being broken?

Show 7 quoted lines
> The affected t5334 cases fail while trying to write MIDX bitmaps. The
> detached layer omits packs above the selected base, and thus the
> resulting MIDX does not have a reachability closure, making it
> impossible to generate reachability bitmaps.
> 
> Mark those tests as expected failures accordingly. The following commit
> will fix the broken behavior and restore these tests.
Okay.
Show 14 quoted lines
> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
> index 00ffb36394d..949bfa796b2 100644
> --- a/builtin/multi-pack-index.c
> +++ b/builtin/multi-pack-index.c
> @@ -224,7 +224,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,
>  	}
>  
>  	ret = write_midx_file(source, opts.preferred_pack,
> -			      opts.refs_snapshot, opts.flags);
> +			      opts.refs_snapshot, opts.incremental_base,
> +			      opts.flags);
>  
>  	free(opts.refs_snapshot);
>  	return ret;

Previously we only passed the base to `write_midx_file_only()`, which is what we use with "--stdin-packs". Here we now update the normal write path to use the incremental base, too.

Show 18 quoted lines
> diff --git a/midx-write.c b/midx-write.c
> index 561e9eedc0e..aa438775ebd 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -1850,12 +1850,14 @@ static int write_midx_internal(struct write_midx_opts *opts)
>  int write_midx_file(struct odb_source *source,
>  		    const char *preferred_pack_name,
>  		    const char *refs_snapshot,
> +		    const char *incremental_base,
>  		    unsigned flags)
>  {
>  	struct write_midx_opts opts = {
>  		.source = source,
>  		.preferred_pack_name = preferred_pack_name,
>  		.refs_snapshot = refs_snapshot,
> +		.incremental_base = incremental_base,
>  		.flags = flags,
>  	};

I was wondering whether there needs to be error checking somewhere so that we only accept an incremental base in case MIDX_WRITE_INCREMENTAL is set. I couldn't find any.

Show 13 quoted lines
> diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh
> index 68a103d13d2..69e96bf8d93 100755
> --- a/t/t5334-incremental-multi-pack-index.sh
> +++ b/t/t5334-incremental-multi-pack-index.sh
> @@ -119,7 +119,7 @@ test_expect_success 'write MIDX layer with --base without --no-write-chain-file'
>  	test_grep "cannot use --base without --no-write-chain-file" err
>  '
>  
> -test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '
> +test_expect_failure 'write MIDX layer with --base=none and --no-write-chain-file' '
>  	test_commit base-none &&
>  	git repack -d &&
>  

Okay. If I understand correctly, the expectation here would be that we generate a complete MIDX as we don't select any base at all. But we don't, and instead we base our incremental MIDX on top of the newest layer by accident.

Show 12 quoted lines
> @@ -128,19 +128,33 @@ test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file
>  		--no-write-chain-file --base=none)" &&
>  
>  	test_cmp "$midx_chain.bak" "$midx_chain" &&
> -	test_path_is_file "$midxdir/multi-pack-index-$layer.midx"
> +	test_path_is_file "$midxdir/multi-pack-index-$layer.midx" &&
> +
> +	echo "$layer" >"$midx_chain" &&
> +	test-tool read-midx --show-objects "$objdir" "$layer" >midx.objects &&
> +	test_grep "^$(git rev-parse 2.2) " midx.objects &&
> +	cp "$midx_chain.bak" "$midx_chain"
>  '

Would it make sense to also test for an object from the first MIDX layer to be included? Otherwise we don't really assert that all layers are included in the new MIDX.

Patrick
Patrick SteinhardtAug 13, 2026, 08:49 UTC in reply to Taylor Blau on lore

Re: [PATCH 3/3] midx-write: include packs above custom incremental base

On Fri, Jun 12, 2026 at 04:07:14PM -0400, Taylor Blau wrote:
Show 19 quoted lines
> diff --git a/midx-write.c b/midx-write.c
> index aa438775ebd..c50fdb5c6d1 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -133,8 +133,17 @@ static uint32_t midx_pack_perm(struct write_midx_context *ctx,
>  static int should_include_pack(const struct write_midx_context *ctx,
>  			       const char *file_name)
>  {
> +	struct multi_pack_index *m = ctx->m;
>  	/*
> -	 * Note that at most one of ctx->m and ctx->to_include are set,
> +	 * When writing incrementally, ctx->m may contain layers above
> +	 * the selected base MIDX, which must be included in the new
> +	 * layer.
> +	 */
> +	if (ctx->incremental)
> +		m = ctx->base_midx;
> +	/*
> +	 * Note that at most one of m and ctx->to_include are set,

Is that true? With "--stdin-packs --incremental --base=<foo>" I'd expect that we have both set now.

Show 10 quoted lines
> @@ -148,10 +157,7 @@ static int should_include_pack(const struct write_midx_context *ctx,
>  	 * should be performed independently (likely checking
>  	 * to_include before the existing MIDX).
>  	 */
> -	if (ctx->m && midx_contains_pack(ctx->m, file_name))
> -		return 0;
> -	else if (ctx->base_midx && midx_contains_pack(ctx->base_midx,
> -						      file_name))
> +	if (m && midx_contains_pack(m, file_name))
>  		return 0;

Okay, previously we were always checking against `ctx->m`, so we would exclude packs that are contained in the current MIDX. And that includes the case where parts of the current MIDX are supposed to be thrown away because we want to write a new layer that excludes all layers starting at the base.

This is fixed by instead always comparing against the base MIDX in case "--incremental" was passed. When the user passes "--base=none" we don't have any base, and consequently we'd include all packs. Otherwise, we'll exclude all packs that are already covered by our base, but include all the other ones.

That feels sensible to me.
Show 23 quoted lines
>  	else if (ctx->to_include &&
>  		 !string_list_has_string(ctx->to_include, file_name))
> diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh
> index 69e96bf8d93..84ff6120978 100755
> --- a/t/t5334-incremental-multi-pack-index.sh
> +++ b/t/t5334-incremental-multi-pack-index.sh
> @@ -119,7 +119,7 @@ test_expect_success 'write MIDX layer with --base without --no-write-chain-file'
>  	test_grep "cannot use --base without --no-write-chain-file" err
>  '
>  
> -test_expect_failure 'write MIDX layer with --base=none and --no-write-chain-file' '
> +test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '
>  	test_commit base-none &&
>  	git repack -d &&
>  
> @@ -136,7 +136,7 @@ test_expect_failure 'write MIDX layer with --base=none and --no-write-chain-file
>  	cp "$midx_chain.bak" "$midx_chain"
>  '
>  
> -test_expect_failure 'write MIDX layer with --base=<hash> and --no-write-chain-file' '
> +test_expect_success 'write MIDX layer with --base=<hash> and --no-write-chain-file' '
>  	test_commit base-hash &&
>  	git repack -d &&
And those two tests pass now.
Thanks!
Patrick
Taylor BlauAug 13, 2026, 19:28 UTC in reply to Patrick Steinhardt on lore

Re: [PATCH 1/3] t5334: expose shared `nth_line()` helper

On Thu, Aug 13, 2026 at 10:48:49AM +0200, Patrick Steinhardt wrote:
Show 18 quoted lines
> On Fri, Jun 12, 2026 at 04:07:08PM -0400, Taylor Blau wrote:
> > diff --git a/t/lib-midx.sh b/t/lib-midx.sh
> > index e38c609604c..b522dbdb0f4 100644
> > --- a/t/lib-midx.sh
> > +++ b/t/lib-midx.sh
> > @@ -34,3 +34,9 @@ compare_results_with_midx () {
> >  		midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted
> >  	'
> >  }
> > +
> > +nth_line() {
> > +	local n="$1"
> > +	shift
> > +	awk "NR==$n" "$@"
> > +}
>
> It feels a bit weird to have such a general function in "lib-midx.sh",
> but so be it.

Yeah, I agree. It mostly just felt weird to write '| awk "NR==$n"' a bunch of times.

Thanks, Taylor

Taylor BlauAug 13, 2026, 20:30 UTC in reply to Patrick Steinhardt on lore

Re: [PATCH 2/3] midx: pass custom '--base' through incremental writes

On Thu, Aug 13, 2026 at 10:49:00AM +0200, Patrick Steinhardt wrote:
Show 10 quoted lines
> > Thread the parsed base through `write_midx_file()`, and update the
> > repack caller to pass NULL for the new argument where no custom base
> > selection is needed.
> >
> > This exposes a pre-existing problem in incremental writes with custom
> > bases: the writer skips packs from the full existing MIDX chain, even
> > when the caller selected an older base or no base at all.
>
> So as the "normal" write path didn't honor this option at all, I assume
> this bug here then refers to "--stdin-packs" being broken?
Yeah, that's right.
Show 16 quoted lines
> > @@ -128,19 +128,33 @@ test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file
> >  		--no-write-chain-file --base=none)" &&
> >
> >  	test_cmp "$midx_chain.bak" "$midx_chain" &&
> > -	test_path_is_file "$midxdir/multi-pack-index-$layer.midx"
> > +	test_path_is_file "$midxdir/multi-pack-index-$layer.midx" &&
> > +
> > +	echo "$layer" >"$midx_chain" &&
> > +	test-tool read-midx --show-objects "$objdir" "$layer" >midx.objects &&
> > +	test_grep "^$(git rev-parse 2.2) " midx.objects &&
> > +	cp "$midx_chain.bak" "$midx_chain"
> >  '
>
> Would it make sense to also test for an object from the first MIDX layer
> to be included? Otherwise we don't really assert that all layers are
> included in the new MIDX.

I don't think that is necessary in this case, but let me know if I am missing something below.

The new layer is written with '--bitmap', and '--base=none' means that there is no base layer from which the bitmap can inherit objects. Since 1.2 is an ancestor of 2.2, writing a bitmap for the new layer already requires that it contain 1.2 and the rest of its reachable history. Otherwise bitmap generation would fail with the missing-closure error before we reached the assertion.

Checking 2.2 confirms that an object from the old tip was pulled into the new layer; the successful bitmap write already establishes that its objects from the earlier layer were pulled in, too.

Thanks, Taylor

Taylor BlauAug 13, 2026, 20:50 UTC in reply to Patrick Steinhardt on lore

Re: [PATCH 3/3] midx-write: include packs above custom incremental base

On Thu, Aug 13, 2026 at 10:49:05AM +0200, Patrick Steinhardt wrote:
Show 23 quoted lines
> On Fri, Jun 12, 2026 at 04:07:14PM -0400, Taylor Blau wrote:
> > diff --git a/midx-write.c b/midx-write.c
> > index aa438775ebd..c50fdb5c6d1 100644
> > --- a/midx-write.c
> > +++ b/midx-write.c
> > @@ -133,8 +133,17 @@ static uint32_t midx_pack_perm(struct write_midx_context *ctx,
> >  static int should_include_pack(const struct write_midx_context *ctx,
> >  			       const char *file_name)
> >  {
> > +	struct multi_pack_index *m = ctx->m;
> >  	/*
> > -	 * Note that at most one of ctx->m and ctx->to_include are set,
> > +	 * When writing incrementally, ctx->m may contain layers above
> > +	 * the selected base MIDX, which must be included in the new
> > +	 * layer.
> > +	 */
> > +	if (ctx->incremental)
> > +		m = ctx->base_midx;
> > +	/*
> > +	 * Note that at most one of m and ctx->to_include are set,
>
> Is that true? With "--stdin-packs --incremental --base=<foo>" I'd expect
> that we have both set now.

That invariant holds for `ctx->m`j, but not for the local m after the assignment above. With '--stdin-packs', `write_midx_internal()` leaves `ctx->m` unset, but can still set `ctx->base_midx` for an incremental write. Once we assign `m = ctx->base_midx`, both `m` and `ctx->to_include` can indeed be non-NULL.

The filtering still does the right thing: packs covered by the selected base are excluded, and the remaining packs are checked against the stdin list. But the comment is wrong, so I'll fix it.

Show 5 quoted lines
> Okay, previously we were always checking against `ctx->m`, so we
> would exclude packs that are contained in the current MIDX. And that
> includes the case where parts of the current MIDX are supposed to be
> thrown away because we want to write a new layer that excludes all
> layers starting at the base.

On the non- '--stdin-packs' path, yes. With '--stdin-packs', `ctx->m` is `NULL` and the old code already checks `ctx->base_midx`. The problem appears when the previous patch starts honoring '--base' on the ordinary write path. Since `ctx->m` still refers to the entire existing chain, it excludes packs from layers above the selected base.

Show 7 quoted lines
> This is fixed by instead always comparing against the base MIDX in case
> "--incremental" was passed. When the user passes "--base=none" we don't
> have any base, and consequently we'd include all packs. Otherwise, we'll
> exclude all packs that are already covered by our base, but include all
> the other ones.
>
> That feels sensible to me.
Exactly.

Thanks, Taylor

Patrick SteinhardtAug 14, 2026, 07:27 UTC in reply to Taylor Blau on lore

Re: [PATCH 2/3] midx: pass custom '--base' through incremental writes

On Thu, Aug 13, 2026 at 03:30:53PM -0500, Taylor Blau wrote:
Show 31 quoted lines
> On Thu, Aug 13, 2026 at 10:49:00AM +0200, Patrick Steinhardt wrote:
> > > @@ -128,19 +128,33 @@ test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file
> > >  		--no-write-chain-file --base=none)" &&
> > >
> > >  	test_cmp "$midx_chain.bak" "$midx_chain" &&
> > > -	test_path_is_file "$midxdir/multi-pack-index-$layer.midx"
> > > +	test_path_is_file "$midxdir/multi-pack-index-$layer.midx" &&
> > > +
> > > +	echo "$layer" >"$midx_chain" &&
> > > +	test-tool read-midx --show-objects "$objdir" "$layer" >midx.objects &&
> > > +	test_grep "^$(git rev-parse 2.2) " midx.objects &&
> > > +	cp "$midx_chain.bak" "$midx_chain"
> > >  '
> >
> > Would it make sense to also test for an object from the first MIDX layer
> > to be included? Otherwise we don't really assert that all layers are
> > included in the new MIDX.
> 
> I don't think that is necessary in this case, but let me know if I am
> missing something below.
> 
> The new layer is written with '--bitmap', and '--base=none' means that
> there is no base layer from which the bitmap can inherit objects. Since
> 1.2 is an ancestor of 2.2, writing a bitmap for the new layer already
> requires that it contain 1.2 and the rest of its reachable history.
> Otherwise bitmap generation would fail with the missing-closure error
> before we reached the assertion.
> 
> Checking 2.2 confirms that an object from the old tip was pulled into
> the new layer; the successful bitmap write already establishes that its
> objects from the earlier layer were pulled in, too.

I think that's a bit roundabout, as it simply tells us that the bitmap was generated correctly, but not that the MIDX contains the objects. It of course should if the bitmap was generated properly, but I would have preferred if we verified the property directly.

Anyway, this is not a huge concern, more of a nitpick. Thanks!
Patrick
Junio C HamanoAug 26, 2026, 21:37 UTC in reply to Taylor Blau on lore

Re: [PATCH 3/3] midx-write: include packs above custom incremental base

Taylor Blau <ttaylorr@openai.com> writes:
Show 5 quoted lines
> `ctx->to_include` can indeed be non-NULL.
> ...
> The filtering still does the right thing: packs covered by the selected
> base are excluded, and the remaining packs are checked against the stdin
> list. But the comment is wrong, so I'll fix it.
Has anything happened since we saw this comment on Aug 13th?
Thanks.
Taylor BlauAug 26, 2026, 23:38 UTC in reply to Junio C Hamano on lore

Re: [PATCH 3/3] midx-write: include packs above custom incremental base

On Wed, Aug 26, 2026 at 02:37:23PM -0700, Junio C Hamano wrote:
Show 9 quoted lines
> Taylor Blau <ttaylorr@openai.com> writes:
>
> > `ctx->to_include` can indeed be non-NULL.
> > ...
> > The filtering still does the right thing: packs covered by the selected
> > base are excluded, and the remaining packs are checked against the stdin
> > list. But the comment is wrong, so I'll fix it.
>
> Has anything happened since we saw this comment on Aug 13th?
Not until you sent this message ;-).

I had a small reroll prepped that I had meant to send a couple of weeks ago but never got around to doing so. When I looked at it just now, I found that I wasn't quite satisfied with the range-diff in that the resulting block comment was somewhat confusing.

Instead of sending a new round immediately, let me instead share the comment that I wrote instead. Patrick (or others): does this comment seem clear, or do you think there are ways to tighten it up further?

--- 8< ---
Show changes to midx-write.c +25 −9
diff --git a/midx-write.c b/midx-write.c
index 66da608370..ff94076104 100644
--- a/midx-write.c
+++ b/midx-write.c
@@ -143,15 +143,31 @@ static int should_include_pack(const struct write_midx_context *ctx,
 		m = ctx->base_midx;

 	/*
-	 * Note that m and ctx->to_include may both be set,
-	 * so we are testing midx_contains_pack() and
-	 * string_list_has_string() independently (guarded by the
-	 * appropriate NULL checks).
-	 *
-	 * We could support passing to_include while reusing an existing
-	 * MIDX, but don't currently since the reuse process drags
-	 * forward all packs from an existing MIDX (without checking
-	 * whether or not they appear in the to_include list).
+	 * Note that it is OK for both ctx->base_midx and
+	 * ctx->to_include may both be non-NULL, but at most one of
+	 * ctx->m and ctx->to_include may be non-NULL.
+	 *
+	 * When ctx->m is NULL we are writing a new MIDX without reusing
+	 * any packs from the previous layer(s). In that case, we care
+	 * that both:
+	 *
+	 *   - the new layer's base MIDX (ctx->base_midx) does not
+	 *     already contain the pack we are considering, or the new
+	 *     layer has no base (i.e., it is a non-incremental MIDX)
+	 *
+	 *   - the pack appears in ctx->to_include, or ctx->to_include
+	 *     is NULL, meaning that we can include any pack provided
+	 *     the above condition is met.
+	 *
+	 * When ctx->m is non-NULL, we are writing a new MIDX that will
+	 * subsume ctx->m and thus includes its packs. In this case, we
+	 * could support respecting ctx->to_include, but currently
+	 * don't.
+	 *
+	 * The only caller of this function which permits
+	 * ctx->to_include being non-NULL restricts setting ctx->m when
+	 * this is the case. So in this setting it is impossible that
+	 * both will be non-NULL.
 	 *
 	 * If we added support for that, these next two conditional
 	 * should be performed independently (likely checking
--- >8 ---

Thanks,
Taylor
Patrick SteinhardtAug 31, 2026, 06:12 UTC in reply to Taylor Blau on lore

Re: [PATCH 3/3] midx-write: include packs above custom incremental base

On Wed, Aug 26, 2026 at 06:38:15PM -0500, Taylor Blau wrote:
Show 43 quoted lines
> On Wed, Aug 26, 2026 at 02:37:23PM -0700, Junio C Hamano wrote:
> > Taylor Blau <ttaylorr@openai.com> writes:
> >
> > > `ctx->to_include` can indeed be non-NULL.
> > > ...
> > > The filtering still does the right thing: packs covered by the selected
> > > base are excluded, and the remaining packs are checked against the stdin
> > > list. But the comment is wrong, so I'll fix it.
> >
> > Has anything happened since we saw this comment on Aug 13th?
> 
> Not until you sent this message ;-).
> 
> I had a small reroll prepped that I had meant to send a couple of weeks
> ago but never got around to doing so. When I looked at it just now, I
> found that I wasn't quite satisfied with the range-diff in that the
> resulting block comment was somewhat confusing.
> 
> Instead of sending a new round immediately, let me instead share the
> comment that I wrote instead. Patrick (or others): does this comment
> seem clear, or do you think there are ways to tighten it up further?
> 
> --- 8< ---
> diff --git a/midx-write.c b/midx-write.c
> index 66da608370..ff94076104 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -143,15 +143,31 @@ static int should_include_pack(const struct write_midx_context *ctx,
>  		m = ctx->base_midx;
> 
>  	/*
> -	 * Note that m and ctx->to_include may both be set,
> -	 * so we are testing midx_contains_pack() and
> -	 * string_list_has_string() independently (guarded by the
> -	 * appropriate NULL checks).
> -	 *
> -	 * We could support passing to_include while reusing an existing
> -	 * MIDX, but don't currently since the reuse process drags
> -	 * forward all packs from an existing MIDX (without checking
> -	 * whether or not they appear in the to_include list).
> +	 * Note that it is OK for both ctx->base_midx and
> +	 * ctx->to_include may both be non-NULL, but at most one of
> +	 * ctx->m and ctx->to_include may be non-NULL.

That reads a bit off. Should that be "Note that it is OK for both ... to be non-NULL" instead?

Show 21 quoted lines
> +	 * When ctx->m is NULL we are writing a new MIDX without reusing
> +	 * any packs from the previous layer(s). In that case, we care
> +	 * that both:
> +	 *
> +	 *   - the new layer's base MIDX (ctx->base_midx) does not
> +	 *     already contain the pack we are considering, or the new
> +	 *     layer has no base (i.e., it is a non-incremental MIDX)
> +	 *
> +	 *   - the pack appears in ctx->to_include, or ctx->to_include
> +	 *     is NULL, meaning that we can include any pack provided
> +	 *     the above condition is met.
> +	 *
> +	 * When ctx->m is non-NULL, we are writing a new MIDX that will
> +	 * subsume ctx->m and thus includes its packs. In this case, we
> +	 * could support respecting ctx->to_include, but currently
> +	 * don't.
> +	 *
> +	 * The only caller of this function which permits
> +	 * ctx->to_include being non-NULL restricts setting ctx->m when
> +	 * this is the case. So in this setting it is impossible that
> +	 * both will be non-NULL.

I feel like this last paragraph could be dropped -- it's something that we could mention as part of the commit message, but in this function here I think it's very likely to go stale fast.

Other than that I think this is good. It's quite long, but I don't have any good ideas for how to tighten this up significantly.

Thanks!
Patrick

Back to recent threads