From: Taylor Blau Date: Wed, 25 Feb 2026 00:21:16 GMT Subject: [RFC PATCH 05/14] midx: support custom `--base` for incremental MIDX writes Message-ID: <9ab735bafe37c2bf3b5e9064d83415e7c4e555c6.1771978829.git.me@ttaylorr.com> In-Reply-To: Both `compact` and `write --incremental` fix the base of the resulting MIDX layer: `compact` always places the compacted result on top of "from's" immediate parent in the chain, and `write --incremental` always appends a new layer to the existing tip. In both cases the base is not configurable. Future callers need additional flexibility. For instance, the incremental MIDX-based repacking code may wish to write a layer based on some intermediate ancestor rather than the current tip, or produce a root layer when replacing the bottommost entries in the chain. Introduce a new `--base` option for both subcommands to specify the checksum of the MIDX layer to use as the base. The given checksum must refer to a valid layer in the MIDX chain that is an ancestor of the topmost layer being written or compacted. The special value "none" is accepted to produce a root layer with no parent. This will be needed when the incremental repacking machinery determines that the bottommost layers of the chain should be replaced. If no `--base` is given, behavior is unchanged: `compact` uses "from's" immediate parent in the chain, and `write` appends to the existing tip. For the `write` subcommand, `--base` requires `--checksum-only`. A plain `write --incremental` appends a new layer to the live chain tip with no mechanism to atomically replace it; overriding the base would produce a layer that does not extend the tip, breaking chain invariants. With `--checksum-only` the chain is left unmodified and the caller is responsible for assembling a valid chain. For `compact`, no such restriction applies. The compaction operation atomically replaces the compacted range in the chain file, so writing the result on top of any valid ancestor preserves chain invariants. Signed-off-by: Taylor Blau --- Documentation/git-multi-pack-index.adoc | 17 +++++- builtin/multi-pack-index.c | 24 +++++++-- midx-write.c | 34 ++++++++++-- midx.h | 5 +- t/t5334-incremental-multi-pack-index.sh | 30 +++++++++++ t/t5335-compact-multi-pack-index.sh | 71 +++++++++++++++++++++++++ 6 files changed, 172 insertions(+), 9 deletions(-) diff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc index 657e0639f6a..635105ad801 100644 --- a/Documentation/git-multi-pack-index.adoc +++ b/Documentation/git-multi-pack-index.adoc @@ -12,8 +12,10 @@ SYNOPSIS 'git multi-pack-index' [] write [--preferred-pack=] [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs] [--refs-snapshot=] [--[no-]checksum-only] + [--base=] 'git multi-pack-index' [] compact [--[no-]incremental] - [--[no-]bitmap] [--[no-]checksum-only] + [--[no-]bitmap] [--base=] [--[no-]checksum-only] + 'git multi-pack-index' [] verify 'git multi-pack-index' [] expire 'git multi-pack-index' [] repack [--batch-size=] @@ -83,6 +85,13 @@ marker). and packs not present in an existing MIDX layer. Migrates non-incremental MIDXs to incremental ones when necessary. + + --base=:: + Specify the checksum of an existing MIDX layer to use + as the base when writing a new incremental layer. + The special value `none` indicates that the new layer + should have no base (i.e., it becomes a root layer). + Requires `--checksum-only`. -- compact:: @@ -97,6 +106,12 @@ compact:: --[no-]bitmap:: Control whether or not a multi-pack bitmap is written. + + --base=:: + Specify the checksum of an existing MIDX layer to use + as the base for the compacted result, instead of using + the immediate parent of ``. The special value + `none` indicates that the result should have no base. -- verify:: diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index ee1ddf1386f..4fc53a5971c 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -15,11 +15,13 @@ #define BUILTIN_MIDX_WRITE_USAGE \ N_("git multi-pack-index [] write [--preferred-pack=]\n" \ " [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n" \ - " [--refs-snapshot=] [--[no-]checksum-only]") + " [--refs-snapshot=] [--[no-]checksum-only]\n" \ + " [--base=]") #define BUILTIN_MIDX_COMPACT_USAGE \ N_("git multi-pack-index [] compact [--[no-]incremental]\n" \ - " [--[no-]bitmap] [--[no-]checksum-only] ") + " [--[no-]bitmap] [--base=] [--[no-]checksum-only]\n" \ + " ") #define BUILTIN_MIDX_VERIFY_USAGE \ N_("git multi-pack-index [] verify") @@ -62,6 +64,7 @@ static char const * const builtin_multi_pack_index_usage[] = { static struct opts_multi_pack_index { char *object_dir; const char *preferred_pack; + const char *incremental_base; char *refs_snapshot; unsigned long batch_size; unsigned flags; @@ -150,6 +153,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, N_("pack for reuse when computing a multi-pack bitmap")), OPT_BIT(0, "bitmap", &opts.flags, N_("write multi-pack bitmap"), MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), + OPT_STRING(0, "base", &opts.incremental_base, N_("checksum"), + N_("base MIDX for incremental writes")), OPT_BIT(0, "incremental", &opts.flags, N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL), OPT_BIT(0, "checksum-only", &opts.flags, @@ -189,6 +194,13 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, options); } + if (opts.incremental_base && + !(opts.flags & MIDX_WRITE_CHECKSUM_ONLY)) { + error(_("cannot use --base without --checksum-only")); + usage_with_options(builtin_multi_pack_index_write_usage, + options); + } + source = handle_object_dir_option(repo); FREE_AND_NULL(options); @@ -200,7 +212,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, ret = write_midx_file_only(source, &packs, opts.preferred_pack, - opts.refs_snapshot, opts.flags); + opts.refs_snapshot, + opts.incremental_base, opts.flags); string_list_clear(&packs, 0); free(opts.refs_snapshot); @@ -228,6 +241,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv, struct option *options; static struct option builtin_multi_pack_index_compact_options[] = { + OPT_STRING(0, "base", &opts.incremental_base, N_("checksum"), + N_("base MIDX for incremental writes")), OPT_BIT(0, "bitmap", &opts.flags, N_("write multi-pack bitmap"), MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), OPT_BIT(0, "incremental", &opts.flags, @@ -289,7 +304,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv, die(_("MIDX %s must be an ancestor of %s"), argv[0], argv[1]); } - ret = write_midx_file_compact(source, from_midx, to_midx, opts.flags); + ret = write_midx_file_compact(source, from_midx, to_midx, + opts.incremental_base, opts.flags); return ret; } diff --git a/midx-write.c b/midx-write.c index 9cf085ca333..c3e70d76d7c 100644 --- a/midx-write.c +++ b/midx-write.c @@ -1245,6 +1245,7 @@ struct write_midx_opts { const char *preferred_pack_name; const char *refs_snapshot; + const char *incremental_base; unsigned flags; }; @@ -1327,11 +1328,32 @@ static int write_midx_internal(struct write_midx_opts *opts) /* * If compacting MIDX layer(s) in the range [from, to], then the - * compacted MIDX will share the same base MIDX as 'from'. + * compacted MIDX will share the same base MIDX as 'from', + * unless a custom --base is specified (see below). */ if (ctx.compact) ctx.base_midx = ctx.compact_from->base_midx; + if (opts->incremental_base) { + if (!strcmp(opts->incremental_base, "none")) { + ctx.base_midx = NULL; + } else { + while (ctx.base_midx) { + const char *cmp = midx_get_checksum_hex(ctx.base_midx); + if (!strcmp(opts->incremental_base, cmp)) + break; + + ctx.base_midx = ctx.base_midx->base_midx; + } + + if (!ctx.base_midx) { + error(_("could not find base MIDX '%s'"), + opts->incremental_base); + goto cleanup; + } + } + } + ctx.nr = 0; ctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16; ctx.info = NULL; @@ -1844,7 +1866,8 @@ static int write_midx_internal(struct write_midx_opts *opts) int write_midx_file(struct odb_source *source, const char *preferred_pack_name, - const char *refs_snapshot, unsigned flags) + const char *refs_snapshot, + unsigned flags) { struct write_midx_opts opts = { .source = source, @@ -1859,13 +1882,16 @@ int write_midx_file(struct odb_source *source, int write_midx_file_only(struct odb_source *source, struct string_list *packs_to_include, const char *preferred_pack_name, - const char *refs_snapshot, unsigned flags) + const char *refs_snapshot, + const char *incremental_base, + unsigned flags) { struct write_midx_opts opts = { .source = source, .packs_to_include = packs_to_include, .preferred_pack_name = preferred_pack_name, .refs_snapshot = refs_snapshot, + .incremental_base = incremental_base, .flags = flags, }; @@ -1875,12 +1901,14 @@ int write_midx_file_only(struct odb_source *source, int write_midx_file_compact(struct odb_source *source, struct multi_pack_index *from, struct multi_pack_index *to, + const char *incremental_base, unsigned flags) { struct write_midx_opts opts = { .source = source, .compact_from = from, .compact_to = to, + .incremental_base = incremental_base, .flags = flags | MIDX_WRITE_COMPACT, }; diff --git a/midx.h b/midx.h index 9f1acd7ace4..e4a75ff2bef 100644 --- a/midx.h +++ b/midx.h @@ -132,10 +132,13 @@ int write_midx_file(struct odb_source *source, int write_midx_file_only(struct odb_source *source, struct string_list *packs_to_include, const char *preferred_pack_name, - const char *refs_snapshot, unsigned flags); + const char *refs_snapshot, + const char *incremental_base, + unsigned flags); int write_midx_file_compact(struct odb_source *source, struct multi_pack_index *from, struct multi_pack_index *to, + const char *incremental_base, unsigned flags); void clear_midx_file(struct repository *r); int verify_midx_file(struct odb_source *source, unsigned flags); diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh index 96449178c07..77fb40ade01 100755 --- a/t/t5334-incremental-multi-pack-index.sh +++ b/t/t5334-incremental-multi-pack-index.sh @@ -112,6 +112,36 @@ test_expect_success 'write non-incremental MIDX layer with --checksum-only' ' test_grep "cannot use --checksum-only without --incremental" err ' +test_expect_success 'write MIDX layer with --base without --checksum-only' ' + test_must_fail git multi-pack-index write --bitmap --incremental \ + --base=none 2>err && + test_grep "cannot use --base without --checksum-only" err +' + +test_expect_success 'write MIDX layer with --base=none and --checksum-only' ' + test_commit base-none && + git repack -d && + + cp "$midx_chain" "$midx_chain.bak" && + layer="$(git multi-pack-index write --bitmap --incremental \ + --checksum-only --base=none)" && + + test_cmp "$midx_chain.bak" "$midx_chain" && + test_path_is_file "$midxdir/multi-pack-index-$layer.midx" +' + +test_expect_success 'write MIDX layer with --base= and --checksum-only' ' + test_commit base-hash && + git repack -d && + + cp "$midx_chain" "$midx_chain.bak" && + layer="$(git multi-pack-index write --bitmap --incremental \ + --checksum-only --base="$(nth_line 1 "$midx_chain")")" && + + test_cmp "$midx_chain.bak" "$midx_chain" && + test_path_is_file "$midxdir/multi-pack-index-$layer.midx" +' + for reuse in false single multi do test_expect_success "full clone (pack.allowPackReuse=$reuse)" ' diff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh index 55b9773568b..f0d1d68d26c 100755 --- a/t/t5335-compact-multi-pack-index.sh +++ b/t/t5335-compact-multi-pack-index.sh @@ -302,6 +302,7 @@ test_expect_success 'MIDX compaction with --checksum-only' ' layer="$(git multi-pack-index compact --incremental \ --checksum-only \ + --base="$(nth_line 1 "$midx_chain")" \ "$(nth_line 2 "$midx_chain")" \ "$(nth_line 3 "$midx_chain")")" && @@ -324,4 +325,74 @@ test_expect_success 'MIDX compaction with --checksum-only' ' ) ' +test_expect_success 'MIDX compaction with --base' ' + git init midx-compact-with--base && + ( + cd midx-compact-with--base && + + write_packs A B C D && + + test_line_count = 4 "$midx_chain" && + + cp "$midx_chain" "$midx_chain.bak" && + + git multi-pack-index compact --incremental \ + --base="$(nth_line 1 "$midx_chain")" \ + "$(nth_line 3 "$midx_chain")" \ + "$(nth_line 4 "$midx_chain")" && + test_line_count = 2 $midx_chain && + + nth_line 1 "$midx_chain.bak" >expect && + nth_line 1 "$midx_chain" >actual && + + test_cmp expect actual + ) +' + +test_expect_success 'MIDX compaction with --base=none' ' + git init midx-compact-base-none && + ( + cd midx-compact-base-none && + + write_packs A B C D && + + test_line_count = 4 $midx_chain && + + cp "$midx_chain" "$midx_chain".bak && + + # Compact the two bottommost layers (A and B) into a new + # root layer with no parent. + git multi-pack-index compact --incremental \ + --base=none \ + "$(nth_line 1 "$midx_chain")" \ + "$(nth_line 2 "$midx_chain")" && + + test_line_count = 3 $midx_chain && + + # The upper layers (C and D) should be preserved + # unchanged. + nth_line 3 "$midx_chain.bak" >expect && + nth_line 4 "$midx_chain.bak" >>expect && + nth_line 2 "$midx_chain" >actual && + nth_line 3 "$midx_chain" >>actual && + + test_cmp expect actual + ) +' + +test_expect_success 'MIDX compaction with bogus --base checksum' ' + git init midx-compact-bogus-base && + ( + cd midx-compact-bogus-base && + + write_packs A B C && + + test_must_fail git multi-pack-index compact --incremental \ + --base=deadbeef \ + "$(nth_line 2 "$midx_chain")" \ + "$(nth_line 3 "$midx_chain")" 2>err && + test_grep "could not find base MIDX" err + ) +' + test_done -- 2.53.0.185.g29bc4dff628