[PATCH 05/16] midx: introduce `--checksum-only` for incremental MIDX writes
- From
Taylor Blau <me@ttaylorr.com>
- Date
- Mar 29, 2026, 21:41 UTC
- Message-ID
- <41cb5471bb96d38a1b9ae75d022e2c5356318939.1774820449.git.me@ttaylorr.com>
- In-Reply-To
- <cover.1774820449.git.me@ttaylorr.com>
When writing an incremental MIDX layer, the MIDX machinery writes the new layer into the multi-pack-index.d directory and then updates the multi-pack-index-chain file to include the freshly written layer.
Future callers however may not wish to immediately update the MIDX chain itself, preferring instead to write out new layer(s) themselves before atomically updating the chain. Concretely, the new incremental MIDX-based repacking strategy will want to do exactly this (that is, assemble the new MIDX chain itself before writing a new chain file and atomically linking it into place).
Introduce a `--checksum-only` flag that:
* writes the new MIDX layer into the multi-pack-index.d directory
* prints its checksum
* does not update the multi-pack-index-chain file.
The MIDX chain file (and thus, the lock protecting it) remain untouched, allowing callers to assemble the chain themselves. This flag requires `--incremental`, since the notion of a separate layer only makes sense for incremental MIDXs.
Signed-off-by: Taylor Blau <me@ttaylorr.com> --- Documentation/git-multi-pack-index.adoc | 4 +-- builtin/multi-pack-index.c | 28 ++++++++++++++++-- midx-write.c | 38 ++++++++++++++++--------- midx.h | 1 + t/t5334-incremental-multi-pack-index.sh | 17 +++++++++++ t/t5335-compact-multi-pack-index.sh | 36 +++++++++++++++++++++++ 6 files changed, 107 insertions(+), 17 deletions(-)
diff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc index 61256830141..657e0639f6a 100644 --- a/Documentation/git-multi-pack-index.adoc +++ b/Documentation/git-multi-pack-index.adoc @@ -11,9 +11,9 @@ SYNOPSIS [verse] 'git multi-pack-index' [<options>] write [--preferred-pack=<pack>] [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs] - [--refs-snapshot=<path>] + [--refs-snapshot=<path>] [--[no-]checksum-only] 'git multi-pack-index' [<options>] compact [--[no-]incremental] - [--[no-]bitmap] <from> <to> + [--[no-]bitmap] [--[no-]checksum-only] <from> <to> 'git multi-pack-index' [<options>] verify 'git multi-pack-index' [<options>] expire 'git multi-pack-index' [<options>] repack [--batch-size=<size>] diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 0f72d96c02d..17ab1525454 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -16,11 +16,11 @@ #define BUILTIN_MIDX_WRITE_USAGE \ N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]\n" \ " [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n" \ - " [--refs-snapshot=<path>]") + " [--refs-snapshot=<path>] [--[no-]checksum-only]") #define BUILTIN_MIDX_COMPACT_USAGE \ N_("git multi-pack-index [<options>] compact [--[no-]incremental]\n" \ - " [--[no-]bitmap] <from> <to>") + " [--[no-]bitmap] [--[no-]checksum-only] <from> <to>") #define BUILTIN_MIDX_VERIFY_USAGE \ N_("git multi-pack-index [<options>] verify") @@ -153,6 +153,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), OPT_BIT(0, "incremental", &opts.flags, N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL), + OPT_BIT(0, "checksum-only", &opts.flags, + N_("write a MIDX layer without updating the MIDX chain"), + MIDX_WRITE_CHECKSUM_ONLY), OPT_BOOL(0, "stdin-packs", &opts.stdin_packs, N_("write multi-pack index containing only given indexes")), OPT_FILENAME(0, "refs-snapshot", &opts.refs_snapshot, @@ -178,6 +181,15 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, if (argc) usage_with_options(builtin_multi_pack_index_write_usage, options); + + if (opts.flags & MIDX_WRITE_CHECKSUM_ONLY && + !(opts.flags & MIDX_WRITE_INCREMENTAL)) { + error(_("cannot use %s without %s"), + "--checksum-only", "--incremental"); + usage_with_options(builtin_multi_pack_index_write_usage, + options); + } + source = handle_object_dir_option(repo); FREE_AND_NULL(options); @@ -221,6 +233,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv, MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), OPT_BIT(0, "incremental", &opts.flags, N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL), + OPT_BIT(0, "checksum-only", &opts.flags, + N_("write a MIDX layer without updating the MIDX chain"), + MIDX_WRITE_CHECKSUM_ONLY), OPT_END(), }; @@ -239,6 +254,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv, if (argc != 2) usage_with_options(builtin_multi_pack_index_compact_usage, options); + + if (opts.flags & MIDX_WRITE_CHECKSUM_ONLY && + !(opts.flags & MIDX_WRITE_INCREMENTAL)) { + error(_("cannot use %s without %s"), + "--checksum-only", "--incremental"); + usage_with_options(builtin_multi_pack_index_compact_usage, + options); + } + source = handle_object_dir_option(the_repository); FREE_AND_NULL(options); diff --git a/midx-write.c b/midx-write.c index 9f7d2bbf4cb..2c6905173ba 100644 --- a/midx-write.c +++ b/midx-write.c @@ -1600,11 +1600,14 @@ static int write_midx_internal(struct write_midx_opts *opts) } if (ctx.incremental) { - struct strbuf lock_name = STRBUF_INIT; + if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) { + struct strbuf lock_name = STRBUF_INIT; - get_midx_chain_filename(opts->source, &lock_name); - hold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR); - strbuf_release(&lock_name); + get_midx_chain_filename(opts->source, &lock_name); + hold_lock_file_for_update(&lk, lock_name.buf, + LOCK_DIE_ON_ERROR); + strbuf_release(&lock_name); + } incr = mks_tempfile_m(midx_name.buf, 0444); if (!incr) { @@ -1725,14 +1728,19 @@ static int write_midx_internal(struct write_midx_opts *opts) } strvec_init_alloc(&keep_hashes, keep_hashes_nr); + if (opts->flags & MIDX_WRITE_CHECKSUM_ONLY) + printf("%s\n", hash_to_hex_algop(midx_hash, r->hash_algo)); + if (ctx.incremental) { - FILE *chainf = fdopen_lock_file(&lk, "w"); struct strbuf final_midx_name = STRBUF_INIT; struct multi_pack_index *m = ctx.base_midx; - if (!chainf) { - error_errno(_("unable to open multi-pack-index chain file")); - goto cleanup; + if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) { + FILE *chainf = fdopen_lock_file(&lk, "w"); + if (!chainf) { + error_errno(_("unable to open multi-pack-index chain file")); + goto cleanup; + } } if (link_midx_to_chain(ctx.base_midx) < 0) @@ -1793,8 +1801,10 @@ static int write_midx_internal(struct write_midx_opts *opts) } } - for (uint32_t i = 0; i < keep_hashes_nr; i++) - fprintf(get_lock_file_fp(&lk), "%s\n", keep_hashes.v[i]); + if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) + for (uint32_t i = 0; i < keep_hashes_nr; i++) + fprintf(get_lock_file_fp(&lk), "%s\n", + keep_hashes.v[i]); } else { keep_hashes.v[ctx.num_multi_pack_indexes_before] = xstrdup(hash_to_hex_algop(midx_hash, r->hash_algo)); @@ -1804,10 +1814,12 @@ static int write_midx_internal(struct write_midx_opts *opts) if (ctx.m || ctx.base_midx) odb_close(ctx.repo->objects); - if (commit_lock_file(&lk) < 0) - die_errno(_("could not write multi-pack-index")); + if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) { + if (commit_lock_file(&lk) < 0) + die_errno(_("could not write multi-pack-index")); - clear_midx_files(opts->source, &keep_hashes, ctx.incremental); + clear_midx_files(opts->source, &keep_hashes, ctx.incremental); + } result = 0; cleanup: diff --git a/midx.h b/midx.h index 08f3728e520..9f1acd7ace4 100644 --- a/midx.h +++ b/midx.h @@ -83,6 +83,7 @@ struct multi_pack_index { #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) #define MIDX_WRITE_INCREMENTAL (1 << 5) #define MIDX_WRITE_COMPACT (1 << 6) +#define MIDX_WRITE_CHECKSUM_ONLY (1 << 7) #define MIDX_EXT_REV "rev" #define MIDX_EXT_BITMAP "bitmap" diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh index c9f5b4e87aa..ac249365f2b 100755 --- a/t/t5334-incremental-multi-pack-index.sh +++ b/t/t5334-incremental-multi-pack-index.sh @@ -96,6 +96,23 @@ test_expect_success 'show object from second pack' ' git cat-file -p 2.2 ' +test_expect_success 'write MIDX layer with --checksum-only' ' + test_commit checksum-only && + git repack -d && + + cp "$midx_chain" "$midx_chain.bak" && + layer="$(git multi-pack-index write --bitmap --incremental \ + --checksum-only)" && + + test_cmp "$midx_chain.bak" "$midx_chain" && + test_path_is_file "$midxdir/multi-pack-index-$layer.midx" +' + +test_expect_success 'write non-incremental MIDX layer with --checksum-only' ' + test_must_fail git multi-pack-index write --bitmap --checksum-only 2>err && + test_grep "cannot use --checksum-only without --incremental" err +' + for reuse in false single multi do test_expect_success "full clone (pack.allowPackReuse=$reuse)" ' diff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh index 40f3844282f..3ad910a9d1c 100755 --- a/t/t5335-compact-multi-pack-index.sh +++ b/t/t5335-compact-multi-pack-index.sh @@ -290,4 +290,40 @@ test_expect_success 'MIDX compaction with bitmaps (non-trivial)' ' ) ' +test_expect_success 'MIDX compaction with --checksum-only' ' + git init midx-compact-with--checksum-only && + ( + cd midx-compact-with--checksum-only && + + git config maintenance.auto false && + + write_packs A B C D && + + test_line_count = 4 $midx_chain && + cp "$midx_chain" "$midx_chain".bak && + + layer="$(git multi-pack-index compact --incremental \ + --checksum-only \ + "$(nth_line 2 "$midx_chain")" \ + "$(nth_line 3 "$midx_chain")")" && + + test_cmp "$midx_chain.bak" "$midx_chain" && + + # After writing the new layer, insert it into the chain + # manually. This is done in order to make $layer visible + # to the read-midx test helper below, and matches what + # the MIDX command would do without --checksum-only. + { + nth_line 1 "$midx_chain.bak" && + echo $layer && + nth_line 4 "$midx_chain.bak" + } >$midx_chain && + + test-tool read-midx $objdir $layer >midx.data && + grep "^pack-B-.*\.idx" midx.data && + grep "^pack-C-.*\.idx" midx.data + + ) +' + test_done
-- 2.53.0.729.g817728289e1.dirty