From: Taylor Blau Date: Tue, 19 May 2026 15:57:36 GMT Subject: [PATCH v4 00/16] repack: incremental MIDX/bitmap-based repacking Message-ID: In-Reply-To: Here is another small reroll of my series to implement the last remaining component of the incremental MIDX/bitmap-based repacking strategy [1]. The changes since v3 are fairly small: - `repack_prepare_midx_command()` now calls its third parameter "subcommand" instead of "verb". - `clear_incremental_midx_files()` now avoids a redundant conditional around `r->objects` and reuses a local `struct odb_source *` when closing any open incremental MIDX handles. As usual, a range-diff is included below for convenience. Thanks in advance for reviewing! [1]: https://lore.kernel.org/git/cover.1777507303.git.me@ttaylorr.com/ Taylor Blau (16): midx-write: handle noop writes when converting incremental chains midx: use `strset` for retained MIDX files midx: build `keep_hashes` array in order midx: use `strvec` for `keep_hashes` midx: introduce `--no-write-chain-file` for incremental MIDX writes midx: support custom `--base` for incremental MIDX writes repack: track the ODB source via existing_packs midx: expose `midx_layer_contains_pack()` repack-midx: factor out `repack_prepare_midx_command()` repack-midx: extract `repack_fill_midx_stdin_packs()` repack-geometry: prepare for incremental MIDX repacking builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK` packfile: ensure `close_pack_revindex()` frees in-memory revindex repack: implement incremental MIDX repacking repack: introduce `--write-midx=incremental` repack: allow `--write-midx=incremental` without `--geometric` Documentation/config/repack.adoc | 18 + Documentation/git-multi-pack-index.adoc | 32 +- Documentation/git-repack.adoc | 44 +- builtin/multi-pack-index.c | 48 +- builtin/repack.c | 102 +++- midx-write.c | 206 ++++--- midx.c | 102 ++-- midx.h | 11 +- packfile.c | 2 + repack-geometry.c | 48 +- repack-midx.c | 710 +++++++++++++++++++++++- repack.c | 58 +- repack.h | 26 +- t/meson.build | 1 + t/t5334-incremental-multi-pack-index.sh | 63 +++ t/t5335-compact-multi-pack-index.sh | 113 ++++ t/t7705-repack-incremental-midx.sh | 525 ++++++++++++++++++ 17 files changed, 1909 insertions(+), 200 deletions(-) create mode 100755 t/t7705-repack-incremental-midx.sh Range-diff against v3: 1: d6c27317c25 = 1: ead11e610c8 midx-write: handle noop writes when converting incremental chains 2: 629c8d23116 = 2: ece55bf2957 midx: use `strset` for retained MIDX files 3: e303bf6a4ac = 3: 5609d1941e6 midx: build `keep_hashes` array in order 4: 42d76c70060 = 4: 13b7c808860 midx: use `strvec` for `keep_hashes` 5: 2c80aa34fac = 5: cac3fd54bf0 midx: introduce `--no-write-chain-file` for incremental MIDX writes 6: 2a05f4b86f3 = 6: 1bbb387d6b6 midx: support custom `--base` for incremental MIDX writes 7: 92aba3d366f = 7: 4a93adb3ad3 repack: track the ODB source via existing_packs 8: d3ac65c1f11 = 8: 8d1b8b1d301 midx: expose `midx_layer_contains_pack()` 9: 1bd2f194c6f ! 9: 42111e5f75d repack-midx: factor out `repack_prepare_midx_command()` @@ Commit message subcommands), so extract the common portions of the command setup into a reusable `repack_prepare_midx_command()` helper. - The extracted helper sets `git_cmd`, pushes the `multi-pack-index` - subcommand and verb, and handles `--progress`/`--no-progress` and - `--bitmap` flags. The remaining arguments that are specific to the - `write` subcommand (such as `--stdin-packs`) are left to the caller. + The extracted helper sets `git_cmd`, pushes `multi-pack-index` and a + subcommand, and handles `--progress`/`--no-progress` and `--bitmap` + flags. The remaining arguments that are specific to the `write` + subcommand (such as `--stdin-packs`) are left to the caller. No functional changes are included in this patch. @@ repack-midx.c: static void remove_redundant_bitmaps(struct string_list *include, +static void repack_prepare_midx_command(struct child_process *cmd, + struct repack_write_midx_opts *opts, -+ const char *verb) ++ const char *subcommand) +{ + cmd->git_cmd = 1; + -+ strvec_pushl(&cmd->args, "multi-pack-index", verb, NULL); ++ strvec_pushl(&cmd->args, "multi-pack-index", subcommand, NULL); + + if (opts->show_progress) + strvec_push(&cmd->args, "--progress"); 10: 2a87a1e4561 = 10: ed76e6efd1c repack-midx: extract `repack_fill_midx_stdin_packs()` 11: 3d32b9c88da = 11: 9665f1b3a64 repack-geometry: prepare for incremental MIDX repacking 12: 1f7a5479bb8 = 12: e0db62b9f10 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK` 13: b155f25d53c = 13: c8c846b1ac1 packfile: ensure `close_pack_revindex()` frees in-memory revindex 14: ef012314930 = 14: 78f1a98c1ea repack: implement incremental MIDX repacking 15: 04cfecd5136 ! 15: 95bf8b21fff repack: introduce `--write-midx=incremental` @@ midx.c: void clear_midx_file(struct repository *r) +void clear_incremental_midx_files(struct repository *r, + const struct strvec *keep_hashes) +{ ++ struct odb_source *source = r->objects->sources; + struct strbuf chain = STRBUF_INIT; + -+ get_midx_chain_filename(r->objects->sources, &chain); ++ get_midx_chain_filename(source, &chain); + -+ if (r->objects) { -+ struct odb_source *source = r->objects->sources; -+ for (source = r->objects->sources; source; source = source->next) { -+ struct odb_source_files *files = odb_source_files_downcast(source); -+ if (files->packed->midx) -+ close_midx(files->packed->midx); -+ files->packed->midx = NULL; -+ } ++ for (; source; source = source->next) { ++ struct odb_source_files *files = odb_source_files_downcast(source); ++ if (files->packed->midx) ++ close_midx(files->packed->midx); ++ files->packed->midx = NULL; + } + + if (!keep_hashes && remove_path(chain.buf)) 16: 1c05dfce579 = 16: 8bd0ec98dc3 repack: allow `--write-midx=incremental` without `--geometric` base-commit: 7bcaabddcf68bd0702697da5904c3b68c52f94cf -- 2.54.0.175.g8bd0ec98dc3