{"thread":{"id":"65071","subject":"[RFC PATCH 00/14] repack: incremental MIDX/bitmap-based repacking","startedAt":"2026-02-25T00:20:45Z","lastAt":"2026-02-27T03:07:15Z","messageCount":21,"participants":["Taylor Blau","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":14},"messages":[{"id":"537044","messageId":"cover.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":null,"subject":"[RFC PATCH 00/14] repack: incremental MIDX/bitmap-based repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:20:41Z","receivedAt":"2026-02-25T00:20:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"[NOTE: this series has not yet been tested thoroughly and is based on\nthe v3 of 'tb/incremental-midx-part-3.2', which is d54da84bd9d (midx:\nenable reachability bitmaps during MIDX compaction, 2026-02-24).\n\nI suspect that there will be non-trivial changes to the approach in this\nseries based on feedback, hence the RFC.]\n\nNote to the maintainer:\n\n * This series is based on 'tb/incremental-midx-part-3.2', I suggest\n queueing as 'tb/incremental-midx-part-3.3'.\n\nOVERVIEW\n========\n\nThis series implements the third and final major component for an\nincremental MIDX/bitmap-based repacking strategy. As a refresher, those\nare:\n\n 1. Refactoring code out of builtin/repack.c into individual compilation\n units outside of the builtin directory.\n\n 2. Implementing MIDX layer compaction, i.e., the ability to combine a\n contiguous sequence of MIDX layers in an existing chain and replace\n them with a single layer representing the union of objects/packs\n among the compacted layers.\n\n 3. (this series) An incremental repacking strategy that, unlike our\n current '--geometric' approach, does not rely on periodic\n all-into-one repacks.\n\nBACKGROUND\n==========\n\nToday, a '--geometric' repack with '--write-midx' rewrites the entire\nMIDX (and, when enabled, its reachability bitmap) on every invocation.\nFor most repositories this is acceptable. In large monorepos, these\ncosts can add up significantly, especially when those repositories are\nrepacked frequently.\n\nThe incremental MIDX support introduced in the earlier parts of this\neffort allows us to append new layers to a MIDX chain without rewriting\nanything. Combined with the support for MIDX compaction in 3.2, we now\nhave all of the building blocks needed to maintain an incrementally\ngrowing and shrinking MIDX chain as part of the repack cycle.\n\nSTRATEGY\n========\n\nThe incremental repacking strategy implemented in this series works\n(roughly) as follows:\n\n 1. Repack non-MIDX'd packs using an ordinary '--geometric' repack,\n optionally including packs from the tip MIDX layer if and only if it\n contains more than 'repack.midxNewLayerThreshold' number of packs.\n\n 2. Prepare to write a new MIDX layer containing the freshly written\n pack(s) (and any surviving packs from the tip layer, if it was\n rewritten).\n\n 3. Perform a compaction pass over adjacent MIDX layers to restore a\n geometric progression on object count among layers in the chain\n (determined by 'repack.midxSplitFactor').\n\n 4. Write the new MIDX chain, link it into place, and remove redundant\n layers.\n\nIn effect, this approach encourages MIDX chains where:\n\n * older layers contain fewer, larger packs, and\n\n * newer layers contain many smaller packs.\n\nAs a result of the compaction pass, we prevent the chain itself from\never growing too long.\n\nThis roughly matches the description of this algorithm I gave at Git\nMerge last year, which is covered on slides 80-131 of this presentation:\n\n https://ttaylorr.com/presentations/git-merge-2025.pdf\n\nTHIS SERIES\n===========\n\nThis series is organized roughly as follows:\n\n * The first three patches perform small-ish quality-of-life refactors\n within the MIDX machinery.\n\n * The next two patches introduce `--checksum-only` and `--base`\n options for `git multi-pack-index write` and `compact`, which are\n needed by the incremental repacking machinery to assemble MIDX\n chains without prematurely updating the chain file.\n\n * The next six patches prepare the repack infrastructure and pack\n geometry code for the new repacking strategy.\n\n * The final three patches introduce the new repacking machinery, expose\n it to users, and then extend it to work with non-geometric repacks in\n that order. The first two are the substantive patches of this series.\n\nWHERE WE'RE AT\n==============\n\nThis series delivers the final substantive component of this overall\neffort to enable the new repacking strategy implemented here. There are\na couple of (comparatively much smaller) items that will be useful as\nfollow-on items, including:\n\n * \"Reachability-infused\" geometric repacking to emit small cruft packs\n to introduce other ways to update the set of cruft objects beyond a\n whole-repository traversal.\n\n * Richer bitmap configuration to determine which bitmap(s) to carry\n forward between adjacent MIDX layers when doing an incremental\n repack, to prevent an endless accumulation of reachability bitmaps.\n\nLike I mentioned earlier, I think that both of those are significantly\nsmaller challenges than this and the previous series. As a general note,\nI am going on vacation for two weeks starting this Friday (2026-02-24)\nand will be away from the list.\n\nI'll plan on picking up any review from this series as well as the two\nother topics listed above as soon as I get back.\n\nThanks in advance for your review!\n\nTaylor Blau (14):\n midx: use `string_list` for retained MIDX files\n strvec: introduce `strvec_init_alloc()`\n midx: use `strvec` for `keep_hashes`\n midx: introduce `--checksum-only` for incremental MIDX writes\n midx: support custom `--base` for incremental MIDX writes\n repack: track the ODB source via existing_packs\n midx: expose `midx_layer_contains_pack()`\n repack-midx: factor out `repack_prepare_midx_command()`\n repack-midx: extract `repack_fill_midx_stdin_packs()`\n repack-geometry: prepare for incremental MIDX repacking\n builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n repack: implement incremental MIDX repacking\n repack: introduce `--write-midx=incremental`\n repack: allow `--write-midx=incremental` without `--geometric`\n\n Documentation/config/repack.adoc | 18 +\n Documentation/git-multi-pack-index.adoc | 19 +-\n Documentation/git-repack.adoc | 44 +-\n builtin/multi-pack-index.c | 48 +-\n builtin/repack.c | 90 ++-\n midx-write.c | 119 ++--\n midx.c | 104 ++--\n midx.h | 11 +-\n repack-geometry.c | 50 +-\n repack-midx.c | 704 +++++++++++++++++++++++-\n repack.c | 23 +-\n repack.h | 25 +-\n strvec.c | 7 +\n strvec.h | 5 +\n t/meson.build | 1 +\n t/t5334-incremental-multi-pack-index.sh | 47 ++\n t/t5335-compact-multi-pack-index.sh | 105 ++++\n t/t7705-repack-incremental-midx.sh | 461 ++++++++++++++++\n 18 files changed, 1730 insertions(+), 151 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\n\nbase-commit: ac6221686db11ba802ced8ea6b8b2fa8388b21b2\n-- \n2.53.0.185.g29bc4dff628\n"},{"id":"537045","messageId":"df37959d2933d84ec61b9ba570c87f9e5213c735.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 06/14] repack: track the ODB source via existing_packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:20:46Z","receivedAt":"2026-02-25T00:20:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Store the ODB source in the `existing_packs` struct and use that in\nplace of the raw `repo->objects->sources` access within `cmd_repack()`.\n\nThe source used is still assigned from the first source in the list, so\nthere are no functional changes in this commit. The changes instead\nserve two purposes (one immediate, one not):\n\n - The incremental MIDX-based repacking machinery will need to know what\n source is being used to read the existing MIDX/chain from that source\n (should one exist).\n\n - In the future, if \"git repack\" is taught how to operate on other\n object sources, this field will serve as the authoritative value for\n that source.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 ++---\n repack.c | 2 ++\n repack.h | 1 +\n 3 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f6bb04bef72..44a95b56f23 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -402,7 +402,7 @@ int cmd_repack(int argc,\n \t\t * midx_has_unknown_packs() will make the decision for\n \t\t * us.\n \t\t */\n-\t\tif (!get_multi_pack_index(repo->objects->sources))\n+\t\tif (!get_multi_pack_index(existing.source))\n \t\t\tmidx_must_contain_cruft = 1;\n \t}\n \n@@ -549,8 +549,7 @@ int cmd_repack(int argc,\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n \t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n-\t\twrite_midx_file(repo->objects->sources,\n-\t\t\t\tNULL, NULL, flags);\n+\t\twrite_midx_file(existing.source, NULL, NULL, flags);\n \t}\n \n cleanup:\ndiff --git a/repack.c b/repack.c\nindex 596841027af..2ee6b51420a 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -154,6 +154,8 @@ void existing_packs_collect(struct existing_packs *existing,\n \t\t\tstring_list_append(&existing->non_kept_packs, buf.buf);\n \t}\n \n+\texisting->source = existing->repo->objects->sources;\n+\n \tstring_list_sort(&existing->kept_packs);\n \tstring_list_sort(&existing->non_kept_packs);\n \tstring_list_sort(&existing->cruft_packs);\ndiff --git a/repack.h b/repack.h\nindex bc9f2e1a5de..c0e9f0ca647 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -56,6 +56,7 @@ struct packed_git;\n \n struct existing_packs {\n \tstruct repository *repo;\n+\tstruct odb_source *source;\n \tstruct string_list kept_packs;\n \tstruct string_list non_kept_packs;\n \tstruct string_list cruft_packs;\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537046","messageId":"d64a799afd620363c1940d7c2e634e78ea553cb6.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 01/14] midx: use `string_list` for retained MIDX files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:20:56Z","receivedAt":"2026-02-25T00:21:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\nbuild a list of filenames to keep while pruning stale MIDX files. Today\nthey hand-roll an array instead of using a `string_list`, thus requiring\nus to pass an additional length parameter, and makes lookups linear.\n\nReplace the bare array with a `string_list` which can be passed around\nas a single parameter. Though it improves lookup performance, the\ndifference is likely immeasurable given how small the keep_hashes array\ntypically is.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 56 ++++++++++++++++++++++----------------------------------\n 1 file changed, 22 insertions(+), 34 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex c1b9658240d..c5e3553e2bb 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -755,8 +755,7 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar **keep;\n-\tuint32_t keep_nr;\n+\tstruct string_list keep;\n \tconst char *ext;\n };\n \n@@ -764,15 +763,12 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n-\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tfor (i = 0; i < data->keep_nr; i++) {\n-\t\tif (!strcmp(data->keep[i], file_name))\n-\t\t\treturn;\n-\t}\n+\tif (string_list_has_string(&data->keep, file_name))\n+\t\treturn;\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n@@ -780,48 +776,40 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash)\n {\n-\tstruct clear_midx_data data;\n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n-\n-\tif (keep_hash) {\n-\t\tALLOC_ARRAY(data.keep, 1);\n-\n-\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n-\t\tdata.keep_nr = 1;\n-\t}\n-\tdata.ext = ext;\n-\n-\tfor_each_file_in_pack_dir(source->path,\n-\t\t\t\t clear_midx_file_ext,\n-\t\t\t\t &data);\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRING_LIST_INIT_NODUP,\n+\t\t.ext = ext,\n+\t};\n \n \tif (keep_hash)\n-\t\tfree(data.keep[0]);\n-\tfree(data.keep);\n+\t\tstring_list_insert(&data.keep, xstrfmt(\"multi-pack-index-%s.%s\",\n+\t\t\t\t\t\t keep_hash, ext));\n+\n+\tfor_each_file_in_pack_dir(source->path, clear_midx_file_ext, &data);\n+\n+\tstring_list_clear(&data.keep, 0);\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t char **keep_hashes,\n \t\t\t\t uint32_t hashes_nr)\n {\n-\tstruct clear_midx_data data;\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRING_LIST_INIT_NODUP,\n+\t\t.ext = ext,\n+\t};\n \tuint32_t i;\n \n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n-\n-\tALLOC_ARRAY(data.keep, hashes_nr);\n \tfor (i = 0; i < hashes_nr; i++)\n-\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t\t ext);\n-\tdata.keep_nr = hashes_nr;\n-\tdata.ext = ext;\n+\t\tstring_list_append(&data.keep,\n+\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n+\t\t\t\t\t keep_hashes[i], ext));\n+\tstring_list_sort(&data.keep);\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n \n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tfree(data.keep[i]);\n-\tfree(data.keep);\n+\tstring_list_clear(&data.keep, 0);\n }\n \n void clear_midx_file(struct repository *r)\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537047","messageId":"50efbbb0fe8d897d7c4cd51489af4cb4c4c49d02.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 02/14] strvec: introduce `strvec_init_alloc()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:01Z","receivedAt":"2026-02-25T00:21:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When the caller knows upfront how many elements will be pushed onto a\n`strvec`, it is useful to pre-allocate enough space in the array to fit\nthat many elements (and one additional slot to store NULL, indicating\nthe end of the list.)\n\nIntroduce `strvec_init_alloc()`, which allocates the backing array large\nenough to hold `alloc` elements and the termination marker without\nfurther reallocation.\n\nSigned-off-by: Taylor Blau \n---\n strvec.c | 7 +++++++\n strvec.h | 5 +++++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/strvec.c b/strvec.c\nindex f8de79f5579..f7f32a53b56 100644\n--- a/strvec.c\n+++ b/strvec.c\n@@ -10,6 +10,13 @@ void strvec_init(struct strvec *array)\n \tmemcpy(array, &blank, sizeof(*array));\n }\n \n+void strvec_init_alloc(struct strvec *array, size_t alloc)\n+{\n+\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n+\tarray->nr = 0;\n+\tarray->alloc = alloc + 1;\n+}\n+\n void strvec_push_nodup(struct strvec *array, char *value)\n {\n \tif (array->v == empty_strvec)\ndiff --git a/strvec.h b/strvec.h\nindex f74e061e141..34cb1f939f0 100644\n--- a/strvec.h\n+++ b/strvec.h\n@@ -43,6 +43,11 @@ struct strvec {\n */\n void strvec_init(struct strvec *);\n \n+/*\n+ * Initializes an array large enough to store `alloc` elements.\n+ */\n+void strvec_init_alloc(struct strvec *, size_t alloc);\n+\n /* Push a copy of a string onto the end of the array. */\n const char *strvec_push(struct strvec *, const char *);\n \n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537048","messageId":"b2fffdfc8910308b6c816dfba9e04196a5aaf4ac.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 03/14] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:05Z","receivedAt":"2026-02-25T00:21:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `keep_hashes` array in `write_midx_internal()` accumulates the\nchecksums of MIDX files that should be retained when pruning stale\nentries from the MIDX chain. For similar reasons as in a previous\ncommit, rewrite this using a strvec, requiring us to pass one fewer\nparameter.\n\nUnlike the aforementioned previous commit, use a `strvec` instead of a\n`string_list`, which provides a more ergonomic interface to adjust the\nvalues at a particular indice. The ordering is important here, as this\nvalue is used to determine the contents of the resulting\n`multi-pack-index-chain` file when writing with \"--incremental\".\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 51 ++++++++++++++++++++++++++-------------------------\n midx.c | 20 ++++++++++----------\n 2 files changed, 36 insertions(+), 35 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 0ff2e45aa7a..7fdc4d31243 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -29,8 +29,7 @@ extern void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t const char *keep_hash);\n extern void clear_incremental_midx_files_ext(struct odb_source *source,\n \t\t\t\t\t const char *ext,\n-\t\t\t\t\t const char **keep_hashes,\n-\t\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t\t const struct strvec *keep_hashes);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -1109,8 +1108,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n }\n \n static void clear_midx_files(struct odb_source *source,\n-\t\t\t const char **hashes, uint32_t hashes_nr,\n-\t\t\t unsigned incremental)\n+\t\t\t const struct strvec *hashes, unsigned incremental)\n {\n \t/*\n \t * if incremental:\n@@ -1124,13 +1122,15 @@ static void clear_midx_files(struct odb_source *source,\n \t */\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n-\tuint32_t i, j;\n+\tuint32_t i;\n \n \tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tclear_incremental_midx_files_ext(source, exts[i],\n-\t\t\t\t\t\t hashes, hashes_nr);\n-\t\tfor (j = 0; j < hashes_nr; j++)\n-\t\t\tclear_midx_files_ext(source, exts[i], hashes[j]);\n+\t\tclear_incremental_midx_files_ext(source, exts[i], hashes);\n+\t\tif (hashes) {\n+\t\t\tfor (size_t j = 0; j < hashes->nr; j++)\n+\t\t\t\tclear_midx_files_ext(source, exts[i],\n+\t\t\t\t\t\t hashes->v[j]);\n+\t\t}\n \t}\n \n \tif (incremental)\n@@ -1266,7 +1266,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = -1;\n-\tconst char **keep_hashes = NULL;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tsize_t keep_hashes_nr = 0;\n \tstruct chunkfile *cf;\n \n@@ -1721,7 +1721,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t} else {\n \t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n \t}\n-\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n+\tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n \n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n@@ -1758,39 +1758,45 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n \t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n \n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.nr++;\n \t\t\t\tm = m->base_midx;\n \t\t\t}\n \n-\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\tkeep_hashes.v[i] = xstrdup(hash_to_hex_algop(midx_hash,\n \t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes.nr++;\n \n \t\t\ti = 0;\n \t\t\tfor (m = ctx.m;\n \t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n \t\t\t m = m->base_midx) {\n-\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n+\t\t\t\tkeep_hashes.v[keep_hashes_nr - i - 1] =\n \t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.nr++;\n \t\t\t\ti++;\n \t\t\t}\n \t\t} else {\n-\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n \t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes.nr++;\n \n \t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.nr++;\n \t\t\t\tm = m->base_midx;\n \t\t\t}\n \t\t}\n \n \t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n \t} else {\n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tkeep_hashes.nr++;\n \t}\n \n \tif (ctx.m || ctx.base_midx)\n@@ -1799,8 +1805,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n-\t\t\t ctx.incremental);\n+\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n \tresult = 0;\n \n cleanup:\n@@ -1816,11 +1821,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n-\tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfree((char *)keep_hashes[i]);\n-\t\tfree(keep_hashes);\n-\t}\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&midx_name);\n \tclose_midx(midx_to_free);\n \ndiff --git a/midx.c b/midx.c\nindex c5e3553e2bb..20732c256a6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"chunk-format.h\"\n #include \"pack-bitmap.h\"\n #include \"pack-revindex.h\"\n+#include \"strvec.h\"\n \n #define MIDX_PACK_ERROR ((void *)(intptr_t)-1)\n \n@@ -19,8 +20,7 @@ int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash);\n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t const struct strvec *keep_hashes);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -791,20 +791,20 @@ void clear_midx_files_ext(struct odb_source *source, const char *ext,\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr)\n+\t\t\t\t const struct strvec *keep_hashes)\n {\n \tstruct clear_midx_data data = {\n \t\t.keep = STRING_LIST_INIT_NODUP,\n \t\t.ext = ext,\n \t};\n-\tuint32_t i;\n \n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tstring_list_append(&data.keep,\n-\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n-\t\t\t\t\t keep_hashes[i], ext));\n-\tstring_list_sort(&data.keep);\n+\tif (keep_hashes) {\n+\t\tfor (size_t i = 0; i < keep_hashes->nr; i++)\n+\t\t\tstring_list_append(&data.keep,\n+\t\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n+\t\t\t\t\t\t keep_hashes->v[i], ext));\n+\t\tstring_list_sort(&data.keep);\n+\t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537049","messageId":"2d31ea3907ff1965302acb089a594746c8dfb149.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 04/14] midx: introduce `--checksum-only` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:10Z","receivedAt":"2026-02-25T00:21:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When writing an incremental MIDX layer, the MIDX machinery writes the\nnew layer into the multi-pack-index.d directory and then updates the\nmulti-pack-index-chain file to include the freshly written layer.\n\nFuture callers however may not wish to immediately update the MIDX chain\nitself, preferring instead to write out new layer(s) itself before\natomically updating the chain. Concretely, the new incremental\nMIDX-based repacking strategy will want to do exactly this (that is,\nassemble the new MIDX chain itself before writing a new chain file and\natomically linking it into place).\n\nIntroduce a `--checksum-only` flag that:\n\n * writes the new MIDX layer into the multi-pack-index.d directory\n\n * prints its checksum\n\n * does not update the multi-pack-index-chain file.\n\nThe MIDX chain file (and thus, the lock protecting it) remain untouched,\nallowing callers to assemble the chain themselves. This flag requires\n`--incremental`, since the notion of a separate layer only makes sense\nfor incremental MIDXs.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 4 +--\n builtin/multi-pack-index.c | 28 ++++++++++++++++--\n midx-write.c | 38 ++++++++++++++++---------\n midx.h | 1 +\n t/t5334-incremental-multi-pack-index.sh | 17 +++++++++++\n t/t5335-compact-multi-pack-index.sh | 34 ++++++++++++++++++++++\n 6 files changed, 105 insertions(+), 17 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 61256830141..657e0639f6a 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -11,9 +11,9 @@ SYNOPSIS\n [verse]\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n-\t\t [--refs-snapshot=]\n+\t\t [--refs-snapshot=] [--[no-]checksum-only]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] \n+\t\t [--[no-]bitmap] [--[no-]checksum-only] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 2f24c113c8f..ee1ddf1386f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -15,11 +15,11 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=]\")\n+\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] \")\n+\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -152,6 +152,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n@@ -177,6 +180,15 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \tif (argc)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--checksum-only\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -220,6 +232,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n \t\tOPT_END(),\n \t};\n \n@@ -238,6 +253,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \tif (argc != 2)\n \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--checksum-only\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(the_repository);\n \n \tFREE_AND_NULL(options);\ndiff --git a/midx-write.c b/midx-write.c\nindex 7fdc4d31243..9cf085ca333 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1598,11 +1598,14 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \tif (ctx.incremental) {\n-\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(opts->source, &lock_name);\n-\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n-\t\tstrbuf_release(&lock_name);\n+\t\t\tget_midx_chain_filename(opts->source, &lock_name);\n+\t\t\thold_lock_file_for_update(&lk, lock_name.buf,\n+\t\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\t\t\tstrbuf_release(&lock_name);\n+\t\t}\n \n \t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n \t\tif (!incr) {\n@@ -1723,14 +1726,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n \n+\tif (opts->flags & MIDX_WRITE_CHECKSUM_ONLY)\n+\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n+\n \tif (ctx.incremental) {\n-\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \n-\t\tif (!chainf) {\n-\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n-\t\t\tgoto cleanup;\n+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\t\tif (!chainf) {\n+\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t}\n \n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n@@ -1791,8 +1799,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY))\n+\t\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n+\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n+\t\t\t\t\tkeep_hashes.v[i]);\n \t} else {\n \t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n@@ -1802,10 +1812,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.m || ctx.base_midx)\n \t\todb_close(ctx.repo->objects);\n \n-\tif (commit_lock_file(&lk) < 0)\n-\t\tdie_errno(_(\"could not write multi-pack-index\"));\n+\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\tif (commit_lock_file(&lk) < 0)\n+\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t}\n \tresult = 0;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex 08f3728e520..9f1acd7ace4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -83,6 +83,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n #define MIDX_WRITE_COMPACT (1 << 6)\n+#define MIDX_WRITE_CHECKSUM_ONLY (1 << 7)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex d30d7253d6f..96449178c07 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -95,6 +95,23 @@ test_expect_success 'show object from second pack' '\n \tgit cat-file -p 2.2\n '\n \n+test_expect_success 'write MIDX layer with --checksum-only' '\n+\ttest_commit checksum-only &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--checksum-only)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n+\ttest_must_fail git multi-pack-index write --bitmap --checksum-only 2>err &&\n+\ttest_grep \"cannot use --checksum-only without --incremental\" err\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 40f3844282f..55b9773568b 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -290,4 +290,38 @@ test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --checksum-only' '\n+\tgit init midx-compact-with--checksum-only &&\n+\t(\n+\t\tcd midx-compact-with--checksum-only &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\tlayer=\"$(git multi-pack-index compact --incremental \\\n+\t\t\t--checksum-only \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n+\n+\t\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\n+\t\t# After writing the new layer, insert it into the chain\n+\t\t# manually. This is done in order to make $layer visible\n+\t\t# to the read-midx test helper below, and matches what\n+\t\t# the MIDX command would do without --checksum-only.\n+\t\t{\n+\t\t\tnth_line 1 \"$midx_chain.bak\" &&\n+\t\t\techo $layer &&\n+\t\t\tnth_line 4 \"$midx_chain.bak\"\n+\t\t} >$midx_chain &&\n+\n+\t\ttest-tool read-midx $objdir $layer >midx.data &&\n+\t\tgrep \"^pack-B-.*\\.idx\" midx.data &&\n+\t\tgrep \"^pack-C-.*\\.idx\" midx.data\n+\n+\t)\n+'\n+\n test_done\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537050","messageId":"9ab735bafe37c2bf3b5e9064d83415e7c4e555c6.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 05/14] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:16Z","receivedAt":"2026-02-25T00:21:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Both `compact` and `write --incremental` fix the base of the resulting\nMIDX layer: `compact` always places the compacted result on top of\n\"from's\" immediate parent in the chain, and `write --incremental` always\nappends a new layer to the existing tip. In both cases the base is not\nconfigurable.\n\nFuture callers need additional flexibility. For instance, the incremental\nMIDX-based repacking code may wish to write a layer based on some\nintermediate ancestor rather than the current tip, or produce a root\nlayer when replacing the bottommost entries in the chain.\n\nIntroduce a new `--base` option for both subcommands to specify the\nchecksum of the MIDX layer to use as the base. The given checksum must\nrefer to a valid layer in the MIDX chain that is an ancestor of the\ntopmost layer being written or compacted.\n\nThe special value \"none\" is accepted to produce a root layer with no\nparent. This will be needed when the incremental repacking machinery\ndetermines that the bottommost layers of the chain should be replaced.\n\nIf no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\nimmediate parent in the chain, and `write` appends to the existing tip.\n\nFor the `write` subcommand, `--base` requires `--checksum-only`. A plain\n`write --incremental` appends a new layer to the live chain tip with no\nmechanism to atomically replace it; overriding the base would produce a\nlayer that does not extend the tip, breaking chain invariants. With\n`--checksum-only` the chain is left unmodified and the caller is\nresponsible for assembling a valid chain.\n\nFor `compact`, no such restriction applies. The compaction operation\natomically replaces the compacted range in the chain file, so writing\nthe result on top of any valid ancestor preserves chain invariants.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 +++++-\n builtin/multi-pack-index.c | 24 +++++++--\n midx-write.c | 34 ++++++++++--\n midx.h | 5 +-\n t/t5334-incremental-multi-pack-index.sh | 30 +++++++++++\n t/t5335-compact-multi-pack-index.sh | 71 +++++++++++++++++++++++++\n 6 files changed, 172 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 657e0639f6a..635105ad801 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,8 +12,10 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=] [--[no-]checksum-only]\n+\t\t\t [--base=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] [--[no-]checksum-only] \n+\t\t [--[no-]bitmap] [--base=] [--[no-]checksum-only]\n+\t\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +85,13 @@ marker).\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n \t\tnecessary.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base when writing a new incremental layer.\n+\t\tThe special value `none` indicates that the new layer\n+\t\tshould have no base (i.e., it becomes a root layer).\n+\t\tRequires `--checksum-only`.\n --\n \n compact::\n@@ -97,6 +106,12 @@ compact::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base for the compacted result, instead of using\n+\t\tthe immediate parent of ``. The special value\n+\t\t`none` indicates that the result should have no base.\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex ee1ddf1386f..4fc53a5971c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -15,11 +15,13 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n+\t \" [--refs-snapshot=] [--[no-]checksum-only]\\n\" \\\n+\t \" [--base=]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n+\t \" [--[no-]bitmap] [--base=] [--[no-]checksum-only]\\n\" \\\n+\t \" \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -62,6 +64,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tchar *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *incremental_base;\n \tchar *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n@@ -150,6 +153,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n@@ -189,6 +194,13 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t\t options);\n \t}\n \n+\tif (opts.incremental_base &&\n+\t !(opts.flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\terror(_(\"cannot use --base without --checksum-only\"));\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -200,7 +212,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \n \t\tret = write_midx_file_only(source, &packs,\n \t\t\t\t\t opts.preferred_pack,\n-\t\t\t\t\t opts.refs_snapshot, opts.flags);\n+\t\t\t\t\t opts.refs_snapshot,\n+\t\t\t\t\t opts.incremental_base, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \t\tfree(opts.refs_snapshot);\n@@ -228,6 +241,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n@@ -289,7 +304,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n \t}\n \n-\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\tret = write_midx_file_compact(source, from_midx, to_midx,\n+\t\t\t\t opts.incremental_base, opts.flags);\n \n \treturn ret;\n }\ndiff --git a/midx-write.c b/midx-write.c\nindex 9cf085ca333..c3e70d76d7c 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1245,6 +1245,7 @@ struct write_midx_opts {\n \n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n+\tconst char *incremental_base;\n \tunsigned flags;\n };\n \n@@ -1327,11 +1328,32 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t/*\n \t * If compacting MIDX layer(s) in the range [from, to], then the\n-\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t * compacted MIDX will share the same base MIDX as 'from',\n+\t * unless a custom --base is specified (see below).\n \t */\n \tif (ctx.compact)\n \t\tctx.base_midx = ctx.compact_from->base_midx;\n \n+\tif (opts->incremental_base) {\n+\t\tif (!strcmp(opts->incremental_base, \"none\")) {\n+\t\t\tctx.base_midx = NULL;\n+\t\t} else {\n+\t\t\twhile (ctx.base_midx) {\n+\t\t\t\tconst char *cmp = midx_get_checksum_hex(ctx.base_midx);\n+\t\t\t\tif (!strcmp(opts->incremental_base, cmp))\n+\t\t\t\t\tbreak;\n+\n+\t\t\t\tctx.base_midx = ctx.base_midx->base_midx;\n+\t\t\t}\n+\n+\t\t\tif (!ctx.base_midx) {\n+\t\t\t\terror(_(\"could not find base MIDX '%s'\"),\n+\t\t\t\t opts->incremental_base);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1844,7 +1866,8 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n-\t\t const char *refs_snapshot, unsigned flags)\n+\t\t const char *refs_snapshot,\n+\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n@@ -1859,13 +1882,16 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags)\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.packs_to_include = packs_to_include,\n \t\t.preferred_pack_name = preferred_pack_name,\n \t\t.refs_snapshot = refs_snapshot,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags,\n \t};\n \n@@ -1875,12 +1901,14 @@ int write_midx_file_only(struct odb_source *source,\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.compact_from = from,\n \t\t.compact_to = to,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags | MIDX_WRITE_COMPACT,\n \t};\n \ndiff --git a/midx.h b/midx.h\nindex 9f1acd7ace4..e4a75ff2bef 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -132,10 +132,13 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags);\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags);\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 96449178c07..77fb40ade01 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -112,6 +112,36 @@ test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n \ttest_grep \"cannot use --checksum-only without --incremental\" err\n '\n \n+test_expect_success 'write MIDX layer with --base without --checksum-only' '\n+\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n+\t\t--base=none 2>err &&\n+\ttest_grep \"cannot use --base without --checksum-only\" err\n+'\n+\n+test_expect_success 'write MIDX layer with --base=none and --checksum-only' '\n+\ttest_commit base-none &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--checksum-only --base=none)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write MIDX layer with --base= and --checksum-only' '\n+\ttest_commit base-hash &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--checksum-only --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 55b9773568b..f0d1d68d26c 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -302,6 +302,7 @@ test_expect_success 'MIDX compaction with --checksum-only' '\n \n \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n \t\t\t--checksum-only \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n \n@@ -324,4 +325,74 @@ test_expect_success 'MIDX compaction with --checksum-only' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --base' '\n+\tgit init midx-compact-with--base &&\n+\t(\n+\t\tcd midx-compact-with--base &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tnth_line 1 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 1 \"$midx_chain\" >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with --base=none' '\n+\tgit init midx-compact-base-none &&\n+\t(\n+\t\tcd midx-compact-base-none &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\t# Compact the two bottommost layers (A and B) into a new\n+\t\t# root layer with no parent.\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=none \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\t# The upper layers (C and D) should be preserved\n+\t\t# unchanged.\n+\t\tnth_line 3 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 4 \"$midx_chain.bak\" >>expect &&\n+\t\tnth_line 2 \"$midx_chain\" >actual &&\n+\t\tnth_line 3 \"$midx_chain\" >>actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bogus --base checksum' '\n+\tgit init midx-compact-bogus-base &&\n+\t(\n+\t\tcd midx-compact-bogus-base &&\n+\n+\t\twrite_packs A B C &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t--base=deadbeef \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" 2>err &&\n+\t\ttest_grep \"could not find base MIDX\" err\n+\t)\n+'\n+\n test_done\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537051","messageId":"aba9ad5174aec361f019eb44da87b04ec9b92679.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 07/14] midx: expose `midx_layer_contains_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:27Z","receivedAt":"2026-02-25T00:21:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Rename the function `midx_contains_pack_1()` to instead be called\n`midx_layer_contains_pack()` and make it accessible. Unlike\n`midx_contains_pack()` (which recurses through the entire chain), this\nfunction checks only a single MIDX layer.\n\nThis will be used by a subsequent commit to determine whether a given\npack belongs to the tip MIDX layer specifically, rather than to any\nlayer in the chain.\n\nNo functional changes are present in this commit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++---\n midx.h | 2 ++\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 20732c256a6..709fe7c3fd7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -665,8 +665,8 @@ static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n \t\t m->pack_names[*(const size_t *)b]);\n }\n \n-static int midx_contains_pack_1(struct multi_pack_index *m,\n-\t\t\t\tconst char *idx_or_pack_name)\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -707,7 +707,7 @@ static int midx_contains_pack_1(struct multi_pack_index *m,\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tfor (; m; m = m->base_midx)\n-\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\tif (midx_layer_contains_pack(m, idx_or_pack_name))\n \t\t\treturn 1;\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex e4a75ff2bef..f211a38b9e7 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -119,6 +119,8 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct multi_pack_index *m, const struct object_id *oid, struct pack_entry *e);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct odb_source *source);\n \n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537052","messageId":"50cf14e1cf4c7e0a00bd72f24d0919f14115faba.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 08/14] repack-midx: factor out `repack_prepare_midx_command()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:33Z","receivedAt":"2026-02-25T00:21:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `write_midx_included_packs()` function assembles and executes a\n`git multi-pack-index write` command, constructing the argument list\ninline.\n\nFuture commits will introduce additional callers that need to construct\nsimilar `git multi-pack-index` commands (for both `write` and `compact`\nsubcommands), so extract the common portions of the command setup into a\nreusable `repack_prepare_midx_command()` helper.\n\nThe extracted helper sets `git_cmd`, pushes the `multi-pack-index`\nsubcommand and verb, and handles `--progress`/`--no-progress` and\n`--bitmap` flags. The remaining arguments that are specific to the\n`write` subcommand (such as `--stdin-packs`) are left to the caller.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 30 +++++++++++++++++++-----------\n 1 file changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 74bdfa3a6e9..7547efc0651 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -283,6 +283,23 @@ static void remove_redundant_bitmaps(struct string_list *include,\n \tstrbuf_release(&path);\n }\n \n+static void repack_prepare_midx_command(struct child_process *cmd,\n+\t\t\t\t\tstruct repack_write_midx_opts *opts,\n+\t\t\t\t\tconst char *verb)\n+{\n+\tcmd->git_cmd = 1;\n+\n+\tstrvec_pushl(&cmd->args, \"multi-pack-index\", verb, NULL);\n+\n+\tif (opts->show_progress)\n+\t\tstrvec_push(&cmd->args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd->args, \"--no-progress\");\n+\n+\tif (opts->write_bitmaps)\n+\t\tstrvec_push(&cmd->args, \"--bitmap\");\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -297,18 +314,9 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \n \tcmd.in = -1;\n-\tcmd.git_cmd = 1;\n \n-\tstrvec_push(&cmd.args, \"multi-pack-index\");\n-\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n-\n-\tif (opts->show_progress)\n-\t\tstrvec_push(&cmd.args, \"--progress\");\n-\telse\n-\t\tstrvec_push(&cmd.args, \"--no-progress\");\n-\n-\tif (opts->write_bitmaps)\n-\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537053","messageId":"e06fa5da501200d8d4fc2ae5884f2dc38214a1fc.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 09/14] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:37Z","receivedAt":"2026-02-25T00:21:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `write_midx_included_packs()` manages the lifecycle of\nwriting packs to stdin when running `git multi-pack-index write` as a\nchild process.\n\nExtract a standalone `repack_fill_midx_stdin_packs()` helper, which\nhandles `--stdin-packs` argument setup, starting the command, writing\npack names to its standard input, and finishing the command.\n\nThis simplifies `write_midx_included_packs()` and prepares for a\nsubsequent commit where the same helper is called with `cmd->out = -1`\nto capture the MIDX's checksum from the command's standard output,\nwhich is needed when writing MIDX layers with `--checksum-only`.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 38 ++++++++++++++++++++++++--------------\n 1 file changed, 24 insertions(+), 14 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 7547efc0651..bc5059927f7 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -300,23 +300,42 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n \t\tstrvec_push(&cmd->args, \"--bitmap\");\n }\n \n+static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n+\t\t\t\t\tstruct string_list *include)\n+{\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tcmd->in = -1;\n+\n+\tstrvec_push(&cmd->args, \"--stdin-packs\");\n+\n+\tret = start_command(cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd->in, \"w\");\n+\tfor_each_string_list_item(item, include)\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_command(cmd);\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *item;\n \tstruct packed_git *preferred = pack_geometry_preferred_pack(opts->geometry);\n-\tFILE *in;\n \tint ret = 0;\n \n \tmidx_included_packs(&include, opts);\n \tif (!include.nr)\n \t\tgoto done;\n \n-\tcmd.in = -1;\n-\n \trepack_prepare_midx_command(&cmd, opts, \"write\");\n-\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n@@ -358,16 +377,7 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = start_command(&cmd);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tin = xfdopen(cmd.in, \"w\");\n-\tfor_each_string_list_item(item, &include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n-\n-\tret = finish_command(&cmd);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537054","messageId":"d81578120945e7bb24aa552a347be57020dbcf62.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 10/14] repack-geometry: prepare for incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:45Z","receivedAt":"2026-02-25T00:21:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach `pack_geometry_init()` to optionally restrict the set of\nrepacking candidates to only packs in the tip MIDX layer when a\n`midx_layer_threshold` is configured. If the tip layer has fewer packs\nthan the threshold, those packs are excluded entirely; otherwise only\npacks in that layer participate in the geometric repack.\n\nAlso track whether any tip-layer packs were included in the rollup\n(`midx_tip_rewritten`), which a subsequent commit will use to decide\nhow to update the MIDX chain after repacking.\n\nSigned-off-by: Taylor Blau \n---\n repack-geometry.c | 37 +++++++++++++++++++++++++++++++++++++\n repack.h | 4 ++++\n 2 files changed, 41 insertions(+)\n\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 7cebd0cb45f..d2065205f87 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -4,6 +4,7 @@\n #include \"repack.h\"\n #include \"repository.h\"\n #include \"hex.h\"\n+#include \"midx.h\"\n #include \"packfile.h\"\n \n static uint32_t pack_geometry_weight(struct packed_git *p)\n@@ -31,8 +32,30 @@ void pack_geometry_init(struct pack_geometry *geometry,\n {\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstruct multi_pack_index *m = get_multi_pack_index(existing->source);\n \n \trepo_for_each_pack(existing->repo, p) {\n+\t\tif (geometry->midx_layer_threshold_set && m &&\n+\t\t p->multi_pack_index) {\n+\t\t\t/*\n+\t\t\t * When writing MIDX layers incrementally,\n+\t\t\t * ignore packs unless they are in the most\n+\t\t\t * recent MIDX layer *and* there are at least\n+\t\t\t * 'midx_layer_threshold' packs in that layer.\n+\t\t\t *\n+\t\t\t * Otherwise 'p' is either in an older layer, or\n+\t\t\t * the youngest layer does not have enough packs\n+\t\t\t * to consider its packs as candidates for\n+\t\t\t * repacking. In either of those cases we want\n+\t\t\t * to ignore the pack.\n+\t\t\t */\n+\t\t\tif (m->num_packs > geometry->midx_layer_threshold &&\n+\t\t\t midx_layer_contains_pack(m, pack_basename(p)))\n+\t\t\t\t;\n+\t\t\telse\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -173,6 +196,20 @@ void pack_geometry_split(struct pack_geometry *geometry)\n \tgeometry->promisor_split = compute_pack_geometry_split(geometry->promisor_pack,\n \t\t\t\t\t\t\t geometry->promisor_pack_nr,\n \t\t\t\t\t\t\t geometry->split_factor);\n+\tfor (uint32_t i = 0; i < geometry->split; i++) {\n+\t\tstruct packed_git *p = geometry->pack[i];\n+\t\t/*\n+\t\t * During incremental MIDX/bitmap repacking, any packs\n+\t\t * included in the rollup are either (a) not MIDX'd, or\n+\t\t * (b) contained in the tip layer iff it has more than\n+\t\t * the threshold number of packs.\n+\t\t *\n+\t\t * In the latter case, we can safely conclude that the\n+\t\t * tip of the MIDX chain will be rewritten.\n+\t\t */\n+\t\tif (p->multi_pack_index)\n+\t\t\tgeometry->midx_tip_rewritten = true;\n+\t}\n }\n \n struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry)\ndiff --git a/repack.h b/repack.h\nindex c0e9f0ca647..77d24ee45fb 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -108,6 +108,10 @@ struct pack_geometry {\n \tuint32_t promisor_pack_nr, promisor_pack_alloc;\n \tuint32_t promisor_split;\n \n+\tuint32_t midx_layer_threshold;\n+\tbool midx_layer_threshold_set;\n+\tbool midx_tip_rewritten;\n+\n \tint split_factor;\n };\n \n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537055","messageId":"a234852eba45cbd988f44c676e0d9e63b27de7e5.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 11/14] builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:51Z","receivedAt":"2026-02-25T00:21:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Change the --write-midx (-m) flag from an OPT_BOOL to an OPT_CALLBACK\nthat accepts an optional mode argument. Introduce an enum with\nREPACK_WRITE_MIDX_NONE and REPACK_WRITE_MIDX_DEFAULT to distinguish\nbetween the two states, and update all existing boolean checks\naccordingly.\n\nFor now, passing no argument (or just `-m`) selects the default mode,\npreserving existing behavior. A subsequent commit will add a new mode\nfor writing incremental MIDXs.\n\nExtract repack_write_midx() as a dispatcher that selects the\nappropriate MIDX-writing implementation based on the mode.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 50 ++++++++++++++++++++++++++++++++++++------------\n repack-midx.c | 14 +++++++++++++-\n repack.h | 8 +++++++-\n 3 files changed, 58 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 44a95b56f23..3a5042491d6 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -97,6 +97,24 @@ static int repack_config(const char *var, const char *value,\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n+static int option_parse_write_midx(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum repack_write_midx_mode *cfg = opt->value;\n+\n+\tif (unset) {\n+\t\t*cfg = REPACK_WRITE_MIDX_NONE;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!arg || !*arg)\n+\t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse\n+\t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_repack(int argc,\n \t const char **argv,\n \t const char *prefix,\n@@ -119,7 +137,7 @@ int cmd_repack(int argc,\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct pack_objects_args po_args = PACK_OBJECTS_ARGS_INIT;\n \tstruct pack_objects_args cruft_po_args = PACK_OBJECTS_ARGS_INIT;\n-\tint write_midx = 0;\n+\tenum repack_write_midx_mode write_midx = REPACK_WRITE_MIDX_NONE;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n \tconst char *filter_to = NULL;\n@@ -185,8 +203,14 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometry.split_factor,\n \t\t\t N_(\"find a geometric progression with factor \")),\n-\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n-\t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n+\t\tOPT_CALLBACK_F(0, \"write-midx\", &write_midx,\n+\t\t\t N_(\"mode\"),\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t PARSE_OPT_OPTARG, option_parse_write_midx),\n+\t\tOPT_SET_INT_F('m', NULL, &write_midx,\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t REPACK_WRITE_MIDX_DEFAULT,\n+\t\t\t PARSE_OPT_HIDDEN),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n \t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n@@ -221,14 +245,16 @@ int cmd_repack(int argc,\n \t\tpack_everything |= ALL_INTO_ONE;\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!write_midx &&\n+\t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t}\n \tif (po_args.pack_kept_objects < 0)\n-\t\tpo_args.pack_kept_objects = write_bitmaps > 0 && !write_midx;\n+\t\tpo_args.pack_kept_objects = write_bitmaps > 0 &&\n+\t\t\twrite_midx == REPACK_WRITE_MIDX_NONE;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) &&\n+\t write_midx == REPACK_WRITE_MIDX_NONE)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (write_bitmaps && po_args.local &&\n@@ -244,7 +270,7 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n-\tif (write_midx && write_bitmaps) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n \t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\",\n@@ -297,7 +323,7 @@ int cmd_repack(int argc,\n \t}\n \tif (repo_has_promisor_remote(repo))\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (!write_midx) {\n+\tif (write_midx == REPACK_WRITE_MIDX_NONE) {\n \t\tif (write_bitmaps > 0)\n \t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n \t\telse if (write_bitmaps < 0)\n@@ -504,7 +530,7 @@ int cmd_repack(int argc,\n \tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n \n-\tif (write_midx) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n \t\t\t.existing = &existing,\n \t\t\t.geometry = &geometry,\n@@ -513,11 +539,11 @@ int cmd_repack(int argc,\n \t\t\t.packdir = packdir,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n-\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft\n+\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.mode = write_midx,\n \t\t};\n \n-\t\tret = write_midx_included_packs(&opts);\n-\n+\t\tret = repack_write_midx(&opts);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \t}\ndiff --git a/repack-midx.c b/repack-midx.c\nindex bc5059927f7..6c28d9acef6 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -323,7 +323,7 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n-int write_midx_included_packs(struct repack_write_midx_opts *opts)\n+static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n@@ -386,3 +386,15 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \n \treturn ret;\n }\n+\n+int repack_write_midx(struct repack_write_midx_opts *opts)\n+{\n+\tswitch (opts->mode) {\n+\tcase REPACK_WRITE_MIDX_NONE:\n+\t\tBUG(\"write_midx mode is NONE?\");\n+\tcase REPACK_WRITE_MIDX_DEFAULT:\n+\t\treturn write_midx_included_packs(opts);\n+\tdefault:\n+\t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n+\t}\n+}\ndiff --git a/repack.h b/repack.h\nindex 77d24ee45fb..81907fcce7f 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -134,6 +134,11 @@ void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\n \n+enum repack_write_midx_mode {\n+\tREPACK_WRITE_MIDX_NONE,\n+\tREPACK_WRITE_MIDX_DEFAULT,\n+};\n+\n struct repack_write_midx_opts {\n \tstruct existing_packs *existing;\n \tstruct pack_geometry *geometry;\n@@ -143,10 +148,11 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tenum repack_write_midx_mode mode;\n };\n \n void midx_snapshot_refs(struct repository *repo, struct tempfile *f);\n-int write_midx_included_packs(struct repack_write_midx_opts *opts);\n+int repack_write_midx(struct repack_write_midx_opts *opts);\n \n int write_filtered_pack(const struct write_pack_opts *opts,\n \t\t\tstruct existing_packs *existing,\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537056","messageId":"abcd8d303eab067b83e6b11b7608a27d48ac9227.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 12/14] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:21:57Z","receivedAt":"2026-02-25T00:22:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the `write_midx_incremental()` function, which builds and\nmaintains an incremental MIDX chain as part of the geometric repacking\nprocess.\n\nUnlike the default mode which writes a single flat MIDX, the incremental\nmode constructs a compaction plan that determines which MIDX layers to\nwrite, compact, or copy, and then executes each step using `git\nmulti-pack-index` subcommands with the --checksum-only flag.\n\nThe repacking strategy works as follows:\n\n * Acquire the lock guarding the multi-pack-index-chain.\n\n * A new MIDX layer is always written containing the newly created\n pack(s). If the tip MIDX layer was rewritten during geometric\n repacking, any surviving packs from that layer are also included.\n\n * Starting from the new layer, adjacent MIDX layers are merged together\n as long as the accumulated object count exceeds half the object count\n of the next deeper layer (controlled by 'repack.midxSplitFactor').\n\n * Remaining layers in the chain are evaluated pairwise and either\n compacted or copied as-is, following the same merging condition.\n\n * Write the contents of the new multi-pack-index chain, atomically move\n it into place, and then release the lock.\n\n * Delete any now-unused MIDX layers.\n\nAfter writing the new layer, the strategy is evaluated among the\nexisting MIDX layers in order from oldest to newest. Each step that\nwrites a new MIDX layer uses \"--checksum-only\" to avoid updating the\nmulti-pack-index-chain file. After all steps are complete, the new chain\nfile is written and then atomically moved into place.\n\nAt present, this functionality is exposed behind a new enum value,\n`REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A\nsubsequent commit will expose this mode via `git repack\n--write-midx=incremental`.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 475 +++++++++++++++++++++++++++++++++++++++++++++++++-\n repack.h | 3 +\n 2 files changed, 476 insertions(+), 2 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 6c28d9acef6..ad78a10378b 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -2,9 +2,12 @@\n #include \"repack.h\"\n #include \"hash.h\"\n #include \"hex.h\"\n+#include \"lockfile.h\"\n+#include \"midx.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n #include \"pack-bitmap.h\"\n+#include \"path.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n #include \"tempfile.h\"\n@@ -301,13 +304,16 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n }\n \n static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n-\t\t\t\t\tstruct string_list *include)\n+\t\t\t\t\tstruct string_list *include,\n+\t\t\t\t\tstruct string_list *out)\n {\n \tstruct string_list_item *item;\n \tFILE *in;\n \tint ret;\n \n \tcmd->in = -1;\n+\tif (out)\n+\t\tcmd->out = -1;\n \n \tstrvec_push(&cmd->args, \"--stdin-packs\");\n \n@@ -320,6 +326,17 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \t\tfprintf(in, \"%s\\n\", item->string);\n \tfclose(in);\n \n+\tif (out) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tFILE *outf = xfdopen(cmd->out, \"r\");\n+\n+\t\twhile (strbuf_getline(&buf, outf) != EOF)\n+\t\t\tstring_list_append(out, buf.buf);\n+\t\tstrbuf_release(&buf);\n+\n+\t\tfclose(outf);\n+\t}\n+\n \treturn finish_command(cmd);\n }\n \n@@ -377,7 +394,7 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include, NULL);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n@@ -387,6 +404,458 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \treturn ret;\n }\n \n+struct midx_compaction_step {\n+\tunion {\n+\t\tstruct multi_pack_index *copy;\n+\t\tstruct string_list write;\n+\t\tstruct {\n+\t\t\tstruct multi_pack_index *from;\n+\t\t\tstruct multi_pack_index *to;\n+\t\t} compact;\n+\t} u;\n+\n+\tuint32_t objects_nr;\n+\tconst char *csum;\n+\n+\tenum {\n+\t\tMIDX_COMPACTION_STEP_UNKNOWN,\n+\t\tMIDX_COMPACTION_STEP_COPY,\n+\t\tMIDX_COMPACTION_STEP_WRITE,\n+\t\tMIDX_COMPACTION_STEP_COMPACT,\n+\t} type;\n+};\n+\n+static const char *midx_compaction_step_base(const struct midx_compaction_step *step)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot use UNKNOWN step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_get_checksum_hex(step->u.copy);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\tBUG(\"cannot use WRITE step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_get_checksum_hex(step->u.compact.to);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static int midx_compaction_step_exec_copy(struct midx_compaction_step *step)\n+{\n+\tstep->csum = xstrdup(midx_get_checksum_hex(step->u.copy));\n+\treturn 0;\n+}\n+\n+static int midx_compaction_step_exec_write(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t\t const char *base)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list hash = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\tconst char *preferred_pack = NULL;\n+\tint ret = 0;\n+\n+\tif (!step->u.write.nr) {\n+\t\tret = error(_(\"no packs to write MIDX during compaction\"));\n+\t\tgoto out;\n+\t}\n+\n+\tfor_each_string_list_item(item, &step->u.write) {\n+\t\tif (item->util)\n+\t\t\tpreferred_pack = item->string;\n+\t}\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\", NULL);\n+\tstrvec_pushf(&cmd.args, \"--base=%s\", base ? base : \"none\");\n+\n+\tif (preferred_pack) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\n+\t\tstrbuf_addstr(&buf, preferred_pack);\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\t\tstrbuf_addstr(&buf, \".pack\");\n+\n+\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\", buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tret = repack_fill_midx_stdin_packs(&cmd, &step->u.write, &hash);\n+\tif (hash.nr != 1) {\n+\t\tret = error(_(\"expected exactly one line during MIDX write, \"\n+\t\t\t \"got: %\"PRIuMAX),\n+\t\t\t (uintmax_t)hash.nr);\n+\t\tgoto out;\n+\t}\n+\n+\tstep->csum = xstrdup(hash.items[0].string);\n+\n+out:\n+\tstring_list_clear(&hash, 0);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tFILE *out = NULL;\n+\tint ret;\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"compact\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\",\n+\t\t midx_get_checksum_hex(step->u.compact.from),\n+\t\t midx_get_checksum_hex(step->u.compact.to), NULL);\n+\n+\tcmd.out = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\tgoto out;\n+\n+\tout = xfdopen(cmd.out, \"r\");\n+\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n+\t\tif (step->csum) {\n+\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep->csum = strbuf_detach(&buf, NULL);\n+\t}\n+\n+\tret = finish_command(&cmd);\n+\n+out:\n+\tif (out)\n+\t\tfclose(out);\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec(struct midx_compaction_step *step,\n+\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t const char *base)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot execute UNKNOWN midx compaction step\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_compaction_step_exec_copy(step);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\treturn midx_compaction_step_exec_write(step, opts, base);\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_compaction_step_exec_compact(step, opts);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step step = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\tuint32_t i;\n+\tint ret = 0;\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tfor (i = 0; m && i < m->num_packs + m->num_packs_in_base; i++) {\n+\t\tif (prepare_midx_pack(m, i)) {\n+\t\t\tret = error(_(\"could not load pack %\"PRIu32\" from MIDX\"),\n+\t\t\t\t i);\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * The first MIDX in the resulting chain is always going to be\n+\t * new.\n+\t *\n+\t * At a minimum, it will include all of the newly written packs.\n+\t * If there is an existing MIDX whose tip layer contain packs\n+\t * that were repacked, it will also include any of its pack\n+\t * which were *not* rolled up as part of the geometric repack\n+\t * (if any), and the previous tip will be replaced.\n+\t *\n+\t * It may grow to include the packs from zero or more MIDXs from\n+\t * the old chain, beginning either at the old tip (if the MIDX\n+\t * was *not* rewritten) or the old tip's base MIDX layer\n+\t * (otherwise).\n+\t */\n+\n+\tstep.type = MIDX_COMPACTION_STEP_WRITE;\n+\tstring_list_init_nodup(&step.u.write);\n+\n+\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\tstrbuf_addf(&buf, \"pack-%s.idx\", opts->names->items[i].string);\n+\t\tstring_list_append(&step.u.write, strbuf_detach(&buf, NULL));\n+\t}\n+\tfor (i = 0; i < opts->geometry->split; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\n+\t/*\n+\t * Now handle any existing packs which were *not* rewritten.\n+\t *\n+\t * The list of packs in opts->geometry only contains MIDX'd\n+\t * packs from the newest layer when that layer has more than\n+\t * 'repack.midxNewLayerThreshold' number of packs.\n+\t *\n+\t * If the MIDX tip was rewritten (that is, one or more of those\n+\t * packs appear below the split line), then add all packs above\n+\t * the split line to the new layer, as the old one is no longer\n+\t * usable.\n+\t *\n+\t * If the MIDX tip was not rewritten (that is, all MIDX'd packs\n+\t * from the youngest layer appear below the split line, or were\n+\t * not included in the geometric repack at all because there\n+\t * were too few of them), ignore them since we'll retain the\n+\t * existing layer as-is.\n+\t */\n+\tfor (i = opts->geometry->split; i < opts->geometry->pack_nr; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tstruct string_list_item *item;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\tif (p->multi_pack_index && !opts->geometry->midx_tip_rewritten)\n+\t\t\tcontinue;\n+\n+\t\titem = string_list_append(&step.u.write,\n+\t\t\t\t\t strbuf_detach(&buf, NULL));\n+\t\tif (p->multi_pack_index || i == opts->geometry->pack_nr - 1)\n+\t\t\titem->util = (void *)1; /* mark as preferred */\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\n+\t/*\n+\t * If the MIDX tip was rewritten, then we no longer consider it\n+\t * a candidate for compaction, since it will not exist in the\n+\t * MIDX chain being built.\n+\t */\n+\tif (opts->geometry->midx_tip_rewritten)\n+\t\tm = m->base_midx;\n+\n+\t/*\n+\t * Compact additional MIDX layers into this proposed one until\n+\t * the merging condition is violated.\n+\t */\n+\twhile (m) {\n+\t\tuint32_t preferred_pack_idx;\n+\n+\t\tif (step.objects_nr < m->num_objects / opts->midx_split_factor) {\n+\t\t\t/*\n+\t\t\t * Stop compacting MIDX layer as soon as the\n+\t\t\t * merged size is less than half the size of the\n+\t\t\t * next layer in the chain.\n+\t\t\t */\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (midx_preferred_pack(m, &preferred_pack_idx) < 0) {\n+\t\t\tret = error(_(\"could not find preferred pack for MIDX \"\n+\t\t\t\t \"%s\"), midx_get_checksum_hex(m));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tuint32_t pack_int_id = i + m->num_packs_in_base;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\titem = string_list_append(&step.u.write,\n+\t\t\t\t\t\t strbuf_detach(&buf, NULL));\n+\t\t\tif (pack_int_id == preferred_pack_idx)\n+\t\t\t\titem->util = (void *)1; /* mark as preferred */\n+\t\t}\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, m->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep.objects_nr += m->num_objects;\n+\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tif (step.u.write.nr > 0) {\n+\t\t/*\n+\t\t * As long as there is at least one new pack to write\n+\t\t * (and thus the MIDX is non-empty), add it to the plan.\n+\t\t */\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\t\tsteps[steps_nr++] = step;\n+\t}\n+\n+\t/*\n+\t * Then start over, repeat, and either compact or keep as-is\n+\t * each MIDX layer until we have exhausted the chain.\n+\t *\n+\t * Finally, evaluate the remainder of the chain (if any) and\n+\t * either compact a sequence of adjacent layers, or keep\n+\t * individual layers as-is according to the same merging\n+\t * condition as above.\n+\t */\n+\twhile (m) {\n+\t\tstruct multi_pack_index *next = m;\n+\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\n+\t\tmemset(&step, 0, sizeof(step));\n+\t\tstep.type = MIDX_COMPACTION_STEP_UNKNOWN;\n+\n+\t\twhile (next) {\n+\t\t\tstruct multi_pack_index *base = next->base_midx;\n+\t\t\tuint32_t proposed_objects_nr;\n+\n+\t\t\tif (unsigned_add_overflows(step.objects_nr, next->num_objects)) {\n+\t\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tproposed_objects_nr = step.objects_nr + next->num_objects;\n+\n+\t\t\tif (!base) {\n+\t\t\t\t/*\n+\t\t\t\t * If we are at the end of the MIDX\n+\t\t\t\t * chain, there is nothing to compact,\n+\t\t\t\t * so mark it and stop.\n+\t\t\t\t */\n+\t\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (proposed_objects_nr < base->num_objects / opts->midx_split_factor) {\n+\t\t\t\t/*\n+\t\t\t\t * If there is a MIDX following this\n+\t\t\t\t * one, but our accumulated size is less\n+\t\t\t\t * than half of its size, compacting\n+\t\t\t\t * them would violate the merging\n+\t\t\t\t * condition, so stop here.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, it is OK to compact the next layer\n+\t\t\t * into this one. Do so, and then continue\n+\t\t\t * through the remainder of the chain.\n+\t\t\t */\n+\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\tnext = base;\n+\t\t}\n+\n+\t\tif (m == next) {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COPY;\n+\t\t\tstep.u.copy = m;\n+\t\t} else {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COMPACT;\n+\t\t\tstep.u.compact.from = next;\n+\t\t\tstep.u.compact.to = m;\n+\t\t}\n+\n+\t\tm = next->base_midx;\n+\n+\t\tsteps[steps_nr++] = step;\n+\t}\n+\n+out:\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+\n+\treturn ret;\n+}\n+\n+static int write_midx_incremental(struct repack_write_midx_opts *opts)\n+{\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct strbuf lock_name = STRBUF_INIT;\n+\tstruct lock_file lf;\n+\tsize_t steps_nr = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tget_midx_chain_filename(opts->existing->source, &lock_name);\n+\tif (safe_create_leading_directories(opts->existing->repo,\n+\t\t\t\t\t lock_name.buf))\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t lock_name.buf);\n+\thold_lock_file_for_update(&lf, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\n+\tif (!fdopen_lock_file(&lf, \"w\")) {\n+\t\tret = error_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\tgoto done;\n+\t}\n+\n+\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < steps_nr; i++) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tchar *base = NULL;\n+\n+\t\tif (i + 1 < steps_nr)\n+\t\t\tbase = xstrdup(midx_compaction_step_base(&steps[i + 1]));\n+\n+\t\tif (midx_compaction_step_exec(step, opts, base) < 0) {\n+\t\t\tret = error(_(\"unable to execute compaction step %\"PRIuMAX),\n+\t\t\t\t (uintmax_t)i);\n+\t\t\tfree(base);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfree(base);\n+\t}\n+\n+\ti = steps_nr;\n+\twhile (i--) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tif (!step->csum)\n+\t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n+\t\t\t (uintmax_t)i);\n+\t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t}\n+\n+\tcommit_lock_file(&lf);\n+\n+done:\n+\tstrbuf_release(&lock_name);\n+\tfree(steps);\n+\treturn ret;\n+}\n+\n int repack_write_midx(struct repack_write_midx_opts *opts)\n {\n \tswitch (opts->mode) {\n@@ -394,6 +863,8 @@ int repack_write_midx(struct repack_write_midx_opts *opts)\n \t\tBUG(\"write_midx mode is NONE?\");\n \tcase REPACK_WRITE_MIDX_DEFAULT:\n \t\treturn write_midx_included_packs(opts);\n+\tcase REPACK_WRITE_MIDX_INCREMENTAL:\n+\t\treturn write_midx_incremental(opts);\n \tdefault:\n \t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n \t}\ndiff --git a/repack.h b/repack.h\nindex 81907fcce7f..831ccfb1c6c 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -137,6 +137,7 @@ struct tempfile;\n enum repack_write_midx_mode {\n \tREPACK_WRITE_MIDX_NONE,\n \tREPACK_WRITE_MIDX_DEFAULT,\n+\tREPACK_WRITE_MIDX_INCREMENTAL,\n };\n \n struct repack_write_midx_opts {\n@@ -148,6 +149,8 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n \tenum repack_write_midx_mode mode;\n };\n \n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537057","messageId":"27d15fd735df9c68e29372678b80b289f1ebca2a.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 13/14] repack: introduce `--write-midx=incremental`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:22:02Z","receivedAt":"2026-02-25T00:22:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Expose the incremental MIDX repacking mode (implemented in the previous\ncommit) via a new --write-midx=incremental option for `git repack`.\n\nAdd \"incremental\" as a recognized argument to the --write-midx\nOPT_CALLBACK, mapping it to REPACK_WRITE_MIDX_INCREMENTAL. When this\nmode is active and --geometric is in use, set the midx_layer_threshold\non the pack geometry so that only packs in sufficiently large tip layers\nare considered for repacking.\n\nTwo new configuration options control the compaction behavior:\n\n - repack.midxSplitFactor (default: 2): the factor used in the\n geometric merging condition for MIDX layers.\n\n - repack.midxNewLayerThreshold (default: 8): the minimum number of\n packs in the tip MIDX layer before its packs are considered as\n candidates for geometric repacking.\n\nAdd tests exercising the new mode across a variety of scenarios\nincluding basic geometric violations, multi-round chain integrity,\nbranching and merging histories, cross-layer object uniqueness, and\nthreshold-based compaction.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 18 ++\n Documentation/git-repack.adoc | 39 ++-\n builtin/repack.c | 38 ++-\n midx.c | 30 ++\n midx.h | 3 +\n repack-geometry.c | 13 +-\n repack-midx.c | 113 +++++++-\n repack.c | 21 +-\n repack.h | 9 +-\n t/meson.build | 1 +\n t/t7705-repack-incremental-midx.sh | 436 +++++++++++++++++++++++++++++\n 11 files changed, 692 insertions(+), 29 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex e9e78dcb198..054de9f8795 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -46,3 +46,21 @@ repack.midxMustContainCruft::\n \t`--write-midx`. When false, cruft packs are only included in the MIDX\n \twhen necessary (e.g., because they might be required to form a\n \treachability closure with MIDX bitmaps). Defaults to true.\n+\n+repack.midxSplitFactor::\n+\tThe factor used in the geometric merging condition when\n+\tcompacting incremental MIDX layers during `git repack` when\n+\tinvoked with the `--write-midx=incremental` option.\n++\n+Adjacent layers are merged when the accumulated object count of the\n+newer layer exceeds `1/` of the object count of the next deeper\n+layer. Defaults to 2.\n+\n+repack.midxNewLayerThreshold::\n+\tThe minimum number of packs in the tip MIDX layer before those\n+\tpacks are considered as candidates for geometric repacking\n+\tduring `git repack --write-midx=incremental`.\n++\n+When the tip layer has fewer packs than this threshold, those packs are\n+excluded from the geometric repack entirely, and are thus left\n+unmodified. Defaults to 8.\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 673ce910837..27a99cc46f4 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=] [--depth=] [--threads=] [--keep-pack=]\n-\t[--write-midx] [--name-hash-version=] [--path-walk]\n+\t[--write-midx[=]] [--name-hash-version=] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -250,9 +250,42 @@ pack as the preferred pack for object selection by the MIDX (see\n linkgit:git-multi-pack-index[1]).\n \n -m::\n---write-midx::\n+--write-midx[=]::\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n-\tcontaining the non-redundant packs.\n+\tcontaining the non-redundant packs. The following modes are\n+\tavailable:\n++\n+--\n+\t`default`;;\n+\t\tWrite a single MIDX covering all packs. This is the\n+\t\tdefault when `--write-midx` is given without an\n+\t\texplicit mode.\n+\n+\t`incremental`;;\n+\t\tWrite an incremental MIDX chain instead of a single\n+\t\tflat MIDX. This mode requires `--geometric`.\n++\n+The incremental mode maintains a chain of MIDX layers that is compacted\n+over time using a geometric merging strategy. Each repack creates a new\n+tip layer containing the newly written pack(s). Adjacent layers are then\n+merged whenever the newer layer's object count exceeds\n+`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n+that do not meet this condition are retained as-is.\n++\n+The result is that newer (tip) layers tend to contain many small packs\n+with relatively few objects, while older (deeper) layers contain fewer,\n+larger packs covering more objects. Because compaction is driven by the\n+tip of the chain, newer layers are also rewritten more frequently than\n+older ones, which are only touched when enough objects have accumulated\n+to justify merging into them. This keeps the total number of layers\n+logarithmic relative to the total number of objects.\n++\n+Only packs in the tip MIDX layer are considered as candidates for the\n+geometric repack; packs in deeper layers are left untouched. If the tip\n+layer contains fewer packs than `repack.midxNewLayerThreshold`, those\n+packs are excluded from the geometry entirely, and a new layer is\n+created for any new pack(s) without disturbing the existing chain.\n+--\n \n --name-hash-version=::\n \tProvide this argument to the underlying `git pack-objects` process.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 3a5042491d6..9e070f35868 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -33,7 +33,7 @@ static int midx_must_contain_cruft = 1;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t \"[--window=] [--depth=] [--threads=] [--keep-pack=]\\n\"\n-\t \"[--write-midx] [--name-hash-version=] [--path-walk]\"),\n+\t \"[--write-midx[=]] [--name-hash-version=] [--path-walk]\"),\n \tNULL\n };\n \n@@ -42,9 +42,14 @@ static const char incremental_bitmap_conflict_error[] = N_(\n \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n );\n \n+#define DEFAULT_MIDX_SPLIT_FACTOR 2\n+#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8\n+\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -94,6 +99,16 @@ static int repack_config(const char *var, const char *value,\n \t\tmidx_must_contain_cruft = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"repack.midxsplitfactor\")) {\n+\t\trepack_ctx->midx_split_factor = git_config_int(var, value,\n+\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"repack.midxnewlayerthreshold\")) {\n+\t\trepack_ctx->midx_new_layer_threshold = git_config_int(var, value,\n+\t\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -109,6 +124,8 @@ static int option_parse_write_midx(const struct option *opt, const char *arg,\n \n \tif (!arg || !*arg)\n \t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse if (!strcmp(arg, \"incremental\"))\n+\t\t*cfg = REPACK_WRITE_MIDX_INCREMENTAL;\n \telse\n \t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n \n@@ -223,6 +240,8 @@ int cmd_repack(int argc,\n \tmemset(&config_ctx, 0, sizeof(config_ctx));\n \tconfig_ctx.po_args = &po_args;\n \tconfig_ctx.cruft_po_args = &cruft_po_args;\n+\tconfig_ctx.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR;\n+\tconfig_ctx.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD;\n \n \trepo_config(repo, repack_config, &config_ctx);\n \n@@ -244,6 +263,9 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n+\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n+\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n+\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n@@ -293,6 +315,10 @@ int cmd_repack(int argc,\n \tif (geometry.split_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--geometric\", \"-A/-a\");\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL) {\n+\t\t\tgeometry.midx_layer_threshold = config_ctx.midx_new_layer_threshold;\n+\t\t\tgeometry.midx_layer_threshold_set = true;\n+\t\t}\n \t\tpack_geometry_init(&geometry, &existing, &po_args);\n \t\tpack_geometry_split(&geometry);\n \t}\n@@ -540,6 +566,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.midx_split_factor = config_ctx.midx_split_factor,\n+\t\t\t.midx_new_layer_threshold = config_ctx.midx_new_layer_threshold,\n \t\t\t.mode = write_midx,\n \t\t};\n \n@@ -552,11 +580,15 @@ int cmd_repack(int argc,\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\texisting_packs_remove_redundant(&existing, packdir);\n+\t\tbool wrote_incremental_midx = write_midx == REPACK_WRITE_MIDX_INCREMENTAL;\n+\n+\t\texisting_packs_remove_redundant(&existing, packdir,\n+\t\t\t\t\t\twrote_incremental_midx);\n \n \t\tif (geometry.split_factor)\n \t\t\tpack_geometry_remove_redundant(&geometry, &names,\n-\t\t\t\t\t\t &existing, packdir);\n+\t\t\t\t\t\t &existing, packdir,\n+\t\t\t\t\t\t wrote_incremental_midx);\n \t\tif (show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\ndiff --git a/midx.c b/midx.c\nindex 709fe7c3fd7..5f024d061e6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -837,6 +837,36 @@ void clear_midx_file(struct repository *r)\n \tstrbuf_release(&midx);\n }\n \n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes)\n+{\n+\tstruct strbuf chain = STRBUF_INIT;\n+\n+\tget_midx_chain_filename(r->objects->sources, &chain);\n+\n+\tif (r->objects) {\n+\t\tstruct odb_source *source = r->objects->sources;\n+\t\tfor (; source; source = source->next) {\n+\t\t\tif (source->packfiles->midx)\n+\t\t\t\tclose_midx(source->packfiles->midx);\n+\t\t\tsource->packfiles->midx = NULL;\n+\t\t}\n+\t}\n+\n+\tif (!keep_hashes && remove_path(chain.buf))\n+\t\tdie(_(\"failed to clear multi-pack-index chain at %s\"),\n+\t\t chain.buf);\n+\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_BITMAP,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_REV,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_MIDX,\n+\t\t\t\t\t keep_hashes);\n+\n+\tstrbuf_release(&chain);\n+}\n+\n static int verify_midx_error;\n \n __attribute__((format (printf, 1, 2)))\ndiff --git a/midx.h b/midx.h\nindex f211a38b9e7..b45da0a3144 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -9,6 +9,7 @@ struct repository;\n struct bitmapped_pack;\n struct git_hash_algo;\n struct odb_source;\n+struct strvec;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION_V1 1\n@@ -143,6 +144,8 @@ int write_midx_file_compact(struct odb_source *source,\n \t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\n int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags);\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex d2065205f87..5b554da89f1 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -251,7 +251,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t\t\t uint32_t pack_nr,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tconst struct git_hash_algo *algop = existing->repo->hash_algo;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -271,7 +272,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t (string_list_has_string(&existing->kept_packs, buf.buf)))\n \t\t\tcontinue;\n \n-\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf);\n+\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n \n \tstrbuf_release(&buf);\n@@ -280,12 +282,13 @@ static void remove_redundant_packs(struct packed_git **pack,\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs(geometry->pack, geometry->split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n \tremove_redundant_packs(geometry->promisor_pack, geometry->promisor_split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n }\n \n void pack_geometry_release(struct pack_geometry *geometry)\ndiff --git a/repack-midx.c b/repack-midx.c\nindex ad78a10378b..5da67f854c7 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -11,6 +11,7 @@\n #include \"refs.h\"\n #include \"run-command.h\"\n #include \"tempfile.h\"\n+#include \"trace2.h\"\n \n struct midx_snapshot_ref_data {\n \tstruct repository *repo;\n@@ -567,6 +568,9 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \tuint32_t i;\n \tint ret = 0;\n \n+\ttrace2_region_enter(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n \todb_reprepare(opts->existing->repo->objects);\n \tm = get_multi_pack_index(opts->existing->source);\n \n@@ -578,6 +582,8 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t}\n \t}\n \n+\ttrace2_region_enter(\"repack\", \"steps:write\", opts->existing->repo);\n+\n \t/*\n \t * The first MIDX in the resulting chain is always going to be\n \t * new.\n@@ -593,13 +599,16 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t * was *not* rewritten) or the old tip's base MIDX layer\n \t * (otherwise).\n \t */\n-\n \tstep.type = MIDX_COMPACTION_STEP_WRITE;\n \tstring_list_init_nodup(&step.u.write);\n \n \tfor (i = 0; i < opts->names->nr; i++) {\n \t\tstrbuf_addf(&buf, \"pack-%s.idx\", opts->names->items[i].string);\n \t\tstring_list_append(&step.u.write, strbuf_detach(&buf, NULL));\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:fresh\",\n+\t\t\t\t step.u.write.items[step.u.write.nr - 1].string);\n \t}\n \tfor (i = 0; i < opts->geometry->split; i++) {\n \t\tstruct packed_git *p = opts->geometry->pack[i];\n@@ -610,6 +619,9 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \n \t\tstep.objects_nr += p->num_objects;\n \t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:fresh:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n \n \t/*\n \t * Now handle any existing packs which were *not* rewritten.\n@@ -638,8 +650,18 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\tstrbuf_addstr(&buf, \".idx\");\n \n-\t\tif (p->multi_pack_index && !opts->geometry->midx_tip_rewritten)\n+\t\tif (p->multi_pack_index &&\n+\t\t !opts->geometry->midx_tip_rewritten) {\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"exclude:unmodified\", buf.buf);\n \t\t\tcontinue;\n+\t\t}\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified\", buf.buf);\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified:midx\",\n+\t\t\t\t p->multi_pack_index ? \"true\" : \"false\");\n \n \t\titem = string_list_append(&step.u.write,\n \t\t\t\t\t strbuf_detach(&buf, NULL));\n@@ -650,8 +672,12 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n \t\t\tgoto out;\n \t\t}\n+\n \t\tstep.objects_nr += p->num_objects;\n \t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:unmodified:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n \n \t/*\n \t * If the MIDX tip was rewritten, then we no longer consider it\n@@ -661,6 +687,11 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \tif (opts->geometry->midx_tip_rewritten)\n \t\tm = m->base_midx;\n \n+\ttrace2_data_string(\"repack\", opts->existing->repo, \"midx:rewrote-tip\",\n+\t\t\t opts->geometry->midx_tip_rewritten ? \"true\" : \"false\");\n+\n+\ttrace2_region_enter(\"repack\", \"compact\", opts->existing->repo);\n+\n \t/*\n \t * Compact additional MIDX layers into this proposed one until\n \t * the merging condition is violated.\n@@ -668,12 +699,27 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \twhile (m) {\n \t\tuint32_t preferred_pack_idx;\n \n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"candidate\", midx_get_checksum_hex(m));\n+\n \t\tif (step.objects_nr < m->num_objects / opts->midx_split_factor) {\n \t\t\t/*\n \t\t\t * Stop compacting MIDX layer as soon as the\n \t\t\t * merged size is less than half the size of the\n \t\t\t * next layer in the chain.\n \t\t\t */\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"compact\", \"violated\");\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"next_objects_nr\",\n+\t\t\t\t\t (uintmax_t)m->num_objects);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"split_factor\",\n+\t\t\t\t\t (uintmax_t)opts->midx_split_factor);\n+\n \t\t\tbreak;\n \t\t}\n \n@@ -693,6 +739,9 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n \t\t\tstrbuf_addstr(&buf, \".idx\");\n \n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"midx:pack\", buf.buf);\n+\n \t\t\titem = string_list_append(&step.u.write,\n \t\t\t\t\t\t strbuf_detach(&buf, NULL));\n \t\t\tif (pack_int_id == preferred_pack_idx)\n@@ -717,6 +766,16 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\tsteps[steps_nr++] = step;\n \t}\n \n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:objects_nr\", (uintmax_t)step.objects_nr);\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:packs_nr\", (uintmax_t)step.u.write.nr);\n+\n+\ttrace2_region_leave(\"repack\", \"compact\", opts->existing->repo);\n+\ttrace2_region_leave(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\ttrace2_region_enter(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n \t/*\n \t * Then start over, repeat, and either compact or keep as-is\n \t * each MIDX layer until we have exhausted the chain.\n@@ -734,18 +793,29 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\tmemset(&step, 0, sizeof(step));\n \t\tstep.type = MIDX_COMPACTION_STEP_UNKNOWN;\n \n+\t\ttrace2_region_enter(\"repack\", \"step\", opts->existing->repo);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"from\", midx_get_checksum_hex(m));\n+\n \t\twhile (next) {\n-\t\t\tstruct multi_pack_index *base = next->base_midx;\n \t\t\tuint32_t proposed_objects_nr;\n-\n \t\t\tif (unsigned_add_overflows(step.objects_nr, next->num_objects)) {\n \t\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n \t\t\t\tgoto out;\n \t\t\t}\n \n \t\t\tproposed_objects_nr = step.objects_nr + next->num_objects;\n \n-\t\t\tif (!base) {\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed\",\n+\t\t\t\t\t midx_get_checksum_hex(next));\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed:objects_nr\",\n+\t\t\t\t\t (uintmax_t)next->num_objects);\n+\n+\t\t\tif (!next->base_midx) {\n \t\t\t\t/*\n \t\t\t\t * If we are at the end of the MIDX\n \t\t\t\t * chain, there is nothing to compact,\n@@ -755,7 +825,7 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t\tbreak;\n \t\t\t}\n \n-\t\t\tif (proposed_objects_nr < base->num_objects / opts->midx_split_factor) {\n+\t\t\tif (proposed_objects_nr < next->base_midx->num_objects / opts->midx_split_factor) {\n \t\t\t\t/*\n \t\t\t\t * If there is a MIDX following this\n \t\t\t\t * one, but our accumulated size is less\n@@ -763,6 +833,13 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t\t * them would violate the merging\n \t\t\t\t * condition, so stop here.\n \t\t\t\t */\n+\n+\t\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at\",\n+\t\t\t\t\t\t midx_get_checksum_hex(next->base_midx));\n+\t\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at:objects_nr\",\n+\t\t\t\t\t\t (uintmax_t)next->base_midx->num_objects);\n \t\t\t\tbreak;\n \t\t\t}\n \n@@ -772,27 +849,43 @@ static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t * through the remainder of the chain.\n \t\t\t */\n \t\t\tstep.objects_nr = proposed_objects_nr;\n-\t\t\tnext = base;\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"step:objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\tnext = next->base_midx;\n \t\t}\n \n \t\tif (m == next) {\n \t\t\tstep.type = MIDX_COMPACTION_STEP_COPY;\n \t\t\tstep.u.copy = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"copy\");\n \t\t} else {\n \t\t\tstep.type = MIDX_COMPACTION_STEP_COMPACT;\n \t\t\tstep.u.compact.from = next;\n \t\t\tstep.u.compact.to = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"to\", midx_get_checksum_hex(m));\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"compact\");\n \t\t}\n \n \t\tm = next->base_midx;\n-\n \t\tsteps[steps_nr++] = step;\n+\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n \t}\n \n+\ttrace2_region_leave(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n out:\n \t*steps_p = steps;\n \t*steps_nr_p = steps_nr;\n \n+\ttrace2_region_leave(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n \treturn ret;\n }\n \n@@ -801,6 +894,7 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \tstruct midx_compaction_step *steps = NULL;\n \tstruct strbuf lock_name = STRBUF_INIT;\n \tstruct lock_file lf;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tsize_t steps_nr = 0;\n \tsize_t i;\n \tint ret = 0;\n@@ -846,10 +940,13 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n \t\t\t (uintmax_t)i);\n \t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t\tstrvec_push(&keep_hashes, step->csum);\n \t}\n \n \tcommit_lock_file(&lf);\n \n+\tclear_incremental_midx_files(opts->existing->repo, &keep_hashes);\n+\n done:\n \tstrbuf_release(&lock_name);\n \tfree(steps);\ndiff --git a/repack.c b/repack.c\nindex 2ee6b51420a..be2cc0e9d44 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -55,14 +55,18 @@ void pack_objects_args_release(struct pack_objects_args *args)\n }\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name)\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct odb_source *source = repo->objects->sources;\n \tstruct multi_pack_index *m = get_multi_pack_index(source);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && source->local && midx_contains_pack(m, buf.buf))\n+\tif (m && source->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n+\t\tif (!wrote_incremental_midx)\n+\t\t\tclear_incremental_midx_files(repo, NULL);\n+\t}\n \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n \tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n@@ -252,23 +256,26 @@ void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \n static void remove_redundant_packs_1(struct repository *repo,\n \t\t\t\t struct string_list *packs,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, packs) {\n \t\tif (!existing_pack_is_marked_for_deletion(item))\n \t\t\tcontinue;\n-\t\trepack_remove_redundant_pack(repo, packdir, item->string);\n+\t\trepack_remove_redundant_pack(repo, packdir, item->string,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n }\n \n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs_1(existing->repo, &existing->non_kept_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n \tremove_redundant_packs_1(existing->repo, &existing->cruft_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n }\n \n void existing_packs_release(struct existing_packs *existing)\ndiff --git a/repack.h b/repack.h\nindex 831ccfb1c6c..d2876f569a6 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -34,7 +34,8 @@ void prepare_pack_objects(struct child_process *cmd,\n void pack_objects_args_release(struct pack_objects_args *args);\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name);\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx);\n \n struct write_pack_opts {\n \tstruct pack_objects_args *po_args;\n@@ -84,7 +85,8 @@ void existing_packs_retain_cruft(struct existing_packs *existing,\n void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t struct string_list *names);\n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void existing_packs_release(struct existing_packs *existing);\n \n struct generated_pack;\n@@ -129,7 +131,8 @@ struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry);\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\ndiff --git a/t/meson.build b/t/meson.build\nindex 2421220917a..4f7c7d5691d 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -941,6 +941,7 @@ integration_tests = [\n 't7702-repack-cyclic-alternate.sh',\n 't7703-repack-geometric.sh',\n 't7704-repack-cruft.sh',\n+ 't7705-repack-incremental-midx.sh',\n 't7800-difftool.sh',\n 't7810-grep.sh',\n 't7811-grep-open.sh',\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nnew file mode 100755\nindex 00000000000..f81c2c67060\n--- /dev/null\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -0,0 +1,436 @@\n+#!/bin/sh\n+\n+test_description='git repack --write-midx=incremental'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+# incrementally_repack N\n+#\n+# Make \"N\" new commits, each stored in their own pack, and then repacked\n+# with the --write-midx=incremental strategy.\n+incrementally_repack () {\n+\tfor i in $(test_seq 1 \"$1\")\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify || return 1\n+\tdone\n+}\n+\n+# Create packs with geometrically increasing sizes so that they\n+# satisfy the geometric progression and survive a --geometric=2\n+# repack without being rolled up. Creates 3 packs containing 1,\n+# 2, and 6 commits (3, 6, and 18 objects) respectively.\n+create_geometric_packs () {\n+\ttest_commit \"small\" &&\n+\tgit repack -d &&\n+\n+\ttest_commit_bulk --message=\"medium\" 2 &&\n+\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t--write-bitmap-index\n+}\n+\n+# create_layer \n+#\n+# Creates a new MIDX layer with the contents of \"test_commit_bulk $@\".\n+create_layer () {\n+\ttest_commit_bulk \"$@\" &&\n+\n+\tgit multi-pack-index write --incremental --bitmap\n+}\n+\n+# create_layers\n+#\n+# Reads lines of \" \" from stdin and creates a new MIDX\n+# layer for each line. See create_layer above for more.\n+create_layers () {\n+\twhile read msg nr\n+\tdo\n+\t\tcreate_layer --message=\"$msg\" \"$nr\" || return 1\n+\tdone\n+}\n+\n+test_expect_success '--write-midx=incremental requires --geometric' '\n+\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+\n+\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+'\n+\n+test_expect_success 'below layer threshold, tip packs excluded' '\n+\tgit init below-layer-threshold-tip-packs-excluded &&\n+\t(\n+\t\tcd below-layer-threshold-tip-packs-excluded &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 4 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create 3 packs forming a geometric progression by\n+\t\t# object count such that they are unmodified by the\n+\t\t# initial repack. The MIDX chain thusly contains a\n+\t\t# single layer with three packs.\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a new commit. Since the layer threshold is\n+\t\t# unmet, a new MIDX layer is added on top of the\n+\t\t# existing one.\n+\t\ttest_commit extra &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs repacked' '\n+\tgit init above-layer-threshold-tip-packs-repacked &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-repacked &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Same setup, but with the layer threshold set to 2.\n+\t\t# Since the tip MIDX layer meets that threshold, its\n+\t\t# packs are considered repack candidates.\n+\t\tcreate_geometric_packs &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Perturb the existing progression such that it is\n+\t\t# rolled up into a single new pack, invalidating the\n+\t\t# existing MIDX layer and replacing it with a new one.\n+\t\ttest_commit extra &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip layer preserved' '\n+\tgit init above-layer-threshold-tip-layer-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-layer-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\ttest_commit_bulk --message=\"medium\" 2 &&\n+\t\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Create objects to form a pack satisfying the geometric\n+\t\t# progression (thus preserving the tip layer), but not\n+\t\t# so large that it meets the layer merging condition.\n+\t\ttest_commit_bulk --message=\"small\" 1 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\ttest_line_count = 1 packs.new &&\n+\t\ttest_line_count = 3 packs.after &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs preserved' '\n+\tgit init above-layer-threshold-tip-packs-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Same setup as above, but this time the new objects do\n+\t\t# not satisfy the new layer merging condition, resulting\n+\t\t# in a new tip layer.\n+\t\ttest_commit_bulk --message=\"huge\" 18 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'new tip absorbs multiple layers' '\n+\tgit init new-tip-absorbs-multiple-layers &&\n+\t(\n+\t\tcd new-tip-absorbs-multiple-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a 4-layer chain where each layer is too small to\n+\t\t# absorb the one below it. The sizes must satisfy L(n) <\n+\t\t# L(n-1)/2 for each adjacent pair:\n+\t\t#\n+\t\t# L0 (oldest): 75 obj (25 commits)\n+\t\t# L1: 21 obj (7 commits, 21 < 75/2)\n+\t\t# L2: 9 obj (3 commits, 9 < 21/2)\n+\t\t# L3 (tip): 3 obj (1 commit, 3 < 9/2)\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tL0 25\n+\t\tL1 7\n+\t\tL2 3\n+\t\tL3 1\n+\t\tEOF\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Now add a new commit. The merging condition is\n+\t\t# satisfied between L3-L1, but violated at L0, which is\n+\t\t# too large relative to the accumulated size.\n+\t\t#\n+\t\t# As a result, the chain shrinks from 4 to 2 layers.\n+\t\ttest_commit new &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'compaction of older layers' '\n+\tgit init compaction-of-older-layers &&\n+\t(\n+\t\tcd compaction-of-older-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a chain with two small layers at the bottom\n+\t\t# and a larger barrier layer on top, producing a\n+\t\t# chain that violates the compaction invariant, since\n+\t\t# the two small layers would normally have been merged.\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 2\n+\t\ttwo 4\n+\t\tbarrier 54\n+\t\tEOF\n+\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running an incremental repack compacts the two\n+\t\t# small layers at the bottom of the chain as a\n+\t\t# separate step in the compaction plan.\n+\t\ttest_commit another &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'geometric rollup with surviving tip packs' '\n+\tgit init geometric-rollup-with-surviving-tip-packs &&\n+\t(\n+\t\tcd geometric-rollup-with-surviving-tip-packs &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create a pack large enough to anchor the geometric\n+\t\t# progression when small packs are added alongside it.\n+\t\tcreate_layer --message=\"big\" 5 &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a small number of objects such that the\n+\t\t# progression is unbothered. Note that the existing pack\n+\t\t# is considered a repack candidate as the new layer\n+\t\t# threshold is set to 1.\n+\t\ttest_commit small-1 &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before\n+\t)\n+'\n+\n+test_expect_success 'kept packs are excluded from repack' '\n+\tgit init kept-packs-excluded-from-repack &&\n+\t(\n+\t\tcd kept-packs-excluded-from-repack &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create two equal-sized packs, marking one as kept.\n+\t\tfor i in A B\n+\t\tdo\n+\t\t\ttest_commit \"$i\" && git repack -d || return 1\n+\t\tdone &&\n+\n+\t\tkeep=$(ls $packdir/pack-*.idx | head -n 1) &&\n+\t\ttouch \"${keep%.idx}.keep\" &&\n+\n+\t\t# The kept pack is excluded as a repacking candidate\n+\t\t# entirely, so no rollup occurs as there is only one\n+\t\t# non-kept pack. A new MIDX layer is written containing\n+\t\t# that pack.\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count = 1 actual.packs &&\n+\t\ttest_grep ! \"$keep\" actual.packs &&\n+\n+\t\tgit multi-pack-index verify &&\n+\n+\t\t# All objects (from both kept and non-kept packs)\n+\t\t# must still be accessible.\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'incremental MIDX with --max-pack-size' '\n+\tgit init incremental-midx-with--max-pack-size &&\n+\t(\n+\t\tcd incremental-midx-with--max-pack-size &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\t# Now add enough data that a small --max-pack-size will\n+\t\t# cause pack-objects to split its output. Create objects\n+\t\t# large enough to fill multiple packs.\n+\t\ttest-tool genrandom foo 1M >big1 &&\n+\t\ttest-tool genrandom bar 1M >big2 &&\n+\t\tgit add big1 big2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -a -m \"big blobs\" &&\n+\t\tgit repack -d &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index --max-pack-size=1M &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count -gt 1 actual.packs &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'noop repack preserves valid MIDX chain' '\n+\tgit init noop-repack-preserves-valid-midx-chain &&\n+\t(\n+\t\tcd noop-repack-preserves-valid-midx-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running again with no new objects should not break\n+\t\t# the MIDX chain. It produces \"Nothing new to pack.\"\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_cmp $midx_chain.before $midx_chain &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'repack -ad removes stale incremental chain' '\n+\tgit init repack--ad-removes-stale-incremental-chain &&\n+\t(\n+\t\tcd repack--ad-removes-stale-incremental-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 1\n+\t\ttwo 1\n+\t\tEOF\n+\n+\t\ttest_path_is_file $midx_chain &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\ttest_path_is_missing $packdir/multi-pack-index &&\n+\t\ttest_dir_is_empty $midxdir\n+\t)\n+'\n+\n+test_done\n-- \n2.53.0.185.g29bc4dff628\n\n"},{"id":"537058","messageId":"d87b360be1442663f3c05687b47d4b5a76571fd3.1771978829.git.me@ttaylorr.com","threadId":"65071","inReplyTo":"cover.1771978829.git.me@ttaylorr.com","subject":"[RFC PATCH 14/14] repack: allow `--write-midx=incremental` without `--geometric`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:22:07Z","receivedAt":"2026-02-25T00:22:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Previously, `--write-midx=incremental` required `--geometric` and would\ndie() without it. Relax this restriction so that incremental MIDX\nrepacking can be used independently.\n\nWithout `--geometric`, the behavior is append-only: a single new MIDX\nlayer is created containing whatever packs were written by the repack\nand appended to the existing chain (or a new chain is started). Existing\nlayers are preserved as-is with no compaction or merging.\n\nImplement this via a new repack_make_midx_append_plan() that builds a\nplan consisting of a WRITE step for the freshly written packs followed\nby COPY steps for every existing MIDX layer. The existing compaction\nplan (repack_make_midx_compaction_plan) is used only when `--geometric`\nis active.\n\nUpdate the documentation to describe the behavior with and without\n`--geometric`, and replace the test that enforced the old restriction\nwith one exercising append-only incremental MIDX repacking.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-repack.adoc | 19 +++++----\n builtin/repack.c | 3 --\n repack-midx.c | 62 ++++++++++++++++++++++++++++--\n t/t7705-repack-incremental-midx.sh | 35 ++++++++++++++---\n 4 files changed, 101 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 27a99cc46f4..72c42015e23 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -263,14 +263,19 @@ linkgit:git-multi-pack-index[1]).\n \n \t`incremental`;;\n \t\tWrite an incremental MIDX chain instead of a single\n-\t\tflat MIDX. This mode requires `--geometric`.\n+\t\tflat MIDX.\n +\n-The incremental mode maintains a chain of MIDX layers that is compacted\n-over time using a geometric merging strategy. Each repack creates a new\n-tip layer containing the newly written pack(s). Adjacent layers are then\n-merged whenever the newer layer's object count exceeds\n-`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n-that do not meet this condition are retained as-is.\n+Without `--geometric`, a new MIDX layer is appended to the existing\n+chain (or a new chain is started) containing whatever packs were written\n+by the repack. Existing layers are preserved as-is.\n++\n+When combined with `--geometric`, the incremental mode maintains a chain\n+of MIDX layers that is compacted over time using a geometric merging\n+strategy. Each repack creates a new tip layer containing the newly\n+written pack(s). Adjacent layers are then merged whenever the newer\n+layer's object count exceeds `1/repack.midxSplitFactor` of the next\n+deeper layer's count. Layers that do not meet this condition are\n+retained as-is.\n +\n The result is that newer (tip) layers tend to contain many small packs\n with relatively few objects, while older (deeper) layers contain fewer,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 9e070f35868..8e187322fe4 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -263,9 +263,6 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n-\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n-\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n-\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 5da67f854c7..3ab64937c2e 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -556,6 +556,58 @@ static int midx_compaction_step_exec(struct midx_compaction_step *step,\n \t}\n }\n \n+/*\n+ * Build an append-only MIDX plan: a single WRITE step for the freshly\n+ * written packs, plus COPY steps for every existing layer. No\n+ * compaction or merging is performed.\n+ */\n+static void repack_make_midx_append_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step *step;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tif (opts->names->nr) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_WRITE;\n+\t\tstring_list_init_nodup(&step->u.write);\n+\n+\t\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\t\tstrbuf_addf(&buf, \"pack-%s.idx\",\n+\t\t\t\t opts->names->items[i].string);\n+\t\t\tstring_list_append(&step->u.write,\n+\t\t\t\t\t strbuf_detach(&buf, NULL));\n+\t\t}\n+\t}\n+\n+\tfor (; m; m = m->base_midx) {\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_COPY;\n+\t\tstep->u.copy = m;\n+\t\tstep->objects_nr = m->num_objects;\n+\t}\n+\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+}\n+\n static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t\t\t struct midx_compaction_step **steps_p,\n \t\t\t\t\t size_t *steps_nr_p)\n@@ -911,9 +963,13 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \t}\n \n-\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n-\t\tret = error(_(\"unable to generate compaction plan\"));\n-\t\tgoto done;\n+\tif (opts->geometry->split_factor) {\n+\t\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\t\tgoto done;\n+\t\t}\n+\t} else {\n+\t\trepack_make_midx_append_plan(opts, &steps, &steps_nr);\n \t}\n \n \tfor (i = 0; i < steps_nr; i++) {\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nindex f81c2c67060..562554e69b4 100755\n--- a/t/t7705-repack-incremental-midx.sh\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -63,10 +63,36 @@ create_layers () {\n \tdone\n }\n \n-test_expect_success '--write-midx=incremental requires --geometric' '\n-\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+test_expect_success '--write-midx=incremental without --geometric' '\n+\tgit init incremental-without-geometric &&\n+\t(\n+\t\tcd incremental-without-geometric &&\n \n-\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+\t\tgit config maintenance.auto false &&\n+\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# A second repack appends a new layer without\n+\t\t# disturbing the existing one.\n+\t\ttest_commit third &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\t\thead -n 1 $midx_chain.before >expect &&\n+\t\thead -n 1 $midx_chain >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit fsck\n+\t)\n '\n \n test_expect_success 'below layer threshold, tip packs excluded' '\n@@ -334,8 +360,7 @@ test_expect_success 'kept packs are excluded from repack' '\n \t\t# entirely, so no rollup occurs as there is only one\n \t\t# non-kept pack. A new MIDX layer is written containing\n \t\t# that pack.\n-\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n-\t\t\t--write-bitmap-index &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental &&\n \n \t\ttest-tool read-midx $objdir >actual &&\n \t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n-- \n2.53.0.185.g29bc4dff628\n"},{"id":"537059","messageId":"aZ5BANxASSKHdqBu@nand.local","threadId":"65071","inReplyTo":"df37959d2933d84ec61b9ba570c87f9e5213c735.1771978829.git.me@ttaylorr.com","subject":"Re: [RFC PATCH 06/14] repack: track the ODB source via existing_packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-25T00:23:28Z","receivedAt":"2026-02-25T00:23:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Feb 24, 2026 at 07:20:46PM -0500, Taylor Blau wrote:\n> Store the ODB source in the `existing_packs` struct and use that in\n> place of the raw `repo->objects->sources` access within `cmd_repack()`.\n\nI have no idea why my scripts sent this patch twice, but there are two\ncopies of [06/14], which are identical. I'll make sure to just send one\nof these in the non-RFC version of this series ;-).\n\nThanks,\nTaylor\n"},{"id":"537224","messageId":"xmqqldgf1c65.fsf@gitster.g","threadId":"65071","inReplyTo":"d64a799afd620363c1940d7c2e634e78ea553cb6.1771978829.git.me@ttaylorr.com","subject":"Re: [RFC PATCH 01/14] midx: use `string_list` for retained MIDX files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-02-26T20:29:22Z","receivedAt":"2026-02-26T20:29:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\n> build a list of filenames to keep while pruning stale MIDX files. Today\n> they hand-roll an array instead of using a `string_list`, thus requiring\n> us to pass an additional length parameter, and makes lookups linear.\n>\n> Replace the bare array with a `string_list` which can be passed around\n> as a single parameter. Though it improves lookup performance, the\n> difference is likely immeasurable given how small the keep_hashes array\n> typically is.\n\nAnd if it the lookup performance turns out to be an issue, we can\nswitch to strmap or something more appropriate.\n\n>\n> Signed-off-by: Taylor Blau \n> ---\n> midx.c | 56 ++++++++++++++++++++++----------------------------------\n> 1 file changed, 22 insertions(+), 34 deletions(-)\n>\n> diff --git a/midx.c b/midx.c\n> index c1b9658240d..c5e3553e2bb 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -755,8 +755,7 @@ int midx_checksum_valid(struct multi_pack_index *m)\n> }\n> \n> struct clear_midx_data {\n> -\tchar **keep;\n> -\tuint32_t keep_nr;\n> +\tstruct string_list keep;\n> \tconst char *ext;\n> };\n> \n> @@ -764,15 +763,12 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n> \t\t\t\tconst char *file_name, void *_data)\n> {\n> \tstruct clear_midx_data *data = _data;\n> -\tuint32_t i;\n> \n> \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n> \t ends_with(file_name, data->ext)))\n> \t\treturn;\n> -\tfor (i = 0; i < data->keep_nr; i++) {\n> -\t\tif (!strcmp(data->keep[i], file_name))\n> -\t\t\treturn;\n> -\t}\n> +\tif (string_list_has_string(&data->keep, file_name))\n> +\t\treturn;\n> \tif (unlink(full_path))\n> \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n> }\n> @@ -780,48 +776,40 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n> void clear_midx_files_ext(struct odb_source *source, const char *ext,\n> \t\t\t const char *keep_hash)\n> {\n> -\tstruct clear_midx_data data;\n> -\tmemset(&data, 0, sizeof(struct clear_midx_data));\n> -\n> -\tif (keep_hash) {\n> -\t\tALLOC_ARRAY(data.keep, 1);\n> -\n> -\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n> -\t\tdata.keep_nr = 1;\n> -\t}\n> -\tdata.ext = ext;\n> -\n> -\tfor_each_file_in_pack_dir(source->path,\n> -\t\t\t\t clear_midx_file_ext,\n> -\t\t\t\t &data);\n> +\tstruct clear_midx_data data = {\n> +\t\t.keep = STRING_LIST_INIT_NODUP,\n> +\t\t.ext = ext,\n> +\t};\n> \n> \tif (keep_hash)\n> -\t\tfree(data.keep[0]);\n> -\tfree(data.keep);\n> +\t\tstring_list_insert(&data.keep, xstrfmt(\"multi-pack-index-%s.%s\",\n> +\t\t\t\t\t\t keep_hash, ext));\n> +\n> +\tfor_each_file_in_pack_dir(source->path, clear_midx_file_ext, &data);\n> +\n> +\tstring_list_clear(&data.keep, 0);\n> }\n> \n> void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n> \t\t\t\t char **keep_hashes,\n> \t\t\t\t uint32_t hashes_nr)\n> {\n> -\tstruct clear_midx_data data;\n> +\tstruct clear_midx_data data = {\n> +\t\t.keep = STRING_LIST_INIT_NODUP,\n> +\t\t.ext = ext,\n> +\t};\n> \tuint32_t i;\n> \n> -\tmemset(&data, 0, sizeof(struct clear_midx_data));\n> -\n> -\tALLOC_ARRAY(data.keep, hashes_nr);\n> \tfor (i = 0; i < hashes_nr; i++)\n> -\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n> -\t\t\t\t ext);\n> -\tdata.keep_nr = hashes_nr;\n> -\tdata.ext = ext;\n> +\t\tstring_list_append(&data.keep,\n> +\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n> +\t\t\t\t\t keep_hashes[i], ext));\n> +\tstring_list_sort(&data.keep);\n> \n> \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n> \t\t\t\t clear_midx_file_ext, &data);\n> \n> -\tfor (i = 0; i < hashes_nr; i++)\n> -\t\tfree(data.keep[i]);\n> -\tfree(data.keep);\n> +\tstring_list_clear(&data.keep, 0);\n> }\n> \n> void clear_midx_file(struct repository *r)\n"},{"id":"537225","messageId":"xmqqh5r31byc.fsf@gitster.g","threadId":"65071","inReplyTo":"50efbbb0fe8d897d7c4cd51489af4cb4c4c49d02.1771978829.git.me@ttaylorr.com","subject":"Re: [RFC PATCH 02/14] strvec: introduce `strvec_init_alloc()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-02-26T20:34:03Z","receivedAt":"2026-02-26T20:34:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> When the caller knows upfront how many elements will be pushed onto a\n> `strvec`, it is useful to pre-allocate enough space in the array to fit\n> that many elements (and one additional slot to store NULL, indicating\n> the end of the list.)\n>\n> Introduce `strvec_init_alloc()`, which allocates the backing array large\n> enough to hold `alloc` elements and the termination marker without\n> further reallocation.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> strvec.c | 7 +++++++\n> strvec.h | 5 +++++\n> 2 files changed, 12 insertions(+)\n>\n> diff --git a/strvec.c b/strvec.c\n> index f8de79f5579..f7f32a53b56 100644\n> --- a/strvec.c\n> +++ b/strvec.c\n> @@ -10,6 +10,13 @@ void strvec_init(struct strvec *array)\n> \tmemcpy(array, &blank, sizeof(*array));\n> }\n> \n> +void strvec_init_alloc(struct strvec *array, size_t alloc)\n> +{\n> +\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n> +\tarray->nr = 0;\n> +\tarray->alloc = alloc + 1;\n> +}\n\nIt is not satisifying that strvec_init() does *not* become a thin\nwrapper around this that says \"my initial allocation is for zero\nelements\", but that cannot be done easily as a strvec that begins as\nan empty one has a small optimization to avoid one-slot allocation\nonly to store NULL. So, ... OK.\n\n"},{"id":"537231","messageId":"xmqq4in3xlvb.fsf@gitster.g","threadId":"65071","inReplyTo":"xmqqh5r31byc.fsf@gitster.g","subject":"Re: [RFC PATCH 02/14] strvec: introduce `strvec_init_alloc()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-02-26T20:58:48Z","receivedAt":"2026-02-26T20:58:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Taylor Blau writes:\n>\n>> When the caller knows upfront how many elements will be pushed onto a\n>> `strvec`, it is useful to pre-allocate enough space in the array to fit\n>> that many elements (and one additional slot to store NULL, indicating\n>> the end of the list.)\n>>\n>> Introduce `strvec_init_alloc()`, which allocates the backing array large\n>> enough to hold `alloc` elements and the termination marker without\n>> further reallocation.\n>>\n>> Signed-off-by: Taylor Blau \n>> ---\n>> strvec.c | 7 +++++++\n>> strvec.h | 5 +++++\n>> 2 files changed, 12 insertions(+)\n>>\n>> diff --git a/strvec.c b/strvec.c\n>> index f8de79f5579..f7f32a53b56 100644\n>> --- a/strvec.c\n>> +++ b/strvec.c\n>> @@ -10,6 +10,13 @@ void strvec_init(struct strvec *array)\n>> \tmemcpy(array, &blank, sizeof(*array));\n>> }\n>> \n>> +void strvec_init_alloc(struct strvec *array, size_t alloc)\n>> +{\n>> +\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n>> +\tarray->nr = 0;\n>> +\tarray->alloc = alloc + 1;\n>> +}\n>\n> It is not satisifying that strvec_init() does *not* become a thin\n> wrapper around this that says \"my initial allocation is for zero\n> elements\", but that cannot be done easily as a strvec that begins as\n> an empty one has a small optimization to avoid one-slot allocation\n> only to store NULL. So, ... OK.\n\nActually, we should do the same optimization if a caller explicitly\nasks\n\n\tstrvec_init_alloc(&array, 0);\n\nSo perhaps we could do this if we wanted to encapsulate the tricky\nbits in a single place for maintainability.\n\n strvec.c | 15 +++++++++++++--\n 1 file changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git c/strvec.c w/strvec.c\nindex f8de79f557..cbe72e9411 100644\n--- c/strvec.c\n+++ w/strvec.c\n@@ -4,10 +4,21 @@\n \n const char *empty_strvec[] = { NULL };\n \n+void strvec_init_alloc(struct strvec *array, size_t alloc)\n+{\n+\tif (!alloc) {\n+\t\tstruct strvec blank = STRVEC_INIT;\n+\t\tmemcpy(array, &blank, sizeof(*array));\n+\t} else {\n+\t\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n+\t\tarray->nr = 0;\n+\t\tarray->alloc = alloc + 1;\n+\t}\n+}\n+\n void strvec_init(struct strvec *array)\n {\n-\tstruct strvec blank = STRVEC_INIT;\n-\tmemcpy(array, &blank, sizeof(*array));\n+\tstrvec_init(array, 0);\n }\n \n void strvec_push_nodup(struct strvec *array, char *value)\n"},{"id":"537265","messageId":"aaEJVDoUyL/QbY9O@nand.local","threadId":"65071","inReplyTo":"xmqqldgf1c65.fsf@gitster.g","subject":"Re: [RFC PATCH 01/14] midx: use `string_list` for retained MIDX files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-27T03:02:44Z","receivedAt":"2026-02-27T03:02:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 26, 2026 at 12:29:22PM -0800, Junio C Hamano wrote:\n> > Replace the bare array with a `string_list` which can be passed around\n> > as a single parameter. Though it improves lookup performance, the\n> > difference is likely immeasurable given how small the keep_hashes array\n> > typically is.\n>\n> And if it the lookup performance turns out to be an issue, we can\n> switch to strmap or something more appropriate.\n\nTrue. In practice these should have at most tens of entries, so I\nsuspect the string_list will be completely fine.\n\nIf we are cleaning up so many stale MIDX files that we need a more\nefficient data structure just to keep track of those files, we likely\nhave far greater problems to worry about ;-).\n\nThanks,\nTaylor\n"},{"id":"537266","messageId":"aaEKYGWi7nd1VLJc@nand.local","threadId":"65071","inReplyTo":"xmqq4in3xlvb.fsf@gitster.g","subject":"Re: [RFC PATCH 02/14] strvec: introduce `strvec_init_alloc()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-27T03:07:12Z","receivedAt":"2026-02-27T03:07:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 26, 2026 at 12:58:48PM -0800, Junio C Hamano wrote:\n> > It is not satisifying that strvec_init() does *not* become a thin\n> > wrapper around this that says \"my initial allocation is for zero\n> > elements\", but that cannot be done easily as a strvec that begins as\n> > an empty one has a small optimization to avoid one-slot allocation\n> > only to store NULL. So, ... OK.\n>\n> Actually, we should do the same optimization if a caller explicitly\n> asks\n>\n> \tstrvec_init_alloc(&array, 0);\n>\n> So perhaps we could do this if we wanted to encapsulate the tricky\n> bits in a single place for maintainability.\n\nAh, that is a very satisfying change. I like it and squashed it into my\nseries with your:\n\n Helped-by: Junio C Hamano \n\n> void strvec_init(struct strvec *array)\n> {\n> -\tstruct strvec blank = STRVEC_INIT;\n> -\tmemcpy(array, &blank, sizeof(*array));\n> +\tstrvec_init(array, 0);\n\nI suspect you meant `strvec_init_alloc()` here and not `strvec_init()`,\nbut I applied the change and adjusted the typo before squashing it in.\n\nThanks,\nTaylor\n"}]}